Accurate Interpolation for Scattered Data through Hierarchical Residual Refinement
Authors: Shizhe Ding
Boyang Xia
Dongbo Bu
Venue: NeurIPS 2023 poster (2023)
Keywords: Interpolation algorithm
scattered data
deep learning
residual learning
TL;DR: We propose HINT, a novel hierarchical residual refining framework for scattered data interpolation.

Abstract: Accurate interpolation algorithms are highly desired in various theoretical and engineering scenarios. Unlike the traditional numerical algorithms that have exact zero-residual constraints on observed points, the neural network-based interpolation methods exhibit non-zero residuals at these points. These residuals, which provide observations of an underlying residual function, can guide predicting interpolation functions, but have not been exploited by the existing approaches. To fill this gap, we propose Hierarchical INTerpolation Network (HINT), which utilizes the residuals on observed points to guide target function estimation in a hierarchical fashion. HINT consists of several sequentially arranged lightweight interpolation blocks. The first interpolation block estimates the main component of the target function, while subsequent blocks predict the residual components using observed points residuals of the preceding blocks. The main component and residual components are accumulated to form the final interpolation results. Furthermore, under the assumption that finer residual prediction requires a more focused attention range on observed points, we utilize hierarchical local constraints in correlation modeling between observed and target points. Extensive experiments demonstrate that HINT outperforms existing interpolation algorithms significantly in terms of interpolation accuracy across a wide variety of datasets, which underscores its potential for practical scenarios.

Accurate Interpolation for Scattered Data through Hierarchical Residual Refinement
Shizhe Ding! Boyang Xial Dongbo 12,3.
Key Laboratory of Intelligent Information Processing,
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China University of Chinese Academy of Sciences, Beijing, China
3 Central China Institute for Artificial Intelligence Technologies, Zhengzhou, China Edingshizhe19e 19s, xiaboyang20s, dbu) aict.ac.........................
Abstract
Accurate interpolation algorithms highly desired in various theoretical and engineering scenarios. Unlike the traditional numerical algorithms that have exact zero-residual constraints on observed points, the neural network-based interpolation methods exhibit non-zero residuals at these points. These residuals, which provide observations of an underlying residual function, can guide predicting interpolation functions, but have not been exploited by the existing approaches. To fill this gap, we propose Hierarchical INTerpolation Network (HINT), which utilizes the resid- uals on observed points to guide target function estimation in a hierarchical fashion. HINT consists of several sequentially arranged lightweight interpolation blocks. The first interpolation block estimates the main component of the target function, while subsequent blocks predict the residual components using observed points residuals of the preceding blocks. The main component and residual components are accumulated to form the final interpolation results. Furthermore, under the assumption that finer residual prediction requires more focused attention range on observed points, we utilize hierarchical local constraints correlation modeling between observed and target points. Extensive experiments demonstrate that HINT outperforms existing interpolation algorithms significantly terms of interpolation accuracy across wide variety of datasets, which underscores its potential for practical scenarios.
Introduction
Scattered data interpolation aims to estimate the unknown values of a continuous function at some target points based finite set of observed data points (IT]. This technique plays significant role in extensive theoretical and engineering domains, e.g. partial differential equations (PDEs) solving, physical analysis and thermal engineering, etc 77 20 28 For instance, in thermal engineering, scattered data interpolation is employed to estimate the environmental temperature distribution of buildings (28) and electronic devices [2] based on temperature sensor readings obtained dispersed points. In many applications, the function distributions are complex and the scattered points are non-uniformly distributed, which poses significant challenges for accurate interpolation algorithms.
Traditional scattered data interpolation algorithms follow a well-established pipeline: firstly a suitable set of basis functions is prepared (e.g. polynomial, Spline and radial basis functions), then possible target function is generated by constructing a linear combination of these basis functions [[]] Basis function are selected depending application scenarios, which highly relies domain knowledge and human experience. This makes traditional algorithms hardly able to generalize between problems
Cooresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

K--66
Hierarchical local
MO
TO
MAIN BLOCK
RESIDUAL BLOCK]
RESIDUAL BLOCK
FT
Interpolation task
Interpolation result
component
component
Residual component
Figure Overview of our HINT architecture. The architecture comprises multiple interpolation blocks, including a main block and several residue blocks. The main block directly estimates the primary components of the target function. Subsequent residue blocks refine the estimation based on the residuals the observed points computed from previous blocks. As the blocks that output finer residuals place more emphasis on capturing fine- tinegrained details of the function, we incorporate hierarchical local constraints on valid range of observed points correlated with target points on these interpolation blocks.
and imposes much burden of searching and tuning. Thus, when targeting slightly more complex functions. these methods usually present limited interpolation accuracy.
has given rise to a plethora of neural neteork-based interpolation algorithms. Notable examples of such models encompass Neural Processes 8 19 18 Tran-Taasfrrmer 22. and NIERT 4 These models leverage neural network modules capture the correlation between observed points and target points, and estimate the target point values based on the correlations and the observed point values. In contrast to traditional methods that rely hand-crafted and rigid basis functions, these neural interpolators can adaptively discern the underlying distribution of the target function from a function set or scattered data set, consequently achieving superior generalization performance.
Recently, the evolution of deep learning technology [7 [5] [6 12
Different from the traditional interpolation algorithms that have exact zero-residual constraints on observed points (target function curve passes through the points) [T], the neural networks-baat approaches exhibit non-zero residuals at these points. We assume these residuals represent a type of residual function, which maintains residual information sharing between observed points and target points. Although existing methods utilize the correlations between observed points and target points conditions, few methods explore the of residual information sharing between them.
To fill this gap, we introduce Hierarchical INTerpolation network (HINT) utilize observed points residuals to guide target function estimation in hierarchical fashion (depicted in Figure [D. Our key insight is that the target function estimation can be decomposed into a main function component and several residual function components. With this in mind, design the framework of HINT hierarchy of lightweight interpolation blocks, among which the first block (main block) estimates the main component of the target function and subsequent blocks (residual blocks) predict the residual components in progressively refining manner. In each residual block, to utilize residual information sharing between observed and target points, predict the residuals of target points from accessible observed point residuals and via correlation modeling between two point domains. Finally, the main component and residual components of target points accumulated form interpolation results.
Furthermore, we consider that finer residual prediction for target point requires more focused attention its spatially neighboring observed points. Thus, we incorporate hierarchical local

---

constraints that progressively narrows down the valid range of observed points correlated with target points, to reduce the noise of correlation modelling during residual prediction.
We summarize our contributions as follows: 1) We propose a novel hierarchical residual refining framework, HINT,, for scattered point interpolation. We introduce series of residual blocks that utilize the residual information of observed points to guide target points prediction via correlation modeling in a coarse-to-iie fashion, to enhance interpolation accuracy. 2) We impose hierarchical local constraints on the range of observed points correlated with target points to conform to hierarchical residual scales, thereby further improving interpolation robustness. 3) We validate our framework on four scattered data interpolation task sets across diverse theoretical and application scenarios, which demonstrates our HINT surpasses existing state-of state-of-the-art thethrrrt interpolation methods in terms of accuracy in various tasks consistently.
2 Method
2.1 Problem Statement and Notations
Given n observed points with known values, denoted as O (e)s,::))==> and m target points with values to be estimated, represented by T {a; vn+m the objective of the interpolation task is to accurately estimate the values fz) for each target point æ T. where f is the underlying function determined by the information from the observed points in O.
Here, the position of point is indicated by x; E Rd, while the value of point is signified by Y: = fx;; E Rdy The function mapping is denoted by f : TRd= -> RRd, with de and dy representing the dimensions of point's position and value respectively. The function f originates from function distribution, which can be implicitly represented by set of scattered data.
For convenience, we denote the positions of the observed points as xo [xx ) the values of the
observed points as yo {u.)_.r and the positions of the target points as Øт = {2: N=R+-
2.2 Hierarchical Residual Refining Framework
The architecture of our proposed HINT is illustrated in Figure Our key insight is the target function estimation can be decomposed into series of function components at different scales, including a main function component and several residual components. In light of this view, we design a hierarchical interpolation architecture, consisting of series of L. neural network interpolation blocks. The first block (main block) serves as a coarse interpolator, estimating the main component of the target function, and subsequent L blocks (residual blocks) function as finer interpolators, estimating the residual parts of the target function.
The main block takes the positions and ground truth function values of the observed points, zo and yo. along with the positions of the target points Ir, as inputs. It outputs the coarse function value predictions at both the observed and target points yo (1) (1)y7"
No (0) yr (0) MainBlock(Fo. (oo. #T.
(I)
Then we can immediately obtain the estimated residuals of the target function at the observed points:
(1)
90 (0)
yo y0
After that, the following L 1 residual blocks predict the residual components of the target function in progressively refining manner. Specifically, each residual block takes the residual of the previous block's predicted function values at the observed points input and outputs the residual function values at both the observed and target points. The predicted residual function values at the observed points are used to compute the remaining residuals for the next residual block, while the predicted residual function values at the target points are employed to calculate the final function estimation values the target points:
(1) vo (1) vr Resid (xo, Vo ,T),
((+1)
(1) - 4o (1)
Vo
(4)
3

---

Prediction stage
si (1)
Mo (9)
8/7
MAC
Correlation modeling
Transformer Encoder
Embedding stage
No
MASK,
MASK,
Attention mask only retaining observed points' impact and integrating KNN graph constraint
(1) MO
30
Figure 2: The proposed interpolation block structure. Given specific interpolation task, each interpolation block starts by duplicating the observed points without values as masked observed points. Then, all scattered points are embedded. For the target points and masked observed points that are without values, a MASK token is used to fill the void. Next, a Transformer encoder is utilized encode these scattered points in a unified fashion and model their correlations. Finally, the predictions are generated. To eliminate the target points' influence on the observed points and target points themselves, we apply a masked self-attention mechanism that exclusively preserves the impact of observed points. Meanwhile, to uphold the locality constraint of the interpolation task, we incorporate K-nearest neighbor graph (KNN graph) constraint in the attention mask.
Ultimately, we obtain the final interpolation function prediction by accumulating the initial main components and subsequent residual components:
fT
yr
(5)
2.3 Transformer-bbsed Interpolation Block
Here we elaborate on the structure of our interpolation block illustrated in Figure which is embodied as the main block and residual blocks of HINT. The 1-th interpolation block takes the positions and values of the observed points and Vo and the positions of the target points ør as inputs, and predicts the values of the target function at both the observed points and target points, i.e., vo (!) and vi To facilitate the implementation of our hierarchical residual refining framework, propose novel encoder-only interpolation block building upon the basic structure of NIERT block [4], state-of-the-ee oh-the-art neural interpolation model. Please refer to Supplementary Material for detailed description of NIERT. Specifically, we make two key modifications to the original NIERT block: 1) set of "masked observed" points are introduced to solve the problem of inconsistent distribution between observed points and target points. 2) Local constraints are imposed on the range of observed points for target points in correlation modeling to accommodate different scales of the output function value. We introduce our interpolation block in detail as follows.
Embedding Stage. In the embedding phase, each point's position and value are separately embedded and concatenated to form the point's 's overall embedding. Following [4, we introduce a learnable token vector MASK, to embed the absent value for target points. Although this avoids structural discrepancy between observed points and target points, there still exists the problem of inconsistent distribution between them. Moreover, observed point residuals and target point residuals serve as the input and output of the block, respectively. Thus consistent distribution between them helps the model to compute residuals more easily. To this end, we introduce masked observed points, which are the replicates of observed points without values. This simulates the absence of value in target points for observed points, ensuring the distribution of them is consistent with target points. After this replication operation. the embeddings of three sets of points can be represented as:
(isear,(zs), Linear, (v) ()))))]
if i [1,7], i in 1, 2m],
>> observed points masked observed points (6)
(Lneer, MASK, y) (inear,((s.n),,MAASK MASK,),
i [2n + 1, 2n + m]
D target points,

---

where i - n denotes index shifting.
Correlation Modeling. Then the embeddings of the points fed into stack of P Transformer layers, to model the correlations between observed points and target points, and produce resultant points encodings. Through the p-th transformer layer, the encoding of the i-th point hy are transformed to N Following NIERT 4 we use attention mask mechanism on vanilla self-attention in each Transformer layer, to avoid the unexpected interference of target points on observed points. Thus, all three set of points are treated as queries { 2 while only observed points are treated as keys {ke M==1 and values {u, ) Ji=1. with these concepts referring to 27
Then, the unnormalized attention score ei can be computed via scaled dot product ciy 9.. Here, vaz dy the dimension of the query and key vectors. Based on prior knowledge that the observed points near the target points have a greater impact on the interpolation of target points than distant ones 47, we impose controllable local constraints on the attention mask to accommodate different scales of residuals. We retain only the influence of the K nearest observed points (under the Euclidean distance metric) of the i-th target point (we denote (i( here), masking out the attention scores of distant observed points (see Figure 2):
Egg E 1 -00,
N
otherwise.
where K a hyperparameter for local constraints. Then the attention score can be obtained by
softmax normalization a = Σد= exp 3 With reasonable K, this design helps lower down noise
exp (RP)
in correlation modelling, thus improves interpolation accuracy.
Estimating stage. For each point [1 + m], we estimate its value & through feeding its representation at the final Transformer layer into an MLP, E, MLP our (hr (h²"). These values are divided into three parts, including 1) predicted values of target function at target points yf (1) 2a+mm ( 2) predicted values of target function at "masked observed points yo
{ ) { We use yo (1) rather than yo (4) to compute residuals vo yo (1) as the input of next residual block. This because yo (1) have consistent distribution with y which helps the model conveniently utilize sharing residual information between two types of points in predict yrr (1) We use yo (!) for auxiliary loss function optimization.
and 3) auxiliary reconstruction of observed points y0 (1)
2.4 Hierarchical Local Constraints
We expand local inductive bias 7 of interpolation in a hierarchical way. We assume the main block focuses on relatively global information for estimates the primary components of the target function, while residual blocks focus on local information for it estimates the residual components with smaller scales. Considering this, we impose hierarchical local constraints on the interpolation blocks by utilizing K-nearest neighbors constraints with decreasing K. Specifically, we establish simple rule, setting two hyperparameters,
(0)) and K where (()) determines the K value for the main block, and K min represents the minimum value for the residual blocks. We recursively set the K value for the 1-th residual block as
K()) max ( (K((-1) /2, Kmin) (see Figure []. It is worth noting that the specific values for K (0) and Kmin depend on the characteristics of the dataset.
2.5 Loss Function
The loss function consists of two components: one for optimizing the interpolation error Lint, and the other as an auxiliary term Lauxs i.e.,
L Lint + XLaxx
(8)
Errorroo (1) (0) the where Lint. = Error(jrr, y/) the primary objective and Laux = + Error(soo auxiliary reconstruction term. The primary objective is to minimize the discrepancy between the predicted function values and the ground truth at the target points. And the auxiliary term serves to constrain the auxiliary reconstruction at observed points for each interpolation module, facilitating
5

---

the accurate reconstruction of both observed and target points. Here, The enror function used in this context can be either Mean Squared Error (MSE) Mean Absolute Error (MAE), depending on the precision metric requirements of the specific scenario. À is the loss balance hyperparameter.
3 Experimental Setting
We evaluate the interpolation performance of HINT on four representative datasets and compare with existing state-of-the-ar approaches. We describe these datasets, the baselines used for comparison and the experimental setup as follows. Please refer to the Supplementary Material for more details.
3.1 Datasets
The four datasets are Mathit-2D and Perlin for synthetic function interpolation, TFRD for temperature field reconstruction, and PTV for particle tracking velocimetry. We provide brief introduction to these four datasets, and further details and statistical information can be found in the Supp.
Theoretical Dataset I: Mathit-2D. Mathit-2D [4] contains interpolation tasks derived from two- dimensional synthetic symbolic functions. 512 data points are sampled within the square region [-1, and are randomly divided into observed and target points to form an interpolation task. The number of observed points randomly chosen from the range [10, 50].
Theoretical Dataset II: Perlin, To evaluate the interpolation performance on functions with fine details, we additionally constructed the Perlin dataset, which is sampled from random two-dimensional Perlin noise functions (23]. Each interpolation task involves randomly sampling from a Perlin function defined [-1, 1J3 resulting in 64 observed points and 192 target points.
Application Dataset I: TERD. TFRD [2] is a two-dimensional interpolation dataset for temperature field reconstruction. It contains three sub-datasets, ADlet, DSine, and HSink corresponding to different field boundary conditions. Each temperature field has 40,000 data points, with 37 points serving as observed points and the remaining points target points form interpolation tasks.
a scattered dataset for real-world two-dimensional particle tracking velocimetry, where each instance corresponds to a set of particles moving in velocity field extracted from the images of laminar jet experiment scenes [25]. Each instance contains 2,000 to 6,000 data points, from which we randomly select 512 points observed points, while the remaining points serve as target points for prediction.
Application Dataset II: PTV. PTV
3.2 Baselines
The baselines for evaluation include: 1) Conditional Neural Processes (CNPs) utilize MLPs to predict function distributions given observed points. 2) Attentive Neural Processes (ANPs)
[4] leverage attention mechanisms to enhance CNPs' performance. 3) Bootstrapping Attentive Neural Processes (BANPs) [19 which employ the bootstrap technique to further improve ANPs' performance. 4) TFR-Transformer (2) incorporates typical encoder-deecoder-sty. Transformers model correlations between observed points and target points. 5) NIERT 4 leverages Transformer encoder-only architecture to unify the processing of observed and target points in the same feature space to obtain better interpolation performance.
3.3 Experimental Setup
Hyperparameter Setting, We carefully select and tune several hyperparameters of HINT, including the number of interpolation blocks L, the number of attention layers each block, the local parameter Ko, and the minimum local parameter Kmin- For the Mathit-2D dataset, we set L = 2, with 6 attention layers in the main block and 2 attention layers the residue block. As for Perlin, PTV. and TFRD datasets, we set L = 4 with 2 attention layers in each block. The choice of L was based ensure a fair comparison, we maintained
on prior research and our experimental observations. consistency between HINT and the baseline models, NIERT and TFR-Transformes in terms of their key hyperparameter settings. Specifically, we keep the number of attention layers, the number of attention heads, and the dimensionality of the hidden space, consistent across all these models.

---

Table Interpolation accuracy on Mathit dataset.
Table 2: Interpolation accuracy Perlin dataset.
Interpolation
MSE (x 10 *) on
Interpolation
MSE (x P) on
approach
Mathit-2D test set
Perlin test set approach
CNP
24,868
CNP ANP
48.642
ANP
14,001
23.731
BANP
8.419
BANP
20.737
TFR-Transformer
5.857
Tranffosformrr
12.101
NIETT
3.167
NIERT
7.185
HINT (ours)
2.903
HINT (ours)
5.848
Table 3: Interpolation accuracy on PTV dataset.
Table Interpolation accuracy on TFRD dataset.
Interpolation approach
MSE (x 10
*) PTV test
Interpolation approach
MAE (x 10 on TFRD test set
HSink
ADlet
DSine
CNP ANP BANP
137.573
CNP
204.351
91.782
92.456
32.111
ANP
164.491
54.684
58.589
33.585
BANP
59.728
28.671
19.107
TFR-Transformer
17.125
TFR- Transformer
64.987
27.074
29,961
NIERT HINT (ours)
5.167
NIERT 23.519
3,473
8.785
3.507
HINT (ours)
13.758
1.761
4.912
Regarding the local parameter (()), we set to the number of observed points in the interpolation task for Mathit-2D, Perlin, and TFRD datasets, as these datasets have a relatively small number of observed points. For the PTV dataset, set ( 128 to accommodate its larger number of observed points. These values were selected to balance the local constraints with the available information in each dataset. Additionally, datasets. This value was chosen to ensure a sufficient level of local modeling and capture fine-grained details in the interpolation process.
the minimum local parameter K Kmin for all
Evaluation metric. Following the precedent set by previous works 2 4 and for the purpose of facilitating comparison, we adopted the MSE the evaluation metric for Mathit-2D, Perlin, and PTV datasets. Additionally, for the TFRD dataset, we utilized the MAE as the evaluation metric.
Experimental results
4.1 Interpolation Accuracy
We present comprehensive comparisons of test set average interpolation accuracy of our interpolation approach, HINT, with state-of-the-att approaches on the four representative datasets. On the Mathit- 2D dataset, we observe a notable enhancement of 8.34% in accuracy compared to the best method NIERT (Table D). Similarly, on Perlin and PTV datasets, our method demonstrates remarkable accuracy gains of 30.23% and 18.61% % respectively (Tables 2 and [3). Furthermore, on three subsets of the TFRD dataset, i.e., HSink, ADlet and DSine, our method exhibits substantial accuracy gains of 41.50% 49.29% and 44.08% (Table over NIERT. These compelling results validate the efficacy of our approach on interpolation tasks across diverse datasets, highlighting the superiority and versatility of our algorithm in both theoretical and real-worls application scenarios.
4.2 Qualitative Analysis
Qualitative Comparison. Here we provide a more intuitive comparison of the interpolation per- formance between HINT and existing interpolation methods. We randomly select an interpolation task from the Perlin test datasets, visualizing the interpolation results and етrоr maps of HINT and existing interpolation methods. As shown in Figure for this interpolation task from the Perlin test set, our proposed HINT achieves the highest interpolation accuracy (MSE x x1.04 x 10-4). The error map of BANP exhibits strong error hotspots TFR- Transfoonfrrmrr and NIERT still have relatively large accuracy difference regions, while HINT has the largest area with minimal discrepancies (dark blue regions) its error map.
7

---

Realbaal blocks
Reatoval block
Reabual alocky
Bbock obtcoortte
Bakk
Edee
NOW
HAY Dhams
182r4
Comdatinn
moo
Figure 3: Qualitative comparison on a 2D inter- polation task extracted from Perlin test set. The top row shows the 64 observed points and the ground-truth function. The second row shows interpolation functions reported by HINT and the state-of-the-ard approaches. The last row shows their error maps (their differences with the ground truth).
Figure 4: Visualization of output from each in- terpolation block on an example task (same as in Figure (3) from the Perlin test set. The top row of figures shows the output for each block, the middle row shows the accumulated output of these blocks, and the bottom row displays the error maps, indicating the differences between the cumulative predictions and the ground truth.
Residual Visualization. To investigate the role of each block in our proposed HINT, we visualized the output components of each block, shown in Figure We find that the main block produces a relatively smooth overall function shape for the target function, lacking local details. The first residual block outputs more localized fluctuations in the target function with smaller scale. The second residual block generates even smaller-scale components that appear to capture more local details and the last block contributes minuscule components. Observing the cumulative sum of the outputs from these blocks, we find that the accumulated function becomes progressively more detailed and closely resembles the ground truth of the target function as the outputs of the subsequent blocks are added. By examining the error maps of each cumulative sum, we observe that the output of main block has strong error hotspots and subsequent blocks gradually refine the prior predictions until the final error map displays uniformly low errors throughout.
4.3 Ablation Study
Ablation Main Design Choices of HINT. We validate the efficacy of the auxiliary loss term Caux. masked observed points, and local constraints in HINT, as shown in the upper section of Table 5 When the auxiliary loss term is omitted (equivalent to setting the weight À of the auxiliary loss term Laux to 0), as indicated in the table as "HINT w/o Laux", HINT experiences a reduction in accuracy on all three datasets, with notably significant impact Perlin and TFRD-ADle (decreasing by 30% and 125%, respectively). Omitting the local constraints ("HINT w/o KNN") also leads to decreased accuracy on these datasets, particularly affecting Perlin and PTV (with a decrease of 20% and 43%, respectively). Additionally, we assess the effectiveness of masked observed points ("HINT
w/o masked obs."). ). In this setup, instead of using yio (1) we utilize yo (!) to compute residuals on the observed points. Experimental results in this configuration reveal decline in accuracy across all three datasets, most notably Perlin (a reduction of 55%), underscoring the significance of maintaining a consistent distribution between observed and target points. These findings highlight the advantages of incorporating these design choices within our framework.
The Jower half of Table presents the ablation results on different local constraints. For straightfor- ward evaluation, we employed two ablation settings where the local constraint parameter K was set to either (0)0) or (1) for all interpolation blocks. Here, k3(0) represents the local parameter of the first block in the hierarchical local constraint, while (()) represents the local parameter of the second block. These two ablation configurations of local parameters represent non-hienenn non-hierarchical approach with similar hy hyperparameters to our hierarchical approach. These results demonstrate the hierarchical design of local constraints has significant superiority over the non-hierarchical one in our method.
Ablation on Interpolation Blocks Number. Table presents the results obtained by varying the number of blocks from I to 5 on two datasets, Perlin and PTV. The results indicate that when there
8

---

Table 5: Ablations main design choices.
Table 6: Ablation on block number L.
Interpolation accuracy on
Interpolation accuracy on
Interpolation
approach
(MSEx10 9) (MSEx10 10
Perlin
PTV
TFRD-ADlet (MAEx10 )
Block number
Perlin (MSE x 10
PTV (MSE 10-9)
HINT w/o frus
7,630
3.656
3.985
HINT w/o KNN
7.017
5.001
1944
21.453
14,362
HINT w/o masked obs.
9,071
3,547
2.773
2
1.279
5.527
HINT
5.848
3.507
1.761
3
8.073
4.386
HINT x((0))
7:017
3.811
1.944
HINT x())
8.083
4.791
2.975
4
5.848
3,507
HINT (Hicrarchical)
5,848
3.507
1.761
6.224
3.618
only one block (i.e., no residual block), HINT performs poorly on both datasets. However, with the introduction of single residual block (i.e., two blocks in total), the interpolation performance of HINT improves significantly. This highlights the effectiveness of incorporating the interpolation module. As the number of blocks increases, the interpolation accuracy of HINT shows incremental improvements with smaller margins. When the number of blocks reaches 5, the interpolation accuracy of HINT shows slight decline on the Perlin dataset and remains relatively stable on the PTV dataset. These results suggest that after a certain number of blocks, the additional blocks no longer contribute the learning of HINT, and the interpolation accuracy plateaus.
Related Work
Traditional Approaches for Scattered Data Interpolation. Traditional interpolation algorithms for scattered data construct the target function through linear combinations of explicit basis functions. For one-dimensional scattered data, techniques such as linear interpolation, Lagrange interpolation,
Newton interpolation [TT] and spline interpolation [9] are commonly employed. For more general scattered data residing in siggherdiimesiinal spaces, Shepard's methods [26). and Radial Basis Function (RBF) interpolation [24 6 are widely used. MIR (29) employs the Taylor series for interpo- lation and determines the weights by optimizing the interpolation error. In engincering applications, hierarchical interpolation methods have also been developed [5]. which are often combined with spatial partitioning techniques or the selection of nearby points to reduce computational complexity. All these traditional methods are inherently non-adaptive, as they don't 't learn the function distribution from data. This restricts their interpolation precision and hampers their generalization across diverse scenarios.
Learning- based Approaches for Scattered Data Interpolation. Recent neural network-based
scattered data interpolation algorithms can be broadly categorized into two groups: probabilistic and deterministic. Probabilistic methods, such as the family of Neural Processes [8 [1 [8 estimate the conditional distribution the target function based on observed points. Deterministic methods, such as TraaTTranffrmer 2 and NIERT 44, estimate the target function based on observed points with a focus on interpolation accuracy. Technically, these approaches use neural network modules to model the correlation between observed and target points, subsequently obtaining the representation and estimation of target point values. For instance, CNPs employs MLPs model the correlation between observed and target points, while ANPs [14], Transfaanformer, and NIERT utilize attention mechanisms and Transformer layers (27) for the same purpose. Among these methods, NIERT employs the architecture of the Transformer encoder to uniformly handle observed and target points, resulting in enhanced interpolation accuracy. stands as the current state-of-the-an neural interpolation algorithm, serving inspiration for our interpolation block design. Notably, these current neural interpolators have thus far neglected the information regarding interpolation residuals at observed points, which constitutes shared limitation.
30 [9, [2]].
Residual Learning and Decomposition Structure. Residual connections in neural networks can be traced back to residual modules in ResNet [10] and DenseNet [13]. In this module, the output of each stack fed into subsequent stacks, for better optimization deep neural networks. For a different propose, in time series forecasting, N-BEATS (22 presents doubly residual framework with two distinct residual branches: one branch for backcast prediction and another branch for the forecast prediction at each layer. The residual between the backcast prediction and history input is

---

exploited to forecast prediction. N-HiTS [ integrates hierarchical sparse prediction into the similar doubly residual framework, enhance both forecasting accuracy and interpretabilityy. However, these techniques are primarily tailored for fundamental time series signal decomposition and display a notable limitation when tasked with learning sparse and scattered data interpolation. Although the exploitation of residual is successfully applied in time series forecasting, it is hard to apply in scattered data interpolation straight-forwane for sparsity and irregularity of scattered data.
6 Conclusion
In this paper, we propose HINT, a novel hierarchical residual refining framework for scattered data interpolation. By employing series of residual blocks in coarse-to-fine fashion and incorporating hi- erarchical local constraints in residual correlation modeling, HINT effectively improves interpolation accuracy. Our extensive validation across diverse theoretical and application scenarios demonstrates the superior performance of HINT compared to existing ttate-of-the-e art interpolation methods. We acknowledge the limitations of our approach regarding the hyperparameter K in hierarchical local constraints, and we plan to explore methods for learning adaptive K values in future work.
7 Acknowledgements
We would like thank the National Key Research and Development Program of China (2020YFA0907000, and the National Natural Science Foundation of China (32370657, 32271297, 82130055, 62072435) for providing financial supports for this study and publication charges. The numerical calculations this study were supported by ICT Computer-X center.
References
[1] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler-Canee and Artur Dubrawski. N-hits: Neural Hierarchical Interpolation for Time Series Forecasting. arXiv preprint 22Xiv:2201.128888 2022.
[2] Xiaoqian Chen, Zhiqiang Gong, Xiaoyu Zhao, Weien Zhou, and Wen Yao. A Machine Learning Modelling Benchmark for Temperature Field Reconstruction of Heat-Source Systems. arXiv preprint arXiv:2 08298, 2021.
[3] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint Ttt::181000400.. 2018
[4] Shizhe Ding, Boyang Xia, Milong Ren, and Dongbo Bu. NIERT: Accurate Numerical Inter- polation through Unifying Scattered Data Representations using Transformer Encoder, arXiv preprint arXiv:2209.09N) 09078, 2023.
[5] Michael S Floater and Armin Iske. Multistep Scattered Data Interpolation Using Compactly Supported Radial Basis Functions. Journal of Computational and Applied Mathematics, 73(1- 2):65-78, 1996.
[6] Bengt Fornberg and Julia Zuev. The Runge Phenomenon and Spatially Variable Shape Pa- rameters RBF Interpolation. Computers Mathematics with Applications, 54(3):379-398, 2007.
[7] Richard Franke and Gregory M Nielson. Scattered Data Interpolation and Applications: A Tutorial and Survey. Geometric Modeling, pages 131-160, 1991.
[8] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and M. Ali Eslami. Conditional Neural Processes. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pages 1704-1713. PMLR, 10-15 Jul 2018.
10

---

[9] Charles A Hall and W Weston Meyer. Optimal enror bounds for cubic spline interpolation. Journal of Approximation Theory, 16(2):105-12,, 1976.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[11] Michael T Heath. Scientific Computing: An Introductory Survey, Revised Second Edition. Society for Industrial and Applied Mathematics, 2018.
[12] Sepp Hochreiter and Jürgen Schmidhuber. Long Short-term Memory, Neural Computation, 18): 1997.
[13] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely Con- nected Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4700-4708, 2017.
[14] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive Neural Processes. In International Conference on Learning Representations, 2019.
[15] Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations,
2014 2014.
[16] Thomas N Kipf and Max Welling. Supi-supe classification with graph convolutional networks. arXiv preprint arXiv: (et::::09900000 2016.
[17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet Classification with Deep Convolutional Neural Networks. Communications of the ACM, 60(6):84 90, 2017.
[18) Byung-Jun Lee, Seunghoon Hong, and Kee-Eung Kim. Residual Neural Processes. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4545 4552, 2020.
[19] Juho Lee, Yoonho Lee, Jungtaek Kim, Eunho Yang, Sung Ju Hwang, and Yee Whye Teh. Bootstrapping neural processes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6606-6615. Curran Associates, Inc., 2020.
[20] GR3543275 Liu. An Overview on Meshfree Methods: For Computational Solid Mechanics. International Journal of Computational Methods, 13(00))1630001, 2016.
[21] Tung Nguyen and Aditya Grover. Transformer Neural Processes: Uncertainty-aware Meta Learning via Sequence Modeling. arXiv preprint arXiv:2207.04 2201224477779... 2022.
[22] Boris Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-BEATS: Neu- ral Basis Expansion Analysis for Interpretable Time Series Forecasting. arXiv preprint 1rXiv 10437, 2019.
[23] Ken Perlin. An Image Synthesizer, ACM Siggruph Computer Graphics, 19(3):287-266, 1985.
[24] Michael JD Powell. Radial Basis Functions for Multivariable Interpolation: A Review. Algo- rithms for Approximation, 1987.
[25] Andrea Sciacchitano, Douglas R Neal, Barton L. Smith, Scott O Warner, Pavlos P Vlachos, Bern- hard Wieneke, and Fulvio Scarano. Collaborative framework for piv uncertainty quantification: Comparative assessment of methods. Measurement Science and Technology. 26(7):074004, 2015.
[26] Donald Shepard. A Two-dimensional Interpolation Function for Irregularly-spaced Data. In Proceedings of the 23rd ACM National Conference, pages 517-524, 1968.
[27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, pages 5998-6008, 2017.

---

[28] Di Wang and Xi Zhang. Modeling of a 3D Temperature Field by Integrating a Physics-Specific Model and Spatiotemporal Stochastic Processes. Applied Sciences, 9(10), 2019.
[29] Qiqi Wang, Parviz Moin, and Gianluca Iaccarino. A High Order Multivariate Approximation Scheme for Scattered Data Sets. Journal of Computational Physics, 229(18):633366311 2010.
[30] Noah Weber, Janez Stare, Arpit Mittal, Roi Blanco, and Lluís Mårquez. Optimizing over Bayesian Last Layer. In NeurlPS Workshop on Bayesian Deep Learning, 2018.
12


Reviews:
[{"review_id": "5vkCnc8XRD", "reviewer": "Reviewer_746Q", "summary": "The paper introduces the Hierarchical INTerpolation Network (HINT) as an accurate interpolation algorithm for various theoretical and engineering applications. HINT consists of lightweight interpolation blocks arranged sequentially, where the first block estimates the main component of the target function, and subsequent blocks predict residual components using observed point residuals from preceding blocks.  Moreover, the authors introduce hierarchical local constraints. Extensive experiments demonstrate that HINT significantly outperforms existing interpolation algorithms.", "questions": "see above", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The method that utilizing the first and subsequent blocks to estimate the main component and residual components is novel. It applies the residual method to  interpolation algorithm, which is also interesting.\n2. The whole method is easy to follow.\n3. This paper conduct extensive extriments on various datasets, demonstrateing its superiority.", "weaknesses": "1. The comparison of the time cost is missing. It's necessary to compare the proposed HINT and other methods including the traditional ones and deep learning ones.\n2. The paper lacks the ablation study of hyper-parameter L and loss scale $\\lambda$. Besides, does the value of $\\lambda$ keep the same for all the datasets?\n3. For Mathit-2D and Perlin, the choich of L and K is (2,6) and (4,2), respectively. Could the author provide some guidlines for the choice ? Or it may Hinder the spread and use of this method. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer 746Q\n\nWe sincerely thank Reviewer 746Q for the insightful comments and suggestions. We will address each comment in the following.\n\n**Comment 1:** The absence of time cost comparison between HINT and other methods and comparison with traditional methods.\n\n**Response 1:** \nIn response to the comment regarding the comparison of time cost, we have performed additional evaluations. Specifically, we assessed the average interpolation time cost of our proposed HINT method and existing interpolation techniques, both traditional and deep-learning-based, on the Mathit-2D test set. Following the work from [1], we selected RBF [2] and MIR [3] as representatives of traditional algorithms. The results are tabulated below:\n\n|Interpolation approach| Average time (ms)|Interpolation accuracy (MSE $\\times10^{-4}$)|\n|:---:|:---:|:---:|\n|RBF|0.66|34.706|\n|MIR|80.30|27.460|\n|CNP|13.53|24.868|\n|ANP|27.55|14.001|\n|BANP|29.19|8.419|\n|TFR-Transformer|26.16|5.857|\n|NIERT|31.99|3.167|\n|**HINT**|36.20|**2.722**|\n\nAs evident from the table, the traditional interpolation method, RBF, offers the highest interpolation efficiency at 0.66ms, but its accuracy is substantially compromised. MIR offers slightly better accuracy, it possesses the longest interpolation time amongst all methods evaluated (80.30ms). For deep neural network-based methods, there is a clear trend of increasing accuracy from CNP to our HINT method and the time cost also gradually rises. Specifically, our HINT method has an average time of 36.20 ms, which, while marginally higher than that of NIERT (31.99ms), is considerably more efficient than the traditional MIR (80.30ms).\n\nMoreover, it's worth noting that, our current HINT implementation employs dense attention computation combined with an attention mask to realize the hierarchical local constraints proposed in our paper. Given that these local constraints are theoretically sparse, there remains room for optimizing the time cost of HINT. Our primary focus in this research was to achieve high interpolation accuracy, and, as of now, HINT appears to be best suited for scenarios demanding high accuracy while being somewhat lenient on time constraints.\n\n**Comment 2:** lacks of the ablation study of hyper-parameter $L$ and loss scale $\\lambda$.\n\n**Response 2:** In fact, we conducted ablation studies on the hyper-parameter $L$ and loss scale $\\lambda$. The ablation study concerning the block number $L$ can be found in Table 6 on Page 8. The ablation experiment for the auxiliary loss weight $\\lambda$ is presented in the first row of Table 5 on Page 8, specifically under the section \"HINT w/o $\\mathcal{L}_{\\mathrm{aux}}$\", where this implies $\\lambda=0$. To clarify further on your query regarding the value of $\\lambda$, it does vary, and these specific values are detailed in the last row of Table 2 on Page 3 in the supplementary material. We arrived at these values after a series of experiments to fine-tune our approach.\n\nWe appreciate your question as it has made us realize that we might not have explicitly highlighted these details in our current presentation. We will emphasize these more prominently in the revised version of our manuscript to prevent any ambiguity.\n\n**Comment 3:** The guidelines for choosing hyperparameters (block number $L$, attention layer number, and local parameter $K$).\n\n**Response 3:** If I understand correctly, the second hyperparameter you mentioned is the number of attention layers in the main block. In fact, the choice of these hyperparameters is not arbitrary. We outline the rationale for setting these parameters as follows:\n\n1. **Block number ($L$)**: In our experiments, we observed that for functions with smoother profiles and fewer details within a set, a smaller block number $L$ is appropriate. Conversely, for functions rich in details and containing higher frequency components, a larger block number $L$ is more suitable.\n\n2. **Attention layers in the main block**: Concerning the number of attention layers in the main block, our findings indicate that larger datasets demand more attention layers to attain sufficient model capacity. On the other hand, for smaller datasets, fewer attention layers can help prevent overfitting.\n\n3. **Local constraint parameter ($K$)**: Setting $K$ is somewhat challenging, requiring insights into the data or even expert knowledge. Such knowledge encompasses spatial auto-correlation of the function. If the function exhibits auto-correlation over a larger neighborhood, a larger $K$ might be more appropriate. In our experiments, we found that determining the optimal $K$ often requires trial and fine-tuning.\n\nFor instance, for the Mathit dataset—a synthetic set comprised of smoother mathematical functions with a significant volume—a smaller block number $L$ and more attention layers seem optimal. The mathematical functions' features correlate over longer distances (periodicity, symmetry, trends, etc.), thus initially a larger $K$ covering all observation points is prudent. Conversely, for the Perlin dataset—a real-world 2D velocity field dataset of smaller volume but intricate, detailed functions—a larger block number $L$, fewer attention layers, and a smaller $K$ are logical. We appreciate the reviewer's constructive feedback and will incorporate this discussion in the Supplementary Material.\n\n------\n\n[1] Shizhe Ding and Dongbo Bu. NIERT: Accurate Numerical Interpolation through Unifying Scattered Data Representations using Transformer Encoder. arXiv preprint arXiv:2209.09078, 2023.\n \n[2] M. J. Powell, Radial basis functions for multivariable interpolation: a review. Algorithms for Approximation, 1987.\n \n[3] Q. Wang, P. Moin, and G. Iaccarino, A high order multivariate approximation scheme for scattered data sets, Journal of Computational Physics, vol. 229, no. 18, pp. 6343–6361, 2010."}, {"review_id": "2uj5HsRydZ", "reviewer": "Reviewer_G7AZ", "summary": "In this paper, a novel hierarchical residual refining framework called HINT (Hierarchical Residual Refining for Scattered Point Interpolation) was proposed to improve interpolation accuracy. The framework utilized residual information from observed points to guide the prediction of target points in a coarse-to-fine manner, employing correlation modeling. HINT consisted of a series of neural network interpolation blocks, including a main block and multiple residual blocks, which estimated different components of the target function at various scales. Local constraints were applied to the interpolation blocks using K-nearest neighbor graph constraints, conforming to hierarchical residual scales. The hierarchical structure of the main and residual components was leveraged, enabling progressively stronger local constraints to enhance interpolation robustness. Experimental evaluations on diverse theoretical and application scenarios demonstrated the superiority of HINT over existing state-of-the-art interpolation methods in terms of accuracy across various tasks.", "questions": "More literature reviews should be provide to demonstrate the necessity of using Transformer in interpolation.\n\nWhat are the main differences of the proposed method when compared with  the Transformer encoder-only method in [4]?\n\nThe MSE results in Fig.5 in Supplementary Material can be separated into a table for better clarity, and the authors can provide a case by case analysis for those figures in Supplementary Material.", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and it provides concise descriptions of the proposed framework.\n\nThe experiments conducted in this paper are comprehensive.  ", "weaknesses": "A couple of assumptions are used, for example in Sec. 2.3, the authors failed to demonstrate that they all holds in experiments.\n\nThe proposed framework is simply a combination of existing modules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer G7AZ\n\nWe appreciate the reviewer's constructive feedback. In the following sections, we address each point raised.\n\n**Comment 1:** The assumptions made in Section 2.3 are not validated experimentally.\n\n**Response 1:** In relation to the assumptions highlighted in Section 2.3, we'd like to offer the following clarifications:\n\n1. **Foundational Work and Validation**:\n   - Our approach is built in the NIERT interpolator [1]. The effectiveness of this methodology is comprehensively discussed in [1], establishing it as a SOTA model for scattered data interpolation.\n\n2. **Our Innovative Design and Verification**:\n   - **Incorporation of masked observed points**. It has been empirically validated in our ablation study (refer to Table 5, Page 8, third row). Our findings indicated a notable decline in precision when leveraging observed points without the inclusion of \"masked observed\" points.\n   - **Local constraints on correlation modeling**. This design have been extensively corroborated in Section 4. For instance, Fig. 4 (on Page 8) elucidates how the interpolation blocks in HINT sequentially yield the main component of the target function, succeeded by residuals of diminishing magnitude. Furthermore, Table 5 (on Page 8) reinforces this through ablation studies, indicating that the performance of HINT, when devoid of local constraints or with non-hierarchical local constraints, is inferior to our proposed HINT.\n\nWe hope that the above elucidations address your concerns. Should there be any oversight or if further clarifications are required, we humbly solicit your guidance.\n\n**Comment 2:** The proposed framework is simply a combination of existing modules.\n\n**Response 2:** We would like to take this opportunity to further elucidate the unique contributions and the value of our research.\n\n1. **Beyond a Mere Combination**: While we have indeed leveraged the N-Beats from the time series forecasting and NIERT, our work is not just about juxtaposing two techniques. We have undertaken substantial modifications. Specifically:\n   1) We significantly altered the NIERT block to enable it to output re-predictions of observed points that are consistent with target point predictions. \n   2)  Furthermore, we incorporated local constraints based on KNN Graph, which facilitates adaptation to the varying local scales inherent in function interpolation.\n   3) For the interpolation blocks, we incorporated progressively stringent local constraints. Early blocks focus on capturing the primary components of the function, while subsequent blocks hone in on the intricate details. This hierarchical fashion allows for a nuanced refining of interpolation.\n\n2. **Pioneering Application of Dual Residual Architecture for scattered Data Interpolation**: Our study is the first to introduce such residual architecture into the realm of scattered data interpolation.\n\n3. **Superior Experimental Results**: Our method has demonstrated exemplary performance in experiments, achieving state-of-the-art results.\n\n\n**Comment 3:** More literature reviews should be provide to demonstrate the necessity of using Transformer in interpolation.\n\n**Response 3:** Indeed, the existing literature specifically addressing the application of Transformers to scattered data interpolation is somewhat limited. This is the very reason our manuscript heavily references works like NIERT [1] and TFR-Transformer [2]. We can elucidate the superiority of Transformer from the following perspectives:\n\n1) **Inductive Bias**: The Transformer architecture is particularly suited for handling scattered point data due to its inherent permutation invariance.\n\n2) **Recent Advancements**: NIERT stands out as one of the most recent and high-performing scattered data interpolation models, effectively showcasing the superiority of Transformer-based models for such tasks.\n\n3) **Theoretical Underpinnings**: Both references [1] and [3] draw connections between the Transformer framework and traditional interpolation algorithms. Specifically, [1] posits that the self-attention in Transformers can be viewed as a neural representation for interpolation basis function learning. Such perspectives provide a theoretical foundation, underscoring the strong alignment between Transformers and interpolation.\n\nWe will endeavor to expand upon and clarify these points in the revised manuscript.\n\n\n**Comment 4:** main differences of the proposed method when compared with the Transformer encoder-only method in [4]?\n\n**Response 4:** NIERT and our proposed HINT exhibit significant differences, which can be summarized as follows:\n\n1. **Architectural Differences**: The Transformer encoder-only method, termed as NIERT, employs a singular Transformer encoder as its main model. In contrast, our proposed methodology, named HINT, integrates a series of lightweight blocks, rooted in the Transformer encoder architecture, and connected through residual links.\n\n2. **Methodological Variations**: NIERT directly predict target point values using a whole piece model. Our HINT adopts a hierarchical refining process. It firstly predicts the main component of the function, followed by a series of residual components incrementally. \n\n**Comment 5:** The MSE results in Fig.5 in Supplementary Material can be separated into a table for better clarity...\n\n**Response 5:** For clarity, we will adjust the presentation in accordance with the reviewer's suggestions.\n\n---\n\n[1] Shizhe Ding and Dongbo Bu. NIERT: Accurate Numerical Interpolation through Unifying Scattered Data Representations using Transformer Encoder. arXiv preprint arXiv:2209.09078, 2023.\n\n[2] Xiaoqian Chen, Zhiqiang Gong, Xiaoyu Zhao, Weien Zhou, and Wen Yao. A Machine Learning Modelling Benchmark for Temperature Field Reconstruction of Heat-Source Systems. arXiv preprint arXiv:2108.08298, 2021.\n\n[3] Cao, Shuhao, Peng Xu, and David A. Clifton. How to understand masked autoencoders. arXiv preprint arXiv:2202.03670, 2022."}, {"review_id": "DrrePMYKQV", "reviewer": "Reviewer_RpoB", "summary": "This work presents a new transformer based approach for scattered data interpolation. By extending NIERT [4] in several aspects, authors achieved improved results in the target task. The extensions include 1) hierarchical residual refinement to produce fine-grained interpolation results and 2) hierarchical local constraints to constrain the set of observed points used in the interpolation block. In short, unlike existing approaches, authors attempt to conduct the scattered data interpolation in a progressive fashion by estimating interpolation residuals within gradually narrowing search space.", "questions": "Please consider the weaknesses 1), 2), 3).", "limitations": "N.A.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) The proposed components were reasonably designed to obtain fine-grained interpolation results.\n2) Intensive ablation studies support the effectiveness of these components.\n", "weaknesses": "1) The datasets used in experiments consists of two-dimensional data. Performance should be evaluated with sparse dataset of much higher dimension.\n2) Some explanations were given in the embedding stage, but it is very difficult to interpret why the masked tokens for the given observed points are necessary. \n3) The local constraint K may be problematic, as it can be dependent on the data properties of each application.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer RpoB\n\nMany thanks to the reviewers for the thorough feedback. We will proceed to respond to each of the comments provided.\n\n**Comment 1:** The datasets used in experiments consists of two-dimensional data. Performance should be evaluated with sparse dataset of much higher dimension.\n\n**Response 1:** We acknowledge the importance of evaluating performance on sparser datasets of higher dimensions. Our initial choice of two-dimensional datasets was driven primarily by their availability and prominence in the literature.\n\nIn response to your suggestion, we synthesized a 10-dimensional dataset, termed as \"D10\". Within this dataset, each function for sampling an interpolation task arises from the summation of several 10-dimensional Gaussian functions, chosen randomly.\n\nWe compared the interpolation performance of HINT against existing approaches on this dataset, and the results can be observed in the appended table. Notably, the outcomes demonstrate that the interpolation precision of HINT can effectively scale to datasets with much higher dimensions.\n\n|Interpolation approach|Interpolation accuracy (MSE $\\times10^{-4}$) on D10|\n|:---:|:---:|\n|CNP|35.623|\n|ANP|12.578|\n|BANP|12.077|\n|TFR-Transformer|7.465|\n|NIERT|5.496|\n|**HINT**|**4.173**|\n\nDue to space constraints, we are unable to provide a detailed description of the D10 dataset here. The comprehensive construction process of D10 and the associated results will be placed in the supplementary materials.\n\n**Comment 2:** It is difficult to interpret why the masked tokens for the given observed points are necessary.\n\n**Response 2:** The primary intention behind using \"masked observed points\" is to ensure consistency with the masked target points being input. By doing so, we can obtain re-predictions on the observed points $\\hat{y_O}^{(l)}$, which is consistent with the predictions on target points $\\hat{y_T}^{(l)}$. This consistency stems from the fact that both predictions emerge from the same function: $\\hat{f}(x_{\\mathrm{masked}}) = \\mathrm{Block}^{(l)}(x_O, y_O, x_{\\mathrm{masked}})$ evaluated at both observed and target points.\n\nConsequently, the residuals from re-predictions on the observed points and the residuals from predictions on the target points are strongly correlated. By easily obtaining the former, we can effectively predict the latter.\n\nWe appreciate the reviewer's keen observation, and we hope this clarification elucidates our methodology more comprehensively.\n\n**Comment 3:** The local constraint K may be problematic, as it can be dependent on the data properties of each application.\n\n**Response 3:** Indeed, the hyperparameter $K$ for the local constraint is application-dependent, an aspect we have given careful consideration. However, we don't perceive this as \"problematic\". In our research, the value of $K$ for each dataset was meticulously determined through a series of experiments to ensure optimal performance. Moreover, we validated the efficacy of this parameter through ablation studies. We believe that, with appropriate selection, the constraint remains highly effective. We hope this addresses your concerns."}, {"review_id": "1Hm7fdGDW6", "reviewer": "Reviewer_ce6Q", "summary": "The authors proposed an algorithm named Hierarchical Interpolation Network (HINT) to predict unseen point values in scattered data to replace the usage of manually designed interpolation algorithms. The HINT used the residuals on observed points to guide target function estimation and the hierarchical local constraints in correlation modeling between observed and target points. Experiments show that it outperforms existing interpolation algorithms in accuracy on 3 synthetic datasets.\n\n-- Post rebuttal:\nI read the rebuttal from the auhtors. It addresses some of my concerns on the comparsions and I would like to keep my score.", "questions": "How many samples should be observed at least to guarantee the accuracy of your algorithm and this interpolation task? Is there any theory supported by its lower limit?\n\nCan it be used for more dimensional or more complex interpolation？ Such as interpolation for pixels (inpainting, super-resolution, video interpolation)? \n", "limitations": "The algorithm and the task setting did not consider the situation of incorrect observation (noisy) points which is common in theoretical and engineering domains.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The algorithm leverages a neural network to adaptively discern the underlying distribution of the target function from a function set or scattered dataset, achieving state-of-the-art performance on four datasets.", "weaknesses": "1. Limited novelty in network design. Mask mechanism and partial attention are used in NIERT. The Embedding Stage and Correlation Modeling in Transformer blocks are very similar to NIERT. \n2. Inadequate method comparisons. The paper compared Neural Processes algorithms proposed in 2018, 2019, and 2020 but the later developments in 2022 not included such as [1] and [2].\n3. It did not use PhysioNet dataset which is compared in NIERT.\n\nIn SM:\n1. Did not explain the reason why it has more FLOPs than NIERT in some datasets.\n2. Mistake： In SM line 105, ‘On the PTV dataset,’ should be ‘On the Perlin dataset,’\n\n[1] Wang, Qi, Marco Federici, and Herke van Hoof. \"Bridge the Inference Gaps of Neural Processes via Expectation Maximization.\" The Eleventh International Conference on Learning Representations. 2022.\n[2] Liu, Huafeng, et al. \"Learning Intrinsic and Extrinsic Intentions for Cold-start Recommendation with Neural Stochastic Processes.\" Proceedings of the 30th ACM International Conference on Multimedia. 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer ce6Q\n\nWe are grateful to the reviewer for the comprehensive comments. The following responses are structured to address each comment.\n\n**Comment 1:** Limited novelty in network design..\n\n**Response 1:** Our HINT and NIERT exhibit significant differences:\n\n1. **On architecture**: NIERT is built upon a singular Transformer encoder as its core model. HINT employs a sequence of lightweight interpolation blocks based on the Transformer encoder, which are interconnected through residual connections.\n\n2. **On methodology**: NIERT's approach involves directly predicting the function's values at target points. HINT strategy employs a hierarchical refinement process. It sequentially predicts the primary components of the function and a series of residual components and combines them.\n\n3. **Adaptations from NIERT**: Our interpolation block is inspired by NIERT. Yet, we've made substantial adjustments to fit our broader architecture:\n   - Masked observed points as input is introduced, enabling a re-prediction of the function values at the observed points. This, in turn, provides us with a function's residual at observed points, which guides the subsequent residual function predictions.\n   - Local constraints applied in correlation modeling, which ensures that our interpolation block can interpolate the function's details at different scales.\n\nWe believe our enhancements distinctly set HINT apart from NIERT, adding unique value and novelty. We trust this clarifies the concerns raised.\n\n**Comment 2:** Inadequate method comparisons..\n\n**Response 2:** 1. After thorough research on recent NPs algorithms, we observed that most either mirror methods we've compared or aren't apt for accuracy comparisons, as outlined below.\n\n1. **Regarding [1]**: We previously examined this method. After a detailed review on its appendix and [review feedback](https://openreview.net/forum?id=A7v2DqLjZdq&noteId=19Gw_u0rbIy), we observed its accuracy weren't notably superior to techniques like CNP. Since we've already benchmarked against CNP, ANP, and BANP, we felt it redundant to include this method in our comparisons.\n\n2. **Regarding [2]**: We noted that this method applies NPs algorithms to recommendation systems. Consequently, it was not considered relevant for comparison in the context of our study due to its weak correlation.\n\n3. **Highlight on Notable Recent Work**: It's worth noting that among the recent developments, Transformer Neural Processes (TNP) [3] stands out as a significant contribution. However, its architecture is almost identical to NIERT. To avoid redundancy, we chose not to include it in our comparisons.\n\nIn summary, we've selected methods for comparison that are leading figures within NPs algorithms. Combined with NIERT and TFR-Transformer, we believe they offer a balanced comparison.\n\n**Comment 3:** It did not use PhysioNet dataset.\n\n**Response 3:** We omitted the PhysioNet dataset primarily because it's a 1D time series. As noted in [4], scattered data interpolation typically involves data points irregularly spaced in 2D or higher spaces. While NIERT assessed their model on PhysioNet, given our research focus, we didn't find it essential for our evaluation.\n\n**Comment 4:** why more FLOPs than NIERT in some datasets?\n\n**Response 4:** The two primary reasons are: \n1. HINT encompasses several blocks, each embedded with an embedding and a prediction layer. NIERT only has a embedding and a prediction layer.\n2. The input token consideration for HINT is greater, encompassing tokens from both the observed and the masked observed points in addition to the target points, leading to (2n+m) tokens, while NIERT uses (n+m) tokens.\n\n**Comment 5:** Typos.\n\n**Response 5:** The error has been corrected as suggested.\n\n**Comment 6:** On the number of observed points at least to guarantee the accuracy. Is there any theory supported by its lower limit?\n\n**Response 6:** We observed that a decrease in the number of observed points correlates with an increase in interpolation error. Thus, to maintain our model's accuracy, the minimum required observed points would align with those in our test sets.\n\nRegarding a theoretical foundation, crafting such a framework is complex and outside our study's current scope. Real-world data often eludes neat theoretical frameworks. Hence, pinpointing a direct theoretical link between observed point counts and our method's accuracy remains challenging. We appreciate your emphasis on the theoretical aspect, which undeniably stands as a crucial avenue for future research.\n\n**Comment 7:** On high-dimensional interpolation or more complex interpolation for pixels.\n\n**Response 7:** Regarding your question on high-dimensional interpolation, Reviewer 746Q posed a similar suggestion. Please refer to our response to 746Q where we've included pertinent experiments. \n\nPixel-related interpolation tasks quite differs from scattered data interpolation. These tasks usually involve data points uniformly distributed on a dense grid, imbued with rich semantic information. Adapting our method, which is tailored for scattered data interpolation, to these tasks would likely necessitate substantial alterations.\n\n**Comment 8: ** Limitation in handling noisy data\n\n**Response 8:** Our study centers on high-precision interpolation, assuming observation points are accurate. Addressing noisy data poses unique challenges and falls outside this research's purview. We recognize its significance and might explore it in subsequent works.\n\n---\n\n[1] Wang, Qi, et al. Bridge the Inference Gaps of NPs via Expectation Maximization. ICLR. 2022. \n\n[2] Liu, Huafeng, et al. Learning Intrinsic and Extrinsic Intentions for Cold-start Recommendation with Neural Stochastic Processes. ICM. 2022.\n\n[3] Nguyen, Tung, et al. Transformer Neural Processes: Uncertainty-aware meta learning via sequence modeling. arXiv, 2022.\n\n[4] Wendland, Holger. Scattered data approximation. Vol. 17. Cambridge university press, 2004."}, {"review_id": "JBSOxxyoFz", "reviewer": "Reviewer_WnAW", "summary": "This paper introduces the Hierarchical INTerpolation Network (HINT), a hierarchical framework that leverages the residuals on observed points to guide the estimation of the target function. HINT comprises multiple lightweight interpolation blocks arranged sequentially. The first block estimates the main component of the target function, while subsequent blocks predict the residual components using the residuals from the preceding blocks. By accumulating the main component and residual components, HINT produces the final interpolation results. Experiments demonstrate the effectiveness of HINT, and the results demonstrate HINT can get better performance compared with existing interpolation algorithms across a wide range of datasets.", "questions": "The motivation of the proposed hierarchical residual refining framework is not clear, could you explain the motivation of hierarchical residual refining framework from the perspective of experiments, such as ablation study, or theoretical analysis?", "limitations": "The limitations of this paper regarding the hyperparameter K in hierarchical local constraints.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) This paper is easy to follow.\n2) The Ablation Study clearly validates the effectiveness of the proposed auxiliary loss term, masked observed points and local constraint in HINT.", "weaknesses": "1) The main concern is the novelty of this paper is not clearly presented. The major contribution of the paper seems to be the proposed hierarchical residual refining framework but its novelty has not been sufficiently addressed. It is worth noting that previous studies, specifically [1] and [22] as mentioned in the last paragraph of the section of Related Work, have already proposed hierarchical residual architectures for time series forecasting. However, the paper does not discuss the challenges in applying the hierarchical residual refining framework to the field of interpolation, regarding that interpolation can also be viewed as prediction from existing observations. It neither emphasizes the distinction between the proposed architecture and existing architectures. As a result, the relation between this paper and existing literature is not clear and the main contribution of the paper cannot be evaluated.\n2) The presentation is not clear and should be carefully revised. For example, the positions of the observed points are defined twice in Line 83 and Line 84. The expression “It outputs and outputs the coarse function value predictions at both the observed and target points” in Line 94 is wrong.\n3) The experimental results provided in the paper are limited. For instance, the authors did not conduct experiments to investigate the performance of the proposed method with varying ratios of observed points. Therefore, it is hard to evaluate the effectiveness of the proposed method under different data conditions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer WnAW\n\n**Comment 1:** The paper's novelty, especially regarding its hierarchical residual refining framework, isn't clearly articulated, with insufficient distinction from previous works on hierarchical residual architectures.\n\n**Response 1:**  Thank you for your thoughtful comments and concerns regarding the novelty of our work. We'd like to address these concerns systematically:\n\n1. **Main contributions of our HINT**:\n    1. Our paper's key contribution is a novel hierarchical residual refining framework for scattered data interpolation. While inspired by N-Beats [22] and N-Hits [1], our model diverges significantly from both them in its application to this specific task (as detailed below).\n    2. The hierarchical local constrained approach progressively refines residual predictions, leading to reduced residuals and improved interpolation accuracy.\n    3. Our model has outperformed existing methodologies across all datasets tested.\n\n2. **Differences between scattered data interpolation and time series forecasting**:\n    1. Data Dimensionality: Time series forecasting is 1D, focusing on modeling long-term dependencies. Scattered data spans 2D or higher, revealing complex spatial relationships.\n    2. Sparsity: Time series data are densely recorded within time frames, whereas scattered data is often sparse.\n    3. Ordering: Time series data is consistently ordered over intervals, whereas scattered data is randomly spaced, complicating holistic approaches.\n    4. Function Priors: Time series often exhibit cyclical and trending patterns, as underscored in [1] and [22]. Scattered data rarely has such clear priors, and when it does, its random distribution complicates application.\n\n3. **Differences between our HINT and N-Beats [22] and N-Hits [1]**:\n    1. On methodology: Both our method and [1] & [22] use a dual residual structure for signal decomposition. However, while [1] & [22] categorize time series signals into components like cyclical and trending, facilitating prediction, our HINT refines by capturing diminishing residual functions, aligning closer with function approximation.\n    2. Basic block implementation: The predictive blocks in [1] & [22], tailored for time series, harness its orderly and dense nature, using MLPs to predict signal base weights. Such networks are not ideal for scattered data interpolation. In contrast, our approach uses a modified Transformer encoder, leveraging self-attention to model correlation between observation and target points.\n    3. On hierarchical approach: [1] decomposes by sampling and interpolating the whole time series signals based on various cycle scales. Our hierarchy, rather than down-sampling scattered data, introduces local constraints during correlation modeling, diverging from [1]'s methods.\n\nWe hope this clarifies the distinctions between our work and prior art, emphasizing the novelty of our approach. We appreciate the opportunity to elaborate on our contributions and look forward to any further queries or comments you might have.\n\n**Comment 2:** Typos at line 83 and 94.\n\n**Response 2:** Thanks for pointing out the typos.\n\n1. The end of line 83 will be corrected from \"the positions of the observed points\" to \"the values of the observed points\".\n  \n2. We'll rectify the redundancy in Line 94.\n\nApologies for any oversight. We'll revisit the manuscript for thoroughness and clarity.\n\n\n**Comment 3:** Lack of experiments to investigate the performance of the proposed method with varying ratios of observed points.\n\n**Response 3:** Thank you for raising this valid concern.\n\nWe concur that evaluating the performance of our method with varying ratios of observed points would enrich our validation process. We have conducted supplementary experiments on the Mathit-2D dataset to gauge the impact of different numbers of observation points on interpolation accuracy, as depicted in the figure in [global response](https://openreview.net/forum?id=8d9wVXri89&noteId=RMsTw2cPZh).\n\nThese findings further underscore the superiority of our approach. Not only does it excel in terms of average interpolation accuracy, but it also outperforms existing interpolation algorithms across all levels of observed point numbers. These results will be appended to the supplementary materials for further clarity.\n\n**Comment 4:** Motivation of the proposed hierarchical residual refining framework is not clear, explain the motivation of hierarchical residual refining framework from the perspective of experiments\n\n**Response 4:** Thank you for inquiring about the motivation of our hierarchical residual refining framework.\n\n1. Experimentally, Sec.4.2's Residual Visualization (on Pages 7 and 8) offers insight, especially Fig. 4. It depicts the outputs of each block, their cumulative effects, and the error map on a Perlin test set interpolation task.\n\n    1. The Main block primarily captures the interpolation task's essence but misses finer details. Subsequent Residual Blocks progressively capture increasing detail levels, with the third block being especially subtle.\n \n    2. As we layer the residual predictions, the cumulative output more closely aligns with the ground truth (refer to Fig. 3's left column), diminishing the error map.\n\n2. Theoretically, scattered data interpolation often involves intricate details. Our framework, based on residual prediction, segments the interpolation task, easing the challenge. This tiered approach better captures the target function, outperforming previous techniques by fully utilizing information of observed point values, enhancing accuracy.\n\n---\n\n[1] Cristian Challu, et al. N-hits: Neural Hierarchical Interpolation for Time Series Forecasting. arXiv preprint arXiv:2201.12886, 2022\n\n[22] Boris N Oreshkin, et al. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437, 2019."}]

Author rebuttal:
### Supplementary Result to Reviewer WnAW

We evaluated the impact of varying observed point counts on interpolation accuracy using the Mathit-2D dataset, as shown in the supplementary figure. On Mathit-2D, both our HINT method and other techniques exhibit a marked reduction in interpolation error (MSE) as the number of observed points increases. Notably, HINT outperforms other methods across all observed point settings, underscoring its superior performance.

Meta-review: The paper introduces HINT, a novel hierarchical residual refining framework for interpolation tasks. Reviewers appreciate its unique aspects, including the use of initial and subsequent blocks for estimation and the application of residual methods to interpolation. The paper's clarity and comprehensive experimental evaluations are strengths. However, concerns about novelty, method simplicity, lack of comparisons, and experimental limitations were raised. After the rebuttal, additional comparisons and clarifications have effectively addressed these concerns. The reviewers now recommend acceptance, given the method's effectiveness while maintaining efficiency, demonstrating practical potential. We have decided to accept this submission after thorough discussion. We encourage the author to thoughtfully incorporate reviewers' feedback into the final version, particularly regarding the clarification of related works.
Decision: Accept (poster)