NIERT: Accurate Numerical Interpolation through Unifying Scattered Data Representations using Transformer Encoder
Authors: Shizhe Ding
Dongbo Bu
Venue: Submitted to ICLR 2023 (2023)
Keywords: numerical interpolation
transformer encoder
mask mechanism
pre-training model

Abstract: Numerical interpolation for scattered data, i.e., estimating values for target points based on those of some observed points, is widely used in  computational science and engineering. The existing approaches either require explicitly pre-defined basis functions, which makes them inflexible and limits their performance in practical scenarios, or train neural networks as interpolators, which still have limited interpolation accuracy as they treat observed and target points separately and cannot effectively exploit the correlations among data points. Here, we present a learning-based approach to numerical interpolation for scattered data using encoder representation of Transformers (called NIERT). Unlike the recent learning-based approaches, NIERT treats observed and target points in a unified fashion through embedding them into the same representation space, thus gaining the advantage of  effectively exploiting the correlations among them. The specially-designed partial self-attention mechanism used by NIERT makes it escape from the unexpected interference of target points on observed points. We further show that the partial self-attention is essentially a learnable interpolation module combining multiple neural basis functions, which provides interpretability of NIERT. Through pre-training on large-scale synthetic datasets,  NIERT achieves considerable improvement in interpolation accuracy for practical tasks. On both synthetic and real-world datasets, NIERT  outperforms the existing approaches, e.g., on the TFRD-ADlet dataset for temperature field reconstruction, NIERT achieves an MAE of $1.897\times 10^{-3}$, substantially better than the state-of-the-art  approach (MAE: $27.074\times 10^{-3}$).  The source code of NIERT is available at  https://anonymous.4open.science/r/NIERT-2BCF.

Under review as a conference paper at ICLR 2023
NIERT: ACCURATE NUMERICAL INTERPOLATION THROUGH UNIFYING SCATTERED DATA REPRESENTA- TIONS USING TRANSFORMER ENCODER
Anonymous authors Paper under double-blind review
ABSTRACT
Numerical interpolation for scattered data, i.e., estimating values for target points based on those of some observed points, is widely used in computational science and engineering. The existing approaches either require explicitly pre-defined ba- sis functions, which makes them inflexible and limits their performance in prac- tical scenarios, train neural networks interpolators, which still have lim- ited interpolation accuracy as they treat observed and target points separately and cannot effectively exploit the correlations among data points. Here, we present a learning-based approach to numerical interpolation for scattered data using encoder representation of Transformers (called NIERT). Unlike the recent bearning-based approaches, NIERT treats observed and target points in a unified fashion through embedding them into the same representation space, thus gain- ing the advantage of effectively exploiting the correlations among them. The ppeciall-designed partial self-attention mechanism used by NIERT makes it es- cape from the unexpected interference of target points observed points. We further show that the partial self-attention essentially a lcarnable interpolation module combining multiple neural basis functions, which provides interpretability NIERT. Through pre-training on large-scale synthetic datasets, NIERT achieves considerable improvement in interpolation accuracy for practical tasks. On both synthetic and real-world datasets, NIERT outperforms the existing approaches, e.g., the TFRD-ADlet dataset for temperature field reconstruction, NIERT achieves an MAE of 1.897 x 10-3, substantially better than the state-of-the-art approach (MAE: 27.074 x 10 3) The source code of NIERT is available https::/ http:////nooomoouusoo 4open. science/ 2BCF
INTRODUCTION
Scattered data consist of collection of points and corresponding values, in which the points have no structure besides their relative positions (Franke & Nielson 1991). Scattered data arise naturally and widely from large variety of theoretical and practical scenarios, including solving partial dif- ferential equations (PDEs) (Franke & Nielson 1991 Liu 2016), temperature field reconstruction (Chen et al.) 2021). and time series interpolation (Lepot al. 2017 Shukla & Marlin 2019). These scenarios usually require numerical interpolation for scattered data, i.c., estimating values for tar- get points based on those of some observed points. For example, in the task of temperature field reconstruction for micro-scale electronics, interpolation methods are used to obtain the real-time working environment of electronic components from limited measurements, and imprecise interpo- lation might significantly increase the cost of predictive maintenance. Thus, accurate approaches to numerical interpolation are highly desirable.
A large number of approaches have been proposed for interpolating scattered data. Traditional ap- proaches use schemes that approximate the target function by a linear combination of some basis functions (Heath, 2018), in which the basis functions should be explicitly pre-defined. To adapt to different scenarios, various types of basis functions have been devised. These schemes can theoret- ically guarantee the interpolation accuracy when sufficient observed points available; however, they have also been shown to be ineffective for sparse data points (Bulirsch et al. (2002). In addition, the schemes can hardly learn from the experience of interpolation in similar tasks.

---

Under review as a conference paper at ICLR 2023
Recent progress has exhibited alternative strategy that uses neural networks to learn target fune- tions directly from the given observed points. For example, conditional neural processes (CNPs) (Garnelo et al. 2018) and their extensions (Kim et al. 2019; Lee et al., 2020b model the condi- tional distribution of regression functions given the observed points, and Chen et al. (2021) proposed to use vanilla Transformer (Vaswani al. 2017) solve interpolation task in temperature field re- construction. All of these approaches use an "encoder-decoder" architecture, in which the encoder learns the representations of observed points while the decoder estimates values for target points. Ideally, observed points and target points should be processed in unified fashion because they are from the same domain. However, these approaches treat them separately and cannot effectively exploit the correlation between them.
Here, we present an approach to numerical interpolation that can effectively exploit the correlations between observed points and target points. Our approach is barning-based approach using the encoder representations of Transformers (thus called NIERT). The key elements of NIERT include: i) the use of mask mechanism, which complements target points with learnable mask tokens as performed by masked language model, say BERT (Devlin et al., 2018), and thus enables process- ing both observed and target points in unified fashion, ti) novel partial self-attention model that calculates attentions among the given data points except for the influence of target points onto the ob- served points each layer, thus gaining the advantages of exploiting the correlations between these two types of points and, more importantly, avoiding the unexpected interference of target points on observed points at the same time, and iii) the use of the pre- vre-training technique, which leverages large-scale low-cost synthetic data to build powerful, general-pune purpose, and transferable pre-trained interpolation models. The use of the pre- pre-trained models significantly improves generalization ability and interpolation accuracy.
The main contributions of this study are summarized follows.
We propose an accurate approach to numerical interpolation for scattered data. On rep- resentative datasets including synthetic and real-world datasets, our approach outperforms the state-of-the-art approaches and shows potential a wide range of application fields.
We propose novel partial self-attention mechanism to make Transformer incorporated with strong inductive bias for interpolation tasks, i.e., it can effectively exploit the correla- tion among two types of points and, at the same time, avoids the interference of one type of points onto the others.
We also demonstrate the essence of NIERT, i.e., learnable interpolation approach ing neural basis functions, through illustrating the deep connection between partial self- attention mechanism and traditional interpolation approaches.
To the best of our knowledge, this study is the first work to propose the pre-trained models for scatter-data interpolation. We have verified that such interpolation pre-trained models can be generalized to a wide range of interpolation tasks.
2 RELATED WORKS
2.1 TRADITIONAL INTERPOLATION APPROACHES FOR SCATTERED DATA
Traditional interpolation approaches for scattered data use explicitly pre-defined basis functions to
construct interpolation function, e-g., Lagrange interpolation, Newton interpolation (Heath) 2018)) B-spline interpolation (Hall & Meyer 1976). Shepard's method (Gordon & Wixom 1978). Kriging (Wackernagel 2003), and radial basis function interpolation (RBF) (Powell 1987; Forbberg & Zuev 2007). Among these approaches, the classical Lagrange interpolation, Newton interpolation and B- splines interpolation are usually used for univariate interpolation. [Wang et al. (2010) proposed a high-order multivariate approximation scheme for scattered data sets, in which approximation error is represented using Taylor expansions at data points, and hasis functions are determined through minimizing the approximation error.
2.2 NEURAL NETWORK- -BASED INTERPOLATION APPROACHES
Equipped with deep neural networks, data-driven interpolation and reconstruction methods show great advantages and potential. For instance, convolutional neural networks (CNNs) have been ap-
2

---

Under review as a conference paper at ICLR 2023
Loss
5/
Norm
Function distribution
Par PNtrEatii
NIERT INTERPOLATOR
Add Norm
Srutts and Portal Selalttnnion
Partial selt-attention
Meshes toksnn
1/
Embeddinng Layer
NIERT INTERPOLATOR
Sampled Interpolation tasks
Observed paints
Torget points
Figure 1: Overview of NIERT training process. Here, I; represents the position of a point, and y represents its value. The predicted values of the point is denoted as yú. We prepare the training interpolation tasks by first sampling functions from function distribution FF and then sampling observed points O and target points ' on each function. NIERT trains an interpolator over this data. The partial self-attention mechanism facilitates exploiting the correlations between observed points and target points and avoiding the unexpected interference of target points on observed points
plied in the interpolation tasks of single image super-resolution (Tai et al. 2017 Li et al. 2021), and recurrent neural networks (RNNs) and Transformers have been used for interpolation of sequences like time series data (Shukla & Marlin 2019; 2021).
Recently, Garnelo et al. 2018) proposed to model the conditional distribution of regression func- tions given observed points. The proposed approach, called conditional neural processes (CNPs), has shown increased estimation accuracy and generalizing ability. variety of studies have been performed to enhance CNPs: Kim et al. (2019) designed attentive neural processes (ANPs), which shows improved accuracy. Lee et al. (2020a) leveraged Bayesian last layer (BILL) (Weber et al., addition, the bootstrap technique was also em- ployed for further improvement (Lee et 2020b). To solve the interpolation task in 2D temper- ature field reconstruction, Chen et al. (2021) proposed a Transformer-ban bassdd approach, referred to as IFF-trranffrmmer which can also be applied to solve interpolation tasks for scattered data with higher dimensions.
2018) for faster training and better prediction.
2.3 MASKED LANGUAGE//MAGE MODELS AND THE PRE-TRAINING TECHNIQUE
The design of NIERT is also inspired by the recent advances in masked language/image models (Devlin et al. 2018 Bao et al. 2021 Brown et al. 2020: He al. 2021) and pre-trained models for symbolic regression (Biggio et al. 2021 Valipour al. (021). The mask mechanism enables the models to reconstruct missing data from their context and effectively learn representations of language and images. In addition, the pre-trained models for symbolic regression were designed to learn the map from scattered data to corresponding symbolic formulas (Biggio et al. 202 Valipour et al.) 2021). Here, NIERT uses the mask mechanism to reconstruct the missing values of target points and uses the pre-training technique to learn the experience of interpolation on similar tasks.
3 METHOD
3.1 OVERVIEW OF NIERT
In the study, focus the interpolation task that can be formally described as follows: We
given n observed points with known values = {(x,, 9:) } and m target points with values to be determined, denoted as T {x, [
n+m finn += Here, F; E denotes position of a point, 3/1 = fa;) e Y denotes the value of point, and : X - Y denotes function mapping positions to values. The function f is from a function distribution F, which can be explicitly defined using a mathematical formula or implicitly represented using a of scattered data in the form (æ,, yt). The goal of the
3

---

Under review as a conference paper at ICLR 2023
interpolation task is to accurately estimate the values f(x) for each target point E T according to the observed points in O.
The main element of our NIERT approach is a neural interpolator that learns to estimate values for target points. We train the neural interpolator using a set of interpolation tasks sampled from the function distribution F (Fig. TT. For each of the sampled interpolation tasks, mask the values of target points, feed the task to the neural interpolator, and train the interpolator to predict the values of target points and observed points well. The training objective is to minimize the error between the predicted values and the ground-truth values of both target and observed points. The trained interpolator can be used interpolate values of target points for the interpolation tasks from the same function distribution.
3.2 ARCHITECTURE OF THE NIERT INTERPOLATOR
The neural interpolator NIERT adopts the Transformer encoder framework; however, to suit the interpolation task, significant modifications and extensions were made in the embedding, Trans- former and output layers, which are described detail below.
Embedding with masked tokens: NIERT embeds both observed points and target points into the unified high-dimensional embedding space. As the position x of a data point and its value are from different domains, use two linear modules Linear, embeds the positions while Lincar, embeds the values.
should be noted that for target points, their values are absent when embedding they to be determined. In this case, we use a masked token as substitute, which is embedded as trainable parameter MASK,, as performed in BERT (Devlin et al., 2018)). This way, the interpolator processes both target points and observed points in a unilying fashion.
We concatenate the embeddings of position and value of a data point the point's embedding, denoted as h2, i.e.,
[L.n Linear, (w:),, if (2,, E h [Linear,xr.), MASK,). g)), if if T
Transformer layer with partial self-attention mechanism: NIERT feeds the embeddings of the points into a stack of L Transformer layers, producing encodings of these points as results. Each Transformer layer contains subsequent sub-layers, namely, multi-head self-attention module, and point-wid wwie fully-connected network. These sub-layers are interlaced with residual connections and layer normalization between them.
To avoid the unexpected interference of target points on observed points and target points them selves, NIERT replaces the original self-attention in Transformer layer with partial self-attention, which calculates the feature of point at the l + 1-st layer as follows:
h 4 LayerNorm( ( + MLP(())).
where of LayerNorm oof + E, wijakyy)
Here, of and a represent the value vector and ordinary attention weights at the /-th layer as cal- culated in Transformer /Vaswani al. 2017). We use an additional term wy to represent which entries of attention weights are neglected in the partial self-attention mechanism, i.c.,
1, if (1), E
wy = 0. if ff ET
For each observed point (Iz, vy) e O, we set wij 1; thus, NIERT can model the correlation between observed points and target points, and the correlation among observed points as well. Rel- atively, the correlation among observed points is easier to learn these points have known values. In addition, this correlation can be transferred onto the target points, thus promoting learning repre- sentations of these data points.
In contrast, by forcing the weight Wiy to be for a target point and any point j, we completely avoid the unexpected interference of target points the other points.
4

---

Under review as a conference paper at ICLR 2023
Estimating values for target points: For each target point i, we estimate its value yi through feeding its features at the final Transformer layer into a fully connected feed-forward network, i.c.,
We calculate the error between the estimation and the corresponding ground-truth value, and com- pose the errors for all points into a loss function to be minimized.
3.3 ENHANCING NIERT USING PRE-TRAINING TECHNIQUE
The existing data-driven interpolation approaches. although powerful, still suffer from the short- comings of limited generality and transferability. To overcome these shortcomings, we propose to use the pre-training technique to build NIERT into pre-trained interpolation models. Due to the lack of large-scale pre-training datasets for interpolation, we constructed the NeSymReS dataset, which contains a large variety of synthetic symbolic mathematical functions (see [.] for further details). These functions approximate general and diverse function distribution, thus gaining our pre-trained model strong generalization ability. Here, we pre-train NIERT using NeSymReS dataset and fine-tune it other datasets in practical scenarios.
3.4 DEEP CONNECTION BETWEEN NIERT AND CLASSICAL INTERPOLATION METHOD
The underlying rational of using partial self-attention mechanism is rooted in its deep connection with classical interpolation methods. We use RBF as representative of classical methods illus- trate this connection below.
To interpolate n observed points O (:)s,, the RBF approach uses the following interpo- lation function
(RBF(x)= = A10(2, 22).
Here, 2x5) represents radial basis function specified by the observed point 2j, and X is the coefficient, which can be determined through solving the linear equations Σy=1 )50(2, xz) = yis
(I;, M1) E O (Sooooon 2015).
We further rewrite the partial self-attention shown in Eq. 2 with simplification but without loss of its essence as:
S (ar) - - ((g((),)))))
(4)
Here, a(q(1), ky) is the normalized attention weight that models the contribution to query vector ((x) by the key vector ky, in which k; and uy are dominated by the observed point Ty-
Comparing Eq. 3 with Eq. 4 we can find that partial self-attention is a general form of RBF interpolation function: Xy in Eq. corresponds to oy in Eq. (4), and ø(x, a.)) in Eq. 33 corre- sponds to (g(()), k;) in Eq. (4). This correspondence indicates that ar (rg(zz) k;) can be treated as learnable basis function and, similarly, U, can be treated as a predictable basis function coefficient. From this point of view, partial self- celf-attention is learnable layer that interpolates the representation of observed points to yield new representations of data points. Together, this insight provides a plau- sible explanation of the partial self-attention mechanism, which is also supported by the contribution analysis of observed points shown Section 4.4
It should be pointed out that the decoder of UFF-traanformer also be rewritten into formula similar to Eq. (4) however, it still differs greatly from the partial self-attention mechanism: par- tial self-attention encodes both observed and target points into the same feature space at each layer whereas tranfrranformer treats these two types of data points separately. This unifying represen- tation of data points gains NIERT the advantages of exploiting the correlations among data points more effectively.
4 EXPERIMENTS AND RESULTS
We evaluated NIERT and compared it with ten representative scattered data interpolation approaches on both synthetic and real-world datasets. We further examined the effects of the key elements of
5

---

Under review as a conference paper at ICLR 2023
NIERT, including the partial self-attention, and the pre-training technique. We also visualized the attention map to exhibit the contribution each observed point to the interpolation function.
4.1 EXPERIMENT SETTING
We evaluated NIERT on three synthetic datasets (NeSymReS, TFRD-ADlet, and D30) and real- world dataset (PhysioNet): N a synthetic dataset of up to 4 dimensions for mathematical function interpolation, TFRD-ADlet is synthetic dataset for 2D temperature field reconstruction, and D30 is a high dimension dataset that contains data points of up to 30 dimensions. PhysioNet is a real-world dataset collected from intensive care unit (ICU) records for time-series data interpolation.
In this study, the NeSymReS dataset was used for pre-training NIERT to further improve its interpo- lation accuracy on TFRD-ADlet and PhysioNet datasets. For the TFRD-ADlet dataset, we directly use the 2D NeSymReS dataset for pre-training. As the PhysioNet dataset has dependent variable with dimensionality of 37, we construct the pre-training instances by stacking random 37 functions from the ID NeSymReS dataset and then sampling interpolation task instances.
When evaluating NIERT and other interpolation approaches, the prediction error of target points calculated as interpolation accuracy. For the NeSymReS, D330 and PhysioNet datasets, we adopted mean squared error (MSE) as the error metric. For the TFRD-ADle dataset, we use three error met- rics: mean absolute error (MAE), MAE in the component area (CMAE) and MAE at the boundary (BMAE) following Chen et al. D). Accordingly, we use L2-form loss function for NeSymReS and PhysioNet dataset and L1 form loss function for TFRD-ADlet dataset for training.
We compared NIERT with state-of-the-aae approaches. Further details of datasets, approaches for comparison, and other experiment settings are provided in the Supplementary material.
4.2 INTERPOLATION ACCURACY ON SYNTHETIC AND REAL-WORLD DATASETS
For each instance of test dataset, we applied the trained NIERT to estimate values for target points. We calculate the errors between estimation and ground-truth interpolation accuracy.
Interpolation approach
MSE (x10 -5) NcSymReeS
Interpolation approach
Evaluntion criteria (x10
ID
2D
3D
4D
MAE
CMAE
BMAE
RBF
215.439
347.066
4433.254
327.775
CNP
96.674
109.419
$6,939
CNP
67.175
248.668
274.601
392.348
342.997
314.311
ANP
54.684
62,511
26.524
ANP
34.558
140.005
206.609
164.751
BANP
28.671
29.450
19.984
84.187
TTR-aranfformer
27.074
29.772
18.835
THR-transfommer
15.556
5X. 566
99.996
90.579
NIERT 3.473
3.947
2.467
NIERT
8.964
45.319
77.664
72,025
NIERT w/ pretraining
1,897
1.971
1.246
Table I: Interpolation accuracy of NIERT and existing approaches on NeSymReS dataset
Table 2: Interpolation accuracy of NIERT and
existing approaches over TFRD-ADlet dataset
Interpolation approach
MSE (x 1xx10m4) ") D30 test
Interpolatonn
Ratio of observed points
100)
200
30D
approach
50%
70%
90%
181.744
112.902
RNN-VAE
03.418=00000
11.470:10.006
MIR
161.474
155.035 155.038
92.278
LODE-RNN
8.132_0.020
8:171::0:000
8.402:1-00022
CNP
35.623
51.166
58.312
L-ODE-ODE
6.721=001099
6.798-10.145
7.142:10.066
BANP
12.077
13.010
17.422
mTTAND-Fall
NIERT
2.868=0.023
4.139=0.029
2.656:20.041
4.157=b00053
2.709-1:01157
4.798-10.000
TER-tranfoomer
NIERT
7.465
5.496
6.211
6.334
1.549
NIERT pretraining
2.831:00021
2.641::0.052
2.596:1-0.159
Table 4: Relationship between interpolation accuracy
Table 3: Interpolation accuracy of NIERT and existing approaches on D30 dataset
(MSE, x10- 33 on PhysioNet with observed ratio
Accuracy NeSymReS and D30 datasets: As shown in Table 1 the 1D NeSymReS test set. RBF shows the largest interpolation eтror (MSE: 215. 315.439). MIR, another approach using ex- plicit basis functions, also shows high interpolation error of 67.281 In contrast, BANP and TFR- transformer, which use neural networks to learn interpolation. show relatively lower errors (MSE: 14.913, 15.556). Compared with these approaches, our NIERT approach achieves the best interpo- lation accuracy (MSE: 8. 8.46... T demonstrates the advantage of NIERT over the existing approach on the 2D, 3D, and 4D instances.

---

Under review as a conference paper at ICLR 2023
10.1
MSE prediction of target points
To examine in depth the interpolation accuracy, we further divide test instances into subsets ac- cording to the number of observed points. As shown Figure 2 as the number of observed points increases, the interpolation emror decreases as expected. In addition, the relative advantages of these approaches vary with the number of ob- served points, e.g., CNP is better than RBF and MIR initially but finally becomes worse as the number of observed points increases. Among all approaches, NIERT stably shows the best perfor- mance over all test subsets, regardless of the num- ber of observed points.
10-
10
10
20
30
40
50
Number of observed points
Figure 2: The relationship between the inter- polation accuracy and the number of observed points. Here we use the 2D instances in the NeSymReS test dataset as representatives
Table suggests that even for D30 dataset with high-dmmensional (up to 30 dimensions) data points, NIERT still shows excellent interpolation accuracy and outperforms all existing methods, which demonstrates the scalability of NIERT to the high-dimensional interpolation tasks.
Accuracy on TFRD-ADlet and PhysioNet datasets: As shown in Table CNP, although employ- ing the neural network technique, still performs poorly with MAE as high as 96.674. In contrast, order of magnitude
NIERT achieves the lowest interpolation error (MAE: 3.473), which is lower than CNP, ANP, BANP and trrtrranfforrr. Moreover, when enhanced with the pre-training technique, NIERT can further decrease its interpolation MAE 1.897. Besides MAE, other metrics, say CMAE and BMAE, also show the superior of NIERT over the existing approaches (Table 2
Table suggests that on the PhysioNet dataset, NIERT also outperforms the existing approaches, e.g., when controlling the ratio of observed points to be 50%, NIERT achieves an average MSE (x10-3) of 2.868, significantly lower than other approaches. Again, NIERT with the pre-training technique shows better performance. The advantages of NIERT hold across various settings of the ratio of the observed points.
Taken together, these results demonstrate the power of NIERT for numerical interpolation in multiple application fields, including interpolating the scattered data generated using mathematical functions, reconstructing temperature fields, and interpolating time-series data.
4.3 CASE STUDIES OF INTERPOLATION RESULTS
To further understand the advantages of NIERT, we carried out case studies through visualizing the observed points, the reconstructed interpolation functions and the interpolation errors in this subsection. More visualized cases are put in Section D.I in the Supplementary material.
Figure 3: An example of 2D interpolation task extracted from NeSymReS test set. The up-left figure shows the ground-truth function while the bottom-left figure shows the 22 observed points. The interpolation functions reported by NIERT and the existing approaches are listed on the top panel with their differences with the ground-truth are list below
Figure show 2D instance in the NeSymReS test set, respectively. As illustrated, RBF performs poorly in the application scenario with sparse observed data. In addition, RBF and MIR, especially ANP, cannot accurately predict values for the target points that fall out of the range restricted by observed points. The CNP approach can only learn the rough trend stated by the observed points, thus leading significant emrrs. In contrast, BANP, IrR-trranfformr and NIERT can accurately
7

---

Under review as a conference paper at ICLR 2023
estimate values for target points within considerably large range, and compared with BANP and TFR-trans tra-tranfformrr, NIERT can produce more accurate results.
AAP Lec
100
15
Figure 4: An example of temperature field reconstruction task extracted from TFRD-ADlel test set. The up-left figure shows the ground-truth temperature field while the bottom-left figure shows the 32 observed points. The reconstructed results reported by NIERT and the existing approaches are listed on the top panel with their differences with the ground-trune truth temperature field are list below
Figure shows an instance of temperature field reconstruction extracted from TFRD-ADlet. From this figure, observe that when using the pre-training technique, NIERT further improves its interpolation accuracy in the whole area.
4.4 CONTRIBUTION ANALYSIS OF OBSERVED POINTS FOR INTERPOLATION
idcalized interpolation approach is expected to effectively exploit all observed points with ap- propriate consideration of relative positions among observed points and target points well. To examine this issue, we visualized the attention weight, i.e., ((g(z), k2) in Eq. (4 of each observed point to all target points. These attention weights provide an intuitive description of the contribution by observed points, which also represents the learned neural basis functions as stated in Section 3.4
sitere Staric MERI JINNCO
Osear paic
scardin
(a)
Figure 5: Contributions by observed points for interpolation. The 2D instance is that used in Figure We randomly select observed points and extract their attention weights from the final attention layer of NIERT and TFR-transformer. The contributions by other 18 observed points are shown in Section D.2 in the Supplementary material
As shown in Figure when using trF-ttransformer, the contributions by observed points are consid- side, some observed points (e.g.. Fig. (aa) might affect their neighboring
erably imbalanced: target points in large region; on the other side, the other observed points (c.g., Fig. (bb) have little contributions interpolation. In contrast, when using NIERT, contributions by an observed point are much more local and thus targeted. More importantly, all observed points have contributions to interpolation. These results demonstrate that NIERT can exploit the correlation between observed points and target points more effectively.
4.5 ABLATION STUDY
The effects of partial self-attention: For a specific interpolation task, the interpolation function is determined by the observed points only. To investigate the effects of partial self-attention in avoiding interference of target points, we evaluated NIERT on the test sets with various numbers of target points. Here, compared NIERT with its two variants, one using vanilla self-attention, and the other using partial self atffaatention with the target points' correlation added. Both them were trained using the same training (the number of target points varies within [206, 246)).
8

---

Under review as a conference paper at ICLR 2023
As illustrated by Figure 6 the two NIERT variants show poor performance for the tasks with few target points, say less than 64 target points or more than 768 target points. In contrast, NIERT, which uses partial self-attention, always performs stably without significant changes in accuracy.
The results clearly demonstrate that the partial self-attention mechanism allows NIERT to be free from the unexpected effects of the target points.
CAP
Surae DOEKZ soraiaticn
atreation
val wwr.ddnnonn
on 1D hadynttud
MDT
te-1
3 Number target polnts
Thiining eportss
The robustness of NIERT to the number of target points. Here, NIERT using three attention mechanisms trained on the ID NeSymReS dataset
Figure 6:
The convergence of NIERT. NIERT with pre-training and the existing ap- proaches. Here, these models are trained on TFRD-ADlet dataset
Figure 7:
The effects of pre-training technique: To investigate the effects of the pre-training technique, we show in Figure 7 the training process of two versions of NIERT, one without the pre-training technique, and the other enhanced with pre-training- As depicted by the figure, even at the first epoch, the pre-trained NIERT shows a sufficiently high interpolation accuracy, which is comparable with the fully-trained BANP and TFR-transformer. Moreover, the performance of the pre-trained NIERT improves at roughly the same convergence speed as the original NIERT. At the final epoch, the pre-trained NIERT decreases the interpolation error to nearly half of that of the original NIERT.
These results clearly suggest that the experience learned by NIERT from the interpolation task in one application field has the potential to be transferred to the tasks in other application fields.
5 DISCUSSION AND CONCLUSION
We present in the study an accurate approach to numerical interpolation for scattered data, The specific features of our NIERT approach are highlighted by the full exploitation of the correlation between observed points and target points through unifying scattered data representation. At the same time, the use of the partial self-attention mechanism can effectively avoid the interference of target points onto the observed points. The enhancement with the pre-training technique is another special feature of NIERT. The advantages of NIERT in interpolation accuracy have been clearly demonstrated by experimental results on both synthetic and real-worke datasets. Analysis suggests that the advantages of NIERT mainly come from the fact that both observed and target points are embedded into the same feature space at each layer, which makes the message passing from observed points to target points more efficient and informative.
The current version of NIERT has computational complexity of O(n(m + n:),, thus cannot handle the interpolation tasks with extremely large amounts of observed points due to the limitations of GPU memory size. Compared with the lightweight traditional methods, our NIERT approach has a larger model with expensive computation to learn complex function distribution, which limits its application in cost-sensitive scenarios.
One possible strategy to improve NIERT's efficiency is sparse partial self-attention: As shown in Figure 5| the contribution by observed points is quite local and targeted. Thus, for target point, its neighboring observed points dominate the estimation of its value. Considering the neighborhood only might greatly reduce the computation cost. We will investigate this strategy in future studies.
We expect NIERT, with extensions and modifications, to greatly facilitate numerical interpolations in wide range of engineering and science fields.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
In the Supplementary material, we provide the anonymous link to the source code for the key ex- periments. Instructions on how generate data and train the models are described in detail in the code base. We also provide the construction method or source of each dataset and list the details of experimental configurations. We have fully checked our implementation and verified that our method is reproducible through sufficient experiments.
REFERENCES
Hangbo Bao, Li Dong, and Furu Wei. BEIT: BERT Pre-Training of Image Transformers. arXiv preprint arXiv:2106.082:: 2021.
Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascan- dolo. Neural Symbolic Regression that Scales. In International Conference on Machine Learning, pp.
PMLR, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Mlererr-Vosss Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Iyya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Roland Bulirsch, Josef Stoer, and J Stocr. Introduction to numerical analysis, volume 3. Springer, 2002.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Dif- ferential Equations. Advances in Neural Information Processing Systems, 2018.
Xiaoqian Chen, Zhiqiang Gong, Xiaoyu Zhao, Weien Zhou, and Wen Yao. A Machine Learning Modelling Benchmark for Temperature Field Reconstruction of Heat-Source Systems. arXiv preprint 00290220088.,298, 2021.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048005 2018.
Bengt Fornberg and Julia Zuev. The Runge phenomenon and spatially variable shape parameters in
RBF interpolation. Computers & Mathematics with Applications, 54(3):379-393 2007.
Richard Franke and Gregory M Nielson. Scattered data interpolation and applications: A tutorial and survey. Geometric Modeling, pp. 131-160, 1991.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional Neural Processes. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1704-1713. PMLR, 10-15 Jul 2018.
William J Gordon and James A Wixom. Shepard's method of "metric interpolation" to bivariate and multivariate interpolation. Mathematics of computation, 32(141):253-266) 1978.
Charles A Hall and W Weston Meyer. Optimal error bounds for cubic spline interpolation. Journal of Approximation Theory, 1652):055-12,, 1976.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
10

---

Under review as a conference paper at ICLR 2023
Michael T Heath. Scientific Computing: An Introductory Survey, Revised Second Edition. Society for Industrial and Applied Mathematics, revised 2nd edition edition, 2018.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive Neural Processes. In International Conference on Learning Representations, 2019.
Byung-Jun Lee. Seunghoon Hong. and Kee-Eung Kim. Residual neural processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.
2020a.
Juho Lee, Yoonho Lee, Jungtaek Kim, Eunho Yang, Sung Ju Hwang, and Yee Whye Teh. Boot- strapping neural processes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6606-6615. Curran Associates, Inc., 2020b.
Mathieu Lepot, Jean-Baptiste Baptiste Aubin, and François HLR Clemens. Interpolation in time series: An introductive overview of existing methods, their performance criteria and uncertainty assessment. Water, 910)) 2017.
Juncheng Li, Zehua Pei, and Tieyong Zeng. From Beginner to Master: A Survey for Deep Learning- based Single-Image Super-Resolution arXiv preprint arXiv:2109. /4335, 2021.
GR3543275 Liu, An overview on meshfree methods: for computational solid mechanics. Interna- tional Journal of Computational Methods, 13(05))1630001, 2016.
Michael JD Powell. Radial basis functions for multivariable interpolation: a review. Algorithuss for approximation, 1987.
Yulia Rubanova, Ricky T. Q. Chen, and David K Duvenaud. Latent Ordinary Differential Equations for Seggularly-Sampler Time Series. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019.
Satya Narayan Shukla and Benjamin Marlin. Multi-Time Attention Networks for Irregularly Sam- pled Time Series. In International Conference on Learning Representations, 2021.
Satya Narayan Shukla and Benjamin M Marlin. interpolation-prediciiie networks for irregularly sampled time series. arXiv preprint 179.02190. 2019.
Ikaro Silva, George Moody, Daniel Scott, Leo A Celi, and Roger G Mark. Predicting in to-hossptal mortality of icu patients: The physionet/comppting in cardiology challenge 2012. In 2012 Com- puting in Cardiology, pp. 245-248 248. IEFE, 2012.
Justin Solomon. Numerical algorithms: methods for computer vision, machine learning, and graph- ics. CRC press, 2015.
Ying Tai, Jian Yang. and Xiaoming Liu. Image super-resolution via deep recursive residual network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3147- 3155, 2017.
Mojtaba Valipour, Bowen You, Maysum Panju, and Ali Ghodsi. SymbolicGPT: A Generative Trans- former Model for Symbolic Regression. arXiv preprint arXiv:2 14,31,1122,2,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Pauli Virtanen, Ralf Gommers, Travis F. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nel- son, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nauur Methods, 17:261-272, 2020. doi: 10. 1038/44 592-019--666--22
11

---

Under review as a conference paper at ICLR 2023
Hans Wackernagel. Multivariate geostatisties: an introduction with applications. Springer Science & Business Media, 2003.
Qiqi Wang, Parviz Moin, and Gianluca Iaccarino. A high order multivariate approximation scheme for scattered data sets. Journal Computational Physics, 229(18))3343-3661 2010.
Noah Weber, Janez Stare, Arpit Mittal, Roi Blanco, and Lluís Märquez. Optimizing over bayesian last layer. In NeurlPS workshop on Bayesian Deep Learning, 2018.
12

---

Under review as a conference paper at ICLR 2023
SUPPLEMENTARY MATERIAL
TABLE OF CONTENT
DETAILS OF IMPLEMENTATION AND EXPERIMENTS
14
A.I AVAILABILITY OF THE SOURCE CODE OF NIERT
14
A.2 HYPERPARAMETER SETTINGS
14
A.3 TRAINING ALGORITHM
14
DETAILS DATASETS
16
B. I SYNTHETIC DATASET I: NESYMRES
16
B 2 SYNTHETIC DATASET II:
17
B.3 SYNTUETIC DATASET III: D30
18
B. 4 REAL-WORLD DATASET: PHYSIONET
18
APPROACHES FOR COMPARISON
19
ADDITIONAL EXPERIMENTAL RESULTS
20
D.I ADDITIONAL CASE STUDIES
20
D.2 CONTRIBUTION ANALYSIS OF OBSERVED POINTS FOR INTERPOLATION
25
D.3 ADDITIONAL ABLATION STUDIES
25
D.4 PREDICTION ACCURACY GAP BETWEEN OBSERVED POINTS AND TARGET POINTS
26
D.5 COMPUTATIONAL TRADE-OFT
26
13

---

Under review as a conference paper at ICLR 2023
A DETAILS OF IMPLEMENTATION AND EXPERIMENTS
A.1
AVAILABILITY OF THE SOURCE CODE OF NIERT
We provide the details of implementation and experiments in this section and deposit source code of NIERT at anonymous. 4open. swwence///NIEIC 2BCF
A.2
HYPERPARAMETER SETTINGS
Table lists the the papaa-parameters of NIERT when running on the four representative datasets, including NeSymReS, TFRD-ADlet, D30 and PhysioNet datasets. For fair comparison, we set up the IFR-transformer with the same H and dmodei. and the number of its encoder layers is set to and that of its decoder set to following (Chen al. 2021
Parameter value in experiments on
Parameter name
Symbol
NeSymReS
TFRD-ADlet
D30
PhysioNet
Number of layers Hidden dimension Number of heads x's embedding dimension
L
6
3
6
3
dmodel H
512
128
128
128
8
4
4
dzemb dyemm
16 x dg
16x d2
16
y's embedding dimension
16
16
16
592
Table 5: Hyper-parameters of NIERT in experiments on the four representative dataset
For experiments on NeSymReS dataset, we trained NIERT and the neural approaches for comparison two NVIDIA GeForce RTX 3090 GPUs for 160 epochs with a batch size 128. Adam optimizer with a learning rate of 1.0 x 10-4 and no schedules are performed for parameters optimization.
Training on NeSymReS dataset:
For experiments on D30 dataset, NIERT and the neural approaches for comparison trained one NVIDIA GeForce RTX 3090 GPUs for 100 epochs with batch size of 128. Adam optimizer with a learning rate of 1.0 x 10-4 and no schedules are performed for parameters optimization.
Training on D30 dataset:
Training on TFRD-ADlet and PhysioNet datasets: For experiments on TFRD-ADlet dataset, we trained NIERT and the approaches for comparison on one NVIDIA GeForce RTX 3090 GPU for 100 epochs with a batch size of 5. Adam optimizer with learning rate of 5.0 x 10 with decay rate of 0.97 is performed. For experiments on PhysioNet dataset, we trained NIERT on one NVIDIA GeForce RTX 3090 GPU for 160 epochs with a batch size of 32. Adam optimizer with a learning rate of 5.0 10-4 and with a decay rate of 0.97 is performed.
Pre-training for TFRD-ADlet and PhysioNet datasets: Pre-trained NIERT on 2D NeSymReS is used fine-tune on TFRD-ADlet dataset. deal with the problem the data distribution too different between NeSymReS data and temperature field in TFle- simply normalize the data of temperature field to the value range of 2D NeSymReS data before fine-tuning.
To apply NIERT on PhysioNet dataset in which each instance has many dependent variables with a number of 37, we increase the number MLPs for dependent variables to 37 in the embedding layer to embed them separately. The pre-training instances for these tasks are correspondingly dif- ferent. For convenience, we stack 37 random ID functions in NeSymReS dataset as an instance for pre-training. Although the data generated using such construction method is considerable dif- ferent from PhysioNet data, we still observe that pre-training technique improves the interpolation accuracy as shown in Table 4
A.3 TRAINING ALGORITHM
The training process is depicted in Algorithm I Specifically, we prepare the training interpolation tasks by first sampling functions from a distribution FF and then sampling observed points and target points T each function. When training NIERT, set the loss function as the error between
14

---

Under review as a conference paper at ICLR 2023
Algorithm NIERT training process
Require: NIERT model Ma with parameters O, epoch number N, batch size B, domain X, function distribution J. and error metric function Error(.,)
for & in {1..N) do
x- for b in {1..B} do
f, [.i)s < sample a function and scatter points from F,X X
{y:}: calculate on {xi);
0, + split and mask scatter points with values {(24, 9:)
{ú: 7.)
J +- J Σ, Error (1s, yi)
end for Compute the gradient VnJ and update 0 end for
the estimated values and the corresponding ground truth. Note that errors acquired on both observed points and target points accounted into loss function.
15

---

Under review as a conference paper at ICLR 2023
B DETAILS OF DATASETS
We evaluated NIERT approach using four datasets, including synthetic datasets NeSymReS, TFRD- ADlet and D30, and real-world dataset PhysioNet. We first clearly list the statistics of interpola- tion tasks of these datasets and then introduce the datasets in detail in the following subsections.
The statistical information of the interpolation tasks used for training in each dataset is shown in Table 6 In practice, for each instance in the training set, i... N scattered points from a certain function, we randomly select of them observed points and the remaining N n target points. For the NeSymReS dataset, we randomly select a fixed range of numbers as the number of observed points (e.g. [5.25] for ID NeSymReS dataset). For the real-world PhysioNet dataset, which has variable number of scattered points for its instances, distributed in the range of [20, 190], we therefore select observed points from all points a fixed rate (say 50% or 90%).
Dataset
Subset tag
# All points N #Observed points
#Target points
ID
256
[10,50]
N
NeSymReS
2D/3D/4D
512
[10,50]
N n2
TFR-ADlet
40000
37
39963
D30
10D/20D/30D
256
64
192
50%
[20,190]
0.5xNN
N
PhysioNet
70%
[20,190]
0.7x
N 72
90%
[20,190]
0.9x
N n2
Table 6: Statistics of the interpolation tasks used for training in each dataset
B. SYNTHETIC DATASET I: NESYMRES
We used the NeSymRes" dataset to test the performance of NIERT for mathematical function inter- polation. This dataset was constructed following Biggio al. (2021) with only slight modification for the sake of numerical interpolation. For clarity, we put the details of construction procedure in the subsection B.I.I below, and list the summarized characteristics of this dataset as follows.
Instance construction: We randomly sampled N points from X = []1, (dz represents the point dimensionality) and calculate the values for these points with mathematical function f. From these points, we randomly picked n points as observed points and used the left-over as target points. In the study, we chose different d, for constructing datasets, including ID, 2D, 3D, and 4D. For ID NeSymReS dataset, we set the scattered points number N as 256. For 2D/3D/4D NeSymReS dataset, we set N as 512. We control observed points number within the range of [5, 50).
Training set and test set: Each instance in training or test set was built using a mathematical function f. These functions were generated using a function sampler as performed in (Biggio al.) 2021]. During the training process, million instances are dynamically sampled at each epoch as training set. We used the L-zformm loss function for this dataset. We also generated 12000 instances and used them as test set.
Dimensionality of points: We evaluated NEIRT using seatter data with various dimensionality, including ID, 2D, 3D, and 4D.
Approaches for comparison: We compared NIERT with five representative interpolation ap- proaches, including radial basis function (RBF) (Powell) 1987; Fornberg & Zuev [007]. MIR (Wackernagel) 2003). conditional neural process cess (ANP)(Kim et al. 2019) and TFRRttranformme (Chen et al. 2021). 1). Among these approaches, RBF and MIR are classical approaches that use explicit basis functions, while CNP, ANP and TFR- transformer use neural networks to learn how interpolate.
Garnelo et al. 2018), attentive neural pro-
Built based on http://wwww.....iiinonnainaaainn
16

---

Under review as a conference paper at ICLR 2023
B.1.1 NESYMRES DATASET CONSTRUCTION
Following the study of Biggio et al. (202T), firstly, we generate equation skeletons which is refer to the symbolic equation where numerical constants are replaced by placeholders (Biggio al. 2021) Each equation skeleton has the configured number (dzz of independent variables symbols. For example, y sin CC ai1 + C223 is possible generated equation skeleton which includes constants placeholders C1 and C2 and two variables 21 and ø2. Such equation skeletons are considered expression trees during generation. Each randomly-generann nodes, i.e, function operators. Unnormalized weighted distribution shown in Table 7 is used for sampling each non-leaf node. Each leaf node has probability of 0.8 of being an independent variable and 0.2 of being an integer. Different from (Biggio et al. 2021) using almost all elementary functions symbols including discontinuous ones like In, arcsin, tan for symbolic regression tasks, we only use the operators listed in Table 7wich guarantee that the generated function is continuous make more suitable for interpolation tasks.
expression tree has up-to 5 non-leaf
in the whole domain X [-1, 1]
Operator
+
X
.2
exp
sin
COS
Unormalized probability
10
10
4
2
4
4
Table 7: Operator and corresponding un-normalized probabilities during the generating process of mathematical functions NeSymReS dataset
Secondly, constants values are independently sampled from a uniform distribution (11,5) and we get a set of completely-defin mathemattiall-xppestit functions. Then normalize those func- tions to make their values range in [0, 1] and multiply it with a random number range [0.9, 1] to produce diversity.
After those completely-defines functions obtained, we sample interpolation task for each function by randomly sampling a of support points { in X, evaluating the function and get the corre- sponding {yi J. and then splitting those data points into a observed points set and a target points set. 512 scatter points are sampled using each function, of which a random number (ranging in [5, 50)) of points are set up as the observed points and the rest are set up as the target points.
During the generation of interpolation tasks. results with invalid values (NaN or inf) are removed. For dz in configurations {1, 2, 3, 4]. we generate datasets separately and use them for training and testing. In particular, when d, is configured, we generate 150 equation skeletons set for testing and 1 million skeletons set for training. Each skeleton in the training set existing in the test set has been removed. At training time, interpolation tasks are sampled using equation skeletons from the training set in real time. At testing time, we use an interpolation task set containing 10000 instances, which are generated by the testing skeletons set.
B.2 SYNTHETIC DATASET II: TFRD-ADLES
TFRD- Chen et al. 2021 is a synthetic dataset for 2D temperature field reconstruction where each instance represents simulated 2D temperature field containing several heat source components and specific Dirichlet conditioned boundary. The goal of each task instance is to reconstruct the whole temperature field according to a limited number of observed points with measured tempera- ture. We use the TFRD-ADlet dataset to test the performance of NIERT for 2D temperature field reconstruction. The characteristics of this dataset are summarized as follows.
Instance construction: Each instance has 200 x 200 regular grid points that represent the temper- ature field in 0.1m x 0.1m square area. Among these grid points, 32 points have their temperate known and used observed points. The other 3968 points are used target points.
Training set and test set: We have a total of 10,000 training instances and 10,000 test instances. For the sake of fair comparison, we also use form loss function for this dataset performed in Ref. (Chen et al. 2021).
Dimensionality of points: The grid points are in 2D plane.
TFRD-Alett is downloadable http://www.aaaaiicccc///////tttr passwor:: 'tfrd',
17

---

Under review as a conference paper at ICLR 2023
Approaches for comparison: For this dataset, compared NIERT with three neural network- based interpolation approaches. including conditional neural process (Ganeo et al. 2018). attentive neural process (ANP)(Kim et al. (2119) and TFR-transforma (Chen et al. 2021). We also compared NIERT with its enhanced version that was pre-trained using the NeSymReS dataset.
B.3 SYNTHETIC DATASET III: D30
In order to verify the scalability of NIERT on higher dimensional data, we specially constructed synthetic dataset (called D30 dataset) containing three subsets whose dimensionalities are 10, 20 and respectively.
Instance construction: Each function for sampling one interpolation task is obtained by the sum- mation K randomly sampled d dd-dmmessional Gaussian functions, which can be formalized as
f(x) = Σ As exp (-1(---0)) o
K
k=1
We fix K as 5. For each Gaussian function using in each function, we uniformity sampled the center Ca from 1-1, weight Aş from [-1, 1] and width 0g from [od,, 20a,] where dd, is configured as 1, 2 and 4 when d is 10, 20 and 30 respectively.
Training set and test set: For each dimensionality of 10, 20 or 30, created training set contain- ing 256K instances and a test set containing 512 cases from this function distribution. Each instance includes 64 observed points and 192 target points, which uniformly sampled from 1-1,1]
Dimensionality of points: The data points in three subsets have the dimension of 10, 20 and 30 respectively.
Approaches for comparison: On D30, we compared NIERT with five representative interpolation approaches, including radial basis function (RBF) (Powell 1987 Forbberg & Zuev| 2007), MIR [Wackernael] 2003). conditional neural process (Greeo ct al. 2018). attentive neural pro-
cess (ANP) (Kim et al.) (2019) and EFR-transformer (Chen et 2021)
B.4
REAL-WORLD DATASET: PHYSIONET
PhysioNef excerpted from the PhysioNet Challenge 2012 (Silva al. 2012). is real world dataset collected from intensive care unit (ICU) records for time-series data interpolation. Each point in an instance represents measurement at specific time, where each measurement contains up to 37 physiological indices. It should be pointed out that this dataset is representative of hard interpola- tion tasks due to the sparsity and irregularity of the records.
Instance construction: Each instance consists of multiple points, each of which represents a mea- surement of patient a specific time. Following the study (Shukla & Marlin 2021), randomly divided the points into observed points and target points. We also set the ratio of observed points five levels, i... 50%, 70%. and 90%, and trained and evaluated NIERT using the thus-acquired datasets.
Training set and test set: We randomly divided the 8,000 instances acquired from the PhysioNet Challenge 2012 2 into training set and test set with a ratio of 4:1. For the sake of fair comparison, we use Lo-form loss function for this dataset as performed in (Shukla Marlin (021).
Dinsionality of points: Each point in an instance represents a measurement at specific time and each measurement contains up to physiological indices; thus, the independent variable while the dependent variable y has dimensionality of 37.
Approaches for comparison: On this dataset, we compared NIERT with four representative ap- proaches designed for time-series data interpolation, including RNN-VAE (Chung al., 2014). L-ODE-RNN (Chen et al.) 2018)) L-ODE-ODE/Rubanoo) et al. 2019), and mTAND-Ful((Suki)
& Marlin 2021). The details of these approaches are provided in Section C We also compared NIERT with its enhanced version that was pre-trained using the NeSymReS dataset.
PPhysioNet downloadable at http:///tyysssccrroooooommmm/ene
18

---

Under review as a conference paper at ICLR 2023
C APPROACHES FOR COMPARISON
Radial basis function (RBF) RBF is one of the most commonly-used scattered data interpolation methods. It adopts specific type of radial basis functions on observed points and uses their linear combination to represent the target function. We use the RBF interpolation implementation in SciPy (Virtanen al. 2020) and multiquadric function as basis function type for the experiments.
MIR MIR is another multivariate interpolation and regression method for scattered data sets proposed by (Wang et al., 2010). MIR represents the approximation error with Taylor expansions and minimizes the approximation error to find the basis functions.
Conditional nerual process (CNP) CNP proposed by (Garnelo al. 2018) is an neural model able to learn to predict distributions of target points values given series of observed points. In order to fully verify the accuracy of interpolation, we let CNP to predict the values of target only and the training criteria function is set to be the prediction error of values of target points in the experiments.
We borrowed the parameter settings from the comparison experiments in
the work of Chen et al. (2021). where the intermediate layer dimensions of the MLP encoder [d2 + dy, 128, 128, 256 512]. and these of MLP decoder are [512 + dg. 256, 256, 128, 128, dy).
Attentive nerual process (ANP) ANP Kim et al. 2019) leverages attention mechanism in CNP and improves the prediction performance. In the experiments, the criteria function are fixed as same CNP above. We borrowed the parameter settings from Chen al. (202T), where the hidden dimension is 512, attention head number is 4, and the encoder and decoder have two attention layers respectively.
Bootstrapping attentive nerual process (BANP) BANP (Lee et al. 2020b) employs hootstrap technique to further improve the performance of ANP. In the experiments, the criteria function are fixed as same as CNP and ANP above. We set its hyperparameters to be the same as ANP, except for its unique hyperparameter, i.e., the number of samples, which is set for computational cost considerations.
TFR- Trr-Traasformer Transformer (Kim al. 2019) is originally proposed to solve tasks in natu- ral language processing. Chen et al. (2021) adopts Transformer in 2-dimensional temperature field reconstruction using scattered observations. Compared with vanilla transformer, TTFR-Transformer removes positional encoding, encodes the observations using encoder, using cross-attention mecha- nism between observations' encoding and target points to represents targets' features at decoder, and using a MLP to map targets' features to values. We set its hidden space dimension, the number of attention layers and the number of attention heads in each experiment to be the same as our method NIERT. These hyperparameters are listed in Table S
RNN-VAE RNN-VAE is a VAE- model where the encoder and decoder are standard RNN models. Gated Recurrent Unit (GRU) (Chung al. 2014) module is configured the recurrent network.
L-ODE-RNN L-ODE-RNN refers to latent neural ODE model where the encoder is RNN and decoder a neural ODE proposed in (Chen et al. 2018).
L-ODE-ODE Rubanova et al. (2019) proposes ODE-RNN model which generalize RNNs to have continuous-time hidden dynamics defined by ODEs. L-ODE-ODE refers to the model where the encoder an ODE-RNN and decoder is neural ODE.
mTAND-Full Shhkkla & Marlin (202T) proposes mTAND-Full for interpolation and classification of sparse, irregularly sampled, and multivariate time series data. mTAND-Full performs time atten- tion mechanism to learn temporal similarity and Bidirectional RNNs to encode temporal features. Mask mechanism makes the representation of missing data and target points convenient and easy to parallel. For the experiments of RNN-VAL VAE, LODE-RNN L-ODE-ODE, and mTAND-FF -uul approaches, we use the same hyperparameter settings as (hukka Marlin 2021)
'MI'' implementation can he found http://wwwhmiicccccggggwwwww////
19

---

Under review as a conference paper at ICLR 2023
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 ADDITIONAL CASE STUDIES
Cases from ID & 2D NeSymReS test set
In each example of ID interpolation task extracted from NeSymReS test set, the blue curve repre- sents the ground-trutt truth function while the red curves represent the interpolation functions reported by NIERT and the existing approaches.
RBF
MIR
ONP
AMP
BAN
TFR-transfommr
NIERT (Ours)
66-
0.5
-1.4
-6.3
-55
-25
-05
MSE: 3.5e-6
MSB 2.19-55
VSE 2,10m3
1.4e-
MSE1 Ne.5
4.85
3.81e-6
04
34
7A
02
6.2
34
12
-1.0
-65
10
35
-16
-05
00
35
10
-33
-15
06
0.5
10
-34
-65
00
05
16
-1.
-25
0.
65
-00
-55
05
-10
-05
25
BBT
TFB-needoorme
(Ours)
3.0
-33
-03
64
MSE 6.7e
64
MSB 5.10e4
34
MSE 1.35e4
1A
MSB 4.33e5
MSB 1.12e5
0.
MS5
1.35e5
MSE: 3.08e
-10
-05
20
25
17
-10
-05
00
75
20
-10
-55
00
05
10
-33
-65
00
05
1.
-2.
-35
05
65
-20
-35
38
05
3.0
-10
-05
25
IFR-tranffomar
(Ours)
MSE: 8.5022
MSD 2.82+-2
VSS: 1,8002
MSG: 3.11e2
1.39m2
A,48ec
1.86ea3
05 45
05
63
TFR-transfomee
(Ours)
6.803
-35
-05
26
4.5
-35
65
-35
3,6763
65
414
-35
Figure 8: Additional from ID NeSymReS
20

---

Under review as a conference paper at ICLR 2023
1.0
RBF
1.2
MIR
6.4
CNP
1.0
AMP
SAN
IHR-trasffomer
10-
NIERT (Ours)
80-
0.5
38
139
08-
08-
60-
566
04
34
3.5
04
06
66
0.8
0.-
-1.0 -05 10
35
12
-10
-05
00
15
20
-10
-55
06
0.5
10
-1.
-6.
00
05
10
-2.0
-25 0.2 65
10
400
-15
26 6.5
3.0
-10
-05
2.0
25
64-
MS5:
5-54e5
MSD 1/44-4
3:03el
MSS:: 8: 8.70bb
MS::: 3: 1.866
MSS:: 0.31e4
MS8: 3.3be-6
62-
622
32
62-
60
1.0
65
00
35
05
00
65
00
05
10
60
05
55
62
0.5
35
0.
10
65
25
RBF
MIR
AMP
BAK
TFR-transformer
NIERT (Ours)
3.0
3.2
5.0
0.6
6.
24
2.
-1.0
1. -10
.05 00 15
-0-
-55
06 65
10 -35
-6.
0%
10" 16:-20
-13 6's
-2--20 -25 6.
-10 -05
64
MSE:
1.73.4
64
NSE 7
VSE 7,5863
MSE 1.018
MSE 1.5124
MSE 4,50c
MSE: 1.18c
02-
0.2
22
02
02
82-
60
-1.0
45
50
55
00
55
.25
65
05
16:-10
6.
10
-35
RBF
THB-ieendormmr
NIERT (Ours)
1.0
W
W
68-
02
-55
-15
-05
MSE: 3.16e33
MSB 4.22e3
MSE 1.92e22
MSB 1.262
MSE 2.95e-4
4.16e-
044
64
34
02
64
32
-1.0
-65
20
-10
-05
00
05
-33
-55
05
-65
60
05
-25
0.
65
-25
05
-L0 -05
TFR:transforart
(Ours)
-10
-33
-03
MSE: 7.1166
MSE
VSL: 3.3644
MSD: 6.22
4,20m3
1.bbeb
-1.0 -05
70
25
-10
-05
00
15
20
-30
-55
00
05
10
-33
00
05
1.
-2.
-35
03
05
200
-35
05
3.0
-10
-05
25
TRR-rranffomar
(Ours)
MSE: 1.7
MSD 1.10
958:1.80ed
8.05
3,47e3
8.3263
00
03
83
0%
03
Figure 9: Additional cases from 1D NeSymReS
21

---

Under review as a conference paper at ICLR 2023
AMP
HINT [ound
254:26003
N52220000
NS2 203e0
PSG:24603
22363
Grardtrch
HIGHT 10am)
2E0:22003
NG2 1223-
N52 112e4
H52 1.4e-
H62:4 43
211
06
00
Courd truhh
AM
NIGIT (0um)
-55 06
05
U0
00 15
06
00
25 00
me site
Neo 2.50
790 2.074
U16 13
45
63-
52
43
45
Figure 10: Additional cases from 2D NeSymReS test set
22

---

Under review as a conference paper at ICLR 2023
HIKT (oun)
7.36-
N52 68244
M80 1.96e3
699ed
M6012 75
10
22 06 01 10
Je 25 23
26 65 08
ME6:2476-4
3
Tikcandome
HIGHT (0am)
HEC:340044
N52
N52 432e4
MSc
M6211 00m3
211
to
00
to
40
Gourd
AM
NIGIT (0am)
-55
06
05
10
60 15
45
06
05
-05 00
<05 68
05
05
45 00
69
PAA 856
Nes Swied
63-
Figure 11: Additional cases from 2D NeSymReS test set
23

---

Under review as a conference paper at ICLR 2023
Cases from TFRD-ADlet
Grunddruhh
BAGP
MISAT (Guno)
MICAT (Bura)
1323
1003
2015
0.8
329
mm
pants
3123
MAD:
3.02e
MAZ: 4.2b
MAG:
833
300
40
3b
32
424
40
3.5
Ground Iruth
BAN
NIERT (Oura)
MISRT (Dura)
1123
3003
3915
0015
0.8
359
pants
1225
MAt:
2.50
2.30
635
3325
3X0
40
ab
04
415
40
455
3.
Ground Iruth
BAMP
WIERT (Ours)
MIGRT (Oues)
3125
3100
3000
3915
0.8
105
ncs
NSS
200
S
2915
mm
2.5
6.5
40
15
0:55
04
014
30
315
33
5.55
Ground outh
BAMP
MIERT (Ours)
MISHT (Oues)
3125
300
3000
6/8
31%
3003
Juka
P 8
3015
635
0.2
6.5
30
15
0.55
06
364
30
3/5
02
33
855
CA
8.45
Groundouth
CMP
AMP
TT Brrefooee
NIIRT (Dari)
MIGRT (Ours)
100
0.8
355
300
3913
$35
455
2333
3.4
8C5
30
10
92
14
02
364
30
312
02
85
08
6.55
Figure 12: Additional cases from TFRD-ADlet test set
24

---

Under review as a conference paper at ICLR 2023
D.2 CONTRIBUTION ANALYSIS OBSERVED POINTS FOR INTERPOLATION
As supplements to Figure 6, all observed points' attention weights extracted from the final attention layer of NIERT and TFR transforrrfr are visualized Figure K3
Obseved pant
Tereraaoomer elanton NETTacarie
Osseved pent 5 Theremteee scardon MERT etiente
Obervec 008137 Teleriibeer etienee mottes NDD atartice
Desevedd pent
clenton NEMtatenton
1e mormmme martan etienes
Moterimeer elimtee
desomeed part
ceeeandiieeeatee attetio sestanvetin
Dvse pert 12 tandere anrelee sane atoottes
Dioeeenieer THe Demmaiime atistes seae aterive
disoeved parnt
anddaneee attetion NRAT amoedia
Goeeeeperree Pendeeeee amerta amootts
15 mareemee
NERT
Figure 13: All observed points and the corresponding extracted attention response
The contributions by observed points are quit imbalanced using TFR TRR--ransformer. In particular, the intensity and area of the contributions of the five observed points 0,2.6.16.J1 cover almost the entire domain, while the remaining 17 observed points only produce negligible contribution. In contrast, when using NIERT, contributions by an observed point are much more local and thus targeted and all observed points have contributions to interpolation. This shows that NIERT can fully exploit the relationship between observed points and target points.
D.3 ADDITIONAL ABLATION STUDIES
The effects of different model depths
We carried out experiments on 2D NeSymReS dataset using NIERT with hyper-parameter L varying from 7. Then evaluate the models the 2D NeSymReS test dataset. Accuracy are listed in below Table g The results show that NIERT with 7 transformer layers has the best accuracy on the test set, and NIERT with 6 transformer layers has reached comparable level. Therefore, in the experiments on NeSymReS data set, we use L 6 to balance efficiency and accuracy.
The effects of different hidden dimensions
25

---

Under review as a conference paper at ICLR 2023
Interpolation approach
Number of transformer layers /.
3
6
NIERT
66.812
60.133
52.098
45.319
44.043
Table The interpolation accuracy of NIERT (MSE x10-9) under various settings of transformer layer number L on 2D NeSymReS dataset
We also carried out experiments 2D NeSymReS dataset using NIERT with smaller hidden di- mensions dmodet. say from 256, 128 and 64. Then evaluate the models on the 2D NeSymReS test dataset. Accuracy are listed below Table 9 The results show that when the hidden dimension is within 512, NIERT's interpolation accuracy is higher when the hidden dimension is larger.
In this experiment, we fix other super parameters, say model depth and number of heads 8.
Interpolation approach
Different hidden dimensions dmodel
64
128
256
512
NIERT 106.193
72.107
51.153
45.319
Table 9: The interpolation accuracy of NIERT (MSE x10-5) under various settings of hidden di- mension dmodel on 2D NeSymReS dataset
The effects of prediction error of observed points in loss function
To verify the contribution of re-predicting values of the observed points to the interpolation task, conducted experiment that puts the prediction error of observed points in the loss function, i.c. only minimizes the estimation eгrоr of target points. The results are shown in Table [0) which demonstrates that only minimizing estimation error of target points make NIERT performing poorly. This indicates that re- re-rrdicting the value of observed points helps NIERT to predict the value of target points more accurately.
Loss contains prediction error of Only target points
All points
MSE (x10-
48.931
45.319
Table 10: Interpolation accuracy (MSE x10 5) of NIERT trained with loss only containing the prediction error of the target points
D.4 PREDICTION ACCURACY GAP BETWEEN OBSERVED POINTS AND TARGET POINTS
Prediction accuracy on
MSE
(x10-5) on NeSymReS test sets
ID
2D
3D
4D
Observed points
1.301 8.964
4.862
3.775
2,662
Target points
45.319
77.664
72.025
Table 11: Prediction accuracy gap between observed points and target points on NeSymReS test set
We carried out an extra experiment for prediction accuracy analysis observed points and target points. Figure II shows that on the observed points, the MSE of prediction relatively smaller when compared with it of target points expected.
D.5 COMPUTATIONAL TRADE-OFF
Our work focuses on improving the accuracy of scattered data interpolation. Considering the practical scenario of efficient computational interpolation, compare and analyze the accuracy-
26

---

Under review as a conference paper at ICLR 2023
computation time trade-off between approach and previous approaches. We collected the erage computation time and MSE per interpolation task for all approaches on the 2D NeSymReS testset. which contains 10,000 test interpolation tasks, and visualize the results in Figure [4 Fig- ure 14] the lower the method the higher the accuracy and the more left the method the more efficient the calculation.
4x10
RBF
MSE on 2D NeSymReS testset
MIR
CNP
2x10
ANP
10:3
BANP
FFR-transformer
5x10-1
NIERT
40
Average time spent per interpolation task (ms)
Figure 14: Computational trade-off comparison on 2D NeSymReS testset
that RBF is the most computationally efficient, but it is also the least accurate. Our approach NIERT has the highest computational accuracy, and its computational effi ciency is only slightly reduced compared to the next best approach, tFFR-transformer.
From Figure 14
should be noted that the traditional interpolation approach RBF and MIR are tested using Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz with 40 cores and the data-driven approaches including CNP, ANP, BANP, TFRttransformer, and NIERT were evaluated on an NVIDIA GeForce RTX 3090 GPU. Here we set the batch size for evaluation keep the comparison fair.
For the training times of these different methods, we collected and put them into the Table 12 We used two NVIDIA GeForee RTX 3090 GPUs for training. On 2D NeSymReS training set dataset containing one million functions, 160 epochs of training were performed. It should be noted that traditional interpolation methods such RBF and MIR do not require training.
Interpolation approach
Training time (h)
RBF
MIR
CNP
14.9
ANP
18.3
BANP
21.1
TFR-transformer
17.5
NIERT
19.1
Table 12: Training time comparison on 2D NeSymReS training set
Table 12 indicates that the training times of these data-driven approaches do differ much. Among them, CNP has the shortest training time (14.9 h) and BANP has the longest training time (21.1 h). The training time of our approach NIERT is 19.1 hours.
27


Reviews:
[{"review_id": "pXmgmzh2a_", "reviewer": "Reviewer_uN6v", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces NIERT which is a new framework for numerical interpolation for scattered data. The key idea of the framework is to use Transformers as a representation encoder and treats observed and target points in a unified fashion by embedding them into the same representation space. In the paper, the authors also propose a new partial self-attention mechanism that could escape from the unexpected interference of target points on observed points. Combining the partial self-attention mechanism,  masking mechanism, and pre-training technique, NIERT outperforms the existing approaches on both real-world and synthetic datasets. For example, NIERT gives the best mean absolute error on the NeSymReS dataset, the TFRD-ADlet dataset, the D30 dataset, and the PhysioNet. The authors also conduct ablation studies to understand the role of partial self-attention and the pre-training technique.", "summary_of_the_review": "The paper proposes the partial self-attention mechanism and applies it to the numerical interpolation problem. The framework achieves great prediction results however the performance in downstream tasks is missing.", "strength_and_weaknesses": "## Strength\n* The proposed framework exploits the power of Transform in the numerical interpolation where traditional interpolation approaches for scattered data use explicitly pre-defined basis functions to construct interpolation functions.\n* The partial self-attention mechanism is new to the best of my knowledge.\n* The authors also discuss the connection between NIERT and the traditional approach by showing that partial self-attention is a general form of the RBF interpolation function.\n* The proposed framework beats all baselines on all datasets.\n\n## Weaknesses\n* The computational trade-off should be discussed e.g, the training time, and the interpolation time.\n* The authors should discuss the statistics about the number of observed and target points in each dataset. I wonder how the proposed framework performs in different settings of sequence lengths.\n* The authors only focus on the mean absolute error. There is no downstream task for showing that NIERT is better than previous frameworks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i9SBSKzVrs4", "reviewer": "Reviewer_rgW7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new partial self-attention layer to improve numerical interpolation for scattered data. The module was used in a modified Transformer to solve the interpolation task. The proposed approach treats observed and target points in a unified way by embedding them in the same representation space.", "summary_of_the_review": "In terms of text quality, the paper is easy to read. There are some minor issues with the text, but nothing major, except that the sentence abruptly ended at the end of page 4.\n\nExperiments and evaluation design are done nicely; the authors conducted many experiments and ablation studies. \n\nHowever, there are some issues with the paper:\n\nThe paper claims that one of the reasons why other methods perform poorly is because they process observed and target points separately. This fact prevents them from exploiting correlations between observed and target points (see page 2, end of paragraph 1, and page 5 before the experiments section). I do not see this claim proven in the paper, although I may be wrong.\n\nThere are a lot of variables (for example, different network parameter sizes) that can influence the outcome of this kind of experiment, so careful experiment design is needed to prove this.\n\nIn section 4.4 in figure 5, the fact that attention maps are imbalanced does not mean much. In my opinion, its area of affection for each point depends on its neighbors' count and their proximity placement. For example, if you have a lot of points in one area, it is natural that each point will contribute, or can either contribute a little to a wider area, or can contribute a lot to a local area. Which option is better is a good question. So the claim in the paper at the end of this section that NIERT can exploit the correlation between points observed and target points more effectively, in my opinion, is not well-supported. \n\nFinally, the simple use of the pre-training technique is not a contribution. \n\nQuestions:\nIs it correct that partly self-attention is just a particular case of self-attention, where observed and target points can’t “look” at the target points(wij=0)? \nPlease clarify, what is input in each batch for your neural network in evaluation time? \n\n\nSmall suggestions:\n1. Section 3.4 is a good addition to the text but can be moved to supplementary.\n2. It would be nice to see a comparison of several hyperparameters for each model for the methods you compare with.\n", "strength_and_weaknesses": "Strength:\nA new mechanism to mask points in self-attention was introduced. \nThe proposed model achieves a better result on the synthetic and real-world dataset from the paper compared to the current state-of-the-art methods.\nExtensive analysis and ablation studies were conducted.\nThe Paper is written in an easy to read manner\nStrong evaluation and comparisons.\n\nWeakness:\nMistypes in the text.\nSome claims of the paper were not well-supported (see summary for more details).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Aws1d8DLZn", "reviewer": "Reviewer_qiuo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a learning-based approach using the encoder representations of Transformers for numerical interpolation of scattered data.", "summary_of_the_review": "This paper combines existing techniques with trivial modification to achieve good results for numerical interpolation. Due to lack of novelty, I believe this is a good technical report for practitioners to apply deep learning methods for numerical interpolation, but by its current form it is not ready to be published a paper in a top machine learning venue.", "strength_and_weaknesses": "Strength: this paper combines a set of existing (and indeed very popular) deep learning methods to deal with the numerical interpolation problem, and it achieves good results by experiments on real data sets.\n\n\nWeakness: there are several concerns, among which lack of novelty is the major concern.\n\n1. This paper combines a set of very popular techniques in deep learning to the numerical interpolation problem, such as transformer with self-attention and enhancement with pre-training. Every component of the proposed method is either existing or trivial, so it is not clear what is the novel method proposed by this paper.\n\n2. The claimed contribution, the partial self-attention mechanism, is mostly trivial which sets zero correlation weights to target points (eq. (2)).  While the authors argued that such partial self-attention mechanism is connected to Radial Basis Function (RBF interpolation), it is really trivial to me: if one sets zero weights to target points, then it is naturally written as a combination of functions on the observed points. In addition, while observed and target points are decoupled, I believe it should be helpful to boost the performance by modeling the correlation among target points by common principles of semi-supervised learning (the target points are given).\n\n3. Why are different metrics applied to different data sets, such as MSE for the NeSymReS, D30 and PhysioNet datasets, and MAE (CMAE,BMAE) for the TFRD-ADlet dataset? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject