NIERT: Accurate Numerical Interpolation through Unifying Scattered Data Representations using Transformer Encoder
Authors: Shizhe Ding
Dongbo Bu
Venue: NeurIPS 2022 Submitted (2022)
Keywords: numerical interpolation
transformer encoder
mask mechanism
pre-training model

Abstract: Numerical interpolation for scattered data aims to estimate values for target points based on those of some observed points. Traditional approaches produce estimations through constructing an interpolation function that combines multiple basis functions. These approaches require the basis functions to be pre-defined explicitly, thus greatly limiting their applications in practical scenarios. Recent advances exhibit an alternative strategy that learns interpolation functions directly from observed points using machine learning techniques, say deep neural networks. This strategy, although promising, cannot effectively exploit the correlations between observed points and target points as it treats these types of points separately. Here, we present a learning-based approach to numerical interpolation using encoder representations of Transformers (thus called NIERT). NIERT treats the value of each target point as a masked token, which enables processing target points and observed points in a unified fashion. By calculating the partial self-attention between target points and observed points at each layer, NIERT gains  advantages of exploiting the correlations among these points and, more importantly, avoiding the unexpected interference of target points on observed points. NIERT also uses the pre-training technique to further improve its  accuracy. On three representative datasets, including two synthetic datasets and a real-world dataset, NIERT  outperforms the existing approaches, e.g., on the TFRD-ADlet dataset for temperature field reconstruction, NIERT achieves an MAE of $1.897\times 10^{-3}$, substantially better than the transformer-based approach (MAE: $27.074\times 10^{-3}$). These results clearly demonstrate the accuracy of NIERT and its potential to apply in multiple practical fields.

NIERT: Accurate Numerical Interpolation through Unifying Scattered Data Representations using Transformer Encoder
Anonymous Author(s)
Affiliation
Address
email
Abstract
Numerical interpolation for scattered data aims to estimate values for target points based on those some observed points. Traditional approaches produce estima- tions through constructing interpolation function that combines multiple basis functions. These approaches require the basis functions to be pre-defined explicitly, thus greatly limiting their applications in practical scenarios. Recent advances exhibit an alternative strategy that learns interpolation functions directly from observed points using machine learning techniques, say deep neural networks. This strategy, although promising, cannot effectively exploit the correlations between observed points and target points as treats these types of points separately. Here, we present a learning- barneng-bbsedd approach to numerical interpolation using encoder representations of Transformers (thus called NIERT). NIERT treats the value of each target point as a masked token, which enables processing target points and observed points in unified fashion. By calculating the partial self-ateention be- tween target points and observed points at each layer, NIERT gains advantages of exploiting the correlations among these points and, more importantly, avoiding the unexpected interference of target points observed points. NIERT also uses the pre-training technique to further improve its accuracy. On three representative datasets, including two synthetic datasets and a real-world dataset, NIERT outper- forms the existing approaches, e.g., on the TFRD-ADlet dataset for temperature field reconstruction, NIERT achieves MAE of 1.897 x 10-3, substantially better than the transformer-based approach (MAE: 27.074 x 10 10-3). These results clearly demonstrate the accuracy of NIERT and its potential to apply in multiple practical fields.
2
4
6
10
11
12
13
14
15
16
17
19
21
23
Introduction
25 Numerical interpolation for scattered data plays important and fundamental roles in a wide range of practical scenarios, including solving partial differential equations (PDEs) [I]. temperature field reconstruction (2]. time series interpolation B 4. In meshfree PDE solvers, the interpolation error often leads to deviations in subsequent calculations, which seriously affeets the solution's ''s accuracy [5) In the task of temperature field reconstruction for micro-scale electronics, interpolation
27
29
30 methods used obtain the real-time working environment of electronic components from
31 limited measurements, and imprecise interpolation will significantly increase the cost of predictive maintenance (2). Thus, accurate approaches to numerical interpolation are highly desirable.
32
33 large number approaches have been proposed for interpolation of scattered data, which can be divided into two categories, namely, traditional non-learning based methods and recent learning- based methods. The typical traditional interpolation schemes construct the target function by a linear
35
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

36
combination of basis functions [6] These schemes require asplicitl-deffneee basis functions to model the target function space, and various types of basis functions have been devised by algorithm designers to adapt to different scenarios. Nevertheless, such methods still suffer the from limitations of high requirement of sufficient observed points, and the limited complexity of the target function.
37
38
39
40
Recent progress has exhibited an alternative strategy that uses neural networks to learn interpolation functions directly from the given observed points. For example, conditional neural processes (CNPs) 77 and their extensions [8-10] model the conditional distribution of regression functions given the observed points. In addition, Chen et 2 proposed to use vanilla Transformer [ solve interpolation task in temperature field reconstruction. All of these approaches use an "encoder- decoder" architecture, in which the encoder learns the representations of observed points while the decoder estimates values for target points. Intuitively, observed points and target points should be processed unified fashion because they are from the same domain. However, this architecture treats them separately and cannot effectively exploit the correlation between them.
41
42
43
44
45
48
47
48
49
Inspired by the recent advances of language/image models, especially BERT [12] and BEIT [13], we designed an approach to numerical interpolation that can effectively exploit the correlations between observed points and target points. Our approach is a learning-basse representations of Transformers (thus called NIERT). The key elements of NIERT include: i) the 53 use of the mask mechanism, which enables processing target points and observed points in a unified fashion, ii) novel partial self-attention model, which calculates attentions between target points and observed points at each layer, thus gaining the advantages of exploiting the correlations between these two types of points and, more importantly, avoiding the unexpected interference of target points on observed points simultaneously. and idt) the use of the pre-training technique, which further improves
50
51
approach using the encoder
52
54
55
56
57
58
the interpolation accuracy of NIERT.
59
The main contributions of this study are summarized follows.
80
1.
We propose an accurate approach numerical interpolation for scattered data, On represen- tative datasets, including both synthetic and real-world datasets, our approach outperforms existing approaches. The experimental results demonstrate the potential of our approach in a wide range of application fields. The source code of NIERT will be released for open source use.
81
62
63
64
2.
We propose novel partial self-attention mechanism to make Transformer incorporated with strong inductive bias for interpolation tasks; i.e., it can effectively exploit the correlation among two types of points but simultaneously avoid the interference of one type of points onto the others.
66
68
3. We propose use the pre- rre-training technique to enhance interpolation approaches. When facing an interpolation task in a newly-appearis application field, we can benefit from the experience learned from low-cost synthesized interpolation tasks.
70
71
72
2
Related works
2.1 Traditional interpolation approaches for scattered data
74 Traditional interpolation approaches for scattered data use explicit basis functions to construct
interpolation function, e.g., Lagrange interpolation, Newton interpolation (6), B-spline interpolation
76
[4], Shepard's method [15), Kriging [16], and radial basis function interpolation (RBF) [17 [8]. Among these approaches, the classical Lagrange interpolation, Newton interpolation and B B-splines interpolation are usually used for univariate interpolation. Wang et al. [19] proposed a high order multivariate approximation scheme for scattered data sets, in which approximation error is represented with Taylor expansions at data points, and basis functions are determined through minimizing the squares of approximation error.
77
78
79
80
81
2.2 Neural network-based interpolation approaches
85 Equipped with deep neural networks, data-driven interpolation and reconstruction methods show great
84 advantages and potential. For instance, convolutional neural networks (CNNs) have been applied
85 in the interpolation tasks of single image super-resolution [20 2T). and recurrent neural networks
(RNNs) and Transformers have been used for interpolation of sequences like time series data 4 22

---

9/
Loss
Function distribution
Pair FINU
NIERT INTERPOLATOR
Add Norm
buttsheed Seifatlention
Partial self-attentnn attention
Nesked tokion
y1
Erbboliiig Layer
NIERT INTERPOLATOR
Sampled Interpolation tasks
Observed points
Targeel piitis
Figure 1: Overview of NIERT training process. Here, 2, represents the position of point, and y represents its value. The predicted values of the point is denoted as ýt We prepare the training interpolation tasks by first sampling functions from function distribution FF and then sampling observed points O and target points T on each function. NIERT trains an interpolator over this data. The partial eeffaattetiion mechanism facilitates exploiting the correlations between observed points and target points and avoiding the unexpected interference of target points on observed points
87 Recently, Garnelo et al. 7 proposed to model the conditional distribution of regression functions given observed points. The proposed approach, conditional neural processes (CNPs), has shown increased estimation accuracy and generalizing ability. Kim et al. [ designed an enhanced model, attentive neural processes (ANPs), with improved accuracy. Lee al. [10] leveraged Bayesian
89
91 last layer (BLL) [23] for faster training and better prediction. In addition, the bootstrap technique was also employed for further improvement
[9]. To solve the interpolation task in 2D temperature field reconstruction, Chen et al. (2) proposed an Transformer-bee based approach, referred to as TFR- tranformer, which can also be applied solve interpolation tasks for scattered data with higher dimensions. Note that although TFR-transe transformer and our NIERT are both based on transformer, they are fundamentally different: TFR- transformer adopts an encoder-decoder structure and treats observed points and target points respectively, while NIERT adopts only a Transformer encoder (equiped with partial self-attention) encode and learn corelatetion of the scattered data in a unified fashion; ùi) NIERT's training objective considers the prediction error of both observation points and target points.
93
94
95
96
97
98
99
is trained by minimizing the prediction error of target points, while
100
101
2.3
Masked language/image models and the pre-training technique
The design of NIERT is also inspired by the recent advances in masked language/image models 24 25 and pre-trained models for symbolic regression [26)
103
[26]27]. (27]. The masked pre-trained models have been shown be successful in learning representations of languages and images from large-scale data and improving the performance of downstream tasks. In addition, the mask mechanism makes the model able to reconstruct missing data from their context. Utilizing large-scale synthetic symbolic functions and sampled scattered data. Biggio et al. [26) and Valipour et al. [27] pre-trained Transformers to learn the map from scattered data corresponding symbolic formulas. Different from these approaches, NIERT uses synthetic data to learn interpolating functions from scattered data numerically.
12
13
104
105
106
107
108
109
110
Method
112
3.1
Overview of NIERT
113 In the study, we focus on the interpolation task that can be formally described as follows: We are
114 given n observed points with known values = {(x,, y:) and m target points with values to be
115 determined, denoted as T (a; |- Here, x; E denotes position of a point, l/ = f(;;) E Y denotes the value of point, and ʃ + Y denotes function mapping positions to values. The
117 function f is from function distribution J, which can be explicitly defined using a mathematical
formula implicitly represented using set of scattered data in the form (z,, v:). The goal of

---

Algorithm NIERT training process
Require: NIERT model Ma with parameters 0, epoch number N, batch size B, domain X, function distribution FF and error metric function Error(:..)
for k in {1..N} do
< for b in {1..B} do
f. {ri); < sample a function and scatter points from F, X
(yiti < calculate on {r;};
0,T < split and mask scatter points with values (ar, vi)
[ < T))
+ J E, Error Mi)
end for
Compute the gradient V and update 0
end for
119
interpolation task is to accurately estimate the values f(x) for each target point T according to the observed points in O.
120
Figure [ depicts the schematic diagram of our NIERT approach. Briefly speaking, approach employs a data-driven approach to numerical interpolation using encoder representations of Trans- formers. The main element of our approach is neural interpolator that learns estimate values for target points. The interpolator is featured by the characteristic that it treats the value of each target point as masked token, thus enabling the unifying fashion to process both target points and observed points in the subsequent encoding and estimation procedures.
122
123
124
125
128
To suit the interpolation task, we design partial self-attention mechanism: one side, calculate the attention between target points and observed points at each layer, which gains NIERT the advantage effectively exploit the correlations between these two types of points. On the other side, the attention is partial one as we do not consider the effects of a target point on all other points. This way. the unexpected interference of target points onto the observed points, and the interference among target points, are completely avoided.
128
130
132
133
The training process is depicted in Algorithm T Specifically, we prepare the training interpolation tasks by first sampling functions from a distribution JF and then sampling observed points and target points 1' on each function. When training NIERT, we set the loss function as the error between the estimated values and the corresponding ground-truthh It should be pointed out that errors acquired on both observed points and target points are accounted into loss function.
134
136
138
3.2
Architecture of the NIERT interpolator
The neural interpolator in NIERT adopts the Transformer encoder framework; however, to suit the interpolation task, significant modifications and extensions were made in embedding, Transformer and output layers, which are described in details below.
140
141
142
Embedding with masked tokens: NIERT embeds both observed points and target points into the unified gighddimessooal embedding space. As the position of data point and its value y from different domains, we use two linear modules: Linear, embeds the positions while Linear, embeds the values.
144
146
should noted that for target points, their values are absent when embedding as they are to be determined. In this case, we use masked token as substitutes, which is embedded as trainable parameter MASK,, as performed in BERT [12]. This way, the interpolator processes both target points and observed points in unifying fashion.
147
148
150
We concatenate the embeddings of position and value of data point as the point's embedding, denoted as hy, i.e.,
151
44 [Linear.(x)). Linear,(su)) if (x, M4) E O i, T
[Linear, MASK

---

152
Transformer layer with partial self-attention mechanism: NIERT feeds the embeddings of the points into a stack of L Transformer layers, producing encodings of these points as results. Each Transformer layer contains two subsequent sub-layers, namely, multi-head self-attention module, and point-wise fully-connected network. These sub-layers are interlaced with residual connections and layer normalization between them.
153
155
156
157
To avoid the unexpected interference of target points on observed points and target points themselves, NIERT replaces the original self-attention in Transformer layer with partial self-attention, which calculates the feature of point i the + 1-st layer as follows:
158
159
h ++ LayerNorm (τ) + +MPP(7))
160
01 LayerNorm o + Euyaady
where of and aly represent the value embedding and ordinary attention weights at the I-th layer as calculated in Transformer [[]]. In this formula, we introduce new term waj that represents the partial self-attention pattern, i.e., Waj
161
163
11, if (T).0)) 0, if
184
By forcing the weight Why to be 0 for target point and any point j. we completely avoid the unexpected interference of target points on the other points.
185
Estimating values for target points: For each target point we estimate its value y; through feeding its features at the final Transformer layer into a fully connected feed-forward network, i.e.,
187
S MLP out (h.").
We calculate the error between the estimation and the corresponding ground-truth value, and compose the errors for all points into a loss function to be minimized.
169
170
3.3
Enhancing NIERT using pre-training technique
The interpolation functions from different applications usually differ greatly in their forms; however, the interpolation tasks might still share some common characteristics, say the correlation between observed points and target points. These common characteristics enable enhancing NIERT using the pre-training technique. Here, we pre-train NIERT using a synthetic dataset (see 4.1 for further details) and fine-tune other datasets in application fields.
172
173
174
175
Experiments and results
176
We evaluated NIERT and compared it with representative scattered data interpolation approaches on both synthetic and real-world datasets. We also examined the effects of the key elements of NIERT, including the partial self-attention and the pre-training technique.
178
179
180
4.1
Experiment setting
The datasets, metrics and approaches for comparison are briefly described below. Further details of experiment settings are provided Supplementary text.
182
183
Datasets:
Three representative datasets in various application fields, including two synthetic datasets S and TFRD-ADlet, and real-world dataset PhysioNet, are used for evaluation.
184
185
S a synthetic dataset for mathematical function interpolation, which built using data generator proposed by Biggio et al. [26] and Lample and Charton [28]. We construct function set with various dimensionality of data points, including 1D, 2D. 3D, and 4D. Scattered points in each instance are randomly sampled and divided into observed points and target points.
187
188
189
TFRD-ADlet is a synthetic dataset for 2D temperature field reconstruction where each instance represents simulated 2D temperature field containing several heat source components and specific
5

---

Dirichlet conditioned boundary. The goal of each task instance is to reconstruct the whole temperature field according to limited number of observed points with measured temperature.
192
199
PhysioNet Challenge 2012 dataset [29] is real world dataset collected from intensive care unit (ICU) records for time-series data interpolation. Fach point in an instance represents measurement at specific time, where each measurement contains up to 37 physiological indices. Following the study [22], we randomly divided the points into observed points and target points by setting the ratio
194
195
196
197
of observed points at five levels, i.e., 50%, 60%, 70%, 80%, and 90%. Note that this dataset is a representative of hard interpolation tasks due to the sparsity and irregularity of the records.
198
199
Pre-training dataset: this study, N dataset was used for pre-training NIERT to further dataset we directly use 2D TFRD-ADlet dataset for pre-training. As the PhysioNet dataset has a dimen- sionality of 37, we construct the pre-training instances by stacking random 37 functions from 11) TFRD-ADlet dataset and then sampling interpolation task instances.
improve its interpolation accuracy TFRD-ADle and PhysioNet dataset. For
201
202
203
When evaluating NIERT and other interpolation approaches, the prediction error of target points are calculated as interpolation accuracy. For the NeSymReS and PhysioNet dataset, we adopted mean squared error (MSE) as the error metric. For TFRD-ADlet dataset, we use three error metrics: mean absolute error (MAE), MAE in component area (CMAE) and MAE at boundary (BMAE) following Chen et al. (2 Accordingly, we use Lg- Le-form loss function for NeSymReS and PhysioNet dataset and L1 -frm loss function for TFRD-ADE dataset for training.
Metrics:
205
207
209
210
Approaches for comparison:
For NeSymReS dataset, we compared NIERT with six representative interpolation approaches, including RBF(3) MIR [19], C A BANP(9] and TFR- transformer 2 For TFRD--Dlee we compared NIERT with CNP, ANP, BANP and TFR- -rr-traanformer.
211
212
213
For PhysioNet we compared NIERT with four approaches designed for time-series data interpolation, including RNN-VAE (31, L-ODE-RNN 332 L-ODE-ODE and mTANN-FFlll 22
4.2
Interpolation accuracy synthetic and real-world datasets
216
For each instance of the test datasets, we applied the trained NIERT to estimate values for the target points. We calculate the errors between the estimation and the ground-truth as interpolation accuracy.
Interpolation approuch
MSE (x100 NcSymReS test
Interpolation apprachh
Evaluation criteria (x 10 3)
1D>
21)
313
41)
MAE
CMAE
BMAE
RBF
215.439
347.060
443.004
327.775
CNP
96.674
109.419
56,939
274-6111
342.997
ANP
$4.684
62.511
CNP
67.176
218.
392,318
314.311
ANP
34,558
140.005
206.699
164.751
BANP
28.671
29.450
19.984
218
HANP
14.913
143.518
140.288
TFR-transfommrr
27.074
29,772
18.835
TFR Pr-eranffoomer
15.556
$8.569
99.986
90.579
NIERT 3.473
3.947
2.467
NIERT 8.964
45.319
77.664
72,025
NIERT w/ pretraining
1.897
1.971
1.246
Table 1: Interpolation accuracy of NIERT and the existing approaches NeSymReS test dataset
Table 2: Interpolation accuracy of NIERT and the
existing approaches over the TFRD-ADlet dataset
Interpolation 50%
Ratio of observed points
approach
60%
70%
80%
90%
RNN-VAL VAE
13.41810.000
12.594110.00
11.887
11.133.10.000
11.470.10.006 0.006
L-ODE-RNN
8.132+0.020
8. 8.140-0..018
8.171+0.030
8.143++0.025
8.402+00.022
L-ODE-ODE
6.721-0.109
6.816:100055
6.798:0.143
6.850=10.066
7.142-10.066
mTAADDFull
4.139
4.008
4.157:10.053
4.410.10.144 149
4,798 10.036
NIERT
2.868=0.021
2.811#0.032
2.656+0.041
2.598+0.077
2.709+0.157
NIERT w/ pretraining
2.831=0.021
2.771
2.641-10.022
2.539
2.596-10.159
Table 3:
The relationship between interpolation accuracy (measured using MSE, x10 3) and the ratio of observed points. Here, we use the PhysioNet dataset as representatives
219
Accuracy on the Ne dataset: As shown in Table [ on the ID NeSymReS testset, RBF shows the largest interpolation error (MSE: 215.439). MIR, another approach using explicit basis functions, also shows high interpolation error of 67.281. In contrast, BANP and TFR- tranforrnforr.r
220
222
which use neural networks to learn interpolation, show relatively lower errors (MSE: 14.913, 15.556). Compared with these approaches, our NIERT approach achieves the best interpolation accuracy

---

224
(MSE: 8.964). aalso demonstrates the advantage of NIERT over the existing approach on the 2D, 3D, and 4D instances.
225
226
Note that the number of observed points varies greatly in test instances, making the average interpolation error calculated over all test in- stances insufficient measure interpolation per- formance. Therefore, we further divide test in-
10-4
228
MSE prediction target
229
10--)
230
231
stances into subsets according to the number of observed points. As shown in Figure 2 as the
232
233
number of observed points increases, the inter-
234
polation error decreases as expected. In addition,
20
235
the relative advantages these approaches vary
10
NEATT
35
20
25
30
40
45
50
with the number of observed points, e.g., CNP is better than RBF and MIR initially but finally be- comes worse the number of observed points increases. Among all approaches, NIERT stably shows the best performance over all test subsets, regardless the number observed points.
Number of observed points
237
Figure 2: The relationship between the interpola- tion accuracy and the number of observed points. Here we use the 2D instances in the NeSymReS test dataset as representatives
239
241
242
Accuracy on the TFRD-ADlet dataset: As shown in Table 2 CNP, although employing the neural network technique, still performs poorly with MAE as high as 96.674. In contrast, NIERT achieves the lowest interpolation error (MAE: 3.473), which is over one order of magnitude lower than CNP, ANP, BANP and TFFR-rranformmer. Moreover, when enhanced with the pre-training technique, NIERT can further decrease its interpolation MAE to be 1.897. Besides MAE, other metrics, say CMAE and MAE,, also show the superior of NIERT over the existing approaches (Table
243
245
247
248
Accuracy on the PhysioNet dataset: Table Table33suee suggests that on the PhysioNet dataset, NIERT also outperforms the existing approaches, e.g., when controlling the ratio of observed points to be 50%,
249
250
NIERT achieves an average MSE (x 10-3) of 2.868, significantly lower than RNN-VAL (13.418), L-ODE-RNN (8. 132). L-ODE-ODE (6.721) and mTAND-Full (4.139). Again, NIERT with the pre- pre-training technique shows better performance. The advantages of NIERT hold across various settings of the ratio the observed points.
252
253
254
Taken together, these results clearly demonstrate the power of NIERT for numerical interpolation in multiple application fields, including interpolating the scattered data generated using mathematical functions, reconstructing temperature fields, and interpolating time-series data.
256
257
4.3
Case studies of interpolation results
To further understand the advantages of NIERT, we carried out case studies through visualizing observed points, the reconstructed interpolation functions and the interpolation errors this subsection. More visualized cases are put in the Supplementary material.
259
SESP sterd
ABR 73h
14
Figure 3:
An example of 2D interpolation task extracted from Ne test set. The up-left figure shows the ground-truth function while the bottom-left figure shows the 22 observed points. The interpolation functions reported by NIERT and the existing approaches are listed on the top panel with their differences with the ground-trutt touth are list below
251 Figure 3 show a 2D instance the NeSymReS test set, respectively. As illustrated by these two figures, RBF performs poorly in the application scenario with sparse observed data. In addition, RBF 253 and MIR, especially ANP, cannot accurately predict values for the target points that fall out the range restricted by observed points. The CNP approach can only learn the rough trend stated by the
7

---

285
observed points, thus leading to significant errors. In contrast, BANP, TFR FRR-tranfformer and NIERT can accurately estimate values for target points within considerably large range, and compared with BANP and TFR-transformer, NIERT can produce more accurate results.
266
FigurdAshowe shows instance of temperature field reconstruction extracted from TFRD--ADet. From this figure, we can observe that when using pre-training technique, NIERT further improves its interpolation accuracy in the whole area.
270
Bass
TILT dlere
Dual
aru
Bill
Me Jate
1e 13red
80
15
Figure 4: An example of temperature field reconstruction task extracted from TFRD-ADlet test set. The up-left figure shows the ground-truth temperature field while the bottom-left figure shows the 32 observed points. The reconstructed results reported by NIERT and the existing approaches are listed on the top panel with their differences with the ground-truth temperature field are list below
4.4 Contribution analysis of observed points for interpolation
272 An idealized interpolation approach is expected effectively exploit all observed points with appropriate consideration of relative positions among observed points and target points well. To examine this issue, visualized the attention weight of each observed point all target points. These attention weights provide an intuitive description of the contribution by observed points.
274
As shown in Figure 5 when using TFR-transformer, the contributions by observed points considerably imbalanced: on one side, some observed points might affect their neighboring target points in a large region; on the other side, the other observed points have little contributions to interpolation. In contrast, when using NIERT, contributions by an observed point are much more local and thus targeted. More importantly, all observed points have contributions to interpolation.
278
279
280
281
These results demonstrate that NIERT can exploit the correlation between observed points and target points more effectively.
Dluwend 21 Felaeieen SIFT Aimik
Figure 5: Contributions by observed points for interpolation. The 2D instance is same to that used in Figure 3. We randomly select observed points and extract their attention weights from the final attention layer of NIERT and TFR- tranfrranffommer. These attention weights provide an intuitive description of the contribution by observed points. The contributions by other observed points are shown in Supplementary material
283
4.5
Ablation study
284
The effects of partial self-attention: For a specific interpolation task, the interpolation function is determined by the observed points only. Thus, an idealized encoding of observed points should not be affected by target points. To investigate the affects of target points, we evaluated NIERT on
286
test sets with various number of target points. Here, we compared two variants of NIERT, one with partial self-attention, and the other with vanilla self-attention. Both of these two variants were trained using the same training sets (the number of target points varies within [206, 246]).
288

---

290 As illustrated by Figure the variant with vanilla self-attention shows poor performance for the tasks with few target points, say less than 64 target points. In contrast, the variant with partial self-attention always performs stably without significant changes of accuracy.
291
293
The results clearly demonstrate that the partial self-attention mechanism allows NIERT to be free from the unexpected affects by the target points.
294
The effects of pre-training technique:
To investigate the effects of the pre-training technique, we show in Figure the training process of two versions of NIERT, one without pre-training technique, and the other enhanced with pre-training. As depicted by the figure, even at the first epoch, the pre- pre-trained NIERT shows a sufficiently high interpolation accuracy, which is comparable with the fully-trained BANP and transfranform.r. Moreover, the performance of the pre- pre-trained NIERT
296
298
improves in roughly the same convergence speed to the original NIERT, At the final epoch, the tra-ted NIERT decreases the interpolation error to be nearly half of that of the original NIERT.
300
301
302
These results clearly suggest that the experience learned by NIERT from the interpolation task in one application field has potential to be transferred to the interpolation tasks in other application fields.
rg varila sellettartion
MSE on 1D NesymRes reatoet
10
MAE on voretvattiii
20-1
30--
30-1
512
Number
Trannnn apochs
Figure 6: The robustness of NIERT to the number of target points. Here, two variants of NIERT are trained on ID NeSymReS dataset
Figure 7: The convergence of NIERT, NIERT with pre-training, and the existing approaches. Here, models are trained on TFRD-ADlet dataset
305
Discussion and conclusion
306
We present in the study accurate approach to numerical interpolation for scattered data. The specific features of our NIERT approach are highlighted by the full exploitation of the correlation between observed points and target points through unifying scattered data representation. At the same time, the use of partial self-attention mechanism can effectively avoid the interference of target points onto the observed points. The enhancement with pre-training technique is another special feature of NIERT. The advantages of NIERT in interpolation accuracy have been clearly demonstrated by experimental results on both synthetic and real-world datasets.
308
310
312
313
314
The current version NIERT has a computational complexity O(n(m + n)). thus cannot handle the interpolation tasks with extremely large amounts of observed points due to the limitations of GPU memory size. Compared with the lightweight traditional methods, our NIERT approach has a much larger model with expensive computation to learn complex function distribution, which limits its application in cost sensitive scenarios. How to reduce the memory requirement and computational cost is one of the future works. Additionally, it is interesting to combine NIERT and the traditional approaches based on basis functions to yield an approach with both high accuracy and interpretability.
316
318
320
321
We expect NIERT, with extensions and modifications, to greatly facilitate numerical interpolations in a wide range of engineering and science fields.
322
References
323
[1] Richard Franke and Gregory M Nielson. Scattered data interpolation and applications: A tutorial and survey. Geometric Modeling, pages 131-160, 1991.
325
326
2 Xiaoqian Chen, Zhiqiang Gong, Xiaoyu Zhao, Weien Zhou, and Wen Yao. Machine Learning Modelling Benchmark for Temperature Field Reconstruction of Hoar-Source Systems. arXiv

---

328
preprint N.Xiv:2008.08299 2021.
329
[3]
Mathieu Lepot, Jean- San-Baptistte Aubin, and François HLR Clemens. Interpolation in time series: An introductive overview of existing methods, their performance criteria and uncertainty assessment. Water, 9(10):79,, 2017.
331
[4] Satya Narayan Shukla and Benjamin M Marlin. Interpolation networks for irregularly sampled time series. arXiv preprint (XXxv:/909.07700 2019.
335
[5] GR3543275 Liu. An overview on meshfree methods: for computational solid mechanics. International Journal of Computational Methods, 13(05):163001,, 2016.
335
[6] Michael T Heath. Scientific Computing: An Introductory Survey, Revised Second Edition. Society for Industrial and Applied Mathematics, revised 2nd edition edition, 2018.
337
[7] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional Neural Processes. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1704-1713. PMLR, 10-15 Jul 2018.
339
341
342
343
[8] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Tch. Attentive Neural Processes. In International Conference on Learning Representations, 2019.
3M
345
348
[9]
Juho Lee, Yoonho Lee, Jungtack Kim, Eunho Yang. Sung Ju Hwang, and Yee Whye Teh. Bootstrapping neural processes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6606 6615. Curran Associates, Inc., 2020.
347
348
349
350
[10] Byung-Jun Lee, Seunghoon Hong, and Kee-Eung Kim. Residual neural processes. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4545-4552, 2020.
352
[11] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, pages 5998-6008, 2017.
354
356
[12] Jacob Devlin, Ming-Wee Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
358
[13] Hangbo Bao, Li Dong, and Furu Wei. BETT: BERT Pre-Training of Image Transformers. arXiv preprint (xXiv22106.082. 2021.
360
[14] Charles A Hall and W Weston Meyer. Optimal error bounds for cubic spline interpolation. Journal of Approximation Theory, 1652):105-122, 1976.
362
363
[15] William J Gordon and James A Wixom. Shepard's method of "metric interpolation" bivariate and multivariate interpolation. Mathematics of computation, 32(141):253-266 1978.
384
385
[16] Hans Wackernagel. Multivariate geostatistics: an introduction with applications. Springer Science& & Business Media, 2003.
388
387
[17] Michael JD Powell. Radial basis functions for multivariable interpolation: a review. Algorithms for approximation, 1987.
369
[18] Bengt Fornberg and Julia Zuev. The Runge phenomenon and spatially variable shape parameters
370
in RBF interpolation. Computers & Mathematics with Applications, 54(3):379-398, 2007.
371
[19] Qiqi Wang, Parviz Moin, and Gianluca Iacearino. A high order multivariate approximation scheme for scattered data sets. Journal of Computational Physics, 229(18):6343-632 2010.
10

---

373
[20] Ying Tai, Jian Yang, and Xiaoming Liu. Image super- resolutioo via deep recursive residual network. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 3147-3155, 2017.
374
2017.
376
[21] Juncheng Li, Zehua Pei, and Tieyong Zeng. From Beginner to Master: A Survey for Deep Learning-based Single-Image Super-Resolution arXiv preprint arXiv:2109. 2021.
377
378
[22] Satya Narayan Shukla and Benjamin Marlin, Multi-Time Attention Networks for Irregularly Sampled Time Series. In International Conference on Learning Representations, 2021.
[23] Noah Weber, Janez Stare, Arpit Mittal, Roi Blanco, and Lluís Marquez. Optimizing over bayesian last layer. In NeurlPS workshop on Bayesian Deep Learning, 2018.
381
382
[24]
Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel MerrerrtVosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners, In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
384
386
387
388
389
390
[25] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.00 1.06777, 2021.
381
382
[26] Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Paras- candolo. Neural Symbolic Regression that Scales. International Conference on Machine Learning, pages 936 945.945.20 PMLR, 2021.
393
394
[27] Mojtaba Valipour, Bowen You, Maysum Panju, and Ali Ghodsi. SymbolicGPT: A Generative Transformer Model for Symbolic Regression. arXiv preprint arXiv:2106. 14131, 2021.
396
397
[28] Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXiv preprint
2019.
[29] Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark. Predicting in- hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In 2012 Computing in Cardiology, pages 245-248. IEEE, 2012.
400
[30] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SeiPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261 2020. doi: 10. 1038003/4111 1e0p441592--00 010-0099966666 2.
403
405
406
407
408
409
410
[31] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 2014.
411
412
413
[32] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. Advances Neural Information Processing Systems, 2018.
414
415
[33]
Yulia Rubanova, Ricky T. Q Chen, and David K Duvenaud. Latent Ordinary Differential Equations for Iregularly-Sampled Time Series. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
417

---

Checklist
419
420
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
422
contributions and scope? [Yes]
423
(b) Did you describe the limitations of your work? [Yes] Limitations of our work are
424
briefly described in Section 5
425
(c) Did you discuss any potential negative societal impacts of your work? [No]
426
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
428
2. If you are including theoretical results...
429
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
430
(b) Did you include complete proofs of all theoretical results? [N/A]
431
3. If you ran experiments..
432
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] In the Supplementary.
434
435
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All the training details are in the Supplementary.
438
437
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We reported error bars in experiments on PhysioNet dataset. See Table
439
440
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Described in the Supplementary.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
443
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Mentioned in the Supplementary.
(c) Did you include any new assets either in the supplemental material
URL?? [No]
446
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating?
447
[No] The datasets used in our work are all open-sourced by their
448
creators.
449
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] The data we are using doesn't contain any personally identifiable information or offensive content.
451
452
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
454
455
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
456
457
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "olqNdbJl3KO", "reviewer": "Reviewer_3o5r", "summary": "In this work, the authors propose an accurate approach for numerical interpolation problems. The proposed NIERT method leverages the benefits of the self-supervised mechanism and pre-training technique to improve accuracy.  Several benchmark datasets are employed here to illustrate the effectiveness, compared with the baseline methods. Also, the authors provide some discussions on limitations and future plans, specifically, computational cost and interoperability. ", "questions": "1. numerical integration is important and several mathematical ways are good enough, what's the real motivation? why do you pursue a higher accuracy, is that necessary enough?  what's the real impact on a specific application?  I just saw a very general description but did not very understand the motivation.\n\n2. Most of the baselines are not competitive. I think the only strong one is TFR[2], what are the benefits compared with this method? why NIERT is better than that if both use transformer-based structures? \n\n3. Just wonder, this work is like transformer + X, kind of work, but how do you think the transformer will work better?  in other words, the study lack interpretability. Could you highlight the origin of the idea? \n\n4. About the correlation, I did not get how to explore the correlation, could you give more details?  \n\n5. The algorithm flowchart is not informative, could you polish it a little bit to show more details? \n\n6. The benchmarks mainly focus on 1d and 2d examples, how about the scalability of this algorithm?  I think the real challenge is high-dimensional numerical integration. If the algorithm can handle that, what's the accuracy-cost trade-off? \n\n", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. interesting topic and well-structured contexts\n2. solid benchmark and baseline comparisons \n\nWeaknesses\n\n1. Lack of real-world high-dimensional applications\n2. Motivation is not clear, only for numerical integration? any benefits for large-scale AI applications?\n3. Something like transformer + X work, what's the core contribution and benefits? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "_CRZGbkWnBX", "reviewer": "Reviewer_YVNJ", "summary": "The paper addresses the problem of scatter data interpolation. For that the authors used the encoder representations of Transformers. Since the observed points and the target points are from the same domain, they processed them in a unifying fashion. The paper used partial self-attention to avoid unexpected interference of target points on observed and other target points. The authors also used pre-training technique to improve accuracy. Experimental results were presented on two synthetic and one real datasets.", "questions": "- Can you please explain how this approach is \"self-supervised\" as mentioned in line 10?\n- I found the following claim to be too strong compared to provided evidence, since the reported pre-training was done on a synthetic data only. \"These results clearly suggest that the experience learned by NIERT from the interpolation task in one 299 application field can be transferred to the interpolation tasks in other application fields.\"", "limitations": "The novelty comes from unifying observed and target data representation and partial self-attention. This is more like the decoder of a transformer. Though it is a good observation, which resulted in promising performance, the novelty of the paper is limited.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Problem addressed in the paper in important and has practical significance.\n- The ablation study was helpful to understand the improvement contributed by each proposed factor.\n- The experimental result on real data is promising.\n\nWeaknesses:\n- Contribution from pre-training is not very significant.\n- \"an\" -> \"a\" in line 146, \"points and observed points in \"an\" unifying fashion\".\n- It will be good to see performance on more real data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "E4ElsIyDU3V", "reviewer": "Reviewer_93ZQ", "summary": "This is a work on numerical interpolation using transformers. The domain of interpolation is 1d-4d and extends to higher dimensions naturally. An input sequence consists of the following: a function is sampled from a distribution comprising a small set of polynomials, sinusoids and exponentials; input points are sampled uniformly from a cube [-1,1]^d; some of them are deemed observed and the rest are said to be target points (akin to masked inputs). Input points are embedded into a 16-dimensional space using a simple linear layer (Wx + b). \n", "questions": "1. How is RBF tuned in the experiments? In some experiments (say Figure 11), it seems the kernel bandwidth is too large. Also, in general, how are other competing methods tuned?\n\n2. Some errors at the boundaries incurred by RBF could be mitigated by using local polynomial kernels. There are also adaptive basis splines (R mgcv package, for example). Does it make sense to compare NIERT against such methods? It is possible that these methods are inferior to NIERT, but I think for fair comparison we should compare against the best-known classical methods.\n\n3. It is interesting how the inputs are embedded into a 16-dimensional space. What do these parameters look like for 1d and 2d inputs? It would be good to show these in the paper, with perhaps a plot in the 2d case.\n\n4. The NIERT model is large and the input sequence length is limited. Is it possible to reduce the model size? Sorry if I missed, but I did not see any ablation study justifying why you need such a large model. It should be made clear that the interpolation by NIERT is expensive compared to classical methods, say, via flops/wall-clock-time comparison. At least, it should be acknowledged that expensive computation is a known limitation of the model.\n\n5. In Table 3, the MSE increases for some methods as a larger proportion of the data is observed. Why is that?\n\n6. In Section E.2, the MSE on observed points is much less than that on the target points. Why is that?\n\n7. In 1d, does it make sense to sort within the observed data points and also sort within the target data points?\n", "limitations": "No red flags.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The empirical results are in favor of NIERT (the proposed method). It beats some interesting/relevant methods. \n\nWeakness: The number of model parameters in NIERT is high and computation is potentially much more expensive compared to classical methods (e.g.: RBF). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject