SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Interpolation accuracy on PTV dataset.
Interpolation approach	MSE (×10-3) on PTV test set
CNP	137.573
ANP	32.111
BANP	33.585
TFR-Transformer	17.125
NIERT	5.167
HINT (ours)	3.507
Table 4: Interpolation accuracy on TFRD dataset.
Interpolation approach	MAE (×10-3) on TFRD test set
HSink	ADlet	DSine
CNP	204.351	91.782	92.456
ANP	164.491	54.684	58.589
BANP	59.728	28.671	19.107
TFR-Transformer	64.987	27.074	29.961
NIERT	23.519	3.473	8.785
HINT (ours)	13.758	1.761	4.912
Regarding the local parameter  $K^{(0)}$ , we set it to the number of observed points in the interpolation task for Mathit-2D, Perlin, and TFRD datasets, as these datasets have a relatively small number of observed points. For the PTV dataset, we set  $K^{(0)}$  to 128 to accommodate its larger number of observed points. These values were selected to balance the local constraints with the available information in each dataset. Additionally, we set the minimum local parameter  $K_{\mathrm{min}}$  to 8 for all datasets. This value was chosen to ensure a sufficient level of local modeling and capture fine-grained details in the interpolation process.
Evaluation metric. Following the precedent set by previous works [2, 4] and for the purpose of facilitating comparison, we adopted the MSE as the evaluation metric for Mathit-2D, Perlin, and PTV datasets. Additionally, for the TFRD dataset, we utilized the MAE as the evaluation metric.
4.1 Interpolation Accuracy
We present comprehensive comparisons of test set average interpolation accuracy of our interpolation approach, HINT, with state-of-the-art approaches on the four representative datasets. On the Mathit-2D dataset, we observe a notable enhancement of  $8.34\%$  in accuracy compared to the best method NIERT (Table 1). Similarly, on Perlin and PTV datasets, our method demonstrates remarkable accuracy gains of  $30.23\%$  and  $18.61\%$  respectively (Tables 2 and 3). Furthermore, on three subsets of the TFRD dataset, i.e., HSink, ADlet and DSine, our method exhibits substantial accuracy gains of  $41.50\%$ ,  $49.29\%$  and  $44.08\%$  (Table 4) over NIERT. These compelling results validate the efficacy of our approach on interpolation tasks across diverse datasets, highlighting the superiority and versatility of our algorithm in both theoretical and real-world application scenarios.
4.2 Qualitative Analysis
Qualitative Comparison. Here we provide a more intuitive comparison of the interpolation performance between HINT and existing interpolation methods. We randomly select an interpolation task from the Perlin test datasets, visualizing the interpolation results and error maps of HINT and existing interpolation methods. As shown in Figure 3, for this interpolation task from the Perlin test set, our proposed HINT achieves the highest interpolation accuracy  $(\mathrm{MSE} \times 1.04 \times 10^{-4})$ . The error map of BANP exhibits strong error hotspots. TFR-Transformer and NIERT still have relatively large accuracy difference regions, while HINT has the largest area with minimal discrepancies (dark blue regions) on its error map.
Figure 3: Qualitative comparison on a 2D interpolation task extracted from Perlin test set. The top row shows the 64 observed points and the ground-truth function. The second row shows interpolation functions reported by HINT and the state-of-the-art approaches. The last row shows their error maps (their differences with the ground truth).
Figure 4: Visualization of output from each interpolation block on an example task (same as in Figure 3) from the Perlin test set. The top row of figures shows the output for each block, the middle row shows the accumulated output of these blocks, and the bottom row displays the error maps, indicating the differences between the cumulative predictions and the ground truth.
Residual Visualization. To investigate the role of each block in our proposed HINT, we visualized the output components of each block, as shown in Figure 4. We can find that the main block produces a relatively smooth overall function shape for the target function, lacking local details. The first residual block outputs more localized fluctuations in the target function with a smaller scale. The second residual block generates even smaller-scale components that appear to capture more local details and the last block contributes minuscule components. Observing the cumulative sum of the outputs from these blocks, we find that the accumulated function becomes progressively more detailed and closely resembles the ground truth of the target function as the outputs of the subsequent blocks are added. By examining the error maps of each cumulative sum, we observe that the output of the main block has strong error hotspots and subsequent blocks gradually refine the prior predictions until the final error map displays uniformly low errors throughout.
4.3 Ablation Study
Ablation on Main Design Choices of HINT. We validate the efficacy of the auxiliary loss term  $\mathcal{L}$  aux, masked observed points, and local constraints in HINT, as shown in the upper section of Table 5. When the auxiliary loss term is omitted (equivalent to setting the weight  $\lambda$  of the auxiliary loss term  $\mathcal{L}$  aux to 0), as indicated in the table as "HINT w/o  $\mathcal{L}_{\mathrm{aux}}$ ", HINT experiences a reduction in accuracy on all three datasets, with a notably significant impact on Perlin and TFRD-ADlet (decreasing by  $30\%$  and  $125\%$ , respectively). Omitting the local constraints ("HINT w/o KNN") also leads to decreased accuracy on these datasets, particularly affecting Perlin and PTV (with a decrease of  $20\%$  and  $43\%$ , respectively). Additionally, we assess the effectiveness of masked observed points ("HINT w/o masked obs.") In this setup, instead of using  $\hat{y_{O}}^{(l)}$ , we utilize  $\tilde{y_{O}}^{(l)}$  to compute residuals on the observed points. Experimental results in this configuration reveal a decline in accuracy across all three datasets, most notably on Perlin (a reduction of  $55\%$ ), underscoring the significance of maintaining a consistent distribution between observed and target points. These findings highlight the advantages of incorporating these design choices within our framework.
The lower half of Table 5 presents the ablation results on different local constraints. For a straightforward evaluation, we employed two ablation settings where the local constraint parameter  $K$  was set to either  $K^{(0)}$  or  $K^{(1)}$  for all interpolation blocks. Here,  $K^{(0)}$  represents the local parameter of the first block in the hierarchical local constraint, while  $K^{(1)}$  represents the local parameter of the second block. These two ablation configurations of local parameters represent a non-hierarchical approach with similar hyperparameters to our hierarchical approach. These results demonstrate the hierarchical design of local constraints has significant superiority over the non-hierarchical one in our method.
Ablation on Interpolation Blocks Number. Table 6 presents the results obtained by varying the number of blocks from 1 to 5 on two datasets, Perlin and PTV. The results indicate that when there
Table 5: Ablations on main design choices.
Interpolation approach	Interpolation accuracy on
Perlin (MSE×10-5)	PTV (MSE×10-3)	TFRD-ADlet (MAE×10-3)
HINT w/o Laux	7.630	3.656	3.985
HINT w/o KNN	7.017	5.001	1.944
HINT w/o masked obs.	9.071	3.547	2.773
HINT	5.848	3.507	1.761
HINT (K = K(0))	7.017	3.811	1.944
HINT (K = K(1))	8.083	4.791	2.975
HINT (Hierarchical)	5.848	3.507	1.761
Table 6: Ablation on block number  $L$  .
Block number	Interpolation accuracy on
Perlin (MSE×10-5)	PTV (MSE×10-3)
1	21.453	14.362
2	11.279	5.527
3	8.073	4.386
4	5.848	3.507
5	6.224	3.618
is only one block (i.e., no residual block), HINT performs poorly on both datasets. However, with the introduction of a single residual block (i.e., two blocks in total), the interpolation performance of HINT improves significantly. This highlights the effectiveness of incorporating the interpolation module. As the number of blocks increases, the interpolation accuracy of HINT shows incremental improvements with smaller margins. When the number of blocks reaches 5, the interpolation accuracy of HINT shows a slight decline on the Perlin dataset and remains relatively stable on the PTV dataset. These results suggest that after a certain number of blocks, the additional blocks no longer contribute to the learning of HINT, and the interpolation accuracy plateaus.
5 Related Work
Traditional Approaches for Scattered Data Interpolation. Traditional interpolation algorithms for scattered data construct the target function through linear combinations of explicit basis functions. For one-dimensional scattered data, techniques such as linear interpolation, Lagrange interpolation, Newton interpolation [11], and spline interpolation [9] are commonly employed. For more general scattered data residing in higher-dimensional spaces, Shepard's methods [26], and Radial Basis Function (RBF) interpolation [24, 6] are widely used. MIR [29] employs the Taylor series for interpolation and determines the weights by optimizing the interpolation error. In engineering applications, hierarchical interpolation methods have also been developed [5], which are often combined with spatial partitioning techniques or the selection of nearby points to reduce computational complexity. All these traditional methods are inherently non-adaptive, as they don't learn the function distribution from data. This restricts their interpolation precision and hampers their generalization across diverse scenarios.
Learning-based Approaches for Scattered Data Interpolation. Recent neural network-based scattered data interpolation algorithms can be broadly categorized into two groups: probabilistic and deterministic. Probabilistic methods, such as the family of Neural Processes [8, 14, 18, 30, 19, 21], estimate the conditional distribution of the target function based on observed points. Deterministic methods, such as TFR-Transformer [2] and NIERT [4], estimate the target function based on observed points with a focus on interpolation accuracy. Technically, these approaches use neural network modules to model the correlation between observed and target points, subsequently obtaining the representation and estimation of target point values. For instance, CNPs [8] employs MLPs to model the correlation between observed and target points, while ANPs [14], TFR-Transformer, and NIERT utilize attention mechanisms and Transformer layers [27] for the same purpose. Among these methods, NIERT employs the architecture of the Transformer encoder to uniformly handle observed and target points, resulting in enhanced interpolation accuracy. It stands as the current state-of-the-art neural interpolation algorithm, serving as inspiration for our interpolation block design. Notably, these current neural interpolators have thus far neglected the information regarding interpolation residuals at observed points, which constitutes a shared limitation.
Residual Learning and Decomposition Structure. Residual connections in neural networks can be traced back to residual modules in ResNet [10] and DenseNet [13]. In this module, the output of each stack is fed into subsequent stacks, for better optimization of deep neural networks. For a different propose, in time series forecasting, N-BEATS [22] presents a doubly residual framework with two distinct residual branches: one branch for backcast prediction and another branch for the forecast prediction at each layer. The residual between the backcast prediction and history input is
exploited to forecast prediction. N-HiTS [1] integrates hierarchical sparse prediction into the similar doubly residual framework, to enhance both forecasting accuracy and interpretability. However, these techniques are primarily tailored for fundamental time series signal decomposition and display a notable limitation when tasked with learning sparse and scattered data interpolation. Although the exploitation of residual is successfully applied in time series forecasting, it is hard to apply in scattered data interpolation straight-forwardly for sparsity and irregularity of scattered data.
In this paper, we propose HINT, a novel hierarchical residual refining framework for scattered data interpolation. By employing a series of residual blocks in a coarse-to-fine fashion and incorporating hierarchical local constraints in residual correlation modeling, HINT effectively improves interpolation accuracy. Our extensive validation across diverse theoretical and application scenarios demonstrates the superior performance of HINT compared to existing state-of-the-art interpolation methods. We acknowledge the limitations of our approach regarding the hyperparameter  $K$  in hierarchical local constraints, and we plan to explore methods for learning adaptive  $K$  values in future work.
We would like to thank the National Key Research and Development Program of China (2020YFA0907000), and the National Natural Science Foundation of China (32370657, 32271297, 82130055, 62072435) for providing financial supports for this study and publication charges. The numerical calculations in this study were supported by ICT Computer-X center.
[1] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler-Canseco, and Artur Dubrawski. N-hits: Neural Hierarchical Interpolation for Time Series Forecasting. arXiv preprint arXiv:2201.12886, 2022.
[2] Xiaoqian Chen, Zhiqiang Gong, Xiaoyu Zhao, Weien Zhou, and Wen Yao. A Machine Learning Modelling Benchmark for Temperature Field Reconstruction of Heat-Source Systems. arXiv preprint arXiv:2108.08298, 2021.
[3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805, 2018.
[4] Shizhe Ding, Boyang Xia, Milong Ren, and Dongbo Bu. NIERT: Accurate Numerical Interpolation through Unifying Scattered Data Representations using Transformer Encoder. arXiv preprint arXiv:2209.09078, 2023.
[5] Michael S Floater and Armin Iske. Multistep Scattered Data Interpolation Using Compactly Supported Radial Basis Functions. Journal of Computational and Applied Mathematics, 73(1-2):65-78, 1996.
[6] Bengt Fornberg and Julia Zuev. The Runge Phenomenon and Spatially Variable Shape Parameters in RBF Interpolation. Computers & Mathematics with Applications, 54(3):379-398, 2007.
[7] Richard Franke and Gregory M Nielson. Scattered Data Interpolation and Applications: A Tutorial and Survey. Geometric Modeling, pages 131-160, 1991.
[8] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional Neural Processes. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1704–1713. PMLR, 10–15 Jul 2018.
[28] Di Wang and Xi Zhang. Modeling of a 3D Temperature Field by Integrating a Physics-Specific Model and Spatiotemporal Stochastic Processes. Applied Sciences, 9(10), 2019.
[29] Qiqi Wang, Parviz Moin, and Gianluca Iaccarino. A High Order Multivariate Approximation Scheme for Scattered Data Sets. Journal of Computational Physics, 229(18):6343-6361, 2010.
[30] Noah Weber, Janez Starc, Arpit Mittal, Roi Blanco, and Lluis Márquez. Optimizing over a Bayesian Last Layer. In NeurIPS Workshop on Bayesian Deep Learning, 2018.