Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this work, the authors propose an accurate approach for numerical interpolation problems. The proposed NIERT method leverages the benefits of the self-supervised mechanism and pre-training technique to improve accuracy.  Several benchmark datasets are employed here to illustrate the effectiveness, compared with the baseline methods. Also, the authors provide some discussions on limitations and future plans, specifically, computational cost and interoperability. 
strengths_and_weaknesses: Strength

1. interesting topic and well-structured contexts
2. solid benchmark and baseline comparisons 

Weaknesses

1. Lack of real-world high-dimensional applications
2. Motivation is not clear, only for numerical integration? any benefits for large-scale AI applications?
3. Something like transformer + X work, what's the core contribution and benefits? 
questions: 1. numerical integration is important and several mathematical ways are good enough, what's the real motivation? why do you pursue a higher accuracy, is that necessary enough?  what's the real impact on a specific application?  I just saw a very general description but did not very understand the motivation.

2. Most of the baselines are not competitive. I think the only strong one is TFR[2], what are the benefits compared with this method? why NIERT is better than that if both use transformer-based structures? 

3. Just wonder, this work is like transformer + X, kind of work, but how do you think the transformer will work better?  in other words, the study lack interpretability. Could you highlight the origin of the idea? 

4. About the correlation, I did not get how to explore the correlation, could you give more details?  

5. The algorithm flowchart is not informative, could you polish it a little bit to show more details? 

6. The benchmarks mainly focus on 1d and 2d examples, how about the scalability of this algorithm?  I think the real challenge is high-dimensional numerical integration. If the algorithm can handle that, what's the accuracy-cost trade-off? 


limitations: yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper addresses the problem of scatter data interpolation. For that the authors used the encoder representations of Transformers. Since the observed points and the target points are from the same domain, they processed them in a unifying fashion. The paper used partial self-attention to avoid unexpected interference of target points on observed and other target points. The authors also used pre-training technique to improve accuracy. Experimental results were presented on two synthetic and one real datasets.
strengths_and_weaknesses: Strengths:
- Problem addressed in the paper in important and has practical significance.
- The ablation study was helpful to understand the improvement contributed by each proposed factor.
- The experimental result on real data is promising.

Weaknesses:
- Contribution from pre-training is not very significant.
- "an" -> "a" in line 146, "points and observed points in "an" unifying fashion".
- It will be good to see performance on more real data.
questions: - Can you please explain how this approach is "self-supervised" as mentioned in line 10?
- I found the following claim to be too strong compared to provided evidence, since the reported pre-training was done on a synthetic data only. "These results clearly suggest that the experience learned by NIERT from the interpolation task in one 299 application field can be transferred to the interpolation tasks in other application fields."
limitations: The novelty comes from unifying observed and target data representation and partial self-attention. This is more like the decoder of a transformer. Though it is a good observation, which resulted in promising performance, the novelty of the paper is limited.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This is a work on numerical interpolation using transformers. The domain of interpolation is 1d-4d and extends to higher dimensions naturally. An input sequence consists of the following: a function is sampled from a distribution comprising a small set of polynomials, sinusoids and exponentials; input points are sampled uniformly from a cube [-1,1]^d; some of them are deemed observed and the rest are said to be target points (akin to masked inputs). Input points are embedded into a 16-dimensional space using a simple linear layer (Wx + b). 

strengths_and_weaknesses: Strengths: The empirical results are in favor of NIERT (the proposed method). It beats some interesting/relevant methods. 

Weakness: The number of model parameters in NIERT is high and computation is potentially much more expensive compared to classical methods (e.g.: RBF). 

questions: 1. How is RBF tuned in the experiments? In some experiments (say Figure 11), it seems the kernel bandwidth is too large. Also, in general, how are other competing methods tuned?

2. Some errors at the boundaries incurred by RBF could be mitigated by using local polynomial kernels. There are also adaptive basis splines (R mgcv package, for example). Does it make sense to compare NIERT against such methods? It is possible that these methods are inferior to NIERT, but I think for fair comparison we should compare against the best-known classical methods.

3. It is interesting how the inputs are embedded into a 16-dimensional space. What do these parameters look like for 1d and 2d inputs? It would be good to show these in the paper, with perhaps a plot in the 2d case.

4. The NIERT model is large and the input sequence length is limited. Is it possible to reduce the model size? Sorry if I missed, but I did not see any ablation study justifying why you need such a large model. It should be made clear that the interpolation by NIERT is expensive compared to classical methods, say, via flops/wall-clock-time comparison. At least, it should be acknowledged that expensive computation is a known limitation of the model.

5. In Table 3, the MSE increases for some methods as a larger proportion of the data is observed. Why is that?

6. In Section E.2, the MSE on observed points is much less than that on the target points. Why is that?

7. In 1d, does it make sense to sort within the observed data points and also sort within the target data points?

limitations: No red flags.
review_title: null

Meta-review: metareview: This work proposed an accurate approach that called NIERT for numerical interpolation problems. The motivation of the proposed NIERT is not explained very well. The major limitation lacks of  real-world  applications in high-dimensional datasets. Due to the main concern of the work, this shoud be necessary.   
confidence: Less certain
Decision: Reject