Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose to derive a technique allowing gradient based learning of splines (or piecewise polynomials) and to apply such a method to standard tasks such as PDE solving/approximation and other function approximation problems.
main_review: Some notations are counter intuitive e.g. f_I(i) since F is a vector and F_I is a restriction (in term of dimension) of this version, it seems unnatural to call it as a function (especially since it is not well introduced/defined).

The problem of both optimizing the spline (per-region) function parameters and the knots of the partition is well studied yet no reference is provided (see all the literature on adaptive spline, and many DeVore references).

Theorem1 is merely a direct calculus result that is straightforward and has been known in the spline community for a while (it is trivial to show the block structure based on the fact that each subset of parameters is only ‘’active’’ within its partition region)

Many english formulation that are not appropriate ‘’the backward pass is a bit more tricky’’ and so on. Additionally the whole organization and choice of organization is odd (e.g. bold paragraph in the intro and so on).

Many sentences seem incomplete e.g. “This is powerful since ….into equivalent energy functionals”.

No comparison is done against alternative spline fitting methods including the ones that allow adaptivity of the partition though mesh refinement (which is the whole direction of PDEs where mesh refinement is able to somewhat compete with adaptive knot splines).


----- AFTER REBUTTAL ------

I thank the authors for their detailed answers and acknowledging the drawbacks I have raised. I am raising my score from 4 to 5 for the following reason. In my opinion, some of the technical contributions that are the pillars of the paper have been obtained before (e.g. using Gaussian (or more generally narrow support function) approximation for gradient computation in https://arxiv.org/pdf/1706.04698.pdf https://arxiv.org/pdf/1901.09948.pdf, gradient sparsity which is (as also pointed out by ZLi8) known and already leveraged in various fields such as PDE approximations (the sparsity of the gradient is what allows most methods to actually be tractable on a fine mesh)). And when considering the remaining contributions, I am uncertain that this paper passes the acceptance threshold. However, some reviewers are pointing out that this paper offers an interesting application and find the submission useful, hence I am raising my score accordingly.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presented a differentiable approximation to splines for use as a differentiable layer in differential programming. The main idea is to exploit the sparsity of the resulting Jacobian matrix for the spline approximation operation, and to use a Guassian approximation for the non-differentiable Direc delta spike functions from the Cox-de Boor base function. The authors used this differentiable spline approximation in three application settings: 2d image segmentation; 3d surface fitting for point clouds using NURBS, and solving a PDE (Poisson Equation).
main_review: The paper is rather math dense and it took me some time to understand the main proposal set forth by this work. The main pros and cons are concluded below.

Pros:
+ I think the differentiable approximation of the spline functions is a very interesting topic, potentially allowing the integration of spline function approximations into the differentiable programming paradigm. The current field of deep learning is heavily dominated by "neural" (i.e., Multi-Layer Perception) approximations for parameterizing high dimensional functions, which are flexible and extensible, but has its own set of limitations. I think generalizing the idea of differentiable splines as function approximations is attractive.
+ The idea of smoothing the discontinuous delta functions using Gaussian with small bandwidth feels intuitive.
+ I like the error intervals reported in the experiments.

Cons:
- Though the paper is quite content heavy, most of the stuff is not new (e.g., when computing the Jacobian, only the gradients corresponding to the specific segments are non-zero). The main contribution seems to be that the authors proposed to smooth the step function by approximating it with a Gaussian kernel. However, this main aspect does not seem very thoroughly studied. 
    - For example, how does the bandwidth of the Gaussian affect the results (with empirical evidence)? I would assume a bandwidth too small will result in nasty gradients, and a bandwidth too large will produce inaccurate approximations. 
    - What if you replace the step function in the base case (Eqn. 4) with two sigmoids of different bandwidth instead of using a Gaussian? What about using two tanh? Different options can be explored in greater depth.
- The experimental section is not very well executed.
    -  Following the point above, ablation studies for the choice of smoothing function are lacking.
    - The experiments lack details and context. For example, I find it difficult to understand how the differentiable spline layer is used in the image segmentation experiments. The only description seems to be "We begin with implementing a 2D piecewise constant splines regression approach for the image segmentation problem using a UNet" (l 242). What does that mean? Are the shape contours represented using NURBS? How is the output parameterized? What parameters are the neural network outputting after the UNet feature map? The baseline method is simply described as "without the DSA layer", with not enough details about the DSA layer for the reader to understand. The evaluation is a little hard to grasp. How is the Jaccard score defined? Is it the same as pixel-wise IoU score? For the PDE experiment, is my understanding correct that the first part looks at directly optimizing the Lagrange polynomial based representation (by minimizing PDE residuals), and the second part looks at training for a learned prior over a dataset (using a UNet)? Similar question to the first experiment, how are the U-Net feature maps associated with the spline polynomials?
limitations_and_societal_impact: In general I think differentiable spline is an interesting topic. A few areas of improvement I would love to see:
1. More comprehensive ablations on the smoothing part for differentiability.
2. Clear details for the experiments. If space is an issue, consider moving more details to the suppmat.
3. Some performance benchmarking would be nice-to-have. How does the algorithm performance scale with number of knots / degree? How fast is this layer compared with other NN layers?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents sparse Jacobians/backward algorithms for several spline formulations and motivates their use in differentiable algorithms. The formulations include general splines, NURBS, and basis functions for FEM solvers. 
The resulting differentiable modules are applied to the three proof of concept tasks of image segmentation, point cloud reconstruction, and solving Poisson equations, where they show promising results.
main_review: Originality:
- To my knowledge, this is the first method that allows backpropagating gradients to the knot vectors of NURBS curves. The authors achieve this by framing the box function as convolutions with Dirac impulses and approximating those Dirac impulses using Gaussian functions. This seems to be an interesting idea, which possibly could be applied to other situations as well.
- The observation that the same principles can be applied to backpropagate through functions in FEM solvers is interesting and has potential. There might be several useful applications.
- Utilizing the sparsity of the Jacobian (induced by the local support property) to develop efficient backward algorithms through splines was already done by previous work (e.g. SplineCNN [1]). However, this is the first work that presents that concept in a principled way for general splines.

Quality:
- The paper is technically sound
- The experiments show that the method is indeed applicable in several domains.
- All three experiments are proofs of concept. I wonder what hinders the application to more sophisticated tasks. At least for image segmentation it should be possible to compare to SOTA methods on common segmentation benchmarks. Instead, the authors only compare to their own baseline.

Clarity:
- The paper is well-written, well structured and nice to read.
- The paper would profit from some explanatory figures for the general splines in Section 2/2.1 and the FEM method in Section 2.3.

Significance:
- This is a paper which can spark a lot of applications in several areas, if the presented methods turn out to be practical. There are several interesting ideas and concepts.

All in all, I recommend to accept the paper for the mentioned reasons. The experiments certainly leave room for improvement, mostly because the authors are not comparing to other SOTA methods. However, the given experiments reach the bar in that they showcase the different potential applications as proofs of concept.

Questions:
- Is the approximation of Basis functions (using Gaussians instead of Dirac impulses) only done in the backward step or also in the forward step? If it is done in both I wonder if the approach would also work if the normal basis is used in the forward computation, while the "surrogate" is used for gradient computation. 
- How efficient is the FEM solver? Would it be possible to apply it to larger scale tasks, for example volumetric reconstruction tasks as in SIREN [2]?
- In the given applications, the knot vectors (and therefore the gradients computed for them) lie in 2- or 3-dimensional space. When performing gradient descent in such low dimensional spaces, one often runs into bad local minima. Did the authors face any of such challenges when optimizing their positions?


[1] Fey et al.: SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernels, CVPR 2018

[2] Sitzmann et al.: Implicit Neural Representations with Periodic Activation Functions, NeurIPS 2020

----------------- After Rebuttal ---------------

I thank the authors for answering my questions. I will keep my score.

The other reviewers correctly note that in this paper some contributions are oversold, as both ideas, utilizing sparsity of the Jacobian for efficient backprop and computing gradients through Dirac functions via smooth approximations are not novel but already present in previous work. I agree and encourage the authors to adjust the statements in future versions. 

However, all in all, I see a lot of value in this work, as it is the first that builds a full differentiable building block for general splines, which can be useful to many researchers.
limitations_and_societal_impact: The authors could further discuss the limitations of their method. This would help to assess the applicability of the presented approach for real-world tasks. I see no immediate societal impact.
review_title: null

Meta-review: Congratulations, the paper is accepted to NeurIPS 2021!
Please reformulate your contributions in light of reviewer's qxvV comments. 
Please incorporate other corrections and additions from the rebuttal/reviews.
Decision: Accept (Poster)