Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a flexible, efficient, and task-agnostic open source library for nonlinear least squares problems. By integrating with PyTorch, the package allows one to learn and solve many tasks in computer vision and robotics in an end-to-end and deep fashion. The library provides support for common objectives, robust functions, optimizers (*e.g.*, GN, LM), as well as useful utilities (*e.g.*, Lie-algebra). The underlying implementation is carefully designed such that it is memory and computationally efficient.

strengths_and_weaknesses: **Strength**
- I am personally a big fan and a practitioner of deep structure models. Many of my previous work have combined deep neural networks with optimization (some are NLS while some are not). In order to train the whole model end to end, I had to write all these things myself, such as unrolling GN step, lie-algebra transformation, etc. It was fun, but indeed a bit cumbersome. It is thus very exciting to see that there is a flexible and modularized library that can greatly improve the speed of our future development. I can foresee the library being adopted by a wide variety of the community.
- The authors provides several case studies (which is just the tip of the iceberg) across computer vision and robotics and showcase how they can be implemented easily and cleanly with the task-agnostic interface.
- Theseus provide additional support/integration for sparse linear solvers, which can be used independently as standalone pytorch functions.
- Thorough analysis on computational cost and memory usage.
- I also like how the library integrates many (latest) techniques, such as projection operator for Lie-algebra, implicit differentiation, etc. It allows one to experiment with more crazy stuff such as unrolling tons of steps, or obtain more stable results with gradient explosion.


**(Minor) Weakness**
- While the aim of the package is to support generic nonlinear optimization, the current state seems to focus majorly on least square problem (*e.g.* GN solver). The title is thus a bit mis-leading imo.
- Following above, many state-of-the-art algorithms in vision and robotics adopt LM solver with learnable, data-driven $\lambda$. I don't see why this function is unavailable, or what's the major difficulty/block on this.

**Correctness** 
- I'm not aware of any correctness issue in this paper.


**Minor**
- I personally gave it a try and use Theseus to replace my personal implementation on GN solver and lie-algebra (for 3D pose estimation). While I didn't observe significant speedup (which aligns with the analyses in the paper), the code is indeed much cleaner ;)
questions: Please see the above weakness section.
limitations: The limitations are properly discussed.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents Theseus, a software library allowing for a new layer in pytorch models based on differentiable nonlinear least squares (DNLS) solves.  That is, the input to the layer specifies a NLS problem which in the forward pass the layer solves and in the backward pass, the layer provides the adjoint derivates, i.e. derivatives for the solution to the NLS problem in terms of the problem parameters.  This creates a nested optimization problem in which an inner NLS solver iterates to find the NLS solution as part of the outer optimization of a pytorch model by gradient descent.  

The authors are clear that they are not introducing this technique but rather provide an extensive list of citations for its use in robotics and vision communities for a solving an array of problems in which combining the flexibility of the deep learning with the constrained structure of NLS leads to good solutions.  However, the authors of this work note that in other works this problem was solved using custom application specific implementations which were in many cases inefficient.  They remedy the problem by providing an efficient and scalable package which can work for many applications.

The paper contains many examples of use cases for the method, explanations of the many options provided for both the forward and backward modes, and empirical profiling comparing the different options and showing improved efficiency versus the previous NLS solver at large scale.   
strengths_and_weaknesses: ## Strengths

- The authors make a compelling case for the usefulness of the method implimented by their package, giving an extensive bibliography of uses in robotics and vision.
- They also implement two novel (in this context) methods for computing the adjoint derivatives for DNLS: Implicit differentiation and Direct Loss minimization.  Experiments in Sec. 5.1 show an improvement in efficiency and accuracy relative to unrolling or truncated differentiation. 
- The library is very flexible providing two different sparse NLS solvers: CHOLMOD and cudaLU, and explaining the advantages and drawbacks of each.  Four methods for the backward mode are provided.  There is also support for optimizing variables in Lie groups such as orientations in SO(3) which are quite important in robotics and vision and tricky to optimize over.   
- Support for batching and automatic vectorization gives the DNLS layer similar levels of parallelism to other pytorch layers.  
- The supplementary material includes the complete code of the package including scripts and instructions to run several different examples in the paper.  The appendix includes nice explanations of the examples and proofs justifying the novel backward modes.  Given that the purpose of the paper is provide a solution which works flexibly across different applications, I found seeing these examples script and explanations helpful to understanding this. 

## Weaknessess
- Some of the empirical evaluations are somewhat hard to interpret.  Figure 2 left is not a very clear plot and it is hard to find a strong conclusion of when to prefer either sparse solver (although the preference over dense is clear.)  
- I understand that it is a prior assumption from the literature that the combination of deep learning and NLS afforded by this method is a worthwhile and superior approach in many applications relative to either DL or NLS independently.  It still would have been convenient if this comparison could be made explicitly for one of the examples. 
- Figure 4, right shows that the choice of backward method inside Theseus have an impact on the ultimate performance of the model.  The authors note this is probably application dependent.  This seems like a critical aspect for end users and should probably be explored more. 


questions: ## Minor Point
- Line 74: $\in$ should be $\subset$.  Do the subsets need to be disjoint? 
- Line 76: parallelism issue.  ``Euclidean vector or a matrix Lie group element''
- Line 91: maybe a small semantic point, but this does not really seem like a second-order method since the 2nd derivatives do not need to be computed, which seems like an advantage.  I have seen the derivation and I understand that $J^T J$ is the second order term of the objective after linearizing the model, but it seems one should be careful to be precise here.
- Line 106: "the lastest"

## Questions
- Figure 1: It seems the larger batch has smaller speed up.   Why do you think that is? 

## Post-Response Edit
I have read the other reviews and author responses.  These strengthen my positive assessment and I maintain my score. 
limitations: The paper discusses limitations of various alternatives throughout the work, for example, the fact that the CHOLMOD spare solver is tied to the CPU.  A limitations section at the end lays out other limitations, for example, incompatibility with hard constraints.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents Theseus, a library for optimizing through differentiable nonlinear least squares. The library allows for end-to-end learning for various structured learning problems like pose estimation, and homography estimation among others. The library is task agnostic, and provides an efficient implementation for dense and sparse problems, along with a variety of optional backprop methods for the underlying second-order optimizer.
strengths_and_weaknesses: **Strengths**: 
1. The proposed library is task-agnostic allowing for end-to-end learning across a variety of robotics and vision problems.
2. The implementation supports a variety of backpropagation approaches for taking gradients including autodiff.
3. The paper shows either comparable or better results as compared to Ceres solver.
4. The motivation is well supported with several examples and has value to the community.


**Weaknesses** 
1. Technically, there do not seem to be any flaws in the proposed approach.
2. The authors cite several papers that already implement related approaches. It will be good to compare a general purpose solver like Theseus with application specific implementations.
3. The writing needs 
questions: 1. Reiterating from above, I would like to see comparisons with existing task specific implementations to see differences if any.
2. In terms of the optimization, the approach builds on a two step optimization scheme. However, this may lead to instabilities in optimization especially with the choice of wrong hyperparams. How much does the choice of hyperparameters affect the overall performance of the proposed approach?
limitations: The authors have been upfront about the limitations and have clearly addressed them in the settings.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper describes a convenient, systematic, and clean DNLS library. It could be possibly used to develop many differentiable algorithms.

It has several useful features, such as Differentiable Lie groups and kinematics. 

Although it’s not novel, this feature makes optimization of matrices and robots kinematics more easily in pytorch. It could be useful when optimizing non-Euclidean variables, for example, joints in robotics, pose matrices in multi-view geometry like SLAM and SfM.

Automatic Vectorization: It could possibly be one novelty, but was not described in detail.  In L232, “we make use of the single instruction-multiple-data (SIMD) protocol by automatically detecting and vectorizing operations of the same type, significantly reducing the overhead for forward and backward passes.” 

GPU Acceleration for a Solver:  They design and implement a differentiable cudaLU solver, which itself could be useful. Cuda solver is usually closed-source. Making a differentiable version of it takes effort.

 Four Differentiation Strategies:  In Sec 4.3, they provide 4 different strategies for users, suitable for different problems. It helps the user to select the most suitable one.
strengths_and_weaknesses: STRENGTH:  potentially high utility for the ML community interested in differentiable programming paradigm

With this library, researchers can more easily design differentiable programming systems while spending less time on the differentiation part (or at least can quickly code up "proof of concept" systems using this general tool), esp. considering that the optimization is one of the hardest parts when making a system differentiable.

WEAKNESS:  novelty in this paper is unclear, if not low, as it is currently presented and described.

Another possible weakness is that, similarly to other methods that achieve differentiation by pytorch, the overhead could be high compared to simple forward C++ code, especially when the data size is small and the advantage of parallelization is not significant. An example is in Figure 3, where the existing Ceres (black dashed) outperforms most of the methods in this new library on small data. 



questions: Can you highlight or give more detail on the  key novelties of the design choices & algorithms in the library itself?
limitations: Not discussed clearly.
review_title: null

Meta-review: metareview: This paper presents Theseus, a software library which provides a new layer in the form of a differentiable nonlinear least squares (DNLS) solver. Forward pass solves the problem and the backward pass provides derivates for the optimum with respect to parameters. The reviewers uniformly appreciated the presentation of the paper and the usefulness and packaging of the proposed library. The library's features such as sparse solvers and Lie group algebra was also appreciated by the reviewers. Overall the paper and the library are a strong contribution to the Neurips community and hence I am happy to recommend for acceptance. I would urge the authors to make sure they follow through on feature requests and other suggestions made by the reviewers. 
confidence: Certain
Decision: Accept