Venue: NeurIPS 2019 Workshop Program_Transformations

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: “The naïve approach to higher order differentiation, supported by most AD frameworks, is to compose derivative operation until the desired order is achieved.”  I think that’s awfully optimistic: my impression is that most AD system either don’t handle nesting at all, or manifest all sorts of bugs (both numeric incorrectness and surprising inefficiency) when nesting is used.

I think, but am not absolutely positive, that some of the equations and definitions in Pearlmutter and Siskind (POPL-2007, doi:10.1145/1190215.1190242), if truncated at 𝒪(𝜀ⁿ), are equivalent to some of the formulas used here. On the other hand, they leave some symmetries on the floor.  Basically, they assume different nested derivatives are always on unique variables, whereas that situation is focused on here and the redundancies caused by permutational symmetries are exploited. It does seem like even when distinct variables are in play, some clever factorization might make the symmetries manifest and allow them to be exploited anyway.

Some of the formulas here are in "big sum" form when they could instead be formulated recursively while giving identical (or sometimes superior, due to factoring out comment subexpressions) numeric operation counts. The big sum is traditional in calculus texts. But, the recursive formulation lends itself to more concise implementation.

- ref [6], title contains proper name which should be capitalized, and maybe a missing accent

Recommendation:

This is clearly on-topic. Might be a bit technical for an oral. In any case, an oral (or poster, for that matter) would beg for a bunch of color-coded examples to really show the way things get redundant in a naïve formulation, and how this can be avoided by the right book-keeping.
review_title: Sweats the details of multivariate higher-order forward mode

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors present their implementation of Taylor-mode automatic differentiation (AD) in the JAX library. Taylor-mode refers to the computation of derivatives using the higher-order chain rule (also known as Faà di Bruno's formula) in a forward AD setting where higher-order derivatives up to a chosen level are propagated simultaneously during the forward evaluation of a function. This yields higher-order derivatives with better scaling compared with the alternative approach of nested application of first-order AD operations.

One of the advantages of this mode is the ability to exploit shared evaluation, meaning that terms computed for lower-level derivatives might also show up in the higher-levels and therefore do not need to be computed more than once as all levels are computed in the same shared evaluation. This is nicely explained in the paper, but I’m also curious whether the same outcome (shared evaluation) could also be achieved in a source-transformation-based nested AD case combined with compiler optimizations noticing and exploiting shared terms in the resulting higher-order derivative code.

I think this paper clearly belongs in this workshop. The technique implemented has been known in the automatic differentiation literature (e.g., Griewank and Walther, 2008) as the authors clearly mention, but not so much by the machine learning audience of this workshop.

I have the following (mostly minor) comments to improve the paper in the camera-ready version:

- I think a brief explanation of “what is JAX?” is needed to aid the general reader, and you need to provide a reference or at least a URL of the project.
- In the notation of, e.g., “\partialf(x)[v]”, it would be better if you introduced what exactly the square brackets are used for.
- I think the symbol “K” in equations 1 and 2 corresponds to “k” elsewhere in the paper. Also in equations 4 and 5, the “end” in “\sigma_end” also refers to “k” if I’m not mistaken. It would be good to use the same symbol everywhere.
- In Section 2, starting the introduction with the case of “f = g o h” and then continuing the rest of the section and appendices A and B with the case of “f o g” is slightly confusing and not ideal. I think the beginning of Section 2 can be improved in general.
- In Section 4 you say “dynamical systems are defined directly in terms of their derivatives”. I think this is correct for flows, but it is an unnecessary generalization. There are dynamical systems that are not defined in terms of derivatives, such as maps, discrete-time systems, etc.
- The paper ends abruptly after Section 4. It would really help to have some unifying conclusion.

review_title: Higher-order derivatives without nested automatic differentiation

Meta-review: This paper contributes strongly to bridging the gaps between the AD and ML communities, which is a primary goal of this workshop.
Decision: Accept (Oral)