Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: I apologize for the late review since I did not realize I was assigned an emergency review for this paper.

## Summary

The paper proposes a new DSL language embedded in Julia that can represent reversible programs. They provide mechanisms to automatically reverse a program given its forward definition in their DSL. Their language would allow reverse mode automatic differentiation to be implemented in lieu of the standard checkpointing schemes. 

## Strengths

* A DSL to express reversible programs and implementation in a well-known language
* Simple description

## Weaknesses

* Very limited evaluation on two benchmarks.
* Presentation issues that distract from the main point of the paper.
* Vague language description and limitations not mentioned.


## General discussion and Questions for the authors

I liked the high-level idea of this paper, however the presentation and writing need to be drastically improved in order to be accepted for publication. Also, the current results section is limited to two examples and I recommend adding more case studies to validate the usefulness of their DSL.

At a high-level the authors can do the following to improve their presentation.

* Some of the language implementation details distract the readers from understanding the design aspects. For example. “One can input “←” and “→” by typing “\leftarrow[TAB KEY]” and “\rightarrow[TAB KEY]” respectively in a Julia editor or REPL”. The authors can omit the implementation details or move it to appendices and focus more on the design decisions they took and the reasoning for those decisions.
* The authors can present their DSL language more formally. For example, they can give operational semantics for their forward program and reversed program using deductive rules.
* What’s supported by the language and what’s not is not clearly mentioned. Specifically, the authors should mention limitations and scope of the language.

Evaluation needs to be improved

* It is not conclusive from the results presented that the DSL (or reverse mode in general) produces better code than a checkpointing strategy. It helps in case of bundle adjustment, however has an overhead in the GMM implementation.
* Coverage - To showcase the expressivity of the DSL, the authors should implement a known NN architecture (e.g., even a small resnet) and show how training and inference speeds vary.

review_title: Evaluation and presentation needs to be improved

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper draws connections between reversible programming and reverse mode automatic differentiation and introduces a reversible DSL in Julia that can be used to calculate first and second order gradients.

I reviewed a previous version of this paper for NeurIPS.

I really like the idea of reversible programming and I think that a clear introduction of reversible programming and its use in automatic differentiation could be of interest to the machine learning community. However, I feel that this paper fails to clearly explain the use of reversible programming and its trade-offs compared to checkpointing and other existing approaches.

As a paper, the first section is great, but then the authors leave me with many questions: How do checkpointing and reversible programming differ in memory usage? Given that the multiplier from listing 1 has 3 outputs, doesn't that mean that a program consisting of n chained multiplications still requires storing n * 2 + 1 outputs, similarly to regular AD? And doesn't binomial checkpointing allow for logarithmic memory usage in exchange for a logarithmic increase in runtime (rather than polynomial)?

Rather than answering these questions, the paper jumps eagerly into Julia code snippets, metaprogramming, and CUDA kernels, which I don't feel actually serve to elucidate the message that reversible programming is of interest to the machine learning community.

Although I feel that this version of the paper is an improvement over the version I reviewed for NeurIPS, I feel that it still fails to clearly introduce reversible programming and shed light on the subtle trade-offs between reversible programming, checkpointing, and regular AD. I encourage the authors to rewrite the paper with less of a focus on the implementation details of their framework, and a stronger focus on the memory and runtime trade-offs provided by all of these methods from a more high-level, theoretical perspective.

Pros

* Very relevant and interesting topic
* Well-written introduction
* Good code

Cons

* Fails to introduce the topic appropriately for an ML audience
* Does not clearly compare to advanced checkpointing methods
* Not well written; too many details about the software implementation that do not contribute to an understanding of the high-level technique
review_title: Interesting contribution but not clearly presented

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Changes after rebuttal
Thanks to the authors for their answers to the questions and their revisions to improve the manuscript. It is useful to have further descriptions of reversible computing for an audience that may be unfamiliar with the topic. I would encourage the authors to make further revisions to more concisely show the scientific value of the work while leaving some of the details to tutorials or other documents.  Other venues more focused on scientific computing, programming languages, or Julia may also be more suitable. If the language also attracts more users and applications built on top of it, then the case for publication will also be stronger (consider that the PyTorch paper was presented at NeurIPS 2019 even though the first release was in 2016).

---
# Summary
The paper presents an embedded domain-specific language in Julia which enables reversible computation and automatic differentiation. In order to compute gradients, we need access to all the intermediate results in a program; typically, it is necessary to store (checkpoint) these intermediate results separately or recompute them from the inputs. With reversible computing, these are unnecessary as we can compute backwards from the output to reach any intermediate result. The paper shows some empirical performance benchmarks on a bundle adjustment program.

# Strengths
- The proposed system shows strong performance compared to the presented alternatives, especially when using GPU kernels.
- There are many examples presented which shows that the language is terse.
- The system is practicality and usable for existing scientific computing applications.

# Weaknesses
- As a significant portion of the paper is dedicated to examples of the language in use, there is not much room to discuss other aspects of the system.
- The novel aspects of the work are not made very clear in the paper. It is hard to compare the benefits and drawbacks of the work compared to the alternatives, other than in the runtime performance.
- ICLR is about machine learning, but there is no evaluation of machine learning workloads. For example, I think the ICLR audience would be quite interested in how such a system might enable very deep neural networks. 

# Comments
- For people not very familiar with reversible computing (like me), it is quite useful to have detailed examples and explanations about the fundamentals. However, it seemed to me that too much of the main body is spent on the fundamentals and the tutorial aspects, and more of it could be moved into an appendix. 
- It would be better to have more comparisons and discussions of which aspects of the system are novel. There is a discussion of some related work on page 2, but it is not very systematic.
- As the paper states at the end of page 2, it is not possible to rigorously reverse operations on floating point numbers. However, there is no further discussion of the implications of this. It would be good to have some further reassurance that the errors are not important, or to have a discussion of what applications are acceptable and sufficiently tolerant of the errors.
- Given the lack of checkpointing required for automatic differentiation, it would seem to me that the language can enable significantly lower memory usage than non-reversible languages. So I was surprised that there are no benchmarks discussing memory usage, or showing that the method can operate with larger datasets/parameters on a fixed memory budget.
- Anonymous submissions should not include acknowledgements and should not include links to non-anonymous GitHub repositories.
- Please use \citet and \citep with natbib so that citations are formatted properly in the text. Citations at the end of sentences, or otherwise not used as a noun in the sentence, should look like (Author, 2020) rather than Author (2020).

review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper adapts reversible computing techniques to compute gradients. The techniques presented are not new though the Julia based DSL is new. The results presented are for differentiating through a GMM. It is not clear if the technique scale to a modern day neural network models and how they will integrate into current frameworks like JAX, PyTorch or TensorFlow.

The paper may be more relevant to a Programming Language focused venue or maybe even JuliaCon
review_title: This paper presents a Julia based DSL to do automatic differentiation using reversible computing

final_score: rating: 6: Marginally above acceptance threshold
review_content: General comments:
- This paper presents a new approach to automatic differentiation (AD), namely the use of reversible programming to achieve memory-efficient function inverse and adjoint. The authors have done a good job reviewing the background and laying out the motivation for the new apporach. The implementation is based on adding an embedded DSL to Julia called NiLang. Through reversible programming, NiLang gets rid of the need for checkpointing and hence is amenable to CUDA execution. NiLang is benchmarked against native Julia, ForwardDiff and Tapenade. The performance of NiLang is slightly worse than other approachs in the GMM benchmark. But in the bundle adjustment benchmark, NiLang outperforms ForwardDiff and Tapenade, especially with CUDA acceleration.	
 
Detailed comments:
- Section 2.3
  - I feel the explanation of the pre and post conditions can be made clearer. While most readers are familiar with the idea of pre condition in if and while control flows, some may not be familiar with what post condition is. The example in Listing 4 isn't presented in a way that's most clear and helpful to resolve this confusion either. Please explain post condition more clearly, perhaps be adding annotations to Listing 4.
  - Also, in Figure 1(a), what does "pre == post" mean? On a related note, in Listing 4, if the post condition of the if statement is a placeholder "~", how does it work when it's treated as the pre condition during reversal? Explain that it's simply treated as an always-true condition.
  - Please add a sentences to explain how NiLang handles the errors (as shown in red in Figure 1) that occur during the control flow and their reversals.
- Section 3.1
  - Please explain what adjoint is briefly before mentioning it first time, possibly by citing the Tapenade paper (Hascoet & Pascual 2013).
- Section 4.1
  - Since the authors emphasize the memory usage advantage of NiLang, why not run memory benchmarking, quantify the results and show them in tables or figures here?
  - Table 1: Why are NiLang GPU results not included in this table?
  - Tables 1 and 2: The timing numbers in these tables to not span many orders of manitude. So using regular decimal points (e.g., 0.009844, 0.0351) might be more visually clear and easier to parse than the engineering notation currently used. Also, consider making plots instead of tables for these numbers, because plots will be much more intuitive and facilitate comparisons between the different implementations. The plots can be in logarithmic scale.
- This manuscript currently lacks a discussion section. Given AD is widely used in machine learning and neural networks (especially within the context of this conference), many readers will be interested in whether NiLang is suitable for training neural networks.

More detailed comments / writing suggestions:
- p 2. "Besides the above “nice” features, it also has some “bad” features to meet the practical needs" The meaning of the second part of this sentence is unclear. Please rewrite it.
- p 8, Section 4.1. Do not repeatedly define the ancronym "BA". It's already defined above in the same page. Also, since you don't really use the acronym anyway, there's no need to define it.
review_title: A promising manuscript on a new approach that combines reversible programming and automatic differentiation; Call for improvements

Meta-review: After reading the paper, reviews and authors’ feedback. The meta-reviewer agrees with the reviewers that the paper touches an interesting topic (reversible computing) but could be improved in the area of presentation and evaluation. Therefore this paper is rejected.

Thank you for submitting the paper to ICLR. 

Decision: Reject