Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper proposes a combined approach based on both imperative and symbolic co-execution  of a program that can handle any imperative Deep Learning program while achieving the performance of symbolic graph execution. They evaluated on real-world models for imagenet and demonstrate their improvement.  
main_review: Originality: 
- In the context of DL program context, combinations of automatically building the symbolic graph from program traces seems new and interesting. I actually learned a lot from the paper.

Quality: 
- The lazy construction of the graph makes sense in this context.
- The evaluation is solid on 10 large models and the gain is significant. 
- The only problem is there is no formal argument of the correctness of the approach. However, it intuitively makes sense. 
- Authors also should consider providing ablation studies showing how much time is required to build the graph.

Clarity:  Well written paper.

Significance: This kind of optimization has the potential to significantly improve the DL models. 


limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper describes a framework based on imperative symbolic co-execution to execute deep learning programs. The main idea is to build a symbolic trace of DL operators while executing imperatively until the current iteration's trace is fully captured in its trace graph. At this point the imperative code and the symbolic code is run asynchronously with necessary communication inserted to guarantee correctness. The results show improved performance compared to AutoGraph (a one time trace only system).
main_review: # Strengths

* The system uses the concept of trace graph to gradually build a symbolic graph of multiple traces. These ideas are exploited in the JIT compilation and dynamic binary instrumentation field (Code Caches) and it is commendable it is being utilized in the context of a DL framework.
* Comparison against other tracing imperative systems such as Autograph

# Weaknesses

* The authors completely overlook statically compiled DL backends such as XLA or TVM. These build symbolic data flow graphs and statically compile them to machine code and exploits better optimizations compared to TensorRT and other runtimes. The authors do not compare the performance of their system against such compiled approaches, which is a significant omission. In Tensorflow, you can easily invoke the XLA compiler by using tf.function regions.
* Related work missing on compiled approaches such as XLA or TVM. Please add them. XLA is only mentioned as a citation for operator fusion, but in fact is a fully featured compiler that does many other optimizations.
* Evaluation mentions it outperforms AutoGraph by upto 1.61x. A breakdown of results per benchmark would give a wholistic picture to appreciate this result in full. It's surprising why a graph is not included for this result.
limitations_and_societal_impact: I have listed the technical limitations in the above section. I do not see any major negative societal impact arising from this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method for obtaining a symbolic (graph) representation of a model, given the imperative Python code. It distinguishes itself from (most) other methods by co-executing the symbolic as well as the imperative representation, as well as merging multiple traces into a single symbolic representation. This allows it to obtain arbitrary representations of 


main_review: Overall, I think this paper tackles a very relevant problem, and some parts of the approach are novel (to my knowledge) and quite interesting. Before I talk about this paper's strengths, I want to note some very related work that this paper should definitely have compared to. Specifically, that it is the LazyTensor approach (https://arxiv.org/abs/2102.13267), which does something similar, in that it builds up a symbolic graph while executing, and will recompile if that symbolic graph doesn't match. In this way, LazyTensor is also able to guarantee that it executes all code correctly.

Back to this paper, I am fairly impressed with the built system as well as the approach. AFAIK, I haven't seen this kind of "trace out program and merge traces into a single graph" approach before. Their results show that their approach is successful on a wide variety of models, including some models with a significant amount of dynamism. 

My primary concern with this approach is quite complicated, and the co-execution of the symbolic and the imperative models seems to me to impair debuggability and UX. This isn't a fatal flaw by any means - if it can execute user code reliably enough then this isn't a problem - but is something to keep in mind with these kinds of approaches.

Another question I had was how the symbolic graph was actually being executed. In TF/PyTorch, one of the primary reasons to create a symbolic graph is to pass it to a compiler like XLA. However, it seems to me like this co-execution structure/python fallbacks in order to resolve control flow may 1. not allow you to use such compilers, 2. inhibit possible optimizations. Could you clarify whether this understanding is correct and whether the baseline AutoGraph was then lowered to XLA?

Overall, I find the approach interesting + think it tackles an important problem. However, I think lack of proper comparison to relevant work are somewhat problematic.


EDIT: Primarily based off of the new results presented with XLA, as well as the preliminary LazyTensor comparisons, I have raised my score from 5 to 7.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes an approach for achieving optimized performance of symbolic graph execution for any imperative DL program. The approach is implemented in a system called Terra. The proposed approach involves decoupling symbolic operations and building a symbolic graph. The skeleton python program is executed using the Python interpreter whereas the symbolic operations are delegated to the optimized symbolic execution. Terra is evaluated on ten imperative DL programs and the performance is compared with AutoGraph. Terra can successfully execute all the ten programs whereas AutoGraph fails to execute five of them. The speed up achieved by Terra is comparable to AutoGraph. The main appeal of the method is that it does not have any Python feature coverage limitations. 

main_review: ### Originality: 
The work tries to combine the "imperative execution" and "symbolic execution" paradigms to get the best of both worlds. Instead of trying to convert the whole program to a symbolic graph, it just decouples the symbolic operations into a symbolic graph. The control flow is governed by the skeleton Python program executed in an imperative way whereas the symbolic graph is executed by a Symbolic Runner. Although the idea of speculative generation of symbolic graphs has been explored in earlier works, the idea of co-execution is novel. Co-execution is crucial for ensuring support for all the host language features.

### Quality: 

The paper is technically sound.

* The work has done a good job of carefully designing the constructs for communicating data and managing control flow across the two runners. 
* The proof that the graph generation algorithm can handle any DAG is described in the supplementary material.
* To support the claim that Terra performs comparably with Autograph on the programs on which AutoGraph works, it would be preferable to see the numbers on the benchmarks from the AutoGraph paper. The rationale for selecting the 10 programs is not discussed. It would be nice if the authors could discuss different dimensions they tried to cover( architectures, type of input data, etc.)
* The paper compares Terra with a single competing method(AutoGraph). I would have liked to see comparisons with other methods from the "single path tracing" and "static compilation paradigms".


### Clarity
The paper is well written and easier to follow along. 

Some minor issues.
* I do not understand when "GraphGenerator" stops collecting the traces (line 190). A little elaboration would be helpful.
* When Terra cancels the execution of the GraphRunner and falls back to the tracing phase (line 212), is the computation done so far lost?
* In section 4.2 (TraceGraph), in almost all the sentences, the word "GraphGenerator" is repeated. "GraphGenerator attempts ...", "GraphGenerator compares...", etc. Since the whole section is about GraphGenerator, I found the repeated usage a bit distracting. It is a style issue; authors may want to improve it.
* Spelling mistake. Line 307. "publicy" and "opended"
* Line 190, "lastest"

### Significance

The paper is trying to address a very important problem. The main appeal of the "define-by-run" imperative programs is the flexibility. The prevalent approaches which try to optimize the performance of imperative programs end up putting restrictions on the program features that can be used. Terra's approach of co-execution allows it to support all the host language features without sacrificing the performance. This paper will inspire more work in this direction. 

Although this is not discussed in the paper, I believe that since the control flow is governed by the python interpreter, this approach will have better debuggability than the approaches that try to convert the whole program to a symbolic graph.

limitations_and_societal_impact: I believe that more discussion is needed on limitations or possible pain points of the proposed approach. For example, one can ask:
- Does the method put any limitations on the kind of graphs that can be generated? Does it miss out on certain optimization opportunities due to the nature of the graph generation.
- Are there use cases where AutoGraph will perform significantly better than Terra?
- For the programs where AutoGraph works, does it make sense to use Terra considering that the performance on Terra would be slightly worse than AutoGraph?
- Does the way the program is implemented have any impact on the performance of Terra? (Possibly due to excessive switching between the two running modes.)

The answers to these questions will either bolster the claims or warn the readers about possible limitations.

I do not believe that there is any potential negative societal impact of the work.
review_title: null

Meta-review: The reviewers appreciated the novel techniques presented in the paper for co-executing the symbolic and imperative representations of a model program. During discussions, one major concern that came up was regarding experimental evaluation and comparisons to other related approaches. The additional experiments of applying XLA to Terra, and comparison with LazyTensor emulation helped clarify some of the advantages of Terra's approach and the corresponding benefits. It would be great if authors can incorporate the detailed feedback from reviews, the additional experiments and results, as well as discussions about limitations and clarifications from the author response in the final version of the paper. 
Decision: Accept (Poster)