Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tries to address the problem of locating runtime error with neural interpreter model. The contribution of this paper includes: (1) a new dataset for the problem, which consists of competitive programs in multiple languages, input data files, input descriptions, runtime error labels. (2) an improved model based on IPA-GNN.
strength_and_weaknesses: Strengths:
1.	This paper an interesting problem of locating runtime error with neural interpreter models. The proposed dataset may be useful for the future study in the area.
2.	The proposed approach can be used for locating the bug, even though the model is trained with only on the labels of error presence and error class.
3.	It is shown that the data description is helpful for predicting RE types and locating RE bugs.
Weaknesses:
The proposed extended IPA-GNN has two key modifications: (1) it takes data descriptions as input. (2) it can model the exception handling. My concerns are: (1) The data descriptions are only available for competitive programs, and may not available for other programs. This limits the usage scenario of the approach. (2) I am worrying that most competitive programs tend to not handle exceptions. How many programs in the proposed dataset have exception handling blocks? If there is not many, the exception handling part of the approach could be insignificant. Also, there is no empirical validation of the exception handling in the experiments.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. The paper has moderate novelty due to the reasons mentioned above.
summary_of_the_review: The paper proposes a new dataset as well as a new approach for the problem of static prediction of runtime errors. I appreciate the work of publishing the dataset, but the technical contribution of the proposed approach seems to be thin. The impact of the modeling of exception handling is not fully discussed or evaluated.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the problem of identifying runtime errors in a program in a static setting. Their proposed model is a modification of an Instruction Pointer Attention GNN (IPA-GNN) proposed in Bieber et al. (2020) [1], which learns to execute the program one instruction at a time in a "continuous" manner using embeddings and probabilistic transitions. They train this model on the problem of classifying a program into either a) one of many defined error classes, or b) no error. Then they show that they are able to use the inner states of the model to localize runtime errors occurring in a specific statement, with impressive accuracy.

Since the IPA-GNN [1] model forms the basis for this work, I provide a quick summary of my understanding of this paper.

------------------
**<IPA-GNN>**

Each statement in a (simple) program is represented with a 4-tuple that constitutes the statement's initial embedding. The execution is simulated in a series of time steps (like an RNN). The model maintains a probabilistic instruction pointer, where $p_{t, n}$ represents the "weight" given to statement $n$ at time step $t$. Similarly, there is a hidden state embedding $h_{t, n}$ for each time step and statement. For a time step $t$ and statement $n$, the "state proposal" is computed as
$$ a_{t, n} = \text{RNN} \left( h_{t, n}, \text{Embed}(x_n) \right) $$

Then :

* To compute the next time step, the model predicts a branch probability using a Dense layer on the state proposal. This is unclear in the paper, but I assume there can only be at most 2 possible next states. I also assume that the Dense layer weights are shared among all such branch predictions.

* The next hidden states $h_{t+1, n}$ and instruction pointer probabilities $p_{t+1, n}$ are computed using a weighted sum over all branch paths that lead *into* $n$, weighted by both the instruction pointer probabilities and the branch probabilities.

**End of </IPA-GNN>**

------------------

The authors of this paper build on the IPA-GNN in the following ways :

1. They perform their analyses on a large annotated dataset of real Python programs, possibly with runtime errors.

1. Instead of a 4 tuple to represent a statement, they run a Transformer over the token embeddings and pool to get an embedding for each statement.

1. Since these programs are long, the computational graph could be very large. So they apply a "rematerialization" trick at each layer to make the model memory efficient (I did not make an effort to understand the details of this).

1. They take a natural language description of the stdin input format and construct an embedding from it using the same Transformer from earlier. This is then combined ("modulated") with the statement embedding at each statement and each time step.

1. Before predicting a branch, they use a separate Dense layer to predict whether there is an exception at each statement. All thrown exceptions lead to a special $n_{\text{error}}$ node.
------------------

The authors then evaluate their approach on a) classifying a program into one of various error types (or no error), and b) localizing the statement corresponding to a runtime exception. They show significant gains over their baselines on these tasks. Further, they perform an interpretability study on one handpicked example program.

[1] David Bieber, Charles Sutton, Hugo Larochelle, and Daniel Tarlow. Learning to execute program with instruction pointer attention graph neural networks. In Advances in Neural Information Processing Systems, 2020.
strength_and_weaknesses: **Strengths :**

1. The extensions from the previous IPA-GNN paper are novel and non-trivial.

1. The evaluation is thorough and the results are convincing. The results on error localization in particular are very impressive considering that the model was not trained explicitly for that task.

1. Extensive details have been provided in the appendix about the experimental setup. Further, I appreciate the principled experiment design with carefully constructed metrics (weighted F1, weighted error F1, etc), and model selection performed on validation data as it should be.

**Weaknesses :**

1. I'm unsure if pylint is an appropriate choice for a static analysis baseline. Wouldn't something based on symbolic execution be closer to the spirit of this paper? After all, this model attempts to simulate the execution of the program, which is closer to symbolic execution than static analysis, in my opinion. Further, it is only natural that static analysis cannot catch runtime errors effectively, whereas symbolic execution approaches might be able to find values that throw exceptions. 

1. How does this scale with the size of the program? The programs in CodeNet are still relatively small programs compared to kernel code etc which could be hundreds of thousands of lines. What is the execution time/memory for the largest programs in CodeNet? And how does the graph of computation cost with size of program look?

**Suggestions/Clarifications :**
(I feel like it wouldn't be accurate to call these points "Weaknesses")

*Point 1*

Regarding Table 4, here is my attempt to understand what's going on : `|S| = 3`, so if `q` is some number `>3`, we could have an EOF error when it tries to read from stdin more than 3 times. The model sees the constraint `|S| = 3` as part of the stdin description, and that is why it is able to predict this accurately.

Could something like the above ^ be put in the paper, either in the table caption or in the text? I know that the end of Section 5.2 has a brief note to this effect, but it would really be nice if the part about `|S| = 3` could be emphasized as that is the key bit of info that helps the model make the decision (at least as per my understanding).

*Point 2*

Regarding the example in Table 4, I feel it would be interesting to evaluate the model with the same resource description, but with `|S| = 3` removed. It would show that *that* is the piece of information that the model is using to make a better prediction.

*Point 3*

When you use a Transformer to generate embeddings of each token (and to embed the resource description) as the first step of E IPA-GNN, do you use a pre-trained Transformer model? What architecture? Do you freeze the weights or do you learn it along with the rest of the model? It would be useful if this information was provided.

*Point 4*

I find the flow of RQ1 and RQ2 a little confusing in the context of Table 2a. When we first read RQ1, it's not apparent that the method of incorporating resource descriptions is as a docstring. Then later in RQ2 that is specified, so we have to jump back to Table 2a to consult it again. I feel like there should be a note in RQ1 specifying that docstrings are used, and other methods like FiLM and Cross-attention will be discussed in the next RQ.

*Point 5*

Does IPA-GNN (not E.IPA-GNN) also have an $n_{\text{error}}$ state? 

If not, how do you use IPA-GNN to predict errors? For E. IPA-GNN the method is clear - use a dense layer on the $n_{\text{error}}$ state in the last timestep. For IPA-GNN, what state do you use?

Similarly for localization. How do you localize a bug using IPA-GNN if there is no $n_{\text{error}}$ state?
clarity,_quality,_novelty_and_reproducibility: **Clarity :**

This paper attempts to convey a very sophisticated method with several "moving parts". Considering that, it is written extremely clearly and concisely. But I still have some questions about certain aspects of the algorithm. I've asked questions to clarify these under the headline of "Suggestions/Clarifications" above.

**Quality and Novelty :**

The experiments are thorough (with several auxiliary details in the Appendix), the design choices are well-motivated, the idea is novel, and the results are very impressive, especially on "unsupervised" error localization.

**Reproducibility :**

The authors have mentioned that their models and dataset will be made available after the review process, so it is not possible to check their reproducibility claims at this time. But the description of experiments in the paper are thorough enough that it should be possible to re-implement their approach with some (possibly significant) engineering effort.
summary_of_the_review: It is my opinion that this paper presents a novel idea convincingly and elaborately. I have a few clarifying questions about the details of their algorithm/experiments, along with a minor concern about the choice of one of their baselines. But on the whole, I feel that the paper, even in its current form, is an excellent contribution to the program analysis literature and therefore I wholeheartedly recommend acceptance.

I give myself a confidence score of 3/5 **not** because I don't understand the paper well-enough, but because I am not sufficiently familiar with the current research in this area to know if there are other potential competing approaches to solve this problem.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors propose an approach to predict runtime errors of programs by using IPA-GNN models with improvements to handle exceptions and provide error locations. They also add support for passing resource descriptions to the model. Authors provide a dataset for testing their approach. They show that their approach outperforms baselines. 
strength_and_weaknesses: Strengths:

- Improvement of IPA-GNN model to handle exceptions and real programs
- Adding handling of resource descriptions to IPA-GNN
- Unsupervised localization of errors
- Ablation analysis
- Improved results compared to baselines and ablations

Weaknesses:

- Authors compare results to baselines, but do not try to identify state of the art results for their testset and compare against state of the art methods

clarity,_quality,_novelty_and_reproducibility: The paper is well written.
I think that the IPA-GNN extensions and improvements are novel.
The paper results should be reproducible using the dataset authors plan to release and the descriptions of the code/methods in the paper. The reproducibility might improve if the code was also released by authors.

-------------------------------------------------------------------------------------------------------
ICLR review form does not provide a section for comments/questions about the paper. I am going to post them here:
- Authors should clarify that the "external resource descriptions" could be comments, problem statement, possible inputs, expected outputs and so on. They also perhaps should analyze and comment on which of these descriptions work better or at all. 
summary_of_the_review: I think that authors present a novel improvement of IPA-GNN to handle exceptions, real programs and external descriptions. Their approach improves prediction of runtime errors vs baselines and I think this may warrant accept to ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper is able to predict runtime errors in a static setting using a form of graph neural network on program points connected by control flow links.  While this has been done before, the paper advances the state of the aret by showing that this can be scaled to larger programs. The paper also shows that this architecture can learn to localize errors without being trained on localization.  Perhaps this is reminiscent of the observation that training a graph network on SAT problems can determine truth assignments when trained only on a satisfiable/unsatisfiable labeling of SAT problems.

A weakness of the paper is that the experiments use home-grown baselines rather than scores from previously published papers on a shared dataset.
justification_for_why_not_higher_score: Two reviewers felt the results acceptable but not exciting.  The weaknesses they point out do not seem serious to me.  However, the paper does not seem to rise to the level of a spotlight.
justification_for_why_not_lower_score: The innovation in the architecture of control-flow GNNs, and the observation that error localization emerges without training, seems sufficiently significant to warrant publication.
Decision: Accept: poster