Static Prediction of Runtime Errors by Learning to Execute Programs with External Resource Descriptions
Authors: David Bieber
Rishab Goel
Dan Zheng
Hugo Larochelle
Daniel Tarlow
Venue: ICLR 2023 poster (2023)
Keywords: program analysis
graph neural networks
recurrent networks
attention mechanisms
source code
program execution

Abstract: The execution behavior of a program often depends on external resources, such as program inputs or file contents, and so the program cannot be run in isolation. Nevertheless, software developers benefit from fast iteration loops where automated tools identify errors as early as possible, even before programs can be compiled and run. This presents an interesting machine learning challenge: can we predict runtime errors in a "static" setting, where program execution is not possible? Here, we introduce a competitive programming dataset and task for predicting runtime errors, which we show is difficult for generic models like Transformers. We approach this task by developing an interpreter-inspired architecture with an inductive bias towards mimicking program executions, which models exception handling and "learns to execute" descriptions of external resources. Surprisingly, we show that the model can also predict the locations of errors, despite being trained only on labels indicating error presence or absence and kind. In total, we present a practical and difficult-yet-approachable challenge problem related to learning program execution behavior and we demonstrate promising new capabilities of interpreter-inspired machine learning models for code.

Published as a conference paper at ICLR 2023
STATIC PREDICTION OF RUNTIME ERRORS BY LEARNING TO EXECUTE PROGRAMS WITH EXTERNAL RESOURCE DESCRIPTIONS
David Bieber Google Research
Rishab Goel Mila
Daniel Zheng Google Research
Daniel Tarlow Google Research
Hugo Larochelle Google Research
ABSTRACT
The execution behavior of a program often depends on external resources, such program inputs file contents, and so the program cannot be run in isola- tion. Nevertheless, software developers benefit from fast iteration loops where automated tools identify errors as early as possible, even before programs compiled and run. This presents an interesting machine learning challenge: can we predict runtime errors in "static" setting, where program execution is not possible? Here, we introduce a competitive programming dataset and task for pre- dicting runtime errors, which we show is difficult for generic models like Trans- formers. We approach this task by developing an interpreter-inspired architecture with an inductive bias towards mimicking program executions, which models ex- ception handling and "learns to execute" descriptions of external resources. Sur- prisingly, we show that the model can also predict the locations of errors, despite being trained only on labels indicating error presence or absence and kind. In total, we present a practical and sffficul-yct--pprrochhbbll challenge problem re- lated to learning program execution behavior and we demonstrate promising new capabilities of saerpreter-inspires machine learning models for code.
be
INTRODUCTION
We investigate applying neural machine learning methods to the static analysis of source code for early prediction of runtime errors. The execution behavior of a program is in general not fully defined by its source code in isolation, because programs often rely on external resources like inputs, the contents of files, or the network. Nevertheless, software developers benefit from fast iteration loops where automated tools identify errors early, even when program execution is not yet an option. Therefore we consider the following machine learning challenge: can we predict runtime errors in a "static" setting, where program execution is not possible?
This runtime eтrоr prediction task is well suited as challenge problem because it is difficult-yet- approachable, has real-world value for software developers, requires novel modeling considerations that we hypothesize will be applicable to range of learning for code tasks, and with this work, now has a suitable large dataset of complex autan-- code with error labels. The task is to predict whether program will exhibit runtime error when it is run, and if so to determine the error; even when static analysis cannot provide guarantees an error in the code, patterns learned from data may point likely errors. Our dataset consists of 2.4 million Python 3 programs from Project CodeNet (Puri et al.) 2021) written by competitive programmers. We have run all programs in sandboxed environment on sample inputs determine their error classes, finding the programs exhibit 26 distinct error classes including "no error". Each program relies on an external resource, the stdin input stream, and we pair each program with natural language description of the behavior of the stream. We make the task and dataset, along with all models considered in this work, available for the research community to facilitate reproduction of this work and further research
To make progress on this challenging task, we identify promising class of models from prior work, interpreter-insse inspired models, and we demonstrate they perform well on the task. Instruction Pointer

---

Published as a conference paper at ICLR 2023
Attention Graph Neural Network (IPA-GNN) (Bieber et al. 2020 models simulate the execution of a program, following its control flow structure, but operating in a continuous embedding space. We make a number of improvements to IPA-GNN: scaling up to handle complex programs requir- ing thousands of execution steps, adding the ability to "learn to execute" descriptions of external resources, and extending the architecture to model exception handling and recover error locations. We evaluate these anterreter-inspiiee ral baselines, and against pylint as a static analysis baseline. Our combined improvements lead to increased accuracy in predicting runtime errors and to interpretability allowing for prediction of er- ror locations even though the models are only trained on error presence and error class, not error location. In total, we summarize our contributions as:
architectures against Transformer, LSTM, and GGNN neu-
We introduce the runtime error prediction task and large accompanying dataset, providing run- time error annotations for millions of competition Python programs.
We demonstrate that IPA-GNN architectures are practical for the complexity of real programs by scaling them to handle competition programs, and there we find they outperform generic models.
We demonstrate that external resource descriptions, such as Japanese or English descriptions of stdin, can be leveraged improve performance on the task across all model architectures.
We extend the IPA-GNN model exception handling, resulting in the Exception IPA-GNN, which we find can localize errors even when only trained on error presence and kind, not error location.
2 RELATED WORK
Program analysis
Program analysis is rich family of techniques for detecting defects in pro- grams, including static analyses which are performed without executing code (Livshits and Lam, 2005] Xie and Aiken 2006
Ayewah et al. 2008 and dynamic analyses which are performed at runtime (Cadar et al. 2008 Sen et al. 2005 Godefroid et al.) 2005). Linters and type checkers are popular error detection tools that use static analysis. Static analysis (e.g. symbolic execution) does not typically use concrete imputs, while dynamic analysis requires concrete inputs and pro- gram execution. Compared with traditional static analysis, our approach is more flexible in its input representation, using general "resource description" abstraction, which can represent the entire spectrum from concrete inputs to input constraints to missing inputs.
Several neural architectures draw inspiration from program interpreters (Graves et al.] 20144) Lukasz Kaiser and Sutskever 2016; Reed and de Freitas 2016 Graves et al.) 2016 BBsnjak et al.
Execution-aware models
2017 Gaunt et al. [017 Dehghani et al. 2019 Bieber et al.) 2020). Our work is most similar to Bieber et al. (2020) and Bošnjak et al. (001 7), focusing on how interpreters handle control flow and exception handling, rather than memory allocation and function call stacks. Other works use program execution data directly, training with textual representations of execution traces as inputs (Nye et al.,
2021aa Pei et al., 202T Nye al., 2021b) or performing execution
during synthesis (Chen et al. 2019 Li et al. 2022) Shrivastava et al. (002]). Compared with these, our approach uses weaker supervision, using only runtime error labels for training.
Fault detection and localization datasets There has been considerable recent interest in applying machine learning to identifying and localizing faults in source code (Allamanis et al., 2018a). Puri et al. 2021) makes large dataset of real world programs available, which we build on in construct- ing our runtime errors dataset. Our dataset (i) is large (it has millions of examples), (ii) exhibits many programming language features, (iii) is written by human authors, and (iv) has error labels from the execution behavior programs. Previous code datasets only exhibit a subset of these properties: large real-world and competition code datasets (Hendrycks et al.] |021 Li et al.] 2022 Kanade et al., 2020 RRayhevv et al., 2016; Husain et al.) 2019 et al. al.] 2022 exhibit properties i,
ii and iii, but not iv, while learning
execute datasets (Zaremba and Sutskever 2014 [ieber al.
2020) exhibit property iv but not i, ii, or iii. Recent program synthesis datasets (Chen et al.) 2021
Austin et al., 2)21) exhibit ii and iii only. Other datasets obtain error labels by injecting synthetic
errors (Allamanis et al.) 20188; Karampatsis and Sutton, 2020 Pradel and Sen 2018) (lacking the
realism of fii or from commit messages (Just et al. 2014 Dinella et al.) 2020 (lacking i and iv).
Fault localization approaches Fault localization approaches vary in (i) level of supervision
weak (error labels) (Li al. al. 2019) vs strong (explicit location labels) (Lou et al. 2021 Zhhng al.
2

---

Published as a conference paper at ICLR 2023
TARGET CLASS
TRAIN #
VALID
TEST
TARGET CLASS
TRAIN #
VALID #
TEST #
205343/
Suisy-AxisEror
2
3
error
1881303
207162
13289!
OSEmer
2
OverflowEroo
AssertionFroo
47
releroor
5
0
D
AttribureError
10026
509
1674
RecursionFror
BOFEmsr
7676
RuntimeError
3
FileNotFoundEruur
259
37
22
Stoplteration
3
1
ImportEroo
7645
285
841
SyntaxEEmor
3
IndentationlEnuur
10
12
TypeFrmr
21414
2641
2603
IndexError
7505
965
733
UnhoundLeeallmmo
KeyErrr
362
39
22
ValueFimor
25087
3087
2828
MemoryError
ZeroDivision
437
125
Timeout
7816
1072
691
NameEEror
21540
2427
2422
Other
8
2
Table Distribution target classes the runtime errors dataset. i denotes the balanced test split.
2019 Zhou al. 2019 Allamanis al. 2021 and (ii) localization granularity statement- (Lou et al. 2021 Zhang et al. 2019 Allamanis et al. 2021) vs method-level (Li et al., [2019) Zhou
al.] 2019). Our approach weak supervision in the form of runtime error labels to indirectly
learn fault localization at statement-level.
RUNTIME ERROR PREDICTION
Task The goal of the runtime error prediction task is to determine statically whether a program is liable to encounter a runtime error when run, and if so, what error kind. The programs cannot be executed directly, they lack unit tests and depend external resources which are not available. A textual description of the external resources, which may be the program's inputs, file's contents, network access, is provided. This makes reasoning about the execution behavior of the programs plausible, even though actually performing the execution is not. We treat this task as one-class classification, with each error type as its own class and with "no error" an additional class.
Dataset We construct the runtime errors dataset using Python submissions to competitive pro- gramming problems from Project CodeNet (Puri al., 202T). Beginning with the 3.28 million Python submissions in Project CodeNet, we filter the submissions to keep only those written in Python 3, which are syntactically valid, which do not make calls to user-defined functions, and which do not exceed threshold length of 512 tokens once tokenized. By running each submission a sandboxed environment, we identify its ground truth runtime error class. Each submission is associated with competitive programming problem whose problem statement a description in either English or Japanese of the inputs the program is liable to receive at runtime. This process results in dataset of 2.44 million submissions, each paired with of 26 target classes. The "no error" target is most common, accounting for 93.4% of examples. For examples with one of the other 25 error classes, additionally note the line number which the error occurs.
parse to obtain
We divide the problems into train, validation, and test splits at a ratio of 80: 10:10 10. All submissions to the same problem become part of the same split. This reduces similarities between examples across splits that otherwise could arise from the presence of multiple similar submissions for the same problem. Since there a strong class imbalance favor of the no error class, we also produce balanced version of the test split by sampling the no error examples such that they comprise roughly 50% of the test split. We use this balanced test split for evaluations. We report the number of examples having each target class in each split in Table [ We describe the full dataset generation and filtering process in greater detail in Appendix A and we evaluate the limitations the dataset quantitatively in Appendix
While there are many datasets in the literature that test understanding of different aspects of code including bug-finding, real-world implications and presents a practical opportunity for improvement using ML-based ap- proaches, and it tests combination of statistical reasoning and reasoning about program execution.
believe
fills gap: it is large-scale (millions of examples), it has
3

---

Published as a conference paper at ICLR 2023
Algorithm Interpreter for which the Exception IPA-GNN is a continuous relaxation
Input: Program
h 2:p f-0
Initialize the interpreter.
2: while p $ {Mexit Nemor do
3:
<- Evaluate(<p: h)
b Evaluate the current statement.
4: if' Raises(Zp. h) then
5: p G--GRRiisNNod((zp.. h)
Raise exception.
6: else
7: if Branches(Ipp, h) then
8:
GetBranchNode((pp
Follow branch.
9:
else
p p+
Proceed to next statement.
APPROACH: IPA-GNNS AS RELAXATIONS OF INTERPRETERS
We make three modifications the Instruction Pointer Attention Graph Neural Network (IPA-GNN) architecture. These modifications scale the IPA-GNN to complex code, allow it to incorporate ex- ternal resource descriptions into its learned executions, and add support for modeling exception handling. The IPA-GNN architecture is a continuous relaxation of the standard interpreter (!) de- fined by the pscudocode in Algorithm | minus the magenta text. We frame these modifications in relation specific lines of the algorithm: scaling the IPA-GNN to complex numan-authored code (Section[4.I) and incorporating external resource descriptions (Section 4.2) both pertain interpret- ing and executing statement Tp at Line 3, and modeling exception handling adds the magenta text at lines 4-6 to yield new interpreter (I') (Section 4.3). We showcase the behavior of both interpreters I and I' on sample program in Figure 1 and illustrate an execution of the same program by continuous relaxation of interpreter I' (I') alongside it.
4.1 EXTENDING THE IPA-GNN REAL PROGRAMS
Bieber et al. (020) interprets the IPA-GNN architecture message passing graph neural network operating on the statement-level control-flow graph of the input program æ. Each node in the graph corresponds to single statement in the program. At each step é of the architecture, each node performs three steps: it executes the statement at that node (Line Equation 2), computes a branch decision (Lines 7-8, Equation 4 and performs mean-field averaging the resulting states and
instruction pointers (Appendix C Equations 10 and
Unlike in Bieber al. (2020) where program statements simple enough be uniformly encoded as four-tuples, the programs in our runtime errors dataset consist of arbitrarily complex Python
n SOURCE
STDIN
-3
STDIN DESCRIPTION
"A SINGLE INTEGER -10..101
x = input ()
if x
(b) The resource description suggests values the program may receive on stdin.
y = 4/3
else:
t
hA,B
Pr
hp
Pp
5 Y = abs (x)
0
{}
1
10000000
z = ==y+ sqrt (x)
1
{x: -3]
2
2
01000000
<exit>
<raise>
2
{x: -3}
5
00001000
3
{x: -3, y: 3}
6
6
00000100
(a) A sample program illustra-
4 ValueError: line 7
8
0000000
tive of Algorithm behavior.
which raises ValueError
(c) Step-by-step execution of the program under interpreters I and I', and continuous relaxation Í' Distinct colors represent distinct emhedding values.
æ line
Figure 1: A sample program and its execution under discrete interpreters / and I' (Algorithm) 1 and under continuous relaxation j' of interpreter I', pı, denotes the instruction pointer under I step
4

---

Published as a conference paper at ICLR 2023
statements authored by real programmers in competition setting. The language features used are numerous and varied, and so the statement lengths vary substantially, with a mean statement length 6.7 tokens; we report the full distribution of statement lengths in Figure
The IPA-GNN architecture operates on a program x's statement-level control-flov graph, and so requires per-statement embeddings Embedd for each statement Z..- We first apply either local or global Transformer encoder to produce per-token embeddings, and we subsequently apply one of four pooling variants span of such embeddings to produce a node embedding per statement in program. In the local approach, we apply an attention mask to limit the embedding of a token in a statement to attending to other tokens in the same statement. In the global approach, no such attention mask is applied, and so every token may attend to every other token in the program. We consider four types of pooling in our hyperparameter search space: first, sum, mean, and max. The resulting embedding is given by
(I) First pooling takes the embedding of the first token in the span of node Sum, mean, and max pooling apply their respective operations to the embeddings all tokens in the span of node n.
Embeed(z,) = Pool (xannfformmrr))sp((sss))
Finally we find that the programs in our dataset require as many 174 steps of the IPA-GNN under the model's 's heuristic for step limit T(x) (Appendix E). To reduce the memory requirements, we apply rematerialization at each layer of the model (Griewank and Walther 2000 Chen et al. 2016).
4.2 EXECUTING WITH RESOURCE DESCRIPTIONS
In our dataset, each program x may be accompanied by a description of what values stdin may contain at runtime. We convert this description into embedding d(x);; the embeddings, vocabulary, and tokenizer used to produce d(x) are shared with those used to produce token embeddings from program source. Analogous
Line of Algorithm I IPA-GNN architectures initialize with per- node hidden states ho.: = O and soft instruction pointer Pa.n = 1n = 0}. Here Pt.n. represents the probability under the model that node n is executing at step f. Following initialization, each step of an IPA-GNN begins by simulating execution (Line 3) of each non-terminal statement with non-zero probability under the soft instruction pointer to propose a new hidden state contribution
RNN(h+-1,.. Modulate( (Embed(x,) h(,,)))
The text in magenta shows our modification to the IPA-GNN architecture to incorporate external resource descriptions. We consider both Feature-wise Linear Modulation (FiLM) (Perez al. (017) and cross-attentin (Lee et al. 2019) for the Modulate function, which we define in Appendix D Modulation allows the IPA-GNN to execute differently each step conditioned on the information in the resource description, whether it be type information, value ranges, or candidate values.
We also consider additional method: injecting the description as a docstring at the start of the program. This method yields a new valid Python program, and so any model can accommodate it.
4.3 MODELING EXCEPTION HANDLING
The final modification we make to the IPA-GNN architecture to model exception handling. In Algorithm [ this corresponds to adding the magenta text to form interpreter I', computing a raise decision (Lines 4-6, Equation 3). We call the architecture that results the Exception IPA-GNN.
Whereas execution always proceeds from statement to next statement in interpreter / and in the IPA- GNN, interpreter I' admits another behavior. Under I' and the Exception IPA-GNN, execution may proceed from any statement to a surrounding "except block", if it is contained in a try/except frame, or else to a special global error node, which we denote n In the sample execution in Figure T we see at step t = 4 the instruction pointer pr updates to nerror 8.
We write that the IPA-GNI makes raise decisions as
(nn) (1 b,,),(())) softmax (Dense(a(t)). (1)))
The dense layer here has two outputs representing the cases that an error is and is not raised. Here r(n) denotes the node that statement n raises to; r(n) = neme if n is not contained in a try/except frame, and o denotes the probability under the model of execution transitioning from to n'.
5

---

Published as a conference paper at ICLR 2023
Next the model makes soft branch decisions in analogous manner; the dense layer for making branch decisions has distinct weights from the layer for making raise decisions.
b ban m,, = (1 - (e)). softmax (Denso(af.))
(4)
The text in magenta corresponds to the "else" at Line 6. The model has now assigned probability up three possible outcomes for each node: the probability that raises exception by (()),(()>>
the probability that the true branch is followed b,, and the probability that the false branch is followed De, In the common case where a node is not control node and has only single successor, the probability of reaching that successor is simply - (t,,)(())
Finally, we assign each program a step limit T(x) using the same heuristic as Bieber et al. (0200) detailed in Appendix F After ((:) steps of the architecture, the model directly uses the probability mass at Rexit and Rerrr to predict whether the program raises an error, and if so predicts the error type using the hidden state at the error node. We write the modified IPA-GNN's predictions as
P(no error) Pea and P(error) PT(x), nemer with P(error k error) softmax (Dense()r(sa
(5)
(6)
We train with cross-entropy loss the class predictions, treating "no error" its class.
4.4 UNSUPERVISED LOCALIZATION OF ERRORS
Since the Exception IPA-GNN makes soft decisions as to when to raise an exception, we aggregate these soft decisions to obtain the model's ' prediction for where a program raises an error. We use this to evaluate the model's localization accuracy despite training without error locations as supervision.
For programs that lack try/except frames, we compute the localization predictions of the model by summing. separately for each node, the contributions from that node to the error node across all time steps. This gives an estimate of exception provenance
p(error at statement n.
Σ Pt.n Dt,m,
For programs with a try/except frame, we must trace the exception back to the statement that origi- nally raised it. To do this we calculate a recurrence as detailed Appendix H
EXPERIMENTS
In experiments evaluate the following research questions:
How does the adaptation of the IPA-GNN to realistic code compare against existing static analysis and against standard architectures like GGNN, LSTM, and Transformer? (Section
RQI:
RQ2: What is the impact of including resource descriptions? What methods for incorporating them work best? (Section 5.2)
RQ3: How interpretable are the soft instruction pointer values in the Exception IPA-GNN for local- izing errors? How does unsupervised localization with the Exception IPA-GNN compare altemn- tive unsupervised localization approaches based on multiple instance learning? (Section 5.3
5.1 EVALUATION OF IPA-GNN AGAINST BASELINES
We describe the experimental setup for our first experiment, comparing the IPA-GNN architec- tures with Transformer (Vaswani et al., 2017). GGNN (Li et al.) 2017)), and LSTM (Hochreiter and Schmidhuber] 1997) baselines. In all approaches, we use the 30,000 token vocabulary constructed in AppendixAA A applying Byte-Pair Encoding (BPE) tokenization (Sennrich et al.) 2016) to tokenize each program into a sequence of token indices. The Transformer operates on this sequence to- ken indices directly, with its final representation computed via mean pooling. For all other models (GGNN, LSTM, IPA-GNN, and Exception IPA-GNNN, the token indices are first combined via a masked (local) Transformer produce per-node embeddings, and the model operates node embeddings as in Section 4.1 Following Bieber et al. (0000) we encode programs for a GGNN using six edge types, and use two-layer L.STM for the L.TM baseline and in all IPA-GNN variants.
these per-

---

Published as a conference paper at ICLR 2023
(a) Accuracy, weighted FI, and weighted error Fl scores.
(b) Localization accuracy (%) for the MIL Trans- formers and Exception INN.
R.D.?
ACC.
W.
PYLINT
60.4
47.9
23.8
MODEL
R.D.?
LOCAL.
BASC-
LINES
GGNN
62.8
58.9
45.8
LOCAL MIL TRANSFORMER
33.0
TRANSTORMUR
63.6
60.4
48.1
LOCAL MIL TRANSFORMER
48.9
LSTM
66.1
61.4
GLOBAL MIL TRANSFORMER
48.2
ADLAND ABLATIONS
GGNN
68.3
66.5
56.8
GLOBAL MIL TRANSFORMER
48.8
TRANSFORMER
67.3
65.1
54.7
E. IPA-GNN
50.8
LSTM
68.1
66.8
58.3
E. IPA-GNN DOCSTRING
64.7
IPA-GNN
68.3
64.8
53.8
E. IPA-GNN FILM
64.5
IPA-GNN
68.7
64.9
53.3
E. IPA-PAANE GNN CROSS ATTENTION
68.8
OURS
IPA GNN
71.4
70.1
62.2
E. IPA-GNN
71.6
70.9
63.5
Table 2: Error classification and error localization results on the balanced test set with and without resource descriptions (R.D.).
In order to compare against the capabilities of a standard static analysis setup, we also consider a baseline based pylint. For this baseline, we map subset of the findings that pylint can identify to runtime errоr classes that they can indicate. The baseline predicts an error class pylint identifies corresponding finding. The purpose of this baseline is to consider standard tool used by Python developers and see how is performing on the task. We provide further details in Appendix G
For each neural approach, perform an independent hyperparameter search using random search. We list the hyperparameter space considered and model selection criteria in Appendix F The models each trained minimize cross-entropy loss the target class using stochastic gradient descent for up to 500,000 steps with a mini-batch size of 32. In order to more closely match the target class distribution found in the balanced test set, we sample mini-batches such that the proportion of examples with target "no error" and those with an error target is 1:1 in expectation. We evaluate the selected models on the balanced test set, and report the results in Table 2a (see rows without check marks). Weighted F1 score (W. FI) performs weighted average of the per-class FI scores by class frequency, and weighted error F1 score (F. F1) does the same while restricting consideration to those examples with runtime error.
We perform additional evaluations using the same experimental setup but distinct initializations to compute measures of variance, which we detail in Appendix F
RQI: The interpreter-inspired architectures show significant gains over the pylint, LSTM, GGNN and Transformer baseline approaches on the runtime error prediction task. We observe that the pylint baseline can make incorrect predictions because it correctly identifies an issue the code under analysis when that code does not result in runtime error in dataset; pylint's lower performance on runtime error prediction is not evidence against pylint's performance for its intended use cases. We attribute the enerpreter-inspiree architectures' relative success over other neural architectures to their inductive bias toward mimicking program execution.
5.2 INCORPORATING RESOURCE DESCRIPTIONS
We next evaluate methods of incorporating resource descriptions into the models. For each archi- tecture we apply the docstring approach of processing resource descriptions of Section 4.2 This completes matrix of ablations, allowing to distinguish the effects due to architecture change from the effect of the resource description. We follow the same experimental setup as in Section 5.] and show the results again in Table (compare rows with check marks to those without).
We also consider the FiLM and cross-attention methods of incorporating resource descriptions into the IPA-GNN. Following the same experimental setup again, show the results of this experiment in Table 3 Note that the best model overall by our model selection criteria on validation data was the IPA- GN-G with -ross-attention, though the Exception IPA-GNN performed better on test.
RQ2: Across all architectures, the results show external resource descriptions improve performance on the runtime prediction task. On the IPA-GNN architectures, further improvements

---

Published as a conference paper at ICLR 2023
BASELINE
DOCSTRING
FILM ACC.
CROSS-AFTENTION F.
MODEL
Acc.
W. F1
E. Fl
ACC.
W. F1
E. F1
W.
E. F1
Acc.
E. FI
IPA-GNN
68.3
64.8
53.8
71.4
70.1
62.2
71.6
70.3
62.9
72.0
70.3
62.6
E. IPA-GNN
68.7
64.9
53.3
71.6
70.9
63.5
70.9
68.8
59.8
73.8
72.3
64.7
Table A comparison of early and late fusion methods for incorporating external resource descrip- tion information into interpreter-inspee inspireed models.
by considering architectures that incorporate the resource description directly into the execution step of the model, but these gains are inconsistent. The pylint baseline is unable to incorporate resource descriptions. Critically, using any resource description method is better than none all.
To understand how the resource descriptions lead to better performance, we compare in Figure 2 the instruction pointer values of two Exception IPA-GNN models a single example (shown Table The model with the resource description predicts that the input () calls will read input beyond the end of the stdin stream. In contrast, the model without the resource description has less reason to suspect an eπror would be raised by those calls. The descriptions of stdin in our runtime errors dataset also frequently reveal type information, expected ranges for numeric values, and formatting details about the inputs. We visualize additional examples in Appendix K
5.3 INTERPRETABILITY AND LOCALIZATION
We next investigate the behavior of the Exception IPA-GNN model, evaluating its ability to localize runtime errors without any localization supervision. In unsupervised localization, the models predict the location of the error despite being trained only with error presence and kind supervision.
Multiple Instance Learning Baselines Unsupervised localization may be viewed as multiple stance learning (MIL) (Dietterich et al.), 1997). Consider the subtask of predicting whether partic- ular line contains an error. an n-line program, there are n instances of this subtask. The available supervision only indicates if any of these subtasks has error, but not which one. By viewing each instance as a bag of subtasks, we have cast the problem as MIL.
Using this view, introduce two variations the Transformer architecture multiple instance learning baselines. The first the "Local MIL Transformer", in which each statement in the program encoded individually, in the local node embeddings computation of Section 4.] The second is the "Global MIL Transformer", in which all tokens the program may attend to all other tokens in the Transformer encoder. In both cases, the models make per-line predictions, which aggregated to form an overall prediction as defined in Appendix
10
15
20
BASELINE
RESOURCE DESCRIPTION
Figure 2: Heatmap of instruction pointer values produced by BASELINE and DOCSTRING Exception IPA-GNNs for the example Table 4 The x-axis represents timesteps and the y-axis represents nodes, with the last two respectively representing nextt and namer- The BASELINE instruction pointer value diffuse, with most probability mass ending at Rexit- The R.D. instruction pointer value is sharp, with almost all probability mass jumping nerrrr from node 2.
8

---

Published as a conference paper at ICLR 2023
STDIN DESCRIPTION
Input: Input is given from Standard Input the fol following format Constraints: Each character of S is A or B. |S| =
n
SOURCE
BASELINE
R.D.
0
a str (input ())
16.9
0.4
1
= int (input ())
3.2
0.3
2
= [input () splitt() for i in (q))
0.5
99.3
3,4 4
for in range (q):
6.4
0.0
5
if int (s[1 [0])
and len (a) >1:
0.0
6
a = a[::-1]
0.7
0.0
7
elif int [0]] [0])=-
and int (s[i] [1])==1]
0.1
0.0
8
a=s [2]+a
0.2
0.0
else:
a=a+s [1] [2]
0.0
10
print (a)
1.1
0.0
Table 4: Per-node localization predictions from the BASELINE and DOCSTRING Exception IPA- GNN models on a sample program from the validation split. The target class EOFERROR, occur- ring on line 2 (n = 2). BASELINE predicts No ERROR with confidence 0.708, while R.D. predicts EOFERROR with confidence 0.988, localized line (n 3). The input description shows the cause for error: there are more input () calls than the number of expected inputs.
we train each of the MIL Trans- former and Exception IPA-GNN models. As before, the models trained only to minimize cross- entropy loss on predicting error kind and presence, receiving no error location supervision. We report the localization results in Table 2b Localization accuracy ("LoCAL.") measures the percent of the test examples with errors for which the model correctly predicts the error line number.
Localization Experiment Using the same protocol as Section 5.1
RQ3: The Exception IPA-GNN's unsupervised localization capabilities far exceed that of baseline approaches. In Figure we see the flow of instruction pointer mass during the execution of a sample program (Table 4 by two Exception IPA-GNN models, including the steps where the models raise probability mass to Temor- Tallying the contributions to nerror from each node yields the exception provenance values in the right half of Table This shows how the model's internal state resembles plausible program executions and allows for unsupervised localization. As beneficial side-effect of learning plausible executions, the Exception IPA-GNI can localize the exceptions predicts.
DISCUSSION
In this work, we introduce the task of predicting runtime errors in competitive programming prob- lems and advance the capabilities of interpreter-inspired models. Our models support the complexity of competition code and demonstrate that natural language descriptions of external resources can re- duce the ambiguity that arises in a static analysis setting. We show that the interpreter-inspiree models outperform standard alternatives and that their inductive biases allow for interesting inter- pretability in the context of unsupervised localization.
Though they perform best, current IPA-GNN models require taking many steps of execution, up to 174 this dataset. A future direction is to model multiple steps of program execution with single model step, to reduce the number of model steps necessary for long programs. Extending the interpreter-insiree models with additional interpreter features, supporting multi-file programs programs with multiple user-defined functions are also interesting avenues for future work.
Learning understand programs remains rich area of inquiry for machine learning research be- cause of its complexity and the many aspects of code. Learning to understand execution behavior particularly challenging as programs grow in complexity, and as they depend on more external resources whose contents are not present in the code. Our work presents challenging problem and advances interpreter-inspired models. both of which we hope are ingredients towards making progress on these difficult and important problems.

---

Published as a conference paper at ICLR 2023
REFERENCES
Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), (1)))881 2018a.
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations, 2018b.
Miltiadis Allamanis, Henry Jackson-Flux, and Mare Brockschmidt. Self-supervised bug detection and repair. Advances in Neural Information Processing Systems, 34, 2021.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.
Nathaniel Ayewah, William Pugh, David Hovemeyer, J. David Morgenthaler, and John Penix. Using static analysis to find bugs. IEEE Software, 25(5):22-25 29, 2008. doi: 10.1 10111099/S.000 2008.130.
David Bieber, Charles Sutton, Hugo Larochelle, and Daniel Tarlow. Learning to execute programs with instruction pointer attention graph neural networks. In Advances in Neural Information Processing Systems, 2020.
Matko Bošnjak, Tim Rocktäschel, Jason Naradowsky, and Sebastian Riedel. Programming with differentiable forth interpreter, 2017.
Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, volume 8, pages 209-224, 2008.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Blizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, 2016.
Xinyun Chen, Chang Liu, and Dawn Song. Execution Execution-guided neural program synthesis. In Interna tional Conference on Learning Representations, 2019. URL httpps::/pppeeeeiiiieieee net/
Eorum??00+
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers, 2019.
Thomas Dietterich, Richard Lathrop, and Tomás Lozano-Pérez. Solving the multiple in- stance problem with axis-parallel rectangles. Artificial Intelligence, 89(1):1177,, 1997. ISSN 0004-3702. doi: sciencedirect.
URL httpps//wwat
Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. Hoppity: Learning graph transformations to detect and fix bugs in programs. In International Conference on Learn- ing Representations, 2020. URL htth:ttt::/// https: httptpps://oenneviee net/ www.moormmioo 2id=
Alexander L. Gaunt, Marc Brockschmidt, Nate Kushman, and Daniel Tarlow. Differenniable pro- grams with neural libraries, 2017.
Patrice Godefroid, Nils Klarlund, and Koushik Sen. Dart: Directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, pages 213-223, 2005.
10

---

Published as a conference paper at ICLR 2023
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines, 2014.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiñisk,, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hass- abis. Hybrid computing using a neural network with dynamic external memory. Nature, 538 (7626):471-476. October 2016. ISSN 00280836. URL http://dx dei.i..iiiii.ooooooooooooo...... wwg//0..000.... 10308/10.138/// 1038/ nature20101
Andreas Griewank and Andrea Walther. Algorithm 799: Revolve: An implementation check-
pointing for the reverse adjoint mode of computational differentiation. ACM Trans. Math. Softw., 26(1): 45, mar 2000. ISSN 0098-3500, doi: 10.1145/347833.344846.. URL https:
//doi aroi.org//00 10.0/.11445335 40/44//4477777 3 347846
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps, 2021. httpttss: arxiv..vvvv...//ii. org/ abs/ abs52 09938.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 98): 1735-1780, November 1997. ISSN 0899-7667. doi: 19t.://co..99.......7... URL
https://doi .org/orgg... /00 0. 166/neeoo 1997. 9. 8.8.7. 1735
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search, 2019. URL https:
//arxiv.orq/aan wwwprr//bb///ae
09436
René Just, Darioush Jalali, and Michael D. Ersst Defects4J: A Database of existing faults to enable controlled testing studies for Java programs. In ISSTA 2014, Proceedings of the 2014 International Symposium on Software Testing and Analysis, pages 437-440, San Jose, CA, USA, July 2014.
Tool demo.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. Hal Daumé III and Aarti Singh, editors, Proceed- ings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5110-5121. PMLR, 13-18 Jul 2020. URL hetps: 7 //proceedings.ooo mlr wttppppswswe/.aaaeeeeeep /ttppeeeecss....ooopee////////aa
html
Rafael-Michael Karampatsis and Charles Sutton. How often do single-statement bugs occur? Pro- ceedings of the 17th International Conference on Mining Software Repositories, Jun 2020. doi: 10.1145/3379599.3388744)
http:://kd
otteioorgg10.. 0eepg/.0.1145//211 /tt.///33399999333333333 3387491
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention- based permutation-innariant neural networks, 2019.
Xia Li, Wei Li, Yuqun Zhang, and Lingming Zhang. Deepfl: Integrating multiple fault diagnosis dimensions for deep fault localization. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 189-180, 2019.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks, 2017.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'tuumee Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode, 2022. URL https::/arxiv.. org/ abtp//33.b2222.... 07814
Benjamin Livshits and Monica Lam. Finding security vulnerabilities in java applications with static analysis. In USENIX security symposium, volume 14, pages 18-18, 2005.
11

---

Published as a conference paper at ICLR 2023
Yiling Lou, Qihao Zhu, Jinhao Dong, Xia Li, Zeyu Sun, Dan Hao, Lu Zhang, and Lingming Zhang. Boosting coverage- based fault localization via graph based representation learning. Proceed- ings of the 29th ACM Joint Meeting European Software Engineering Conference and Sympo- sium on the Foundations of Software Engineering, pages 664-676, 2021.
Flemming Nielson and Hanne Riis Nielson. Interprocedural control flow analysis. In ESOP, 1999.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Au- gustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021a. htLhttp::/ /tttpp:://rxiv..om org/ abs/ 2112. 00114
Maxwell Nye, Yewen Pu, Matthew Bowers, Jacob Andreas, Joshua B. Tenenbaum, and Armando Solar-Lezama. Representing partial programs with blended abstract semantics, 2021
Kexin Pei, Jonas Guan, Matthew Broughton, Zhongtian Chen, Songchen Yao, David Williams-King Vikas Ummadisetty. Junfeng Yang. Baishakhi Ray, and Suman Jana. Former: Fine-Gree Grained
Type Recovery from Binaries Using Generative State Modeling, page 690-702. Association for Computing Machinery, New York, NY, USA, 2021. ISBN 9781450385626. URL https://
doi org 10.
3468607
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer, 2017.
Michael Pradel and Koushik Sen. Deepbugs: A learning approach to name-based bug detection.
Proc. ACM Program. Lang., 2(0OPSLA), oct 2018. doi: 10.1145/3276517. URL https:
//doi. 114.1326551777
Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: large-scale ai for code dataset for learning diversity coding tasks, 2021.
Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. SIGPLAN Not., 51(10):331-477 oct 2016. ISSN 0362- doi: 10.1145/3022671 1.244041.. URL /tt.tss:///ootto http://///006l..000011111000//00 012.59222271..9444.4 2984041]
Scott Reed and Nando de Freitas. Neural programmer-interpreeerrs 2016.
Koushik Sen, Darko Marinov, and Gul Agha. Cute: A concolic unit testing engine for c. ACM SIGSOFT Software Engineering Notes, 30(5):263-21) 2005.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units, 2016.
Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. Learning to combine per-exxple solutions for neural program synthesis, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need. arXiv e-prints, art. arXiv:1706.03762, Jun 2017.
Yun Wang, Juncheng Li, and Florian Metze. Comparing the max and noisy-or pooling functions in multiple instance learning for weakly supervised sequence learning tasks, 2018
Yichen Xie and Alex Aiken. Static detection of security vulnerabilities in scripting languages. In USENIX Security Symposium, volume 15, pages 179-192, 2006.
Wojciech Zaremba and Ilya Sutskever. Learning to execute, 2014.
Zhuo Zhang, Yan Lei, Xiaoguang Mao, and Panpan Li. Cun-ft: An effective approach for localiz- ing faults using convolutional neural networks. 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 445-455. IEFF, 2019.
12

---

Published as a conference paper at ICLR 2023
Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulner- ability identification by learning comprehensive program semantics via graph neural networks. Advances in neural information processing systems, 32, 2019.
Lukasz Kaiser and lla Sutskever. Neural gpus learn algorithms, 2016.
13

---

Published as a conference paper at ICLR 2023
A PYTHON RUNTIME ERROR DATASET DETAILS
We describe in detail the construction of the Python Runtime Error dataset from the submissions Project CodeNet (Puri et al. 202T). The Project CodeNet dataset contains over 14 million sub- missions to 4,053 distinct competitive programming problems, with the submissions spanning more than 50 programming languages. We partition the problems into train, valid, and test splits at an 10:10:10 ratio. By making all submissions the same problem part of the same split we mitigate concerns about potential data leakage from similar submissions to the same problem. We restrict our consideration to Python submissions, which account for 3,286,314 14 of the overall Project CodeNet submissions, with 3,119 of the problems receiving at least one submission in Python. In preparing the dataset execute approximately million problems in sandboxed environment collect their runtime error information, we perform two stages of filtering on the dataset, syntactic and complex- ity filtering, and we construct a textual representation of the input space for each problem from the problem description.
A.1 SYNTACTIC FILTERING
In this first phase of filtering, we remove submissions in Python as well as those which fail to parse and run from our dataset. We remove 76,888 programs because they are in Python 2, 59,813 programs because they contain syntax errors that prohibit parsing, 2,011 programs that result in runtime errors during parsing, and 6 additional programs for which the python-graphs library fails to construct a control-flow graph. A program may result in runtime error during parsing contains return, break, continue keywords outside of appropriate frame.
A.2 PROGRAM EXECUTION
We attempt to run each submission in a sandboxed environment using the sample input provided in the Project CodeNet dataset. The environment is custom harness running on Google Cloud Platform (GCP) virtual environment. This allows us to collect standard out and standard error, to monitor for timeouts, and to catch and serialize any Python exceptions raised during execution. We restrict execution of each program to second, marking any program exceeding this time as a timeout error. If the program encounters Python exception, the target class for the program. If an error type occurs only once in the dataset, we consider the target class be Other. Programs not exhibiting an error or timeout are given target class "no error".
the name of that exception as
In addition to collecting the target class, we record for each runtime error the line number at which the eиror occurs. We use these line numbers as the ground truth for the unsupervised error localiza- tion task considered in Section 5.3
A.3 EXTRACTING RESOURCE DESCRIPTIONS BY PARSING PROBLEM STATEMENTS
For each problem, we parse the problem statement extract the input description and input con- straints, if they exist. These two sections of the problem statement together form the external re- source description that accompanies that problem. The problem statements in our dataset are each written either in English or Japanese, and so we write our parser to support both languages. When one or both of these sections are present in the problem statement, we construct the external resource description for the problem by concatenating together the headers and contents of the sections that are present. For the experiments that use the resource description as a docstring, we prepend each submission docstring containing the resource description for the problem that goes with that sub- mission. Similarly these serve as the resource descriptions in the experiments that process resource descriptions via either cross-attention or FiLM.
A.4 VOCABULARY CONSTRUCTION AND COMPLEXITY FILTERING
All experiments use the same vocabulary and tokenization procedure. For this, we select the stan- dard Byte-Pair Encoding (BPE) tokenization procedure (Sennrich et al.) 2016). We construct the vocabulary using 1,000,000 submissions selected from the training split, along with the input space descriptions constructed for all problems in the train split. We use vocabulary size of 30,000.
14

---

Published as a conference paper at ICLR 2023
200000
150000
Count
100000
50000
100
Program
Statement length (Takens))
Figure 3: A histogram showing the distribution of program lengths, measured in lines, repre- sented in the runtime errors dataset train split.
Figure 4: The distribution of statement lengths, measured in tokens, in the runtime errors dataset train split.
We then apply bazee filtering, further restricting the set of programs considered. First, the program length after tokenization not to exceed 512 tokens, the number of nodes and edges in the control-flow graph are each not to exceed 128, and the step limit T(x) for program computed in Appendix E is not to exceed 174. We select these numbers to trim the long tail of exceptionally long programs, and this filtering reduces the total number of acceptable programs by less than I 1.. To achieve consistent datasets comparable across all experiments, we use the longest form of each program (the program augmented with its input space information docstring) when computing the program sizes for size-based submission filtering.
We further impose the restriction that user-defined functions (UDFs) are called in submission; this further reduces the number of submissions by 682,220. A user-defined function is a function defined in the submission source code, as opposed to built-in function imported from third party module. Extending the IPA-GNN models to submissions with UDFs called at most once trivially achieved by replacing the program's control-flow graph with its interprocedural control- flow graph (ICFG) (Nielson and Nielson 1999). We leave the investigation of modeling user-defined functions further work.
A.5 FINAL DATASET DETAILS
After applying syntactic filtering (only keeping Python 3 programs that parse) and complexity fil- tering (eliminating long programs and programs that call user-defined functions), we are left with dataset of 2,441,130 examples. The division of these examples by split and by target class is given in [able Figure shows the distribution of program lengths in lines represented in the completed dataset, with an average program length of 14.2 lines. The average statement length is 6.7 tokens, with the full distribution shown in Figure
A.6 DATA LICENSE
The Project CodeNet (Puri et al. 2021) data that we use is available under the CDLA Permissive v2.0 license, and we release our derived dataset under this same license.
B UNDER- AODER-APPROXIMATION OF ERROR LABELS
As described in Section 3 the ground truth error targets in our dataset are obtained by running each submission only single input. We do this because we only have a single input available from the online judges with which execute the programs. As a result, the error labels we obtain under-approximate the full set of errors liable to appear at runtime. Metadata obtained from (Puri et al. 2021) indicates whether each submission encountered a runtime error on larger set of inputs, though it does not indicate the kind or location of these errors when they are present. We this metadata to determine the degree to which our labels are an under-approximation. We find that on the balanced test set there 1,076 submissions (4%) which, per the metadata, encounter error,
15

---

Published as a conference paper at ICLR 2023
but for which our evaluation finds no error. These likely examples for which the program without error on the input we have, but for which the program fails on some additional unavailable input.
We next measure generalization from the labels in our dataset to the labels suggested by the metadata without retraining. Since these labels are only binary indicators of error presence, to perform binary classification by summing the predicted probabilities of all error kinds. The model predicts "no error" on 76.2% of the examples for which our dataset finds error. On the examples for which the metadata indicates no error, this drops to 75.9%, and on the examples for which the metadata indicates there is an error, this rises to 80.9%. These examples, where single input detects no error but multiple inputs detect an eπrоr, are difficult for the model to classify. We hypothesize that the types of errors our labels omit systematically differ from those our labels include as an explanation for this 4.7% discrepancy.
model
C IPA-GNN ARCHITECTURE
We provide concise and precise definition of the IPA-GNN baseline architecture, following the notation of Bieber al. (2020). The IPA-GNN operates on the statement-level control-flow graph of the input program maintaining per-node per-step hidden states he and soft instruction pointer Pt.n- At each step i, each node Iy participates in execution, branch prediction, and aggregation. First, the IPA-GNN models executing the statement at each node to produce per-no per-node state proposals
tnn = RNN (h4-1.n, (mb
(8)
Then, the model uses these inform soft branch decisions at every control flow juncture, given
m Denn,ng softmax (Dense (2)
where {ni, n Nour when (Noru(
When (on(()))) 1 have bt,,,,' 1 for
n' E (Tn) indicating straight-line code. For all other n, n', br,,,' 0. The state proposals and branch decisions in turn feed into the computation of the new hidden states
hen
(1) Pt-1n- ben at,
(10)
n' (iENi(())
and new instruction pointer values
Ptn
Pt-1.1 br
(11)
The hidden state at final time step T(x) at the program's 's exit node nexit given by mxx) are used for downstream predictions.
D INPUT MODULATION
In Section 4.2
consider both cross-attention (Lee al. 2019) and Feature-wise Linear Modu-
lation (FiLM) (Perez et al.) (017)) as options for the Modulate function. We provide the definitions of these operations here. First, croos-aatention modules the input as:
MultiHead (Embed(zz)) d(x), hn-) SSunat(Concat(heee
heada)Wo EEmeemdedr....
(12)
QQ'') Vdl
where head, softmax
(13)
Q = W.O Concat(Fmmbed(a) h 11-15)
(14)
K W.K d(x)
(15)
V W.Y d(x)
(16)
16

---

Published as a conference paper at ICLR 2023
Here, WO E Rh x tlooks W.O. E Rdox X (dwwrtteeebbb W Rdaxdaa) and W." Rdlxdaa are learnable parameters. Similarly, for FiLM we modulate the imput with the resource description as follows:
(L.)(Embe((z.))d2a) 1,n) = Concat(3 d(z) + Embedd(r.)) where B ( Concat 14-1,)) + hg), 7 a oW, Concal h1-1, + by),
(17)
(18)
(19)
where W, E Ra x(dmas (2. and Wg E R x (fuoss RRmbed(ss.)) are learnable parame- ters.
E TRAINING DETAILS
Hyperparameter selection We select hyperparameters by performing random search indepen- dently for each model architecture. The hyperparameters considered by the search are listed in Table 6 All architectures use Transformer encoder, and the Transformer sizes considered in the search are listed in and defined further in Table S
HYPERPARAMETER
T- 1-128
T-256
T-512
EMBEDDING DIMENSION
128
256
512
NUMBER OF HEADS
4
4
8
NUMBER LAYERS
2
QKY DIMENSION
256
MLP DIMENSION
512
1024
2048
Table 5: Hyperparameter settings for each of the three Transformer sizes.
Step limit For the IPA-GNN and Exception IPA-GNN, the function T'x) represents the number of execution steps modeled for program æ. We reuse the defimition of T(x) from Bieber et al. (2020) as closely as possible, only modifying to accept arbitrary Python programs, rather than being restricted to the subset of Python features considered in the dataset of the earlier work.
Parameter counts We provide in Table the total number of parameters in each model, for the best performing hyperparameters in each model class. For all model classes, the maximum number of parameters considered is roughly equal (approximately 8.8 million).
Compute usage and model speeds All models trained Google Cloud Platform using TPUv2 accelerators. We use approximately one TPU-week of compute in training each IPA-GNN model. At inference time, IPA-GNN compute is proportional to the number of model steps, which is up 174 for examples in our dataset. We measure the average inference time on the test set: 0.43
VALUE(S)
ARCHITECTURE(S)
OPTIMIZER
(SGD)
ALL
BATCH SIZE
[32]
ALL
LEARNING RATE
(0.01, 0,03, 0.2, 0.3)
LSTM, TRANSFORMERS, IPA-GNNS
LEARNING
(0.001, 0.003, 0.01, 0.03)
GGNN
GRADIENT CLIPPING
[0, 0.5, 1, 2)
ALL
HIDDEN
(64, 128, 256]
RNN LAYERS
(2)
LSTM, IPA-GNNS
GNN LAYERS
(e, 16, 24)
GGNN
SPAN ENCODER
[FTRST,
S3M)
ALL
CHOSS-ATTENTION NUMBER OF HEADS
{1, 2}
PA-GNNs WITH CRUSS-ATTENTION
[Nax, NEAN, LOGSUREXP]
TRANSFORMERS
TRANSFORMER DROPOUT
[0, 0.1l
ALL
TRANSFORMER ATTENTION DROPOUT RATE
(0, 0.13
ALL
SIZE
(2-128, T-256, 2-512)
Table 6: Hyperparameters considered for random search during model selection.
17

---

Published as a conference paper at ICLR 2023
MODEL
PARAMETER COUNT
TRAIN LATENCY
INFERENCE LATENCY
GGNN
4,831,903
0.055
0.040
TRANSFORMER LSTM
8,578,975
0.054
0.051
4,361,822
0.058
0.057
IPA-GNN
4,368,161
0.727
0.294
E. IPA-GNN
8,856,099
1.167
0,435
Table 7: The parameter count, training lateney (sec/step), and inference latency (sec/batch) for the best performing instance of each model variant. Training and inference latencies use batch size 32.
seconds per batch of 32. We also measure the training speed in seconds per step for each method, which we report in Table We observe that the IPA-GNN train times slower than those of the generic models, a drawback of the IPA-GNN model family in its current implementations. That said, we also note that the IPA-GNN models do benefit from the same optimizations basic implementations of the well known general purpose models (GGNN, Transformer, and LSTM), and with further optimizations the IPA-GNN performance can be improved.
F METRIC VARIANCES
Under the experimental conditions Section 5.1 we perform three additional training runs to cal- culate the variance for each metric for each baseline model, and for the Exception IPA-GNN model using the doestring strategy for processing resource descriptions. For these new training runs, we the hyperparameters obtained from model selection. We vary the random seed between runs (0, 1, 2), thereby changing the initialization and dropout behavior of each model across runs. We report the results in Table g + values are standard deviation.
METHOD
R.D.?
Acc.
W. F1
E. FI
GGNN
61.98 1.24
56.62 2.96
41.24 = 5,51
TRANSFORMER
63.82 = 0.62
59.86
46.75 = 0.93
LSTM
66.43 = 0.60
62.33 1.12
50.10 = 1.94
EXCEPTION IPA-GNN
71.44 0.15
70.78 1 0.07
63.54 1: 0.03
Table 8: Mean and standard deviation for each metric is calculated from three training runs per model, using the hyperparameters selected via model selection.
G STATIC ANALYSIS BASELINE
Our work builds towards developer tool that predicts runtime errors in programs without running the program, treating the task as static analysis. Existing static analysis tools already inspect Python source code for possible issues, though they are not generally designed with runtime error prediction in mind. Among the most popular such tools are the linters pylint and flake8, the type analyzer pytype, and the formatter black. We elect to compare against pylint it is the most common of these tools and hence most representative of a modern developer workflow. Additionally, a formatter not well suited to the task of predicting errors, and type analysis benefits from type annotations which are rarely utilized in competition code. In our comparison of machine learning methods against pylint (Section 5) we build a runtime error classifier based on pylint's output. For each kind of error warning that pylint detect, determine whether it is indicative runtime error class. For example, pylint's s error no-member (E1101 101) indicates the AttributeError runtime error. The pylint baseline predicts runtime error class whenever pylint's errors warnings indicate that error class, and "no error" otherwise. Table shows the mapping from pylint findings to runtime error.
Only eleven of the twenty-six runtime classes (those listed in Table 9 and "no error") can be predicted by this baseline. Additionally, the presence pylint finding that corresponds to an error does not guarantee the error will actually be present when running the program; for example
18

---

Published as a conference paper at ICLR 2023
ERROR CLASS
PYLINT FINDING
ERROR CLASS
PYLINT FINDING
AssertionFrror
had-threat instantiation (W1500)
TypeFrror (cont.)
invalid index returedd (F0005)
invalid-rer--retumee (E0306)
AuributelErrer
misplaced function (E0119)
invalid-st--eturred (B0307)
no-member (E1101)
invalid- saalid-bbles-reiurree (E0308))
not context manuger (E11299)
invalid-hasl--ctunns (E0309)
missing formar attrihute (W1306)
invalid-length- rettrre-ee (E8310)
(E1701)
invalid. format returned (EII11)
ImporEEror
import arror (F0401)
seliid-getnewarg--eeemnnn (F00312)
ellrive-beyonn--o--eeeee (E0402)
mwalli-getnvvang-----tnumee (B031) (B0313)
mamoll--mooulll (B061
unpacking- -oo----e----eeeeeeeeee (E06333
risinn had lype (E07002
IndexError
potential- settiiall-nde-eeeeee index-eror (B0643)
bad-csoeption-cuuse (E0705)
fo--tow-format-args (E1306)
ruising non esception (E37
invalid- format-indes (W1307)
patimplementedl-aasee (F5071
KeyEmrrr
missing format argument key (W1303)
catching-non-ee nonee-ooonncccttioon (E0712)
missing format-string-key (F.134)
bad-super-call (E1003)
not-callable (E1102)
NamefError
before assignment (E0660)
isinstance.... valid---ieeee (W1116)
undefined-
variable undifined-variabll (F0602)
no-value- forpue-fr--arrmmeter (E1120)
many function args (E1121)
RuntimeError
misplaced hare raise (B0704)
umexpecied.Kkywwrddddk (R1123)
madified-iteraai iteratinng (F44022)
redundant-koyworddagg (E1124)
modiiicd-itertiinge (B4703)
missing-Kwoa (E1125)
SyntaxFrror
syntax emoir (F0001)
invalid sequence index (E1126)
catam-oorsideffunttie (E0104)
invalid-slice ince--indxxxx
imvaió-umarr--pprna--epe (EI130)
yield outside function (FR1055
deplicane-rguumenn-nnne (E0108)
unsippootted-binnry appaatrrrrrrrrroooonnnooo (EI13)))
too-
(E01
repeated-keyyword (El
slur wws-aasentttttrggee (10113))
bot-an-iterable (E1133)
star needs ussignment turget (E:0114)
ussupponeed-meemeeessip-pccce (E:1155
nonlocal-and- landall-addggooall (5)
unsabscriptable ohjje (E1136)
minloaal-iitbeut-bnlddee (E0117)
unsupporetaasiigneenn.nnnnnnniii (37)
used used-prior- glohal-declaration (F:118)
unsipportoo-ddiit--oeeaaooo (EI188)
owait-outidc-assyn (E1142)
diet-iter-missingg itemms (F.1
yield-inside-ssynne sasss---sya--tunctiio (E1700)
unbashable-member (El
invalid unicude codec (E2501))
Nod-format-characce (E13000
bidirectional unicode uniiccd (F25002)
mixed formal (EI3302)
format needs- mapping (03)
Typelireer
abstract-class instantiated (E0110)
bad-string-fonnat--y (EI3
bad reversed sequence (R)
invaiidcanvas--valee (B1507)
invalid-slos--ohecct (B0236)
invalid-env intiidi-nnaa--fiiillttt (W1558)
invalid slots (B02238
inherit-non-clle (E02399)
ValuelEmor
return in init (E0101)
inconsistent-mro (E0240)
satesssariable-lots-sooomm (E0242)
duplicate- bases (EU241)
unbalasesd-uull- hatlallnsss--ppll--ppekkinng (W0632)
invalid extension (B0244)
bad-format- string (W1302)
invalid cealldd-lnngtheere lew.teennttgeeeedddddd (P0303)
format combined specification (W1305)
nnvaidd-bool-etuumed (E0304)
had-open-made (W1501)
Table 9: The pylint baseline for runtime error prediction predicts the error class shown when encounters any of the corresponding pylint findings. Many of pylint's 235 finding types do not indicate runtime errors. This table shows the mapping used by the pylint baseline.
an undefined variable may appear on an unused control-flow path, benign at runtime. The results of this baseline are reported in Section
H LOCALIZATION BY MODELING EXCEPTION HANDLING
For programs that lack try/except frames, we compute the localization predictions of the Exception IPA-GNN model by summing, separately for each node, the contributions from that node to the global error node across all time steps. This gives estimate of exception provenance as
p(error at statement n.) = Σ Pt.n" Dt.n.
(20)
For programs with try/except frame, however, we must trace the exception back to the statement that originally raised it. To do this, we keep track the exception provenance at each node at each time step; when an exception raises, it becomes the exception provenance at the statement that it raises to, and when a statement with non-zero exception provenance executes without raising, propagates its exception provenance information to the next node unchanged.
19

---

Published as a conference paper at ICLR 2023
Define Ve,,,,' the amount of "exception probability mass" at time step at node n' attributable to an exception starting at node n. Then we write
Vt,,,n'
U1-1,n bLk Pe,k + (1
Σ U2-1;n)) de Pt, In' r(n)). (21)
RE (t
The first term propagates exception provenance across normal non-raising execution, while the sec- ond term introduces exception provenance when an exception is raised. We then write precisely
p(error at statement n)) = (
(22)
allowing the Exception IPA-GNN to make localization predictions for any program in the dataset.
I LOCALIZATION BY MULTIPLE INSTANCE LEARNING
The Local Transformer and Global Transformer models each compute per-statement node embed- dings Ex)ed( given by Equation !
In the multiple instance learning setting, these are trans- formed into unnormalized per-statement class predictions
é(class k lineno = [) (ense(Embed(zz.))
(23)
We consider three strategies for aggregating these per-statement predictions into an overall predic- tion for the task. Under the logsumexp strategy, we treat á as logits and write
log p(class = k) x log
Σ exp (kk,))
(24)
log p(lineno =) x log
Σ KEK
exp ((k.))
(25)
where is the set of error classes.
The max and mean strategies meanwhile follow Wang et al. (2018) in asserting p(class = lineno = =) = softmax(o((:!)).
(26)
compute the location probabilities
plinenoo = x Σ
p(class = k lineno = l),
(27)
AEK
and compute the outputs as
log p(class k:) x log max p(class = k lineno = l), and
(28)
1 log lccass k) x log T, Σ p p(class = lineno =
(29)
respectively, where L denotes the number of statements in As with all methods considered, the MII. models are trained to minimize the cross-entropy loss in target class prediction, but these methods still allow reading off predictions of p(lineno).
J BROADER IMPACT
Our work builds toward improvements to developer tools, suggesting the possibility of future tools that predict runtime errors in code even when that code lacks unit tests. However, the false positive rate under the current best models present challenge. A developer tool built using these models may present the developer with incorrect predictions. This could cause the developer to make mistakes, or to lose trust in the tooling, lowering productivity in the short term and making it harder to win back trust in the long term when tools are built upon higher quality models with fewer errors. We therefore recommend that tool developers use a combination of cautious judgement and data driven evaluations when deciding when to implement features that rely models like the ones present.
20

---

Published as a conference paper at ICLR 2023
K EXAMPLE VISUALIZATIONS
We sample three examples at random from the Python Runtime Error dataset validation split, and visualize them here. As Figure 2 show instruction pointer heatmaps for both the BASELINE and DOCSTRING Exception IPA-GNN model variants. The values in the tables show the total error contribution predicted by the model (Appendix H) associated with each line of code.
In the heatmaps, the x-axis represents time steps and the y-axis represents nodes, with the last two rows representing the exit node nexit and the error node Rerre-- Note that for loop statements are associated with two spans in the statement-level control-flow graph, one for construction of the loop iterator, and a second for assignment to the loop variable. Hence we list two indexes for each for loop statement in these figures, and report the total emrr contribution for the line.
Input: Input is given from Standard Input in the following format: N a.l a.2 a.N
STDIN DESCRIPTION
Constraints: All values in input are integers. 1 N a.i <= 100
n
SOURCE
BASELINE R.D.
0
int (input )))
1
1ist (map (int, input
2
res 0
3.0
63.3
3,
for i in range (1, (A) 2):
9.8
6.3
5
Alil
0.3
0.1
6
print (res)
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
BASELINE
RESOURCE DESCRIPTION
Figure 5: The target error kind is INDEXERROR, occuring on line 5 (n = 5). BASELINE incorrectly predicts No ERROR with confidence 0.808. DOCSTRING correctly predicts INDEXERROR with con- fidence 0.693, but localizes to line 3 (n = 2). Both BASELINE and DOCSTRING instruction pointer values start out sharp become diffuse when reaching the for-loop. The BASELINE instruction pointer value ends with most probability mass at Mexit- The DOCSTRING instruction pointer value has a small amount probability mass reaching Rexits with most probability mass ending at Rerro-
21

---

Published as a conference paper at ICLR 2023
Input: Input is given from Standard Input in the following format: H N A.1 A.2
STDIN DESCRIPTION
A.N
Constraints: 1 <= H <= 10 1 <= N <= 10 <= A.i 10"4
values in input are integers.
SOURCE
BASELINE R.D.
0
H, A, ss (map (int, input () .ppl ()))
9.7
3.4
for A[N]:
43.7
83.0
3
if H C- <- 6:
2.9
break
else:
H A[i]
6.0
0.0
5
if set (A):
6
print ("Yes")
9,3
0.7
else:
7
print ("No")
3,3
0.2
10
15
20
10
BASELINE
RESOURCE DESCRIPTION
Figure 6: The target error kind is VALUEERROR, occuring on line (n = 0). BASELINE incorrectly predicts INDEXERROR with confidence 0.319 on line (n 0). DOCSTRING correctly predicts VALUEERROR with confidence 0.880 on line 2 (n = 1), corresponding to A [n]. Both BASELINE and DOCSTRING instruction pointer values start out sharp and quickly shift most of the probability mass to the error node.
22

---

Published as a conference paper at ICLR 2023
Input: dl d2 dm Two integers and are given in the first line. The available dencminations are given in the second line.
STDIN DESCRIPTION
Constraints: 1 <= n <= 50000 1 <= <= 20 1 <= denemination <= 10000 The denominations are all different and contain 1.
7
SOURCE
BASELINE
R.D.
0
from itertcols import
40.1
1.3
as C
1
map (int, input .ppp ())
2.3
7.1
coin sorted (1ist (int, input sppit
7.2
2.8
3
if coin:
4
print (1)
2.0
1.5
else:
end // coin[0]
0.3
0.1
False
0.3
7,8
for i in range (2, end) :
2.4
0.7
9,10
for tup in 1ist (C 1)):
3.4
1.2
11
if sum (tup) == (rint(i)
0.3
0.0
12
0.3
0.1
13
= break
True
0.9
if b: break 14
0.1
1.4
BASELINE
RESOURCE DESCRIPTION
Figure 7: The target error kind is No ERROR. BASELINE correctly predicts No ERROR with confi- dence 0.416. DOCSTRING also correctly predicts No ERROR with confidence 0.823. The BASELINE instruction pointer value makes its largest probability mass contribution to Tleror at n and ends up with mass split between Nex and Remorr The DOCSTRING instruction pointer value accumulates little probability in nere and ends up with most probability mass in Nexit-
23


Reviews:
[{"review_id": "qxc1DFRBW2", "reviewer": "Reviewer_xTjd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to address the problem of locating runtime error with neural interpreter model. The contribution of this paper includes: (1) a new dataset for the problem, which consists of competitive programs in multiple languages, input data files, input descriptions, runtime error labels. (2) an improved model based on IPA-GNN.", "summary_of_the_review": "The paper proposes a new dataset as well as a new approach for the problem of static prediction of runtime errors. I appreciate the work of publishing the dataset, but the technical contribution of the proposed approach seems to be thin. The impact of the modeling of exception handling is not fully discussed or evaluated.", "strength_and_weaknesses": "Strengths:\n1.\tThis paper an interesting problem of locating runtime error with neural interpreter models. The proposed dataset may be useful for the future study in the area.\n2.\tThe proposed approach can be used for locating the bug, even though the model is trained with only on the labels of error presence and error class.\n3.\tIt is shown that the data description is helpful for predicting RE types and locating RE bugs.\nWeaknesses:\nThe proposed extended IPA-GNN has two key modifications: (1) it takes data descriptions as input. (2) it can model the exception handling. My concerns are: (1) The data descriptions are only available for competitive programs, and may not available for other programs. This limits the usage scenario of the approach. (2) I am worrying that most competitive programs tend to not handle exceptions. How many programs in the proposed dataset have exception handling blocks? If there is not many, the exception handling part of the approach could be insignificant. Also, there is no empirical validation of the exception handling in the experiments.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L3SZCdaPl4", "reviewer": "Reviewer_NFFC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of identifying runtime errors in a program in a static setting. Their proposed model is a modification of an Instruction Pointer Attention GNN (IPA-GNN) proposed in Bieber et al. (2020) [1], which learns to execute the program one instruction at a time in a \"continuous\" manner using embeddings and probabilistic transitions. They train this model on the problem of classifying a program into either a) one of many defined error classes, or b) no error. Then they show that they are able to use the inner states of the model to localize runtime errors occurring in a specific statement, with impressive accuracy.\n\nSince the IPA-GNN [1] model forms the basis for this work, I provide a quick summary of my understanding of this paper.\n\n------------------\n**<IPA-GNN>**\n\nEach statement in a (simple) program is represented with a 4-tuple that constitutes the statement's initial embedding. The execution is simulated in a series of time steps (like an RNN). The model maintains a probabilistic instruction pointer, where $p_{t, n}$ represents the \"weight\" given to statement $n$ at time step $t$. Similarly, there is a hidden state embedding $h_{t, n}$ for each time step and statement. For a time step $t$ and statement $n$, the \"state proposal\" is computed as\n$$ a_{t, n} = \\text{RNN} \\left( h_{t, n}, \\text{Embed}(x_n) \\right) $$\n\nThen :\n\n* To compute the next time step, the model predicts a branch probability using a Dense layer on the state proposal. This is unclear in the paper, but I assume there can only be at most 2 possible next states. I also assume that the Dense layer weights are shared among all such branch predictions.\n\n* The next hidden states $h_{t+1, n}$ and instruction pointer probabilities $p_{t+1, n}$ are computed using a weighted sum over all branch paths that lead *into* $n$, weighted by both the instruction pointer probabilities and the branch probabilities.\n\n**End of </IPA-GNN>**\n\n------------------\n\nThe authors of this paper build on the IPA-GNN in the following ways :\n\n1. They perform their analyses on a large annotated dataset of real Python programs, possibly with runtime errors.\n\n1. Instead of a 4 tuple to represent a statement, they run a Transformer over the token embeddings and pool to get an embedding for each statement.\n\n1. Since these programs are long, the computational graph could be very large. So they apply a \"rematerialization\" trick at each layer to make the model memory efficient (I did not make an effort to understand the details of this).\n\n1. They take a natural language description of the stdin input format and construct an embedding from it using the same Transformer from earlier. This is then combined (\"modulated\") with the statement embedding at each statement and each time step.\n\n1. Before predicting a branch, they use a separate Dense layer to predict whether there is an exception at each statement. All thrown exceptions lead to a special $n_{\\text{error}}$ node.\n------------------\n\nThe authors then evaluate their approach on a) classifying a program into one of various error types (or no error), and b) localizing the statement corresponding to a runtime exception. They show significant gains over their baselines on these tasks. Further, they perform an interpretability study on one handpicked example program.\n\n[1] David Bieber, Charles Sutton, Hugo Larochelle, and Daniel Tarlow. Learning to execute program with instruction pointer attention graph neural networks. In Advances in Neural Information Processing Systems, 2020.", "summary_of_the_review": "It is my opinion that this paper presents a novel idea convincingly and elaborately. I have a few clarifying questions about the details of their algorithm/experiments, along with a minor concern about the choice of one of their baselines. But on the whole, I feel that the paper, even in its current form, is an excellent contribution to the program analysis literature and therefore I wholeheartedly recommend acceptance.\n\nI give myself a confidence score of 3/5 **not** because I don't understand the paper well-enough, but because I am not sufficiently familiar with the current research in this area to know if there are other potential competing approaches to solve this problem.", "strength_and_weaknesses": "**Strengths :**\n\n1. The extensions from the previous IPA-GNN paper are novel and non-trivial.\n\n1. The evaluation is thorough and the results are convincing. The results on error localization in particular are very impressive considering that the model was not trained explicitly for that task.\n\n1. Extensive details have been provided in the appendix about the experimental setup. Further, I appreciate the principled experiment design with carefully constructed metrics (weighted F1, weighted error F1, etc), and model selection performed on validation data as it should be.\n\n**Weaknesses :**\n\n1. I'm unsure if pylint is an appropriate choice for a static analysis baseline. Wouldn't something based on symbolic execution be closer to the spirit of this paper? After all, this model attempts to simulate the execution of the program, which is closer to symbolic execution than static analysis, in my opinion. Further, it is only natural that static analysis cannot catch runtime errors effectively, whereas symbolic execution approaches might be able to find values that throw exceptions. \n\n1. How does this scale with the size of the program? The programs in CodeNet are still relatively small programs compared to kernel code etc which could be hundreds of thousands of lines. What is the execution time/memory for the largest programs in CodeNet? And how does the graph of computation cost with size of program look?\n\n**Suggestions/Clarifications :**\n(I feel like it wouldn't be accurate to call these points \"Weaknesses\")\n\n*Point 1*\n\nRegarding Table 4, here is my attempt to understand what's going on : `|S| = 3`, so if `q` is some number `>3`, we could have an EOF error when it tries to read from stdin more than 3 times. The model sees the constraint `|S| = 3` as part of the stdin description, and that is why it is able to predict this accurately.\n\nCould something like the above ^ be put in the paper, either in the table caption or in the text? I know that the end of Section 5.2 has a brief note to this effect, but it would really be nice if the part about `|S| = 3` could be emphasized as that is the key bit of info that helps the model make the decision (at least as per my understanding).\n\n*Point 2*\n\nRegarding the example in Table 4, I feel it would be interesting to evaluate the model with the same resource description, but with `|S| = 3` removed. It would show that *that* is the piece of information that the model is using to make a better prediction.\n\n*Point 3*\n\nWhen you use a Transformer to generate embeddings of each token (and to embed the resource description) as the first step of E IPA-GNN, do you use a pre-trained Transformer model? What architecture? Do you freeze the weights or do you learn it along with the rest of the model? It would be useful if this information was provided.\n\n*Point 4*\n\nI find the flow of RQ1 and RQ2 a little confusing in the context of Table 2a. When we first read RQ1, it's not apparent that the method of incorporating resource descriptions is as a docstring. Then later in RQ2 that is specified, so we have to jump back to Table 2a to consult it again. I feel like there should be a note in RQ1 specifying that docstrings are used, and other methods like FiLM and Cross-attention will be discussed in the next RQ.\n\n*Point 5*\n\nDoes IPA-GNN (not E.IPA-GNN) also have an $n_{\\text{error}}$ state? \n\nIf not, how do you use IPA-GNN to predict errors? For E. IPA-GNN the method is clear - use a dense layer on the $n_{\\text{error}}$ state in the last timestep. For IPA-GNN, what state do you use?\n\nSimilarly for localization. How do you localize a bug using IPA-GNN if there is no $n_{\\text{error}}$ state?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5zWQqvh0k7S", "reviewer": "Reviewer_Gdcd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose an approach to predict runtime errors of programs by using IPA-GNN models with improvements to handle exceptions and provide error locations. They also add support for passing resource descriptions to the model. Authors provide a dataset for testing their approach. They show that their approach outperforms baselines. ", "summary_of_the_review": "I think that authors present a novel improvement of IPA-GNN to handle exceptions, real programs and external descriptions. Their approach improves prediction of runtime errors vs baselines and I think this may warrant accept to ICLR.", "strength_and_weaknesses": "Strengths:\n\n- Improvement of IPA-GNN model to handle exceptions and real programs\n- Adding handling of resource descriptions to IPA-GNN\n- Unsupervised localization of errors\n- Ablation analysis\n- Improved results compared to baselines and ablations\n\nWeaknesses:\n\n- Authors compare results to baselines, but do not try to identify state of the art results for their testset and compare against state of the art methods\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster