CRABS: A syntactic-semantic pincer strategy for bounding LLM interpretation of Python notebooks
Authors: Meng Li
Timothy M. McPhillips
Dingmin Wang
Shin-Rong Tsai
Bertram Ludäscher
Venue: COLM 2025 (2025)
Keywords: notebook understanding
LLM
YesWorkflow
data flow
provenance
TL;DR: This paper proposes a notebook understanding task yielding an information flow graph and corresponding cell execution dependency graph for a notebook, together with a Capture and Resolve Assisted Bounding Strategy (CRABS).

Abstract: Recognizing the information flows and operations comprising data science and machine learning Python notebooks is critical for evaluating, reusing, and adapting notebooks for new tasks. Investigating a notebook via re-execution often is impractical due to the challenges of resolving data and software dependencies. While Large Language Models (LLMs) pre-trained on large codebases have demonstrated effectiveness in understanding code without running it, we observe that they fail to understand some realistic notebooks due to hallucinations and long-context challenges. To address these issues, we propose a notebook understanding task yielding an information flow graph and corresponding cell execution dependency graph for a notebook, and demonstrate the effectiveness of a pincer strategy that uses limited syntactic analysis to assist full comprehension of the notebook using an LLM. Our Capture and Resolve Assisted Bounding Strategy (CRABS) employs shallow syntactic parsing and analysis of the abstract syntax tree (AST) to capture the correct interpretation of a notebook between lower and upper estimates of the inter-cell I/O set$\textemdash$the flows of information into or out of cells via variables$\textemdash$then uses an LLM to resolve remaining ambiguities via cell-by-cell zero-shot learning, thereby identifying the true data inputs and outputs of each cell. We evaluate and demonstrate the effectiveness of our approach using an annotated dataset of 50 representative, highly up-voted Kaggle notebooks that together represent 3454 actual cell inputs and outputs. The LLM correctly resolves 1397 of 1425 (98%) ambiguities left by analyzing the syntactic structure of these notebooks. Across 50 notebooks, CRABS achieves average $F_1$ scores of 98% identifying cell-to-cell information flows and 99% identifying transitive cell execution dependencies. Moreover, 37 out of the 50 (74%) individual information flow graphs and 41 out of 50 (82%) cell execution dependency graphs match the ground truth exactly.

Published as a conference paper at COLM 2025
CRABS: A syntactic-semanti pincer strategy for bounding LLM interpretation of Python notebooks*
Meng Li', Timothy M. McPhillipst1, Dingmin Wang?, Shin-Rong Tsai', Bertram Ludäscher!
"ccooo of Information Sciences, University of Illinois Urbana- crbana-Champaign
Department of Computer Science, University of Oxford
Abstract
Recognizing the information flows and operations comprising data science and machine learning Python notebooks is critical for evaluating, reusing, and adapting notebooks for new tasks. Investigating notebook via re- execution often is impractical due to the challenges of resolving data and software dependencies. While Large Language Models (LLMs) pre-trained on large codebases have demonstrated effectiveness in understanding code without running it, we observe that they fail to understand some realistic notebooks due hallucinations and long-context challenges. To address these issues, we propose a nolebook underslanding lask yielding an information flow graph and corresponding cell execution dependency graph for a notebook, and demonstrate the effectiveness of pincer strategy that uses limited syntactic analysis to assist full comprehension of the notebook using an LLM. Our Capture and Resolve Assisted Bounding Strategy (CRABS) employs shallow syntactic parsing and analysis of the abstract syntax tree (AST) to capture the correct interpretation of notebook between lower and upper estimates of the inter-cell I/O set-the flows of information into or out of cells via variables- then uses an LLM to resolve remaining ambiguities via cell-by-cell zero-shot learning, thereby identifying the true data inputs and outputs of each cell. We evaluate and demonstrate the effectiveness of our approach using an annotated dataset of 50 representative, highly up-voted Kaggle notebooks that together represent 3454 actual cell inputs and outputs. The LLM correctly resolves 1397 of 1425 (98%) ambiguities left by analyzing the syntactic structure of these notebooks. Across 50 note- books, CRABS achieves average F, scores of 98% identifying cell-to-cell information flows and 99% identifying transitive cell execution dependen- cies. Moreover, 37 out of the 50 (74%) individual information flow graphs and 41 out of 50 (82%) cell execution dependency graphs match the ground truth exactly.
1 Introduction
Computational notebooks, such as Jupyter notebooks (Kluyver et al., 2016), provide readable and executable medium combining code, results, and explanations that facilitates sharing of notebook functionality among users. Generally, users find it necessary to execute a notebook to discover how works in detail, confirm that works correctly, and assess its potential for reuse. However, only one in four notebooks discovered on GitHub that specify both Python version and execution order can be re-executed without errors (Pimentel et al., 2019). Most of these errors stem from module dependency and data accessibility
"Code and data available https: //gi wwwptthh.....ssstteeeeeeeeeeeeee cww./cirss/craas/rreeee......... *Corresponding author: tmephill@illinoiseetu 'Among 1,081,702 unique Python notebooks discovered on GitHub, Pimentel et al. (2019) attempted to execute the 863,878 notebooks for which both Python version and execution order could be deter- mined, and found that only 24% of the latter notebooks could be executed without errors using either the declared dependencies comprehensive, preconfigured Anaconda environment.
1

---

Published as a conference paper at COLM 2025
issues that are often difficult or even impossible to resolve. For this reason, facilitating interpretability of notebooks in the absence of re-exeution would represent a major step toward improving reproducibility and lowering barriers to reuse. Such interpretability requires understanding what individual variables mean, where they are defined, and where they are subsequently used. Answers to such questions are critical for evaluating, reusing, and adapting a notebook for new tasks.
Workflow management systems and frameworks (e.g., Galaxy (Goecks et al., 2010), Ke- pler (Ludäscher et al., 2006), Taverna (Oinn et al., 2004), RestFlow (Tsai et al., 2013), and Snakemake (Köster & Rahmann, 2012)) make these dataflows explicit, thereby facilitating evaluation, reuse, and repurposing. However, these systems generally require that code be structured or restructured to fit the workflow model each implements. In contrast, YesWork- flow (McPhillips et al., 2015) provides means to reveal the dataflows implicit in scripts without requiring users to modify their code. Instead, the author of the code declares the dataflows via annotations included in comments. The YesWorkflow annotation vocabulary models scripts as information flow graphs, i.e, collections of program blocks connected by channels representing paths of data flow. When applied to Python notebooks, code cells naturally correspond to the YesWorkflow program blocks, while variables assigned in one cell and used in another correspond to channels. Viewing notebook this way suggests that it should be feasible to generate meaningful information flow graphs for notebooks automatically, thereby making it clear what a notebook does and how it works running it.
Extracting the information flow graph from a computational notebook requires not just iden- tifying how data flows between code cells, but also how user-defined functions and classes declared in one cell are used another. Fortunately, recognizing code cells that define or call user-defined functions and classes is relatively straightforward via syntactic parsing and analysis of the abstract syntax tree (AST). In contrast, identifying which cells actually assign new values to variables or update them is challenging without comprehensively analyzing the source code of imported modules or executing the cells. For example, the statement phone data. dropna (inplace=True) updates the phone data object in place, while the state- ment phone data. truncate (before=1, after=3) returns information from the phone data object without modifying it. Without understanding the mechanisms of these two methods,
or comparing the phone data object before and after executing the code, it is not possible to determine whether the phone data object is updated not. Therefore, how data flows between code cells can be ambiguous if only the syntactic structure apparent in code cells is taken into account.
We observe that experienced data scientists often can resolve these ambiguities without accessing source code for imported modules and without executing the cells by hand. This raises the question: can Large Language Models (LLMs) do the same thing? LLMs pre- trained on large codebases have been shown to be effective on code-related tasks, e.g., bug detection (Wang et al., 2024), code understanding (Nam et al., 2024), code completion (Zhang et al., 2023; Cheng et al., 2024), code generation (Yin et al., 2023), and code summarization (Haldar & Hockenmaier, 2024). For these reasons, we hypothesize that LLMs can help resolve ambiguites left by purely analyzing the syntactic structure of the notebook itself, iee, just the code in its cells.
At present, however, LLMs cannot reliably analyze entire notebooks on their own. As we show in Section 5.3, LLMs face hallucination issues and long-con long-context challenges with the result that they sometimes identify variables that do not exist, and for longer notebooks frequently return information flow graphs with the wrong number of code cells. To address these issues, we adopt an approach that reduces the overall workload assigned to the LLM and limits its tasks to those that entail semantically informed analysis of the code.
In this paper, we propose the Capture and Resolve Assisted Bounding Strategy (CRABS), syntactic-semantic hybrid strategy for extracting the information flow graph from Python notebook. Figure 1 is an overview of the steps employed by CRABS to generate an information flow graph for a given Python notebook. CRABS depends critically on the concept of a notebook's 's inter-cell 1/O set, defined as the set of all flows of information into or out of individual cells via variables. We first produce lower and upper estimates of the
2

---

Published as a conference paper at COLM 2025
Lower Estimate
Upper Estimate
cell 1
(cell 1, phone,
out)
(cell 1, phone,
out)
(cell survey, out)
(cell 1, survey, out)
cell 2
(cell ec, (cell 3, phone,
out)
(cell 2, ec,
out)
(cell 3, phone,
in
cell 3
(cell 3, ec,
in
(cell 3, ec,
in
(cell 4, phone, in
(cell 3, phone,
out)
cell
(cell 4,
in
(cell 3, ec,
out)
(cell survey, in
(cell 4, phone,
in
cell 5
(cell 5, survey, out) (ceLL 6, phone, in
(cell 4, ec,
in
(cell 4, phone,
out)
cell 6
(cell 6, survey, in
(cell 5, survey, in
(cell 5, survey, out)
A Python Notebook
(cell 6, phone, in
Capture (Syntactic Phase)
(cell 6, survey, in
survey cell
Is phone oulput candidate cell 3?
cell 1
phone phone
survey
No
phone cell
cell
[x] (cell 3, phone, out)
[X]
(cell 3, out)
ec
ec
M
(cell 4, phone, out)
cell 2
Resolve (Semantic-awane Phase)
Output (information flow graph)
An overview of CRABS using the example notebook in Figure 2. CRABS first produces lower and upper estimates of the inter-cell I/O set in syntactic phase, then resolves the differences between these estimates using an LLM in a semantic-aware phase, and finally constructs an information flow graph with thick blue edges indicating this resolved set.
Figure 1:
inter-cell I/O set based on the syntactic structure of the notebook, then prompt LLM to resolve the differences between these two estimates (i.e., ambiguities) one by one. To evaluate this approach, we apply CRABS to 50 representative, highly up-voted Python notebooks from the Meta Kaggle Code dataset (Plotts & Risdal, 2023).
Our main contributions are: (1) annotating 50 notebooks, manually identifying the 2437 cell-to-cell information flows and 7998 transitive cell execution dependencies in these notebooks to serve as the ground truth for our analyses; (2) producing lower and upper estimates of inter-cell I/O sets that bound the ground truth represented by our notebooks; (3) demonstrating that LLMs correctly resolve individual ambiguities in how data flows between code cells in these notebooks with 98% accuracy; (4) showing that the CRABS strategy of prompting LLMs to answer simpler, highly specific questions about notebook mitigates long-context challenges and eliminates hallucination of variables; and (5) generally demonstrating the effectiveness of a pincer strategy that places bounds both on the total workload assigned the LLM, and the individual responses that can return.
2 Related Work
Information flows in computational notebooks. Existing approaches for revealing in- formation flows in computational notebooks depend on human annotations (McPhillips et al., 2015; Ramasamy et al., 2023), structuring code to follow particular set of guidelines (Carvalho et al., 2017; Koop & Patel, 2017), or using runtime information (Koop Patel, 2017; Wenskovitch et al., 2019; Brachmann al., 2020; Li et al., 2024).
3

---

Published as a conference paper at COLM 2025
# cell
cell 1
survey
cell
import pandas as pd
phone = readeed_coomm "redd_cv(""phoee .ewponeeeeerdddeeeeeeeeeeeeeeeee csv") survey pd.read_csV www.esssesssssseeeeeeeeeueeeeeee ver.roo.cccvserr.ssvvsss..s. surveys.csv")
phone
phone
survey
phone
cell
cell
cell 6
cell # exclusion criterion
= "yalidhourscount 12"
ec
cell 2
# cell (en(phone.query(ec))
query(ec)) / Lennphhne)
Figure 3: Ground-truth information flow
# cell 4
graph for the example notebook (Figure 2).
mmone.dropt.
chone.query(C. index, inplace=Trve)
cell
cell
# cell symptons ["pain", "fatigue", "anxiety"] scores survey[synptons [ mean(axis=1) Survey(""sore"" = scores
cell
cell
# cell
data phone. meroe (survey, on="date")
direct dependency
cell 24
indirect dependency
Figure 2: An example Python notebook with code cells, representing simplified data preparation pipeline for symptom burden prediction using smartphone records.
Figure 4: Ground-truth cell execution depen- dency graph for the example notebook (Fig- ure 2).
LLMs and computational notebooks. Pre-trained language models have been leveraged to generate feedback on answers to open-ended questions about instructional coding or math notebooks (Matelsky et al., 2023); and fine-tuned for code generation (Huang al., 2022; Yin et al., 2023; Li et al., 2023; Huang et al., 2024). Cao et al. (2024) explore multimodal agents for data science notebook generation.
Dataflow, LLMs, and code-related tasks. Dataflow analysis is fundamental to compiler design. Guo et al. (2021) leverage dataflow structure of code train a language model for code-related tasks. Wang et al. (2024) employ LLMs to generate customized AST-based parsers for dataflow analysis in support of bug detection. Cheng et al. (2024) employ an AST-based parser and dataflow analysis to construct repo-specific context graph for repository-level code completion.
3 A Model of Information Flow in Python Notebooks
CRABS views Python notebook as sequence of n code cells C (C1, Cn., where 9j is the jth code cell (markdown cells are ignored). CRABS further assumes that cells are executed in order, i.e, from the top to the bottom of the notebook. This assumption limits the possible interpretations of information flows in a notebook; i.e., value assigned to a variable in cell çj cannot be used in cell cj-1 or earlier. We represent the m distinct units of information (i.e., variable values and code references) that flow between cells as (i, ..., im and define an information flow graph as set of information flows F C (c)s, The information flow graph in turn implies a cell execution dependency graph, i.e., the set of transitive dependencies D C ((,,, indirectly depends on cell c, The cell execution dependency graph can be derived from the information flow graph by (1) recognizing direct dependency between two cells if at least one information flow exists between them in the information flow graph; and (2) computing the transitive closure of this graph to infer indirect dependencies.
<
S n), where iz flows from source cell to target cell сy-
< < S n1,, where cell directly or
The model of information flow that CRABS adopts for Python notebooks aims to support number of crucial use cases. Using the information flow graph, it is straightforward
4

---

Published as a conference paper at COLM 2025
to answer questions such "Which cells may have contributed to the final value of particular variable?", "Which cells may have been influenced by a particular variable?", and "Which cell defines particular function class and which cells ct?" it?" Similarly, using the cell execution dependency graph, one can answer questions such as "What cells depend (directly indirectly) on particular cell?" and "What cells influence a particular cell?"
As an example, consider the simple, six-cell data preparation notebook represented in Figure 2 and the corresponding information flow and cell execution dependency graphs in Figure 3 and Figure 4. The information flows (C1, C5, survey) and (es, C6, survey) can easily provide cell 1 and cell 5 as the answer to "What cells may have contributed to the value of the survey dataframe received by cell 6?" The transitive dependency (C6, C5) can easily provide cell 6 as the answer to "What cells depend on cell 5?"
We. refer to the problem of identifying the information flow graph and corresponding cell execution dependency graph for a notebook as the notebook understanding task. In the remainder of this paper, we describe and evaluate how CRABS performs this task.
4 Elements of CRABS
Constructing the information flow graph and corresponding cell execution dependency graph for a notebook depends crucially on determining its inter-cell I/O set. The approach taken by CRABS to determine the inter-cell I/O set is based the observation that some members of the set are certain by shallow syntactic parsing and analysis of the AST, while others are uncertain but may be resolved by taking the semantic meaning of these code cells into consideration. CRABS therefore performs the notebook understanding task in two phases: syntactic phase and semantic-aware phase. The syntactic phase employs shallow syntactic parsing to extract an AST based purely on the notebook itself, and analyzes the extracted AST to yield lower and upper estimates of the inter-cell I/O set. The semantic- aware phase resolves the ambiguities between these lower and upper estimates using an LLM.
We represent the inter-cell I/O set as S C {(c, it lag) |c E (I,, lag E {out, in]], where the out tag denotes that information i flows out of source cell Cs, and the tag denotes that information i& flows into target cell c,. Therefore, it is critical to identify four items for each cell, i..,, data inputs, data outputs, code declarations, and code references. If data in is defined in cell c, and used in cell C,, then is is a data output of C3 and a data imput of Ct- However, when cell Cs is examined in isolation, we cannot determine whether there is in fact subsequent cell e; that uses iç- Because we do not provide any information about subsequent cells when asking the LLM to resolve ambiguous information flows associated with particular cell, we refer to dala oulput candidales rather than dala outputs in these prompts. Code references, e.g., functions and classes declared in cells and used in later cells, are treated analogously data. In the rest of this section, we describe the syntactic phase and semantic-aware phase based on these four items. Relevant examples are attached in the Appendix.
4.1 Syntactic Phase
The syntactic phase of CRABS performs limited static analysis of the code in a notebook to yield two estimates of the inter-cell I/O set, representing lower and upper estimates of the actual flows of information into or out of each cell. It produces two estimates because CRABS aims to interpret Python notebooks without accessing the source code for any dependencies, e.g., code defined outside the notebook and accessed via function calls. Deeper or more sophisticated syntactic parsing and analysis of the AST that descends into each function called from cell likely would achieve tighter lower and upper estimates of the inter-cell I/O set, leaving fewer or even no ambiguities to be resolved by an LLM. However, because we restrict our syntactic analysis to code explicitly included in notebook's 's cells, CRABS does not need to resolve software dependencies. Moreover, even comprehensive static analysis will not yield unique prediction of the dataflow between cells when conditional control
5

---

Published as a conference paper at COLM 2025
import pandas as pd
train = pd. read_ cedd__ss_ee (esarddsssttaa.cvv
test = pd. read_csv teeedddsswetetttteeee c"test.csv")
4
datasets [train, test]
train. dropna
Figure 5: An example shared reference issue.
flow structures are employed, e.g., if variables are assigned, used, or updated conditionally (see Appendix A.1.1).
CRABS makes three basic assumptions about the code in Python notebooks relevant to the syntactic phase. First, execution flow follows a strict top-to-bottom order, starting from an initial state with all variables, imports, and other runtime information cleared from memory. Second, all code is reachable. And third, function and class names are not reused as variable names or vice versa (see Appendix A.1.2). In addition, our current implementation has four further limitations. Global variables that are accessed or modified within a function, without being explicitly passed as arguments, will not be recognized on this basis data inputs or data output candidates by our system (see Appendix A.1.3), Lines of code employing languages other than Python, e.g., shell commands invoked using the syntax, or IPython magic commands invoked using the % syntax are ignored (see Appendix A.1.4). A third limitation related shared data references (aliased variables) is more complex. This issue arises when multiple variables reference the same mutable object in memory, and the object is modified. For example, the first code cell in Figure 5 defines two Pandas DataFrames train and test. The datasets object is list that references two mutable objects, train and test. In the second code cell, train is modified in place, resulting in an implicit change hidden modifications, wherein variable
in datasets. We define this phenomenon name (i.e., datasets) is not explicitly written in a code cell but should still be considered data output candidate. The current implementation of CRABS detects only those shared references explicitly indicated by assignment (e.g., x = y) or by inclusion in collections (i.e., mutable objects stored in list, tuple, or a dictionary). This covers all instances of shared references in our dataset; we will extend CRABS to detect shared references established by other mechanisms in the future. Finally, CRABS produces a single information flow graph per notebook, even when multiple execution paths are possible (e-g., due to conditional statements) given different input data. When different execution paths yield different information flows, the output of CRABS represents the union of those flows.
The lower estimate computed by the syntactic phase represents the set of cell inputs and output candidates considered certain (i.e., unambiguous) given just the code apparent in the cell. This lower estimate is computed using the following rules: (1) Variables explicitly passed to function are regarded as data inputs only. (2) The iterator of a for loop, while loop, or a comprehension is regarded as a data input only, unless is directly reassigned in the loop body. (3) Because CRABS yields information flows for all possible execution paths, it generally treats code in every branch of a conditional statement as contributing to the inputs and the output candidates of the cell. (4) However, when a conditional statement occurs inside a loop, the intersection of all possible execution orders is considered (see Appendix A.1.5 for an illustrative example). (5) The effect of shared references (e-g, aliased variables) is not considered. Figure 6a and Figure 6c show the information flow graph and the corresponding cell execution dependency graph generated from the lower estimate.
The upper estimate represents the set of cell inputs and output candidates that are either certain or ambiguous, which is a superset of both the ground truth and the lower estimate. This upper estimate is computed using the following rules: (1) Variables explicitly passed to a function are regarded as both data inputs and data output candidates.
(2) The iterable of
(e.g., the collection traversed by) a for loop, while loop, or a comprehension is regarded both as a data input and a data output candidate. (3) As with the lower estimate, code on all branches of conditional statement are generally considered to contribute to the inputs and the output candidates of the cell. (4) But unlike the lower estimate, when a conditional statement occurs inside a loop the union of all possible execution orders is considered.

---

Published as a conference paper at COLM 2025
survey
survey
cell
Cell
cell 1
cell 5
phone
phone
survey
phone
survey
phone
phone
phone
Coll 3
cell
cell
CeLL
cell
cell
ec
ec
ec
ec
cell
CeLl
(a)
(b)
cell
cell 5
cell 1
cell
Gell 3
cell 4
cell
cell
cell 4
cell 6
identified
correct incorrect
cell
missing
cell 2
(c)
Figure 6: Information flow graphs derived from the (a) lower and (b) upper estimates of the inter-cell I/O set for the notebook in Figure 2; (c) transitive dependencies (solid black) extracted from Figure 6a augmented with missing edges (dotted red); and (d) transitive dependencies extracted from Figure with correct (black) and incorrect (red) dependencies.
(5) The effect of shared references is taken into account. Figure 6b and Figure 6d show the information flow graph and the corresponding cell execution dependency graph generated from the upper estimate.
Figures 6c and 6d demonstrate the subgraph and supergraph relations between the cell execution dependency graphs generated from lower and upper estimates, respectively, and the ground truth. The identified dependencies (black) in Figure represent subgraph of the ground truth, whereas the correct (black) and incorrect (red) edges in Figure 6d together represent a supergraph of the ground-truth cell execution dependency graph.
Semantic-aware Phase
We hypothesize that pre-trained language models can help resolve ambiguous members of the inter-cell I/O set identified by the syntactic phase. We resolve two different kinds of ambiguities, (1) checking if data an input of a given code cell and (2) checking if data is an output candidate of given code cell. An LLM resolves these ambiguities cell by cell using a zero-shot in-context learning approach. Example prompts are included in Figure 15 and Figure 16 of the Appendix. Requesting binary answers (i.e., "yes" "no") eliminates hallucinations of variables; all data inputs and output candidates are selected from the given syntactic phase results. Prompts include information about shared references defined in previous cells only when shared references are among the ambiguous data. Since the names of shared references are identified in the syntactic phase, the LLM can be prompted for each code cell independently, improving time efficiency.
Experiments
5.1 Dataset
Our dataset comprises set of 50 annotated Python notebooks. We assembled the dataset from the most up-voted Python notebooks in Kaggle that met the following selection criteria:
7

---

Published as a conference paper at COLM 2025
per notebook
sum
avg
min
25%
50%
75%
max
code cells lines
1493
30
20
25
40
76
10576
212
26
125
288
1111
information flows
2437
49
23
32
78
153
transitive dependencies
7998
160
55
75
157
922
Table 1: Summary statistics for the CRABS dataset.
1. Notebook implements a machine learning workflow. Notebooks containing basic tutorials and other instructional materials were excluded.
2. Variable use is explicit and clear. We excluded Python notebooks that used global variables within functions, reused variable names function class names (and vice versa).
3. Information flows comprise a single connected graph. Notebooks excluded on this basis either implemented multiple independent workflows, routed cell-to-cell dataflows through external files.
We determined which notebooks met these criteria by carefully inspecting their contents. We inspected the 104 most up-voted Python notebooks to find 50 meeting these criteria. Reasons for excluding the other notebooks are indicated in Table of the Appendix. For each notebook selected, we extracted the corresponding kernel version ID from the Meta Kaggle dataset (Risdal & Bozsolik, 2022), and retrieved the notebook from the Meta Kaggle Code dataset (Plotts & Risdal, 2023).
We annotated each of these notebooks by hand, identifying the inputs and outputs of each cell. These annotations serve the ground truth for evaluating CRABS. Table 1 summarizes the characteristics of the selected 50 notebooks. These notebooks generally are nontrivial, average containing 49 information flows and 160 transitive dependencies each. Detailed statistics for each notebook are reported in Table 6 of the Appendix. As these are highly up- voted Kaggle notebooks representing both high-quality tutorials and real-world notebooks for machine learning pipelines, we consider them representative of general data science and machine learning notebooks.
5.2 Evaluation Metrics
We evaluate how well CRABS performs the notebook understanding task at three different levels. The Exact Match (EM) metric captures the notebook-level performance, and is defined as the percentage of notebooks for which the generated information flow graph or cell execution dependency graph exactly matches the ground truth. At the next level of detail, we evaluate how well CRABS predicts inter-cell information flows and execution dependencies using the metrics Precision, recall, F, score, and accuracy. These metrics are widely used evaluate information retrieval (Kobayashi & Takeda, 2000), classification (Hossin & Sulaiman, 2015), and link prediction (Daud et al., 2020) tasks, and are easily adapted to our task by treating the returned information flows and transitive dependencies as predicted links, and the human annotated flows ground truth. Finally at the level of resolving individual syntactic ambiguities, we evaluate the accuracy of LLM responses defined the fraction of questions answered correctly.
5.3 Main Results
Results were obtained using the 2pt-40-2024-08 2024-08-06
version of the GPT-4o model (OpenAT, CRABS against
2024a) with temperature set We compared the performance baseline approach which employs zero-shot prompting of an LLM for the whole notebook with carefully designed chain-of-thought prompts (Wei et al., 2022). An example prompt is shown (in part) in Figure 14 of the Appendix. We observe that the LLM completely fails to understand 10 out 50 (20%) notebooks, returning the wrong number of cells and earning values for all metrics. These 10 notebooks are long with 52 cells on average, where the
8

---

Published as a conference paper at COLM 2025
Information Flows (%)
Transitive Dependencies (%)
Method
Prec.
Rec.
F1
Acc.
EM
Prec.
Rec.
F1
Acc.
EM
baseline
73.51
67.89
70.10
64.39
26
79.45
68.95
72.90
68.59
36
upper estimate
56.31
56.39
56.35
42.70
8
53.06
100
66.08
53.06
8
lower esfimate
92.05
89.91
90.79
85.19
42
100
86.65
91.98
86.65
42
CRABS
98.61
98.53
98.57
97.32
74
99,42
99.95
99.67
99.37
82
Table Main results for the notebook understanding task.
LLM faces long-context challenges. For other notebooks, the LLM sometimes hallucinates, describing variables not present at all in the reported cells.
Table summarizes the performance of CRABS for the notebook understanding task. These results average scores across 50 notebooks in our dataset. We observe percentage-point increase of 28 in F1 score, 32 in accuracy, and 48 in EM for information flow graphs; and percentage-pon increases of 26 in F1 score, 30 in accuracy, and 46 in EM for cell execution dependency graphs. These results demonstrate the effectiveness of CRABS for the notebook understanding task. We include in this table metrics for the lower estimate and upper estimate components of CRABS for comparison with the full CRABS approach. For the transitive dependencies, the precision for lower estimate and the recall for upper estimate are each 100%, consistent with the subgraph and supergraph relations described in Section 4.1. We note that contrast to the baseline, CRABS yields plausible outputs (i.e., earns non-zero contributions for each metric) for all notebooks; attribute this improvement to the cell- by-cell prompting strategy CRABS employs which avoids the long- congecontext challenges faced by the baseline method.
CRABS yields information flow graphs that exactly match the ground truth for 37 notebooks. Of these, 4 return an identical lower and upper estimate. As LLM resolves ambiguities between these estimates, the LLM is not used for these notebooks. The metrics in Table thus do not fully reflect the capability of the LLM for resolving ambiguities. We evaluate such capability at finer level of granularity, finding that 1397 out of 1425 (98%) ambiguities are correctly resolved by the LLM.
5.4 Ablation Studies
Because cell execution dependency graphs are extracted from information flow graphs, we only discuss the information flow graphs in this section. For completeness, the results of the ablation studies with respect to cell execution dependency graphs reported in Table 8 and Table 9 of the Appendix. Detailed results for each notebook are reported in Table 10 of the Appendix.
S1. Effectiveness of the syntactic phase. Instead of using the syntactic-semantic hybrid approach, we test the performance of an approach without the syntactic phase, asking the LLM to identify cell inputs and output candidates without providing any hints from the syntactic processor. An example prompt in Figure 17 of the Appendix. Table shows percentage-point decreases of in F, score, 10 in accuracy, and 36 in EM, demonstrating the effectiveness of syntactic parsing.
S2. Effectiveness of cell-by-cell prompting. Instead of using the cell-by-cell prompting approach, we test the performance of using single prompt for the whole notebook. Part of the example prompt is shown in Figure 18 of the Appendix. Table shows percentage- point decreases of 16 in F1 score, 20 in accuracy, and 38 in EM. The main reason it drops dramatically is the long-context challenge, which causes the LLM to fail to handle multiple code cells using a single prompt. Five notebooks, each containing an average of 54 code cells, return the wrong number of cells. Detailed metrics for the remaining 45 notebooks are shown in Table 7 of the Appendix. These results demonstrate that cell-by-cell prompting outperforms the single prompt approach.

---

Published as a conference paper at COLM 2025
Information Flows
Method
(re
Rec.(%)
(16%
Acc.(%)
EM(%%
CRABS
98.61
98.53
98.57
97.32
74
w/o syntactic phase w/o cell-by-cell prompting
95.29
89.38
91.22
86.37
38
82.97
80.93
81.71
76.85
Table 3: Ablation study results.
Conclusion
We have demonstrated that the notebook understanding task is feasible even in the absence of comprehensive code analysis and without executing notebook cells. CRABS achieves this by employing simple syntactic analysis to limit the tasks given to LLM and to delimit its possible answers. This success suggests that such a pincer strategy may represent promising general approach to combining neural and symbolic methods.
Future Work
Work is underway to enhance CRABS to extract and represent multiple distinct information flow graphs for single notebook. This refinement will enable more precise and comprehen- sive analysis by capturing multiple alternative information flows that may emerge under different conditions.
We are currently investigating approaches for optimizing the overall execution performance of CRABS, including reducing the total latency of the notebook understanding task. Prelimi- nary analysis shows that sequential implementation of CRABS achieves lower latency for this task than the baseline for notebooks with small number of ambiguities, but that this latency increases beyond that of the baseline approach as the number of ambiguities grows. Further experiments demonstrate that concurrent implementation of CRABS consistently outperforms the baseline in terms of latency, regardless the number of ambiguities. These findings (see Appendix A.2) point to the potential for CRABS to achieve the performance re- quired for practical, interactive applications. Smaller LLMs may offer further improvements in execution speed; see Appendix A.3 for early results evaluating CRABS on such models.
Finally, in addition to serving as an effective neuro-symbolic technique for notebook under- standing, view CRABS as the foundation for experimental platform for investigating the information used, the skills employed, and the sub-problems that must be solved for any intelligence, artificial or natural, understand a Python notebook via its information flow graph and the corresponding cell execution dependency graph.
References
Michael Brachmann, William Spoth, Oliver Kennedy, Boris Glavic, Heiko Mueller, Sonia Castelo, Carlos Bautista, and Juliana Freire. Your notebook crumby enough, REPlace it. In 10th Annual Conference on Innovative Data Systems Research (CIDR 2020), 2020.
Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Wenjing Hu, Yuchen Mao, et al. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? Advances in Neural Information Processing Systems, 37:107703-107444 2024.
Lucas AMC Carvalho, Regina Wang, Yolanda Gil, and Daniel Garijo. Niw: Converting notebooks into workflows to capture dataflow and provenance. In K-CAP Workshops, pp- 12-16, 2017.
Wei Cheng, Yuhan Wu, and Wei Hu. Dataflow-guided retrieval augmentation for repository-level code completion. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulational
10

---

Published as a conference paper at COLM 2025
Linguistics (Volume 1: Long Papers), pp. 7957-7977, Bangkok, Thailand, August 2024.
Association for Computational Linguistics. doi: 10.18653 2tt/5////1222..aa-oogg.. URL http:://aclaanhhoog..orroo
Nur Nasuha Daud, Siti Hafizah Ab Hamid, Muntadher Saadoon, Firdaus Sahran, and Nor Badrul Anuar. Applications of link prediction in social networks: A review. Journal of Network and Computer Applications, 166:102716, 2020. ISSN 1084-8045. doi: https:
/ 2bbbioorg//...000///mmmm22222110
/ //doi.orr /tt///000011111/ccccc artiche/pp1/S114444522222 10276.. URL https: http:s://wwwwwww// /tww:/sciieneeereetttt.t. httpeee/ddreetcccoeeoeeoeeeeeeee /coonccienee/////
Jeremy Goecks, Anton Nekrutenko, James Taylor, and Galaxy Team team@ galaxyproject. org. Galaxy: comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome biology, 11:1-13, 2010.
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. Graphcodebert: Pre-training code representations with data flow. In International Conference on Learning Representations, 2021. URL https: ctctps://peerreieew..r meee/foorummd=jjoocee4433
Rajarshi Haldar and Julia Hockenmaier. Analyzing the Performance of Large Language Models on Code Summarization. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp- 995-1008, Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanthology/ org/2024. Irec-main.89) 89/.
Mohammad Hossin and Md Nasir Sulaiman. review on evaluation metrics for data classification evaluations. International journal of data mining & knowledge management process, 5(2):1, 2015.
Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin Clement, and Nan Duan. Execution-based evaluation for data science code generation models. In Eduard Dragut, Yunyao Li, Lucian Sopa,,looodai Vucetic, and Shashank Srivastava (eds.), Proceedings of the Fourth Workshop on Data Science with Human- in-thheLoop (Language Advances), pp. 28-36, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL https: /ttp::////cllnthooogyyy org/2022. 1.
Junjie Huang, Daya Guo, Chenglong Wang, Jiazhen Gu, Shuai Lu, Jeevana Priya Inala, Cong Yan, Jianfeng Gao, Nan Duan, and Michael R. Lyu. Contextualized data-wrangling code generation in computational notebooks. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE '24, pp. 1282-1294, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400712487. doi: 10.1145/3691620. 3695503. URL https::///ol 0rg/10. 1145/3691620. 3695503.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang. Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024. URL https://arxiv. wttpv/..rrg/bs//4555// 12186.
Thomas Kluyver, Benjamin Kelln- Fernando Pérez, Brian E. Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B. Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, Carol Willing, and Jupyter De- velopment Team. Jupyter notebooks publishing format for reproducible compu- tational workflows. In International Conference on Electronic Publishing, 2016. URL https://api. weeemntiischolllo wwwpolaarorg//opppuuu orggCCorpsID::: 36928806..
Mei Kobayashi and Koichi Takeda. Information retrieval on the web. ACM computing surveys (CSUR), 3242)114-177, 2000.
11

---

Published as a conference paper at COLM 2025
David Koop and Jay Patel. Dataflow notebooks: encoding and tracking dependencies of cells. In Proceedings of the 9th USENIX Conference on Theory and Practice of Provenance, TaPP'17, pp. 17, USA, June 2017. USENIX Association.
Johannes Köster and Sven Rahmann. Snakemake-a scalable bioinformatics workflow engine. Bioinformatics, 28(19):520--2522 08 2012. ISSN 1367-4803. doi: 10.1093/ bioinformattcc//bs4500 URL httpt: org/10. htt.t///reeeoooo.ooeo//////////r
Haau-Sing (Xiaocheng) Li, Mohsen Mesgar, André Martins, and Iryna Gurevych. Python code generation by asking clarification questions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 14287-14306, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653 (006653/v122233.00 /1l/202..cllloog... URL hatpps://aclaa http//ooccltooooyyy.oooo org/ wwwgg//223..22 acl- 799/.
Zhaoheng Li, Supawit Chockchowwat, Hanxi Fang, Ribhav Sahu, Sumay Thakurdesai, Kan- tanat Pridaphatrakun, and Yongjoo Park. Demonstration of elasticnotebook: Migrating live computational notebook states. In Companion of the 2024 International Conference on Management of Data, pp. 540-543, 2024.
Bertram Ludäscher, Ilkay Altintas, Chad Berkley, Dan Higgins, Efrat Jaeger, Matthew Jones, Edward A Lee, Jing Tao, and Yang Zhao. Scientific workflow management and the kepler system. Concurrency and computation: Practice and experience, 18(10):1039-1065, 2006.
Jordan Matelsky, Felipe Parodi, Tony Liu, Richard D Lange, and Konrad P Kording A large language model-assisted education tool to provide feedback on open-ended responses. arXiv preprint Xiv:2330 00839939,,, 2023.
Timothy McPhillips, Tianhong Song, Tyler Kolisnik, Steve Aulenbach, Khalid Belhajjame, R. Kyle Bocinsky, Yang Cao, James Cheney, Fernando Chirigati, Saumen Dey, Juliana Freire, Christopher Jones, James Hanken, Keith W. Kintigh, Timothy A. Kohler, David James A. Macklin, Paolo Missier, Mark Schildhauer, Christopher Schwalm, Yaxing Wei, Mark Bieda, and Bertram Ludascher. YesWorkflow: A User-Oriented, Language- Independent Tool for Recovering Workflow Information from Scripts. International Journal of Digital Curation, 10(1):298-313, May 2015. ISSN 1746-8256. doi: 10.2218/ /ii/d..v0l11.3.70....... wwwpeee.ppppjjjjjjeeeeeee hhtp/iidd/artiile/viee///////
URL
298. Number: 1.
Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. Using an 11m to help with code understanding. In Proceedings of the JEEF/ACM 46th International Conference on Software Engineering, ICSE '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400702174. doi: (0.1145/359755 3639187. URL httpp:::/dii.oggg////// 11211.0..1445 1145/3597533 3639187.
Tom Oinn, Matthew Addis, Justin Ferris, Darren Marvin, Martin Senger, Mark Greenwood, Tim Carver, Kevin Glover, Matthew R Pocock, Anil Wipat, el. Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics, 20(17):045-3354, 2004.
OpenAl. Hello gpt-4o, 2024a. URL httpps//apeaa..ccom//nneee/lllle Accessss 27 March 2025.
OpenAL. Hello gpt-4o mini, 2024b. URL https://openai. http:penai.commnnndeei. com/index:
gpt-4o-ming advancin-n cost-efficient -iteeeiiiiiee--neelleeeneee Accessed: 27 March 2025.
Joāo Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. A large-scale study about quality and reproducibility of jupyter notebooks. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pp. 507-517, 2019. doi: 10.1109/ /tt////.R.200000077777
Jim Plotts and Megan Risdal. Meta kaggle code, 2023. URL https:tt//// //www. http://wwwwwwwwccccc/ com/ds/ 3240808. Accessed: 27 March 2025.
12

---

Published as a conference paper at COLM 2025
Dhivyabharathi Ramasamy, Cristina Sarasua, Alberto Bacchelli, and Abraham Bernstein. Visualising data science workflows to support third-party notebook comprehension: an empirical study. Empirical Software Engineering, 28(3):58, March 2023. ISSN 1573-7616. doi: 10.1007 s10664-0233333 10289-9. URL httpp:://dn org/10..///.. wwtgg/...00/////./ 00000//11664--22-1222939
Megan Risdal and Timo Bozsolik. Meta kaggle, 2022. URL https: http://wwwwkagleeooooo Accessed: 27 March 2025.
Yingssu Tsai, Scott E McPhillips, Ana González, Timothy M McPhillips, Daniel Zinn, Aina E Cohen, Michael Feese, David Bushnell, Theresa Tiefenbrunn, CDavid Stout, et al. Autodrug: fully automated macromolecular crystallography workflows for fragment- based drug discovery. Acla Crystallographica Section D: Biological Crystallography, 69(5): 796-803, 2013.
Chengpeng Wang, Wuqi Zhang, Zian Su, Xiangzhe Xu, Xiaoheng Xie, and Xiangyu Zhang. LLMDFA: Analyzing Dataflow in Code with Large Language Models. In A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, Tomczak, and C. Zhang (eds.), Ad- vances in Neural Information Processing Systems, volume 37, pp. 131545-131574. Curran Associates, Inc., 2024. URL https::/proceeedini neuripp neurips cc/paper htce/ppprrfflseeeee pdf.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-Thoughh Prompting Elicits Reasoning in Large Language Models. Advances in Neural Informatioon Processing Systems, 35:24824-2444 24837, December 2022. URL hwwph:tpp///eeee /RR.nttts///rroceednn.nnnn http:p/rweeeeiinneeerris.s...//e wwtppeerrrrpppppppeaeeeeeepeeeee fataapprriileeepppprrreeee p20pp//ppr//222aaaaa 145699/332244441ff1aa6771111-6-0
htmll. html.
John Wenskovitch, Jian Zhao, Scott Carter, Matthew Cooper, and Chris North. Albireo: An Interactive Tool for Visually Summarizing Computational Notebook Structure. In 2019 IEEE Visualization in Data Science (VDS), pp. 1-10, Vancouver, BC, Canada, October 2019. IEEE. ISBN 978-1-72815-047-5. doi: 10.1109 /VDS/8997..009989777777 //ieeexplore.c ieee. wttp///ccmme///9733330008
URL https:
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL
org/ abs/ 2505.09388.
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural Language to Code Generation in Interactive Data Science Note- books. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 126-173, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653
wttp:/33//112222lllll//// 222/2.2.22.lloog..99 URL https: / htttp:://allattholgge org. /ttrg/2222..rrm http:/2222....1oo..... 9/.
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian- Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 2471-2484, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 vv/l vllvv122.e2mmm.. http:///2222emmmmmmii.i...5.5 URL / //cl://thllaoaoooyyyoyoo org/o02g/22 10023.emllp... manlp-main. 151/.
13

---

Published as a conference paper at COLM 2025
A
Appendix
A.1 Illustrative Python notebooks
A.1.1 Notebook with information flow dependent on input data
The notebook illustrated in Figure 7 exhibits different patterns of information flow de- pending on the input dataset. Because line in cell will be executed only when the pmone_recore csv file has more than 100 rows, information flows from cell 2 to cell only for runs on such input files.
# cell
2 import pandas as pd phone_data pd. re
# cell 2
if phone_data aroon_data... shape[0] 100: phone_data = mhone_data sample (n=100)
# cell
phone_data head()
Figure 7
A.1.2 Notebook with clashing variable and function names
Figure 8 shows an example case that violates the assumption of distinct data and code names. Cell 1 defines variable named add _on,, cell 2 reuses the variable name function name, and cell 3 calls the function. Our system cannot identify that the add_one variable is data output candidate of cell 1 and data input of cell resulting in further failure to extract the data flow (cell 1, cell 2, add_one) and the cell execution dependency (cell 2, cell 1).
# cell add_one =
# cell
add_one = add_one +
# Reassign the variable name to function
def add_one(v):
return +
# cell add_oned
Figure
A.1.3 Global variable accessed in a function
Cell the notebook in Figure 9 defines a global variable data-file .path and cell accesses it directly from the load_data function. Our system cannot identify that data.file.path is data input of cell 2, resulting in a further failure to recognize the information flow (cell 1, cell 2, data file path) and the cell execution dependency (cell 2, cell 1).
14

---

Published as a conference paper at COLM 2025
# cell
Sata_file_path "data. csv"
# cell
import pandas pd def load_
data pd.
return data
data baad_data()
Figure 9
A.1.4 Notebook with non-Python statements
Cell 1 in Figure 10 uses the XXcapture magic command to capture the standard output of the code cell, storing it as a variable named captured. stduut that is later used by cell Our system skips cell due to the magic command, and fails to extract the information flow (cell 1, cell 2, captured stdoutt and the cell execution dependency (cell 2, cell 1).
# cell
xXcapture captured_stdout out =
print(out + 1)
# cell captured_stdout, show()
Figure 10
A.1.5 Conditional statements in a loop
Cell 1 in Figure 11 defines count and cell 2 has an if statement inside a loop, assigning to count on the first iteration and incrementing count by one during each of the later two iterations. While the lower estimate considers the intersection of all possible branch execution orders, finding cell and cell 2 to be independent; the upper estimate considers the union of all possible branch execution orders, finding the information flow (cell1, cell2, count) and cell execution dependency (cell 2, cell 1).
# cell count
# cell for in range(3):
if i == 0: count
else:
count count +
Figure 11
A.2 Execution time analysis
Figure 12 shows that, when using the sequential implementation of CRABS, the execution time for notebook, defined the total of the latencies of the individual requests involved in analyzing the notebook, is proportional to the total number of ambiguities. The ordinary least squares (OLS) linear regression yields slope of 0.94 seconds per ambiguity and R² of 0.96. In contrast, the OLS fit of the baseline execution time against the number of
15

---

Published as a conference paper at COLM 2025
Execution Time Number of ambiguities
Baseline R 0.539 CRABS R' 0.961
120
100
Execution Time (seconds)
40
CRABS
Baseline
60
100
Number ambiguities
Figure 12: Scatter plot of execution time vs. number of ambiguities per notebook with least squares lines.
ambiguities is substantially weaker with R² of 0.54, though its slope of 0.22 seconds per ambiguity remains informative. The intersection point of these two regression lines supports the observation that CRABS is generally up to 5 times faster than the baseline for notebooks with fewer than 14 ambiguities, and as much as 7 times slower than the baseline when there are more ambiguities. A separate regression against the total number of information flows in a notebook (Figure 13) better predicts the baseline execution time with an R² of 0.76. For the 4 notebooks containing no ambiguities (i.e., for each cell in these notebooks the parser-generated lower and upper estimate the inter-cell T/O sets are identical), CRABS is approximately 3 orders of magnitude faster than the baseline because no prompting of an LLM is necessary; these notebooks were excluded from the linear regressions discussed above.
While the sequential implementation of CRABS resolves exactly one ambiguity per request in sequential manner, the concurrent implementation sends requests to an LLM without waiting for prior responses and achieves performance improvements ranging from 4 to 24 times faster than the baseline ranging from 0.6 to 8.3 seconds.
per-notebook basis, with per-notebook execution times
A.3 Effectiveness of CRABS LLMs other than GPT-4o
Table 4 validates the effectiveness of CRABS smaller models, including open- and closed- source, general-purpose and code-focused language models-GPT-44 mini (OpenAI, 2024b), Qwen3-8B (Yang et al., 2025), and Swven25--oode-77-IInstru (Hui et al., 2024). We observe that smaller models face greater long-context challenges. For example, the GPT-4o mini model fails to understand 25 notebooks under the baseline setting and 20 notebooks without cell-by-cell prompting, returning the wrong number of cells and resulting zero values for all metrics in each of these notebooks.
16

---

Published as a conference paper at COLM 2025
Execution Time Number information flows
CRABS Baseline
Baseline R' 0.764 CRABS R 0.777
120
100
Execution Timee (secands)
20
40
140
Number of information flows
Figure 13: Scatter plot of execution time vs, number of information flows per notebook with least squares lines.
Information Flows (%)
LLM
Method
Prec.
Rec.
F1
Acc.
EM
GPT-4o-mini
baseline
43.19
40.65
41.53
37.40
12
(2024-07-18)
CRABS
83.01
82.87
82.94
74.85
syntacii w/o cell-by-cod prompting
w/o syntactic phase
71.58
65.16
67.73
55.09
10
54.69
53.57
54.01
50.55
28
Qwen3-8B
baseline
45.75
38.92
41.36
34.77
6
CRABS
75.04
73.75
74.31
63.48
22
w/o syntactic phase
87.56
45.77
57.21
43.78
8
w/o bell- prompting
37.07
36.86
36.95
32.29
Owen2.5-Coder (7B-Instruct)
baseline CRABS
19.47
13.87
15.80
12.64
2
77.13
76.01
76.52
66.21
w/o syntactic phase
64.90
56.64
59.81
47.24
6
w/o cell-by-cell prompting
35.55
33.49
34.36
31.71
16
Table 4: CRABS results for LLMs other than GPT-4o.
17

---

Published as a conference paper at COLM 2025
A.4 Additional figures and tables
Given Python Jupyter Notebook, please extract input output variables, custon defined code (including functions and classes), referred code (including functions and classes) of each code cell and return a JSON string which is a list of dictionaries. Each dictionary has "inputs", "outputs", "defines_code" "refers_code keys. Each dictionary is code cell in the notebook. All modules, builtin function names should be ignored. The order of these dictionaries in the List should be the same as the order the code cells the notebook.
and
Here are the steps:
Step 1: extract the inputs and output candidates of each code cell. An input of cell is variable that is used in the cell. Usually it is not defined in current cell, but some previous cells.
exception could a variable that defined in the current cell but used before it is defined. An output candidate of cell is a variable that is defined, updated, or mutated in the cell.
Important
for output candidates:
1. Method Calls (object.methhdd) or Function Calls (funetion(object))
me thed (or a function) modifies the object in place (e.g., updating data, changing the structure, or adjusting properties in place), the object is an output candidate.
If met (or a function) does not modify the object (e.g., retrieving data, describing data, visualizing data, accessing properties), the object is output candidate.
method (or a function) returns new object (e.g., creating copy) without modifying the original, the original object is not output candidate.
2. Reassignment (variable = ...)
If variable is assigned new value, is an output candidate.
3. Shared References (Aliased Variables)
- If multiple variables reference the same mutable object (e.g., through assignment or being stored inside a data structure), modifying the object place through one reference
references to that object output candidates.
Modifying the container in place makes only the container an output candidate, but not its elements.
Modifying an element inside a container in place makes both the container and the modified elenent output candidates.
Operations like retrieving data, describing data, visualizing data, accessing properties, and creating copy not considered modifying the object in place.
Step 2: extract outputs by filtering output candidates which not used in any subsequent cells.
classes "defines_code" used functions and classes as "refers_ce _cde"" to each cell. function or class is considered custom defined if it is defined the cell. function or class considered used it is called in the cell. Note that modules and builtin function should included.
Step 3: add custom defined functions
Step 4: return JSON string.
Given Python Jupyter Notebook: [omitted due to space limitations]... Cell-3:
python
[en cattone__dtt... query (iaphone_dataa.uurr(eee)) / Len(phone)
[omitted due space limitations]
Think it step by step and return the JSON string as described above. Do not include any code cell content the answer.
Figure 14: An example baseline prompt (in part). The real prompt includes all code cells.
18

---

Published as a conference paper at COLM 2025
NotebookID
C1
C2
C
x
kanncaal httpaa/llllcllccceeeettttooooeer
X
V
X
htwp://eeemiiiiiniiiiiiiiiiiiiie stacking-in-pyttinn
X
dansbokerr/how-moddl-wvo
X
hlowadd//ugyter-notbboo----
X
jhoward/gettinngg
colinmorris/fune colnmorris/funciiossnn httss//unnnooonnnnntttttii -eett
thebrownviking20/ htwprriinnnneeeeeiiiiinnngn
kanncaal / sateep--lamming-tttrrreee cetorial-for-beginneee
X
ing moruuriistt-mcccie--eeee learning-
shivamb shivamb//data--cinnee -attaa-ieeeeneeelllssrr
kanncaal /wwhmmma-aaciiiinnee www.ehhiin--laannnnnttttrrraa-aa
gzuidhof pruplppprpppooessiggttttoraall
therealcyberlord,
ibtesama/ www.g//wwww...i.iieiiooeeeeeeeee
ybifoundation/ wwwpd//ddddddnnngrrroooooooooa/a
residentmario /raaraatin--raddi---g-ddd-----ie
x
kanncaal
colinmorris/lists
ybifoundation/
dansbecker haa.eeeeeeeeeeiicccaaaeeeppooooo
masumrumi masumruui/a--tatiitenn
anokas/ hhanules/baiicddd--ceeiinnnnnno
chaataao/utoriillon-reaaaaa larg----attl
vbmokin
X
X
jhoward /howard//linar-moddee /tt.e//i--mddddeeeeeeeeeeeaentee
ybifoundattion/encodingg
X
dansbeccerr/exxlloreyyoorre
colnmmorti//mmmn---ddddddoooonnt
bebiendniie/csstomrrssgeetttt
X X
muassineeen/cooidd---llerrtttre
X
kanncaal Ganncaal/seabornn wwwecaabootutooorrcoooooe cottorriallfo--bginneet artgor/ eda-and- modelss
Table 5: Reasons for discarding a top voted notebook. These notebooks can be accessed via https:/ //www/ kaggle. l. com/ com/code// [NotebookII)) on Feb. 25, 2025. C1, C2, C3 are criteria 1, 2, and 3. "/" is used when a criterion met; "x" used when a criterion is violated.
19

---

Published as a conference paper at COLM 2025
ID
NotebookID
L
F
D
01
alexisbcook/ titanic-tutoria
6
02
pmarcelino/comprehen hwtp:////w....i...aa..oaaoaaoaar
32
127
32
108
03
startupsci/
52
256
133
576
04
65
332
91
773
05 vesstnghoozza//mmddddddtoooororr
213
35
118
06
1111
103
146
07
63
928
153
442
08
62
666
94
424
09
43
285
105
317
ash316 www-eed---ddecceeeee
358
117
461
11
yassineghouzam/ toppaa/ttttiiiiott-ttt-o-
76
439
113
922
12
dansbecker/ dansbeeker//ouu-fisssmmmmcii yourrffssttmaccii-eelmmming
9
29
10
25
13
ybifoundation/staan htt.atitoonntttsssllssiccattooo
20
38
22
57
14
mansbeckerrmodel-valleee
3
15
abhishek hatpleckkr//meerfiittinnn-----oo
566
137
187
16
3
31
17
46
644
84
494
18
212
40
205
19
aarssayah/stocc-mrkkke--ee http:///ww/oaaaaaaaaaaaaaaaarrrn
282
59
20
benhamner raahammeer/yytoonnddt--vvvuuuiiu
15
78
14
14
21
kaurccll/eeuurrreeeoooooddddaava ts
29
205
47
79
22
jagangupta/timent hta.ap/aaaammaeesssss-aaaeeeeree
40
369
35
85
23
gusthema/ http:///mmmmooteeersssssrrrrrrts
24
100
23
81
24
saapiu/regullrizze-llieeenn
41
131
25
dansbecker/rane saabeckkrrrandom--fore forests
2
27
26
dansbecker/sule submitttr ceatinn--fo-------e from-a-kernel
3
27
4
27
madintamer/titanni wwt.aaaiiissuivivlaaaddiiiiitiii
41
286
96
369
28
23
61
29
kauccal//pooty-uuuonnoooooggggn
21
439
18
22
30
16
49
12
15
31 resieettorno//neeennissaccitties
22
26
21
22
32 bifoundttionnisss-eeegggprriroot
20
31
24
57
33
bbfeunddiion/cmmeeppreeccconn
31
26
34 bblooundtion/chaane---aaiisstttt
21
31
26
64
35
jblfoundtiionmmullll-l--------ss
21
34
26
64
36
onarelaaryyarouuuryy-hoouugggh
20
289
31
93
37 bpfouuddtioo/puuuhisseeeeetooood
35
26
65
38
dansbecker paanbeekker/andddiggmmiinnnnn-i
5
17
39
ybifoundation/ wettoo///eedtttrrd-eee credit-card-defaull
22
33
27
65
40
ybifoundation/tunn ieaattiinnntttriialaaaa
29
73
42
148
41
deffro/edat edd-iss-un
40
168
46
70
42
gpreda/ spwcch/snttandeeeddaaaaa /wtp///anndeccd--ddd--eeedeeddde
329
111
160
43
39
79
37
125
44
27
188
37
42
45
henjee/ttianii-ppgstteeeaappppp
58
390
115
381
46
123
30
82
47
157
31
48
36
153
53
127
49
ybifoundation/binarta soaiudaattinbiinayyddaaiifttttti
21
31
26
64
50
ybifoundation/ mwt.t/iicatteorrycassiaiiiccacee
21
31
26
64
Table 6: Individual notebook statistics. These notebooks can be accessed via https://www. kaggle. com/code (oteeeNNooebbooO)) on Feb. 25, 2025. The C column denotes number of code cells, the column denotes number of lines, the F column denotes number of information flows, and the D column denotes number of transitive dependencies.
20

---

Published as a conference paper at COLM 2025
Given a Python program block, determine if an object is an input. An input is variable that is used the program block but not defined within it, or variables used before being reassigned.
Important Cases:
1. Conditional Statement Within Loop
A variable may be initialized during an early iteration of loop and then utilized in subsequent iterations. In such cases, the variable is not considered an input,
its value originates from the loop's execution rather than external sources.
2. Shared References (Aliased Variables)
If multiple variables reference the same object (e.g., through assignment or being stored inside data structure), and of then is input, then all variables referring to that object are also inputs.
container is an input, then elements inside the container are also inputs.
If an element inside a container is an input, then the container itself is also an input.
Preceding context (for shared reference):
python
datasets [train, test]
Question:
In the following Python progran block, is "train" input? Return the final answer in the form "Yes(No), "[x}" is (not) an input" where "(x)" is the actual name of the variable.
python
for dataset in datasets:
Answer:
Figure 15: An example CRABS prompt for resolving ambiguous inputs
Information Flows
Method
(rec.(%)
Rec.(%)
(%
Acc.(%)
EM(%%
CRABS w syntactic phase
98.59
98.52
98.56
97.30
95.28
89.49
91.18
86.49
w cell-by-clll prompting
92.19
89.92
90.79
85.39
40
Table 7: Information flows: ablation study results for 45 selected notebooks.
Transitive Dependencies
Method
Prec.(%)
Rec.(%)
F,(%)
Acc.(%)
EM(%)
CRABS
99,42
99.95
99.67
99.37
82
w/o syntactic phase
95.80
87.88
89.20
84.13
42
w/o cell-y-cell prompting
89.96
77.40
82.22
77.37
Table Transitive dependencies: ablation study results for 50 notebooks.
Transitive Dependencies
Method
Prec.(%)
Rec.(%)
(16%)
Acc.(%)
(M(%))
CRABS
99.36
99.94
99.64
99.30
w w/o syntactic phase
95.60
88.50
89.31
84.49
44
w /o cell-by-cell prompting
99.96
86.00
91.36
85,97
40
Table 9: Transitive dependencies: ablation study results for 45 selected notebooks.
21

---

Published as a conference paper at COLM 2025
Given a Python progran block, determine if object an output candidate. output candidate is variable that is defined, updated, mutated in the program block.
Important Cases:
1. Method Calls bbbject.methodee or Function Calls (function(object)
If method (or function) modifies object place (e.g., updating data, changing the structure, adjusting properties in place), the object is output candidate.
If method (or a function) does not nodify the object (e.g., retrieving data, describing data, visualizing data, accessing properties), the object is not output candidate.
If method (or function) returns new object (e.g., creating copy) without modifying the original, the original object is not output candidate.
2. Reassignment (variable ...)
If variable is assigned new value, is an output candidate.
3. Shared References (Aliased Variables)
If multiple variables reference the same mutable object (e.g., through assignment or being stored inside data structure), modifying the object in place through one reference makes all references to that object output candidates. Modifying the container in place makes only the container an output candidate, but not its elements.
Modifying element inside container in place nakes both the container and the modified element output candidates.
Operations like retrieving data, describing data, visualizing data, accessing properties, and creating copy not considered modifying the object in place.
Preceding context (for shared reference):
python datasets [train, test]
Question:
In the following Python program block, is "datasets" output candidate? Return the final answer in the form "Yes(No), "(x)" is (not) an output candidate" where fx?" the actual name of the variable. python
for dataset in datasets:
droppa (inplace=rre)))
Answer:
Figure 16: An example CRABS prompt for resolving ambiguous output candidates
22

---

Published as a conference paper at COLM 2025
Given Python program block, return JSON string with "inputs", "output candidates" "defines_codd" Da not include any nanes built-in functions "inputs" variables that used in the program block but not defined within it, or variables used before being reassigned. Besides shared references, one important case about conditional statement within loop, i.e., when variable loop
"refers_code" and reffrnceeeferencee keys.
variables.
is initialized during early iteration iterations. In such cases, the variable is not considered an input, as its value originates from the laop's execution rather than external sources. "'outuutccandidates
then utilized in subsequent
variables that are defined, updated, program block. Besides shared references, there are two important cases:
mutated in the
1. Nethod Calls "object.methone ()) or Function Calls (Fbnetionubbjet)) place (e.g., updating
If method (or function) modifies the object data, changing the structure, or adjusting properties in place), the object is an output candidate.
If method (or function) does not modify the object (e.g., retrieving data, describing data, visualizing data, or accessing properties), the object is not an output candidate.
object (e.g., creating copy) without modifying the original, the original object is not an output candidate.
IIf me thod (or function) returns
2. Reassignment (variable ...)
value, it is output candidate. -"defines_ce "coee" is list of functions or classes defined in the progran block. "refers_code" is list of dser-defined functions or classes used in the program block, selected from given list.
If variable is assigned
is a string, which contains descriptions about the shared references (aliased variables) based on previeus context and current program block. If there
ceniared_refeenneee
shared references, return "NA". Otherwise, it explicitly
include:
How multiple variables reference the same object (e.g., through assignment or being stored inside data structure).
The relationship in both directions, meaning if one variable is modified, how it affects the others and vice versa.
Previous context:
python
The variables 'train' and 'test' stored inside the list 'datasets', 'train' or 'test' is modified, the change will be reflected in 'datasets' as well, and vice versa, since 'datasets' holds references to these objects.
Current progran block: python
for dataset in datasets:
dataset.drapne dropna
From the following given list, select only the functions and classes that are used and store them "refers_code" Given functions and classes: ()
JSON string: (Think it step by step, then return the final results only)
Figure 17: An example prompt for the ablation study S1 (i.e., w/o syntactic phase).
23

---

1. Compute ambiguous inputs by excluding definite inputs from possible inputs. Published as a conference paper at COLM 2025
Given a Python Jupyter Notebook with definite and possible annotations, extract the input and output variables for each code cell and return JSON string.
Each dictionary in the JSON list represents a code cell and contains: "inputs": List of input variables.
"outputs": List of output variables.
Rules for Extraction:
Step 1: Include definite inputs and outputs
1. Add all definite inputs to the final inputs list.
2. Add all definite outputs to the final outputs list.
Step 2: Process anbiguous inputs
2. Add an anbiguous input only if follows these rules: It is used in the code cell but not defined within it.
It used before reassignment in the same cell.
Special Cases:
Conditional Statement Within Loop: If a variable is initialized during an early iteration of a loop and then utilized in subsequent iterations, it is not an input. Aliased/ shared references:
If multiple variables reference the same object (directly or via container), all variables referring to that object are also inputs.
If container
input, then all elements inside the container also
inputs.
If an element inside a container is an input, then the container itself is also input.
Step 3: Process anbiguous outputs
1. Compute ambiguous outputs by excluding definite outputs from possible outputs. Add an anbiguous output only if it follows these rules:
-It is defined, updated, or mutated within the code cell. Special Cases:
Method Calls & Functions: If function or method modifies an object in place, the object an output.
Reassignment: If variable is reassigned, it is output.
ALiased/shares references:
If a nutable object is modified in place, all references to it should be outputs.
Modifying the container in place nakes only the container an output candidate, but not its elements. Modifying element inside container place makes both the container the modified element output candidates.
4: Filter outputs Remove output candidates that are not used as inputs in subsequent code cell.
Step 5: Return the JSDN Output
Fornat the final List as JSON string. Maintain the same order of code cells as in the notebook.
Given a Python Jupyter Notebook with annotations:
... ...[oteeeed due to space Linitations].
Cell-3:
python
definite inputs: [ec, phonel; outputs:
# possible inputs: [ec, phone]; outputs: [ec, phone]
(gg(phooneqqery(ee)) Len(phone)
... Conitted due to space Linitations]...
Think it step by step and return the JSON string as described above. Do not include any code cell content in the answer
Figure 18: Part of example prompt for the ablation study S2 (i.e., w/o cell-by-cell prompting). The real prompt includes all code cells.
24

---

Published as a conference paper at COLM 2025
Information Flows (%)
Transitive Dependencies (%)
ID
baseline
CRABS
S1
S2
baseline
CRABS
S1
S2
01
100
100
100
100
100
100
100
100
02
0
100
100
100
0
100
100
100
03
64.25
96.58
57.29
60.34
67.59
99.91
32.80
58.13
0
97.24
94.51
100
78.84
95.55
05
91.43
97.14
85.71
91.43
83.17
100
85.33
83.17
06 59.74
94.17
65.03
61.11
92.65
98.65
58.25
81.30
07 90.34
93.46
98.00
94.77
91.67
97.08
94.64
92.06
08
0
91.49
92.82
96.81
99.30
96.21
99.88
09
0
94.29
94.58
89.52
98.14
89.74
83.03
10
0
100
95.54
0
0
100
72.65
0
11
0
100
100
0
0
100
100
0
12
100
90
100
100
100
100
100
100
97.67
95.45
100
100
99.12
96.36
100
100
100
100
100
100
100
100
15
0
98.54
97.42
100
0
99.47
95.41
100
16
100
100
100
100
100
100
100
100
17
43.24
93.41
83.44
45.78
100
98.87
0
18
0
90
87.32
100
99.03
82.18
100
19
73,79
100
68.09
70.21
97.14
100
88.66
95.15
20
100
100
100
100
100
100
100
100
21
98.92
93.62
96.70
97.87
100
92.40
100
98.06
87.10
95.65
90.32
94.74
23
73,91
100
100
73.91
66.12
100
100
66. 12
24
89.16
100
95.12
100
91.23
100
89.29
100
25
100
100
100
100
100
100
100
100
26
66.67
100
85.71
100
100
100
100
100
94.62
99.48
91.01
92.55
99.73
100
40.95
90.50
28
86.96
100
100
86.96
85.98
100
100
85.98
29
100
100
100
100
100
100
100
100
30
100
100
59.26
100
100
100
75.00
100
66.67
100
84.62
87
100
100
87.50
86.00
100
100
86.00
33
88.46
100
98.04
88.46
86.73
100
99.21
86.73
34
88.46
100
100
88.46
86.73
100
100
86.73
35
88.46
100
100
88.46
86.73
100
100
86.73
82.86
93.55
100
100
85.19
95.51
37
100
100
100
88.46
100
100
100
87.93
38
90.32
100
100
100
100
100
100
100
39
88.89
100
100
88.89
86.96
100
100
86.96
78.57
97.56
78.57
79.67
76.15
79.67
41
0
100
100
100
100
100
100
42
70.87
100
83.78
90.09
88.50
100
82.35
97.11
43
98.63
100
83.33
100
100
74.40
0
44
100
100
98.63
100
100
100
98.80
100
0
93.45
100
87.40
46
69.39
100
96.77
93.33
59.83
100
94.25
87.67
47
60.38
100
27.78
26.67
100
100
13.33
40
48
98.11
96.23
92.45
90.57
99.22
99.61
93.04
91.45
88.46
100
88.46
86.73
100
100
86.73
50
88,46
100
96.00
88.46
86.73
100
98.41
86,73
Table 10: Detailed results for notebook understanding task. The S1 and S2 columns denote ablation study S1 (i.e., w/o syntactic phase) and ablation study S2 (i.e., w/o cell-by-celll prompting). Values indicate the F2 score of notebook, where the notebook can be referred to via the ID column and Table 6.
25


Reviews:
[{"review_id": "GN7affE02B", "reviewer": "Reviewer_dByL", "summary": "This paper introduces a new approach, CRABS (Capture and Resolve Assisted Bounding Strategy), for understanding the intricate information flows and operational dependencies within Python notebooks. It proposes a hybrid syntactic-semantic methodology that combines Abstract Syntax Tree (AST) parsing with Large Language Models (LLMs) to construct information flow graphs and cell execution dependency graphs. The paper also contributes an annotated benchmark dataset, comprising 50 highly-upvoted Kaggle notebooks, for evaluating notebook understanding.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed CRABS method effectively reduces hallucination in notebook understanding.\n- The creation and annotation of a dataset of 50 real-world Kaggle notebooks provide a valuable resource for future research in notebook understanding and LLM application in code analysis.\n- The writing is pretty clear and well-organized, offering rich context and technical details.", "reasons_to_reject": "1. **Lack of Problem Significance and Practicality:** While the paper articulates the challenges of re-execution due to data and software dependencies, the fundamental importance of deriving information flow *without execution* could be more thoroughly substantiated. For example, in many practical scenarios, especially during development or debugging, direct execution or a robust execution environment might offer a more straightforward and reliable approach for understanding notebook behavior. The paper's argument for avoiding re-execution, while valid, may not universally outweigh the benefits of direct execution for comprehensive analysis. While safety concerns could strongly favor static analysis over execution, this aspect is not prominently highlighted in the Kaggle-based benchmark. Furthermore, the very high success rate (98% F1) of the current method suggests the benchmark might not be challenging enough for evaluating new LLMs. It would be beneficial to include more challenging and compelling examples, both in the benchmark and within the paper's discussion, to better demonstrate the problem's significance.\n\n2. **Lack of Novelty in The Method**: The core method, which combines AST (min and max I/O) parsers and LLMs, while effective, represents a common practice in prompt engineering, leveraging symbolic tools alongside neural LLMs, e.g., Dater (Ye et al., 2023), Binder (Cheng et al, 2023), Maieutic Prompting (Jung 2022). The approach, though functional, might be considered a straightforward application of current capabilities rather than a significant leap in technological contribution.", "questions_to_authors": "1. Given the high performance on the benchmark, I am curious whether advanced reasoning models like OpenAI o1/o3,  Gemini-Pro, and Deepseek-R1 could solve this completely without syntactic prompting.\n2. There are existing code execution (understanding) benchmarks like CRUX-Eval (Gu, 2024) and Code I/O (Li, 2025). While not identical, they share some common spirits. Could a comparison with these in the related works section be considered?", "extra_scores": {}}, {"review_id": "HGH0KtNksB", "reviewer": "Reviewer_rU2s", "summary": "The paper proposes a notebook understanding strategy that models information flow and transitive dependencies. The paper motivates the problem by stating an interesting challenge of understanding api calls in Python notebooks, specifically invocations that may or may not change the state of variables and data contained within the notebook, for which the code is not present in the notebook but rather defined in a library that is imported. The technique relies on leveraging LLM to understand which variables are being mutated without actually executing the code within the notebook. The proposed technique (CRABS) sets out to under-approximate and over-approximate the input and outputs to each Python notebook data cell and then uses an LLM to resolve the ambiguities between the parsing strategies. Finally the paper emperically shows the the CRABS strategy can improve the performance of gpt-4-o. The authors also manually provide a dataset with 50 annotated notebooks for the evaluation set.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors motivate the problem well.\n2. The authors empirically validate the long context challenge in understanding notebooks. \n3. The information flow analysis for Python notebooks is an interesting approach.", "reasons_to_reject": "1. There is a strong assumption that global variables cannot be used in the notebooks. \n2. Further, there is an assumption that a given notebook only connects a single given workflow, whereas in real situations, there might be multiple workflows that a given user might need to perform that would contain numerous disjoint graphs. \n3. The hidden modification of datasets, as pointed out by the authors, is a severe limitation of the technique as they do not truly capture information flow, but rather only the assignment flow, and hence, reasoning, the technique can only reason in cases where there are direct assignments. This makes the technique brittle and might cause hallucinations in the LLM output as well. \n4. Also the authors claim that static analysis tools might not recognise output variables in conditional statements, but do not provide evidence that the LLM can successfully (based on the MinIOParser and MaxIOParser) make informed decisions during such cases.\n5. It would be interesting to test this strategy not just on GPT-4-o but also other open (Qwen, Llama) as well as closed source(Gemini, Claude) general and code LM (QwenCoder, CodeLlama) to validate the the technique generalises.", "questions_to_authors": "1. Could you please provide insights into the other cases that fail (apart from the 4 where the MinIOParser and MaxIOParser results match) where the model fails given that even though the LLM resolves 98% ambiguities correctly, the Exact match number over the dataset is only 74% ? Does that mean that there are inherent limitations in the MinIOParser / MaxIOParser -- that model the information flow graph wrong/differently despite the LLM performing well on disambiguating ? \n\nPlease refer to the weaknesses.", "extra_scores": {}}, {"review_id": "ixVJyxSKCV", "reviewer": "Reviewer_TcJh", "summary": "This paper describes CRABS (Capture and Resolve Assisted Bounding Strategy), an algorithm that parses the information flow and execution dependencies between code cells in Python Notebooks based on AST parser and zero-shot prompting of LLMs. The authors manually annotated the 50 high-quality and highly-up-voted public notebooks and use the dataset as the basis for evaluation of the CRABS algorithm and comparison with baselines. CRABS is a hybrid approach that combines deterministic, explainable ASR parsing with LLM's pattern understanding capabilities. It shows significantly improved parsing accuracy compared to the baselines, as demonstrated by using the precision/recall/F1 metrics for data dependency edges and the overall exact-match (EM) rate. \n\nThe paper is well written and easy to read and follow. It clearly explains the background and importance of this problem (dependency parsing in Python notebooks) and why it is a nontrivial problem, i.e., challenges such as unclear side effect of Python function calls and hidden modifications of values within data structures such as Python lists. These challenges render the problem beyond the reach of traditional, non-ML algorithms, at least without the execution of the code. The authors also explained the shortcomings of previous effort at tackling this problem using LLMs, including the tendency to hallucinate nonexistent information flow and dependencies. \n\nThe originality of the authors' approach (CRABS) is apparent but not to the degree of groundbreaking. The CRABS strategy can be briefly summarized as \"divide and conquer\", i.e., augmenting AST analysis with prompted LLMs at the level of individual code cells and individual variables and methods, instead of relying solely on an LLM to analyze the notebook as a whole. This hybrid approach achieves clear advantage over the LLM-only baseline as well as over the AST-only baseline, which is a significant contribution.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths of this paper that argue for acceptance.\n\nS1. This paper focuses on an important problem in AI-assisted code analysis that has important practical applications in data science and other applicable domains of Python Notebooks. \n\nS2. The authors constructed a dataset with manually-annotated ground truth to enable evaluation of the CRABS algorithm and the contrasting baselines. The authors selected a set of clear and informative evaluation metrics, including precision/recall/F1 values for information flow between code cells and the EM rate on the overall notebook.\n\nS3. A moderately innovative algorithm that combines the strengths of syntactic parsing (AST) and LLM (gpt-4o as used by the authors), which achieved the highest accuracy among all tested solutions. An overall EM rate of 74% was achieved on the information flow graph; 84% for the transitive dependency. The F1 scores on the cell-wise information flow were shown to be at 97% or higher.\n\nS4. The authors performed ablation studies to demonstrate and quantify the contribution of different components of the CRABS algorithm: AST parser and cell-by-cell prompting.", "reasons_to_reject": "Weaknesses of the paper that argue against acceptance.\n\nW1. The dataset that the author constructed is relatively small in size (50 notebooks) and skewed towards the high-quality end of the space of all possible Python notebooks. For practical applications, many users (such as typical data scientists and ML engineers) will perform dependency analyses on their own notebooks, which may have lower code quality and messier (more complex) dependencies compared to the high-quality ones. This casts some doubt on the generalizability of the results to \"average quality\" notebooks. \n\nW2. The authors did not characterize the manual annotation process in several aspects: a) whether the annotation result is unique. For instance, whether multiple code cells can each be considered the execution dependency of a downstream code cell. In those cases, how did the annotation result accommodate this possibility? b) The reliability of the manual annotation process, e.g., whether some of the notebooks were annotated by multiple human annotators and the agreements among them quantified. c) Are the annotation results be made available so that future studies can build on the dataset and compare their results with the current paper?\n\nW3. The authors relied solely on zero-shot prompting in CRABS. It is possible that using few-shot examples can further improve the accuracy of CRABS, but the authors did not mention or discuss that in the paper. The same goes for the potential approach of LoRA tuning the foundational model on a reasonable amount of annotated data.\n\nW4. This paper does not report the computational cost of CRABS and how it compares with the baseline methods including AST (which is expected to be very small in comparison) and the whole-notebook prompting. This question is especially relevant given the large number of code cells (30 on average) and information flow (49 on average) within each notebook according to Table 1. For practical application of such a system, the users of the Python notebooks would be editing and running the notebook in real time. Therefore the latency matters.", "questions_to_authors": "Q1. See my weakness point W2 above: are the annotation of the dependency graph of a notebook unique, given the possibility of multiple parallel upstream dependencies of a given code cell? If not, how was that possibility accommodated? Were all the options in the parallel set of dependencies considered dependencies? \n\nQ2: Also related to W2 above: How did the authors ensure the quality of the manual annotation? Was there any measurement of inter-rater reliability?", "extra_scores": {}}, {"review_id": "kmL9ovfg8T", "reviewer": "Reviewer_TYCf", "summary": "This research paper presents a study on understanding Python notebook execution dependencies using the CRABS (Cell-by-cell Reasoning About Python notebook Structure) strategy. The empirical foundation of the work is generally solid, built upon a dataset of 50 annotated Python notebooks from Kaggle that were specifically selected for their relevance to machine learning workflows. The authors employed a well-defined experimental procedure using a cell-by-cell prompting strategy that effectively addresses long-context challenges and hallucinations in large language models. However, the study's methodological rigor is somewhat undermined by the lack of a detailed explanation regarding the sampling strategy for selecting the most up-voted notebooks, which may introduce selection bias and limit the generalizability of findings.\nThe technological contribution of this work is significant, with the CRABS strategy demonstrating clear effectiveness in addressing the complex problem of understanding notebook execution dependencies without actual code execution. The solution shows substantial performance improvements across multiple metrics, including a 28 percentage-point increase in F1 score, 32 points in accuracy, and 48 points in the exact match for information flow graphs compared to baseline approaches. The innovative combination of neural and symbolic methods represents a promising approach to resolving ambiguities in code analysis, though concerns remain about the scalability and broader applicability of the approach beyond the specific domain tested.\nFrom a forward-looking perspective, the research addresses a crucial gap in understanding Python notebooks without execution, which is essential for improving usability and adaptability in data science workflows. The study design demonstrates innovation by leveraging pre-trained language models to resolve data flow ambiguities between notebook cells, contributing to the broader field of automated code analysis.\nWhile the technical content is robust, the manuscript suffers from presentation issues that detract from its overall impact. The paper contains frequent grammatical errors, particularly related to verb tense consistency and subject-verb agreement, indicating a need for thorough proofreading. These clarity issues, while minor individually, collectively impact the professional presentation of otherwise solid research contributions.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper merits acceptance due to its innovative approach and significant technological contributions. The research presents the CRABS strategy, which effectively addresses the complex challenge of understanding notebook execution dependencies without requiring actual execution—a crucial advancement for improving data science workflow usability. The methodology demonstrates technical excellence through its robust data collection process utilizing 50 annotated Python notebooks from Kaggle specifically chosen for machine learning relevance. The cell-by-cell prompting strategy successfully mitigates long-context challenges and hallucinations, resulting in impressive performance improvements: a 28 percentage-point increase in F1 score, 32 in accuracy, and 48 in exact match for information flow graphs compared to baseline approaches. Particularly noteworthy is the paper's principled approach combining neural and symbolic methods, representing a promising general methodology for resolving ambiguities in code analysis. While there are minor grammatical issues requiring proofreading and some limitations regarding sampling strategy explanation and broader generalizability beyond Python notebooks, these do not diminish the paper's substantial contributions to the field. The research fills a specific gap in computational understanding of non-executed notebooks, making it a valuable addition to the literature that deserves publication.", "reasons_to_reject": "Despite the paper's promising approach to notebook understanding, we should consider several critical methodological limitations. The study fails to address potential biases introduced by the zero-shot in-context learning approach, which significantly undermines the reliability of the LLM's ambiguity resolution capability. Furthermore, the experimental design lacks essential control conditions to compare the cell-by-cell prompting strategy against alternative approaches, severely limiting the robustness of the findings. Another major concern is the unvalidated nature of the syntactic parsers used to bound the ground truth, potentially compromising the accuracy of the entire notebook understanding task. The heavy reliance on syntactic parsing without adequately capturing semantic nuances represents a fundamental weakness, as does the absence of context-aware parsing techniques to manage context-dependent data flows. Finally, the manuscript misses valuable opportunities to explore the broader implications of LLMs in transforming data science roles and education. These substantial shortcomings collectively indicate that the research requires significant refinement before it can make a meaningful contribution to the field.", "questions_to_authors": "- The methodology does not address potential biases introduced by the zero-shot in-context learning approach, which may affect the reliability of the LLM's ambiguity resolution. Suggestion: Provide a clear rationale and criteria for notebook selection to ensure representativeness and reduce bias\n- The experimental procedure does not include control conditions to compare the effectiveness of the cell-by-cell prompting strategy against other potential strategies, limiting the robustness of the findings. Suggestion: Implement control experiments using different prompting strategies to establish a comparative baseline\n- The validity of the syntactic parsers used to bound the ground truth is not thoroughly evaluated, which may impact the accuracy of the notebook understanding task. Suggestion: Conduct a validation study to assess the accuracy and reliability of the syntactic parsers used in bounding the ground truth\n- While the manuscript focuses on resolving ambiguities in Python notebooks, it does not explore the broader implications of LLMs in transforming data science roles and education\n- The manuscript does not extensively address the potential for LLMs to handle context-dependent translations. Suggestion: Integrate context-aware parsing techniques to enhance the CRABS strategy's ability to manage context-dependent data flows in Python notebooks\n- The manuscript's approach relies heavily on syntactic parsing, which may not fully capture semantic nuances or context-dependent information flows, a limitation that may be addressed by dynamic prompting", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers generally view the proposed CRABS method positively. Some of the highlighted strengths include: a well-motivated problem with an innovative approach combining syntactic parsing and LLM, demonstrating significant performance improvements and providing a valuable resource for computational understanding of non-executed notebooks.

While the reviewers also point out some limitations of this work, they are satisfied with the detailed responses from the authors, which provide more explanations and additional results. Including them in the paper will no doubt improve the paper's quality further, and thus is highly encouraged.
Decision: Accept