Sharing State Between Prompts and Programs
Authors: Ellie Y Cheng
Logan Weber
Tian Jin
Michael Carbin
Venue: ICLR 2026 Poster (2026)
Keywords: natural language programming
large language model
programming languages
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: We present a novel programming abstraction, shared program state, that enables programmers to use natural language prompts to to directly write program variables, compute with objects, and implement control flow in programs.

Abstract: The rise of large language models (LLMs) has introduced a new type of programming: natural language programming. 
Users write prompts, which are instructions in natural language, to direct LLMs to perform tasks such as natural language processing, code generation, reasoning, etc.

An emerging area of research enables interoperability between prompts and programs.
We present a novel programming abstraction, shared program state, that removes the manual work required to enable interoperability between prompts and program states.
With shared program state, programmers can write prompts that directly access program variables, compute with program objects, and implement control flow in the program.
We present a schema for specifying natural function interfaces that extend programming systems to support programs with prompts and leverage this schema to specify shared program state as a natural function interface.

We implement shared program state in the Nightjar programming system.
Nightjar enables programmers to write Python programs containing prompts that share the Python program state.
We show that Nightjar programs achieve comparable or higher task accuracy than manually written implementations (+4-19\%), while decreasing the lines of code by 39.6\% on average.
The tradeoff is that Nightjar may incur runtime overhead (0.4-4.3x manual implementations).

Published as a conference paper at ICLR 2026
SHARING STATE BETWEEN PROMPTS AND PROGRAMS
Ellie Y. Cheng
Logan Weber
Tian Jin Michael Carbin
MIT CSAIL
fellieyhc, loganweb, tianjin, mcarbin)@csail mit.edu
ABSTRACT
The rise of large language models (LLMs) has introduced a new type of program- ming: natural language programming. Users write prompts, which are instructions in natural language, to direct LLMs perform tasks such as natural language processing, code generation, reasoning, etc.
An emerging area of research enables interoperability between prompts and pro- grams. We present a novel programming abstraction, shared program state, that removes the manual work required to enable interoperability between prompts and program states. With shared program state, programmers can write prompts that directly access program variables, compute with program objects, and implement control flow in the program. We present schema for specifying natural function interfaces that extend programming systems to support programs with prompts and leverage this schema to specify shared program state natural function interface.
We implement shared program state the NIGHTIAR programming system[] NIGHTIAR enables programmers to write Python programs containing prompts that share the Python program state. We show that NIGHTJAR programs achieve comparable higher task accuracy than manually written implementations (+4- 19%), while decreasing the lines of code by 39.6% on average. The tradeoff is that NIGHTJAR may incur runtime overhead (0.4-4.3x manual implementations).
INTRODUCTION
Writing prompts, which are instructions in natural language, for large language model (LLM) systems to evaluate is new form of programming: natural language programming. LLMs have become
increasingly capable, leading to widespread adoption of prompting for tasks such as natural language
processing, code generation, and mathematical reasoning (Achiam et al. 2023 Chang al. 2024
Chen et al. 2021; Guo et al.] 2025 Liang et al. 2023 OpenAl 2025 Qin et al. [023 Wei et al.
2022). Prompts enable higher level of abstraction in comparison traditional programs, as prompts require fewer operational details.
Prompt & Program Interoperability. Programs offering precision and structure. Programs also support rich programming abstractions, including variables, data structures, and control flow. An
emerging trend of programming systems enables interoperability between prompts and programs, in
which the prompts and the programs operate on common data (Adepi) 2024; Anthropic [b] Beurer
Kellner al. 20233 BoundaryML 2025 Chase 2022 Dinu 2022 Dong al.] 2025 Firebase
Team Gat Guidance A1, Khattab et al.) 2024: Liu & Contributors |024 Microsoft b) Okuda &
Amarasinghe 2024; OpenAl PrefectHQ Vaziri al. 20244) Willard & Louf [023]] Yuksekgonul
et al., 2025 Zheng et al. 2024). Programmers develop programs containing prompts, and these
programming systems orchestrate the interface between the prompts and the programs. Figure la
shows an example Python program containing prompt.
Existing systems evaluate prompts in a separate environment from the program. Programmers must therefore manually implement the transfer of data and, broadly, program state across the boundary between the environments. We term such design as isolated program state. Figure[Ta da the manual work required to transfer program state in the isolated program state design.

---

Published as a conference paper at ICLR 2026
from pydantic import BaseNodel
import nightjar
from 11m_ wrapper import 11m
class Graph:
class Graph: ... EdgeSchena (BaseNodel):
nodes: set set [int]
edgea: dict [int, aet [int]]
arc: int
@nightjar.ff
tgts: list[int]]
def main():
class GraphSchema (Base//oee))))))
graph - Graph(....)
nodes: 1ist list[int]
8
while True:
edges: 1ist [BdgeSchema]
9
query input("Q: ")
10
def serialize(g: Graph) -> str:
10
"" "proupt
11
def rgify(gs: GraphSchems) -> Graph:
11
Perform the <query> with reapect to <graph>, where
12
nodes net(gs.nodes)
are paper IDs and edges point from cited paper of papers that cite it. Break if <query> indicates termination. Else, save <:response> and update <graph> to answer <query>.
13
edges {e.sre: (e.
for ga.edge)
Graph (nodes=noss,, edges=edges) def main():
16
17
graph Craph(...)
17
18
while True:
18
19
query = ("puut(Q:: ")
19
print(f"A: [response]")
20
class OutputSchema (BaaeMoeee)
20
print (Papper that cite paper 14:") print (ginphhrpp..ede edges [14])
21
response: Optional [str]
21
22
break_flag: bool
main()
23
graph: Optional [GraphSchena]
q_out 11m(
(b) Example Program with Shared Program State
""="Perform the <query> with reapect to <graph>, where nodes paper IDs and edges point from cited paper to set of papers that cite it. Return break_flag aa True if <query> indicates termination. Else, return "response" . <graph> was updated, 'graph'.
26
8 /example.pp
27
Update the graph paper cites A: Graph updated. Papers that cite paper
28
29
30
{
31
Q: Exit, please.
32
return
33
(erreea(eerall //grab>>"", rutput..rpprrDDtpptSchemmm
(c) Example Program Execution Console
36
if brout.break_flag:: break
Figure 1: Figure La shows program with iso- lated program state that executes natural language queries graph modeling paper citations. Blue highlighted lines show the prompt; gray high- lighted lines show the work to transfer the pro- gram state between the prompt evaluation and the program. Figure [bb shows the task implemented using shared program state with NIGHTJAR. Fig- ure Ic shows the input and output of an example execution. See Appendix A. A.
37
38
response q_out. response
39
if q_out.graph:
40
graph = out.fyqq_ot.graape
41
print (f"A: (response)")
print (f"Papers cite paper 14:")
print edges [14])
main()
(a) Manual Implementation (Isolated Program State)
for the full program.
Shared Program State. We present a novel programming Dbstraction-shure program state-in which prompts have direct access the program state-the variable scopes, the heap, and the control state of the formal program. The prompts read and write program variables (shared scopes), compute with program objects (shared heap), and implement control flow in the program (shared control state). Figure Tb shows an example program written with shared program state.
Natural Function Interface. Shared program state and isolated program
are both instances of natural function interface. A natural function interface specifies how prompts interact with the host program execution. We formalized and designed schema for designing natural function interfaces based effect and handler program constructs (Bauer & Pretnar 2015 Leijen| 2016)] Plotkin & Power 2003; Plotkin & Pretnar 2013). Effects define the actions to performed, and handlers define the implementations of the actions. Effects and handlers are generalization of tool use that enables using to implement context switches in system execution.
Shared Program State Natural Function Interface. Using schema, we specified shared program state as a natural function interface. The interface operates over variables, references, labels, and program values. It enables prompts to emit effects the host language read and write variables from shared scopes, reference and dereference values from the shared heap, and utilize program labels in the shared control state.
2

---

Published as a conference paper at ICLR 2026
Contributions. We present the following contributions in this work:
We present the shared program state abstraction. Shared program state enables developers to write more concise programs and focus on the core program logic. Figure Ia illustratees the extra work a developer would otherwise need to implement themselves to deliver the program with the same functionality that of Figure lb
We present a schema for specifying natural function interfaces that extend programming systems to support programs with prompts. Using this schema, we define the shared program state natural function function interface, language-agnostic formal specification that defines the mechanisms by which prompts interact with host programming system to realize the shared program state abstraction. System developers use our specification to implement the shared program state abstraction in their own programming systems.
We implemented the shared program state natural function interface as the NIGHTJAR pro- gramming system to demonstrate the efficacy of programming with the shared program state abstraction. We show that NIGHTJAR programs achieve comparable or higher task accuracy than manually written implementations (+4-19%).. while decreasing the lines of code by 39.6% on average. The tradeoff to using NIGHTJAR is that may incur runtime overhead (0.4-4.3x runtime of manual implementations).
Implications. Our work introduces shared program state, a new abstraction for programming with prompts in programs. Shared program state enables programmers to focus on program logic by delegating the low-level work of enabling interoperability to the natural function interface imple- mentation. Our work highlights the emerging computing paradigm brought by the rise of LLMs in which computations are dynamically and adaptively planned, implemented, and executed on-the-fly according to the live context.
RELATED WORK
Natural Language Programming. The idea of integrating natural language in programs dates back to literate programming (Knuth, 1984) and has been popularized by the use of Jupyter and IPython notebooks (Pérez & Granger (0077). This line of work does not treat natural language as executable code, but instead as documentation describing the behavior of the program or as macros for programmer-wwrn written code. The popularity of using natural language prompts to perform computation has risen as LLM capabilities improve. Several works coin prompting
form of programming (Beurer- et al., 2223) Feng et al., [0224] Khattab et al. 2024 Liang et al. 2025). Our work takes this step further by unifying the state between prompts and program.
Language Interoperability. Language interoperability is the task of enabling a software system to leverage multiple different programming languages computing with common data in its implemen- tation. Language interoperability has a long history (Benton & Kennedy] 1999: Breg et al.) 1998]
Burroughs Corporation 1961 Davis et al. 1994] Finne et al. 1999 Graham 1966; Jeffery et al.
1999] Konstantas] [993] Laddad & Sen 2020 Matthews Findler [000 Microsoft] a Nelson 1981
Object Management Group. 1991 Oracle| 2017
Osborne al. 2024 Python Software Foundation 2024), dating back at least 50 years as the software engineering community has sought to build increasingly sophisticated software systems that span increasingly more components, each suited to development in particular programming languages.
2025 The Math Works, Inc. 2025
Wagner et al.
Isolated Program State. In recent years, enabling interoperability between prompts and programs have become an area of active research. Most prior work (Anthropic) b) Keeurer-Klllne' et al. 2023
Chase 2022) Dinu 2022 Dong et al. Firebase Team; Gat Guidance Al Mecrosoit by OpenAl] Vaziri et al.) | Willard & Louf 20233] Yuksekgonul et al.] 2025] Zheng et al. (024) isolates prompts evaluation from program execution. Specifically, prompts have no visibility to the program state. In general, the programmer writes code to serialize program data to pass to the LLM in the prompt. The LLM returns a string value by default, and the programmer optionally writes schema to define how the output string is reified into program data. Systems that enable code interpreter tools to answer a natural language prompt (Anthropic Chen al.) 2023 Gao et al. 2023 OpenAl) isolate the code execution from the program that the natural language prompt is embedded i. Our work enables a new programming model of interoperability in which the prompt evaluation shares the program state of the host program.
BoLndaryML 2025
Khattab et al.
2024 Liu & Contributors
3

---

Published as a conference paper at ICLR 2026
Frogrammer-Definet Tools. In systems that support interoperable programs with isolated program states, programmers can define custom functions and implement tool use to manually convert LLM responses into operations the program state. Several systems automatically make tool calls using
programmer defined functions (Beurer- (Beluer-Kellner et al., 2023 Chase 2022 Dong et al. 2025 Hirebase Team]. Our shared program state abstraction does require programmers define custom functions for the prompt to perform operations on the program state.
Partially Shared Program State. Other systems enable partially shared program state (Adept 2024; Dantanarayana et al. [025] Huang et al. 2023 Okuda & Amarasinghe 2024 Qr
Marvin ) and MTP (Dantanarayana
al. (025) support reading function arguments and scrialization/reiicattoo of program data, but do not enable prompts to manipulate mutable objects
in-place. AskIt (Okuda & Amarasinghe 2024) and ANPI. (Huang et al. 2023) replace prompts with L..M-generated functions. The generated functions can access object references passed as function arguments. None of the above methods enable prompts to read variables outside of the function scope (and thus cannot support closures), write variables, nor implement control flow. AWL (Adept) (2024) a domain -omaain-ppecific language for web interactions; prompts evaluate on shared webpages with the program but not shared program state.
SHARED PROGRAM STATE BY EXAMPLE
NIGHTIAR integrates prompts as first-class code in Python programs: The prompts can read and write Python variables, compute with Python objects, and manipulate the control flow of the surrounding Python program. Figure shows an example program that uses shared program state via NIGHTJAR.
3.1 EXAMPLE PROGRAM
The program implements an agent that manipulates a directed graph of paper citations. The nodes are paper identifiers, represented by a set of integers. The edges connect cited paper to papers that cite it, represented by dictionary of source nodes to sets of target nodes.
The main function uses a while-loop (Line X to continuously read in user queries from standard input. A triple-quoted string with the language identifier prompt demarcates a prompt, as shown on Lines 10 and 18 The prompt specifies the agent's behavior in natural language: The agent performs the given query by writing to variable named response and updating the graph. Execution breaks out of the while-loop if the query signals the user is finished with the program; otherwise, the agent prints the response and the papers that cite paper 14 on Lines 19 to 21 Figure Ic shows program outputs with the inputs of a query that adds a new edge and a query that ends the program.
3.2 SHARRD PROGRAM STATE
At first glance, the prompt in Figure lb might look like a comment to be filled in by the programmer, but prompts with shared program state first-class code. Prompts evaluate with shared scopes, shared heap, and shared control state with the Python program state.
Shared Scopes. Prompts read and write Python variables in the shared scopes. Prompts can refer to any live Python variables within scope using the syntax <var>, For example, on Line [I] the prompt reads the query variable, which on the first loop assigned paper 5 cites 14". <:var> denotes variable assignments. Line [6] writes string response "Graph updated. to the response variable that is printed to standard output on Line [9 Shared scopes enable the response variable to be used in Python like any other variable in that scope.
"Update the graph so
Shared Heap. Prompts access Python objects by reference in the Python heap. The prompt in Figure [b] adds an edge in the graph object with an in-place update to the existing graph. edges dictionary of the existing graph object in the program's heap, equivalent to the Python code edges (S). The update persists after the prompt evaluation. By referencing the existing graph pointer, Lines t0 2T prints out that paper is in the set of papers that cite paper 14.
Shared Control State. Prompts also manipulate the evaluation context in the shared control state through program labels. When the query variable indicates the user is finished with the program, like the second example query "Exit, please", the prompt updates the evaluation context after the
4

---

Published as a conference paper at ICLR 2026
Python Interpreter
main(: while True:
Program State
Scapes
query = input ("Q: )
query; "Exit, please."
quezy: *Eait
Heep
Perform the <query>
8:: Geaph I.d
Gontrol State
main(
Prompt Evaluation
Perform the equery>
"Bxi
Lookup (queryl
quezy please.
122
LLM
resume
"Exit,
please.
Handler
Goto (break)
Figure 2: a) An example execution of Figure lb in which the prompt uses shared program state. b) The Python interpreter reads and writes to the program state, such as by writing the variable query. <) The Python interpreter hands the execution control to the LLM evaluate the prompt. d) The LLM emits a Lookup request to read the query variable. e) The handler reads query from the shared scopes, which has the value "Exit, please." g) The handler resumes prompt evaluation with the value of query. h) The LLM emits another request, Goto, to update the evaluation context to the break program label (i.e. outside the while-loop). Finally, the handler manipulates the control state accordingly without resuming prompt evaluation, implementing control flow.
loop, in effect breaking out of it. Correspondingly, the print-statements on Lines 19 to 21 executed, and the program gracefully exits.
are not
Without Shared Program State. Without shared program state, prompts only read and write serialized string values with no other effects on the Python program. Programmers have to write code to manually serialize input data and convert serialized output values into operations on the program state. In Figure Ia we show an illustrative example of how programmer might implement the task manually without using shared program state. Without shared program state, the programmer has to write Python code to manually serialize and reify data structures, update objects in the heap, and invoke control flow primitives. Shared program state provides an abstraction over the intermediary connecting code between prompt and the program state.
3.3 EXECUTING PROGRAMS WITH SHARED PROGRAM STATF
We implemented the NIGHTJAR programming system with shared program state. NIGHTIAR uses an LLM agent evaluate prompts. The LLM agent issues effects, which are requests to the programming system for manipulations to the Python program state. NIGHTIAR implements handlers, which receive these effects and interface with the Python program state. Figure 2
shows diagram of NIGHTJAR
evaluating a prompt with shared program state.
Table I: Average pass rate and runtime of
Table shows the performance of the Figure La pro- gram (manual programmer implementation) and the (using NIGHTIAR) on graph of 25 nodes with 6 different natural language queries (Appendix A.22 NIGHTIAR achieves higher pass rate in this task than the manual implementation, because the LLM agent adapts to program inputs and inter- mediate program values. For example, for the query Update the graph paper 5 cites 14, the LLM decides to issue effects to update only the dictionary entry for paper 14. Without shared program state, the programmer must decide and implement the specific mechanisms for the prompt to interact with the program state. In Figure Ial the programmer implemented updates to the graph by querying the LLM to generate the entire updated graph, which is prone to hallucinations, Shared program state offloads the work of reasoning about and implementing updates to the program state to the LLM agent. The tradeoff to the abstraction is that the LLM agent incurs runtime overhead. Our work shows that engineering efforts reduce the overhead, offering opportunities for future work to improve the accuracy and runtime of shared program state programming systems.
Figure programs, with time ranges in gray.
Method
Pass Rate
Time (s)
Manual Impl
0.33+10.00
31.3 (28.7 35.7.3
NIGHTIAR (Ours)
0.83-10.12
77.2 (72.7-83.0)
5

---

Published as a conference paper at ICLR 2026
4 A NATURAL FUNCTION INTERFACE FOR SHARED PROGRAM STATE
We developed shared program state to enable a higher level of abstraction when using prompts within programs: The prompts manipulate the host program state directly. To define the mechanisms by which a prompt interfaces with the host program state, we first describe the natural function interface schema for extending programming systems with support for prompts in programs. Then, we formally specify shared program state as a natural function interface.
4.1 NATURAL FUNCTION INTERFACES
natural function interface is a specification of how prompts transfer data or utilizes services of a host programming language. A natural function interface provides definitions of 1) the values that cross the boundary between the prompt and the program, 2) effects, which are distinguished requests by the prompt for interacting with the surrounding code, and 3) handlers that define the behavior of the requests as implemented in the host system. Natural function interface is a generalization of tool use, as defined by OpenAI documentation (OpenAl) and MCP (Model Context Protocol). In Appendix B we formalized natural function interfaces using effects and handlers (Bauer & Pretnar 2013
Leijen 2016
Plotkin & Power 2003 Plotkin & Pretnar 2013).
Values. The values of natural function interface are the data types that the prompt takes input and output from the host system (NFI Values in Figure 3). While LLMs typically take strings as input and output, natural function interface can specify richer data types and leverage serialization and reification such as through JSON formatting and Pydantic (Colvin et al., 2025) to automatically convert between string and program data types.
Effects. The effects of a natural function interface are the requests that the prompt expects the host language fulfill (NFI Effects in Figure 3). The prompt evaluates to either a value an effect, denoting the request the host language should fulfill so that the prompt evaluation can continue.
Handler. The handler of a natural function interface defines the implementation of the effect requests by the host language (NFI Handler in Figure (3. When the prompt evaluates to an effect, the handler that wraps the prompt catches the effect and fulfills the request.
As described, effects and handlers are similar in behavior to elements of the OpenAl function calling API. In the function calling API, the LLM can request function to be called and the client of the API will interpret the function and provide the value back to the API, whereupon the LLM resumes its evaluation. Where natural function interface extends the function calling API is with the ability for the handler to explicitly either resume evaluation (as is typical) or cancel the continued evaluation of the prompt (new). Cancelling the prompt evaluation enables the host program and the prompt to coordinate rich control flow patterns.
natural function interface a contract between the host language and the prompt. Specifically, the LLM system that evaluates the prompt must support 1) serialization/reificatoo of the defined values, 2) checkpointing of the prompt evaluation when emitting an effect (as execution control switches to the handlers), 3) restoration of prompt evaluation from checkpoint when the handler initiates resumption, and 4) cleanup of the prompt evaluation when the handler cancels the prompt evaluation.
4.2 SHARED PROGRAM STATE AS NATURAL FUNCTION INTERFACE
Now we turn to specifying shared program state as natural function interface, as shown in Figure We assume a general host language that supports variable bindings (shared scopes), mutable state (shared heap), and control flow constructs (shared control state).
Shared Scopes. The values to support shared scopes are the names of variables in the host program. The effects are Lookup and Assign. For Lookup requests, the handler retrieves the value of the requested variable in the current scope and uses the resumee keyword to restore prompt evaluation with the value. The requested variable must be in the set of variable names the LLM is allowed to read, denoted X₁- For the Assign request, the handler makes the variable binding in the current scope then resumes prompt evaluation with the variable binding still in place. The variable must be in the set of variable names that the L.M is allowed write to, denoted X.-

---

Published as a conference paper at ICLR 2026
Variables X
References A
Labels E L
Host Values
Host Language
ef :==
...
values
let in
variable bindings
le ref e
mutable state
label & goto &
control flow
Prompts
n
Serialization
a: V > VN Na. UN
Reification 7. UN -> v op X;, X.,Xu.
Prompt Evaluation S
NFI Values
NFI Effects
op
Xo No.k
(2)
Assign(Zo: VN
2t E Xa E
Deref (UN) Ref(uN) Set(a, uN)
Goto(E)
DEL
NFI Handler
by === handle
finally v/ + (ex); Lookup(z) resume a(x);
get variable
Assign(z, uy: +
let (wN) ) in resume of ());
set variable
Deref(#x) + resume a(h(ww))
get value at reference
Ref(ux) resume a(ref ((v)))
create reference with value
Set a, ux) -
resuine ((>(a) (a) ;= >=>)))); Goto(E) goto ((f)
set value at reference
set evaluation context to label
Program
prompt with shared program state Figure Shared Program State Natural Function Interface.
=== est
by xn. a(v)))
Shared Heap. The values to support shared heap are references. Deref, Ref, and Set are the effects. The handler fulfills these effects by resuming prompt evaluation with the value evaluated from the dereference, reference, and set host program construct, respectively.
Shared Control State. The values to support shared control state are program labels. The corre- sponding effect is Goto. The handler dispatches the Goto effect by jumping the evaluation context to the specified label in the effect. Notably, the handler does not resume prompt evaluation while handling this effect, which implements control flow. The label must be in the set of labels that the LLM is allowed to jump to, denoted as L.
NIGHTIAR PROGRAMMING SYSTEM
We implemented the shared program state natural function interface as the NIGHTJAR programming system with Python the host language. NIGHTTIAR uses LLM agent to translate prompts into effects. In this section, we discuss implementation details for NIGHTTAR.
5.1 IMPLEMENTATION
Shared Scopes. NIGHTIAR distinguishes Python variables from regular words within a prompt using the syntax <var> for variable references and <:var> for variable assignments and parses them into X; and X,, respectively. Specifically, the handler enforces that the LLM agent only lookup <var>- denoted variables. If the LLM attempts to look up any other variable, the handler provides the LLM with an eтror message. This prevents the LLM agent from reading hallucinated variables or variables the programmer does not want the LLM to read. The handler also enforces that <:var>-denoee variables must be defined the end of the prompt evaluation, providing the L.LM with an error message if attempts to terminate evaluation when these variables undefined. This check enables programmers to reason about which variables are live after prompt.
Shared Heap. NIGHTTAR represents all Python objects data in the shared heap, represented by references. This enables prompts to manipulate mutable Python objects in-place. Only plain old data types that are immutable, such strings, integers, floats, booleans, and null-types, are passed directly through the NFL. Additionally, the handler provides the LLM agent with an error message it attempts to read invalid references.
7

---

Published as a conference paper at ICLR 2026
Shared Control State. Because Python does not support the low-level goto program construct, NIGHTJAR performs a source code transformation to install try-except blocks as program labels. NIGHTJAR exposes program labels corresponding as break, continue, return, and raise to the L.MM to use with the Goto effect. NIGHTIAR only exposes program labels that are valid within the evaluation context of the prompt. For example, the program label corresponding to break is only exposed when the prompt is within a loop.
Python built-in control flow primitives such
5.2 SYSTEM ENGINEERING
One implementatin of shared program state in a programming system is by querying the LLM agent to generate the effects exactly as defined in Figure 3 While this implementation is aunguag--ggnosicc not designed to be performant. Realizing shared program state as performant abstraction requires additional system engineering. To improve performance, we specialize NIGHTJAR to Python and implement optimizations for reducing runtime.
Specialization. We designed shared program state natural function interface to be general to capture the underlying semantics of the shared program state abstraction. The tradeoff of the general approach is the loss of specialized host-language features such as first-class functions, abstract methods, and high-level control flow constructs. It also relies entirely on the LLM to perform data transformations. We specialize NIGHTJAR to Python so that it supports these Python-specific language features and enables Python- based computation. In this implementation, the effects Eval (evaluates a Python expression and returns the result) and Exec (executes block of Python code in the evaluation context) replace the Lookup, Assign, Deref, Ref, and Set effects. The effects Break, Continue, Return, and Raise replace Goto, corresponding to the respective Python primitives.
Eager Variable Loading. During execution, the LLM selects a subset of the variables mentioned in the prompt and inspects their value and type by issuing effects, However, each LLM agent step incurs runtime overhead. We apply an optimization to eagerly load the values and types of the mentioned variables to the LLM, to reduce the number of effects the LLM issues and thus reduce the runtime. The tradeoff is that the value and types of variables the LLM would otherwise not select for inspection are also placed in context, introducing additional token cost.
Caching. We implement simple caching scheme where the LLM API client stores pairs of LLM agent effect trace history and the response. When given identical agent effect trace histories, the client returns the stored response rather than querying fresh responses from LLM APIs. This enables time savings until the LLM reads program inputs it has not seen before.
EVALUATION
In this work, presented shared program state, new programming abstraction that delegates the work of enabling interoperability between a prompt and the program state to the natural function interface implementation. We implemented the NIGHTIAR programming system to execute programs with prompts using shared program state. We next perform an evaluation to answer the following:
RQ1. Does shared program state enable more concise programs?
RQ2. Can programs with shared program state perform as well as manual implementations?
Benchmarks. Since there are no existing benchmark suites that evaluate shared program state, we introduce SPSBench. SPSBench is suite of 25 programs that contain prompts using shared program state. We adapted several of the programs from the documentation of LLM-related Python packages. We constructed additional programs to evaluate more complex manipulations on the program state by prompts, such as creating closures, in-place state mutations, using generators, defining subclasses, and raising errors. See Appendix for program descriptions.
Baselines. For each program, we implemented the same task using a simplified LLM API that takes a prompt and an optional Pydantic (Colvin et al., 2025) model or JSON schema; we call these Manual Implementation programs. Manual Implementation (Code Interpreter) executes the manual implementation programs with official code interpreter tools from OpenAl and Anthropic enabled in LLM API calls, which are isolated from the host program but enable the LLM to perform Python- based calculations. The third baseline, NIGHTJAR (Baseline), executes shared program state programs using NIGHTJAR without specialization optimizations.
8

---

Published as a conference paper at ICLR 2026
Table Average pass rate with standard deviation and average runtime with ranges in gray.
Model
Method
Pass Rate
Time (s)
Sonnet 4 Manual Impl
0.78.10.03
8.0
(1.3-44.8)
Sonnet 4 Manual Impl (Code Interpreter) 0.66110.04
36.8
(3.2-171.5)
Sonnet 4 NIGHTIAR (Baseline)
0.60110.04
55.6
(9.6.721.0)
Sonnet 4 NIGHTIAR
0.85
25.9
(3.0)
GPT-4.1
Manual Impl
0.74:1:0.03
4.5
(0.8-15.0)
GPT-4.1 Manual Impl (Code Interpreter)
0.72.1.0033
50.3
(10.9-364.9)
GPT-4.1 NIGHTIAR (Baseline)
0.61110.02
39.0
(4.7-957.4)
GPT-4.1 NIGHTIAR
0.78++00.03
19.6
(3.0-72.1)
RQ1: Methodology. We compared the Manual Implementation programs against shared program state programs. We counted the number of lines to implement the task in each implementation, not counting scaffolding for testing nor empty lines and comments. We used the Black Formatter linter with a maximum width of 120 characters to standardize the formatting.
RQ1: Results. Manual Implementation has 42.0 lines of code on average, with a range of 207 lines. In contrast, shared program state has 25.1 lines of code on average and range of 7 to 166 lines. Shared program state enables a decrease of 17.0 lines of code on average, with a range of 0 to 60 lines of code decreased. Shared program state enables programmers to write more concise programs.
RQ2: Methodology. We compare the performance of the shared program state programs executed using NIGHTJAR against Manual Implementation, Manual Implementation (Code Interpreter), and NIGHTIAR (Baseline). We used We used the default temperature value of and executed each program implementation of the benchmark times. We computed the average pass rate averaged over runs, where the average pass rate is the average percentage of passing assertions of each program. We set limit of 300 tool calls or effects and a timeout of 1000 seconds. See Appendix D for the system prompts.
4onnet---2025051 14 and GPT-4.1-2025.. 14 as the LLMs.
RQ2: Results. Table 2 shows the average pass rate with standard deviation and the average runtime with range. NIGHTIAR achieves comparable or higher average pass rate than Manual Implementation and Manual Implementation (Code Interpreter), demonstrating that the shared program state abstraction can be realized as executable programs. The tradeoff to the shared program state abstraction is that NIGHTIAR incurs a runtime overhead compared the chat completion-only usages of Manual Implementation because NIGHTJAR uses an LLM agent to evaluate prompts. However, the Manual Implementattin (Code Interpreter) also incurs a runtime overhead.
The NIGHTIAR (Baseline) achieves lower pass rate compared to Manual Implementation and Manual Implementation (Code Interpreter). This because NIGHTIAR (Baseline) is language-agnostic and thus cannot correctly complete tasks that require Python-specte functions and abstract classes. NIGHTIAR (Baseline) is also slower than Manual Implementation (Code Interpreter), because the LLM agent spends additional time, attempting to find work around for tasks that cannot complete. NIGHTIAR bridges this performance gap by specializing the system to Python to support these features and enabling Python- computations.
data types such as first-class
The additional engineering in NIGHTIAR reduces the runtime overhead by 2.0-2.1x 1x compared to NIGHTJAR (Baseline). This suggests that the performance of the shared program state abstraction can be improved with more sophisticated implementations in future work. Appendix D contains additional ablation results and Appendix presents a failure analysis of NIGHTIAR executions.
LIMITATIONS & FUTURE WORK
Shared program state is novel programming abstraction that automates the work of enabling inter- operability between prompts and the program state. As a novel abstraction, numerous opportunities remain for extending the design that are beyond the scope of this work.

---

Published as a conference paper at ICLR 2026
Safety & Security. Shared program state enables prompts to directly read and write objects in memory.
This is a departure from the existing systems that do not enable shared program state (Anthropic) [b]
Beurer-Kellner et al., [2023] MhoundaryML... [2255] Chase, 2022]
Dong et al. 2025 Firebase Team
Guidanned AI; Khattab et al. 2024 Liu & Contributors 2024 Microsoft, b. OpenAAl PrefectHQ
Vaziri et all]]]] 2024| Willlrd & Louf 2023 Yuksekgonul al. 2025 Zheng et al. 2024]; these systems
provide isolation in that the prompts are completely separated from the program. In existing systems, programmers are in full control over how prompt interacts with the program state. Shared program state blurs this boundary. While NIGHTJAR provides partial safety mechanisms in the handlers, these mechanisms are not complete.
This tradeoff between usability, safety, and performance is well studied in the operating systems and language- memory isolation literature. Heap isolation (Back & Hsieh| 2005] Von Eicken et al. 1999), while offering safety from inconsistent states, requires marshaling and copying data which introduces high overheads (Emmerich et al. 2019). Work that develops safety mechanisms with
abstractions such as single ownership (Hunt& & Larus 2007 Lafrance et al. |2023 Narayanan et al.,
2020) bridges these tradeoff's by enabling zero-copy communication on shared heap. We anticipate fruitful, future research that develops richer safety mechanisms for shared program state, potentially inspired by this previous work. Exploring other execution substrates, such as using a subset of Python that can be compiled to DSL with security and reliability benefits (Mell et al., (025) may reveal additional points in the tradeoff space between execution time, correctness, security, and reliability.
Safety also extends beyond isolation. Work in the approximate computing literature on distinguishing reliably and unreliably computed data is also relevant in that the developed techniques can enable users to reason about the downstream effects of values computed by the LLM (Carbin al. 2013
Sampson et al. 2011). Our work also emphasizes the importance of defending against vulnerabilities
in LLM-based software (Chang et al. 2024)), especially inference- time defenses against malicious
user inputs, such as pre-processing of imputs to LLMs (Li et al.) 2023 Mo et al.
2025
Wei et al.
2026) and post-processing of L.LM generated data and code (Phute et al., 2023
Qi et al. 2021).
Performance. NIGHTJAR opens up new line of work on improving the execution time and
correctness of programming systems enabling shared program state. We foresee future work in optimizing effects akin to bytecode optimizations (Clausen 1997 Vallée-Rai et al. 1999 Xu & Kjolstad 2021), applying parallel generation techniques (Jin et al. 2025 Liu et al. 2024] Pan et al. 2025; Yang et al., 2025), ) and sophisticated caching schemes to decrease the execution time. We also anticipate future system designs that improve correctness, such as by incorporating automatic natural
language prompt optimizations (Chen et al. 2024 Fernando et al. 2024 Khattab al., 2024 Pryzant
ct al. 2023 Schnabel & Neville 2024 Yuksekgonul et al. 2025 Zhou et al. 2022).
Program Development Tools. As new programming abstraction, shared program state calls for future work on program development tools, such as debuggers, testing tools, and program analyses, that assist programmers in writing prompts that achieves low execution time and high accuracy (Liang et al. 2025). These tools would need to combine both traditional program development needs (e.g. identifying inputs that cause bugs (Cadar et al., [2008a b)) as well as the needs of prompt development (e.g. testing the prompt robustness with respect to a given LLM (Sharma et al. 2025)).
CONCLUSION
Shared program state enables programmers to focus on core program logic when using prompts in their programs. The abstraction offloads the work of enabling interoperability between the prompts and the host program to an LLM. By using an LLM agent to plan, implement, and execute high- level natural language instructions, NIGHTJAR enables live dynamic computations that adapt to program inputs and intermediate values, which is not scalable to implement manually in traditional programming. This emerging paradigm of adaptive computing, in which software is dynamically programmed to the context, enables a new promising class of systems that remain robust under changing contexts without large-scale manual engineering.
ACKNOWLEDGMENTS
This material is based upon work supported in part by the National Science Foundation Graduate Research Fellowship under Grant No. 2141064 and the CSAIL-Felicis Research Program. Any
10

---

Published as a conference paper at ICLR 2026
opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. We thank Jesse Michel, Charles Yuan, Eric Li, Linlu Qiu, and Daniel Pfrommer for helpful feedback on this work.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.088 0,77744 2023.
Adept. Building powerful agents with adept, 2024. URL https://uww.adegg: wwt.bllppp..aabboog...... adept-
Anthropic. Welcome to claude, a. URL /ttp:ppp///oooosattto.o.c/////o/
Anthropic. Code execution tool agents and tools (anthropic api documentation)) http://iiioottttttta//t:aa
anthropic. ctccropic.com// en/ wt docs///c///// http///ccs/aetssseeaseesssessss/ and todd codeoll---u-cceeeeooeeeeeeee
b.
Godmar Back and Wilson C Hsieh. The kaffeos java runtime system. ACM Transactions on Programming Languages and Systems, 27(4), 2005.
Andrej Bauer and Matija Pretnar. Programming with algebraic effects and handlers. Journal of Logical and Algebraic Methods in Programming, 84(1), 2015.
Nick Benton and Andrew Kennedy. Interlanguage working without tears: Blending sml with java. In International Conference on Functional Programming. 1999.
Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting is programming: A query language for large language models. In International Conference on Programming Language Design and Implementation, 2023.
BoundaryML. Baml: The framework that adds the engineering to prompt engineering.
https
// tigit wtt./o/o//a/aeeeee comm/ooundarmml/aan 2025.
Fabian Breg, Shridhar Diwan, Juan Villacis, Jayashree Balasubramanian, Esra Akman, and Dennis Gannon. Java rmi performance and object model interoperability: Experiments with java/hpe++. Concurrency: Practice and Experience, 10(11-13), 1998.
Burroughs Corporation. The descriptor: A definition of the b 5000 information processing system. Technical report, Burroughs Corporation, Sales Tech. Serv. Sys. Doc., 1961.
Cristian Cadar, Daniel Dunbar, and Dawson Engler. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. Symposium on Operating Systems Design and Implementation, 2008a.
Cristian Cadar, Vijay Ganesh, Peter M Pawlowski, David Dill, and Dawson R Engler. Exe: Automatically generating inputs of death. Transactions on Information and System Security, 12(2). 2008b.
Michael Carbin, Sasa Misailovic, and Martin C Rinard. Verifying quantitative reliability for programs that execute on unreliable hardware. In ObjecttOriintted Programming, Systems, Languages and Applications, 2013.
Yupeng Chang. Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3). 2024.
Harrison Chase. Langchain, 2022. URL https:tt:////thhub htttshhu..ooom//em wonm/langchannaai/aaae
Lichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang, and Tianyi Zhou. Instructzero: Efficient instruction optimization for black-box large language models. International Conference on Machine Learning, 2024.
11

---

Published as a conference paper at ICLR 2026
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023.
Lars R Clausen. A java bytecode optimizer using side-effect analysis. Concurrency: Practice and Experience, (11),, 1997.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 10.1668,, 2021.
Samuel Colvin, Eric Jolibois, Hasan Ramezani, Adrian Garcia Badaracco, Terrence Dorsey, David Montague, Serge Matveenko, Marcelo Trylesinski, Sydney Runkle, David Hewitt, Alex Hall, and Victorien Plot. Pydantic, 2025. URL
Jayanaka L Dantanarayana, Yiping Kang, Kugesan Sivasothynathan, Christopher Clarke, Baichuan Li, Savini Kashmira, Krisztian Flautner, Lingjia Tang, and Jason Mars. Meaning-typed programming: Language abstraction and runtime for model-integrated applications. In Principles of Programming Languages, 2025.
Harley Davis, Pierre Parquier, and Nitsan Séniak. Sweet harmony: the talk/c++ connection. In Lisp, 1994.
Dinu. Symbolicai: A neuro-symbolic peppioo-yymbbbiiepppeeeetieeee on large language models (Ilms), 2022. URL http::/:/gggitubb..cm///teeeeee/
Honghua Dong, Qidong Su, Yubo Gao, Zhaoyu Li, Yangjun Ruan, Gennady Pekhimenko, Chris J Maddison, and Xujie Si. Appl: A prompt programming language for harmonious integration of programs and large language model prompts. In Association for Computational Linguistics, 2025.
Paul Emmerich, Simon Ellmann, Fabian Bonk, Alex Egger, Esaú García Sánchez-Torija, Thomas Günzel, Sebastian Di Luzio, Alexandru Obada, Maximilian Stadlmeier, Sebastian Voit, et al. The case for writing network drivers high-level programming languages. ACM/IEEE Symposium on Architectures for Networking and Communications Systems, 2019.
Li Feng, Ryan Yen, Yuzhe You, Mingming Fan, Jian Zhao, and Zhicong Lu. Coprompt: Supporting prompt sharing and referring in collaborative natural language programming. In Conference on Human Factors in Computing Systems, 2024.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self- aaf-immprveeent via prompt evolution. In International Confer- ence on Learning Representations, 2024.
Sigbjorn Finne, Daan Leijen, Erik Meijer, and Simon Peyton Jones. Calling hell from heaven and heaven from hell. In International Conference on Functional Programming, 1999.
Firebase Team. Genkit: An open source framework for building ai ai-powered apps. https::/gitie
com/firebase/genkii
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aide aided language models. International Conference on Machine Learning, 2023.
Noam Gat. In-format-enforree: Enforce the output format (json schema, regex etc) of a language model. htttp:://github.commmm
Robert M. Graham. System Module Interfaces (PL/I Subset for System Programming). 1966.
Guidance AL. Guidance: A guidance language for controlling large language models.
https
www.aa/cco/gguuadneeaiiiiiiaa ai/guddanceeee
12

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in llms through reinforce- ment learning. Nature, 645(8081), 2025.
Horace He and Thinking Machines Lab. Defeating nondeterminism in Ilm inference. Thinking Machines Lab: Connectionism, 2025.
Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang, and Sadid Hasan. Does prompt formatting have any impact on Ilm performance? arXiv preprint arXiv:2411. 10544, 2024.
Di Huang, Ziyuan Nan, Xing Hu, Pengwei Jin, Shaohui Peng, Yuanbo Wen, Rui Zhang, Zidong Du, Qi Guo, Yewen Pu, al. Anpl: towards natural programming with interactive decomposition. In Advances in Neural Information Processing Systems, 2023.
Galen C Hunt and James R Larus. Singularity: Rethinking the software stack. ACM SIGOPS Operating Systems Review, 41(2), 2007.
David Jeffery, Tyson Dowd, and Zoltan Somogyi. Mcorba: corba binding for mercury. In International Symposium on Practical Aspects of Declarative Languages, 1999.
Tian Jin, Ellie Y Cheng, Zack Ankner, Nikunj Saunshi, Blake M Elias, Amir Yazdanbakhsh, Jonathan Ragan-Kelley, Suvinay Subramanian, and Michael Carbin. Learning to keep promise: Scaling language model decoding parallelism with learned asynchronous decoding. In International Conference on Machine Learning, 2025.
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang. Keshav Santhanam, Sri Vard- hamanan, Saiful Haq. Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-improving pipelines. In International Conference Learning Representations, 2024.
Donald Ervin Knuth. Literate programming. The Computer Journal, 27(2), 1984.
Dimitri Konstantas. Object oriented interoperability. In European Conference on Object-Oriented Programming, 1993.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Symposium on Operating Systems Principles, 2023.
Shadaj Laddad and Koushik Sen. Scalapy: scamless python interoperability for cross-platform scala programs. In International Symposium on Scala, 2020.
Arthur Lafrance, David Detweiler, Zhaofeng Li, Xiangdong Chen, Vikram Narayanan, and Anton Burtsev. Extending rust with support for zero copy communication. In Proceedings of the 12th Workshop on Programming Languages and Operating Systems, 2023.
Daan Leijen. Algebraic effects for functional programming. Microsoft Research Technical Report, 2016.
Linyang Li. Demin Song, and Xipeng Qiu. Text adversarial purification defense against adversarial attacks. In Association for Computational Linguistics, 2023.
Jenny T Liang, Melissa Lin, Nikitha Rao, and Brad A Myers. Prompts are programs too! under- standing how developers build software containing prompts. Foundations of Software Engineering, 2025.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan. Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023.
Jason Liu and Contributors. Instructor: A library for structured outputs from large language models, 2024. RRI.hhttss://// wtwettthb..ommiittttttttt http://///nntttttooaaaa/aaeeeaee
13

---

Published as a conference paper at ICLR 2026
Mingdao Liu, Aohan Zeng, Bowen Wang, Peng Zhang. Jie Tang. and Yuxiao Dong. Apar: Lims can do auto-parallel auto-regressive decoding. arXiv preprint arXiv:2401.06661, 2024.
Jacob Matthews and Robert Bruce Findler. Operational semantics for multi-language programs. In Principles of Programming Languages, 2007.
Stephen Mell, Botong Zhang, David Mell, Shuo Li, Ramya Ramalingam, Nathan Yu, Steve Zdancewic, and Osbert Bastani. A fast, reliable, and secure programming language for Ilm agents with code actions. arXiv preprint 22x.::25506./222,,2222 2025.
Microsoft. The component object model windows/win32. htte:/occcccccc// componnett-objeet-mooeee -oo- a.
Microsoft. Typechat: Build natural language interfaces using types.
https://github.com com/
microsoft/TypeChar. b.
Wenjie Mo. Jiashu Xu, Qin Liu, Jiongxiao Wang, Jun Yan, Chaowei Xiao, and Muhao Chen. Test- time backdoor mitigation for black-box large language models with defensive demonstrations. In Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, 2025.
Model Context Protocol. What is the model context protocol (mcp)?
https://
modellontexxprotoool... http:/ppootooolioooo//////////te io/docs/gettinggggg..e.oo///c
Vikram Narayanan, Tianjiao Huang, David Detweiler, Dan Appel, Zhaofeng Li, Gerd Zellweger, and Anton Burtsev. Redleaf: Isolation and communication in a safe operating system. In USENIX Symposium on Operating Systems Design and Implementation, 2020.
Bruce Jay Nelson. Remote procedure call. 1981.
Object Management Group. Common object request broker architecture (corba), version 1.1. Techni- cal report, Object Management Group, 1991. httpps//ww// omg. wttpsppcc/COB//LL.....
Katsumi Okuda and Saman Amarasinghe. Askit: Unified programming interface for programming with large language models. In International Symposium on Code Generation and Optimization, 2024.
OpenAI. Openai api documentation code interpreter. guides/tools- -ntre//tooles-
OpenAl. Introducing openai o3 and 04-mini, 2025.
URL https::/opeeaa..comm
introducing- 03- and -ww.--n--------------------- o4--o4-
Oracle. Java remote method invocation: 2 distributed object model, 2017. URL https://docs html
Ianna Osborne, Jim Pivarski, and Jerry Ling. Bridging worlds: Achieving language interoperability between julia and python in scientific computing. In International Workshop Advanced Computing and Analysis Techniques in Physics Research, 2024.
Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. Learning adaptive parallel reasoning with language models. In Conference on Language Modeling. 2025.
Fernando Pérez and Brian Granger. Ipython: system for interactive scientific computing.
Computing in Science & Engineering, 9(3), 2007.
Mansi Phute, Alec Helbling, Matthew Hull, Sheng Yun Peng. Sebastian Szyller, Cory Coreelus,, and Duen Horng Chau, Llm self defense: By self examination, 1lms know they are being tricked. arXiv preprint 21X.::2308..700 2023.
Gordon Plotkin and John Power. Algebraic operations and generic effects. Applied Categorical Structures, 11(1), 2003.
14

---

Published as a conference paper at ICLR 2026
Gordon D Plotkin and Matija Pretnar. Handling algebraic effects. Logical Methods in Computer Science, 9, 2013.
PrefectHQ. Marvin:
Ai agents that spark joy. htttps///gthhu..cco////feeee/l/p
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. In Conference on Empirical Methods in Natural Language Processing, 2023.
Python Software Foundation. Extending and Embedding the Python Interpreter. Python Software Foundation, 2025. URL.tttss://dccs pyt.on..hhnooommoo org/3/ www.endiiingeeeenniidii html
Fanchao Qi. Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. In Conference on Empirical Methods in Natural Language Processing, 2021.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a reneral-purpost natural language processing task solver? In Conference on Empirical Methods in Natural Language Processing, 2023.
Adrian Sampson, Werner Dietl, Emily Fortuna, Danushen Gnanapragasam, Luis Ceze, and Dan Approximate data types for safe and general low-power computation. In International Conference on Programming Language Design and Implementation. 2011.
Grossman. Enerj:
Tobias Schnabel and Jennifer Neville. Symbolic prompt program search: structure-aware approach to efficient compile-time prompt optimization. In Conference Empirical Methods in Natural Language Processing, 2024.
Reshabh Sharma, Jonathan De Halleux, Shraddha Barke, and Benjamin Zorn. Promptpex: Auto- matic test generation for language model prompts. arXiv preprint arXiv:2503.0500 055070, 2025.
Zhi Rui Tam, Cheng- Kuang Wu, Yi-Lin Tsai, Chieh- Yen Lin, Hung-yi Lee, and Yun-Nung Chen, Let me speak freely? study on the impact of format restrictions on large language model performance. In Conference on Empirical Methods Natural Language Processing: Industry Track, 2024.
The MathWorks, Inc. Matlab engine api for python. htttp:///github..commmaattt 2025.
matlab-enginn-for-pve
Raja Vallée-Rai, Phong Co, Etienne Gagnon, Laurie Hendren, Patrick Lam, and Vijay Sundaresan. Soot: A java bytecode optimization framework. In Cascon, 1999.
Mandana Vaziri, Louis Mandel, Claudio Spiess, and Martin Hirzel. Pdl: A declarative prompt programming language. arXiv preprint arXiv:2410.191:: 2024.
Thorsten Von Eicken, Chi-Chao Chang, Grzegorz Czajkowski, Chris Hawblitzel, Deyu Hu, and Dan Spoonhower. J-kernel: A capability- based operating system for java. Secure Internet Programming: Security Issues for Mobile and Distributed Objects, 1999.
Andrew Wagner, Zachary Eisbach, and Amal Ahmed. Realistic realizability: Specifying abis you can count on. In Oriented Programming. Systems, Languages and Applications, 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026.
Brandon T Willard and Rémi Louf. Efficient guided generation for 1lms. arXiv preprint arXiv:2307.09702, 2023.
Haoran Xu and Fredrik Kjolstad. Copy-and-parte patch compilation: a fast compilation algorithm for high-level languages and bytecode. In Principles of Programming Languages, 2021.
15

---

Published as a conference paper at ICLR 2026
Xinyu Yang. Yuwei An, Hongyi Liu, Tianqi Chen, and Beidi Chen. Multiverse: Your language models secretly decide how to parallelize and merge generation. In Conference on Neural Information Processing Systems, 2025.
Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. Optimizing generative ai by backpropagating language model feedback. Nature, 639(8055), 2025.
Lianmin Zheng. Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, al. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Prucessing Systems, 2024.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In International Conference on Learning Representations, 2022.
16

---

Published as a conference paper at ICLR 2026
A EXAMPLE DETAILS
A.1 FULL MANUAL IMPLEMENTATION PROGRAM
from pydantic import BaseModel
from praperr import 11m
class Graph:
nodes: set [int]
edges: dict [int, set [int]]
class EdgeSchema (BaseMoeel:
int
10 tgts: 1ist [int]
11
12 class GraphSchema (BaseModel):
13 nodea: 1ist [int]
14 edges: list [EdgoSchema]
15
18 def serialize(g: Graph) str:
17
"nodes": lide().noees),
19 "edges": [("sre": erc, "tgte": (igtt) for are, tgts in edges items ()]
20
21
return (s)ps()))
22
23
def reify(gs: GraphSchema) -> Graph:
24
modes set(ge ngddees
25 edges fe.arc: tg.t)) for e in ga.
28
return Graph (nodes=nood,,,, edges=edges)
28
def main():
29
graph Graph(...)
30
while True:
31
query (nput("Q: ")
32
33
class OutputSchema (BazeModel):
34
reeponse:
Optionsl [str, int, bool] fraak_flag: bool
35
36
graph: Optional [GraphSchema]
q_out 11m(
39
1" """"Perform the <query> with respect to <graph>, where nodes paper IDs and edges point from cited paper to set of papers that cite it. Return "break_ flag" as True if the <query> indicates termination. Else, return "response". If graph was updated, return as "graph". <query> qqurr>)<<query>
40
41
42
43
44
<graph> (ser output_ tutputt.ypo--uttputtchmm.
45
q_out. break_flag break
49
response q_out. response
50
if g.out.graph.
51
graph reify(q_our (quttttggu.uueooooooo graph)
52
53
print (f"A: [response]")
54 main()
A.2 EXAMPLE PROGRAM INPUTS
'Give
papers 'Does paper 23 directly/indirectt get cited by paper 4.' paper paper "Update the graph paper cites paper 14."
paper
many papera
'Renove paper from the graph complesely." please.'
17

---

Published as a conference paper at ICLR 2026
A.3 EXAMPLE TRACE
Rffect: eva (ra(((er((ypp((qurr))))
Result: *<class 'str'>"
Effect:
(quer)
Result: "Update the graph so paper 5 cites paper 14."
Effect: eval (str(tta
Result: *<elase '<run_pth>>> Graphi>"
Effect: eval (aaaaatttrrgrrap "__doc__'))
Result: True
Effect: oval
Reault: directed graph. dictionary of source node value to set of target node values."
represented
Edges represented by
Effect: eval (dIr
Result:
[I_.ssnotaation__..
clacs_.. containne__. deleterr dict '__dir_..
doc_.
1'-99--' firstlineno_ "fommat._.' -- getatttribure getstatte..... st
bash.__ init_ init_aubelass__'. "__o__ lan_
nodule
00 _e__", "__nex__'' reduco reduce___r. r..repr aetttttr
aetattr....... aizeof_
_.slotaeeee_.... _ebaalic,etrriuuee.... '..str..' "_eucclaessoooo_. weakreff 'edges'
moder") '}
Effect: (typeeer
Result: *cclase 'ses!>"
Bffect: (ral( (frerpr)))))))))))eeee)))))
Reault: *<class 'dict!
Effect:
(graaph nodee))
Result:
2, 4,
7, 9,
11.
12,
16.
21,
22,
24)"
Rifect: eva (gges).
Result:
*(16: (4, 10, 11, 17, 19, 21. 12: (1,
5, 6,
7,
9,
10,
11,
14.
15,
17,
18
19.
23, 24), 9: (1.
3, 5, 6,
20,
21, 22, 23)
10,
10,
15,
17,
20)
18:
(0,
3, 6,
17, 20, 21, 22, 23), {3, 4,
11, 1, 13, 14, 20,
21, 22,
243
5:
12,
3,
7,
8
11,
13,
14, 21, 23,
242, 10:
7, 8, 11, 28, 24), 11: (1, 3, 17, 23), 17: 12, 3,
22,
23)
1:
(3,
4,
8,
20,
22). 14:
(2, 7, 13, 24),
13),
124,
13
7,
20,
23
24
0,
{24, 8, 7), 24: (0, 3, 20), 8: (0, 3, 20), 3: [203)"
Offect: exec(N Planning phase: understand the query and how to execute it
The query "Update the graph paper cites paper 14.
In the graph, adges point from cited paper to papera that cite it if paper
edge 14 Looking at current edges, 14 -> (2. 7, 8, 13, 24)
paper 14,
there should
need
the
papers cite paper
nj...houghht
The quary asks update graph paper 5 cites paper 14. In this graph representation, adgea point from cited paper to citing papars. paper Currently, paper 14 has edges to (2, 7, 8, 13, 24).
So paper
edge
Ne
add
The strategy in:
1. Check if is already in graph.edgea (it 1s)
2. Add the adapn
3. Set response to indicate success
will
une)
Result:
exec Execute the update: paper cites paper 14 Thie neans adding to the set papers that cite paper graph add(5)
Effect:
response confirm update zesponse "Updated")
Reault:
Effect: eval (grapp..egee[(44)
Result: (2, 5, 7, 13, 24)
Effect: done()
18

---

Published as a conference paper at ICLR 2026
Rffect: (eype(u(())))) *<class 'str'>"
Result:
Efrect: eval (quary)
Result: "Exit, please."
Effect: eval (str (sre((()p((grrap))
Result:
'<run_path>, Graap''
Effect: eval (gaaat '._doc_.'))
Result:
Effect: eval (graaphhh....oc.....
Reault: directed graph. Rodes are represented by set node valuee. Edges are represented by dictionary source node value to Effect: (val (dir (graph)))
node values."
"'..smnotatione.. class__.. containee...
dict..', dir....'' doc_.
fornat.. '--80.- ...ttaatribbuee.. ...gettato.... --
bosh init__' ..b.laaauells..... '__le_"' lep___ I....... no_ule.
'__nox__'' reduce...' reduce_ "..repr._'s aetattr aizeor
__ootteeee_... sttrsacc "..str...', "..eubclassbooo... "...eekref_."' 'edgee". !
Effect: (treerrgp
Result:
'set'>"
Offect:
eval eral(errttpppgggrp..eeooo
Reault:
'dict!>"
Effect:
(xec thought "The query 'Exit, please. which indicates termination. need break according the instruction. The graph is Graph object with nodes (set) and edges (dict) representing citation relationchips where edges point from cited papers citing papers.")
Result: Success
Bffect: break()
19

---

Published as a conference paper at ICLR 2026
B NATURAL FUNCTION INTERFACE SYNTAX AND SEMANTICS
B.I NATURAL FUNCTION INTERFACE
Host Values
Serialization a:
Reification Y u/ + Prompt Evaluation nat UN + V op
Prompts
7
NFI Values
UN C
NFI Effects
Op
NFI Handler Clauses
== finally I - e
exit NFI handler op N $ f
xp (2) e:
NFI Handler
by == handle {n)
Program
CH by (nat((rrr (n, a a(v)))
host language prompt
Figure 4: Natural Function Interface.
E =
[] I.. hN (n)) l E (E, hN)
k =
finally + hN (p. o. hN (n)(v)) (p|z + v).0,c)
FINALLY
(pw(2) + e E hx (p, hw (E JOp (x)Ii 8p/w + resume (E,AN)),o,e)
HANDLE
RESUME
(p (F, hN) (p o, hw (F)(F(
NAT-EX (n 0) (n' Np
(n 2) (n' uN) (p, o, nat(n, v)) (p UN)
NAT-RET
(p, a. nat v)) (p. or nat(n', opx))
Scope
p: X A
Heap A
Figure 5: Operational semantics of the Natural Function Interface. The represents LLM call prompt.
B.2 NATURAL FUNCTION INTERFACE FOR TooL. USE
Tools
JEF
Host Values n
Serialization
a W >> on
Reification
7: UN +
Prompts
Prompt Evaluation nats uN v op
NFT Values
UN
NFT Effects
opx Call(f,, NN)
f
NFI Handler
hx == handle {
finally uN - (wx);
Call(f, uN) resuoe a(f(((v))))
call tool
}
Program
=:=
CH
host language
hy (natr((n (v)))
prompt with tool use
Figure 6: Natural Function Interface for tool use. The host system or the programmer supplies a list of available function/tool names to functions for prompts to use, denoted as F.
20

---

Published as a conference paper at ICLR 2026
B.3 NATURAL FUNCTION INTERFACE WITH ISOLATED PROGRAM STATE
Host Values
Scrialization 0: + UN Reification T: Uy +
Prompts n
Prompt Evaluation nats x UN + v op
NFI Values
NFI Handler
by = handle {
finally UN ((x);;
Program
CH
host language prompt
hw (nat(n, a())))
Figure 7: Syntax of the Natural Function Interface without shared program state. Prompts only take constant value and returns a constant value. There are no program state effects, so the NFI handler no-op.
21

---

Published as a conference paper at ICLR 2026
C
BENCHMARKS
Table 3: SPSBench programs with their descriptions.
Program
Description
Python-
Specific
Features
avg ofg_of_the three
Average three input numbers, represented numbers and words in multiple natural languages
balance_ parenthess
Semantically halance parentheses in an expression based on context (e.g. order of operations or regex) Updating list of objects based on natural language description of player action.
battle
calendar
Object state mutation using only public methods. Shows how to regular tool use with Nightjar. Adapted from Microsoff Creation of object of predefined dataclass
character_builn buildder coffceshop
Parse input into items in a coffee order, limited to the options if imput includes unparseable
available in the shop. Throws items. Shows how to Structured output with Nightjar. Adapted from Microsoff
maail_classification
Classify email with enum and raise exception if no fitting category. Adapted from Anthropic (ak: PrefectHQ
enemy
Dynamic object creation and manipulation.
fact_checking
Gathers claims from string and appends fact-check results to a list
in place
feed_ jed__fiterrng
Functional string list filtering based preference
filter_numbers
Filter list of numbers by semantic property
get_ enemy
Closure creation returning object of unspecified class
graph
Program shown in Figure lb with natural language queries
gsm8k
Using eval to solve gsm8k (Cobbe et al. 2021 math problems
health_date
Chatbot that helps the user input health data. Adapted from Mi-
crosoft (b)
mask_sensitive_int
Create string with sensitive information masked with asterisks Object state mutation to update inventory
menu
pos_tagee jos_tagiin
Dynamic closure creation for translation generation based on input language
saposer_verifie
Proposer-verifiee loop on riddles, using only functional generations
sentiment
Sentiment analysis. Adapted from Chase 2022) Microsoft (b):
OpenAl PrefectHQ)
stores
Create subclasses based on input, and instantiating these subclasses.
Adapted from Chase (2)22) Microsoft] (b)
stream_ _estam__eerttorrr
Create generator relevant to ippu
task completion
Mutate object state to mark tasks as complete
tiny bookshop
Retrieve book for user based on request
wordle, solver
Nested function that prompts solve wordle
22

---

Published as a conference paper at ICLR 2026
D SYSTEM PROMPTS
NIGHTIAR system prompt:
helpful assistant.
<goal>
Execute instructions efficiently accurately possible tool as poseible, folloving the Execution Protocol.
</goal>
<ayntax?
Python variables being used inside variable> to denote variables that can be used in the Python code after the block. Do not use the using The instructions night also include the type and values (and/or references to values) to ingut variables "[(var?) [type: ((type_name)]3:: ((val_ref)) (Value: [[val_11 immitabla)))".
natural instructions <variable>
the
</syntax>
cexecution, protocol>>
Follow this protocel to execute the instructione:
Discorery Phase: Explore
type, values, and attribates are not already given in the natural instruction. Skip this phase if the information already given in the natural instruction.
understand
structures
Do inspect (via "eval") type, values, and attributes already given in the natural instructions.
Tip: (stp(t))e(varr)" get type variable.
Tip: Reference _odd.__"' attribute for docunentation.
Planning
strategy to with the least number of tool calla and tokens. Only use Python to perforn computation you don't can't calculate it directly.
instructions,
highest accuracy
3. Execution Phase: Execute the originally
natural instruction
few tool calls and fex
tokens as possible.
Tip: you
already without any computation (remenber, you're really agent with common sense world knouleige and reasoning capabilitier) directly use the answer you know. already
For exampla, da
Tip: Any Fython builtina are also in the heap and can be used
Tip: Do not use any nonstardard Python libraries.
Tip: "eral" only
immutable values (strings, integers, numbers, booleans, None). Everything else (including lists, dictionaries, tuples, etc.) is returned as an object reference. Use "ett" into string (e.g. "str([x specific attribute.
serielize
my_list))") read use 'getattr'
Python statements. 4. Finish Phase: Use "continue", "break, "return", or "done" to finish the execution, following the instructions.
Tip:
only evaluates Python expressiona.
REQUIRED: Use 'continue" if and only the instruction says 'contimue" and the conditions for continue" are ust.
REQUIRED:
only instructions says and
"break"
are met.
REQUIRED: Use "return" and only if instruction saya the word "return" and the conditiona for "return"
REQUIRED: Othervice use "done'
REQUIRED: When using "raise" label, make the "val" is Exception object
You are failure if you cheose the krong tool to use.
Tip:
object
"eral' object
not use "id"
</execution_protoool>
23

---

Published as a conference paper at ICLR 2026
NIGHTJAR (Baseline) System Prompt:
You are a helpful seeistant. Please compute the following instructions using the provided toole to interact with the context.
Execute natural instructions officiently and accurately possible in for tool calls as possible, Protocol. Required stepa alsays folloved.
follosing
Syntaz
Nightjar versien of Python that allovs <nstural> urapped natural language instructions in the code. The <ontural> block uses evariable> to denote Python variables baing used inside the block and <: Dnly variables
the Python code after donoted as <ivariable> can be used outside of the <natural> black it belongs to. <natural> block doea not return anything. Valuea have to aasigned to variables accessible outside the block.
variable> denote
be
Execution Protocol
YOU MUST follos this protocol to execute the instructions in order and by the lettar: band. Follow
Discovery
Explore
required steps:
Required: Look at the input variables see what objecta the point to.
Required: Inepect the nested structure to ensure all the actions you will take valid.
Required: Look at attribute type annotations and doc__. to see chat's expected attributes of object
2. Planning Phase: Plan out the best strategy to execute the inatructions.
current strategy is taking tea tool
Think
you
strategies the tool calls. Alvays use the least mumber of tool calls possible.
the
Estimate the number of tool calls you will need execute the instructions. Pick the strategy that will take the least number of tool calls.
3. Execution Phase: Execute the originally given natural instruction in as fez tool calls as possible. initial instruction confirm you executed the instructions correctly. Inspect errora to see shere you went wrong.
Reflection Phase: Reflect
instruction,
execution, plan out
stepe.
Check if you have made nistake.
If you have made a mistake, go back to the Discovery Phase repeat the procees. If you have
fulfilled every piece of the instruction, back to the Discovery Phase and repeat the process.
inatructions saya error uessages from tools and do whatever you can to perform the instructed computation, whatever incoupatible data types. LLN capabilitiee.
Avoid "raise"
You ahould alvaya try address
about dealing
about the semantice values
rather than abide rigid rules. Use
4. Finish Phase: Use "goto" or "done" to fimish the execution, depending on the instructions.
REQUIRED: Use 'goto" with program label "continue" if and only the instruction says "contimue"
and the conditions for using "continua" in the instructions are nat.
"goto" with program the conditione for using 'break" in the instructions are net. only
REQUIRED:
"break"
only if instructions says "break" and
instruction saye word return" and the conditions for using 'return" the instructions are net.
REQUIRED: 'goto'
program
REQUIRED: Othervice use 'done.. . You must use "done" for output variables to be written REQUIRED: When using "raise" label,
the "val" is Exception object You are a failure if you cheese the vrong tool to use batzeen "goto" and "done".
Tips for execution
Alvays pick
strategy execute
correctly.
Never assume the data structure of the objacts are working with.
Feel free to perforn actions in place, unless the instruction specifies othervise. the resulta your conputations the context and assign all variables the correct references as instructed for the output variablea before you continue to the next instruction.
Nake
Strings, integers, fleats, boolean, Nonetype, tuples are imutable data types.
Dictionaries, lists, sets, objecta, classes are mutable referenced.
allocated
heap
The names of classes are not surfaced as variables; the referance of the data must be assigned to referred by variable
variable
Deref classes and objects to inspect their attributes
The generic object class
available
"Object" is not a valid type acnotation, just put "Any" you're 'r for object, then you must sane.
object. Objacts and dictionaries not the
"done' given an arror, fix the issue. it mays a variable is undafined, define it. types ("MoSBpportedDaaatyppee" either find different strategy
coning acrosa unsupported that doeen't t. use that data or rsise an error. try the
Address errors,
and over again.
## Tool Call Limits
You only fnax, tool_ callec total tool calls. Use few possible.
24

---

Published as a conference paper at ICLR 2026
NIGHTJAR (Shared Python) System Prompt:
You are a helpful seeistant. Please compute the following instructions using the provided toole to interact with the context.
Execute natural instructions efficiently and accurately possible in for tool calls as possible, Protocol. Required stepa alsays folloved.
follosing
Syntaz
Nightjar versien of Python that allovs <nstural> urapped natural language instructions in the code. The <ontural> block uses cvariable> to denote Python variables baing used inside the block and <: Dnly variables
the Python code after danoted as <ivariable> can be used outside of the <natural> black it belongs to. <natural> block doea not return anything. Valuea have to aasigned to variables accessible outside the block.
variable> denote
be
Execution Protocol
YOU MUST follos this protocol to execute the instructions in order and by the lettar: Discovery
Explore
band. Follow
required steps:
Required: Look at the input variables see what objecta the point to.
Required: Look at what methods and properties
have Required: Look at the object type (e.g. "sr((yye
to understand what they are. Required: Look the "__doc__"" attribute the objact understand what the objects
Required: Inspect the neated atructure to enaure all the actions you sill take (vith toola or with Python code) valid.
2. Planning Phase: YOU MUST ALLOCATE A THOUGHT PROCESS STRING to plan out the best strategy to execute
the instructions.
fever tool calls to achieve the effect. Pivot strategies if the current strategy is taking too many teol calls. Always use the least number of tool calls possible. instructions. the strategy
Think about how you
Katinate nunber of tool calls you will need
that will take the least number of tool calls.
strategy to 3. Execution Phase: Execute the originally given natural instruction in as feu tool calle as possible.
Required:
plannings thought process string figure
the Instructions. And asaign this string the variable thought""
Follow the folloving steps. They
in order of priority:
1) If the instruction is to do sonsthing in Python, run the Python code.
11) you
already without any computation (remenber, you're really agent with comon-sense world knowledge and reseoning capabilities) directly give the ansver with assigns.
111) Look at the initial instruction to confirn you have executed the instructions correctly.
iv) Inepect errers to see where you vent wrong.
4. Reflection Phase: Reflect the instruction, the execution, and plan out the steps.
Check if you have made a nistake. you
repeat process. If you have not fulfilled every piece of the instruction, back to the Discovery Phase and repeat process.
mistake,
Discovery
repal Avoid "raise" errors unless the instructions saye do You should alveye to addrees error nossages from tools and do whatever you can to perform the instructed computation, whatever takes. Be clever about dealing with incompatible data types. Think ahout the sonantics of values rather than abide by rigid rulea. Use your LIN capabilitien. Finish Use "continue", "break, instructions. Use "contimue" if and only if the instruction says 'contimue" Use "break if and only instructions says raturn". Othervise, you must use "done..
or "done" finish execution, depending the
"return' only instruction says
25

---

Published as a conference paper at ICLR 2026
Taps execution
Alvaya pick the most efficient strategy to execute the instructions correctly.
Tasks/subtasta doean't need LLMa (1.0, can done easily correctly Python) ahould be done in Python code.
Never assune the data atructure of the objacts you are sorking sith.
perforn
place, unless the instruction specifies othervise.
Nake sure to store the results of your conputations in the context and assign all variables to the the output variables
you
instruction.
You can use python's a "type" claes/function to get the type of object.
Aay Python builtins also in the and can be used
Do not use any nonstandard Python Libraries
valuee (strings, integere, numbere, booleane, None). Everything elee including lists, dictionaries, tuples, etc.) is returned as an object reference. To inspect objects 'repr"
only
getattr" get apecific
the object string
When using "eval"
claes BaseModel, you should look at its echena by calling "etr" on the reculte of calling model_json_schens to understand the schena. Then
the JS0K string (adhering the schena). Then, use podel_valldate.. to validate the atring into the Bazellodel Address errore,
try
over again.
Limita
You only (max_ tool_callst total tool calls. Use few possible.
26

---

Published as a conference paper at ICLR 2026
NIGHTJAR (Isolated Eval) and (Shared Eval) System Prompt:
You are a helpful seeistant. Please compute the following instructions using the provided toole to interact with the context.
Execute natural instructions efficiently and accurately possible in for tool calls as possible, Protocol. Required stepa alsays
follosing
Syntaz
Nightjar versien of Python that allovs <nstural> urapped natural language instructions in the code. The <ontural> block uses evariable> to denote Python variables being used inside the block and <: Dnly variables
the Python code after donoted as <ivariable> can be used outside of the <natural> black it belongs to. <natural> block doea not return anything. Valuea have to aasigned to variables accessible outside the block.
variable> denote
be
Execution Protocol
YOU MUST follos this protocol to execute the instructions in order and by the lettar: Follow
Discovery
Explore
required steps:
Required: Look at the input variables see what objecta the point to.
Required: Inepect the nested structure to all the actions you will take valid.
Required: Look at attribute type annotations and doc__. to see chat's expected attributes of an object
2. Planning Phase: Plan out the best strategy to ezecute the inatructions.
current strategy is taking tea tool
Think
you
strategies
tool calls. Alvays use the least mumber of tool calls possible.
Estimate the number of tool calls you will need execute the instructions. Pick the strategy that will take the least number of tool calls.
3. Execution Phase: Execute the originally given natural instruction as fez tool calls as possible. initial instruction confirm you executed the instructions correctly. Inspect errora to see shere you went srong.
Reflection
Reflect
instruction,
execution, plan
stepe.
Check if you have made nistake.
If you have made a mistake, go back to the Discovery Phase repeat the procees. If have fulfilled every piece of the instruction, back to the Discovery Phase and repeat the process. Avoid "raise"
inatructions saya error uessages from tools and do whatever you can to perform the instructed computation, whatever dealing incoupatible data types. rather than abide rigid rules. Use your LLM capabilitiee.
ahould alvaya try
about the semantice values
4. Finish Phase: Use "contique" "break, 'return', or "done" to fimish the execution, based on shat the instructions Use "contimse" if and only if the instruction says "continue". Use "break" if and only if the instructions saya "break". Use "raturn' if and only if the instruction says the word "done".
you
REQUIRED:
When using 'raise" label, make sure the "val" is an Exception object wrong
you Tip: "return" and "raise" take variable
use.
27

---

Published as a conference paper at ICLR 2026
Taps execution
Alvaya pick the most efficient strategy to execute the instructions correctly.
Never assune data structure the objecta you are korking
Feel free perforu actions in place, unless the instruction specifies othervise.
Nake aure to store the resulta of your corpatations in the context and assign all variables to the the output veriables
you continue the
instruction.
Strings, integers, fleats, boolean, Nonetype, tuples
types.
Dictionaries, lists, sata, objacta, classes are mutable and mast be allocated on the heap and referenced.
The nanes of classes not surfaced variables; the reference of the data must sesigned to variable to be referred to by a variable nane.
objects inepect The generic object class is not available
Deref
"Objact"
type arnotation, just put "Any"
you're asked for object, then you must derine object. Objecta and dictionaries are not the zane,
class BaseModel, you should look its schena calling "str' the results of calling model__joon_scle achena" to understand the schena. Then create the JSOK string (adhering to the achena). validate If "done" gives error, fix the issue. If it string variable is undefined, define it. types ("otllsportadaatatyyy"" ), either that doeen't 't use that data raise error.
Then, 'model_ valellleliate_eeee
BazeModel.
coning across unsupported
strategy
"eval" cannot execute Python statenents, everything muet only one line Python expression. This import, assiguments.
Address the errore, do not try the same tool over and over again.
## Tool Call Limits
only get fmax_ call_ tool calla. Use possible.
28

---

date-tine Published as a conference paper at ICLR 2026
Prompt for LLM Code Generation:
You are a Python programer.
Your tack is implenent the Python code replace the natural language
The comment will specity variable references with the syntax <variable> and variable definitions with the syntax <: Only return code for the particular comment that is being replaced. Do not regenerate any of the other code in assune any code what shoun
variable>,
sure
definitions
generated
Python
source coda. Do exista. The final code should be fully implemented valid Python progran.
any unimplamented.
The code make of the 'nj_1lm" function, which nakes call LLM. take JSON schens as output, using only features enabled by OpenAI's structured outpat J80N schema docunentatioon.
the function signature the "nj_lln" function:
iinpopppttt:: str, output_ Optional [Dict] None)
The JSON schena be the format: {{ "type": "json_schema", "json_schema": {( "strict": "nane "..., "schana": ... }3 }}
Supported J808
Supported types
following types are aupported for Structured Outputs:
String
Number
Boolean
Integer
Object
Array
Enun
anyOf
Supported properties
In addition specifying the type proparty, you specify selection additional constraints: Supported string properties:
pattern regular expression string
fornat Predefined fornata for strings. Currently supported:
time
date
duration
etail
1pv4
ipv6
uuid
Supported nunber propartias:
miltipleff The number must be multiple of this value.
equal ixclueiveMarinu. The nunber nust be less than this value. minimum The number must be grester than or equal to this value. exclusivaMinimu. The nunher be Supported array proparties:
than this value.
minItens
The array must have at least this many itens. many itens.
array must Objects have limitations on nesting depth and size schema may have up to 5000 object properties total, with up to levels nesting.
Linitationa on total string size schems,
string length cannot exceed 120,000 characters.
property names, definition nanes, values,
Linitations enun size schema may have to 1000 enu values across all emun properties.
For single enun property with string values, the total string length of sll enun values cannot exceed 15,000 characters
more
emm values.
additional Propprrtierrrrr additiooallPopprtteee controls whether it is allovable for object contain additional keys / values that were not definad in the JSOK Schana.
alvays
objects
Structured Outputs only supports genersting specified keye / valuee, 8o we require developers to set indlppiieeeee to opt
Dutputs.
29

---

Published as a conference paper at ICLR 2026
Key ordering
When using Structured Dutputs, outputs will be produced in the same order the ordering of keys in the schema.
Sone type-spacific kayoorda are not yet aupportad Composition: sl10r, not, dependeutRequired, dependentScheuss, then, For fine-tume tuned models, additionally do not support the following:
For strings: ninlength, maxlangth, pattern, format For numbers: ninima,, naximun, multipleer
For objects: pattermProperties
For arrays: ninitems, naxItens
you Structured Outpute by supplying strict: True Schena, will receive an error.
API unsupported
Example Schema: (<"type": "Jaon_schema", "jaon_achema": (("name": "math responsee" "schena": {("type":
object", "properties": (("steps"; {("sype"; "array", "itens"; {("type": "object", "propertiee": (("
explanation': (("sype"; "string")), "output"; (("tyee": "string")) ),, "zequired"; ('esplanation", output"], "additionaaPrgeetieee"" Falee)), "Tinal "Tinal anaver"; (("type"; Wetring">,, "required"; [" stepe", siinel_ "sdditienlPPopertiee"" False)), "etrict"; Irwe))1))1133 (source_code)
language [natural_text]
replaced:
30

---

Published as a conference paper at ICLR 2026
E ADDITIONAL EVALUATION RESULTS
E.
AVERAGE PASS RATE BY BENCHMARK
Sonnet
1.0
Average Pass Rate
0.6
0.4
0.2
0.0
average, three
balance parenth eree
tiny booksthpp
battle
calendar tool
gomik
charaster builder
email classification
coffesstop
fact, checking
enemy
feed fitering
litter rumbers
get enamy
mask sensitive informatio
graph health data
pos, tagiging
menu
proposer veriieer
sentiment
stream, generator
stores
task, completee completion
wordle salver
Benchmark
Manual Manual (Coda Interpreter)
Nightjar (Naive)
Nightjar
Figure 8: Average pass rate over runs for each benchmark in SPSBench with standard deviation for each method using (aude=SSomett-22025555
GPT-4.1
1.0
Average Pass Rate
0.8
0.6
0.4
0.2
0.0
average_o of three balance gerentheres
battle tiny bookshop
calensar tool
gimdik
character. builder
email classification
coffeestop
fact, checking
enemy
feed, fitering
filter numbers get enemy
mask sensitive information
graph health data
pos, teeging
menu
proposer verifeer
sentiment
stream, generator
stores
(mok completion wortte salver
Benchmark
Manual
Nightjar (Naive)
Interpreter)
Nightar
Figure 9: Average pass rate over runs for each benchmark in SPSBench with standard deviation for each method using GPT-441-2025-044,
E.2 SHARED PROGRAM STATE IMPLEMENTATION ABLATIONS
We performed ablations of the optimization features we implemented in NIGHTJAR. We evaluated the following system designs:
31

---

Published as a conference paper at ICLR 2026
NIGHTJAR (Baseline): Baseline implementation of NIGHTJAR using the shared program state NFI, without any specialization or optimizations.
NIGHTJAR (Isolated Eval): Baseline implementation of NIGHTIAR with Python computation enabled using IsolatedEval evaluate Python expression with isolated state. It also uses Python-specialized control flow tools. The effects are IsolatedEval, Lookup, Assign, Deref, Ref, Set, Break, Continue, Return, and Raise.
NIGHTJAR (Shared Eval): Implementation of NIGHTIAR with Python computation enabled using Eval evaluate Python expression on the shared program state, replacing Lookup and Ref. It also uses "yyton-specialiin control flow tools. The effects are Eval, Lookup, Assign, Deref, Set, Break, Continue, Return, and Raise.
NIGHTJAR (Shared Python): Python-specialized implementation of NIGHTIAR. The effects are Eval, Exec, Break, Continue, Return, and Raise.
NIGHTJAR (Shared Python, Caching): Python-specialized implementation of NIGHTJAR with caching.
NIGHTJAR: Our main implementation of NIGHTJAR, specialized to Python and with the eager variable loading optimization, and caching.
LLM Code Generation: An alternative approach realizing the shared program state abstrac- tion by using an LLM to statically generate Python code (that can contain LLM API calls) to replace the prompt given the source code of the program.
shows the average pass rate with standard deviation, the average runtime, compile time, and total time with ranges on SPSBench over runs.
The LLM code generation approach can only inspect the program statically, which means the LLM cannot adapt the computation to program inputs intermediate program values. This results poor accuracy, despite a faster runtime than NIGHTJAR'S 's interpreter approach.
NIGHTIAR (Baseline) does not perform as well as the tions in pass rate because it does not support language-specifee features of the host-language, such as first-class functions and abstract methods. It also only enables L..M-based computation to transform data. Table shows the performance of the ablations and the manual implementation baselines on the subset of benchmarks in SPSBench that do not require these language-specific features. In terms of pass rate the benchmarks that do not need language-specific features, NIGHTJAR (Baseline) is within standard deviation from the manual implementation for GPT-4.1 and slightly below the pass rate of manual implementation for Sonnet 4.
NIGHTIAR implementa-
NIGHTJAR (Isolated Eval) enables the use of Python- based computation rather than rely- ing only LLMM-based computation to transform data. However, it still cannot utilize language-specific data types. For example, the LLM in NIGHTIAR (Isolated Eval) issues the following effects to compute one of the test cases of the average_of three benchmark:
Effect: (p)
Result: "ona*
Effect: Lookup(q)
Result: "two"
Offect: Loskup(x)
Result: "three"
Effect: (aolatedEvall(("onn
"tuo": "three": 3) 30.get ("onna""", 0) {"ona": "tuo": "three": 3).
get("buo", 0) {"one": ; 1, "two": ; 2, "three": """get("thre"", 0)) / 3.0)
Reault: 2.0
The LLM uses IsolatedEval to reliably compute the math expression, but because the scope is not shared, has to use the values of p. q, and r in the Python expression.
In the subset of benchmarks that does not require these data types, for Sonnet 4, NIGHTIAR (Isolated Eval) achieves parity with the manual implementation pass rate. For G NIGHTIAR (Isolated Eval) achieves slightly higher pass rate than NIGHTIAR (Baseline) and the manual implementation with isolated code interpreter enabled. This indicates that GPT-4.1 could not use EvalatedEval effectively but Sonnet 4 could.
The NIGHTIAR (Shared Eval) ablation enables Eval to read from the shared program state. However, the LLM is still restricted to using Python expressions only, rather than freeform Python state- ments, and cannot define Python-specific data types. This enables the LLM to use more compact syntax in the form of Python code to represent sequence of effects. For example, for the same
32

---

Published as a conference paper at ICLR 2026
program and test above, the LLM NIGHTJAR (Shared Eval) issues the following effects:
Effect: [p)
Reault: "one"
Effect: (v)
Result: "two"
Offect: (v)l(r)
Reault:
Effect: (val((('one: 1, 'tvo'; 2, 'three": 37[p] ('one': 1, 'Deo': 'three'; 3+[4] ('one's 1,
tvo': 2, 'three': 3) (x)) 3.0)
Reault:
Specifically, the L.M refers to the variables P. 9, and r directly in the code because the program state shared. Python code also more familiar interface than the novel natural function interface. In the subset of benchmarks that does not require Python-specific features, with Sonnet 4, NIGHTJAR (Shared Eval) achieves parity with NIGHTJAR pass rate, which better than the manual implemen- tations. However, for GPT-4.1, NIGHTJAR (Shared Eval) has slightly lower pass rate than manual implementations and NIGHTIAR. The disparity indicates that Sonnet is more capable utilizing Python expressions with shared program state than GPT-4.1.
NIGHTJAR (Shared Python) improves pass rate by enabling language-specifie features and the use of freeform Python statements, but the runtime overhead is maintained. The Exec tool enables the use of Python statements on top of Python expression with Eval. This enables the LLM to generate freeform Python code that is not subject to constrained decoding, which has been found reduce LLM reasoning performance (He et al. 2024 Tam et al. 2024). Additionally. Python code is more familiar interface than the novel natural function interface. This is shown by NIGHTIAR (Shared Python) performing better than the programmer implementation on the complete SPSBench for both Sonnet 4 and GPT-4.1.
Adding optimizations reduces the runtime. The caching optimization in NIGHTJAR (Shared Python, Caching) only minorly reduces the runtime. This is because the test cases SPSBench programs consist of different program inputs. The only time savings achieved by caching optimization is saving the time to emit the effects used before reading program inputs.
The main implementation NIGHTIAR demonstrates that the runtime overhead of shared program state can be reduced with additional engineering efforts and optimizations.
33

---

Published as a conference paper at ICLR 2026
Table 4: Performance on SPSBench. Average pass rate and runtime with ranges in gray.
Model
Method
Pass Rate
Runtime (s)
Compile Time (s)
Total Time
Effects
Sonnet
Manual Impl
0.78+0.003
8.0
8.0 (1.3-44.8)
(1.3-44.8)
Sonnet
Manual Impl (Isolated Code Interp)
0.66++0.04
36.8
0.0
36.8
2.6
(3.2-171.3)
(0.0-0.0)
(3.2-171.5)
(0-14)
Sonnet
NIGHTIAR (Baseline)
0.60++0.04
55.6
0.0
27.8
(9.6-721.0)
(0.0-0.0)
(9.6-721.0)
(6-300)
Sonnet
NIGHTJAR (Isolated Eval)
0.72++0.06
100.1
0.0
100.1
34.7
(14,9-870.0)
(0.0-0.0)
(14.9-870.0)
(6-30)
Sonnet
NIGHTIAR (Shared Eval)
0.78+0.04
118.1
0.0
118.1
40.6
(10.6:550.6)
(0.0-0.0)
(10.6-550.6)
Sonnet
NIGHTIAR (Shared Python)
0.85=00002
75.8
0.0
75.8
27.3
(10.1-236.4)
(0.0-0.0)
(10.1-236.5)
(2-93)
Sonnet
NIGHTIAR (Shared Python, Caching)
0.84
70.6
0.0
70.6
27,9
(5,5-244.8)
(0.0-0.0)
(5.5-244.8)
(2-106)
Sonnet 4
NIGHTIAR
0.85
25,9
0.0
25.9
8,9
(6.2-83.0)
(0.0.0.0)
(6.2.83.1)
(2:33)
Sonnet
LLM Code Generation
0.3611.006
7.9
7.3
15.2 (1.4-55.6)
(0.0.41.0)
(1.4-17.)
GPT-4.1
Manual Impl
0.74=1..33
4.5
4.5
GPT-4.1
Manual Impl (Isolated Code Interp)
0.72+10.03
50.3
0.0
50.3
(10.9:36499)
(0.0-0.0)
(10.9:365.0)
(1.0-21.0)
GPT-4.1
NIGHTIAR (Baseline)
0.61
39.0
0.0
39.0
24,9
(4.7-957.4)
(0.0-0.0)
(4.7-957.4)
(4-162)
GPT-4.1
NIGHTIAR (Isolated Eval)
0.61100002
51.7
0.0
$1.7
32.3
(7.8:282.1)
(0.0-0.0)
(7. (7.8-282.1)
(6.0-174.0)
GPT-4.1
NIGHTIAR (Shared Eval)
0.62:10.08
75.0
0.0
75.0
49,2
(6.9.579.4)
(0.0.0.0)
(6.9.579.4)
(6.0-289.0)
GPT-4.1
NIGHTIAR (Shared Python)
0.81
71.8
0.0
71.8
63.1
(4.0-417.0)
(0.0.0.0)
(4.0-417.0)
GPT-4.1
NIGHTIAR (Shared Python, Caching)
0.78++0.03
66.2
0.0
66.2
68.6
(7.9-406.1 1)
(0.0-0.0)
(7.94406.1)
(7-300)
GPT-4.1
NIGHTJAR
0.78+10.03
19.6
0.0
19.7
16.6
13.0-72.1)
(0.0-0.1)
(3.1-72.1)
(2-9)
GPT-4.1
LLM Code Generation
0.45=0002
2.8
5.0
7.7
(0.0-3188)
(0.9-23.7)
(0.9-36.9)
34

---

Published as a conference paper at ICLR 2026
Table 5: Performance results of subset of SPSBench that does not require first-order functions or abstract methods. See Appendix for which programs are in this subset. Average pass rate and runtime with ranges in gray.
Model
Method
Pass Rate
Runtime (s)
Compile Time (8)
Total Time (s)
Effects
Sonnet
Manual Impl
0.78
8.2 (1.3-448)
(1.3-44.3)
Sonnet
Manual Impl (Isolated Code Interp)
0.63:10.04
42.5
0.0
42.5 (3.2: 171.5)
3.1
(3..57
(0.0.0.0)
(0
Sonnet
NIGHTIAR (Baseline)
0.69:10.05
57.8
0.0
57.8
29.6
(9.6/721.0)
(0.0-0.0)
(9.6-721.0)
(6:300)
Sonnet 4
NIGHTIAR (Isolated Eval)
0.76=t0.04
103,2
0.0
103.2
36,2
(14.9-870.0)
(0.0-0.0)
(14.9-870.0)
(6-300)
Sonnet
NIGHTIAR (Shared Eval)
0.85
104.1
0.0
104.2
36,6
(10.6-164.2)
(10.6-464.2)
Sonnet
NIGHTIAR (Shared Python)
0.83
85.6
0.0
85.6
(21.6-236.4)
(0.0-0.0)
(21 (1.6-336.5)
(7-93)
Sonnet
NIGHTJAR (Shared Python, Caching)
0.83++0.03
75.6
0.0
75.7
30.5
(19.0:244.8)
(0.0.0.0)
(19.0-244.8)
(8-106)
Sonnet 4
NIGHTJAR
0.88+00033
29.3
0.0
29.3
10.3
(9.1-83.0)
(0.0-0.0)
(9.1-83.1)
(4-33)
Sonnet
LLM Code Generation
0.41
7.9
7.5
15.4
(0.0-38.6)
(3.6-17.8)
(3.6-55.5)
GPT-4.1
Manual Impl
0.75:10.04
4.6
4.6 (0.8-15.0)
(0.8-15.0)
GPT-4.1
Manual Impl (Isolated Code Interp)
0.71
52.5
0.0
52.5
3.0
(10.9-36499)
(0.0-0.0)
(10.9-365.0)
(1.0-21.0)
GPT-4.1
NIGHTIAR (Baseline)
0.70.10.03
44,9
0.0
44.9 (4.7-957.4)
27.7
(4.7-957.4)
(0.0-0.0)
(4-162)
GPT-4.1
NIGHTJAR (Isolated Eval)
0.72+00004
55.0
0.0
55.0
35.3
(7.8-282.
(0.0-0.0)
(7.8-262.1)
(6.0-174.0)
GPT-4.1
NIGHTJAR (Shared Eval)
0.70+0.00
71.7
0.0
71.7
48.6
(6.9-579.4)
(0.0-0.0)
(6.9-579.4)
(6.0-289.0)
GPT-4.1
NIGHTIAR (Shared Python)
0.79=10.05
79,9
0.0
79.9
67.2
(11.4-417.0)
(0.0-0.0)
(11.5-417.0)
(12-299)
GPT-4.1
NIGHTIAR (Shared Python, Caching)
0.76=t0.05
68.4
0.0
68.4
70,9
(11.2-406.11 1)
(0.0-0.0)
(11.2-406.1)
(11-300)
GPT-4.1
NIGHTIAR
0.76+0002
22.7
0,0
22.7
18.3
(3.3-72.1)
(0.0-0.1)
(3.3-72.1)
(2-96)
GPT-4.1
LLM Code Generation
0.48.10.06
0.3
5.0
5.3
(1.6-13.4)
E.3 LOCALLY HOSTED LLM
We evaluate the performance of NIGHTIAR using smaller, locally hosted model as the base LLM on SPSBench. We used GPT-OSS 20B served through VLLM (Kwon et al. 2023) on an A100 80GB GPU. Table/Sshows the average pass rate with standard deviation, the average runtime with ranges, and the average number emitted effects with ranges over runs. Using GPT-OSS 20B, the pass rate of NIGHTJAR lower than the manual implementation. This is because the NIGHTIAR implementation relies heavily on the agentic capabilities of the base LLM. Further system prompt engineering, optimization, finetuning could improve the pass rate when using smaller model. Nevertheless, NIGHTJAR is still more performant than using an LLM to statically generate Python code (LLM Compiler).
Table 6: Average pass rate, runtime, compile time, and number of effects using GPT-OSS 20B with ranges shown in gray.
Method
Pass Rate
Runtime (s)
Compile Time (s)
Total Time
Effects
Manual Impl
0.68-1004
5.3
5,3
(0.3-80.2)
(0.3-89.2
LLM Compiler
0.13+00.0
1.2
4.3
5.3
(0.00-55.24)
(0.39-16.46)
(0.39-57.75
NIGHTIAR (Shared Python)
0.47=00.3
54.2
0.0
54,2
28.1
(3.2-581.8)
(3.2-581.8)
NIGHTIAR
0.40-10.02
36.5
36,5
8.0
(1.4-385.7)
(0.0-0.0)
(1.45-385.68)
(0-16)
35

---

Published as a conference paper at ICLR 2026
E.4 TEMPERATURE 0
NIGHTIAR uses closed LLMs and temperature for performant LLM agents, which means program executions may be -on-metermiiittcc. We performed an ablation of the evaluation with temperature of 0 on GPT-4.1. Table shows the average pass rate with standard deviation and average runtime with ranges over runs. As shown, NIGHTIAR also achieves parity pass rate to manual implementations with temperature O. We note that even using temperature 0 does not result in deterministic executions because the base LLM is a closed model (He Lab 2025). Using temperature on a locally-hoster open model and controlling for LLM non-determinism (He & Lab 2025) or caching and re-using responses to the same effect trace would enable deterministic NIGHTIAR executions.
Table 7: Average pass rate and runtime with ranges in gray.
Model
Method
Pass Rate
Time (s) Effects
GPT-4.1
Manual Impl
0.7410.01
5.0
(0.8-23.0)
GPT-4.1
NIGHTIAR (Shared Python)
0.73
82,9
79.9
(3.4-548.6)
(4.0:300.0)
GPT-4.1
NIGHTIAR
0.77=+0.04
28.9
20.2
(2.9-267.3)
(2.0-149.0)
E.5 LARGE DATA STRUCTURE
The representation of program data as they are accessed by an LLM factors into the program performance enabled by programming system. There are two methods pass program values to and from LLMs: pass-by-copy and pass-by-reference.
Program values are typically encoded as strings in JSON format and included in the prompt context for the LLM to process. This method is called pass-by-copy. It is intractable when computing with large objects, because LLMs have a limited context length. The length of the encoded object string linear to the size of the object, so sufficiently large object would be beyond the context length of the LLM. This also the case for updating or writing an object, where the size of objects the LLM can compute with is limited by the maximum output length.
In contrast, pass-by befreeeee avoids the need to encode objects into strings. For an LLM read objects, it first receives reference to the object, which is only few digits in length and constant in the size of the object. During program execution, the LLM inspects only the relevant parts of the object it needs for the prompt evaluation. This is more efficient when the prompt requires the use of only a part of the large object.
We examine how pass-by-reference affects performance compared to pass-by-copy with case study using the Figure pprogr/pssr Figure Ia is implemented manually by a programmer. It uses pass-by-copp by encoding the graph object into JSON format (Line T0p and passing it to the LLM in the prompt (Line (44.. The output graph is also decoded via JSON format and reified into the Graph
object (Lines 11
15
Methodology. We compare the performance of executing the Figure 1b with 6 different natural language queries (Appendix (.2) using NIGHTIAR and NIGHTIAR (Baseline) against the perfor- mance of executing the pass-by-copy implementation by programmer (the full Figure Ia] program; Appendix A.I. With shared heap. NIGHTIAR (Baseline) uses baff-be-reeerence to represent mutable data structures NIGHTIAR also uses pass- berfeeee,,,,,,,ce but also additionally enables the use of Python functions. We execute each program on generated graphs of exponentially increasing sizes 5 times each, using GPT--411-202504-111 on machine with an MI Max chip and 64GB memory. Due to cost constraints, we set a limit of 100 steps per query for the NIGHTIAR agent loop.
Results. Figure 10 shows the performance. The pass-by-cops implementation generally achieves lower accuracy than NIGUTIAR implementations. On small graphs with fewer than 100 nodes, the pass-by-copy implementation generally achieves lower execution time, lower token usage, and lower cost. Since the graph is encoded into JSON format and appended into the prompt, the number of input tokens increases linearly with the number of nodes. After 1000 nodes, the encoded graph exceeds the context window size, resulting in failure to answer any queries. The execution time, output token usage, and cost also increase as the graph size increases. In comparison, NIGHTJAR
36

---

Published as a conference paper at ICLR 2026
Average Pass Rate
Execution Time
Cost
1.0
Time (5) flog scale)
10a
10a
USD (log scale)
Rate
10
10
0.5
101
10
10-1
0.0
10-1
10
10a
10a
10a
10a
10a
10a
Input Tokens
Output Takens
Uncached Input Tokens
Tokens (log scale)
10a
Tokens (log scale)
10a
Tokens (log
20
10
20a
104
10*
10a
10a
10a
102
10a
10a
10a
Number of Nodes (log scale)
Number Nodes
Number of Nodes
(log scale)
(log scale)
Nightjar
Nightjar (Baseline)
Manual Impl (Pass-by-Copy)
Figure 10: Performance of the Figure programs.
The shaded area shows the range of values. The black dotted line marks when the encoded graph exceeds the context window size with pass- mass-by-copy.
and NIGHTIAR (Baseline), using pass-by-reference to represent mutable objects, query pieces of the graph as necessary. Thus, the token usage and execution time of NIGHTIAR and NIGHTJAR (Baseline) increase at a slower rate compared to the pass-by-copy implementation as the graph size increases. This enables NIGHTIAR (Baseline) to answer queries on large graphs when the pass-by-copy implementation cannot, resulting in the higher pass rates. The specialization to Python of NIGHTJAR enables the of Python functions, which improves further the pass rate the baseline implementation. While programmers can manually implement using pass- pass-by-referenn NIGHTIAR supports this data representation style of the box without additional configuration by the programmer.
E.6 NUMBER OF EFFECTS TO RUNTIME
There is a direct relation between program runtime executed using NIGHTIAR with the number of effects used, as each emitted effect corresponds to one inference pass through the base LLM. Figure I a scatterplot of the number of effects used and the runtime each program execution of SPSBench. It shows that the program runtime is proportional to the number of effects emitted during program execution, with different LLMs exhibiting different ratios of runtime to number of emitted effects.
37

---

Published as a conference paper at ICLR 2026
400
300
Runtime (s)
200
100
100
Number of Effects
Sonnet 4 Nightjar
GPT-OSS 20B Nightjar
GPT-4.1 Nightjar
Figure 11: Number of emitted effects in program execution to runtime.
E.7 ADDITIONAL RESULTS ON LLM MATH REASONING BENCHMARK
evaluate the performance of NIGITIAR on the full evaluation set of GMSSK (Cobbe et al. 2021) consisting of 1319 math word problems using G.T-4.1. GSM8K is benchmark for evaluating LLM math reasoning, and thus does not require complex data structures nor meaningfully use formal program state solve. In this sort of application, using NIGHTJAR is equivalent using LLM agent with code evaluation tools with isolated program states. We evaluate the performance of NIGHTJAR on GSM8K to empirically validate this and show that NIGHTJAR's performance is competitive.
We compare against the manual implementation of program, which queries the LLM for string expression that the program then invokes the Python eval function on. We also compare against version of manual implementation that employs tool with the OpenAl Code Interpreter (which evaluates code in an isolated container) and version that implements tool use with custom, user-defined code evaluation tool that maintains isolated program states.
Table ( shows the pass rate, the average runtime, and the average number of tool calls or effects over 3 runs. NIGHTIAR achieves higher or equal accuracy the manual implementations, demonstrating that the NIGHTIAR system and the shared program state setup do not detract from LLM capabilities on standard LLM tasks. NIGHTJAR incurs slightly higher runtime overhead than the basic manual implementation program, because NIGHTJAR requires the LLM to emit effects. However, this runtime overhead is similar to the overhead incurred by utilizing tool use with the OpenAl code interpreter tool. Additionally, NIGHTIAR's 's runtime overhead can be reduced with further system engineering as shown by the optimized version NIGHTIAR.
Table 8: GSM8K results using GPT-4.1.
Method
Pass Rate
Avg Time (s)
# Tool Calls or Effects
Manual Impl
0.84
1.4
(0.8-50.4)
Manual Impl w/ OpenAl Code Interpreter
0,92=10.01
5.1
0.9 (0-3)
(0.1-304.7)
Manual Impl w/ Custom Code Interpreter
0.70=00.0
2.6
1.6 (0-4)
(1.0:24.))
NIGHTIAR (Shared Python)
0.93++0.00
5,9
6.6
(2.3-58.)
NIGHTIAR
0.93
(1.7-36.3)
(0-16)
38

---

Published as a conference paper at ICLR 2026
F FAILURE ANALYSIS
We analyzed the 125 effect traces of NIGHTIAR on execution programs from SPSBench using GPT-4.1 and Sonnet 4. We labeled the reason the LLM failed get perfect score the program for each run of the program (25 program x 5 runs each = 125 runs in total). Table shows the summary. Most program runs pass all test cases (100% Pass). It is only in few instances that the LLM gives up on evaluating a prompt when it encounters an error (Give Up), hallucinates data (Hallucination)) or incorrectly performed the program state operation specified in the prompt (Incorrect State Op). Most of the time, the LLM fails test cases due to making incorrect reasoning conclusions or math computations (Reasoning). Decreasing failures due to incorrect reasoning requires more capable LLM models or additional prompt engineering on the program's 's prompts or the L.M system prompt.
Table 9: NIGHTJAR failure analysis summary.
Category
Sonnet
GPT-4.1
100% Pass
(65.6%)
(55.2%)
Reasoning
41 (32.8%)
47 (37.6%)
Give Up
0 (0%)
4 (3.2%)
Incorrect State Op
(0.8%)
(4.0%)
Hallucination
(0.8%)
0 (0%)
Total
125
125
We also further analyzed the types of error messages the LLM encounters when issuing effects and how it attempts to recover from these errors. Table T0 and Table [1 show the summaries. Sonnet 4 encountered very few errors, most of which are due to incorrect syntax or incorrect data type assumptions. GPT-4.1 encountered many more errors, most of them are due to confusing which effects use (State Op Error). For example, it often tried correcting its mistake to use EXEC successfully. Var) by confusing references for variable names or using as placeholder inappropriately. When encountering errors, the LLM generally corrects its mistake. GPT-4, sometimes ignores the error and performs different operation instead or finds a workaround to performing the same computation.
EVAL instead of EXEC, before also often hallucinates variables (Forbidden
Table 10: Types of errors encountered by emitted effects.
Error Category
Sonnet 4
GPT-4.1
State Op Error
0 (0%)
219 (67.8%)
Forbidden Var
0 (0.0%)
70 (21.7%)
Type Error
(36.4%)
15 (4.6%)
Syntax Error
(36.4%)
(4.0%)
Function Call Error
(18.2%)
2 (0.6%)
Index Error
(9.1%)
4 (1.2%)
Total
11
323
Table 11: Method of recovery by the LLM when encountering an error during effect handling.
Recovery Method
Sonnet 4
GPT-4.1
Conrection
(63.6%)
199 (61.6%)
Ignored Error
(9.1%)
59 (18.3%)
Workaround
(18.2%)
34 (10.5%)
Retry
(0%)
22 (6.8%)
Not Recovered
(9.1%)
9 (2.8%)
Total
11
323
We show below the effect trace of the example program (graph in SPSBench) that did not pass the test with the input query "update the graph paper cites paper 14." The execution did not
39

---

Published as a conference paper at ICLR 2026
correctly update the graph as it confused the direction of the graph edges, and instead updated the graph such that paper 14 cites paper 5:
Rffect: eval ovar (cyr((t(p(quuur)))
Rasult: *<class 'str'>"
Effect: (al(esr(tyyp((graab))
Result: "celase path> Graph'>"
Offect:
eval (gaaaattrr(gra,, '._doc__.)))
Result: True
Effect: eval (ggotattrggraph, '__doc...') '))
Result: directed graph. Nodes represented
nade values. Edges represented by dictionary of aource node value to a set of targat node values."
Effect: eval(str (dir(gggh))))
Result:
"'..snaotatione_." "__ccass__.' _..ootaine_.'' __delattr_..' _idct..." '__dir.." doc__
1--09-- firstlineno.. ''._orratt_'' ---50-- gettttrribbre... ...gettatt.... -t
haah -..int..'' init anttt_uuellsss_." -_.l__.. lan__', '._.t..' nodule.
... "__.ee__... reduce.. reduce_.x..." "e.repr.. aetattrt... sizeof
_elotnemes__.' _tstatic_striiuuee_.. '__str_.' "___ubclaesbook_." _..vekrrff_. 'edges'
"odea'""
Effect:
(valeeet(tppe ()))
Result: *<class 'str'>"
Bffect: eval (allarqu..stii (tt.s(((()))oo((()
Reault: "update the graph so paper cites paper 14."
Effect:
eval(graph.
Result:
Ref18
Effect: eval (gal(sssggraaaeedeee edges))
Result:
19, 21).
7,
15,
2. 3, 4, 5, 8, 7, 10, 15, 20,
21, 22, 23)
19: {0,
4,
10,
14,
15,
17,
20)
18:
<0,
3, 6,
17, 20, 21, 22, 233, 13, 4,
14, 20, 21,
243
(2,
3,
8,
13,
14, 21,
24), 10: [0,
8,
14, 21, 24)
15: 00,
2,
4,
13,
17,
23,
24
21: 42, 3,
7. 8, 11, 23, 24), 11: (1, 3, 4, 17, 23), 17: 12, 3,
22,
23)
1:
(3,
4,
20,
22). 14:
(2. 7, 13, 24), 22:
20, 13) 13: 124,
71, 2:
(3,
7,
20,
23,
24)
23:
(24,
0, 4),
(24, 8, 7), 24: (0, 8, 20),
8: (0, 8, 20), 8: (20))"
Bffect: eval (graph.gg.p...cco...
Result: "EO, 1, 2, 3, 4, 6, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24
Effect: @v4 in graph. nodea and graph.nodes)
Result: True
Offect: eval(5 in edges)
Result: True
Rffect: eval(14 in graph (5])
Result:
Effect: (sat heggsh.ddgssaai
Reault: (Af(111((bbj])
Offect: eval eval(sorted
(rreph.edges(S)))
Result: Ref (1152 (Obj])
Rffect: eval (sortad(11at (1rpheddiii((gphhhcceeee)))))))
Result: *[2, 3, 7, 8, 11, 13, 14, 21. 23, 20]
Rffect: evaa("break
query.strip() lover()
quary.strip() llovr O)
Reault: Falae
Effect:
(graph (ggee(grgrppeeeee((s))))) (5)))
Result: Red (1372
Bffect: eval (sortad(listt (graph (gre 14, 21, 23, 24]
Reault: [2,
Effect: eval (reeponee sorted (graph edges (d) (5])))
Result:
Effect: done()
G LLM USAGE DISCLOSURE
We used I.LMss to correct grammar, to find related works, and to brainstorm conceptual terminology.
40


Reviews:
[{"review_id": "Js4UXSvpg0", "reviewer": "Reviewer_8swZ", "summary": "This paper introduces the idea of shared program state, essentially an LLM black box that can read and manipulate the program's internal state which can be steered somewhat by specialised syntax. The paper walks this idea through with an example, defines the proposed construct in eBNF and further explains the functionality of each component. The paper concludes with a reference implementation, nightjar, of the proposed idea and a lightweight empirical study with 25 programming examples which suggests that programmers using nightjar are about as proficient as programmers without it, while sacrificing execution time for conciseness.", "questions": "No questions at this point.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "I think the general idea of natural language programming interfaces is intriguing and has a rich history that goes well beyond the current LLM hype - indeed beyond deep learning.", "weaknesses": "I am not convinced ICLR is the best avenue to publish this work. The empirical study is at best preliminary and I am not convinced at all that programmers would like to give up control and lose out on execution time for saving a few extra lines of code - with LLM-assisted code writing and analysis tools these come essentially for free.\n\nI suggest to greatly expand the scope of the empirical analysis, include evaluations on known public benchmarks as well as to incorporate qualitative feedback, analysis of failure modes, etc, to increase the scope of the contribution commesurate with ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t8d0gdDWxI", "reviewer": "Reviewer_M1qm", "summary": "This paper introduces shared programs states between formal and natural programming languages. This is formalized with the concept of Natural Function Interfaces (NFIs), through values, effects and handlers for sharing scopes, heaps, and control states. The proposed system, NIGHTJAR, implements this shared program and enables natural language code to directly read, write, and manipulate the program state of a host language such as Python. The work is supported by theoretical grounding and a new benchmark suite (SPSBench) evaluating program conciseness and performance.", "questions": "- **Execution times may be conservative**: Rather than relying only on LLM APIs, consider adding evaluations with smaller, locally hosted LLMs. \n- **Discuss failure modes**: Consider adding some analysis of where failures occur when they do.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Novel conceptual contribution**: The idea of exposing host-language state and control to an LLM in a principled and programmable way is novel. This moves beyond existing systems that treat LLMs as isolated components that make tool calls using programmer defined functions.\n- **Strong formal framework**: The authors present a formal framework for NFIs, including variable scopes, heap references, and control state.\n- **System implementation**: The NIGHTJAR system demonstrates that the abstraction is realizable and yields substantial code reduction  compared to manually writing interoperable code.", "weaknesses": "- **Limited empirical evaluation**: The evaluation on program pass rates and conciseness, while adequate as proof of concept, does not deeply explore scalability, robustness, or user experience. \n- **Benchmarks are synthetic**: SPSBench appears to consist mainly of small programs adapted from documentation examples. It would be great to have analyses on real-world user code or larger-scale applications.\n- **Safety and correctness**: Though acknowledged in discussion, the implications of allowing natural code direct memory access (even through abstraction) deserve stronger treatment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qq0v9VWuzx", "reviewer": "Reviewer_83Tw", "summary": "The authors propose to enable LLM \"execution\" agents to directly interact with a host language program state to facilitate easier interaction between \"natural code\" and formal language code (programming languages). They define a Natural Function Interface (NFI) along with its operational semantics, and show how NFIs generalise tool use/MCP to a richer interaction semantics (shared state, shared control flow).\nThe authors then curate a benchmark dataset with natural code to evaluate their approach in terms of lines of code (after normalisation), pass-rate, and execution time. They show a significant reduction of code size (by making much of the serialisation, reisation implicit to the paradigm) while matching the manually written implementations in a majority of cases.", "questions": "- Q1: The idea of shared state requires the model to be aware of how data can be manipulated (`graph.add_edge((14,5))`, etc.) and moving data via the handler requires serialisation and conversion back to the expected data-type (similar to FFI dtypes vs the host language dtypes). Who has the burden of defining these serialisation operations, Function symbol discovery, etc.?\n\n- Q2: While the implicit to the paradigm operations vs explicit boilerplate is explored in the paper, how does this compare to generating a single PL script that solves the task instead of switching PL/NL context?\n\n- Q3: How does NL execution handle stochasticity? Is the temperature set to 0?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Generalisation of tool use to shared program state (memory, control)\n- competitive results even with what seems like a naive implementation", "weaknesses": "- It is unclear how/where serialisation/data marshalling is to be implemented; it is also unclear if grammar-based sampling is compatible with the approach or if any such grammar file should be augmented to enable emission of effect tokens.\n- perhaps out of scope, but the handler loop seems unintuitive: if emitted effects are side-effect free wrt the program state that is observed by the NL code, why not stage an execution plan and delay the interrupts until the last moment possible to reduce overhead?\n- perhaps missing reference to literate programming (of ye olden days): Knuth, Donald Ervin. \"Literate programming.\" The computer journal 27.2 (1984): 97-111. Not directly the same concept, but it follows the spirit of literate programming\n- There is an unexplored trade-off frontier: in the paper, the comparison is the paradigm handling the boilerplate vs the user-written boilerplate. A fully FL/PL program vs a hybrid program would be another comparison of interest, but the cost normalisation is more difficult.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2L3wnLsdTJ", "reviewer": "Reviewer_KmWC", "summary": "The paper introduces **Shared Program State (SPS)**, a novel abstraction allowing natural-language code (prompts) and traditional programming languages (like Python) to **share variables, objects and control flow**. It defines a schema called the Natural Function Interface (NFI) that models this sharing via operations like `Lookup`, `Assign`, `Goto` etc. The authors implement this in a system called NIGHTJAR, which co-executes Python and natural code. Empirical evaluation (on a 25-program benchmark) shows comparable accuracy with much shorter code (23-82% fewer lines), albeit with higher runtime overhead.\n\n**Key Contributions**\n1. Definition of the Shared Program State abstraction and its semantics for mixing natural and formal code.\n2. Formalization of the Natural Function Interface (NFI) schema for the interaction.\n3. Implementation (NIGHTJAR) demonstrating how natural language code can manipulate program state in a host language.\n4. Empirical study (SPSBench) showing productivity gains (code conciseness) and feasibility of SPS, with noted runtime cost.", "questions": "* Is it possible to include a larger scale benchmark with more complex stateful programs (e.g., classes, inheritance, mutable collections, parallel threads) and evaluate how the shared state abstraction performs there (both correctness and latency)?\n* Is it possible to compare the performance (accuracy, conciseness, runtime) on tasks used in ANPL [A] or similar to increase external validity?\n* Is it possible to provide a deeper latency cost-benefit analysis (e.g., user-perceived delay, scalability with number of state operations, caching strategies for repeated NL manipulations)?\n* Is it possible to include a dedicated \"robustness\" evaluation: fault injection (NL commands that try invalid operations), tracking state invariants, measuring how many errors occur, how they are detected and recovered? Also are there guidelines or restrictions on NL code allowed (e.g., sandboxing, type checks, access control)?\n* Is it possible to add a discussion on how debugging mixed NL+code state will be supported (logging, visualization, stepping)?\n* Is it possible to deepen the literature comparison section: explicitly list prior abstractions (e.g., prompt-programming, tool-calling interfaces, NL-to-code) and highlight exactly how the shared -state model differs (e.g., direct variable reference, control flow manipulation)?\n* Is it possible to include a user study (even small-scale) with software engineers or prompt-programmers: measure how well they'd adopt the shared-state model, what errors are common, what cognitive load is involved?\n\n```\n[A] ANPL: Towards Natural Programming with Interactive Decomposition, NeurIPS 2023\n```", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper introduces a novel abstraction; the idea that prompts (natural-language code) and traditional programming languages can share program state; variables, heap objects, control flow. This moves beyond prior work that treated prompts as black-box tool calls or code generation endpoints. For example, [A] considers prompts as programs but does not formalize shared variable/state semantics between NL and formal code). [B] supports collaborative prompt engineering, but again focuses on prompt sharing/management rather than state-sharing semantics between NL and host code.\n* The combination of techniques is also creative: using an LLM as a \"natural interpreter\", defining an interface (Natural Function Interface) that models state-manipulation effects (assign, lookup, goto) and handlers in a host language. Prior works on NL to code generation (e.g., prompt-based learning, chain-of-thought) separate the code generation from program execution; here the \"code\" (NL) directly manipulates execution state. For instance, the \"Program of Thoughts\" [C] work uses LMs to generate a program which is then executed externally (i.e., separate stages). In contrast, this paper merges the NL/code boundary via shared state. That creative combination marks a strong originality.\n* The clarity of the interface model (Natural Function Interface), and the way the paper formalises values/effects/handlers, suggests technical rigor. This goes beyond more HCI-oriented prompt papers (e.g., CoPrompt [B]) which focus on usability rather than formal semantics.\n\n```\n[A] Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts, FSE 2025\n[B] CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming, CHI 2024\n[C] Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks, TMLR 2023\n```", "weaknesses": "* The empirical evaluation uses a 25-program suite (e.g., SPSBench) that demonstrates the abstraction in a limited number of tasks. It is unclear how well the approach scales to larger, more complex programs, especially ones with deeply nested state, concurrency, or rich object graphs.\n* The implementation incurs a substantial runtime penalty (11.7–15.3× slower) due to the involvement of an LLM interpreting natural code. While the paper acknowledges it, the evaluation does not fully explore how this overhead affects usability in real-world settings. Many practitioners may find this cost prohibitive.\n* Allowing natural-language code to manipulate host program state (variables, heap, control flow) introduces risks: unintended side-effects, state inconsistency, security holes, debugging difficulty. The paper acknowledges safety issues but does not deeply evaluate them (e.g., what happens if the NL code assigns to a variable incorrectly, or dereferences invalid references).\n* While the “shared program state” abstraction is compelling, some of its components (NL interfacing with program state, prompts-as-programs) overlap substantially with prior work. For instance, [A] already frames prompts as programs. The novelty claim would be stronger if the paper explicitly differentiates itself from these works and shows how state-sharing goes beyond \"prompts as programs\" and prior tool-calling frameworks.\n* Although the formalism is strong, there may be a steep learning curve for typical developers: understanding effects/handlers, bridging NL syntax with host language semantics, maintaining coherence in hybrid code. The paper could provide more empirical evidence on developer usability: how quickly developers adapt to writing natural code that manipulates host state, what errors or misunderstandings occur. In contrast, works like CoPrompt [B] focus on developer workflows and collaboration in prompt engineering and report usability metrics.\n```\n[A] Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts, FSE 2025\n[B] CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming, CHI 2024\n```", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)