L2MAC: Large Language Model Automatic Computer for Extensive Code Generation
Authors: Samuel Holt
Max Ruiz Luyten
Mihaela van der Schaar
Venue: ICLR 2024 poster (2024)
Keywords: Code Generation
Memory-augmented LLMs
Large Language Models (LLMs)
LLM coder agent
LLM Agent
Stored-program computer
von neumann architecture
Primary area: general machine learning (i.e., none of the above)
TL;DR: Introducing L2MAC, pioneering the first practical LLM-based stored-program automatic computer (von Neumann architecture) framework in an LLM-based multi-agent system, for solving complex tasks through generating extensive and consistent outputs.

Abstract: Transformer-based large language models (LLMs) are constrained by the fixed context window of the underlying transformer architecture, hindering their ability to produce long and coherent outputs. Memory-augmented LLMs are a promising solution, but current approaches cannot handle long output generation tasks since they (1) only focus on reading memory and reduce its evolution to the concatenation of new memories or (2) use very specialized memories that cannot adapt to other domains. This paper presents L2MAC, the first practical LLM-based general-purpose stored-program automatic computer (von Neumann architecture) framework, an LLM-based multi-agent system, for long and consistent output generation. Its memory has two components: the instruction registry, which is populated with a prompt program to solve the user-given task, and a file store, which will contain the final and intermediate outputs. Each instruction in turn is executed by a separate LLM agent, whose context is managed by a control unit capable of precise memory reading and writing to ensure effective interaction with the entire file store. These components enable L2MAC to generate extensive outputs, bypassing the constraints of the finite context window while producing outputs that fulfill a complex user-specified task. We empirically demonstrate that L2MAC achieves state-of-the-art performance in generating large codebases for system design tasks, significantly outperforming other coding methods in implementing the detailed user-specified task; we show that L2MAC works for general-purpose extensive text-based tasks, such as writing an entire book; and we provide valuable insights into L2MAC's performance improvement over existing methods.

Published as a conference paper at ICLR 2024
L2MAC: LARGE LANGUAGE MODEL AUTOMATIC COMPUTER FOR EXTENSIVE CODE GENERATION
Samuel Holt University of Cambridge s ackuk
Max Ruiz Luyten University of Cambridge 971 @cam. ac.
Mihaela van der Schaar University of Cambridge mv 4 72 cam .uk
ABSTRACT
Transformer- bassed large language models (LLMs) are constrained by the fixed context window of the underlying transformer architecture, hindering their ability produce long and coherent outputs. Memory-augmentedd LLMS are promising solution, but current approaches cannot handle long output generation tasks since they (1) only focus on reading memory and reduce its evolution to the concate- nation of new memories or (2) use very specialized memories that cannot adapt other domains. This paper presents L2MAC', the first practical LLM-based general-purpose stored-program automatic computer (von Neumann architecture) framework, an LLM-based multi-agent system, for long and consistent output generation. Its memory has two components: the instruction registry, which is pop- ulated with a prompt program to solve the user-given task, and file store, which will contain the final and intermediate outputs. Each instruction in turn is executed by a separate LLM agent, whose context is managed by a control unit capable of precise memory reading and writing ensure effective interaction with the entire file store. These components enable L2MAC to generate extensive outputs, bypassing the constraints of the finite context window while producing outputs that fulfill a complex user-specified task. We empirically demonstrate that L2MAC achieves state-of-the- thetth performance in generating large codebases for system design tasks, significantly outperforming other coding methods in implementing the detailed user-specified task; we show that 1.2MAC works for general-purpon extensive text-based tasks, such as writing an entire book; and we provide valuable insights into L2MAC's performance improvement over existing methods.
INTRODUCTION
Transformer-base based Large Language Models (LLMs), such as GPT-3 (Brown et al., 2020), Instruct GPT (Ouyang et al., 2022), and the most recent GPT-4 (OpenAI, 2023), have achieved unprecedented success in generating high-quality user-directed text. Despite their impressive capabilities, these models are inherently restricted by a fixed context window of size c, which limits the number of tokens and, consequently, the characters they can process. This limitation manifests itself critically in tasks that require the generation of extensive and logically coherent output structures, such as the generation of large codebases. Therefore, although existing LLMS excel in generating short isolated code snippets (Chen et al., 2021), they struggle to produce codebases that are both extensive and internally consistent due to the information loss and contextual truncation imposed by the finite context window. But practical code is rarely a short snippet, so this limitation undermines the applicability of LLMS solving real-world tasks that require the dynamic integration of multiple outputs and attendance to distant information.
natural solution is extend the LLM agent with an external memory store. However, existing methods primarily serve to extend the implicit knowledge of LLMs through an external corpus (Zhong et al., 2022), to facilitate long conversational or document summarization tasks (Liang et al., 2023), or to maintain precise values for variables (Hu et al., 2023). In the latter case, the approaches simply interface the LLM with a dictionary or database, which does not adjust to other tasks. In other settings, current approaches adopt simplistic memory management strategies that append new
Full code https: /gi gightub........ com/ com//ammhlll 1/1.2MCC.

---

Published as a conference paper at ICLR 2024
L2MAC: Large Language Model A Automatic Computer
Instruction I.
Instruction 2.
Instrction 21.
A User Task
Agent
Agent
Agent
Gode an seline
Context Window
Context Window
Context Window
beb vercion
@oContoo Unit
File Store
3 Control Unit
File Store
Co Unit
File Store
pettlegs
struccure
meglatrtion
Soade ratos
E LLM Resp.
Iqg
LLM Resp
[wite]] file: app-17
Read] File: app 77
LLM Resp
LLM Gen.
tie flam Lapart reass
dau avers Fink
Cooplate
Instructions
[erite] fi3e: spp-p9
tm File Store
D.
P. rreats rogieteation
Create profiles
Valld
teterisin
Instrertian corpieted!
tests
Sanearizo
[br1s0] #29.27
Core application
boe
ceog wite
Figure I: L2MAC Overview. Code-L2MAC is an instantiation of the LLM automatic computer (L2MAC) framework, an LLM-based multi-agent system, here for extensive code generation. First, it breaks down the user task into sequential instructions I. The Control Unit (CU) manages the LLM's context window for each instruction (agent) and interacts with the external memory file store through read, write, and evaluate tools. It identifies and reads relevant files from the memory to generate or update files per instruction (P2). This ensures proper conditioning of existing files without losing vital context (PI). Automatic checks evaluate the LLM's outputs for correctness and completion (P3), with iterative error corrections involving both code syntactical checks of the code and running self-generated unit tests to check desired functionality. Overall, this produces a complete large codebase that fulfills the detailed user task in the file store. See Figure 5 for an expanded overview.
information sequentially without any provision for in-place modification. This makes any error essentially irreversible, a critical limitation when generating codebases, and restricts the possibility of adapting previously generated code as the task progresses. Compounding these shortcomings, these methods do not include mechanisms for maintaining syntactic or semantic consistency within the memory store, vital requirement for the generation of coherent and interdependent code structures. tasks.
Thus, existing methods are ill-suited for extensive large
An effective method for extensive code generation requires the following three core properties: (P1) Task-Oriented Context Management: At each computational step, the context for the LLM agent should contain the information required to complete the current instruction. The context is dynamically managed to prevent exceeding the fixed context window size of tokens.
(P2) Precise Read/Write Tools for Entire Memory: The LLM agent should possess read/write operations that can interact precisely with the memory store to fetch and update relevant files.
(P3) Checking the Generated Output: The outputs of the L.MM agent are checked for both mistakes and when the current instruction has been completed. When mistakes are detected, such as syntactically invalid code, or failing self gelf-generated functional unit tests, they can attempt to be fixed by iterating the discovered errors with the LLM agent.
With these considerations, we introduce L2MAC, the first practical LLM-based general- stored- proogrampmaammmmmmm computer (von Neumann architecture) framework, an LLM-based multi-agent system, and instantiate it for long code generation tasks as Code-L2MAC. A Control Unit (CU) orchestrates the execution of the individual L.M agents and their interaction with the memory store, thus satisfying the three stipulated properties. As outlined in Figure 1, an LLM agent first generates a task-oriented instruction list I from detailed user-specified task. The CU tailors the LLM agent's context (PI), so that it always includes the next unresolved instruction in Z and information about the execution of past iterations (agents), and declutters the context when approaching its limit. It also endows the LLM agent with the ability to read and update any existing region of the memory store or extend it with new outputs (P2). Furthermore, the CU plays a crucial role in checking the generated output (P3). It feeds the LLM agent with syntactical checker errors and requests the LLM agent to generate checks alongside generating output, here unit tests when generating code, which are verified at each update of the memory file store to trigger corrective actions if needed, thereby ensuring that the extensive output in memory is both syntactically and functionally consistent.
Contributions: ! We introduce the L2MAC framework, the first practical LLM-based general- purpose stored-program automatic computer (von Neumann architecture) framework, an LLM--bsed multi-agent system, for long output generation tasks. ② We provide a practical implementation of this framework for code generation tasks called Rode-L2MACC This uses a Control Unit to control the input and output of the L.M and the use of entire memory file store read/write tools and highlights
2

---

Published as a conference paper at ICLR 2024
that code checks are key to correcting generated code that has syntactic and functional errors within whilst conditioning on and integrating with the existing codebase. 3 We empirically validate Code-L2MAC, demonstrating its oftte-of-the-art capabilities in generating large codebases to solve for the first time high-level system design tasks of creating entire applications and for generating code achieving 90.2% Pass@1 score the HumanEval benchmark (Chen et al., 2021). Also, we show L2MAC works for general purppsse extensive text-based tasks, such as writing an entire book. Additionally, we gain insight and understanding of how Code-L2MAC can leverage tools execute and verify code by self- sefff-eneraiing its own unit tests and use these to correct for generation errors whilst generating large interrelated code structures that fulfill detailed complex user-specified task feature requirements.
2 BACKGROUND
Large Language Models (LLMs). Essentially, an LLM is probabilistic function 1 E - P(E), ingesting a list of elements of the alphabet Σ of lengths e and outputting a distribution over the elements from which one is drawn (Vaswani et al., 2017). We refer to c the length of the context. For example, GPT-4 and GPT-3 have fixed context windows of 8, 192 and 4, 097, respectively.
Consequently, discussed in Schuurmans (2023), LLM inference time can be formalized as a finite-state machine (Dolfing & Hetherington, 2001). Given the well-known and exceptional extension from finite-state machines to Turing machines Turing et al. (1936) that leads to the practical framework of stored-program computers, it is natural to envision a parallel extension for LLMss
Stored-Program Computer (SPC). Originally termed the Neumann architecture (Von Neumann, 1945), is a combination of processor (arithmetic and logic unit), a Control Unit (CU), and memory capable of storing both instructions and data, SPCs set the foundation for modern computer design. Where the control unit manages the interaction between the processor and the data. A fundamental property of a general-purposs SPC is that it can be reprogrammed automatically execute a program (ideally to solve a specified task) without manual intervention Specifically, the CU extracts from memory, data and instructions and correspondingly sets the input and state of the processor (PI) and then overwrites a memory register with the output (A (Appendix A). Thus, the SPC extends the processor's 'ss arithmetic and logic abilities with the ability to manipulate memory (P2). An often overlooked but vital function of the CU is error-checking of the processor's output (P3). "Stochastic" effects of temperature, voltage change, or radiation can cause errors in a processor (Nicolaidis, 2010), including misspecified inputs leading to overflows. Therefore, CUs implement output checks such as parity bit checks and usually corresponding error-correcting mechanisms (Harris & Harris, 2010).
3 L2MAC FRAMEWORK
Now we outline the L2MAC framework for the first practical LLLM-based SPC, with an instantiation for coding illustrated in Figure 1. L2MAC consists of three main components: the LLM processor, the memory file store, and the Control Unit (CU) that controls the flow of the execution, thus endowing the L.M agent with read-and-write capabilities, among other capacities Some choices are deliberately left open to differentiate the key functionalities needed from how we tackle them in our implementation for code generation, which we detail in the next section.
this illustrated in Figure 2.
3.1 LLM-BASED PROCESSOR
An inherent element of L2MAC is the Large Language Model (LLM), which is responsible for the actual generation of the output for the task. An LLM can be visualized as a more complex atomic unit of computation f n + P(E), where, for instance, |Σ| = 50, 257 tokens (Radford al., 2019);; rather than being restricted to only deterministic arithmetic and logical operations on binary sequences, that is, f (0, 1 1 x f0, 1,61 > 1)44 64 e.g., for standard 64-bit processor (Hennessy & Patterson, 201 This allows for more flexible and powerful (yet more expensive) computation unit, which can be used to solve a different range of tasks.
*As a side note, the Church- Church-Turing Thesis posits that both Universal Turing Machines (UTMs) and SPCs are capable of simulating each other and, therefore, are models of the same class of computational processes Von Neumann (1945) (Appendix A. Recently. it has been shown that augmenting LLM with an external read-and-wrice memory allows it simulate UTM (Schuurmans, 2023). However, we do aim reproduce the capabilities of the current hardware SPC arithmetic and logic operations but rather explore the powerful abilitics of LLM-SPCs as automatic computers.
'The token vocabulary size implementation detail of the underlying LLM used.
3

---

Published as a conference paper at ICLR 2024
G
Control Unit: Control Flow Diagram
File Store
Evalume
Pass?
Yes
Summarize step output to Myx and elear Ct c- Load next 215-1) and start C+
(2
M..)
Rend/Wiiee Evallunte D
Yes
doun denus Myr
LLM Processor
M
Invoke Tool!
Yes
M4
Called Completed?
No.
Append eyele message M, devi Acrer @ M,-
Yes Would exceed context!
1o Aqus
(C))((C))
Aguu (M-]
Aguns
Aguis Mj
No
No
Context Window Ct : 111C SC C (I(), Mr, MJ...., M
messages till [C4] Summmrize progress My, and clear C* e- Restart same z!-) with C +- (2 .M.a)
Append messegos
C Agrss
Figure 2: Control Unit-Control flow diagram for dialog turn t. Here this executes one current instruction ( It starts by loading the first instruction into the context window C <- { and iterates it automatically until all instructions in I have been executed. First, Co is processed by the LLM Processor PLLM (C*) to output M, The CU stores this in a buffer Act+1 < (M, J. and checks if M, has called a tool, and if so, it executes the tool with the specified imput in My. which includes reading, writing and evaluating EDD the file store D D-oupputing M,, which is appended to the buffer Ac The CU performs additional control flow as detailed in Section 3.3, for checking if an instruction has been completed, continuing an instruction beyond the context window (PI), and continuing executing the current instruction.
Leveraging an LLM within the L2MAC framework offers distinct advantages to exploit and challenges to overcome, categorized to the properties of Pl, P2, and P3. Here, L2MAC benefits from the LLM's awareness of external tools with which it can interact with assisted by the Control Unit. This capability enables requests for memory reads/writes (P2) and additional output checks (P3).
In contrast, the use of LLM also imposes constraints that need to be addressed, such as the impediments of a limited context window to prevent incongruencies caused by the lack of attention distant information; that is. we have to handle context correctly (PI). Furthermore, the stochastic nature of LLM's output (P(E)) is a significant risk in situations where precision and correctness are key. Thus, crucial to effectively updating an interrelated memory the ability to enforce periodic checks on its output to ensure a level of correctness and consistency (P3) (Liventsev et al., 2023).
3.2 MEMORY
Following an SPC, we distinguish between two types of memory, that of the instruction registry I and the file store D. On the one hand, the instruction registry I contains the prompt program that will be used to determine the state of the processor. In L2MAC, given the LLM processor, this corresponds mainly to the strings that will be incorporated into the context of the LLM agent each execution. In a basic implementation, I lists the sequential steps needed to complete the task, either specified by the user automatically senerated-wher each step will be executed by separate LLM agent. On the other hand, the file store 'D stores the rest of the information relevant for the processor to read, write, and evaluate, with the final output ultimately stored in D.
3.3 CONTROL UNIT
The control unit (CU, ef. Figure 2) is responsible for managing the context window for the LLM, encompassing both its inputs and outputs, executing the LLM, checking its outputs for errors, and enabling it to call tools ffunctions--inclee including a block diagram figure and pseudocode for its operation in Appendix B. First, we start with how the CU interacts with the LLM.
reading and writing. We further detail the CU,
3.3.1 TASK-ORIENTED CONTEXT MANAGEMENT (P1)
Context formalism. The CU uses the LLM as multi-turn dialog system, filling its context window C' with a combination of messages m which can come from the user M,,, an LLM response M,, a function (tool) output M, or the CU M., so that E {M,, Me. My, M.). Consequently, at turn t then the context à Ct E List(M) is of the form C* = (m² _m² ..... .n?....m..)
To make L2MAC an automatic computer', the CU prompts the LLM to fill the initially empty instruction registry I with a list of instructions (I ,...,, [(K) (I()) where each will be executed in
'Given a set A, let List(A) = ((a (a.") E AVi E [n). n E N) denote the set of lists arbitrary length composed of elements in A.
We define what we mean by an 'automatic' computer in Appendix A.2.
4

---

Published as a conference paper at ICLR 2024
the LLM processor". L2MAC then loads an empty context window of an LLM agent with the first instruction CO <- {I(() } and iterates the CU control flow loop (Figure 2) until all instructions have been achieved. The LLM can signal when the current instruction I has been completed through calling a special function "step_come cosp__ctmetee at which point the CU evaluates the file store D using its evaluator module & (discussed in Section 3.3.3) to check for any introduced errors. If none are found, asks the LLM to summarize the generated output in the current context window C as a message Mrs and resets the context window as C < (I(++)), Mys).
Overcoming the fixed context window constraint. The input to the LLM cannot exceed the context window constraint c: the combined length of the initial context C and the additional messages buffer Ac = {m" ..... m²" must fit in the context window, that is?, |C @ Ac++1| S c. However, the length of Ac is not known priori, so the CU should have a way of handling the cases where Act+1 exceeds the context margin - |C*|. This can be achieved through combination of three different strategies:
(1) minimize the occurrence by promoting the task at each time step to be small enough and economizing the filling of the context C: and if the situation occurs, (2) store in the file store DD as much relevant output as possible from the current C and (3) update or include a new summary message with (k) as in-context tuning for the next iteration.
Regarding (I), through appropriate crafting C², the CU can prompt the LLM to plan sub-steps for the current instruction (most likely the original task prompt given by the user) and then target each sub-step in the following iterations. For illustration, in coding setting, (2) can be achieved by storing the generated code so far to avoid rewriting it in the next iteration, and (3) by initializing a new prompt with summary M,s of the current progress and helpful information to complete the current instruction, e.g., which files should be read or modified, or the current progress made fixing errors (3) errors-(3) is further detailed the bottom right of Figure 2.
3.3.2 PRECISE READ/WRITE TOOLS FOR ENTIRE MEMORY (P2)
The need for a reading mechanism that retrieves the relevant information at each iteration is evident and has been reasonably explored in previous literature. In contrast, previous work memory (as shown in Section 5) has paid little attention to the writing component, which gets mostly reduced to the appending of new prompts and LLM outputs (Liang et al., 2023; Zhong et al., 2022; Cheng et al., 2023; Wu et al., 2022) or updating the values of very structured and thus restrictive forms of memory (Modarressi et al., 2023), e.g., variables or tables (Hu et al., 2023).
These approaches make sense for summarization, dialogs, and database manipulation tasks but are not suitable for long interconnected output generation tasks, such as generating large codebases for system design tasks. Indeed, in such settings, the possibility of downstream subtasks I(J) demanding extensions of previous outputs (such as modules in a codebase) due to imperfect planning, plus the non-determinism and possible hallucination of LLMss, make it probable to require modifications of previously stored memories 'D to rectify these defects, as shown in Section 6.2.
In 1.2MAC it is thus key implement read/write interactions with any part of the memory. We want the agent to be able to scan on demand D, retrieve parts of the memory that it considers relevant, and potentially update them. In the next Section 4, we detail our implementation of an LLM with write component that allows it not only to add new information to D but also to delete and update any of its contents, an essential element that allows L2MAC to succeed in long output generation tasks.
3.3.3 CHECKING THE GENER ATED OUTPUT (P3)
As discussed in Section 3.1 and 3.3.2, the intrinsic stochasticity of LLMs and the well-known phenomenon of hallucination (OpenAI, 2023) make it likely that incoherent erroneous outputs occur during long interactions, which can be disastrous, for example, in coding. More profoundly, changes (c.g., to a function) to satisfy a given instruction IO) can hamper the solution to formerly completed instructions I(0, j. Therefore, is essential to incorporate two key checks, one to check the generated outputs for errors using given evaluator module E, and the other to check when the current instruction has been completed in the current context C' (c.f. top diamond in Figure 2).
'e consider the simplest case of sequential instructions of prompt program and discuss more complicated control flow paradigms in Appendix I.
We use @ List( A) x List(A) (A) + List( A) the concatenation of two lists on the set A. We abuse notation by considering any a a singleton {a).
5

---

Published as a conference paper at ICLR 2024
Error checking and error correction. Using given evaluator module &, which can process the existing file store D, i... (D), allows when run, errors to be detected and returned to the LLM as evaluator message M The evaluator is domain-speciffe can correspond to syntactical code checkers or self-generated unit tests that verify the correctness of the output for an instruction Z((), Crucially, these self-generated unit tests also help test the existing functionality of previously generated instructions I Naturally, evaluation checks should be enforced on the file store 'D after each writing operation to ensure that new additions are correct and consistent with previous files. These result in messages Mfe that are provided for in-context learning so that the LLM can correct the errors, Acu1 Ace1 @ Me and iterate by rewriting D until the evaluator checks pass, if any are present.
for example, in coding tasks, this
Checking for current instruction completion. To ensure continued execution in a multi-turn dialogue LLM system until completion, we request the LLM to decide on the next step to take, which can involve executing a tool (Wang et al., 2023a). This is achieved through cycle prompt message Moe that also asks the LLM if the instruction has been completed. Cycle prompting is necessary to account for different instructions requiring a variable number of turns complete, and to protect against hallucinations where the LLM agent only discusses the instruction I and does not generate a solution or store it in memory. Overall, this ensures that the LLM provides a solution for the given instruction within the current context (P3).
4 CODE-L2MAC
Now, we use our LLM automatic computer (L2MAC) framework and instantiate it to complete large codebase generation tasks, which we call Code-L2MAC. We distinguish the general-purposs task long-generation framework from the code instantiation to detach the core components from task domain decisions that can be appropriately adapted to other task domains. We provide the full details of the implementation in Appendix C; yet here we highlight some notable design decisions on the memory layout (in particular. D) and the read logic. There are different potentially valid alternatives for the read component in L2MAC (e.g.. Wu et al. (2022)). However, to promote transparent read component, we make the following choices.
We specify the memory file store D, be composed solely of files, each shorter than the residual margin of the context window after accounting for preliminary messages in any iteration and instruct the L.MM agent to assign each file semantically meaningful descriptive path. For example, a file name models s /userr py suggests that it contains the data model for the user class. This choice is not only valuable for human reader but also crucial for our Read-and-Writ the LLM to infer the content of existing files D and prioritize those that it might need to read to complete its current instruction. Given the list of file paths, the LLM can request that the contents of any be appended into its context C². Although the path-to-content link is not absolute, empirically, this read implementation can perform well (Section 6), when coupled with the context management logic of the CU. Specifically, if the L.LM reads the content of certain files and reaches the context window limit, it can include in its subsequent iteration summary indications of which files should be read and which should be excluded in order to generate the code necessary to complete the current instruction. This approach thereby enables the LLM to systematically scan all memory, with the scanning order guided by priorities previously established from the file path names.
implementation,
it allows
5 RELATED WORK
In the following, we review the existing approaches applicable to completing large-generation coding tasks-and summarize their main differences in Table 1. We provide an extended discussion of additional related works, including those of other single-turn code generation methods and augmented LLMs in Appendix D.
LLM-based approaches, focusing on
Single LLMs. These are limited to generating an output of the maximum size of the context window and form the most popular baseline widely used for coding. Examples include Codex (Chen et al., 2021), GPT4 (OpenAI, 2023), GPT-Engineer (Osika, 2023) and Code LLama (Rozière et al., 2023).
Reflecting LLMs. These enhance the performance of LLMs by having the LLM reflect retrospectively on the output of an evaluator and the actions taken (Shinn et al., 2023; Madaan et al., 2023). These verbal reflections are used as in-context learning (Dong et al., 2022) to improve performance when attempting the task again. By using a validator, such as self- unit tests, this simple idea has
"More precisely, no longer than half such margin to ensure the L.M rewrite any file within its context.

---

Published as a conference paper at ICLR 2024
Table I: Comparison of related works. Columns: Context Management (P1)-can -an it generate a larger output than e, and if so, can access the necessary files generate the next file correctly without knowledge of the file falling out of context? Precise Read/Write for Entire Memory (P2) it write to the full memory and update in any order? Checking the Generated Output (P3)?- can it check for mistakes in the output? Referencee:[II[iicaaarr I(Chen al., (s(Osia, ( al., [7 et al., 2023).
22(hrraaaaa & Watkins, 2
Approsch
Methods
Contest Management (Ply)
Procise Read/Write for Entire Memory (P2y?
Checking the Geneeatedd Ouiput (P3y? X No checks
Autrnonons Agent LLMss with memory
[1.2.3]
X forgets which files were generated
Single LLMs
[4.5]
X X-limited to
Reflecting LLMs
[6.7]
limited append-only read always memory
Code-12MMC
This work
provided state-of-the-ar small code snippet generation performance (Shinn et al., 2023). Still, the output of these approaches is also confined to the context window, whereas Code-L22MCC uses an evaluator not only for in-context learning but also for consistency of the codebase.
LLMs. Most work on this topic focuses on long conversational/text summa- rization tasks (Liang et al., 2023; Zhong et al., 2022; Cheng et al., 2023), or mathematical/ reasoning (Hu et al., 2023). Methods in the first group usually focus exclusively on the read com- ponent of memory. The same happens with (Wu ct al., 2022), which, through k-NN retrieval for next-token predictions, enhances perplexity for multiple tasks, including coding. However, read-only implementations are sensitive to the inconsistency and "stochasticity" of LLMs, which might break code, that the method cannot then fix. Modarressi et al. (2023) proposes updating memories, but they use tuples with two subjects and their relation, so they have no clear application to coding tasks. Similarly, the memory of mathematical/dataaass reasoning methods use a dictionary of values or a structured database, which again is not applicable to different settings (e.g., coding).
Memory-augmentt
Autonomous agent LLMs with memory. Richards (2023); Shrestha & Watkins (2023); Nakajima (2023) formulate fully autonomous LLM-based agent in a reinforcement learning environment to complete a given task (Wang et al., 2023a). They build and update their own plans from a few user-given high-level goals. When coding, these agents reprogram and reformulate their step plan at runtime without safeguarding the original intentions, resulting in frequent deviations from the task (Wang et al., 2023a; Sato al., 2023). In comparison, Code-L2MAC is designed to be automatic, i.e. it fulfills the detailed user-specified task without altering or forgetting parts. The only existing autonomous agent LLM with applicable memory capable of writing, reading, and executing code is AutoGPT (Richards, 2023). It tackles the context window constraint by iteratively summarizing previous actions completed at every k-tth turn into a rolling summary kept in the context. Like other agents, the first step of AutoGPT is to summarize the user-specified task into name, role, and five one-sentence goals for the agent to complete; this summary is kept in context throughout execution. AutoGPT and associated methods have two key limitations compared to Code-L2MAC (ef. Figure 3): (1) they compress the user-specified task into a mere six-sentence description, and (2) they compress the previous action history, thus losing crucial information. In code generation tasks, (2) indicates forgetting which files exist and their content, which is compounded by continual re-planning.
EXPERIMENTS AND EVALUATION
and verify that it significantly outperforms state-of-the-an methods for generating code and large codebases for system design tasks. Due to the absence of an existing benchmark for large codebase generation tasks, we introduce one to enable comparison with other methods. Furthermore, there are also no automated tools to validate the generation of codebases for system design tasks either, therefore we propose evaluation metrics to compare the methods.
In this section, we evaluate
Benchmark tasks. We evaluate against three standard system design codebase generation tasks whose prompt questions are taken from real-world system design interview questions (Xu & Lam, 2020; Martin, 2023), and the HumanEval benchmark (Chen et al., 2021); with all details in Appendix E.
Evaluation metrics. Large-scale codebase generation is unique in that the generated code can satisfy the high-level user-specified task feature requirements through various possible implementation approaches. To quantify the degree to which the user-specified features in the initial prompt are effectively implemented in the generated code, we introduce a performance metric named Features %. This metric numerically represents the proportion of input features that are fully and functionally implemented in the output codebase. The Features % is obtained by using a separate GPT-4 API call, which iteratively examines the entire generated code to verify the functional implementation of
7

---

Published as a conference paper at ICLR 2024
Table 2: Codebase generation system design task results showing the percentage of functional features specified that are fully implemented (Features %), ), the number of syntactical errors in the generated code (# Errors), the number of lines of code (LOC), and the number of passing tests (Tests Passed). Code-L2MAC fully implements the highest percentage of user-specified task feature requirements across all tasks by generating fully functional code that has minimal syntactical errors and a high number of passing self- gefereeee unit tests. The results are averaged over 10 random seeds.
URL Stottrer Agp Errors
Ocline Secial Modic App
Dvline Clat Apo Krroes
Mothol
Fentures 9
LOC
Teats Passed
Features
Krroos
LOC
Teats Prssed
Features
LOC
Tests Pasted
C2714
0.00
1992211
2.56.0.5
19.518.25 28
1161S15
0818.00.255
18::2.25
0:5.0034
127422.1
12:11
CrdeT
52998114
D05. Its
110111
36100313
19:1.. 19
0.4180017
106.1127
26.117
10S14.11
01.0
916:15.
1.221.57
Salf-ltaiize
45:918.5
005.0010 105
124:189
365-1:15
16412282
110:199
1.11:0038
18234.19 19
0.211100304
1111158
14210927
15.3
0.00
196:44.9
$.5:1.11
31.3818
0.610.99 369
1461555
31:298
13:1211
18812.47
220:88.8
30811334
91.6 K-12
530 47.6
14:6711
K2.4:14A
0:0
3951 42.9
183366 6.8
59.6-253
01.0
324-123
18819.11
all input features, counting the number of fully implemented features. We quote this as a percentage of the features implemented over the total features specified by the user. A detailed description and a motivating example of this implementation are provided in Appendix G. We also incorporate standard code generation evaluation metrics (Hasemer, 2018), such as the number of lines of code generated LOC and the number of errors # Errors in the codebase determined by code syntactical analyzer (Thénault, 2023). Furthermore, each method is instructed to generate unit tests to check that the generated code is valid; therefore, we quote how many of these self- ge-eeeeeertdd unit tests pass as Tests Passed. We give each metric with their 95% confidence intervals throughout. Moreover, we detail these metrics and the experimental setup in more detail in Appendix G.
Benchmark methods. To assess whether Code-L2MAC is state-of-the-art, we compare it with the most competitive and popular autonomous agent LLM with memory AutoGPT (Richards, 2023), based on GPT4 and capable of reading, writing, and persisting, using vector embedding memory to complete given tasks autonomously. We also compare with code reflecting LLM methods of CodeT (Chen et al., 2022), Self- Self-Refine (Madaan et al., 2023), Reflexion (Shinn et al., 2023) and a single GPT4 LLM (GPT4) (OpenAI, 2023)-we make all these competitive by providing them with the same tools that Code-L2MAC uses. We provide the method implementation details, hyperparameters, and experimental details in Appendix F.
6.1 MAIN RESULTS
We evaluated all the benchmark methods across all our system design tasks with results tabulated in Table 2. Code-L2MAC fully implements the highest percentage of user-specified task feature requirements across all tasks by generating fully functional code that has minimal syntactical errors and a high number of passing self-generated unit thers-e,,,,,foe,, Code-L2MMC is state-of-the- art for completing these system design large codebase generation benchmark tasks. We further evaluated Code-L2MAC on the standard HumanEval benchmark (Chen et al., 2021) and observe that it achieves a state-of- state-of-the-art score of 90.2% Pass@ I (Appendix H. Also, we show L2MAC works for
extensive text-based tasks, such as writing an entire book (Appendix H.2).
6.2 INSIGHT EXPERIMENTS
This section provides in-depth empirical analysis of the efficacy of Code-L2MAC compared to its benchmark counterparts, AutoGPT and GPT-4. Specifically, we examine the core properties we suggested an effective extensive code generation framework should possess: task-oriented context management (PI), precise read/write tools (P2), and output error checking and correcting (P3).
Can Code-L2MAC correctly perform lask-oriented context management? (P1). To explore if the benchmarked methods during operation contain the information within their context to complete the task directly, we adapted our Features % metric to count the number of user-specified task feature requirements that are retained within the methods task instructions instead, i.e., those instructions that are eventually fed into its context window during its operation, as shown in Figure (a). Empirically, we observe that Code-1.2MAC is able to retain a high number of user-specified task feature requirements within its instructions I and perform instruction-orieeneed long-running tasks. We note that AutoGPT also initially translates the user-specified task feature requirements into task instructions; however, does so with higher compression- -onpressiooncnndnnninnnnn the information into a mere six-sentence description. This process results in the loss of crucial task information necessary for completing the overall task correctly, such that it aligns with the detailed user-specified task.
Can Code-L2MAC perform precise read/write operations for the entire file store (P2)? We wish to understand, during the operation of executing a task instruction, if Rode-L2MAC can understand the existing generated code files within the codehase-whin could have been created many instructions
8

---

Published as a conference paper at ICLR 2024
100
50
Rearre Retamee
80
Aeevdrry CPPS
Leruss
Amt:PPT
40
GPT4
80
20 40
60
80
100
8.0
0.2
0.4
0.8
0.8
1.0
Dptoode Compottion (Nocmolltod. Stope)
38 Eprsodo Completon (Nommllied Steps)
96 Epiiode Completion (Normalieed Strps)
(a)
(b)
(c)
Figure 3: Experimental Insight during operation of generating a codebase for an Online Chat
App task. (a) Percentage of user-specified feature requirements that are retained within the methods task instructions and used in context. (b) Number of syntactical errors within the codebase. (c)
Stacked histograms of passing and failing self-generated unit tests.
ago, and through its understanding, create new files that interrelate with the existing files, and most importantly update existing code files as new features are implemented. To derive insight, we plot heatmap of the reading, writing, and when files are created at each write operation step during one episode in Figure 4. We observe that Code-L2MAC has an understanding of the existing generated code that allows it to update existing code files, even those originally created many instruction steps ago, and can view the files when it is not certain and update the files through writing the files. In contrast, AutoGPT often only writes to files once, when initially creating them, and can only update files that it knows about that are retained within its current context window. Although it also has a read file tool, it often forgets about the files that it created many iterations ago due to its context window handling approach of summarizing the oldest dialog messages in its context window, i.e., a continual lossy compression of the previous progress made during operation of completing the task.
Can Code-L.2MAC check the generated output and error correct (P3)? When using probabilistic model (LLM) generator to output code, errors can naturally occur in its out- puts. Therefore, wish to verify if, when errors do appear, the respective benchmark methods can error-correct the codebase. We plot the number of syntactical errors in the codebase during a run where errors are made in Figure 3 (b). We observe that Code-L2MAC can correctly error correct the previously gen- erated codebase that has errors contained within, which could arise from syntactical errors from the last file written or other files that depend on the most recent file written, which now contain errors. It does this by being presented with the error output when it does arise and modifying the codebase to re- solve the error whilst still completing the current instruction. In contrast, AutoGPT cannot detect when error in the codebase has been made and continues operating, which can compound the number of errors forming within the codebase.
Code L2MAC
0
Epieode Steps
AmtoGPT
No 6
10
20 Epieode Stepe
30
Figure 4: Heatmap of file access. Indicating reading, writing, and when files are created at each write operation step during one episode for the Online Chat App task.
Moreover, L22C- generates unit tests alongside the functional code and uses these as an error checker inspect the functionalities of the codebase as it is generated and can use these errors to fix the codebase to pass unit tests that now fail after updating part of an existing file. We show this in Figure 3 (c) and observe that AutoGPT, whilst prompted also write unit tests for all code generated, is unable to use these tests as an integrity error check, which could be compounded by the observation that AutoGPT forgets which files it has previously created and hence unable to modify the existing forgotten code files as new modifications are made, leading to incompatible code files.
7 CONCLUSION AND FUTURE WORK
In this paper, we present L2MAC, the first LLM-based ceneral-purposs framework that effectively and scalably augments LLMs with a memory store for long output generation tasks where this was not previously successfully achieved. Specifically. CMMA- an application for long code generation tasks, surpasses existing solutions-an is an immensely useful tool for rapid development. This work is not without limitations. L2MAC's performance is inherently bounded by its underlying LLM across various tasks, from planning to tool use. Further, the current design imposes an implicit constraint on the scale of the codebase by listing all file names within the context window, which could readily be improved. These, and among many other aspects such as complex instruction flows, re- programming the instructions, and interpretable prompt programs pose exciting open directions for future work to build upon as detailed in Appendix I.
computer

---

Published as a conference paper at ICLR 2024
Ethics statement. This paper proposes the first practical framework for LLM-based stored- program computer and instantiates a version to generate large codebases, named Code-L2MAC. However, misuse or of such tool with incorrectly specified task could lead to undesirable outputs. Furthermore, due to the use of an LLM,, such tool is prone to hallucinations, and hence, such results should always have content filter on the final output.
Reproducibility statement. All code is available at https: / //g/tgttthbbb com samholt/LMM AC. Seeking full reproducibility, we include an extensive appendix with all implementation and experimental details to re-create the benchmark tasks and baseline methods. These are detailed in the following: for benchmark task environment details, see Appendix E; for benchmark method implementation details, see Appendix F; regarding Code-L2MAC implementation details, for high- level details, see Appendix C, and for low-level details, see Appendix F.I; and for evaluation metric details, see Appendix G.
eeknowledgements. The authors would like to acknowledge and thank their corresponding funders, where SH and ML are funded by AstraZeneca. Moreover, we would like to warmly thank all the anonymous reviewers, alongside research group members of the van der Schaar lab (www , vander schaar-late lab. com) and Andrew Rashbass, for their valuable input, comments, and suggestions as the paper was developed.
REFERENCES
Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. arXiv preprint arXhv:2305.16867 2023.
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https:/ http::://wwwwt/// anthr pic. com/ news/ clwcde-aaeee// 3- f-mamll...
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint arXiv:2 007332 2021.
Ned Batchelder and Contributors to Coverage.py. Coverage.py The code coverage tool for Python. https:: httpgttthhbbuee//gg hwem///dbbattt. ctoptttttooeeeee....mmmo 2023.
Jonas Bayer, Christoph Benzmüller, Kevin Buzzard, Marco David, Leslie Lamport, Yuri Matiyasevich, Lawrence Paulson, Dierk Schleicher, Benedikt Stock, and Efim Zelmanov. Mathematical proof between generations, 2022.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer, 2020.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmmann, Trevor Cai, Eliza Rutherford, Katie Mil- lican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens, 2022.
Maarten Breddels. Solara. https: : / g htt.ub..cmwwwwwegettettee
2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 11079-11091. Curran Associates, Inc., 2022. URL https: //pproceddings igg neur pss cc wwtppppppppppppptpppccppteeee..e fi fiffelessl.. pttps//pppppe///22 filee/iee////// 878228629288 pdf.
99 996a1 70e1777550bbbbooc 660e
Tianle Cai, Xuezhi Wang, Tengyu Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint ww.Xiv:235511/222 2023.
10

---

Published as a conference paper at ICLR 2024
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guam Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397, 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained code. arXiv preprint arXiv:2107.08: 033744 2021.
Xin Cheng, Yankai Lin, Xiuying Chen, Dongyan Zhao, and Rui Yan. Decouple knowledge from paramters for plug-and-play language modeling. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 14288 14288-14308, Toronto, Canada, July 2023. As- http:////22.....ngsssssaae findings- acl.901. f23. URL
sociation for Computational Linguistics. doi: 10. 10.18633/v1/0022 .001.
https:: happ:://aclatthte wow.hooooo.o...o.o.o..oooo.oo... wwt.//23.........o
Cheng-Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? arXiv preprini arX1v:2305.00137, 2023.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, 2019.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers, 2022.
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif Saurous, Jascha Sohl-Dickstein, al. Language model cascades. arXiv preprint arXiv:2207.10342, 2022.
Hans JGA Dolfing and Lee Hetherington. Incremental language models for speech recognition using finite-state transducers. In IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. ASRU'OI., pp. 194-197. IEEE, 2001.
Qingxiu Dong, Lei Li, Damai Dai, Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factual- ity and reasoning in language models through multiagent debate. arXiv preprint trXiv:2305..1335. 2023.
Angeliki Giannou, Shashank Rajput, Jy yong Sohn, Kangwook Lee, Jason D. Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers, 2023.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines, 2014.
Mandy Guo, Joshua Ainslic, David Uthus, Santiago Ontanon, Jianmo Ni. Yun-Hsuan Sung, and Yinfei Yang. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 724-736, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 000/86653/11/222/ 2tt.://w///2....nnnnnnnnaae findingg URL https: http///caaaaathlooooooco org/ / 2022. findings naacl. .5....555... 55.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.cm 6800 2024.
Rui Hao, Linmei Hu, Weijian Qi, Qingliu Wu, Yirui Zhang, and Liqiang Nie. Chatllm network: More brains, more intelligence. arXiv preprint arXiv:2304.1 12998. 2023.
David Harris and Sarah Harris. Digital design and computer architecture. Morgan Kaufmann, 2010.
Tony Hasemer. Syntactic debugging of procedural programs. In Novice Programming Environments, pp. 227-241. Routledge, 2018
John L Hennessy and David Patterson. Computer architecture: a quantitative approach. Elsevier, 2011.
11

---

Published as a conference paper at ICLR 2024
Samuel Holt, Alihan Hüyük, Zhaozhi Qian, Hao Sun, and Mihaela der Schaar. Neural laplace control for continuous-time time delayed systems. In International Conference on Artificial Intelligence and Statistics, pp. 1747-1778. PMLR. 2023.
Samuel Holt, Alihan Hüyük, and Mihaela van der Schaar. Active observing in continuous time control. Advances in Neural Information Processing Systems, 36, 2024.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang. Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations. 2024. URL https: http://pppreeeeeeeeeeee hwtppooewwwweeeeeeeeeeeeeeeeeeee nww. wtwpr/wwwwwt/toumm/eeeeeea.aaaee wwweeeeoornidweeeeeeeeee -d
Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. Chatdb: Augmenting Ilms with databases as their symbolic memory. arXiv preprint arXiiv:2306003991, 2023.
Inflection AL Inflection-2, 2023. URL Utt.htppee///hleecioooaaaaaaeeee
Peter Justin. Flaskbb. https://githiie http://////ttttbb..oo//aaaaabbbt
2023.
Kwanwoo Lee, Kyo Kang, and Jaejoon Lee. Concepts and guidelines of feature modeling for product line software engineering. In International Conference on Software Reuse, pp. 62-77. Springer, 2002.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36, 2024.
Wenda Li, Lei Yu, Yuhuai Wu, and Lawrence C. Paulson. Isarstep: a benchmark for high-level mathematical reasoning, 2021.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624);10992-1999 2022.
Xinnian Liang, Bing Wang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. Unleashing infinite-length input capacity for large-scale language models with self-controlled memory system. arXiv preprint Stti::230413344, 2023.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models long contexts. arXiv preprint arXiv:2307.03112. 2023a.
Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and Soroush Vosoughi. Training socially aligned language models in simulated human society. arXiv preprint arXiv:2305. 16960, 2023b.
Vadim Liventsev, Anastasiaa Grishina, Aki Härmä, and Leon Moonen. Fully autonomous program- ming with large language models. arXiv preprint arXiv:2304. 10423, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXh:23303.1651,
Donne Martin. System design primer. https: https://github. http://///.ooomeetann////seeeeee beign-priime 2023.
Joan Miller and Clifford J Maloney. Systematic mistake analysis of digital computer programs. Communications the ACM, 6(2):58-6) -3,, 1963.
Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schütze. Ret-Ilm: Towards a general read-write memory for large language models, 2023.
12

---

Published as a conference paper at ICLR 2024
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint arXtn:2308..77124 2023.
Yohei Nakajima. Babyagi. https://github.... http::///stthbb..ccoyohheaaaalll imm
babyagi, httaaaaaammaaaiaiii...a 2023.
Michael Nicolaidis. Soft errors in modern electronic systems, volume 41. Springer Science & Business Media, 2010.
Brian Okken. Python Testing with pytest. Pragmatic Bookshelf, 2022.
OpenAL. Gpt-4 technical report, 2023.
Anton Osika. Gpt-engineer. https:/ / thub. Ccoom/Anttnnsstt.cc..c.o /atta//nnttooosk//tttteegngg neer, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez. Memgpt: Towards iims as operating systems. arXiv preprint arXiv:2310.08560, 2023.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-22, 2023.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. Random feature attention, 2021.
Jason Phang, Yao Zhao, and Peter Liu. Investigating efficiently extending transformers for long input summarization, 2022.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL
https: //ttervv//peeeevieew net/ wtwp:r/wwwww..eeeeottuiiieeee..e
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Toran Bruce Richards. Autogpt. https://ggt httbb/ghhuuu.c.o.om.. ingii f LaaattGrrviitts/AAe -GPT, 2023.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Megan Kinniment Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin, Lawrence Chan, Luke Harold Miles, Tao R Lin, Hjalmar Wijk, Joel Burget, Aaron Ho, et al. Evaluating language- model agents on realistic autonomous tasks. https: pettps://evals all iigigneent org/ Evalua Siing_LMAs_Red 1 sti c_Tasks ff 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023.
Dale Schuurmans. Memory augmented large language models are computationally universal. arXiv preprint arXiv:2301 .0455,900.0058,, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforeement learning. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Asim Shrestha and Adam Watkins. http:///////ttaaa..////////////e 2023.
13

---

Published as a conference paper at ICLR 2024
Chris Tabor. Flask http://////ttttoo/ooo/oo//aaa/aa 2023.
Leandros Tassiulas and Anthony Ephremides. Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks. In 29th IEEE Conference on Decision and Control, pp. 2130-213222 IEEE, 1990.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, 2022.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint 1xxi::23/2./1805. 2023.
Sylvain Thénault. Pylint, 2023. URL https::/pyyliiiie readthedocs Available at: http: http://yyyyinn......oo.ogo.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288 2023.
Alan Mathison Turing et al. On computable numbers, with application the entscheiduuggproolem.. J. of Math, 58(345-363):,, 1936.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
John Von Neumann. First draft of a report the edvac, 30 june 1945. Moore School of Electrical Engineering, University of Pennsylvania, Philadelphia, PA, USA. Available online: https://library. si. bluddiid---bbryyyb/o///riirdifdd
(accessed on October 2022), 1945.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang. Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308. 1432, 2023a.
Lei Wang, Chen Ma, Xucyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):1-26 2024.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self- collaboration. arXiv preprint arXiv:2307.05300, 1(2):3, 2023b.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback, 2021.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint 2xXi:23308008/55 2023.
Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers, 2022.
xai-org. Open release of grok-1, 2024. 2024. URL https:/// httttps://x.aa//blogggooo
Alex Xu and Sahn Lam. System Design Interview: An Insider's Guide, volume 1. Byte Code LLC,, 2020.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
14

---

Published as a conference paper at ICLR 2024
Dani Yogatama, Cyprien de Masson d' A'Atumee and Lingpeng Kong. Adaptive semiparametric language models, 2021.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences, 2021.
Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, and Rui Zhang. Summ": A multi-stage summarization framework for long input dialogues and documents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 1592-1604, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v h/2/2//2c22c2llloo 112. URL https: acl http///aaaantoooogoooooo org/2022 acl-long 112.
Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 5657 5657-5673, 2022.
Mingehen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, Róbert Csordás, Anand Gopalakr- ishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Hermaann Kazuki Irie, et al. Mindstorms in natural language-based societies of mind. arXiv preprint arXiv:2305.17006 2023.
15

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Stored-Progrann Computers (SPC)
17
A.1 SPC is Universal Turing Machine
17
A.2 What is an Automatic Computer?
17
B Control Unit Operation
18
C Code-L2MAC High Level Implementation Details
19
C.I Read / Write Implementation Details
21
D Extended Related Work
22
D.1 Vanilla LLMs
22
D.2 Transformers for Long Inputs
................................
................................
22
D.3 Memory Augmented LLMs
23
D.4 Tool Augmented LLMs
24
D.5 Multi-step Reasoning & Reflecting LLMs
24
D.6 Turing Machines, Stored Program Computers and Machine Learning
25
E
Benchmark Task Environment Details
25
E.1 Codehase Generation System Design Task
25
E.2
HumanEval Benchmark
29
F Benchmark Method Implementation Details
30
F.I
Code-L2MAC Low Level Implementation Details
31
G Evaluation Metrics
34
G.1 Motivation for Feature %
40
H Additional Results
40
HumanEval Benchmark Results
................................
40
H.2 L2MAC for Writing Entire Book
................................
41
H.3 Additional Main Results
47
H.4 Code L2MAC in Action.
52
I1.5
Additional Diverse Programming Code Generation Tasks
55
H.6 Human Expert Validation of Features Implemented Percentage Metric
56
H.7 Challenges and the Evaluation Huttan-wittenn Test cases
56
Generating 1,000+ Lines of Code with Code-L2MAC
Code-L2MAC Ablation with No Instruction Summarization Message
H.10 Additional Tasks Implementing New Feature in Existing Large Code Base Code-L2MAC of up 165,000 LOCs
58
I Future work
59
1.1
L2MAC in Non-coding Domains
61
Code. All code is available at https: broader research group codebase at https: h/tptttttbb////g/ com/vanderscchaarllaa L2M
We have
AC.
16

---

Published as a conference paper at ICLR 2024
A STORED-PROGRAM COMPUTERS (SPC)
Stored-Progrant Computer (SPC). Originally termed the von Neumann architecture, was first proposed by Von Neumann (1945) a combination of a processor (arithmetic and logic unit), a Control Unit (CU), and memory capable of storing both instructions and data, SPCs set the foundation for modern computer design. With a control unit (CU) managing the interaction between both components, this design is capable of being reprogrammed to solve various tasks without manual intervention '. In more detail, the CU uses instructions from the memory to set the processor to specific state (e.g., load two numbers into the processor's (arithmetic logic unit) input registers and then execute addition and store the result in memory. The computer is not only able to perform arithmetic and logic operations, but it can also manipulate the memory. One unique aspect often overlooked in this paradigm that crucial to the reliable operation of the processor is error-checking of the processor's 's outputs. Errors can arise in typical processor from hardware influences such as temperature, voltage variations, or radiation (e.g.. cosmic rays) (Nicolaidis, 2010), or from miss- specified inputs leading to overflows (e.g., divide by zero errors). Therefore, typical processors involve output checks, the simplest being parity bit checks, and can combine these with error-correcting mechanisms (Harris & Harris, 2010).
A.1 SPC IS A ALIVERSAL TURING MACHINE
Stored-progran computer (SPC) and Automata Theory. We can consider an SPC as specific type of Turing machine T. Formally, a Turing machine is defined by the 7-tuple T = (Q, Σ, r, go. b, F), where:
Q a finite set of states.
Σ is the finite input alphabet not containing the special blank symbol b.
I' is the finite tape alphabet, where b e I and Σ C l.
8 : Q X + Q x I' x {L, R} is the transition function. Given a state and tape symbol, it specifies the next state, the symbol to write on the tape, and the direction in which the machine should move its head (Left or Right).
E Q is the start state.
C Q is the set of accepting states.
In the context of a stored-program computer, the memory holds both the instructions (akin to the transition function 5) and the data (akin to the input sequence from Σ). The program counter, which determines the next instruction to execute, can be represented by the machine's current state in Q. The execution of program on an SPC is then analogous the computation performed by a Turing machine when transitioning from one state to another based on the input read and the defined transition function 8.
The universality of an SPC, similar to universal Turing machine, means that given the appropriate transition function (or set of program instructions) in its memory, it can simulate any other Turing machine and perform any computable task (Turing et al., 1936).
A.2 WHAT IS AN AUTOMATIC COMPUTER?
The term 'automatic computer,' as used in this paper, comes from the original conception presented by Turing et al. (1936), where Turing introduced the concept of an automatic-machinn commonly known today as Turing machine. In this context, 'automatic' refers to system capable of executing a pre-defined set of instructions without human intervention (Turing et al., 1936; Von Neumann, 1945).
"As side note, the Church-Turing Thesis posits that both Universal Turing Machines (UTMs) and SPCs are capable of simulating each other and, therefore, are models of the same class computational processes Von Neumann (1945) (Appendix A) Recently, it has been shown that augmenting an LLM with external read-and- -nd--ii- memory allows it to simulate UTM (Schuurmans, 2023). However, we do not aim to reproduce the capabilities of the current CPU-SPC in arithmetic and logic operations but instead explore the powerful abilities of L.M-SPCC as automatic computers.
17

---

Published as a conference paper at ICLR 2024
In the context of LLMs, there are methods whereby LLMs can autonomously execute tasks, as reviewed by Wang et al. (2023a). However, it is crucial to differentiate between the concepts of "autonomous' and 'automatic'. Autonomous agents characterized by their ability to independently devise and adapt plans, implying a level of self-determinationn and initiative in executing instructions. In contrast, our focus with 1.2MAC is on developing system that meticulously and automatically executes pre-defined instructions contained in the prompt program I. This execution requires no human intervention during the process until all instructions have been successfully executed. In this regard, L2MAC aligns more with the traditional concept of an automatic computer. It functions as general-purpose, Instruction-followe to efficiently executing the provided instructions, thereby ensuring adherence to the completion of the originally specified prompt program. The term "general-purpose here implies that 1,2MAC can accommodate any prompt program as input and execute it without necessitating system redesign for each new program introduced.
LLM-based computer (Von Neumann, 1945) dedicated
B CONTROL UNIT OPERATION
The Control Unit, manages the entire context of the LLM and is itself a finite state machine. Here we expand on Section 3.3, provide pscudocode in Algorithm 1, provide an extend block-diagram figure of L2MAC in Figure 5, and a data flow diagram in Figure 6.
User
Requirements
Instruction
1.
Instruction 2.
Instruction 9.
LLM
LLM
Wrip
LLM
Read
LLM
Context Window C³
Figure 5: Code-L2MAC Extended Overview. Code-L2MAC is an instantiation of the LLM automatic computer (L2MAC) framework, here for extensive code generation. First, breaks down the user task into sequential instructions I. The Control Unit (CU) manages the LLM's context window C for each instruction and interacts with the external memory file store D through read, write, and evaluate tools. It identifies and reads relevant files from the memory to generate or update files per instruction (P2). This ensures proper conditioning of existing files without losing vital context (P1). Automatic checks evaluate the L.M'' 's outputs for correctness and completion (P3), with iterative error corrections involving both code syntactical checks of the code and running self- generated unit tests to check desired functionality. Overall, this produces complete large codebase that fulfills the detailed user task in the file store D.
18

---

Published as a conference paper at ICLR 2024
File Store D
Control Unit
Fetch instruction I((k
CU
Instructions I
Function: Write!
Files
CU
LLM
Evaluate
LLM
CU
LLM
M
M
Mr
M
Mr
M
Mr
Context Window
CtC
Figure 6: Data flow of L2MAC executing prompt instruction Z(k) from self-programmed prompt-program Z. The first practical LLLM-bsed wored-program automatic computer (L2MAC) framework. The L2MAC consists of file store D, entire memory write component W, entire memory read component R. and a program control flow mechanism that of a Control Unit CU (c.f., Figure 2) manage the LLM's context window-encompasee encompassing both its inputs and outputs. Here, the context window C* (dashed gray box) consists of sequence of messages, where each message has one of the following message types: control Mc. user Ma, LLM response M, and function response Mf.
C CODE-L2MAC HIGH LEVEL IMPLEMENTATION DETAILS
In the following we describe the high-level implementation details of Code-L2MAC, and we provide the verbose low level implementation details, such as the function JSON definitions in Appendix F.I. We follow the same structure as the L2MAC outline and focus on the specific design choices we made for each item. For the LLM Processor, we use GPT-4-0613 OpenAI (2023), which trains GPT-4 to receive in-context descriptions of functions (tools) and call them as needed, where the descriptions follow JSON format (OpenAI, 2023).
The functions we provide are the following:
"provide_detailed sub_task_ tast_stees_________ _ttesssorr____________ss_sp_ees_ for_sub, sa___uubaaeeee maa_gettts' For producing a step-by-step plan, where each step paragraph is a detailed sub-task step for separate sub-agent LLM to complete.
2. 'sub_tan suu__ask__stt book_step_oome compleee.. For the LLM to call when the user specified sub task step has been completed.
3. 'view_file'. Prints out all the file contents into the response to view.
4. 'run_python_ file'.'' Runs python file and returns the output to the response to view.
5. 'write_files" Writes out multiple files that will be combined into the existing codebase.
6. 'delete_fike files'. Deletes files, where their names are specified.
Next, we discuss specific choices to develop a method based on our LLM Automatic Computer (L2MAC) framework specifically designed for large code generation tasks, which we call Code- L2MAC.
For the LLM Processor, context descriptions of functions and call them as needed following a JSON format (OpenAI, 2023). See Appendix F.1 for low-level JSON definition of the functions that provide to the LLM,, most of which we also mention below.
GPT-4-0613 OpenAI (2023), which trains GPT-4 to receive in-
In terms of the instructions, initialize ! with To 6, and the CU queries the LLM only in the first iteration with the user-specified feature requirements (task description) and requests the LLM to define a sequential prompt program I =
Regarding the data D, determine that it encompasses only the code generated far so that upon termination Code-L2MAC, 'D corresponds to its output. However, we impose the codebase D to
Rigorous,,, I is initialized with set of pre-defined messages for the I.I.M to be effective as part of Code-LLLM (including the functions the CU provides the L.MM).
19

---

Published as a conference paper at ICLR 2024
Algorithm 1 Control Unit Pseudocode for L2MAC
1: Input: Initial user input prompt Pmi file store D: Domain Evaluator unit &; Domain system message Ma; Domain bootstrap message Mes; Context window constraint e; Unwind token marign b
2: Output: External memory store D.
3: I < M, f
4: Mu < Pin
Self-program I, by bootstrap instruction 6: (I, < process_sii ciccess_intt_____ee cattto_romp_ppppe progam_pr D)
5: M, <- LLM((MM; Meb, Pmn)
7: while I is not empty do
8: C + (M,, Mrs}
D Clear Context Window: It always contains the system and
summary message
9:
pop (Z( + pop__rron front of D)
> Fetch: next instruction from external memory D
10: C² C" + I(k)
has instruction cettriioeeemeeeeeedeeee <- False
12:
while has innptrusstutonnnc cetpructtooocollteeeeeeeeeeeeeee is False do
13:
M, + PC*
14:
Act+ < {M,}
if 'function_call
M, then
16:
(M,, D) t- execute function(MM,, D)
17:
Invoke tool: Includes functions (tools) to read/write and evaluate the file store D A D011 Açt M,
18:
19:
"step_commp cosp_coppeeed""
M, then
20:
Mfe - EDD) if 5e is then
Run evaluator module on file store D If evaluator checks pass Summarize instruction output to
21:
while |C' Actes) > b do
Mrs. Unwind oldest messages to make room for summary control message Mes and summary response M,s
23:
C.C)) + hop_back_nne of_""("C""" Mrs PCIM(C @ Act+) @ Mes) has instruction_ _owpleteeedeeeeeee t- True
24:
25:
26:
Act1 Ac Me
Agr+ + Crtt ( Mec
Append the cycle message Would exceed context check Summarize progress to Mre. Unwind oldest
28:
if C @ Aco41 > then
29:
while C + Act+o) > do messages to make room for summary control message Mcs and summary response M,s
30:
(,C") pap_ppppeeeeeppeepppppppe
pop_back_oot back_ack back_oof("((C)
31:
Mrs + (C $ Aguin Mce)
32:
O++1 + {Ma,
33:
else
34:
C++1 +- Ct Aot
Append Messages
return D
be partitioned into files no longer than the context "i and we name the files with path from a root directory /where the name of folders and files suggests its content. This choice not only useful for human reader but will be crucial for our Read-and- Arrte implementations.
The remaining details are in the arguably richest component of the method, the control unit.
Execution flow of the LLM. In Codc-L2MAC, iteration. For each instruction (b) E I, while (()) is not satisfied, do the wwwlinggfollowing the CU pseudocode Algorithm First, load the fixed messages that provide in-context learning to the LLM, the description of the current instruction (h)) and the summary of the last execution My.. We
describe the execution flow the following
More precisely, no longer than half the margin left in the context after the initial messages are provided in any iteration in the execution. This is ensure that the I.I.M can read and rewrite any given file inside its context.
20

---

Published as a conference paper at ICLR 2024
let the LLM iterate (work completing the instruction) while in context, and if approaching out of context, summarize its current context and iterate.
Interfacing with tools. During execution, the LLM might request reading and writing operations. After each write (which possibly introduces a new test), a syntactical checker and all existing tests in DD are run, and the output is provided to the L.M-through M fe- If, after receiving this feedback, the LLM considers the task completed, it can signal it to the CU through calling a special function which summarizes the current context output into Mrs and continues executing the next instruction with this summary M,s. After all K instructions are completed, the process halts (stops), and D returned.
We only left to detail the process of reading and writing. Given the instruction I(k), the LLM must be able to know what modules in 'D it can call, how to call them, and where in the directory to write new code. However, the size of D might exceed the context window.
Due to the restrictions mentioned on the files in D. the LLM can read and write any single full file as desired and, if necessary for context preservation, summarize this information for the next iteration. However, the decision regarding which files are read can be heuristically based on the semantic information provided by the path and name of the file, which encodes its role in D.
Specifically, we enable the read component R of Code-L2MAC by implementing the two functions. 1ist_fi fifeelels (), which lists the file paths in breadth-fin search order. This could be refined to only list a subset of files within a subdirectory. but for simplicity. in our implementation, we automatically list all file paths in D for the LLM as contexts to enable it to be aware of existing code and help determine where to include newly generated files. And view_ few_flle files (fill (files), which takes a list of file paths and returns strings with their content-appending this to the context window.
We also provide the LLM at all times with system message M, to inform it that it is to complete a coding task, that it must act without human input, and a few details about the coding environment that exists within-for example, that it can use the python test framework Pytest, and can only mock in- in-memory databases as none are installed in the local environment. Although this could be resolved by installing and providing clean, fresh version for most common database providers, we leave this extension as a future work to explore.
We also provide the LLM with pre-prompe message that it uses with the user-specified requirements, to construct the initial plan, we also detail this and the system message in Appendix F.
C.I READ / WRITE IMPLEMENTATION DETAILS
Here we expand on the read/write implementation details of Code-I.2MAC. This builds on the existing descriptions contained within Section 4 and appendices C and F.I.
The LLM interfaces with the control unit (CU) through calling functions, and thus, the read and write implementation can be fully described through a discussion of the functions that are provided to Code-L2MAC to that end, which are 'read files' and write_files'.
is worth noting the control unit always exposes the LLM to the list of file paths that are already part of the codebase (c.g.. "file paths: l'app-py", 'tests/test peesst//est_app-py"p app-ppppp" I"). The name of directories and files (i.e. file path) provides a semantical prior for the order in which the LLM will read each file, depending on the functionality it is looking for (among other elements, including always the current instruction I, and possibly previous dialog turn outputs and error responses).
Note that if the LLM initially reads a file that does not contain what it was looking for the desired functionality is spread across multiple files, the LLM can continue reading other files. Recall that when the context window is full, the CU prompts the LLM to summarize the relevant parts of the context window for the current instruction I, In this case, it could summarize which are the relevant files needs read if the functionality is spread across multiple files.
Assume the LLM has already determined the name of the file 'file_path that it wants to read. Let us zoom into the functions (tools) provided the LLM to signal intentions to the CU.
read files(file "aess requests the CU to output into the context window C' the contents of 'file path'.
21

---

Published as a conference paper at ICLR 2024
"write_ files(file fath, content)" requests the CU to create overwrite 'file path" and popu- late with 'content'.
As the empirical results demonstrate, these functions perform well components of Code-I2MACC The future work appendix (Appendix 1) discusses how these functions could he optimized.
We find Code-L2MAC's reading and writing functionality different from previous memory-augmente LLM (where we refer to the extended related work for more details, Appendix D) as these can be categorized as:
Append only memory: These methods explicitly extend the implicit knowledge of LLMs through an external corpus (Zhong et al., 2022) facilitate long conversational or document summarization tasks (Liang et al., 2023).
Key-value memory: These methods interface the LLM with a dictionary or a database (Hu et al., 2023), which does not apply our automatic coding tasks.
D EXTENDED RELATED WORK
D.I VANILLA LLMS
The usage of only a single LLM without memory augmentation approach is limited to only producing an output that can be maximally as long as the underlying LLM's context window constraint c. Although not directly applicable, they are still related since the main focus of all coding tasks to date in the literature focuses on using an LLM to solve code generation task within the context window constraint Notable examples include Codex (Chen et al., 2021), GPT4 (OpenAI, 2023), GPT-Engineer (Osika, 2023) and Code LLama (Rozière et al., 2023).
Vanilla LLMs with longer context windows. One might try directly extend the context win- dow. However, such an approach scales quadratically in computational complexity (Vaswani et al., 2017), an impracticality worsened by the observation that LLMs with long context windows attend information from the middle of the window poorly, substantially decreasing their task performance as the length increases (Liu et al., 2023a). Another naive approach would be to fine-tune the LLM in the environment at hand so that it implicitly memorizes it in its weights, but this should be done recurrently the state evolves, which is also computationally expensive to be practical (Brown et al., 2020).
D.2 TRANSFORMERS FOR LONG INPUTS
There architectural approaches to directly allow LLMs to ingest larger windows. Tay al. (2022) offers a comprehensive review on this topic of refficiencc-fllvorre transformers.
Child al. (2019); Beltagy et al. (2020): Zaheer al. (2021) present sparse attention mechanisms that scale subquadratically with the length of the sequence. Guo et al. (2022); Phang et al. (2022)
draw on local/global attention mechanisms to scale longer inputs, Wang al. (2020) approximate the attention mechanism with a low-rank matrix to achieve linear complexity, Choromanski et al. (2022) provide an unbiased estimator of the attention that converges uniformly, Peng et (2021) use random features to approximate soft-max attention with decreased computational burden. Although this list not exhaustive, the precise impact of these optimizations on the performance of LLM not yet fully understood, and the adoption of these approaches in practice is not widespread.
In Press et al. (2022), the authors discuss biasing attention scores with a penalty proportional to their distance instead of using positional embeddings, which allows them to extrapolate to longer sequences in inference time effectively. Consequently, they achieve significantly more efficient training. However, this does not address the challenges of long context windows at inference time.
22

---

Published as a conference paper at ICLR 2024
D.3 MEMORY AUGMENTED LLMS
D.3.1 LLMS WITH BUILT-IN MEMORY
These are works that expose the LLM to memory at training time. Yogatama et al. (2021) extend the usual attention architecture in two ways. First, they extend the context window without much burden at training time by omitting the backward step for distant tokens. Second, they update the next token probability distribution by incorporating information about the next token for historical data for the nearest neighbors in latent space. Borgeaud al. (2022) essentially incorporates for each training example the k-NN (based on an embedding fixed through training) from an external corpus. Zhong al. (2022) propose boost to the next token probability distribution based on the similarity of the current embedding to the previous embeddings where this token appeared. These enable the LLM to be trained with this memory component but can essentially be thought of in the same spirit as embedding NN lookup memory. However, none of these methods enables the memory to be updated.
Finally, (Bulatov et al.. 2022) embeds into a sequence of tokens memory for the next inference. This approach, however, has a low-interpretable, non-symbolic, and limited memory, thereby not fully addressing arbitrarily long tasks.
D.3.2 INFERENCE TIME
LLMS
The plainness of the limitations imposed by a limited context window has motivated several efforts to overcome it, which has inevitably led to the concept of memory. However, most of such works focus on long conversational / text summarization tasks (Liang et al., 2023; Modarressi et al., 2023; Zhong et al., 2022; Cheng al., 2023), or mathematical/dataaass reasoning (Hu et al., 2023). The methods from the first group. due to the domain, focus exclusively on the read component of memory. The same happens with (Wu et al., 2022), which explores the k--N lookup to retrieve memory for next-token predictions and observes improvements in perplexity for multiple tasks, including coding. However, such read-only implementations are more sensitive to the erratic fallibility and the "stochastic" nature of LLMs to break code and cannot interact with memory in any other way. An exception to this is Modarressi et al., 2023, but its memory (and thus update capabilities) is composed of tuples of the form (Subject 1, Subject 2, Relationship), for which there is no apparent application to coding tasks. Similarly, mathematical / database reasoning methods such as (Hu et al., 2023) reduce memory to dictionary of values structured database, which again are not applicable different settings (e.g., coding).
D.3.3 AUTNOMOUS AGENT LLMS WITH MEMORY
These works formulate the LLM as an agent in a reinforcement learning environment, where it observes a state and then takes an action, often calling a particular tool with an argument and interacting with the environment until the task is solved (Wang et al., 2023a). Unique to these works they act fully autonomously, where they are often given few high-level goals and construct their own plans, which are continuously updated to have self-guided operation without external input, to complete their goals or run continuously (Richards, 2023; Shrestha & Watkins, 2023; Nakajima, 2023). Regarding code generation tasks, this implies that these agents can re- re-program and re- formulate their step plan at run-time; however, this can often lead to the agents going off-topic instead of completing the specified given task and getting stuck in infinite loops (Wang al., 2023a; Sato et al., 2023). However, in L2MAC we specify that it is automatic rather than fully autonomous so that the specified user input commands can be performed without manual intervention whilst achieving the input specified given task by the user. Furthermore, of the existing Autonomous Agent LLMs with memory, the only directly applicable method capable of writing, reading, and executing code of files is AutoGPT (Richards, 2023). This overcomes the context window constraint to complete long-running tasks iteratively lossly summarizing the previous actions completed after given number of turns and only keeping the rumning summary in the context window. Prevalent with AutoGPT and others is that they contain a first step of taking the user-specified task, and lossly summarizing it into a name, role, and five one-sentence goals for the agent to complete; this goal message also is always in the agent's 's context window. At each dialog step, the LLM is prompted with a ReACT (Yao et al., 2022) message template to reason, plan, and act by selecting the next tool to call until the goals have been achieved, then stops. Overall, these methods, such as AutoGPT have two key limitations: (1) they lossily compress the initial detailed user-specified task into six-sentence
23

---

Published as a conference paper at ICLR 2024
description (a sentence role and five goal sentences), and (2) they lossily compress the previous action history, which for code generation tasks equates to forgetting which files were created and their use, which is further compounded by re-planning with separate new plan at dialog step. Whereas proposed L.LM automatic computer (I (1.2MAC) can (1) create a detailed plan initially and use that single plan throughout to align complete the original user-specified task, and (2) I.2MAC is aware of the existing files generated, which encapsulates all the previous actions up to the current state (where the files track the current memory state), allowing it to know and integrate new code with the existing files already generated.
D.4 TOOL AUGMENTED LLMS
Schick et al. (2023) teach LLMs to use external tools through API calls, for example, to use a calculator, a search engine, or a translation system cover LLM weaknesses or reduce computation cost. The methods by which the LLM chooses when to read from memory (instead of automatically receiving the information obtained from memory) can basically be regarded as providing new tool to the LLM (e.g., Hu et al. (2023); Modarressi et al. (2023)). This is the case for the 1.2MAC implementation of reading and writing, but not all the functionalities of the control unit can be seen under this lens, as is the case with the monitoring of the context window or the automatic evaluation of the file store, unless we stretch it excessively. In any case, as argued in Modarressi et al. (2023), this similarity with Schick et al. (2023) does not undermine the relevance of the development of new and influential tools.
D.5 MULTI-STEP REASONING & REFLECTING LLMS
Reflecting LLMs. These methods can improve the task performance of existing single LLMs by having access to an evaluator and using the LLM to reflect the actions taken (current trajectory part) to derive actionable verbal feedback that is persisted in an append-only memory (Shinn et al., 2023; Madaan al., 2023). These verbal reflection learnings et al., 2022) that are then used to improve the task or sub-task performance when the task or sub-task is restarted anew. This simple, and practical reflection idea, when having access to a possible external validator, such as seff-generated unit tests for code generation, have demonstrated improving single LLMs for small code snippet generation tasks performance to be state-of-the-arr (Shinn et al., 2023). A unique characteristic our our checks is that they are not only aimed at validating the immediate output but rather impose coherence on the memory as a whole. This implies that while previous settings, a failing test would require a change in the LLM output, in L2MAC, it can also motivate revisiting and refactoring preexisting component in the file store.
form of in-context learning (Dong
Specifically, we find the following differences between Code and reflecting LLM methods, such as Self-Refine and Reflexion to be:
Self-Refine (Madaan et al., 2023) refines the most recent output from the LLM and Reflexion (Shinn et al., 2023) the recent outputs that are only within the LLM's current context window C"; whereas Code-L22MAC can refine the entire file store, encompassing all previous outputs from the LLMM-allown back, outside its context window, which are now erroring due to a new code implementation change. This enables Code-L2MAC to generate long, consistent, and interrelated code structures.
it to fix or improve earlier outputs from multiple instructions
Self-Refine and Reflexion are constrained to operating on an output within the context window constraint whereas Code-L2MAC manage total output greater than the context window constraint through the L2MAC framework.
Multi-step reasoning. Methods that leverage hierarchical flow of information between LLMs can also handle information that does not fit the context length, thanks to the distribution of information. For example, (Zhang et al., 2022; Wu et al., 2021) iteratively and hierarchically perform the sum- marization of long documents. We refer to (Dohan et al., 2022) for the unification of such methods under a common formalism. Code-L2MAC uses simple instantiations of this family to populate I from the initial prompt to summarize the execution when cleaning the context window, but future iterations could build on more complex cascades to improve our implementation of L2MAC. Multistep reasoning is only small part of 1.2MAC, and we envision I.2MAC enhancing multistep
24

---

Published as a conference paper at ICLR 2024
reasoning methods through the CU and memory (c.f. Appendix 1). Furthermore, there exist standard RL multi-step planning frameworks (Holt et al., 2024; 2023) which could be adapted to LLMs in future works.
D.5.1 LLM-BASED MULTI-AGENT SYSTEMS
LLM-based Multi-Agent Systems, comprise of multiple LLM- 1.M-pooered agents that interact with one another to achieve desired task (Wang et al., 2024; Guo et al., 2024). L2MAC is an example of an LLM-based multi-agent system, as it consists of separate LLM--ased agents that work sequentially each getting a separate next instruction (where the instructions were generated by an initial LLM- agent) and persisting memory the previous agent's 's outputs between the agent's, as illustrated in Figure 1. Broadly LLM-based multi-agent systems, facilitate sophisticated interactions and decision- making (Wang ct al., 2024; Guo et al., 2024), proposed through collective intelligence and specialized profiles and skills of multiple agents, to allow them collaboratively complete tasks. Such works include (Wang et al., 2023b; Du et al., 2023; Zhuge et al., 2023; Hao et al., 2023; Akata et al., 2023; Liu et al., 2023b; Park al., 2023; Cai al., 2023; Hong et al., 2024; Wu al., 2023; Li et al., 2024; Packer et al., 2023).
D.6 TURING MACHINES, STORED PROGRAM COMPUTERS AND MACHINE LEARNING
There are instantiations of Universal Turing Machines and Stored-Progret Computers with LLMs (Schuurmans, 2023; Giannou et al., 2023), respectively, which are theoretically insightful. However, in both cases, their method with a transformer at the core only aims to simulate UTM or SPC, thereby not achieving a method functionally superior to either UTM or SPC (and none of them new), which can be implemented much more cheaply. In contrast, we capitalize on the SPC framework to boost the LLM's unique capabilities as a stochastic NLP to automatically solve potentially long tasks from single user prompt, which is altogether different focus.
Although (Graves et al., 2014) does not use LLMs, and the focus is also completely different to ours (since aims to reproduce algorithms by supervised learning), it is worth mentioning that they extend a Recursive Neural Network with memory to achieve a Turing Machine or von Neumann architecture capable of inferring simple algorithms when trained reproduce them.
E
BENCHMARK TASK ENVIRONMENT DETAILS
E.
CODEBASE GENERATION SYSTEM DESIGN TASK
Given the absence of pre-existing benchmark task environments suitable for evaluating long code generation tasks, we have introduced a new benchmark tailored for standard system design tasks, which are typically assigned to humans. Traditional system design tasks often require high-level conceptualization, such as devising the architecture, modules, components, interfaces, and data structures needed for an online application to meet specific user feature requirements (Xu & Lam, 2020). Unlike these conventional tasks that predominantly focus on high-level system outlines, our benchmark mandates the implementation of fully functional code. This code aims to instantiate a minimal system satisfying the user-specified requirements in a practical manner.
For meaningful and realistic evaluation, our benchmark encompasses three standard system design codebase generation tasks. The prompt questions for these tasks are derived from actual system design interview questions, providing a realistic basis for assessment (Xu & Lam, 2020; Martin, 2023). The tasks included in the benchmark are as follows:
1. URL Shortener App:
This task necessitates the implementation of a comprehensive online URL shortening service, a utility that enables users to submit lengthy URLs and subsequently receive a shortened variant for simplified sharing. The user-specified feature requirements for this task are extensive. The envisaged system should facilitate URL shortening services, allowing users to input, validate, and, desired, customize their URLs. The shortened URLs should redirect users to the original addresses. Users should have access analytics, revealing data such as click counts, timestamps, and geographical locations linked to each URL.. The platform should also support user account creation and management, enabling
25

---

Published as a conference paper at ICLR 2024
users edit, delete, or view analytics related to their URLs. An administrative dashboard for monitoring user activity, user accounts, and overall system performance should be in place. Furthermore, users should have the ability to set expiration parameters for their shortened URLs.
2.
The task entails the implementation of web-based microblog- ging service where registered users can post short textual messages, follow or unfollow others, and interact with the posts. Each user can create, edit, and manage their profile with options to set it as private or public. Users can post text messages limited to 280 characters, optionally include images in the posts, and have the ability to delete their posts. The platform will support interactions with posts, such as liking, retweeting, and replying, with nested structure for the comments. Users can also search and filter posts or other users using keywords, hashtags, mentions, or view trending topics. Furthermore, users can follow and unfollow others, with a timeline view displaying posts from the users they follow and receiving notifications for new followers, likes, retweets, replies, and mentions. The service will also facilitate private messaging between users, with options to block or unblock users. Finally, the platform will offer user recommendations for following based on interests, activity, and mutual followers and showcase trending topics, which can be viewed globally sorted by location.
Online Microblogging App:
3. Online Chat App: This task involves implementing a real-time online chat application. The GCS must support user registration and authentication functionalities, including email sign-up and forgotten password recovery. Users should be able to set profile pictures and status messages, with privacy settings to control visibility. The application should provide options for contact management (block/unblock contacts and group management), real-time text messaging with read receipts, end-to-end encryption, image sharing, and support for emojis, GIFfs and stickers. Group chat functionality should allow the creation of group chats with assigned names and pictures, participant management, and admin role assignments and permissions. Users should be able to post image statuses that are visible for limited duration, with the ability to control viewer access. The service should be accessible as a web application and support offline messaging, with queued messages sent upon connectivity restoration and visible online/offline user statuses.
4. Recipe App:
This task focuses on creating service that allows users to submit, share, and discover recipes. Key features include the ability for users to submit recipes complete with ingredients, instructions, and images; options for categorizing recipes by cuisine type or dietary restrictions; and functionalities for editing or deleting submissions. The platform also supports searching for recipes based on various criteria such as ingredients or recipe name, and categorizing recipes by meal type or dietary needs. User account creation and management are integral, allowing for saving of favorite recipes and displaying user profiles with their submitted and favorite recipes. Additional features include rating system where users can rate recipes on 5-star scale and write reviews, a community aspect allowing users to follow others and see updates in feed, and the sharing of recipes on social media platforms. An administrative dashboard for managing content, monitoring site usage, and user engagement is also included, alongside system for generating recipe recommendations based on user preferences and past activity.
5. Event Planner App: This task involves developing a web-based application to assist users in organizing and managing various aspects of event planning. The application enables users to create and manage events, specifying details like event type, date, time, and customizations such as themes and color schemes. A calendar view for managing multiple events is included. Users can search for and book venues through the application. with integration of maps for venue locations. Guest list management is facilitated with features for creating, importing, exporting, and RSVP tracking. The platform connects users with event service providers like caterers and decorators, offering vendor profile viewing, comparison, and in-app messaging for coordination. Budget management tools allow users to set, track, and receive alerts for budget overruns. User account features include personal profile creation, customization, and access to past and upcoming events. Automated notifications and reminders for event milestones and tasks are part of the system. Reporting and analytics capabilities are provided for event success metrics, along with feedback collection from guests and vendors post- post-event. An administrative dashboard allows for the monitoring and management of user activities,
26

---

Published as a conference paper at ICLR 2024
system performance, and vendor listings. Additionally, the tool emphasizes security and data privacy, including secure payment gateway integration.
6. Financial Tracking App:
This task involves developing a comprehensive tool for managing personal finances. It should include functionalities for tracking expenses, incomes, and investments, as well as setting and adjusting budget goals. Key features include the ability for users to create and manage personal account with secure linking of bank accounts and multi-factor authentication. The application should support both manual and automatic import of expenses and incomes, categorization of these entries, and visualization of their history. Budget management a crucial aspect, with capabilities to set monthly budget goals, receive alerts when nearing these limits, and analyze spending patterns suggest adjustments. The integration with investment accounts to track performance and balance, along with overview of asset allocation, is also required. Additionally, the application should generate financial reports, like monthly summaries, and provide customizable alerts for unusual spending or important reminders.
Each method receives the same user-specified feature requirements, which are given below for each environment task, and produces codebase output, which we then evaluate using our evaluation methodology, which is fully detailed in Appendix G. The following user-specified input prompts for each task are:
URL Shortener App, user-specified feature requirements input prompt:
1d O-Oalide Shoctening Sersicess
+*Oserview
A service that allows users aebmit long URLa and then receive shortened version that URL for case of sharing.
sonncctional Requinomests implement.e:
URU Shostening ax
1.1. Users can ispot a URL to be stortesed.
1-2 The system validates that the LRE is active and legitimate.
1.3. sssiem zeserates ssique sherteaed
1.4.
can choose castom short lioks (subject availability..
Analyticss:
[ 3.1. Users can view statisties about their shortened URL..
[1 3.2. View namber clicks.
[ 3.3. View date/time of esch click.
[ 3.4. View geographical location of the clicker.
u<erer Accoents
Usere can create accomnts.
-[] Accnent balders cas view their sbortzzed UBI
[ Account holders eaa or delete their shortened
Account holders cca anolyties all their shortened URLs.
Washbozzdee:
Admieletrarass delete LBI, accoust.
5.3.
Adminiesraraa
anstyrics.
Expiirationree
[. 6.1. Users can set expiration date/time for the shortened URI.
Online Microblogging App, user-specified feature requirements input prompt:
Microblogging Service (086) Descripticn Requirementere
Description
post
massages, mossages
and
provides features for individual administrative menegeneat.
Puactional Regsiremests implementt:::
User Masugencnties
I. Registrarins & Autrrnticatinn:ss
Allew usrre to register asing email, suarame, and passoard.
Option to resei lorgutten passards.
Secare outhenticatinn using JRT or similer protocols.
2. PaPofflle Mateponentis
- Users can their profile information: profile picture. bio, website lisk, rnd location.
Option
profile private or pablic.
Posting
Matagement
Allow iaclude text-based
limit characters.
Option
Usere dalete their
2. relateracting vill Posis iss Users like. retweet, and reply to posts.
Nested comminl viraciore for post replics.
3. Content Filisring & Searchise
Usera can cearch for specific posts or ssers sting keywords.
Filter option based on hashrags. user montions, or trending toplos.
27

---

Published as a conference paper at ICLR 2024
vediiil Interaction
Following
followed
aotifications
Private converiation threads betwenn
Option block/astloct
from
3. NNotifications
Uhere are natified of lites, retwenss replics, and mentions.
Treading Discoveix ...
1. Trenndiag Topicsine
Systen identifies displays treading aishiags topics volume velocity of mention. Trending topics csa sorted based location globally.
Recommanddiiomm
activity
Online Chat App, uper-fie feature requirements input prompt:
Globbll Chat Service (003)..
real-lime online chat application slloxing users text menaags, images, and create group chars.
Regsirements implomente::
Registration Anthenticatios:
1.2. Forgoteen email.
User Profile
-1122 2.1. Allow ssers sot profile pictures and statos messages.
-I 1 2.3. Pricacy sattings for who the see exer dxtalls, Tast sora srarus.
Contect Mangement:
3.1. Block/unsiuck contacts.
[ 3.1. Creste, and mangge groupt.
Messaging:
massaes.
equivcleat).
4.5. Enoja, ulb, and stickers apport.
Grop Chats
-1 -11 5.1. Create group chals with nume and plature.
-1] 5.2. Ade or remose participazzs.
[ 5.3. Minn roles and permissions.
Statox/ Story Feature:
pest image stateses visible for Costrol cas see states.
7.1. Wab-based version accessible browsers.
Connectivity Offline Made:
s.1. Messige quezing for ubes user is offline; seet connectivity restered.
8.2. Display oxlise/ offline status.
Recipe App, user-specified feature requirements input prompt:
A service that allows
share and discover recipex. includex features for categerizing
seviewe, and persenul
Pusctional Reguiremests imploment
Recipp Subaission and Mangemmaaie:;
L.I. Usess can vobmit recipes with ingredients, instructions, cad images.
1.1. Rtccips biiieiinn inclade options for categerisation (6.g.. cuisine type, dietary restrictions). delete their submitted recipes.. 1.3. emn validation to casrre complets informatios.
1.4. Recipe
Cateporizations
Usess Categorization free).
based ingredients,
breakfost, luach.
3. User Accousts snd Profiles
1.1. tisess can create and manage accounts
[] 5.2. Accoint holders ran save favorite recipes.
5.3. Profile pages showing suhmitted recipes and favorite recipex.
Ratisgs Resiews.e:
Userx em rate recises on S-star
4.2. Usesa can reviews recipes.
4.3.
Display avecuge roting en recipe pages.
follow
Feed showing retivity followed
ratiags).
social media platforme.
evidiii Dasibozdde::
6.1. Adminstraiore can mangee all sabmitted racipes.
6.2 Adminstraiors can remoe inaggropriate content.
6.3. Motitoring of site mage statixties and user eagagement.
Rocipp Brommadiiionss
28

---

authestication cahasced Published as a conference paper at ICLR 2024
7.1. System zenerates recipe recomnendatioss based aser preferences and past activity. notifications recipes
Event Planner App, user-specified feature requirements input prompt:
Carton Planner Tooles
app ppliiatiin desigeed
organizing mataging aspeots planning.
provide
type.,
Punctional Reguiremests implement
Exentt Creation and Masagomeatas"
L.I. Users can create now exant spacifying detaile like event type date and time.
1.2. The system allows for csstomitaion events le.k.. thenes color schemees)
1.3. Uses can update or modity event details as uceded.
1.4, A celendar view is available for esecs to marage neltiple events.
V<Vense Socrcing ee:
capacity, type
Integration mage
directly thruugh applscation.
Guest List Marageneites:
1.1. 3.1. Users create and matare zeeat lists.
3.2.
list feature. 3.3. RSVP tracking cad mmagencat.
smport/export
4. to Vender Coordiration ost
1 4.1. Platform to consect with various preat service prosiders (eaterors. decorstors).
4.2. Users can view and conpare vendor profiles rad rexiews.
-I te-app messaging gyten for vender communication.
BBBgget Menegenentee;
by catzgorize catering.
aether Accoents and Profites-e:
1.1.1
6.1. Ustes om create personal profiles.
6.2. Profile costonication to reflect crent plaacies prefereaces..
-[1 6.3. Saving cdd accessing past and upcoting eveats.
Notifications rnd keminders
1 7.1. Autemmted ESail/MM sotifications for evest milestones.
1 Castomizable reminders tasks and deadlines.
Reportins Andlytias
Geserate reports
success netries (attendance, bedget adirennce. collection fren guests venders post-event.
Adnin Dashbozad
and Syslem parformance analytize and vender listings platform costent.
Administraiors
activities.
statiatics..
10. Security and Data Privacy vo:
10.1. Ensoring user data protection and privacy.
10.2. Secore payment gatcway inleutation for transctioos.
Financial Tracking App, user-specified feature requirements input prompt:
MF exPesonnal Pleasce Tracking Application.s
compretansive toal for manging personst finances, including tracking expenses, incomes, investments; setring hedgei posts
Punctional Reguiremests implement
exAcconnt Security
1.3. Milli
Expencee and Incone Trackingss:
-! 2.1. Menui end astomatic import of espeases and lacomte.
2.2 Categorization of esgenses and income sources.
- 1 3.3. Visnalization of repanse and income sistory.
Budget Menggmentts;
3.1. Seiting and sejusting monthly budget goals.
3.2. for nearisg budget limits.
3.3. Analy speadiss pilteras suzzest badact adjestments.
investment balance. allocation.
Trecking
4.3. Overview
Reports and Aleris
5.1. Grazration of finascial reports (eig montily summoriss)
4.2. Custemizable slaris for unmeal spanding important reminders.
E.2 HUMANEVAL BENCHMARK
We use the standard HvmanEval benchmark, as introduced by Chen et al. (2021). This benchmark evaluates the task of the underlying L.L.- method to generate standalone Python functions
29

---

Published as a conference paper at ICLR 2024
from a given doestring, and evaluates the correctness of the generated code function through held out unit tests. The benchmark consists of 164 hand-written programming problems with unit tests, which aim to assess language comprchension, algorithms, and simple mathematics, of which the authors (Chen et al., 2021) compare these to simple software interview questions.
F BENCHMARK METHOD IMPLEMENTATION DETAILS
In the following, we detail all implementation details of our benchmark methods, including that of the full low-level implementation details of Code-L2MAC in Appendix F.1.
AuT.GPT. We chose to compare against the most competitive and popular autonomous agent LLM with memory method, that of AutoGPT (Richards, 2023). This uses GPT4 as the underlying LLM. Like other autonomous agents, the first step of AutoGPT is to summarize the user-specified task into a name, role, and five one-sentence goals for the agent to complete; this summary is kept in context throughout execution. At each step, until the goals have been achieved, the underlying LLM receives a ReACT (Yao et al., 2022) message template to reason, plan, and act by selecting the next tool to call. Specifically, we used the latest stable version of AutoGPT, 0.4.7, enabled local tool execution, and disabled searching the web for answers to ensure a fair comparison. It is interesting to note that AutoGPT also has tool functionality and has tools to read, write, and list files in local file store making it the most competitive and applicable baseline to compare against. Although other autonomous agent LLM methods exist, they do not possess the ability to read or write to an external file store in the same manner AutoGPT. Importantly, AutoGPT, when running, would sometimes have runs where it would fail to complete the task, i.e., fail to output any code or get stuck in an infinite loop. When such failures occurred, we classified these runs failures and excluded them from the results. This method defaults to use the LLM with a temperature of O.
GPT4. We also chose compare with the single GPT4 LLM (GPT4) (OpenAI, 2023), which adapt to make it competitive by providing it with the same tools that Code-L22MCC uses, forming an ablation of just using LLM processor without the control unit. Specifically, we provide it with exactly the same messages and experimental setup as Code-L2MAC, including the same system message and prompt format message; however, it is limited to only respond with one long single response that can fill up the entire context window C
CodeT. We build upon our GPT4 method implementation above and implement CodeT a recent state-of-the-arr method for code snippet generation (Chen et al., 2022). CodeT samples independent generated code outputs, and uses the same L.MM to generate test cases for the code samples, it them executes the code samples using the generated test cases and performs a dual execution agreement, to return the output code sample that performs the best against the generated tests. To make this method more competitive when applied to large codebase generation tasks, it is possible generate given codebase statisfying the user given feature requirements, however with many different possible implementation approaches (of splitting components, classes and across files and folders)-thereforr therefore instead of sampling tests independently of the generated codebase. we follow our setup with GPT4 and Code-I.2MAC and generate codebase and unit tests at the same time so they match the given implementation. We repeat this generation independently for n = 3 times, and run the codebase against the generated tests, and rank the codebases in order of the number of tests that pass, and return the codebase that passes the highest number of self generated tests.
Self-Refine. Similarly, we also build upon our GPT4 method implementation above and implement Self-Refine, another recent state-of-the-as method for code snippet generation (Madaan et al., 2023). Self-Refine, uses iterative self-refinement that alternates between using an underlying LLM to provide feedback and refine a given generated LLM output, to create a higher quality output. Specifically, given an initial LLM generated codebase output, we use the same LLM to provide verbal feedback on how it could have improved the codebase to the given original task, and then using this verbal feedback within the same context window, we refine the previously generated codebase. This process is repeated = 3 times, as in line with n 3 used in by Madaan et al. (2023). We followed the same coding task setup as in Madaan et al. (2023) to make this method competitive. Furthermore to make it more competitive, after generating the codebase, Code-L2MAC and include in context the result of any code errors found and the result of any tests generated that are failing, it can also use this signal to improve upon the generated codebase.
use the same eπror checking methods in
30

---

Published as a conference paper at ICLR 2024
Reflexion. Likewise, we also build upon our GPT4 method implementation above and implement Reflexion, another recent Reflexion, another reflecting LLM approach, uses the same LLM to reflect on the trajectory (outputs of an LLM) once an environment episode (task) has been completed to provide verbal feedback on how the task could have been completed better. These verbal feedback signals are then persisted in an episodic append-only (list) memory buffer and are given in context upon the next episode of completing the task from scratch (re-starting the environment). Reflexion converts a binary or scalar feedback from the environment into its verbal feedback, that is then used as additional context for the LLM in the next episode. To make Reflexion more competitive, after generating the codebase we use the same error checking methods in Code-L2MAC and include in context the result of any code errors found and the result of any tests generated that are failing appended to the context after generating the codebase. Specifically, Reflexion follows an iterative process where it generates codebase given the same initial prompt as Code-L2MAC and existing methods, then it is asked to evaluate the codebase for the number of features fully and functionally implemented as listed in the input prompt, giving a scalar score (the reward signal for the environment), which is apended to the context window. This is followed by the self-reflection step, where it is asked analyze the trajectory (codebase, the evaluated scalar feedback) to produce a summary on how it could improve (verbal experience feedback for that episode). This summary is then stored in external buffer. We then reset the context window window, starting with the initial task prompt plus the verbal lessons from the external buffer, and repeat this iterative process n 3 times be be competitive. We follow Reflexion's 's code programming setup and prompts, adapting them to our task to be more competitive.
method for code snippet generation (Madaan et al., 2023).
Code-L2MAC. Our proposed method uses the message and function definitions defined in Ap- pendix F.1. Using these, it then follows the pseudocode for the control unit, as outlined in Algorithm 1, and generates a prompt program of instructions I and iterates each dialog turn of the control unit control flow loop until all instructions are complete. There are a few other minor implementation details, which we also detail here. We impose a maximum number of times an instruction can be re-tried TMax. with new context window when, during the execution of that instruction, it attempts to exceed the context window, forcing the context window to restart the instruction with the summary of the current progress made
empirically set M'ax 30. When the current number of retries exceeds FMax, we class this run as a failure and exclude its results. Importantly, at such a failure, the method could ideally either await human input for correction continue with an errored state. We used the LLM of GPT4-0613, and when using it throughout, set the temperature to 0.01. Additionally, another implementation detail is that when the CU detects that it is stuck in loop repeating the same two messages over again by comparing the most recent two messages in the context window, it increases the temperature of the LLM by 0.1 and continues until the temperature caps at 1.0, and then after it exits the loop reducing the temperature back to 0.01. This is to have a form of simulated annealing of introducing randomness to escape the local minima. In the following, Appendix F.1, we detail special message templates exactly.
F.1 CODE-L22MC Low LEVEL IMPLEMENTATION DETAILS
We follow the same setup as outlined in Appendix C, and in the following, we provide exact low-level implementation details. We use GPT-4-0613, which has fine-tuned support for function calls, and use the function definition file of the following.
Function definitions:
"sume" to "provide detailed, cu ttssttss__eepppee for for__aus_sgeene "dsecription to For preducing ates stystee plan, where each step parazraph is desailed sub-task step for separate ssh-agent (large language modal agent) in Within each detallzd step paragraph. always Include Tast sentence to create and ras trsts wtea implementing or writing code in that same stop.
"paramcters"
"Iype" "ebject".
properties":
steps".
type "array"
description" "List of strings. where each string is separate step sab task paragrapá for separate step paragraph. always agent complate. implementing writing
step.
"ieess:
"tppe" "string".
reguired is (' (eeepp""
"esh task step complete"
31

---

Published as a conference paper at ICLR 2024
"Jserription" "Call this function when the specified tack step has completed."".
"parameters"
"ype""
"view "view_files'
deseription + *Print our the file contents into the respoase to view.",
paramaters
Type" objezt
propecties".
files
tyee "array"
description":
of files view".
items
"string" aceuming reprosented string
"reguired" (".
some "rus_pyihon fite".
deseription "Knn python file and relorn the output to the resposse to view. That in with (yython) Eil_ztaee str.sammeeeooooooooeeoo ron
"prraeterss"
"ype" "object".
propectiea""
"file sane, aaneettou tourxe"
type "cering"
"argemens""
description "aptional ron orgomeats"
iteus [ lype "string"
"eguired to run "].
"pytest_fille""
scaponse
parameters
"ype" "ebjezt"
propecties".
riiles tost
type to "array"
description" "flle nunes to con prlest on".
steus " lype "string"
"weite, "iles"
"dcecription " "Wiit multiple files and will be combized into the existing codebsse. Always the whole
file. You alwass incent ende with tabs." "paramzters".
"ype"" "bjeet""
properties
filles and__ contects"
type " "array"
descriptioz".
of filee and their contents.",
"sspe "object".
propartize
-ile_paiah".
"Ippe
Pash the file"
centecsntssts
lage "string"
description"" "Contests the file"
"equired"; l'filee path "file contesss)
"requited": ['files_end_oottee
"aome "dalate files".
"deccription "Neteie filex. Specify the file these filex will deleted If specify the file "-1"
all files in the folder will be deleted.". "parameteis ";
"Tye"" "ebject".
"roperties":
frles"
type "array"
descriptiis". "Iist of fites delete. If you provide tome "-1" filex the folder be delated.",
Type"" "string"
32

---

Published as a conference paper at ICLR 2024
required ('filexs), "I
With Code-122MAC the following system message (OpenAI, 2023) M,, and include it all context windows Cº, which is standard to do so (OpenAl,
System message Mg:
Objective: Write code for a large system design task. Please note that the code should be rally fanctiond. No placeholders. only th the firctions you have been provided with. Oziy use "ilite
output cods.
You mast aet extonomossly cad you will receive ne henm inpat any stage. have reture is extput the complete code for correctly incorporace
You completing
esieting
Please always view files before writing always with profix
writing the files.
when writing
mike the filename
Provide the minimal code azcessery to achieve the task coacitioned on the existing renerated code mode--inellee includiag changies the existing genersted code.
You exsnot vissalize myy rraphical outpat. You zsist withis a Actor Mocel machins, snd when you list out steps, each step will be taken by a nev separalc wubChaitPi model. Ween sou list out sab-task steps, you can optionally specify the sub-1ask validation to check that it has been complated successfully.
You caanot use any datoboses note are setup in local envirosmnnt, insterd mack a detabase with in in menny dictionary to store deta. data saxed to disk will persist beiwera steps or wiite operations.
failing
incorrect, feel
change the they
incorrect,
pass.
The the fonetions provided. when catling funcrions provide RICKS compliant ISON request following this format withorr deviation.
When Code- Code-L2MMS L2MAC self- peff-prorsams its instructions initially, we provide it with the following prompt template to do so, which encapsulates the user message My, as 'user_ user__pecified
Mes bootstrap instruction:
You get Instroctions for write. byy names
every classes. fenctions, mattods be necessory. he fully functional. placesalders.
quick conment their perpose.
Please
You will with the "estrypoint" file.
the that imposted file. Please that the code should be fully fanctiuzzl, placeholdere.
Fallow language and frsmework appropriate best practice file samin convention.
Mike sure that files continn all imports, types etc. The code ceeld be rully canctional, Mike sore that code in diffarent fills ar compa compotibla with cant other.
Whe writing code if you are caiore. erite a plausible implenentation. Inclede modale depeadency or package masager depeadency definition tile.
Oseful to know:
For Pytionn - you always create asproptiste requirencatts, ti file. briefly descrihing perpose
explaining peactices
complex
Alwas follow package ard my
lagagges for folder/file strecture and package the project.
You
iastall, in loal enviromment instead moct databasee with meney dictiorary to store data. No data saved to disk will persis betwees steps wrile operations.
packages
You
databasts
What writing test. mule the filename start with the prefix "test_".
Python canihelt prefenences:
pylest
dateclosses
flack
Objective:'*
seceesspecfieeeffeteeececc reqsire_.qsssrrmmuusss Tee
Understaad groMen, oreating oxtromely daiziled ste--by-step plan. where
is long (multiple Intal includes every cizgle feature requirmmnnt specified abave. feel free copy direcily from It. the mare than 10 the plan Creste additional texrs, cherks and evalnatios implementatioe. whese all the code is fully feactional. Use best software desiga prectices. and you cus ouipst large amounts of code at once. Please include last seatence to create and run tests wher implementing or writing code is that sone step. You will esceive so hamas input at any stage. to you eascol use a suman to teet. Only create a detailed plan to berin with which includes designing and rassing tests to cbeck that they all pass. Please be sure to include all of the specified feature requiremeats in the followinz plan.
and in
applicable
excallent code
Code-L2MAC also involves control messages of the following form.
Moe Control cycle message of starting the instruction:
Osjective: Execute oub tack step: [inctruction] tatn Note: Condition aay new code filze the existing code filex: [lid_ filex(1 fully implement these features in the code, no placchalders. You cas nos optionally view existing files if you need to
them complete current step. You have limited context window so be selective about which files you view, only
files you you night peevios_insttuccion aatpouosnanttpptnsnnee (""ssss fasciioo_nanees)
AateSemnne outpot presious step: "*I followinz fenctions provided:
code
"writeffiles fusstion
33

---

Published as a conference paper at ICLR 2024
Here instruction is the current instruction of operation, 'list files()' a list files tool function, previ- ous jossissttuction petructtooouupuutttt summary is the summary message Mrs, and 'function names() lists the current function names that the LLM can request.
Mac Control cycle message of continuing the instruction:
Hax the wab rack step bera completed (inatraction)
Vala yes, call fanction task step complese otherwise reflect cerrect full code complete the Ouly you
extput oxsper only "writ__files If yeu AtRespped
files fenction eall of the following fanctions provided: (fiotios namex and you ourput code code.
the before
only the "write, fillxs fuxction
Here instruction is the current instruction of operation, 'sub tarbbttsk_stt_ step_ seep_comlette" is the function the LLM calls to indicate that it has completed the current instruction, 'list files() list files tool function, and 'function names()" lists the current function names that the LLM can request.
Mes Control summarization message for restarting the same instruction:
Please provide onz or two sentence summary of the output of this step which is usefal for the aest step Your response will be used when starting the aext step without any of the presious messages.
Mes Control summarization message, for summarizing the instruction output-which is used when continuing the next instruction.
You exhausted your context vindow. Reflect your pregress. Provide short coseise Jespoase, of two seatences maximut, this will restart this step from the beginsing withest previous messeges.
G EVALUATION METRICS
There are no automated tools to validate the output of system design tasks, so we propose evaluation metrics to compare the output of different methods. Large-scale code generation is unique in that the generated code can satisfy the high-level user feature requirements specified in the input through various possible implementation approaches.
To quantify the degree to which user-specified features in the initial prompt are effectively imple- mented in the generated code, we introduce performance metric named Features %. This metric numerically represents the proportion of input features that are fully and functionally implemented in the output codebase. The Features % is obtained by using separate GPT-4 API call, which iteratively examines the entire generated code to verify the functional implementation of all input features, counting the number of fully implemented features. We quote this as a percentage of the features implemented over the total features specified by the user. Specifically, we implement this metric by first collecting all the generated code files from a method's generated output and outputting all the code files as named strings of the format file name, followed by the file contents, and pass into the prompt template below for the GPT-4 API to count the number of fully implemented features. We then use regex parser extract the final number from the LLM response message. We performed a human study in Appendix H.6 and find that this evaluation metric correlates to that performed by human professional software engineers performing manual code review counting the same number of features implemented.
We now provide this prompt template and then provide example of it in action.
Features % prompt template
Objective: Based on nombered features giver, you are evaluste following code and return numeric value for how many (a count) thoss numbered features are imsiemented is provided code. Give the numeric snsver "EEXTUMS,PUUNCIOON" features furctions!"
Numbered Spectfied:".> loser_specified weaffidddeeatee_eeeeeeeeeeeeeeee Ceatore_reqsiremennts the
Cude to evaluste for the anoust of features fally implemented:"""
[ilee_
Here 'user specified feature requirements' is the eser-speeified task description, which provided in full in Appendix E, and 'code, files' is the entire contents of the codebase printed out with filenames and the contents.
34

---

Published as a conference paper at ICLR 2024
A worked example, be seen in the following, for example Code-L2MAC for one of the runs for the environment task URL Shortener App, with the complete user-specified feature requirements is given, including the GPT4 call'' response.
GPT4 API call.
requicements
Shortening you following
For
-BBsslatiion Chzcklist for Galine US, Shoriening Servicess:
+HUR Shortening sx:
-[] 1.1 Users can ispor URI to be shortzzed.
-[] 1.2. The sssiem validates that the URL is setive and lexitimate.
[[] 1.3. The system seserates a ssique shericzed URL.
[.] 1.4. Users can choose castom short lioks (subject to availability..
Kedirzzeii
2.1. Accessing the shortcard redirects the original
statisties their shortened View maler clicks.
date/tine
click.
3.4.
View geographical location of clicker.
eetheer Acconntess so:
-1 [[] Uhere can create secoints
-
3 4.2. Accoent halders as view all thzir stortzsed URLe
-
[E] 4.3. Accocnt holders caa edit or delete their shortened URLs
1 4.4. Accocnt holders can view analyties for all their shortened URLs.
Dashbozader:
Administiators
performance anslyries.
Expiccation:::
Usere
aspiration date/time for the shortened UR..
Total Implemeated Featureseo: [Cozst the checked boses gel the aumeric value]
You can 2o through the provided code and check off caca tien that has been implemented. the end. count the checked boses to the nomeric value for rumber of features implemented.
eveluste anoust feateres filly implomested sealytiss. by
datetime
def teack (lick
Test that click events are traeked cocrectly "**
analytics track, track_ abe123". "6.8.888")
data analstics aet_analyyiccs ("abeli ('abe123')
axcert (en(datz)
axeert ialesiance [0]] time "). datetime. datetims)
axeert data [o][ ip_address "E.....5"
def Test analyties
retriexed oet_onelytiiss abe123")
data
sasert len(dats)
essert istestance (date 10]1" une une'], JI. datetime. detetime)
assert data 1011 'p address "I an "....68"
analytica. PY "yythos
import dstetime
def crack_click (sberi_url. . ip_addres))
coaYrakk click event. if shorr sel not anstytice dora" analyties dats shori
Record the click event dlick event
datetime. datetime aow().
ip_address ip_addeess
dallytice bath apperd(clie_.evenn)
def onalytics short 1r)):
Retrieva auzlyrics
sbort URI
reture unalytica, brl,, (I)
def get system, WoLLeystem_perireeee performance
Reirieve system performance data.
Returnst
35

---

Published as a conference paper at ICLR 2024
diet: dictionzry containing system performance data. simplicity just
leatancyytics data), "aun_clicks"
seolyties O))
rEst EECT, accoIzIe
""yytho
import pylest
Uxcr_n import UscrAccount
test_ aCCOUEL)
euccesially.
"jossword" "Usersone already
def view uris::
accouac UserAccoutte)
accounc create, Eccounte
"eest" teast""""""""
"passwerd"
sacer: neer account viEw, arist arix("tost sacert user_accouni view_arikt sonextstent "Dernam does not eaiat."
def test test_ode odd orl:)
Eser account UserAccounte) see__acconi..eeeae crestt. ECOEENC Leat "paaswrrd
Gsseit useraacount add_ucll" test "http:l/ examppe
seccessfelly.
Gssert uecraccout add_ conexistent"
bttp excapppppeecoo "Usemnae
UserAccout())
aecountt accoint art' crrer 'hip esaplam com Gseet accouat delete_u:l uf
"pacesord")
"attp ereeouple conn') "URL removed successfully..
Gssett oser, accoust. delete uull"" test", "htip aaanexistentt.co "URL does not cxist."
Essert oser, accoust, delete uull"" a'nnexisttntt 1, "hmp dresample .com con')) "Usereante does not exist."
arl ebo_eborieeerr.r 75
"pythos
import
Import requests
Madk databaxe to the mapping sbortened URlx and origiazl URL arl_mapping
def garorare short arl (erl expieation miruresalfy 211): Geaerate taique shortesed URL.
Arge:
(st]]: The original
espiration, minstes (ini): number misstes expirea.
shortened URL.
bosklib, adScrd encede())
url bobectject (ztdigent()
Store the mapping betwer the shortoned URI and the original UR.. arl, popping short url] a ard erl. "xpprcction" detetiace.oon linedelto (minuteoeexpication minetes))
return sbori_url
uel Catels The URL validate.
Returas:
bool: True the URS is valid, Palsz otherwise.
try: resporse requesis pet(urt)
relnre response aratne code
except requeats exceptions. RequestExcepiion: reture False
def (short___u)
Gen the origieal LRS associztzd with the chortened URI.
Args
oro (ste): ): The shertened URL.
Returns:
sir: The original URL, or None if the shoctened URL does not exist or has expired.
drla uri, mapping, gett chirshari.uz)) dria datal expiration 1I datetime now():
reture dal_d u'u
def aebetc
Delete shortened URL
36

---

Published as a conference paper at ICLR 2024
short_adl wel.mapping
act get all urie::
Get all shorteand URIA.
Retucas:
list: A list all shorteeed URLs.
reture suppterl_
imgort shortezer
foo datetime impori ditetime, timedelia inport lime
def tess generate arl()
Test the geserate short url fenction,
hhrps Hhnn, examplr oon
sbort_url url bbrrteer (esteaiccchhoortalllts))))) assett cahrrtrtaarl
cssert isiestance (sl sto)
def vasi_yalidate
Tor validare orl fenction
calld "hippe: s/roe invalid_arl arl "hitps dtww. invaliducl cliducl.com Gssert orl shortener. validate arll valid_url) is Tuee essert orl shortener validate arll (nvalid__un ort) False
def text ()
Tear that expire correcily.""
hhrperltonn exmppic
short_ocl shortener, get original short,
generate_ turt expiratio_naisuten
shoct_ucl sri_sberteer scl_shociessr get_osigioal__
el
Wall for the UR lime (61)
sssert orl shortener, get, original (horr, edl) is Note
DEEr SCr_ECC 77 pythos
class UserAccount:
create_accone accomntiself,
'Usemane already existe.
self accornls/ paaswedd password, 'uris's DF
reture 'Aceeant created suscessfully.
def view aristseff sermame):
if esersame not in self, acconnts
retuca "Usemene does not exist returs accousis astrmame II
URR added
def Lelete_urliself zscrname nrl):
If useraams not self. accounts:
retuca "Usemane does aot exist."
if arl soi in 12lf accounte] sxermae orls 'I: retura URL coes not rxist.
self, accouets uscomnss Inseormee urls aem)ve(arl)
reture: 'URL mooved enccessfelly
selete_accountiself. oscraame):
osersame eccorsts:
'Uamann dots exier
def aet_ell_ sellassesssssll self):
reture list accounts ())
reguiremente
pytest
moxminddb moxnindd--gelllee?
37

---

Published as a conference paper at ICLR 2024
"pyyhhe
Pytrat
del client()
confiat i TESTING Tree with rest e cllent: ylold chert
def test hone pepel celinc)::
response client zett (")
sssert response. code
easett respoase
Hello.
def test_cedirect, original Ladiiialldeeleen response elient. gett ")
codo
def test, chorten, ari (client):
responce client postl "Jatories" datasl onl
hhiips c/wwww google
sasert respoase cistes_rode mn 200
assemt ssert orl resposse.gel gel. Json
viow, anclytics clicat):
respoase elient, gett"l anooytties/abbll2 respense.
analytics resposse gelpsse.gel_.jonn
def test. accoent cclient::
responic client. pest( "Jecconnt". duta=]"xction "create" "oserneme" !
"lestuser", "passwod""; "testpess '3
Gssett respoase, steles, code =- 200
essert respoase, get geo_ison () ('nnnegee""
"nessgee'] II "Accontt created seccessfally."
responic client, peatl /eccount data=|"sction" setion "view_arls "asereame" : "testeser ']]
axcart resporse cranns rode axcer responee ger (son.))
respoase pest(
data=l"<ctton ccction "delete" "osecocae" " "testuser"
respoase.
deleted
def teat dosia dachhoard: client):
client. 'aamii"
essert respoase. stetes code 200
essert "arls" is response Json(] ascert "asere" in resgance ger (sea()
Gasert "performance is response Ist.
delet_e articiennrtc
cliest, post Vaadmin dataml action 'doleie_arl" "shori_url" "abe123 response ger("rabe123"" response.
def delete, aser(elient):
cliest. post:" faddinn dalan] action "delete_user'' ", 'uremame'; "textuser ']) responie client, gett Vadmie " ascer: "restuser" not rosponse jusec)
app P
pythos
import redirect, ucl_for. request
import original,
get_all generate
asslycies gei gelayysee..prfoomseee gel_analysiess
Flask(
user_ accouat UserAccount))
Stpp ("")
del bonc()
roture "Hello. Word""
Stpp teetor or)
def redisect_to, tortc_____oiiiiioee (sbeent
original_uri get_origiool, ochsher.. original_url
track (sbars
redirocte original.nr))
Zup- coste contee"/s ('sstorte" methods= "POST")
def shorten, (1:
uel request form, getl if not validate_aril crl(u:l1:
raturs Invalid
chort.ure generate shori, (url) usernams request. form getl" asername")
coseeramm,, short_xrl)
wup.roste/
38

---

Published as a conference paper at ICLR 2024
view.aaaayiieaaaceree (aheriii ('analytics get (soly urls]
acthods- ()
def
acticn fonn. "serraam"
request
action "creaic
password a requett form gett
pet("pacseod" "pacssord
rature ("mossage" use account creare, aecomnn password)]
clif action = view arls
reture ('arls" uscr_accoeat siew, ucls (ssersame))
elif action a "delete"
reture massage's Excr_ delete aecouunt
Sup coute! "Tadnia" aethods-
"OSTT))
if delete form ('shor or
delete_ari (Shoortt_r
elif action "delete_ user
mername request form ger( l'marramme))
seer account delzte_acooant (ssermam)
reture redirectiari, for 'ndmin_dnthhrrd
else:
nrts get all eris()
all_usera user_ acoost--ptt_alll allerel pasformaice_ seluyyscen,prrorraae
all_orls, "osers" alll osces. "perfermanae" performonce__dee
debug-True)
GPT4 API response
The prosided code implements the Tellowing features:
+.UK
Shortening sx:
[x]
1.1. Users can iaput a URL to be skortesed.
[x]
1.2. The sysiem validates that the LRL is setive and legitimete.
[4]
1.3. system generates stique chericaed U8M.
1.4.
Users can choore castom thot links (subject availsbility).
Accrssing
original
Andlyticse...
3.1.
Users
statixtiex about their choriened Uils.
3.2. View namber clicks.
[x] 3.3. View date/time esch click.
[ 3.4
View gengraphical Ioration of the clicker.
usUrr Accousts
[x] 4.1.
Users can create accounts
[x] 4.2. Accocnt holders cca siew all their shortered URLs.
[s] Accoeni holders cas edit or delete thiir shorcened
[s]
Accoeni holders caa view analytics
their shortened URL..
delete
Admilnistrarors
onslyties.
Exsiratiio
[X] 6.1. Users can set expication date/time for the shortened UR..
Counting the chocked boxes, get total of 15 features implemented.
FEATURES 15
The final line of the response, 'FEATURES FUNCTIONAL is then extracted by regex parser. We note that the use of the feature requirements specified as checklist helps the LLM check which features are fully implemented and which features are not and aids in counting the number of features that have been fulfilled in the generated codebase according to the user-specified input feature requirements.
Furthermore, we also incorporate standard code generation evaluation metrics (Hasemer, 2018). such as the number of lines of code generated LOC and the number of errors # Errors in the codebase determined by code syntactical analyzer (Thénault, 2023). Specifically, we use PyLint, a well-known Python syntactical analyzer, and configure it to only detect and count the number of errors in the code, which becomes static syntax errors that arise from the generated outputs. We configure it with the following options of -disable=all -enable=E'.
Moreover, as the methods are instructed to generate unit tests to check that the generated code is valid, we also quote how many self-generated unit tests pass with the method as Tests Passed. We compute this metric by rumning Python test tool, that of PyTest (Okken, 2022). This is configured to automatically find all relevant python tests (default configuration), and the LLM can call this separately function only code path input.
39

---

Published as a conference paper at ICLR 2024
We also include the metric of code coverage (Miller & Maloney, 1963) Cov %. This measures the percentage of lines of code that are executed during running all the self-generated unit tests for the generated codebase out of the total executable lines of code. A codebase with high test coverage percentage has more of its code executed during running the unit tests, indicating that it has lower chance of containing undetected software bugs compared percentage. We practically test this with a python code coverage tool, that of Coverage (Batchelder Contributors Coverage.py. 2023), and only run this at evaluation time when evaluating the codebase.
codehase with low test coverage
Experimental setup. We complete random seed runs for each task environment for each benchmark method. We then compute each metric alongside its 95% confidence intervals throughout.
G.1 MOTIVATION FOR FEATURE %
To elucidate the motivation of Feature %, we note that the metric "Is it fully functional?" is very sparse and does not reflect the degrees of correctness that implementation case, it is tough to prove that a given codebase contains no bugs, and in practice, most codebases contain bugs. This does not imply that they not functional, although each time bug is correctly fixed, we can agree that the codebase has gotten more functional. Consequently, asking, "To what degree it functional?" is more reasonable. A way quantify this is to ask "Of the functions/features that expect this codebase to fulfill correctly, how many are actually fulfilled?". This is what feature quantifies. As a proxy, and in line with previous literature (Chiang Lee, 2023), we quantify this by having GPT4 assess what features specified by the user were implemented fully as code (see Appendices and for example prompts).
have. As extreme
H ADDITIONAL RESULTS
H.1 HUMANEVAL BENCHMARK RESULTS
We also evaluated Code-1.2MAC the standard HumanEval benchmark (Chen al., 2021) and found that achieves a state-of-the-an score of 90.2% Pass@1, We briefly describe the benchmark here (with complete description given in Appendix E.2), and provide comparative analysis of existing methods here.
Pass@1 of HumanEval (%)
100
90.2
84.9
85.9
86.6
74.4
Pass@1 (%)
63.2
65.8
67.0
44.5
40
29.9
17.1
20
AlphaCode (1.1B)
LLAMA-2
Inflection-2
Grok
Codex CodeT
GPT-4
Gemini Ultra
Claude 3 Opus
MetaGPT
OctorCoder (GPT-4)
Code-L2MAC
Figure 7: HumanEval Benchmark- -Pass rates with a single attempt.
HumanEval Benchmark. As introduced by Chen et al. (2021), consists of 164 tridtwritten programming problems, where each task provides doestring from which the L.LM-based method
40

---

Published as a conference paper at ICLR 2024
needs generate the rest of the python function. Each task generation is then evaluated for functional correctness by testing with held-out unit tests, which they all have to pass for that task to marked complete.
Evaluation Metric. To evaluate the functional correctness of the programs generated the benchmark uses the @ metric presented by Chen et al. (2021), evaluate the functional accuracy of the top-k generated programs.
pass @ass@@ == E
Problems
Baselines. As HumanEval is an established benchmark, we quote the existing baseline results from respective papers, from the current global leaderboard. Such methods include AlphaCode (Li et al., 2022), LLAMA-2 (Touvron et al., 2023), Inflection-2 (Inflection AI, 2023), Grok-1 (xai-org, 2024; Team et al., 2023), Codex + CodeTT (Chen et al., 2022), GPT-4 (OpenAI, 2023), Gemini Ultra (Team et al., 2023), Claude 3 Opus (Anthropic, 2024), MetaGPT (Hong et al., 2024) and OctorCoder (Muennighoff et al., 2023).
We modify the prompt include the function docstring and provide the instruction to only write code to complete the given function for each problem within the benchmark. We also instruct the initial prompt that all tests should be comprehensive, interpretable, and cover edge cases. We also performed this evaluation using the GPT-4 model of 1106-Preview.
Main Result. Figure 7 shows that Code-L2MAC outperforms all baselines listed above the HumanEval benchmark. Crucially, Code-I.2MAC that uses GPT-4 within its multi-agent framework significantly outperforms standalone GPT-4 model on the pass@) metric on the HumanEval benchmark.
H.2 L2MAC FOR WRITING AN ENTIRE BOOK
In the following we show that general-purpose purpose L2MAC works for completing general-purpose long output text- based tasks, such as writing an entire book. Specifically, we posed the task to write complete book based on the following input prompt below of
Write complete recipe book for the following book citle of "Twirks Tastes: A Journey Through Icalian Pasts"
Deseription: "wirds Tastes" invites you on flavorfat expedition accoss Iuly exploring the diverse gasta landscage from the son-drenched bills of Toscany to the bustling streets of Naples. Disoover regional spreialties Teara the stories belind each dish and master the are of pasts miking with cas--ro-tlllew recipes that promisr to delight your censes
Specifically, as we use the same L2MAC framework, as presented in Sections 3 and 4, where we now generate output text instead of code. Specifically, we replace the error checking mechanism {D) from code specific one as in Code-L2MAC to general consistency check, for coherence of the generated output to the existing generated previous outputs. We additionally use the GPT-4 model of 1 1106-Prewe (06-Preview.
The full result of the raw generated book is given here:
page
Twidss Josrney Througa Itelion
|Autho''s Nine]
forwwrd rwirts Issros.1xE
Twirls & Tistes: A Josrecy Throngs Italian Pasts
Foreword
Readet.
twtrling transport you place sistory. very ltsly heritage
pasts just and
thoough
of the North an-kissed shores the South. You will discover the srlistey behind the bumble soodle and the mvriad zrandmothets, chefs, and artisens slike.
thess
will emhart jourtes
been transformed the kands of
I. invite you to savor the tales of pasta 's pat, to revel is the creativity of its presest, and to axticipate the inseaation that will thaps its fucure, This bunk is a tribute to the iraditions that have made Italian pista corneratone of the culimary world and a celebration of the joy it brings to tables aroud the globe.
41

---

Published as a conference paper at ICLR 2024
My own love affair with pasta began is my sonns kitchen, where learned that the secret ingredient to say the love with made hope to share piece taxe with you throuph the recipes stortes costained within these pages,
Prepure to be delightted the diversity of flavors and textares that you, From the zobust zagus of the North the zesty South, your palate your passport the regious Italy.
sexfoed pastos
Boon visggio and basa sppetite!
[Author's 's Nono]
introduction_ wwwiiiioo_iiiiii_ooeeeeooeieeese wiirss_aastes.om
Twirls Tastex: Jesrney Through Haliian Pasta
Introdaction
pastac tapesiry
reltare, culinary aetistry. Frm rolling Tuerany bustling of story tell.
the heactbeat Itelins cuisice. evolved myriad of shapes and
it has
In. this book. we delve isto the rich tepestry of Itelias pasts exploriag its storied past, the diversity of its forms and its revered place Italian cultare. We will jeurney through the regions of Hally discoverins haw scography ind histery hare ahaped the nation beloved dich Along the way, we will racounier traditional recipes passed down through generations, a5 well as iszovative creations that continue to pach the bourdaries of flavor and forn.
The strecture of this book designed to guide you through Itlly 's pasta landscage ore repion it a time. Each chapter focasos or different area, providing Nistorical context celtural rntodures. and a eelection of recipos that highlight the regles's 's zaique costributiens lo paste caisine. Fen the heary butter-labe diskes of the North to the vibrasi, olive oil biseb recipes the South, you will gain comprehensive inderstundiag of Iialy" pasta irdditios.
chapters simply begiz
stage your culinary masinn proceed through the skills create your pasta musterpieses,
introduction
puce, savorisg
section equi vocabalary
glossary help you navigate
Teatian pasta
Prepare to eabark or Elavorful joursey through time and taste. "Teidh & Tastes' is more then cookbook; it's passport to the soul of Ftalion exisine. Boca viagzio buon appetito!
chapteri_twiell.c www.tiettttsttessssssosssss tastes. IXE
Twirls& Tastext Jasracy Through Italian Pasts
Chapter 1: Heut Italy- Tescony
Nestled zolling lisly, Tuseany and gcoves, land isself.
excellence. for breathiskinz landseapes adventure deeply
intertwined
Paxta playe pivntal role Tutcan caisine, embodying the cimplicity and rustis elegance that the region knows for. Ouz cannor think of Tescony without envistoning the wide. flat ribhons of pappardelle. often served with heatty. game- based sauces such as the iconie Pappardelle al Cirghiale a wild bosr rage that is robast in flaver as the Tuscan countrysid..
Recipes:
Papparietll al Cinghtade
Ingredients:
400g poppardelle pasts +tahlesgoons stra-virgin
onion. finely
fiaely chopped
celery stalks. fieely chopped
2 parlic cloves. minced
500x vild boar mest minced
1 cup red wite
4009 conned tomannss. cruched
Salt ind pepper to taste
Grated Parmeson cheese. for serving
Instructions:
Hest the olive oil large pan over mediam hest. Add the onion. carrots. celery. . and parlic, snd saute until softeaed. Incresse heat high. add the wild bost meat, and cook antil browaed.
let
by pepper. put
hours, stirring occaciosally.
Toss pappardelle
with
ragu
sprinkle Parmesan Cheese.
This disk tells story of the Tuscan bunt, tradition. snd the deep connection between the lasd and the table. The robust flavors and hearty texture emblenatic of the resion' cuisine, offering taste of Texcaay 's exlistty herilage.
At you recreate these resiges, imagine the Tiemn lazdscape and the geserations of cooks win have perfected these dishec over zenturics. Each ingredient each step, is a testsmeal to the love and respect for the feed that is at the heart of Turem life.
NNre: lighquiaity imagee the Tascra landscape and the complated disher will be included in the final publication to rahance the reader's is expesience.]
accuracy. clarily. consistency
"Tiits promises conducted that chapter sligas
journey throggs Italian ...
chapter2 rxtris tastes. Ist
Twirl& & Tastes: A Josreey Throgga Italion Pasta
Chapter 1: The Riches of Emilia-Romare
Emilis -ompaa regias celehrated for calinary contribations, is treasura trove Iralian grstmnomy. The ferile plains and the Po River create lush envireement for agriculture giving rixe to variety of ingredients that form the backhese its rich cuixine.
Emilla Romegra, pasta not just it's art. The region biethplace many pasta stapes sizes, each Taglistelle loctellini. lasagne lew examples of worldwide repuistios.
42

---

Published as a conference paper at ICLR 2024
Recipes:
Tagllatelte Rage Rolagsese
lagredizats:
taglictelle
2 tablespoons -ttravirgin ulive oil onion. fincly chepped
carret. fieely chopped
celory unit finely chopped
parlic claves, minced
300g groudd beef
159g groudd pork
vise
cansed lomslocs, crushed
pepper taste
Pammigos--Rggione serving
Isstructions
Heat olive oll
mndiam beat. Ad rook browsed.
and
until softered.
Increase red vise und let reduce by helf.
beef
Pour
Lover the heat, corer snd simmer for least houss stieriag occasionalis, 5. Cook the ingliatelle a large put of salted bailing water until at ceste then drais.
Ald the crosked tomatoes. salt. and
6. Tore the pasts with the rage and cerve with sprinkle of Famigaao-Rggian
This classic dish is restament to the culieary expertise -Ronegte where the slow cooking of the rigs brings out the depth Havors that this region know for.
Ecbuzzone (Spincch
caupped
ParmiglanoReggglan grared
Salt and pepper to taste
2 sheets of puff
Instructions:
Prehest the eves to 180 C (350 F),
Lay bool the spleach. ricntts, Pammiglan-RRgggaa egps calt, peppor.
paff pactry baking theer Spread the spiesch misture over the pasry. leaving small border. Cover with sccoed sheet paff pastry. seal the edges, make misstes,
the top allow stemm excape.
golden
Emilia Emilia-eemmna's
[Noce: lighqqaaltty
the Enilia-oomaga landecape and the completed dishas will incladed in the fital pablication cohance the reader esperience..
Rosizo for accuracy clarity and consistency with the book's nome has bees conducted to easere that this chaptor aligas with the josreey throug Iralian pasta that "Teids & Tastes" promises to deliver.
chapteri_Dxiiis, Liirix taxtes, Ist
Tastes: Jasrscy Througi Pasta
Chapter
Campaia
with drenched coostlines The regine
and freshueis
and of prodice. including famon San Mareamo tomsioes and the revered baffalo muzzrella..
reflected
contribuee
Pasta in Compania is cnlebrated in its many forms, from the dry pasts of Gragnano to the hand-shaped delights like arnochiette. Seafand plays a significani rale in the regional dishes. with recipes that have ham passed doan throngh geserations.
Recipes:
Spaghetti alle Vesgole
Ingredicats:
spaphasti
tahiisspooss extra-virgin
chill. chopped
cleaned
1/2 white winz
A handfal of fresh parsley. chopped Sult lasis
Iastructions:
Hest The olive oll large pan over median best. Ad the garlie and chiti and saste antil frogrant.
Ads the clans and white wine. cover, cook woril the clams opm about 3-7 ninores,
dexto then drain reserxing some the pasta water. spagherrt large par of calted bailing water pasta water needed loosen souce. clams, cddiag
apaghetti the pan patsley serve inuedictely.
Pizza Napaletans
Ingredicats:
500g pizza dough
2002 San Marzano tomatoes. crashed
2002 buffale mozzarella. sliced
Fresh basi! leases
Exira-cirgie olive ail
lastz
setting.
43

---

Published as a conference paper at ICLR 2024
Distrinste mozaarella slices basil leaves top the tomstees.
oll and sprintle poldra
bohhling.
Pizza Napoletasa, with its
fresh toppings. belored Compasian
Iteliox
food culture
woild.
[Soe:: Eirhqqaaltty images the Campatia lazdscape und the completed dishes will inclzded is the finsl pablication rabance the reader is esperizace.]
Rosiew for accuracy clarity and conerstency with the book 's tome has boea conducted to ensere that this chaptor aligns with the jozrney throug1 Italian gaste that "Twils & Tostes promises to deliver.
chapterd_ wwwris_ttttsssss Ixt
Josrney Throng?
Chapter islaad Flovers Steily
Siclty
cullnary landscage diverse and vibrant history Iafleences from Greek, Arth and Nerman conquerers have seven rich tapesisy of flavers that define Sicilias cuisist. The isleed i fertile soil and boustifel scus provide abundeace feesh ingrediznts that inspire the local dishes.
ishand
Mediterraness,
Pasta in Sicily is oftes pained with the hold fiavors of the sea and the exeetnees of the island 's produce Dicher like hasa sils Neem and Spughetts at Ricet di Mare showeaee the crcativity and resourcefuiness of Stciiian cooking.
Recipes:
Pasta
into ouhes
parlic claxes,
400g consed lomeloes, crusked
A handral of fresh basil. torn
Salt and pepper to taste
Ricotta salats, grated, for serving
Inctrnction:
Heat olive
large pan over medium hest. Add the egoplant caber and fry until golden bown, Romove and eet aside. ome pan. add the garlic and sauts fragrazi- pepper. until dente then
tomatoes. basil,
minates.
gasts large pot
belling
Pasta alla Norna is celchration Sietly's agriculteral with the aggplant and rizatia olis creating harmotion blend flavors.
Caponato
Ingredicats:
lorge eagplent, cat isto cubes
celery stalke, chopped
onien choppe
tablespoons cupers.
green
pitted chopped
tahlespoons pine
conced
crushed
Sutra-virgis olive oil
Salt and
taste
Instrnctions
Pry the eggplant cabos in olive all eatil golden hrows Reems and sat sside.
2. la the some pes. saate the celery and onise entil softened.
3. Ad the orpers. olives. pine nuts, sugar, ard vieega and cook for few misstes.
crusted tomsincs and fried egsplant, semes salt and pepper, and simmer for minutes.
Caponata quimismeatial Siciliua dish. sweet and vegetable medley that om served an appetizer side.
amhodying
comples calisary heritzge.
Iigh-quality
Sicilian landscape expericace.]
completed
pablicstion
Review for accuracy. clarity and consistency with the book's Geme has been conducted easere that this chapter aligas with the journey thrugg Italian that TTiils & Trates promises deliver.
chapters_ wwirix_tastee ist
Twirts Tastes: Josreey Throng) Italian Pasts
Chapter 5: Pasta Making
making
lisly.
generations. accompanying imagts, you perfection..
craft
Typex of Partn Bough:
1. Basic Egs Pasta Dough Tagredieats: 400x '00" flour
4 lurge est Insirnctions
flour clean surface create well the center.
Crack egps isto the and gradually incorporate the flour fron incide rim the well.
deugh about minutes smooth clastic.
Woap plastic
temperatere.
Semolina
ingredicats:
44

---

Published as a conference paper at ICLR 2024
semolina flour
Kneed semeliia flour and
clean serface ustil temperature.
about
plastic let
Shaping Techniques:
Rolling and Corting
Kall nut the rasted dougs using rolling pin or pasta machire to the dexired thicknest.
Cut into strips for togliatelle. or squaces for cavioli.
Band-Shaping
For shapes like cracchiette, , sinch
sieces dough shape your fingers.
For cavatelli. roll small pieces doesh against the surface gaecchi board create ridges.
Cooking
Boiling of salted holling
Use
cooking
Cook usiil al dente sssall 2-4 miszies for fresh
Drain toss with favorite seec.
2. Storing Pacta
Fresh pasta zan be dried ea a rack and stored in an sitlight contaiser.
can also be frozes for up to north
[Noce: Hith quality imsges the pasts making process end the varions shapes will incladed pablication guide reader.]
chapter
practical
pasia lilly. Enjoy steps, you'll making pasta
family
Rosicu for areuracy clarity and concistency with the book Dome hasa conducted ensers that this chapter allgas with journey through Italian that "Twils & Tuates promises to delives.
slassay_-vitbb_teee Jastess. 1st
Twirls Tastest Jasrsey Through Italian Pasts
Giaxary of Italian Pasta Tome
glossary designed help you ingredients,
terminology throughout
incledes deffnitions pasta shapes.
taiil fim Belognase:: mral-based sauce originating from Balagas, typically malt with mixture heaf
dente:
ideal
Ciaghisle: Wild boar, oftee
hearly Toran resembles bes lies or busterfly wings.
Farfalle: "Betterflies
Italian, this Fesilli: cortscrew-ai -haged pasta that is excelleat for holding otto sauces
shope
Grocral: Small damplings made from patsioes, flour, and somatimes ricatta cherse.
Graea Padaso: hadd, slow ripcaed, seni-fai fai chesse from Itlly similar to Farmigiano Regsiono.
Leargne: Wide, flat posta sheets tied in layetiaz dishes with souce sad cheese.
Oreschieste: Listlz ca Italisn,, this pasts dape
smil, cirllike round
Pappardetle: Broad,
pasts noodizs, often served granclar cherse lamn Parmossn, aeed grated over pasts dishes. Tube-chaped pasts angled shoep's
rich meaty sauces
Parmirisso-Raaee Rezgino:
pasaa
is considered may Sparhetti: Long, cylindrizal. of Italian
Ricotta:
Gillings
the aiking
simila to fettuccine. traditionally served with Belognese Torrelliel: Ring-shaped sufted pasts rppically filled ein neat, chrese, or spincch.
Tagliatelle: Lous, risbo-llike
This glocarr will seciet you in batter underetanding the recipes and the culteral cigrificance behind each dish. As you explore the diverse world of Italiae peste, cefer to these terms to cahance your colinary kiowledge end appreciation.
tudes_tvints_tastee txl
Josrncy Thronga
Recipes:
Papzardelle al Ciazhiale. Chapier
Teglistelle al Raga Belogsese. Chapter
Ecbezzone (Spinchh and Cheese Pie). Chapter
Spaghetti alle Vangals, Chapize
Pizza Napeletans. Chapter 3
Posta alla Nomo, Chapter
Caporata, Chopter
Bisic Egg Daugh, Chapter
Semoline Douph, Chapter
Bologsose... Gloscrry
Bafrate mazxarella Chapter
Ciazliale. Glossery
Clam. Chapter
Epsplint. Chapter
Eage. Chapter
Eatra vitzie olive oil, Chapters 1. 3. 4.
Partalle Glassary
Prsilli. Glesrary
Garlic. Chaplers I,
Glessary
Chaster
Glessary
45

---

Published as a conference paper at ICLR 2024
Mozzarelia. Chapier
Glossary
Pappordelle
Pronigieso Regsiate. Chapter Glossory
Pecorino Glossery
Peane Giowary
Rar. Glossrry
Red wine. Chaptors 1.
Ricotra Chapter 2, Glovarr
Ricorta calata Chaptes 4
Soa Mazzase loanlees, Chapter 3, Glossery
Semoline floar. Chapte:
Spaphetti. Chapter 3. Glanary
Taglistelie Chapter Glossry
Tomstoes, Chapters
Glassar
wild
Chaptei
Torms:
Al desie Glossury
Bologsese. Glossary
Cinghiale Glascary
Earialle. Glossrr
Fasilh. Glessrry
Gzocebi. Glessary
Graa Padano Glossary
Lesagee Glessary
Orecchielle. Glossary
Pappardelle Glossery
Glossess
Ragle00/9 Glesery
Ricntra Glestary
Sus Maczato loneloes, Glossass
Spaghetti Glossery
Tegliatelle. Glossary
Tortellini. Glossary
Note: Page numbers, will added oncn fleal layost the book complete
For this implementation of L2MAC we modified the system prompt to be
fellowing
with.
"wrire_filee
text filex.
You mast act estonomossly edd you will receive ne bannn inpat stage. Yos have reture output the complete text for completing this task, and correctly incorporate it into the existing generated text files.
You alwys write eat ite whole rile contents.
Please always view the files before wriling to them, to mike sure you are eriting to the correct files.
Provide text to achieve the task conditioned on the ecisting generated ineelading changing the existing generated lest where necessary.
ayy grophical outpat
you steps, stop step,, you optionally spacity
separate
you
volidation check
data disk persist
wiite
Use the functions proxided. What calling fanctions provide RIC229 compliani ISON request following this format withost deviation
And the initial prompt to be
You will get iastroctions for a book to write. structurd outline of book. and chapters with detailed descriptions what each chapter contain. quick conment the purpose of each chapter. aot comment on what every pizco of test doss.
lay
Feel
text filly complete. No placeholders.
chapters chronslagical
Please
shorld
Fallow best practices for writing boot, and
Mike sure that files corsectly conditsened the subequent chapiers and oxtline(s), The text skoeld be fully complete. Male sure that seat differeat files ure conpatisle with each other.
Whe writing text if you are casure, write the most plaucible test.
Useiul to lnsw:
helgfel to write detailed nuttine of book first, and Dee write the chapters order.
Alveys comment brieily deseribinz purpose Alveys follow best peactices for soqueated structure to package the completed
Osjective: coe [ese__ddeerp
Underctaed the problen. creating extremsly drisiled crp--y-sstep plan wher toil inclades every siagle feature requirement specified above, feel feee cops direcily foom it. Use mase tha 10 vies applicable to help adle excalleat colerent book, where oll the text is fully complete. Use best book design practices, and you con catpet large ameants of test osce. Please includ: a lus sentence to perform checks who inslementigs or writiae text is that ane step. You will receive so huma isput at sa9 stags so you caasol ue bomn to perform any checks. Only create detailed plas to bezin with, which inciudes performing consisrency checks. Please ore to isclude all the spacified feature requitamenis plan.
long (mulriptz conrencee) and in
steps the plan. Peiforn additionl. checks ud eveluction euch
following
46

---

Published as a conference paper at ICLR 2024
We further modified the control unit prompts replace instructions of generating and checking code to generating and checking text.
H.3 ADDITIONAL MAIN RESULTS
In the following, we provide more detailed main results table, including additional test metrics to show the number of failed and tests that passed, tabulated in Table 3. We also provide a codebase output example from each of the methods and observe that they illustrate the difference in quality between the respective approaches. Notably all methods were given the exact same user-specified feature requirements for the given long code generation task, Appendix
Table 3: Codebase generation system design task results showing the percentage of functional features specified that are fully implemented (Features %), the number of syntactical errors in the generated code (# Errors), the number of lines of code (LOC), number failing tests (Tests Failed), and number of passing tests (Tests Passed). of user-specified feature requirements across all tasks by generating fully functional code that has minimal syntactical errors and a high number of passing self- seff-generated unit tests. The results averaged over 10 random seeds, with + indicating 95% confidence intervals.
fully implements the highest percentage
Outer
Media
Reattase
Tat-Raled Raled
Tass Paned
Tecl Fale
Ristare
#Braas
Teaa Falle
Pasad
136449
3534th
BACEOW
14813%3
360
2334915
231/09
3183334
(336:1091
8:6
3M-476
658
16:671
124-146
061924
#441369
6:1
185:5111
L2MAC URL Shortener App example output
app. config.py
requirements.c
services/
admin_dashboard.pp
init init__py py
url_shortener.py
user_acccunt..99
tests/
cest_admin__
test_app.py
test fest__ur___
www_uur__hhorteeee -y
test_ test_user_ccount.
# opp
from flack impart Flack rediract or1 for request from services.uil. shl_shorteset import UrISheriener from services, use,, user_ Jecoust import UserAcceuut from services admis, dnshbard import Adminliadbunnl
app Flask( none UriShartanar))
daskboord AdminDesbbordd))
contec"!")
del komr():
return "Helle.,Wodde Wodd!"
Stpp routed Vicsbort, arls')
del redirect, to, arl ar..arsshe_uuurr url
ocl uil shorteser originig ond (shoct.dd))
If is not Noro:
url_iaoreeeer. record_click recors.ciickssson..ttt 'mocked_lecation
return redirectç
I'analytiss/echont ocls") def anslytics short. arl): analetics if anclytics asi Nore: analyticx(shor
return analytics
alee
return Anslytiss found 404
Sapp rostel Vatcoust/ createe methodcal 'RIST"
def create accoust()
47

---

Published as a conference paper at ICLR 2024
ascenamic reqsest 2emm esernsmec
(count(ssemane)
def "uieousi/uris'" metlods
"ssecoente))
return uxer_accuut.
(username)
sapp roster ("areomatteddi url mechodsal "PTT
daf edir arl
asername requeet form geet "ssernmme"
old_arl seques). feom. gell old_arl")
sew_ uil request ferm. gell Lew arl")
return uscr_account edil_uri [msorname, old_arl, new_erl)
Supp contei "accovat/delete_cu methods of DELETE
requeet pet("ael"
Sup contei Taccousi/maalll mnalytics mothods of ()T
def view sew_onnlytics (32
aornnae request args "ssernmme""
return user_account View, www.y/analliieeeeeeee analyii<(ssemman))
Sup ronted C Vadmic/ prts'. methedsa "GET" def all sris return dniibrd wic.oaarddiioellll..le..oo erl_
Sup conted "Tadiiel delete_url', methodse] "DELET
rostei "Iadmin/ delete, bear". mathadset TIERR def delzie_ ()e
Eser request, forai gett"" gett" "sser": return admis dasiiboard, delete (user)
Supp. coster Tadiii moniter" aethods- GET
def monitar s)stem return dashboard, mosttor mostior_cysiem
nene main._.
app dreeppeee
class Configt objecti:
DEBUG
False
TESTING False
STEET KEY urandom
P!
(sel():
self.
def aiw__allarriaceeell
return salr urls
def delzre, arl self orll:
if arl self. odls
del selr. arts [ud]]
return
delete_ eser serr):
monitor
retorn users lentself users). 'url''s lelf arla))
servicts/ servicos_.eiie (ei
# services/arl att.e///aaaaaarterrcccc
import string
import madon
from daratime import datrtime
sclf.
diet
def validate, validate_ecdlssee srlfs udl.:
It ail, startswith: bbig ( or uit, startswith ("igssss"""" :f"):
return Tree
return False
genarate shan arl self. srl. expiration dateaNore. coston chart trlaNone): custon ston uri custom, stor
dizi:
return Thrarr,This,,ae thon JRI, is,,,is,, no, available"
short_orl
coston_shert,
""joinciosdon choicet sttciig ascii_letters string digits) range
dixits) raaxe(5))
salf.uri_diet dei_diit[shoct... l'ad'' 'espiration espisation_datee
onalyties, aaalyties,iiciishontt.. ("clicks's "click_detoils
48

---

Published as a conference paper at ICLR 2024
return abort_url
original self, "'xpiration_ date" datetiune.nee ucl_data| deepirattio__dan
and datad
"Eror: data None
return daia
def recerd zlick shelf short url location):
if short art is calf analyties diet:
celf anslytics dizd short, oelll "clicke") +
self. onalyties fict[short_ddl Slick_detaila "I. appead]] 'iieestomp' datetinee now(). isoformat "location" location)
att. aaors_er)); return self. aaalytics__ii weccc--ttsshhrrtt.oooeee..r Note)
Jeit (selr):
self
def cecutteccconntsllll (seraame): if oserzams is telf nerre
reiars UUammae attredyy_ssiits self. peernjuuemane
uria :11
retorn "Accoutt created successully.
(sclf sicmane): asers:
usersam: "Usemsne,ddrs" doc___ot___siiii
esell If norram "thaman, dozs
It old not self asers [sserame [I" farts' retura does,
exist.
selr users oseemane II" urls renove cold anl) szif osers usemanel II urls uppend sew_asl) return
delete, [slff esarname srl.:
userzamz nut acers:
"Useraan does exist
(sseraame 1I" urts"]
self. "ees pxtst
urls Lennvetur))
"URL,deletedd
def aic__analyiieaaaa uxersamz):
It osersame not in self. asers:
retura "Usomame does extss
return oorl" urll "asstycios"] for ort in snfd ssers|morrame [Jrrte"] if "analyties" ts arl).
tests/test, adviis aawiiaainnnnsbbbboeeeeeeeeee Jushbardd.ps
from services admir, dashbaard import Adminiiadhooal
vien_ail_urlx()
def deleiseees__arii arl:):
admin (bminDabbozzd)
admin urls ("rest srl": "test")
admin delete arl; testt hest.rr "r!")
ancert admin, urle ()
test, delete, sser():
AdminDaskbuzrdd
admia users Dest_ "test")
def nositor
admin AdminDedbbozdd
admia, users teett_seer "tesc")
admia, uils I'test_ scl" "test")
sncert admin, monitur systeme] an ( ("mzrret d. Parlalo 1)
rests/test opp-19 sy Import pytest
Import
tpp import upp flask_opp
Gryyest.
Oryytat fietnez def client app):
retorn elee
def tol_konn((eieett)
respoace client "er(")")
axcert responce. datus. rode
redirect
49

---

Published as a conference paper at ICLR 2024
analgtiostellent
resporer pww""aanlyyiissiimi art')
def test_ercatte acceatc_cccouiiCc..... (ccensl(clien))
responie client.pee pestt pestt"//cceeeee /accca//a//cc/aaaa create", datam) "ciemame", "text_eaer"
asser respoase. states code sa 700
def test vien, (eleeclie)): respoase client jac(''acccat//arl query query_strin string] "sseramee" "rest_user"
saeer response codi
arlceliznt]:
responce
datant "test_aser" "wld_arr""
resposse.
def deal_delets, arlt cllentt:
responic client. deleter /ecooe///lleteeuur datem] "usernanee" ! "texi_oser"" "orl" "er"" aacer respoase. siates rode =s 200
def test view pelytios(eleent) respoase a ollent, gerl" (facccnnt/arrallioos query stringal "ssernmme": "rest user")]
sssett respoase. code
response
def test_edmia_ delete, orll client):
responic delete Essert respoase. states code
Jelete d'eleii/delete.uu "ard" "arE"))
200
def lexr delate searcellerr:: respoace ellent Seleie("/atmin delnie//eleee_seeee seer', darast loxer's "nser" axeer responce. datert rode
def
testr/tes__vr prrruuur_raatterrr.ppp
Import patest
from services, orl shorteser import UriShortener
from daratime import datetime. rinedalia
def text erl th: arl thoricerr UriSborceerr)
hhtp exsmlee
geserate (rl()r))
get original
oct_ositina__ull "'ivalld'))
short tR
cestom short__oor_rn "suton"
sasett url sbboiieer geacrate_sbortt url castom_abert prrlcuston_short eustoms_skort_ut
asser nrl chortener gexerate sbort ort(url, castom short urlscnston shor orl) "Rrror", Thiss short, available"
Test anvistics
ssse orl shorteaer get, analyties url3 l'elicks"; 0, "olick, details's "s DI
url thoricer recurd
"mocked bil_iberieerr... sel_amalytii
esent cazistica clicks'' II
"imesaap" analytice|
Test espiratios
urp ui srl_shorieserr generate jeseratt_sbeer_uee urlri_ daletime aow() (ox(( timedelta(dassst) Gseit stl_shociearr.c sei.eriginll..lllsspiee..
'Ercor: This, Thi,,UR,, orppiiee
test//enn are.eaaareccccoooo accoust
fomm services mer acouusi impart UserAcceant
def text create Recoust)::
accous: UserAscount()
secoaccoun crecte_ccounnt "tees_use:"" (esl_
"Accouat_ inseeddsuscssslll..... already
def arle:)
ExrrAcomnit)
accouac crecte, eccounti "est_user""
Essert user, accouat, view, arls( ("test_eser"
essert nser account, view, arls con, existent_usee user")
def Texl edit arl()
seer account ExerAccmnnt()
sser, sser_ account creois_ text_uxer") u)
users! "see__uee"" "arls" cppend 'ait test.com")
account test, user".
'hrip succeddsdd_ucceesseeeee con'.
sul_dnei_sol__ciie
useraacousi. ['so__ssistet__ee
50

---

Published as a conference paper at ICLR 2024
"aris" ocl: test "hrpp ifl com') "RR_deleted_ssceesffii
"lagged "lagpedd tertesteeo))
saxert user_cccoutt deleitt deleie_url ("tess_eerr" hitp of conestetent ieteenttine cun dess_
useraccount
delette delete_url aat.ic/_uiissnceiitteeni.. aon_esistent_uuer
aupp test.com") "Usemane, cxiiet
def text. test_view view, aralytice
Eser_ accoun: UserAcconate)
Eser account create aesoent ("test user")
sser, accr_account usrrs text_user II "aria" appende] 'hepp cem".com" ][]]
"ssalyticx's l'elicks's 10, "locatione's ('SA)
case uscraccouni ( "est_uues") ('aupp test.con" ['clicks", "locations"; U'8W', "UXII (
AutoGPT URL Shortener App example output
ana analytics.p py
main.py url url_shortener.py
enstyrres. Py
class Analytics:
def treck clicksssssssssff., abbr
pass
tellcccicll,, srlr)_erl): pass
shortener URLShoreeer
main.
stl_sbertener URLShorieeer)
# Tert the isplenented fanctionaitty
# sel wel_sdertener.pp
Import madon
Import string
anslyties import Aaslyties
Analyriex() self
def validate, (rett, arl): Add URL salidation logic pass
def generate koneste__hontssed_uue sholsssed
Generatt randon c-chraaterr striez
chori "".jolemmddon chmeezsssttingg ascii_latters lntters string digits. ke5) return thori_url
orliself. arl3:
artrucl):
chorteeed_ddd
and
muppiaz
siee
ValueErrar Invalid_LMii))
def rediract to original critalt, Shor erl:
Retziasa the originat ard foa de shor vel
Track clicks and zeolocation
self caalyties track, (licks relf anslyties irck www.lnaaiinssserr.....
pass
accosnr[self.
paseword)
def shortened arls(setr):
def sdit solf. short arl. new orl: pas
def delere_orI(sede or](s short_ady: pass
ticv_usslyticexsellf sbort_ozl):: pass
tel_espiirtioon_co (ott
espiration_ddit)::
class Adein:
def view uel__lll pass
def deicte.anl_or_veerreel identifier): pass
nesitar performancetsatry:
51

---

Published as a conference paper at ICLR 2024
pass
AutoGPT URL Shortener App example output
app.
rest_app.py py
app F7
jsenify. redicect
dataclasses Botcelass
datetime dotetine
pyiz
import ueid
ap Flask name
y Mork dutalone
18
clicks:
datetime
expires_al: datetime
Stpp route ( "Jsharren.. mechodesl (IST")
def <horten, ():
cata requeat.geiljenn (so_)
origiaal urt data per2"art")
Eser data get (ateer""")
expizes_at
gell eapires__at")
sbort_url oiizinol-r-iiiinnl espires_at))
(iid4())1:8)
shoctenddssbort.ul usermaser, clicks-0, sceated_ot-datttine utc). expices_
short_url)),
rostet tiesbors atz", nethodes"((R"" def redirect, aill (clorshot__ulll port)
url DE jei(Short__n))
If arl and arl. expices_at datetime pytep.y...tt url clicks en
return redirectiari. orizinal, code=302)
raturn joonify (C "erro'' foo_d assppie 3),
nene "_iiaaa___
app dreeppe)
1ept__pp--P
from flack
@gytest risture
def cllant ()
app app. config TESTINGG I Twu
with app aps, elient) aa clieat: yield cheeh
resparer 3UT23:599 content_tyse application orl" zesposse.el.joor))
datanjeen,
'Kripe google
Trapines
def rest redirect orl coliintt))::
respoase client pest d'strttt darsnjeon dampet( wel": 'hripe c/owe, google mmn "meerts "reat" "eapires, arth HI, content tygez seplicattin//ss /joon" shortodl response. () (etteeeeeeee""aa scottshoct_u
2002-12-31
response client 2/ short arl"") sssett response. ditta_ code
foand ellent):
respance gert
"error"
H.4 CODE L2MAC IN ACTION.
We provide example transcripts of L2MAC in action below. For the environment task of the Online Chat App.
Initial instructions planning.
[Syste]]
Usjective: Write code for a large system design task.
Please note that the code should be fally fanctional, . No placeholders. Oxty use the Firctions, you have heen provided with.
Oxly "wrire fites ourpot coda.
52

---

Published as a conference paper at ICLR 2024
complating whole corractly incorporate inpat any stage, reture ontpet complete
astonomoesly you
You always files before
with labs.
Please
the files.
Wher
the filenome
Provide the mininal code accessary achieve the task concitioned on the existing generated code cnde-iincluding changiss the existing generated codo.
You casnot viesalixe my graphical ourpat. You zsint withis a Actor Mucel machinz, and when you lis out steps, each etep will he taken by e aew separate sub ChaiPT model. When you list out sib tosk steps. you con optionally specify the sub task
volidation to check that it las been completed sucessfolly.
You casnol any datahases note are setup in the local envirosment. insteed mack a database with in in meniyy dictiorary to saved to disk will persist beiwera sleps or write operations.
store data.
change
they
incosrect, mnke
pass.
Use the funetions proxided. when calling fenctions proside RFCK25 complaant ISON request following this format withort deviation.
[CI]]
You will get instructiene for cndz to write.
First lay cat the naner of the core clasees fenctions , mothods that will be necessary. At well as quick comment on their perpose. De not comment on what every file does, Please aete that the code should be folly Tuactional. No placeholders.
You start the "catrypoint"
thea so the osss imperted by file, so en.
Please note that the shorld fally fanctioazl. placeholdzrs..
laguage famework appropriate
imports types
fully fanctional,
differest
Whe cole
write plancible implementation. Inclade modele deperdincy package depeadency definition file.
Useful to lnow:
For Pytoon. you always create an appropriste reauirements, txt file. Alveys add a comment briefly describing the purpose of the luactios definition. AMl con memnsass explaining very complex hits of logic.
Always follow
brdt pracilees for the repurated lanuages for folder/fite srrectrr and how to package the project.
You can my package and my oiber packagos yeu
intall.
You ceanot any datoboses note setup in loaal enviroiment.
databbase with on in meney dictiorary 10 persis beteee steps operations.
tooibalt
dataciassex
flask
Gellbbll Chat Service (003)..
A real-lime online chat application slluxing uxers text menaaos, images, and create group chars.
Regsiremests imploment
Registration Anthemticaties:
I.I. Sign
email.
recovery.
User Profile:
-I] 2.1. -11. Allow seers to sot profile piatures and statns messages.
-I 1 2.3. Pricacy sattings for wio the see eeer datails, Taxt sora starus.
Contact Management:
3.1. Block/unblock contects.
3.1. Creste, edit, and mange groups.
Messszing:
massaees.
Mecsage receipte
eqsivalznr).
encryption uscurity.
4.4. GIPs. and stickers ougeet.
Group Chais:
11 5.1. Create group chais with ause and picture.
-1] 1 5.2. Ade or remose participazis.
-I] 5.3. Admn roles and permissions.
Starus/Strry Featore:
6.1. Allow seers to pest image siateare visible for limited tine.
6.2. Costrol sho can see the status.
Application:
secessible from
Consectivity and Oeftlar Mate:
S.1. Mescage
for wher thr aser is offline; messages aexr connecticty restered. 8.2. Display oxlise] offline status.
Understand the prblln, by creating in extrenzly drisiled stes ses-by-step plan, where eich step is long (multipiz sentences) and in total inclades every sisgle (eature rectirement specified above. feel free to cops direcily from it. Use to more this 16 steps in the plan Creste additional texrs, checks and evaluatio each step when applicable tn help make accallent code implementatior. where
the cade fully fenctional. of code oace. Please incinde last sesience te create and nan laste loma test. Only create
best catrumrs design praciices. and you aa nuppst large smounts
implementnng or writigg cods that sam stop
input on stoge. you csstot includes designing raceing tests cleck that they pass. Please sure include all specified plan.
ceceive
plas begin
RESPONSE MESSAGE)
53

---

Published as a conference paper at ICLR 2024
[Function p det sttp:/ikkkkopp.o..sb..caco {
stops
"Stap
database usia creating application (app. dictienary nessige. database setting Flask apslication. information sec email. passeed. profile pictuse.
satingg
stracters application
dictienary should be able
settings, blocked codueets
Write store and retriess data carrectly''.
eastre
application setip sorcest
SStop 3: Inplemest sser registration and authestication. This incledes creating roures for ner siga up and forgatten passoard recovary. The sign up raste sooald sacapt usr smail and passorrs and store this Information in the mak ditabare The tergeties paswoard recuvery roste shordd accept ser email and send a pacexend rexet lint (for the purgost of this tsk just display the liek), Wiite tests to casure that the registcatios and anthentiaation costes wak is expected.".
Step 3: Inplemest sser profile masgenent This inclides cresting restes for selling profile pictures and sisies messzee, rnd updating privasy setlings. The profile picture and ststus messuge resies shezld accest the respective data spdsic the user's 's information is and dalabase. The privccy sellings route thoud secept the ase sellings tad spásie the user's 's Information
database.
to
the profila waggene romes
expacted
SSep 4. Inglenest contect
This includes creating and update the pictere participants) ard update the user's groups the database. Write tests emare thtt the costact mingenest routes work expected ",
for unbbocig centacts end matoging g00us.. blocked
Mock/umblock
sheeld the the detzils soold
detabase.
The group
Stop a: Inglement mensagng. This iscludes creating rasize for crading snd racalving messages, updating read receipts
enerypling messages and sharing image,. The ses/recciie rozize shorld setepi the meseage data and updaiz the respeetive oser i messages in the mork database. The read receipt route shoule secept the meuage in and update the massage' is read status in the mock datobase. The exeryption roste shorld aceept the nessage data. encrypt it, and store it in the nock datzhase. The image sharing roste shoild sccept the image dats and opdrte thr respective sser' 's nossages in the moct datebase. Weite tests to casure that the messaging costes week expected.",
Step 6: Inplemest 2ooup chets. This includes creating rostes for creatiag grong chats. uddiag/removiag participants and sceept group detsils (tme, picture. accept
managing
permissions The group roue sed/remove participanis participante) speate kreap participant should
group greup's the group admin's smail, and pernissooss and update the group" admin parmissions the met druabese Write
parmissions
that the grop chat moies wore expected
Step 3: Inglemest the status/story featere, This includes creatiag roxtes for posting imnge stateses end mesoging who con see the states. The post status roste shorld sccept the image data add update the user 's statuses in the mick database. The minge vixibility route shoul secept the ataras ID mnd visibility selliags aad updale the states 's sisibility in the mock databuse. Writs tests to casuce that the states /story rostes woek as expected.".
'Step 3: Inplemest the wit application. This includex creating a bade IMM/SS// frosiend that interacte with the Plask backend The fremted thould have papes for sign up. cign in. profile mataemen, contact mangementt menaging. group chats. and states story. Weite tests to ensure that the frentend con successfally sead requesis to the backend and displey the returned date.".
Step 9: Implemest consrctivity
offline mode, includes creating
for apdating
online ouline/offliae shoeld mestages
for smail and new
offlise. The offlin/ status should
spdate the stsius is the meck is offline. and send then when the database. The ontine. weiz lexte cesure that the
queue whes cosnectivity rnd offline mode festures work expected.".
"Stop 10: Finally. parform a comprehencive test of the entire application. This includes resting all rostes, the frontaed and the nock database. Fix any bugs that are louad, and casure that all tests pass, Alse enssre tast the application meets all the specified fecturs requirencats."
Then the first instruction is loaded and is executed, for example,
[8ysten]
Objective: Write code for a large system dexign task.
Please nots that codz abould fully fanctiunal. No placcholdera. Only use farctions you been provided with.
only use "write files ourput code.
You completing this
extonomonsly sad you will correctly incorporate
inpat stage. You reture ouiput the complete colebuse.
You always grite the whole file contents. alwaye indent code with labe Please always view the files before uriting is them, mike
writing the correct filex.
Whe writing test make the Iilename start with the profis "cest." ",
Provide the minimal code ascesory to schiave the tack conditianed on the extating gesersted code -includinnng changing the existing aenerated code.
You careot vissalize my graphical output. You csist within a Actor Musel michins, and whm you lidt steps, each step will be taken by a nww separaic wubCChitPt model. When you list out st-itat stepe, you can optionally specify the sub-lack validation check that it has been completed successfully.
You cassot datahaszs
datahass with in menery dictiorary io lorst steps wsite operations. data. data to disk persist
samup
test is failing the estor could be code. the incorrect, feel free oxeresite and Change the tests when they incorrect, make all texts pas.
The the functions provided Wha catling fancrions only provide a RICWss complian and ISON request following this format withort deviation.
[CU]
Objectise:
Execute sub task step: Step I: Start by setting up the basie structore of the application. This includes creating the maia application file (159 syl snd settinz up Fiaak asplication. Also, creste a mos dalabase usiss is in is-memery dictsosary to store user data. This distionsrs should abis to store user information uec ta emsil. passwadd prefile picture. states message, privacy settings. blocked contazts, datshase
groups. Write tests to essure that the applicaties sctup is corsect ressieve cormeclly...
Note: Condition new code files
existing files: HI. Fully impletent these featuces in code, placeholders. optionally view the existing files need to view costext uindow
You have linited colective about which files you vira, only the files you think you night and vins
Suntury output of previoss step:
Respond now only with a function call of one of the following functions provided: "ssh_task ctep_complesss ', "view files ran_asiaon_filt file" fiilest_files" ', files_ ", deleieefitess ", and if you want to output code only ase the wille_file fonotion to ourptt code.
54

---

Published as a conference paper at ICLR 2024
[MMM] [Punction Call) [irese contents":
"Eile_path" ". "app.py".
"Cile_coatent"" "hon flask Elask. request Fleskc__.mme Hoiet Mock datchase bousers_ Walteeap route
toltusers dataa""" email 'II datetalttettrn ('mesage" "Uher created "), 2oitslnvarp reste! /usere/<smai > methods "GE" "Gr"" laded (e..ueeeeaaaiiie ciale sscrs_db. (atenaillaaa not eser trtttr menuugee" : "User not found ", creter
nethods- "aosssl"PPPpoddd Dinde
user(.letdda request,
200"
retle parh " "rp BY
costens : "import pytest eimpott app apllaaepptess fixtureeohndd clieat O daaippo app up-coafial ('TESTING') - Troelatrwath
app tpp. test cleate is client Sestityyeld dienttateinde oreetteee__aaa and pet_ vierfelize) vaatessr data (talesi'
email : tist etamplc cm -aatil passorr "paoward' Jlasii _piecrs_
"picture 135 "Anicsi statos message "public "Jnissi" cocdedd Il.tneehi zroups ": [] postt Tissers pon-user, date Moliaasert ressonse statos code 20heltaseet Exposse.sctlieee gart "fuar Prest con't Ialtpasers response atatux_oode 20000
Hello Julett' privacy_
client.
m'mmssage" UVer
Brtaaimngoe response. get, get_ () sser_dais""
[CU] [Fazetion] (' sittee "success", "messaze" " "All tests "assed.")
[CU]]
Hat the set takk dtp bees completed of:
Step 1: Start by setting up the basic stroctare of the application. This incledes creating the mui application file topp 17) and
cetting up a Flask application. Also create a mock database using a ln-menory dictiearry to store nser data. This dictionary should
dble to stare user information such a email, pasiordd profils piciare. states message, privacy settings. blocked contacts, groups. Write lests io easure that the application selup is correct and the nock dutebase cos store and retriere data correctly.
complete tusk. codo. the "wrire filex firction "I fully implement these features in the code, placeholsere If you then. please view den, to mike sure Respoad now only with fonction call of ore of the following fanctions provided; "sub,tukk sttpssttt.ccoplee ", "viee_files" ren penion file", ', "pytost_ files ', "write_ files" ', "delete_ files ', and if you wast to ourput code only use the "write_ riles" fenction to ouiput code.
you, call fanction sub_tasksig wts___iepccomplee
the
fencrions you have bost provided with and you want to
Condition
asieting zodes ('app
"test
hare sei vieved the files before emilaa
writing the correct files.
[IM] [Panction, Call] Ipstest_ filex] "Glec__e_teet" ("teu__ppppp
[Fasetion] ("oxipst":
iaplatforn Python pytsst pluses 0.a0oooo::: Imphhanniii haslusiis haslugias: kydra -13.2, anyio 3.7.Dlneoleecced [100R]aa'a
tstatest__spppp
paxced 0.04x feel free overurite change the texts wam they incorrect. mike all texis paax.")
If text is failing the srrer could be the cnde. the text incorrect.
[CU] Has the sbb task stop been completed of:
Stag I- Start by cotting the basie strictare of the application. This incledos creating the main application file tapp 1y3 and
selling up a Flask upplication. Alse, create i mock database usiaz a in nemarr dictionary to store user date. This dictiourry shezld be able to store user information such is email, pessord, profile pietare. siates message, grivacy ceitinge. biocked contactc. and groupe. Write trete to enture that the application selup is currect and the mork database can scare and relrieve data correctly.
yeu. fenctions you
step complate prosided with. and Condition esistias code: d'app-py" "pest_opg Fully implement
"iilte fasction Only placeholders. you writing the files.
complate
have viewed the before
please view
Respond only with fenction call of of the following fenctions provided: "suh_task_ sask__tt__complle com_lconplets "view_files ren_asi2on_ file". "filesi_filex" ', "iitt_filee"" "delete_files files ", and you wast to output code only use the wiiit_ffiles fonotion to omriut code.
[IM] [ [Pusction Call] [sub_ tttp_itttt__sspee_eeeeeeee siep_.cmpplte. 1)
[CU] Please proxide eae or two scatesce sumnary of the ourput of this step which is usefol for the next step. You respouse will he used when starting the nest step without any of the previoux messagex.
[LM] The basic structure the applicutios iss been set up, including main apslication file (asp.ay) Flask application. using in-monory dicriussry applicatios catep correct
passed
em resrieve commatly
The second instruction is loaded, and the process repeats.
H.5 ADDITIONAL DIVERSE PROGRAMMING CODE GENERATION TASKS
We extended the codebase generation system design benchmark with three new, additional diverse programming code generation tasks. These are a recipe application, an event planner application, and financial tracking application. Each task consists of user prompt of listed features to implement, and Code-I.2MAC produces the entire codebase from scratch. This is tabulated Table 4.
We observe that Code-L22MCC continues to fully implement the highest percentage of user-specified feature requirements across these new diverse tasks, while its code contains minimal syntactical errors and passes high number of unit tests therf-te, Code-L2MAC still achieves state-of-the-art for completing these system design large code generation benchmark tasks. Moreover, we also have implemented new standard code quality metric of code coverage percentage of the unit tests (Miller & Maloney, 1963), labelled Cov %, and similarly observe Code-L2MAC also has a high code coverage percentage to its substantially larger amount of generated lines of code.
55

---

Published as a conference paper at ICLR 2024
Table 4: Codebase generation system design task results showing the percentage of functional features specified that are fully implemented (Features %) ), the number of syntactical errors in the generated code (# Errors), the number of lines of code (LOOC), number of passing tests (Tests Passed), and the unit test code coverage percentage (Cov %). Code-L2MAC fully implements the highest percentage of user-specified feature requirements across all tasks by generating fully functional code that has minimal syntactical errors and a high number of passing self- geffeeeeertddd unit tests. The results are averaged over 10 random seeds, with + indicating 95% confidence intervals.
Rodko-4p 409
Drti Age
Paanac Tricong App
CETE
20.312.12
810
203 1835
3350035
5:03 UnLOE
553155
3631466
0,20311000
$151855
6.15.8111
8724824
194333
025u0339
5891145
18848822
$24-035
22.14438
2244312
273636
92548.75
619
699
12.55
619
243127
619
40.303
24.3
713180
03:13
30.144
RASEER
H.6 HUMAN EXPERT VALIDATION OF FEATURES IMPLEMENTED PERCENTAGE METRIC
We hired two professional software engineers as human experts, separate from the authors of this work, to perform code reviews of the generated codebases for each method against the user-requested task feature checklist, counting only features that they verified are correctly and fully implemented. We regard the resulting metric, labeled human expert features percentage Human Expert Features %, as the ground truth.
We tabulate in Table this metric below across three random seed runs. We highlight two conclusions. CM significantly outperforms other baselines based on Human Expert Features % The human and LLM counterparts, Human Expert Features % and Features % strongly correlate (p = 0.976), thereby establishing Features % as good proxy for the ground truth. This validates our usage of Features % scalable and cost-effective way evaluate the number of features implemented in codebases from new method- task pairs. This conclusion aligns with existing literature on using LLMs as a proxy for human evaluators (Chiang & Lee, 2023).
Table 5: Codebase generation system design task results showing the percentage of functional features specified that are fully implemented (Features %), and ground truth metric of human experts counting the functional features specified that are fully implemeneted Human Expert Features %. C2de-L2MAC fully implements the highest percentage of user-specified feature requirements across all tasks by generating fully functional code. The results are averaged over three random seeds.
URL Shortcner App Features %
Onling Sucial Media App Human Expert Features % Features %
Online Chui App Human Expert Features %
Method
Human Export Features &
Features S
GPT4 AmtoGPT
31.4
53.6
11.1
19.5
10
33.3
CodeL2MAC
78.4
91.6
61.9
82.4
60
59.4
H.7 CHALLENGES AND THE EVALUATION OF WRITTEN TEST-CASES
We also explored using a priori hand-written test cases that are consistent across the different methods. We implemented this metric and present the results in Table 6 below, which shows it is correlated to our proposed main evaluation metric of Features %.
is relevant to discuss some challenges this approach presents:
Hand written test cases assume and impose a known interface or a stringent imple- mentation structure. In small code snippet tasks, such as those in HumanEval (Chen et al., 2021) or MBPP (Austin et al., 2021). an explicitly defined function interface is explicitly presented to the L.M, and the LLM only responds with the code for that function body. In this situation handwritten test cases can assume the pre-defined implementation struc- ture. However, in codebase generation tasks defined through feature requirements, there is freedom about the segmentation into components, modules classes which must be appropriately determined by the code generation method. E.g., allowing a user to register an account can be achieved with many different code implementations. By specifying tests, filter this ambiguity into given implementation approach and we cannot account for all other possible code implementation approaches to implement a functional feature correctly.
56

---

Published as a conference paper at ICLR 2024
Requires expert-crafted task-specific test cases priori. This hinders the scalability of this approach.
We added these hand written test cases into each method's context window C* throughout all stages of generation. Once the method generated codebase, we then randomly changed the wrriten test case parameters to still be the same test, just with different test parameters, to avoid the method memorizing the test examples, c.g. changing the initial user_ide to random value. Since all methods often generated codebase that did not exactly match the implementation of the test-cases, while still having a codebase that would conceptually pass the purpose of the test, we used GPT4 to port the test cases to each specific codebase implementation. We term the proportion of such tests that pass human-written tests as HT %, which is the percentage human tests that pass for given codebase. As tabulated in Table 6, which is computed over 5 random seed runs, observe that this metric correlates to our Feature % evaluation metric (p 0.695), which further provides empirical evidence for such metric.
It is important to point out that providing the test cases to the LLMs is not ideal since it hints at how design the implementation.
Table 6: Codebase generation system design task results showing the percentage of functional features specified that fully implemented (Features %) percentage of human written tests that pass (HT %), the number of syntactical errors in the generated code (# Errors), the number of lines of code (LOC), number of passing tests (Tests Passed) and the generated unit code coverage percentage (Cov %), ). L2dC- fully implements the highest percentage of user-specified feature requirements across the task by generating fully functional code that has minimal syntactical errors and high number of passing self- sefff-enerated unit tests. The results are averaged over 5 random seeds, with + indicating 95% confidence intervals.
URL Shortener App Errors
Method
Features
HT %
LOC
Tests Passed
Cov % t
GPT4
25:79.66
0::0
3.75+110.9
134::19.9
6.7517.16
14.5
CodeT
13.2-+42.1
11.1433.44
0+:0
146+
7.75
86.8+t4.57
Self-Refine
30.61:3.77
33.3:141.4
0.2:0.555
14019.83
91:
74.611.85
Reflexion
30.9+-20.8
33.3+11.44
0+-0
84.5++33.9
3.590.911
96.5+
Code-L2MMC
76.5-+33.3
41.7+154.7
0+-0
286-1172
10:19.09
83-8.72
Furthermore we also experimented with not including the hand written test cases each method's context window throughout all stages of generation. If we otherwise follow the same setup as outlined above, we observe the following results as tabulated Table 7. We observe that the HT metric correlates to our Feature % evaluation metric (p 0.928), which again further provide empirical evidence for such a metric.
H.8 GENERATING 1,000+ LINES OF CODE WITH CODE-L2MAC
By removing the restrictions we imposed upon Code-L2MAC to economize api-calls, such as limiting the amount of instructions 10, we get variation we term Code-L2MAC-Lare that we tested on the Online Chat application task where it reached reached over 1,000+ LOCs (5x the LOC of AutoGPT, the next highest) as shown in Table 8 below.
H.9 CODE-L2MAC ABLATION WITH NO INSTRUCTION SUMMARIZATION MESSAGE
Code-L2MAC does not depend on the summary message Mrs from completing the previous in- struction when loading My.. Indeed, all the outputs regarding previously completed instructions are contained in the file store/extemaal memory and can be accessed demand.
instruction and can tackle each instruction from scratch, without
We empirically verify this by performing an ablation of Lode- that removes this summariza- tion message step, without affecting the quality of the output code by much shown in Table 9.
57

---

Published as a conference paper at ICLR 2024
Table 7: Codebase generation system design task results showing the percentage of functional features specified that are fully implemented (Features %), percentage of human written tests that pass (HT %) the number of syntactical errors the generated code (# Errors), the number of lines of code (LOC), number of passing tests (Tests Passed) and the generated unit test code coverage percentage (Cov %). Code-L2MAC fully implements the highest percentage user-specified feature requirements across the task by generating fully functional code that has minimal syntactical errors and high number of passing self seff-eeeertedd unit tests. The results are averaged over 5 random sceds, with + indicating 95% confidence intervals.
URL Shortener App Errors
Method
Features % t
HT % 1
LOC
Tests Passed
Cov % 1
GPT4 CodeT
37.6++3.3
204-29.9
0+-0
94.2+
2+11776
85.6++18.
17.1
42.2:11..5
0
98=19
4.843.45
91.8=7.15
Self-Refine
50.6#16,8
46.7=49.2
0:10
109:
3.442.26
9211.99
Reflexion
55.3+-8.3
37.8+33.2
0.6+11.67
1244-31.3
3.44-2.99
87.6++128
Code-L2MAC
89,41:12
71.1
0
2831:100
8.6.19522
77.21:3.77
Table 8: Codebase generation system design task results showing the percentage of functional features specified that are fully implemented (Features %). the number of syntactical errors in the generated code (# Errors), the number of lines of code (LOC), and number of passing tests (Tests Passed). Code-L2MAAC-arge can generate 1,000+ lines of code. The results averaged over 10 random seeds, with + indicating 95% confidence intervals.
Online Chat App
Method
Features %
Errors
LOC
Tests Passed
Code-L2MAC-Large
53.3 19
0.333 1.44
1,030:140.8
5.67 13.7
We included this summarization message step to steer the LLM to find the correct files faster since we expect more similarity and interdependence between contiguous instructions than between more distant ones.
Table 9: Codebase generation system design task results showing the percentage of functional features specified that are fully implemented (Features %). ). the number of syntactical errors in the generated code (# Errors), the number of lines of code (LOC), and number of passing tests (Tests Passed). The results are averaged over 10 random seeds, with + indicating 95% confidence intervals.
URL Shortener App Errors
Method
Features %
LOC
Tests Passed
Code-L2MAC (Ablation, without instruction output summarization) Code-L2MAC
89. 89.4119.8
0::0
274.143.3
8.2.1.122
91.6:18.22
330.147.6
14:1..71
H. ADDITIONAL TASKS ON IMPLEMENTING A NEW FEATURE IN AN EXISTING LARGE CODE BASE WITH CODE-L22MC UP TO 165,000 LOCS
We added new three tasks of implementing a new feature in an existing large codebase with Code- 1,2MAC, where each codebase has at least 87,000+ LOCs, and one task up to 165,000 LOCs. We highlight that the task of taking an existing large codebase and implementing new feature is common software engineering (Lee et al., 2002). These tasks involve the following three existing open source codebases of Dynamic Dashboard App (Tabor, 2023), Community Forum App (Justin, 2023) and a Data Exploration App (Breddels, 2023). For each task, we take that existing codebase and implement feature, which new button to delete all dashboards, new front page statistic to show the number of newly registered users within the last 30 days, and to show the current time on the application banner, respectively. Unique these tasks is that instead of Code-I 1.2M starting
58

---

Published as a conference paper at ICLR 2024
from scratch, it starts with the existing codebase which it must correctly condition the generation of new code on. The results are tabulated in Table 10.
We observe that Code can implement the feature, and still has high feature implementation percentage for these new feature implementation tasks, whilst working with large codebase of up to 165,000 LOCs.
Table 10: Implementing new feature in existing large codebase task results showing the percent- age of the runs that implemented the feature as specified that are fully implemented (Feature %) and the number of lines of code (LOC). Code-L2MAC can implement the feature whilst working with a large codebase up to 165,000 LOCs. The results are averaged over 10 random seeds, with = indicating 95% confidence intervals.
Dynamic Dashboard App
Community Forum App
Data Exploration App
Method
Features %
LOC
Features %
LOC
Features %
LOC
t
Code-L2MAC
80130.22
165,811.61143.8
304
88,878.441.668
10010
87,044.5118.5
FUTURE WORK
We envision the possible exciting future directions for future work.
Error Checking. More advanced error checking, checking the output of the LLM to control for hallucinations, and to prematurely avoid out-of-context errors thrroug the potential networking
adoption of backpressure techniques, as employed in existing message-base systems (Tassiulas & Ephremides, 1990). Furthermore, possible future direction control the generation of correct code is to use further tools to run and verify the code, for example, static code linter (to pick up initial syntax or similar errors) as feedback when the code has been generated for sub-task instruction.
2. Recursive planning and Reprogramming, Another possible way to avoid out-of-context errors is enable recursive planning of instruction into smaller sub-instructions and modify existing stored prompt program instructions I to include these. In addition, future work could explore the possibility of replacing the sub-steps altogether if the original instructions prove to be an ineffective plan. This could involve some reflection of the reason that rendered the current plan ineffective. Furthermore, replanning or reprorgramming could also be seen as replacing the sub-steps altogether, however the degree to which (percentage) the original prompt program instructions that are re-programmed is an exciting future work direction.
3. Multi-processing with prompt-program. Investigating and supporting multi- multi-proocssing program flow patterss, where we execute and have access to two or more LLMs to use, and similarly with the associated tools as well.
4. Support for comprehensive control flow (code flow) paradigms: Investigating and supporting wider range of control flow (code flow) paradigms, encompassing constructs like 'while' loops, 'switch' statements, 'if" statements, as well as advanced patterns such as asynchronous and coroutine patterns.
5. Optimize the expected subtask length of a sub-task. Given LLMs struggle with longer context windows (Liu et al., 2023a), perhaps better overall task performance can be achieved by reducing the average sub-task context length, of which there could be optimum. Furthermore, there exists a further tradcoff with a smaller context window to optimize performance of both the overall task output performance and the compute performance of the LLM.
6. LLM may only be able to solve tasks that can be expressed in natural language (text) that it has been trained on. To complete tasks that it has not been trained on, such as implementing a new application in a new programming language, it may need to be further trained by fine-tuning related examples. This may also apply to the use of specialized tools as well, such as the use of special optimization tools, to leverage when planning how to decompose task into sub-tasks.
59

---

Published as a conference paper at ICLR 2024
7. Generating prompt-programs with human machine collaboration. Unique to prompt programs is that they are naturally human-readable, allowing a human to inspect them and interpret them, of which the human may make changes to the prompt program I, by modifying or adding new parts to it. Furthermore, the architecture can also incorporate human input as tool decide the next control flow of the prompt program and use that as part of the state to execute a sub-task instruction.
8. Interpretable operation for srieta-critical applications. The entire process and flow of operation is implemented natural language, allowing humans inspect the operation and be interpretable to a human. This allows the operations to be debuggable and potentially reason about corner case states in the operation of such prompt program.
9. Building up semantic knowledge. For example, building up detailed, up-to-date live semantic knowledge of the large codebase detailin how it works and what components do what. This semantic knowledge of how the codebase is structured could then be leveraged when completing new sub-task instruction to have minimal additional read operations and can be further updated when the sub-task is complete, persisting this semantic knowledge of the generated code.
10. More tools and varied formats of memory Depending on the domain of the task, we could include more tools, such the ones described in (Schick et al., 2023; Hu et al., 2023). Furthermore, the memory could hold other forms of information, such as images or speech, and LL2MAC could still interact with such information through image2text, text2image. image2image (or the parallel ones for audio and any other form of information).
11. Specialized LLM agents. We could consider having specialized LLM processors, each tuned for different task such as planning, coding, summarizing, writing tests, etc. Then, the CU would choose which to instantiate at each step depending on the current goal.
12. Other methods for reading. It is worth investigating the impact on the performance of using other reading methods such as the common embedding k-NN (Borgeaud et al., 2022).
13. Scalability and deployability. Future work could investigate the token efficiency of exccution. That is, minimizing the total number of tokens generated for task completion. A first effort could investigate fine-tuning LLM to write code in the form of diffs instead of overwriting full files or making more fine-grained reads of files (sometimes, instead of reading the whole implementation, it could be enough to only read function and class names and possibly their descriptions).
14. Implementation optimization. There are two inherent limitations to the length of the generated code files in the current implementation of Code-L2MMC. Both can be readily addressed, providing fertile ground for future work. These are:
Reading a file involves outputting all its content into the LLM's context and writing a file implies rewriting This means that to avoid falling out of context, the maximum length for file is the LLM context window, but, in fact, it is undesirable to have a file of length more than half the LLM's context window, since this will imply that the LLM cannot modify it without falling out of context. This limitation can be overcome by endowing the LLM with the capacity selectively read and write parts of the file (e.g.. function names and headers) in the same spirit as diffs are conducted in git. This is in line with the previous item on "Scalability and deployability".
All the code file paths are listed in the context window C². Therefore, the maximum number of file paths (for example, ['app-py,' rest_app-py". '...) listed in C0de-L2MAC can have in memory is strictly less than the context length. This can be readily solved by allowing the LLM only to list the file paths inside a folder and the possibility to navigate inside sub-folder. In such case, the constraint would happen only regarding the degree in the file tree, but we could theoretically have infinite depth in the file store.
15. Improving Code-L2MAC's outputs' efficiency. It is straightforward to measure efficiency. The runtime computational and memory complexity of test could be provided as feedback to the LLM, which could use this measure to reflect upon the efficiency of the current implementation and optimize it if necessary.
60

---

Published as a conference paper at ICLR 2024
1.1 L2MAC NON-CODING DOMAINS
The general-purpose property of stored-program computers and the motivation of L2MAC in such framework suggest that L2MAC could inherit this general purpose.
First, it should be noted that the coding tasks cover a broad set of impactful applications, which, combined with our empirical results, suggests that L2MAC might encompass wide applicability.
Apart from that, we can consider other applications for this framework. As discussed in Section 3.1, crucial consideration for L2MAC is the acknowledgment that LLMs are imperfect. Thus, the evaluation tools for instantiating L2MAC play an important role in its effectiveness (note that this role could become less and less crucial as LLMs progress). Consequently, additional applications that would render most immediate for the usage of this framework are the ones for which we have effective evaluation tools or where there is less stringent requirement on the output structure. This might include AutoMI, generating mathematical proofs (L.i et al., 2021), or writing long textual documents; the first two have natural or existing verification tools (Bayer et al., 2022); the latter imposes less strict requirements, where an LLM could also be used as evaluator.
The materialization of experiments on these tasks is beyond the scope of this paper; however, each of these poses exciting directions for future work.
61


Reviews:
[{"review_id": "Cta1Uxf4Ub", "reviewer": "Reviewer_mgbF", "summary": "Authors propose L2MAC tool that implements automatic generation of large code bases using LLMs. Authors implement context management to keep relevant context and summarize/compress previous context to keep it within context size bounds. They implement approach to read and write file data across all created files. They also implement generated output checker that runs static analysis and unit tests and processes error messages. Authors evaluate their tool on a benchmark set and demonstrate improved results compared to state-of-the-art baselines.", "questions": "- Figure 4 (c) - could you explain why for Code-L2MAC the figure shows the number of tests passed, while for other two tools it shows the number of tests failed? Are these numbers comparable? If so, why and how?\n- Does the \"checking of generated output\" part contain novel contributions? It seems to me that other tools and approaches also have feedback loops where code is regenerated on errors. It would be good if this was clarified. (This is possibly a minor weakness).\n- If I understand correctly, unlike autonomous tools that can add additional subtasks dynamically L2MAC creates a subtask list once at the beginning and does not subdivide or add additional tasks later. It seems that authors consider this to be a strength, because L2MAC will not go into hallucination subtask loop. However, this could also be a drawback since L2MAC may create subtask that needs to be subdivided later and it will not be able to do so. This should be evaluated. This also connects to the weakness of benchmark set that only has 3 benchmarks: perhaps other benchmarks would show the strength or weakness of this approach.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Structured framework for LLM-based computation that can deal with limited context, file input/output and output evaluation and testing.\n- Context handling that preserves information needed for the tasks and limits context to the context size\n- Read and write implementation for files generated during subtasks. Demonstrated capabilities to write, then read and update files.\n- Strongly improved results on benchmark tasks compared to strong baseline models/tools.", "weaknesses": "- File read/write implementation details are not clear. Please explain how your system decides what files to write, read, and update and how this is different from previous systems that did not have this functionality.\n- Benchmark set is not described. It is not clear if the benchmarks are representative of large code base creation tasks. Evaluation is done on only 3 tasks. The number of tasks should be increased to show the versatility and that the results are not outliers.\n\nMinor comments:\n- Stored-program computer subsection does not seem to contribute much to the paper. Probably shorten or remove.\n- Code-LLMatic is used instead of CodeL2MAC in couple places. Did not update old name?\n- Footnotes on page 4 mostly do not add to the narrative. Remove?\n- Figure 3 is placed after Figure 4 for some reason.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROc6A5DNvy", "reviewer": "Reviewer_v4fH", "summary": "The paper investigates the challenges and capabilities of Large-Language Models (LLMs) in the context of code generation from natural language instructions. A central issue identified is the difficulty LLMs face when generating extensive programs due to the limitations imposed by context window length. To tackle this problem, the authors introduce a system, L2MAC, which augments LLMs with an instruction registry and a file store. Through the orchestration of a control unit, L2MAC breaks down code generation into smaller, more manageable steps, and permits the modification of previously generated code via the file store. Evaluations presented in the paper indicate that L2MAC delivers superior performance in three system design tasks compared to two established benchmarks, GPT-4 and AutoGPT.", "questions": "1. Have you considered incorporating ground truths, such as human-written test cases, into your evaluation metrics? Were there any challenges in doing so? Additionally, have you conducted any evaluation using more established metrics, and if so, could you share the results?\n\n2. Might the inclusion of more sophisticated code generation models such as CodeT, Self-refine, and Reflexion have provided a broader comparison of L2MAC's performance? If this is the case, could you possibly incorporate some of these comparative results in the paper?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Relatively Novel Approach**: The paper presents a novel idea of employing a control unit, instruction registry, and a file store to enhance LLMs. Although the individual components have been introduced in prior work (planning, test case generation, using external tools, refining with code execution feedback), the application of these to a stored-program computer in this context seems to be a fresh approach.\n\n2. **Detailed Descriptions**: The paper provides a thorough description of all the crucial modules. The inclusion of example prompts and turn-by-turn actions in the appendix is an added advantage, as it supports better comprehension and reproducibility.\n\n3. **Addressing Key Challenges**: The paper tackles an important and complex problem - the generation of comprehensive and cohesive programs. It makes significant observations, such as emphasizing the need to enable LLMs to revise previously-generated code, which contributes to the understanding of the problem.", "weaknesses": "1. **Questionable Evaluation Metrics**: The paper employs several evaluation metrics that are based on LLMs rather than ground truths like human-written test cases. This approach raises concerns about the representation of these metrics in terms of code quality. For instance, 'Features %' is determined by a GPT-4 call and not by running and testing the code. Similarly, 'Tests Passed' is based on the LLM-generated test cases, which may not accurately reflect test coverage or code quality. The paper could improve its evaluation by incorporating well-established metrics, such as human-written test cases, and applying a consistent set of tests across different methods. [partially addressed in the rebuttal].\n\n2. **Overclaiming**: Although the paper introduces an intriguing concept, it also overstates several claims. For example, it suggests that L2MAC could enable LLMs to generate virtually unbounded code, but in practice, it is still limited by the context window when breaking down tasks and reading/writing code files. Moreover, while the paper claims that L2MAC can generate large codebases, the evaluation only presents a modest code length (300-400 LOCs).\n\n3. **Weak Baselines**: The paper fails to incorporate more state-of-the-art code generation baselines like CodeT (Chen et al., 2022), Self-refine (Madaan et al., 2023), and Reflexion (Shinn et al., 2023). The use of GPT-4 alone as a baseline appears to be a strawman argument, while AutoGPT is not renowned for superior code generation capability in any popular benchmark, such as HumanEval or MBPP [addressed in the rebuttal].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MQqLfxzuO5", "reviewer": "Reviewer_4zmV", "summary": "The paper presents a LLM-based computer that uses LLM in its core as the `computation` engine and enables the model to interact with two different memories, `instruction registry` as a storage for prompts and user-defined instructions and `file store` as a means to store intermediate outputs. The authors create an interesting synergy with how conventional computers (e.g. Von Neumann architecture) operate and aims to replace the core compute and control engine with LLMs. As one of the applications, the author explored how such LLM-enabled computing platform can productively be used for the task of code generation.", "questions": "This is an interesting and timely idea. While the authors only explore one application for such general purpose computer, it would be interesting to see what other applications this computer can enable. To be honest, I found the writing of the paper/formulating the idea challenging to follow and that makes contributions less clear. \n\n(Q1) I appreciate the authors providing a comparison with the related work in Table1. I am wondering if it would make sense to have a comparison with those in the scope of code generation. I am curious to see how your approach compares with reflecting LLM techniques. \n\n(Q2) I understand the choice of target application for your model, but how do you think L2MAC can be extended to cover other applications and domains?\n\n(Q3) Can you clarify how do you generate test programs for each applications? Do you have another verifier to ensure the correctness of the unit test? What would happen if the unit test are limited coverage in testing the target application?\n\n(Q4) I also find the metric of `feature %` to be confusing. As an end-user, I would like my program to be fully functional and correct. Why do you need to design a new metric for the evaluation? How do you relate your metric with correctness of the program? Is there any correlation here?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "$\\mathtt{+}$ I found the synergy between conventional von Neumann architecture and L2MAC interesting and how the authors created a 1to1 mapping between different components in conventional computing platforms and their proposed design.\n\n$\\mathtt{+}$ The results for code generation tasks is promising.", "weaknesses": "$\\mathtt{-}$ While I think the paper proposes an interesting idea, but I found the writing very challenging and difficult to understand and follow.\n\n$\\mathtt{-}$ While the general-purpose computers can excel work in a variety of task, L2MAC focuses on one particular task and it is not clear how such model can generalized to different application and programs.\n\n$\\mathtt{-}$ While the core idea is still new, most of the explored idea like self-refinement, using external memory, etc. have been explored before in the literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YTMmJeRGgo", "reviewer": "Reviewer_4yCT", "summary": "The paper proposes L2MAC, a practical LLM-based stored-program automatic computer for long and consistent code generation. The experimental results show L2MAC outperforms GPT-4 and AutoGPT for a variety of tasks, including URL shortening, online microblogging, and online chat applications.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a practical LLM-based stored-program automatic computer framework-- L2MAC, for long code generation tasks. L2MAC can generate code for a variety of tasks, including URL shortening, online microblogging, and online chat applications, and outperform SOTA works.", "weaknesses": "1.\tThe performance of L2MAC is evaluated on URL shortening, online microblogging, and online chat applications. Can this method be used for other applications, such as programming?\n2.\tIs there a limit on the length of the generated code files? \n3.\tHow to ensure the efficiency of generated code files?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mNALhmROkR", "reviewer": "Reviewer_Sz28", "summary": "The paper presents L2MAC, a framework for using large language models (LLMs) as automatic computers for long and consistent code generation. The framework consists of an LLM, an external memory (which stores both instructions and data),  and a control unit (which manages the interaction between the LLM and the external memory). The CU enables the LLM to execute a prompt program that contains a list of instructions to solve a user-given task. The CU also provides the LLM with precise read and write operations to access and update the memory, as well as error checking and correction mechanisms to ensure the quality and coherence of the generated code. The paper demonstrates the effectiveness of L2MAC by implementing Code-L2MAC, a practical instantiation of the framework for code generation tasks. Code-L2MAC can generate large code bases for system design tasks that require multiple components and features, outperforming existing SOTA methods such as GPT4 and AutoGPT.", "questions": "1. In Section 3.3.1, the authors mention that if none of the introduced errors are found, CU asks the LLM to summarize the generated output in the current context window. The degree of summarization determines whether contextual information will be lost, which is crucial for long context handling. Can you provide a detailed and clearer explanation?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ The paper presents a novel approach with the introduction of the L2MAC framework, which augments LLMs by integrating memory and control mechanisms. This innovation stands as the first practical LLM-based stored-program computer.\n+ The authors have instantiated the LLM SPC framework as Code-L2MAC, specifically tailored for intricate tasks like long code generation. The proposed method exhibits superior performance compared to state-of-the-art techniques.\n+ The introduced benchmark and evaluation metrics for long code generation tasks are valuable for further research in this field.", "weaknesses": "+ In the manuscript, there appears to be an inconsistency in the usage of the terms \"L2MAC\" and \"Code-L2MAC.\" The authors aim to differentiate between the overarching framework, denoted as L2MAC (which stands for LLM-based SPC), and its specific instantiation, referred to as Code-L2MAC, designed for long code generation tasks. Notably, the title of the paper erroneously employs \"L2MAC\" when \"Code-L2MAC\" would be more appropriate for the context of long code generation. Similar discrepancies are observed in the abstract, the first summarized contribution, Figure 1, and its accompanying text, among other sections. It is recommended that the authors undertake a meticulous revision to clearly delineate between these two terms. Additionally, while the L2MAC framework's primary focus is on the long code generation task, its potential applicability to a wider array of experiments should not be overlooked. To underscore the framework's versatility and generalizability, it would be advantageous for the authors to incorporate additional tasks.\n+ Regarding the comparative analysis, the manuscript omits some pivotal baselines, notably the Reflecting LLMs. While the related works section enumerates three methodologies—Single LLMs, Reflecting LLMs, and Autonomous Agent LLMs with memory—the results presented in Table 2 only encompass those of GPT-4 and AutoGPT. This omission should be addressed to provide a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)