When Prompt Engineering Meets Software Engineering: CNL-P as Natural and Robust "APIs'' for Human-AI Interaction
Authors: Zhenchang Xing
Yang Liu
Zhuo Cheng
Qing Huang
Dehai Zhao
Daniel SUN
Chenhua Liu
Venue: ICLR 2025 Poster (2025)
Keywords: large language model
human-LLM interaction
API
prompt engineering
software engineering
controlled natural language
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: By integrating prompt engineering and software engineering principles, this paper introduces CNL-P which advances a more natural and robust SE paradigm serving as an AI infrastructure to enhance human-AI Interaction.

Abstract: With the growing capabilities of large language models (LLMs), they are increasingly applied in areas like intelligent customer service, code generation, and knowledge management.
Natural language (NL) prompts act as the ``APIs'' for human-LLM interaction. 
To improve prompt quality, best practices for prompt engineering (PE) have been developed, including writing guidelines and templates. 
Building on this, we propose Controlled NL for Prompt (CNL-P), which not only incorporates PE best practices but also draws on key principles from software engineering (SE).
CNL-P introduces precise grammar structures and strict semantic norms, further eliminating NL's ambiguity, allowing for a declarative but structured and accurate expression of user intent. 
This helps LLMs better interpret and execute the prompts, leading to more consistent and higher-quality outputs. 
We also introduce an NL2CNL-P conversion tool based on LLMs, enabling users to write prompts in NL, which are then transformed into CNL-P format, thus lowering the learning curve of CNL-P.
In particular, we develop a linting tool that checks CNL-P prompts for syntactic and semantic accuracy, applying static analysis techniques to NL for the first time.
Extensive experiments demonstrate that CNL-P enhances the quality of LLM responses through the novel and organic synergy of PE and SE. 
We believe that CNL-P can bridge the gap between emerging PE and traditional SE, laying the foundation for a new programming paradigm centered around NL.
To further demonstrate the effectiveness of CNL-P, we develop the AI-native CNL-P IDE, UGAiForge that leverages CNL-P to enable users to Think, Describe, and Build with AI and for AI.
For detailed information, please refer https://ugaiforge.ai.

Published as a conference paper at ICLR 2025
WHEN PROMPT ENGINEERING MEETS SOFTWARE ENGINEERING: CNL-P AS NATURAL AND ROBUST
"APIS" FOR HUMAN-AI AI INTERACTION
Zhenchang Xing CSIRO's Data61 zhenchang- xing @data61 css ro.
Yang Liu," Zhuo Cheng, Qing Huang
Jiangxi Provincial Key Laboratory for High Performance Computing, State International Science & Technology Cooperation Base of Networked Supporting Software, Language Intelligence Research Center, Jiangxi Normal University (bgly, zcheng, qh jxnu. edu. on
Dehai Zhao CSIRO's s Data61 dehai zhao@data61, csi au
Daniel SUN
Institute of AI, Newcastle University, UK and UGAiForge Daniel sswn@eewcastte..ea.
Chenhua Liu
Jiangxi Normal University
programmer iu jxnu, edu. cn
ABSTRACT
With the growing capabilities large language models (LLMs), they are in- creasingly applied in areas like intelligent customer service, code generation, and knowledge management. Natural language (NL) prompts act as the "APIs" for human-LLM interaction. To improve prompt quality, best practices for prompt en- gineering (PE) have been developed, including writing guidelines and templates. Building this, propose Controlled NL for Prompt (CNL-P), which not only incorporates PE best practices but also draws on key principles from software en- gineering (SE). CNL-P introduces precise grammar structures and strict semantic norms, further eliminating NI.s ambiguity, allowing for declarative but struc- tured and accurate expression of user intent. This helps LLMSs better interpret and execute the prompts, leading to more consistent and higher-quality outputs. We also introduce an NL2CNL-P conversion tool based on LLMs, enabling users write prompts in NL, which are then transformed into CNL-P format, thus low- ering the learning curve of CNL-P. In particular, we develop linting tool that checks CNL-P prompts for syntactic and semantic accuracy, applying static anal- ysis techniques to NL for the first time. Extensive experiments demonstrate that CNL-P enhances the quality of LLM responses through the novel and organic syn- ergy of PE and SE. We believe that CNL-P can bridge the gap between emerging PE and traditional SE, laying the foundation for new programming paradigm centered around NL. To further demonstrate the effectiveness of CNL-P, we de- velop the Al-native CNL-P IDE, UGAiForge that leverages CNL-P to enable users to Think, Describe, and Build with AI and for AI. For detailed information, please refer https://ugaiforge.ai.
INTRODUCTION
In the context of AI, particularly with models like GPT-4 OOeeAA 2024). "prompt" refers the input that directs the model's 's output. Prompt engineering is the process of designing these inputs
Zhenchang Xing and Yang Liu co-first authors.
Zhuo Cheng is a corresponding author.
Qing Huang corresponding author.

---

Published as a conference paper at ICLR 2025
to guide the model toward generating the desired output. This can involve specifying parameters, providing examples, or framing instructions in a particular way. Prompts enable humans to interact with AI systems. This interaction is akin to how use software to interface with operating system via APIs (Application Programming Interfaces). If we consider models like GPT-4 as an "Al operating system", prompts can be viewed
new form of API for interacting with this system. The rationale is straightforward: (i) Interfacing with AI: Prompts act as the mechanism through which users interact with and instruct the AI. just as APIs facilitate communication and command execution between software applications. (ii) Guiding Behavior: Prompts shape the output and behavior of the Al model, in the same way that APIs define the possible interactions with software. (iii) Accessibility: Similar to how APIs allow developers to access the pre-built functionality of software application, prompts allow users to leverage the capabilities of Al models without needing to understand or manipulate their underlying workings.
However, we need to be mindful of certain nuances. (i) Expressive Limit: While traditional APIs can handle complex, chained requests, prompts may struggle to express such complexity manage varied tasks a single interaction. (ii) State Management: Traditional APIs often have mechanisms to manage state and persistent data across interactions. In contrast, prompts, especially in stateless models like GPT-4, do not retain state or memory across interactions. (iii) Response Predictabil- ity: APIS typically provide defined and predictable responses based on documentation, whereas Al models like GPT-4 may produce responses with some degree of variability and unpredictability.
To address these nuances, the research community has introduced several effective approaches. A technical direction focuses on using programming languages (PL) to drive Al models, Methods such as LangChain (hangChain] 2024). Semantic Kernel (Microsoft) 2024a), and DSPy (Khattab al., 2023) attach abstraction and templates to PL, while others, like Guidance (Al 2024), LMQL (SRI, 2024. SGLang (Zheng et al. 2024), and APPL (Team 2024), extend PL natively. These approaches harness the expressive power, robust state management, and precise control capabilities of PL to tackle the aforementioned challenges.
However, they also bring new challenges, including: (i) Complex NL-PL. Conversion and Wrap- ping: By "forcing" models to output in specific formats (e.g., JSON), these methods require complex parsers to capture and fix many edge cases of the output. (ii) Prompt-Code Tight Coupling: Prompts are often tightly integrated with code, requiring extensive debugging and optimization to achieve desired outcomes. However, the code implementing the prompt is often simple, merely acting logical connector, making it difficult to separate the roles of prompt engineers and developers, lead- ing to inefficiency. (iii) Unfriendliness to Tec-nechl Personnel: Prompt debugging typically requires language and domain experts to iteratively optimize semantics. However, traditional devel- opment methods encapsulate prompts within code, preventing language experts and non-technical experts from participating in the development and optimization process, and limiting their ability to contribute. This development approach makes prompt engineering difficult to implement effec- tively. To better utilize the expertise of language experts and non-technical experts, we need to decouple prompts from code, enabling prompt engineering to adopt role division and collaborative development like other software engineering processes.
Programming languages (PI.) have solved many challenges but also introduced new ones. We should consider using natural language (NL) to take on some of the functions traditionally handled by PL. This leads to an alternative technical path, which involves using NL templates or style guides, e.g., RISEN (Bammer 2024), RODES (Sebo, 2024), to drive AI models. While this approach can improve response predictability, it still struggles with issues like expressive limits and state management. This is because current methods only provide micro-level incremental upgrades NL. If we treat prompts as a new form of software, we should integrate software engineering (SE) principles at macro level into NL. To tackle these challenges, we can apply first principles thinking to seek solutions. Identifying and Defining the First Principles: (i) Core of Traditional Coding: Writing instructions in programming language is the fundamental method for creating software. (ii) Key SE Principles: Modularity, abstraction, encapsulation, and maintainability form the basis of good software design and development. Deconstructing the Problem: (i) Complexity of Traditional Coding: Traditional coding can be difficult and inaccessible for those without specialized training, limiting wider participation in software development. (ii) Al's Impact on Coding: AI, particularly generative AI, is changing the landscape of coding by automating or simplifying certain aspects of the process. (iii) Need for Intuitive Programming: There is growing demand for more intuitive and accessible programming methods, which could democratize software development and increase
2

---

Published as a conference paper at ICLR 2025
efficiency. Reconstructing the Problem: (i) Introducing Controlled NL (CNL): Propose CNL as an alternative to traditional coding, allowing users to write instructions in format closer to natural language, making programming more accessible. (ii) Adhering to SE First Principles: Design CNL to support core SE principles like modularity and abstraction, while ensuring remains easy to understand and use. (iii) Al Interpreter: Use Al to interpret CNL CNL and translate it into executable code, positioning AI not just as an automation tool, but as bridge between human language and machine-readable code. These insights culminate in our vision of advancing a more natural and robust SE paradigm as an Al infrastructure to enhance human- Al Interaction.
2 DESIGN PHILOSOPHY FOR CNL FOR HUMAN-AI INTERACTION
2.1 PRINCIPLES OF SOFTWARE ENGINEERING
The first principles of SE are fundamental concepts and truths that form the backbone of the SE discipline. These principles are not specific any language, technology, or framework but universal guidelines that shape good software development practices. They also inform the macro- level design of Controlled Natural Language (CNL). Here are some key SE first principles:
Modularity Divide software into distinct, independent components. This improves understanding. development, maintenance, and debugging. Each module should serve specific function and interact with others in a well-defined manner.
Abstraction Simplify complexity by exposing only the necessary aspects. It represents essential features without including extraneous details, allowing developers to focus on high-level concepts without being bogged down by minutiae.
Encapsulation Encapsulation bundles data (variables) and the methods (functions) that operate on that data into a single unit (like a class object-oriented programming). also restricts di- rect access to some of the object's 's components to prevent misuse or unintended interference with the methods and data.
Separation of Concerns (SoC) Manage different aspects of the software separately, organizing code so that each part addresses a distinct concern. This makes the system more scalable, manageable, and easier to understand.
2.2 INSPIRATIONS FROM PROMPT ENGINEERING
CNL-P introduces several best practices from PE that enhance interaction with Al models. Some typical practices are outlined below:
Persona A "persona" refers to the specific role or identity assigned to the model in the prompt. Defining persona helps the model adopt a certain tone, style, or perspective, improving the rel- evance and quality of its responses. For example, specifying that the model should respond a subject matter expert (e.g., "As medical expert, explain...") can make the response more authorita- tive. This helps the model better understand the context and adjust answers accordingly, leading to more engaging and appropriate outputs.
"Constraints" are guidelines or limitations imposed on the model's output to shape the response more effectively. Constraints focus the model's creativity, ensuring the results meet specific expectations. For instance, a format constraint might specify how the response should be structured (e.g., "List five benefits in bullet points"), while content constraint might dictate what to include or exclude (e.g., "Discuss renewable energy without mentioning solar power").
Constraints
Chain of thought (CoT) CoT refers technique where the model is encouraged to articulate its reasoning process step-by-step before arriving at a final answer. This method helps break down complex tasks into manageable parts, improving clarity and accuracy.
3

---

Published as a conference paper at ICLR 2025
Y17
ONUP PRONPT TEND, AGENTT
CEOLENTAL BLOCK ESEJENTIIN BLOCK (COMMAND)
AGENT, JAVE muors
F BLOCK TF CONDITION CCOVMNND) (TLSEE CONDITION SCCMMAND
TENO
ASPEST OPPNONALL ASP NoY ULSCMPIKR WI MEPERENO25 ASPSCT HAVE
DOWMAND NONSONN
OPIIONLL ASTECT NAME
COMMAND IEEA "OOWWAE
REQLEST INPTTI
PERSONA TOSTIN PERSONA PERSONA APPOCTS TOUR PERSONA
IESPLAY MESSUGE
PRRSORIL ASPERTA ROLT ASPECT (UPPCTT
OFNERN c0mtooa
WIT SEFERENEE PRESU
ROI ABREST ASPFOT
AEPERTNOES
(PST)) sAPPENaT
ROI AGREOT
DESP MESIMNE Titde
REPORINCES
PEPERENCE
COMMAND RESULT (OST) SSPENOT
CONSTRANT - OPFIONAL ASFECT NAVE DESCRUPTION WITh JEFERDD
ARGUNIENT LSTT PRESPONSE COUSAND SESULT PSSTT "AFENOO")
ARSUNENT LSST STRUCTURED, TEXT
TYPES "EEHER Trree
DECLANOS STRUCTORED DATA, DEGUAGATINN ONI 1YTES
MESULT
NEPERENNS
ENUM DECUAMATION DECLAVED
EMMM TYPE
WITH
DESCRETTION ELEVENT
STRUCTUREE BATA, TITE DECLAVATION DIELARDD
MESC ELEVENT:
STRJCTURED TITE
DCSCRPTION ON CLENENT STATIC DESCRPT ON PEEEEEENCE STATE PRON
JAME
RIPTION
GEFEREMATE -NRRO MASE "ChESSO
ACCE806 DIT AC0088
DATA THEE AAWAY DATA, TYPE REPEISRRRTTUREER DATA, ENUV, [YP]]
SIMPLE RAME: -AOU
TYPE NAME SMPLE TYPE MANE (DCOLARED TYPE
DICT JOCESS
ENUM ML TYPE "I AADRD Aacne
DIST ACCESS NAVE MPLE MASE
STRLA AC CUUSRRL DATA TYRE - STRTICT RED TYPE SCDVY TP
STRUCTI REC, TEXT STRUCTL RED, TEXT JODY STRUCTUREED TEXT BOOY FORVAT ELEMENT (MATT ELEVEENT"
TYPE
ELEVENT ELEMENT"
STRUCTURRED BODY
ELEMENT ELEVENT
TYPE
POIRAT waad-
VALUE VALUE
ELSMENT JANE:
VALUE DESCRITTON REEEREECES ARWY STRUCTURED, TEXT
VANNRLESS DE TEEE RE UAN AALESI I CARIAGLE E DNGLMATINTTON TONO, VANIOL por
ARNLY 1 ARNAY EURMENIST
DAIA THC
(RAN (OUTTNTT MAR LOW 'FHE WORKER!
coves schneee cedwnndnns
TNITUT DATAI UTT
OUTTUTE TT JEFFERENCE DATAI IEND, DUTPUTS
REPEREN
MAME
MAILFLOW FLOW THANIL FLOWE BILOOOO TEND XWAL FLOWT
Figure 1: Syntax of CNL-P Expressed in BNF (Backus-Naur Form)
3 CONTROLLED NATURAL LANGUAGE FOR PROMPT (CNL-P)
3.1 CNL-P SYNTAX
The syntax of CNL-P is shown in Figure with its execution facilitated by the CNLP_AGENT Within this framework, the CNLP _ROMPPT conveys the complete prompt, defining the prompt information and configuration for the intelligent agent. This serves as a comprehensive descriptor that specifies the agent's 's behavior, constraints, data types, variables, workflows, and more, ensuring its actions well-defined, consistent, and predictable, which aids in smooth integration and interaction with various systems and components. Here a detailed explanation of its role:
Defining Agent's Prompt Information PERSONA:
Specifies the agent's role and characteristies. CONSTRAINTS: Defines limitations or rules the agent must follow. DATA TYPE: Describes the types of data the agent will handle. VARIABLES: Lists the variables that the agent will use or manipulate. WORKER: Outlines the main flow of tasks and operations the agent will perform. API: calling APIs that the agent can interact with. INPUTS, OUTPUTS: Defines inputs and outputs for the worker. REFERENCE DATA: References a vari- able. COMMAND: Defines a command, which may include general commands, API calls, input requests, or display messages.
Ensuring Consistency and Clarity By defining these components within CNLP PROMPT, develop- ers ensure agent's behavior is consistent and clear. This enhances predictability, allowing agent's 's actions to be anticipated based on defined prompt. It also simplifies debugging, as well-defined configurations facilitate easier identification and resolution of issues. Addi- tionally, it provides clear documentation of agent's s capabilities and limitations.
As mentioned in section 2 the design of CNL-P is inspired by SE and PE. More explanation for the alignment analysis be found in Appendix A.2
4

---

Published as a conference paper at ICLR 2025
Fursey Like
Votee Like
CNLO
CNI-F ASPLAM
100
Super
Figure 2: Overview of CNL-P Linting Tool
3.2 TRANSFORMER AGENTS
We have developed three transformer agents, RISEN, RODES, and CNL-P transformers. for the automatic conversion of NI. to corresponding templates or CNL--P These frameworks share sys- tematic approach to prompt generation, focusing on the meticulous extraction and structuring of user input. Specifically, the NL to CNL-P transformer agent simplifies CNL-P syntax interactions for non-experts, benefiting end-users by balancing user-friendliness and cognitive load. For more detailed prompts of these agents, please refer to the section A.3 of the Appendix.
3.3 CNL-P LINTING
Based on the precise grammar structures and strict semantic norms outlined in Section 3.1 and inspired by TypeChat (Microsofi) (2024b) and Pydantic (Pydantic) [024]) we developed a linting tool for CNL-P that includes both syntactic and semantic checks, making static analysis techniques applicable to NL. The methodology is illustrated in Figure 2
We draw inspiration from the code compilation process and divide the static analysis of CNL-P into two stages, akin to compilation: syntactic analysis and semantic analysis. The syntactic analysis abstracts CNL-P into an AST Like structure, physically representing it in ISON format. This tool to perform this series of steps is hereinafter referred to as Parser Like. The semantic analysis, based on CNL-P AST Like structure, gathers information by traversing the JSON and introduces background data. Precise checking of certain semantic information (particularly variable types) in CNL-P is entirely implemented through programming. The tool to perform this series of steps hereinafter referred to as Node Visitor Like. For Parser Like, we use text identifiers to distinguish structures in CNL-P. Unlike keywords, these identifiers can appear in the NL of CNL-P and are hereinafter referred to as keyword_Like. Some keyword Like frequently occur in NL, (e.g., CALL), while others are less common (e.g.,
In the static analysis process of CNL-P, relationship between LLM and Program is not either-or; instead, they complement each other. the future, we will further organically combine L.M and the Program conduct more comprehensive and detailed semantic checks on CNL-P. In addition, note that the entire process continues uninterrupted even if errors are encountered, ensuring that all semantic checks are fully executed. A more detailed description of the linting tool can be found in the Appendix A.4
In order to better demonstrate full workflow: (i) automatic conversion from NL to CNL-P conducted by NL to CNL-P transformer agent described in section 3.2, and (ii) applying linting tool described in this section for static checking, running example is illustrated in section A.5 of the Appendix.
5

---

Published as a conference paper at ICLR 2025
Evalustion Busuanly
Technical Evailuztion Wodkeelly
Non-Techoical
Wisla ty
Reedebery
Readabatr
Protass
Figure 3: Overall Evaluation Radar
EXPERIMENTS
4.1 RESEARCH QUESTION 1: ASSESSING THE QUALITY OF CONVERSION
When converting NL prompts to conform to CNL-P or NL style guides using transformer agents like RISEN and RODES, evaluating the quality of the transformation is essential. Thus, propose our first research question:
RQI: How to evaluate the quality of the conversion from natural language prompts to CNL-P or NL style guides.
To assess this, we outline five evaluation dimensions based on the prompting effect of LLMs:
Adherence to Original Intent (D1) Evaluates whether the structured prompt accurately reflects the original natural language intent, retaining core details without necessitating structural similarity. Structured keywords should enhance clarity and module boundary definition without distorting or negatively impacting the original meaning.
Modularity (D2) Measures how well the structured prompt extracts and organizes implicit struc- tured knowledge-llik data structures, variable types, and logical relationships-ing in- dependent modules with minimal coupling.
Extensibility and Maintainability (D3) Evaluates how easily the structured prompt can be modi- fied, extended, or updated, ensuring stability and adaptability of the overall structure with clear rationale and guidance for changes.
Readability and Structural Clarity (D4) Assesses the logical organization and clarity of the prompt's structure, ensuring that structured keywords highlight module boundaries and aid comprehension. Structured keywords should make the prompt more readable to an LLM by clearly delineating different sections without causing confusion or redundancy.
Process Rigor (D5) Assesses the structured prompt's workflow integrity, including the clarity of iterative steps, variable passing, and input-output management, ensuring precise and log- ical flow that an LLM can follow.
The specific criteria of these five evaluation dimensions can be found appendix A.6.1 We discov- ered highly popular project (Akin 2024), on GitHub, which has garnered 113K stars. This project dedicated to collecting various prompts used with ChatGPT. To better structure these prompts, we selected 93 prompt instants with sufficient content and rich steps from the project. The results were then independently evaluated by LLM (GPT-4o) and human evaluators (4 technical personnel with over four years of programming experience and 2 non-technical evaluators from Education and Business School), using fundamentally distinct yet individually justified approaches aligned with the five evaluation dimensions.
Before the evaluation, we did not provide any explanations related to CNL-P or Few-Shot Learning to the LLM, nor did provide any additional training to the evaluators, allowing them to assess
All the data and codes used in the experiment section can he found in the repository through link http://gthuub.com/rrssoCCCCC

---

Published as a conference paper at ICLR 2025
based on their understanding. For LLM evaluation, we asked the LLM to evaluate all 93 3 (each initial NL prompt was converted into CNL-P, RISEN, and RODES) converted prompts according to the scoring criteria scale of 100, the detailed results are displayed in Figure For human evaluation, because assessing "Adherence to Original Intent" and "Process Rigor" requires significant amount of time for comparison with the original text, and considering that performing a detailed evaluation on a 100-point scale is difficult (Hamel, 2024), we randomly selected 30 3 converted prompts for evaluation. Evaluators assessed them on "low, medium, high" scale according to the scoring criteria, and the detailed results were also displayed in Figure
Based on the results, CNL-P showed the most significant advantage in the technical group's eval- uation and also had certain advantages in the non non-technical group and LLM evaluations. The non- technical group's 's results aligned more closely with the LL-M-based evaluation, which suggests that the LLM's s's understanding of the effects of CNL-P is similar to the perception of the target user group for CNL-P.
On the dimensions of "Modularity" and "Extensibility and Maintainability." all three groups consis- tently showed CNL-P's clear advantage. Combined with feedback from the technical group: "When evaluating these two dimensions, the scores for CNL-P, RISEN, and RODES generally do not change due to specific content differences in the Prompt but are determined by their internal structure," suggests that this is an inherent advantage of CNL-P, grammar-dri -drven language, over simple NL Templates in terms of underlying design.
On the "Process Rigor" dimension, CNL-P also demonstrated advantages in all three groups. Non- technical evaluators noted that after a brief adaptation, CNL-P'ss use of fixed characters to distinguish modules and different line indentations define hierarchy made the entire process description much more rigorous. It was very similar to the structured documents they were familiar with but with clearer semantic definitions than those structures.
However, on the "Readability and Structural Clarity" dimension, non-technical evaluators, due to a lack of relevant technical knowledge, rated CNL-P lower in readability, even though they found its structure visually clearer. They needed more time to understand the application of Types and Variables in the Prompt, which led to lower readability scores in the non-technical group. (This threshold could be reduced by an AL-to-CNL-P Transformer Agent, but it is beyond the scope of this experiment.)
On evaluating "Adherence to Original Intent," we observed that RISEN and RODES (especially RISEN) received significantly lower scores in the technical group compared to CNI.-P. After dis- cussions with the technical group, we found that their perspective on NL was different from that of the non-technical group. The non-technical group viewed "Adherence to Original Intent" literally, while the technical group considered the NL text as an "Agent definition for a type of task" and "a specific task that the Agent is supposed to solve." " They believed that these two aspects needed to be clearly distinguished and that narrowing the Agent's scope would violate "Adherence to Original Intent." RISEN, in particular, narrows the scope of the Agent due to the presence of the "Expecta- tion" module, which limits the Agent's ability to only address a specific task. For a more detailed discussion of RISEN, please refer to Section A.6.3
From the results of all three groups, we observe that CNL-P was more favorably recognized com- pared to RISEN and RODES, especially with a strong preference shown by the technical group for CNL-P.
4.2 RESEARCH QUESTION 2: UNDERSTANDING OF CNL-P BY LLMS
Since CNL-P introduces precise syntax and semantics, it is more complex in form compared to sim- ple template-based methods. We want to know whether this complexity will affect the understanding and execution of CNL-P by LLMs. Thus, we set up RQ2.
RQ2: Can LLMs understand and execute CNL-P without additional explanations or Few- Shot Learning?
A recent paper (msclar 2024) explored the performance fluctuations in LLMs due small changes in prompt formats, such as changing "Passage: Answer: " to "PASSAGE: ANSWER:". The re- sponses from LLMs reflected performance fluctuations in classification tasks within their train-
7

---

Published as a conference paper at ICLR 2025
3pt4omini
100- 012e 2000-a1pe-577773000000 1577453300gg
22:11
4yy50/3yy3 97,889622
1231 $1.10-33003111 $422
75-
40109
CR.
CRc.
A1409
most
A0300
Figure 4: Comparison of NL, CNL-P, RISEN, and RODES Scores Across Models
ing datasets, caused by slight format variations. CNL-P differ significantly in "appearance" from other natural language prompts (NL Prompts). We aim to investigate whether significant changes in prompt structure and content at the macro level result in sharp fluctuations in model performance without providing any explanations or using Few-Shot Learning.
In this experiment, Sclar et al.] (2024). These tasks are with standard answers in a broad sense and different levels of difficulty. We hope obtain objective and fair conclusion for the RQ2's main objective through such tasks in this first work. There are some relatively complex tasks among these classification tasks.
selected six classification tasks from dataset AllenAA 2024 presented at
For example, Task 1162 asks the model to assess whether the title of a given paper paragraph is appropriate. Some instants of this task with many technical terms and the language is rigorous and precise, testing the model's 's reasoning and summarization abilities. Tasks 1424 and 1678, on the other hand, are related mathematical reasoning. Task 1424 involves answering questions related to probability, while task 1678 is a computational problem. Tasks' details are shown in Figure of the Appendix. There are some relatively complex tasks among these classification tasks. Detailed example can refer Figure 17
For the underline LLMs, GPT-4o, Gemini- 1.5.ii-.--Prr-002. GPT-4o-Mini, Llama3-70B-8192 and Claude-3-Haiki are chosen.
The experimental process was as follows: (i) Instance Sampling: We set random number seed and selected 50 instances from each task, ensuring that the same batch was used across different models during testing. (ii) Complete task instants: For each task, we generated three prompt types: CNL-P, RISEN, and RODES. Along with the NL prompt, these four prompts were used as system prompts respectively, to drive the LLMs to complete the tasks. (iii) Task Evaluation: Finally, we assessed whether each task completed correctly by comparing the LLMs' responses the corresponding standard answers. We employed five L.LMs across six tasks, resulting in 30 groups of accuracy rate results for NI., CNL-P, RISEN, and RODES, respectively. We selected the prompt(s) with the highest accuracy rate from each group as the best prompt(s), assigning it a score of 100. The performance values of other prompts were then expressed relative this best prompt. The results are detailed in Figure 18 and Figure 4 The experimental results show that although CNL-P introduces precise syntax and semantics, its simple and intuitive design allows LLMs to understand its syntax and semantics without the need for providing CNL-P language additional explanations or Few- Shot Learning. Overall, this allows LLMs achieve performance comparable to natural language prompts in output generation.
4.3 RESEARCH QUESTION 3: STATIC CHECKING CNL-P
The proposed linting tool enables static checking in CNL for the first time. In this experiment, we explore the third research question.
RQ3: Can the designed linting tool effectively conduct static checking in natural language prompt that conform to CNL-P syntax?
Dataset Construction and Description We compared compilation emrors in code against the syn- tactic features of CNL-P, identifying distinct task types. Initially, we manually created three error templates and applied them to these tasks using GPT-4o. By leveraging GPT-4o, generated
8

---

Published as a conference paper at ICLR 2025
dataset of 47 task instants, each with at least three instances, ensuring comprehensive coverage and accuracy for evaluating CNL-P's effectiveness in handling various coding errors.
Summary of Results Among the 47 task instants verified with the linting tool, the accuracy rate was 100% with a redundancy rate of 0%. In contrast, verification with GPT-4o revealed cases of incorrect location and reason, 4 cases with correct location but incorrect reason, and 1 case with incorrect location but correct reason, resulting in an overall accuracy of 76.60% and a redundancy rate of 19%.
Results Analysis CNL-P linting demonstrated a significant advantage in meticulousness during checks. For instance, in task 007, when the global variable auser_account.o _f tness used parameter for the API get_ certttoooouuuppp -plan, the key region with the value 'JP' did not conform to the optional fields ['US', 'CA', 'AU', 'UK'. 'IN']. Despite multiple tests, the LLM consistently over- looked this error, whereas the linting tool identified it. Regarding redundant error content generation, the LLM's inherent flaw of "making things up" due to its NTP (Neural Text Processing) mechanism is evident. In multiple experiments, whether across different tasks with the same template or within the same task with different templates, redundant content generation randomly appeared in such clusters of instances. Nevertheless, CNL-P maintains a 76.60% accuracy and 0% redundancy rate, indicating that the GPT-4o can indeed understand this structured natural language. This suggests GPT4-o remains capable of interpreting and processing structured prompts effectively. Therefore, we conclude that the designed linting tool can effectively conduct static checking for NL prompts conforming CNL-P syntax. In Appendix A.8] we provide example of actual instance exam- ined by the Linting tool. More analysis of the experiments can be found in Appendix A.9
DISCUSSION
CNL-P an executable requirement enhance clarity, testing, and education. In the up- coming version's 's development, CNL.P will be treated as an executable requirement, integrating ALTERNATIVE FLOWS and EXCEPTIONS inspired by use case modeling, along with SCENARIOS spired by Gherkin user stories. Additionally, the API section can include a construct FUNCTION, which clearly defines intended use and potential misuse directly within its definition, serving as in- line documentation. This feature is particularly beneficial in educational contexts, emphasizing cor- rect function usage and common pitfalls to avoid. Furthermore, by incorporating Python's doctest and Hypothesis, CNL-P executable requirements simplifies the specification and testing process, ensuring reliability and clarity, making it an invaluable tool for developers and educators.
CNL-P has the potential to support the CNL-P to code paradigm through a full-stack compiler, ensuring a clear distinction between "What" (high-level intent) and "How" (low-level implementa- tion). This CNL-P compiler will handle all compilation stages, ensuring efficient and error-free code generation. CNL-P's structured syntax enables developers to specify desired behavior without con- cerning themselves with implementation details, which the compiler will seamlessly translate into optimized machine code. Unlike NL-PL. systems, CNL-P maintains a clear separation between spec- ification and implementation, eliminating ambiguity and potential errors. This thorough approach ensures precise, efficient, and maintainable code, setting CNL-P apart from the tight coupling often found in NL-PL conversions.
The CNL-P agent represents a novel software integration form that extends beyond traditional pro- gramming paradigms. By seamlessly integrating with broader SE activities, such as generative user interface (UI) development and technical documentation (e.g., reStructuredTexx), the CNL-P agent enhances the efficiency and coherence of software development processes. This integration stream- lines the creation of dynamic and responsive Uls and ensures that technical documentation is both comprehensive and easily maintainable.
However, it is important to note that CNL-P alone cannot address all the challenges associated with LLM-based agent development. Just as programming languages depend on compilers and suite of supporting tools such as linters and testing frameworks, CNL-P requires comprehensive ecosystem of software engineering tools to facilitate the development, testing, maintenance, and operation of CNL-P agents. We envision broader suite of LLLM-ssisted software engineering tools to support the entire CNL-P development lifecycle, including: A CNL-P tutorial agent that teaches CNL-P, provides relevant examples, and answers syntax and semantic questions. A documentation agent

---

Published as a conference paper at ICLR 2025
that converts CNL-P requirements into technical documentation (e.g.. reStrrccture integrating CNL-P into the broader software DevOps context. Such a software engineering ecosystem around CNL-P aligns with core software engineering principles and best practices, such early error detec- tion, which reduces the cost of fixing issues at later stages of development. By identifying errors or inconsistencies at the requirements level, CNL.-P can prevent costly implementation mistakes. To- gether, these tools will form a CNL-P-centric SE4AI infrastructure, leveraging LLMs to empower more people integrate AI into their lives and work.
6 RELATED WORK
Recent research in SE for AI (SE4AI) focuses on applying SE principles, such as modularity, ab- straction, encapsulation, and maintainability, to PE, transforming it into new form of SE. This shift enhances the predictability and quality of Al outputs while integrating Al capabilities into traditional SE workflows. The proposed CNL-P framework embodies this concept by incorporating SE prin- ciples to reduce NL ambiguity through precise grammar structures and semantic norms, enabling more accurate and consistent interpretation by LLMs. In line with SE4AI, CNL-P bridges the gap between PE and SE, laying the groundwork for a new NL-centered programming paradigm.
Another closely related area is AI4SE, which leverages Al to enhance SE tasks such as API design, code generation and automated testing. In this context, prompts serve as interfaces between humans and AI systems, akin APIs in SE. By optimizing prompt design with CNL-P, which eliminates NL ambiguity through structured grammar, CNL-P enhances the clarity and effectiveness of these interactions, functioning as APIs for AI systems like GPT-4
Compared to programming language (PL) extensions such as Guidance Al 2024), LMQL (RI 2024), SGLang (Zheng et al. 2024), APPL (Team] 2024), and LangChain (Lang 2024), Se- mantic Kernel [Microsoft] 2024a). and DSPy (Khattab et al.) 2023 which attaching abstraction and templates to PL, use PLs to manage state and structure in LLM prompts, CNL-P offers a more ac- cessible and natural solution. While PL extensions provide robust control and expressiveness, they encounter challenges like complex NL-PL conversion and tight prompt-code coupling, making them less suitable for non-technical users and collaborative development. In contrast, CNL-P decouples prompts from code, providing clear role separation and enabling collaborative PE by both developers and language experts, thus addressing key limitations of PL-based methods.
When comparing NL-based methods, such as RISEN (Balmer 2024) and RODES (eboo 2024 which aim to enhance Al model performance through NI templates and style guides, CNL stands out by offering a macro-level transformation. While RISEN and RODES provide micro-level im- provements NL, they fall short in terms of expressive power and state management. In contrast, CNL-P introduces a macro-level transformation by integrating SE principles to manage complex- ity and reduce NL ambiguity, resulting more consistent and high-quality outputs from LLMs. By merging the strengths of NL and SE, CNL-P represenns significant advancement in PE, overcoming the limitations of both PL and NL-based approaches.
In summary, CNI.-P aligns with SF4AI and A14SE by introducing a controlled NI. framework that enhances the precision, accessibility, and collaboration of PE. It effectively bridges emerging PE methods with traditional SE practices, offering novel, NL-centered programming paradigm that propels both fields forward.
7 CONCLUSIONS
As LLMs progress, we introduce CNL-P to address the inherent ambiguity in NL prompts. By merging best practices from PE and SE, CNL-P utilizes precise grammar and semantic rules. To- gether with the developed NI. to CNL-P transformer agent and a CNL.-P linting tool, CNL. opens the door to achieving the ambitious vision that is advancing a more natural and robust SE paradigm as an Al infrastructure to enhance human-Al Interaction.
Comprehensive Comparative Evaluation of CNL. versus DSPy, LangChain, and Semantic Kernel can be
found in Appendix 10
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by National Natural Science Foundation of China (Grant No. 62102171, 62262031), the Distinguished Youth Fund Project of the Natural Science Foundation of Jiangxi Province (Grant No. 20242BAB2301 I), Jiangxi Provincial Department of Education Science and Technology Youth Project (Grant No. GJJ210340).
REFERENCES
Guidance AL. Guidance: guidance language for controlling large language models. https:
//github. aigithub.com. wiwpnnneeeaigdidinnnneeeeea September 2024. Accessed: 2024-9-29.
Fatih Kadir Akin.
Awesome chatgpt prompts.
httpss://ithub.comm/
Swesomee-hhttppt-prroeee 2024. Accessed: 2024-10-01.
AllenAI.
Natural instructions.
htpp://ggitubb.commaaaenn.
natural-instrucet
2024. Accessed: 2024-10-1.
Kyle Balmer.
Risen. htte:s///eellaooooccc//errrereer
ai propootpts 2024.
Accessed: 2024-10-01,
Hamel. Creating a llm-as-a-judge that drives business results, 2024. URL https://hame htw//bloo/pposss/1mm--deee
Accessed: 20241126.
Omar Khattab, Amny Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-
improving pipelines, 2023. URI. https:ttp/..... /tttps://ariiit wwwp:ss.///ooooooto /21// 033714
Tobias Kuhn. A Survey and Classification of Controlled Natural Languages. Computational Linguistics, 40(1):121-170, 03 2014. ISSN 0891-2017. doi: a 00168. URL
httpes://ol..orr///.1166/
LangChain. Applications that can reason. powered by langchain. https: www www.://ca....nc...n..
http://wwwwwww.
com/ September 2024. Accessed: 2024-9-29.
Microsoft.
Semantic kernel.
https://eearn.mmcrroofff.....
us/
Semantic--eerel/oveevvi September 2024a. Accessed: 2024-9.29.
Microsoft. Typechat.
httpp////tthaa.cooo/.coooett//t/ 2024b. Accessed:
2024-10-1.
msclar. Formatspread. http:p///gthhb.coo/m///////////s 2024. Accessed:
2024-11-28.
OpenAl. Gpt-4: Openai's 's next-generation language model. gpt 4/ September 2024. Accessed: 2024-09-29.
Pydantic. Pydantic. httpes//gttabb.c////aaatttttppgp 10-1.
2024. Accessed: 2024-
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sen- sitivity to spurious features in prompt design or: How learned to start worrying about prompt formatting, 2024. / 11324
Sebo.
http:://ww..hepppppraarriioo. 5 prompt - frameworks level prompt 2024. Accessed: 2024-10-01.
Rodes.
ETH SRI. Lmql: A query language for language models. httpp:://gtthub..ommeeeesss1
Imql September 2024. Accessed: 2024-09-29.
APPL Team. Appl: A programming language for ai applications. https://github. com/ appl-team/ape 1 September 2024. Accessed: 2024-09--9.

---

Published as a conference paper at ICLR 2025
Lianmin Zheng, Liangsheng Yin. Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024. URL https: //aaiiv/....v
orc/abs/2312..71
A APPENDIX
A.1 PENS ANALYSIS
To establish a principled classification scheme for CNLs, Kuhn (2014) introduces a four-dimensional framework called PENS, representing Precision, Expressiveness, Naturalness, and Simplicity. Preci- sion assesses the ambiguity, predictability, and formality of CNL definitions. Expressiveness indi- cates how well a CNL can represent concepts compared to natural language. Naturalness combines grammar modifications, understandability. and natural aesthetic (look-and-feel) The fourth di- mension can be called Complexity or to have dimension of the type "more is better" Simplicity. PENS creates a conceptual space for CNLs with five-level classification (I 5) across these di- mensions, blending natural and formal languages such as English and propositional logic. Based on this classification, CNL-P denoted as s² Its specific meanings include:
Reliably interpretable languages (P3), The syntax of these languages is heavily restricted, though not necessarily formally defined. The restrictions are strong enough ensure reliable automatic interpretation.
Languages with maximal expressiveness (E*). Capable of conveying any message exchanged be- tween two humans, covering all logical statements.
Languages with natural sentences (N*). Consisting of valid natural sentences recognized and un- derstood by native speakers without need for instructions or training.
Languages with short descriptions (s*). Require comprehensive description spanning more than one but fewer than ten pages.
Under this classification, CNL-P is characterized by concise grammar and semantics (S*), predom- inantly expressed in NL (N²), matching the expressiveness of NL (B5), and offering moderate inter- pretability (P*).
A.2 ALIGNMENT ANALYSTS OF CNL.-P WITH INSPIRATION OF SE & PE
The design of CNL-P embodies comprehensive approach to SE principles, ensuring a flexible, maintainable, and extensible framework for defining and configuring intelligent agents. By adhering to principles such modularity, abstraction, separation of concerns, and encapsulation, CNL-P offers developers a robust "API" platform.
Modularity CNL-P is organized into independent components, each responsible for specific func- tionalities. This modular design allows for independent development, testing, and mainte- nance, promoting reusability and scalability. Clear interfaces between components prevent unintended side effects, simplifying system maintenance and extension.
bstractionn High-level concepts in CNL-P abstract complex implementation details, enabling de- velopers to focus on essential features. Non-terminal symbols and component abstractions provide a concise way to describe agent behavior and attributes. Well-defined interfaces and the abstraction of complex operations further streamline the development process, making the language easier to use and extend.
Separation of Concerns Adhering to the single responsibility principle, each component in CNL- P handles a specific function. Clear interfaces and boundaries ensure that changes one component do not affect others, enhancing extensibility and of maintenance. This design keeps the system flexible and maintainable, allowing developers to manage different functionalities independently.
12

---

Published as a conference paper at ICLR 2025
Encapsulation CNL-P employs component- based design where each component encapsulates its functionalities and data. Data hiding prevents direct modification component's ' inter- nal state, maintaining integrity. Well-defined interfaces serve contracts between compo- nents, promoting modularity and reusability. This encapsulation simplifies maintenance, as bugs and issues can be isolated and fixed without impacting the entire system.
CNL-P's design is also inspired from emergent practices in PE. The CNL-P syntax itself includes constructs for the persona and constraints. which inherently support best practices in PE. More- over, its sequential workflow definition, conditional execution, and command execution resonate with the step-by-step reasoning typical of the CoT approach. These elements enable developers to define and configure agents that can perform complex tasks by breaking them down into smaller, manageable steps and executing them in logical sequence. This also conforms to the idea of divide and conquer. The specific explanations are as follows.
Sequential Definition
The WORKER component facilitates the definition of sequential workflows,
ensuring that tasks and operations executed in specified order. This sequential execu- tion reflects the step- by--ep reasoning characteristic of the CoT approach.
Conditional Execution The IF BLOCK component enables conditional execution commands, al- lowing the agent to evaluate specific conditions and execute different commands accord- ingly. This aspect of conditional reasoning is crucial the CoT approach, as it involves assessing intermediate states and making decisions based on those evaluations.
Command Execution The COMMAND component defines individual commands that the agent can execute. These commands represent distinct reasoning steps within the CoT process, en- hancing the agent's 's ability to perform complex tasks systematically.
A.3 TRANSFORMER AGENTS
We have developed three transformer agents, RISEN, RODES, and CNL-P transformers, for the automatic conversion of NL to corresponding templates or CNL-P shown in Figure 5 Figure 6 Figure 7 and Figure g
These frameworks share a systematic approach prompt generation, focusing on the meticulous extraction and structuring of user input. This common methodology can be broken down into several
key stages.
Role and Context Identification Each framework starts by defining the Al's role or persona, en- suring that it operates within the intended context. This step is essential for establishing the tone and scope of the Al's interaction with the user.
Objective Extraction
The frameworks then extract the primary goal or intent from the user's input, creating a clear foundation for the prompt and ensuring that the AI understands the main task or goal.
Detailed Instructions and Steps Each framework provides of detailed, actionable steps to guide the AI through the execution process. This stage ensures the Al follows a struc- tured path to achieve the desired outcome, improving the accuracy and effectiveness of its responses.
Expectations and Examples
To align the Al's responses with user intent, the frameworks specify expected outcomes or offer examples. This step clarifies the desired output, ensuring that the Al's response meets the user's needs.
Constraints and Limitations Finally, the frameworks account for constraints and limitations, tai- loring the AT's output specific user requirements. This ensures the response stays focused and relevant, avoiding unnecessary deviations.
Conversion Process The frameworks follow a step-by-step conversion process that transforms user input into structured format, prioritizing accuracy and clarity. This process guarantees that the AT's response is precisely aligned with user input.
This systematic approach ensures that user imput is converted into structured and coherent prompts, guiding the AI to perform tasks efficiently and accurately. Specifically, the NI. CNL-P transformer
13

---

Published as a conference paper at ICLR 2025
#en **aask Overview**
3. **Steps*;; Outline clear, actionable steps the AI should follow, in sequence.
Transform the user's input into RISEN framework prompt, ensuring each section is clearly defined and accurately reflects the user's intent.
4. *Exppetation**: Determine the type of output the user wants.
*Narrowing**: Include any constraints, limitations, specific focus areas.
=#Understadding RISEN**
Role: [Define the AI'S role. E.g., Advisor, Creator]
#h#
*eey considerations**
Keep it concise and accurate.
Input: [Provide detailed input. E.g.,
Ensure each section directly reflects the user's s input.
Specific question topic]
Steps: [Outline clear steps. E.g-, First, provide an overview, then delve into details] Expectation: [State your desired outcome. E.g-, A comprehensive guide, brief summary] Narrowing: [State any limitations, restrictions, what to focus E.g., word limit, focus on cost-effective options]
Maintain clarity and logical flow.
**Prompt Architecture**
Role: [AI's role, e.g., "Analyst"]
Input: [User's detailed question topic]
Steps:
#tn **Conversion Process**
1. [First action]
1. "*Role*"; Identify and specify the role based on the user's 5 task (e-g., "Advisor," "Content Creator").
2. [Next action]
*IInpt**; Extract key topics, questions, Expectation: [Desired outcome] data provided by the user.
Narrowing:
[Any restrictions or focus areas]
Figure 5: Transformer Agent for NL to RISEN Template
Examppes (E)**;
You are tasked with genersting a RODES franswokk proupt
Identiff and provide any examples givon by the usar
based user input, ensuring that all sections are clearly accurately intont.
to clarify expectations.
exemples are provided, nention section should be completed basad on the context.
*Understanding RODES**
(S)**:
The RODES franswork consists of five main components:
Ensure that final step asks the AI understands
the task and if additional details are needed.
(R)**;
persona identity the
should adopt.
sua *Kay Considerations*>
2. ""bbjective (0)*; Specifies the nain goal task.
Keep the prorpt clear and concise. directly
**Details (0)*n; Outlines specific requirerents,
user's intent. Maintain logical flow and avoid unnacessary complexity.
each
constraints, or steps for the task.
4. **Exarples (E)*:; Provides examplas of the destred input and output.
"*Prospt Architecture*
5. **Sense theck (5)*:; Confires understanding and checks for additional clarifications needed.
1. Role (R): [Define Specialist")
task. E.g.,
The final RODES prorpt should achere to this structure, enabling the AT to perform the task effectively.
Objective (0):
#as "CCnversion Process for Each RODES Companent* 1. **Role (R)*::
[State the goal. E.g., "Analyze and provide seg strategles for new wehsite."]
Identify persona
based on user input.
Exarple: If the user asks for sarketing advice, the could "Digital Harketing Expert."
3. Details (0):
[Include any requirenents, constraints, steps. E.g., "Focus on keyword research and link-building techniques.")
2. "*bbjective (0)**;
Extract the nain geal or task from the usen's input. insights social nedia trends, the objective could be "Provide an anelysis of current social redia trends."
Exasple: If
4. Exarples (E):
[Provide input-output examples available, specify
**petalls (D)*s:
then.)
Include any specific instructions, requirenents, or steps provided by the usen.
Exasple: If
specifies, "Focus
Sense (S): "Da you understand the task, or do you need any further clartfication?"
related to TikTok and Instagras," include this the details.
Figure 6: Transformer Agent for NL to RODES Template
agent simplifies CNL-P syntax interactions for non-experts, benefiting end-users by balancing user- friendliness and cognitive load.
14

---

Published as a conference paper at ICLR 2025
moou are tasked with converting natural longuage user input Into a Sonsin-Specific Language (05L) based on the DSL BNF structure provided. follow these guideline liss carefully:
"*Derine Variables
Types:"-
"3,11 Identify Known Variableg:"a Extract Input and output requi regurtennts fron the user's input; check If the usen's recudrements involve API calls, extract the irputs and outputs of these APIS, and treat all these inputs and outputs as variables.
ase 1. Understanding DSL BNF:
Learn the syntex of oSi BMF will guide the transforsation process.
+.3.2 Infer variable Types: Based on the context of the user's input, Infer the types of the identified variables, deternining whether they sirple data number, boolean) defined the data types (SmurTutED _MPTA__VPE)) 33,33 Focus on VARIABLES and TYPES eNF Provide appropriate definitions based the identified variables and inferred data types. Mote that TYPES is not required part, only complex data TYPES with multiple attributes are defined in TVPES.
Prerequisite Knowledge: Backus content indicates that appean either on times. the other hand, content signifles that "contene" can appear B any number of tiees. It's ieportant to distingutsh between the use of brackets '[" and "I" in this notation. specifically, "I content I denotes that []" appears as literal string, whereas "content" within brackets does not imply this.
(BNF),
"cortent" is optioonal
can
(such complex
<REF>CNL- Exaattx__xppesseed_ Expressed_ (RE-FF
4. **Dofine the KORKER:**
+sconversion Pracecs:**
*4,1 Detormine the WORKER'S INPUTS and OUTPUTS** and corresponding BMF, +4, Focus the FLDN SNF; attention identifying explicit irplicit conditions in imput. present, define
Identify descriptions related PERSONA, describe the privary ROLE this agent plays and its key attributes (describe provided by user) for its functionality ity.
"eFouus
gonerate their definitions according
_F_Bo otherwise, define
EIQUENTIAL__BOCE
*4.3 Define commiND:: Carefullyy distinguish between different types of cowwan poDy and strictly nap the relevant content Froe the user's input to the corresponding cOMPANO_ Bo0Y.
2. +eFouu on the CONSTRAINTS INF; Extract all constroint-neilate information fron the usen's input and define it within the CONSTRAINTS sMF.
Figure 7: Transformer Agent for NL to CNL-P Template (Partl)
*1DSL Generation considerations:"* *SStep-hy-Stee
[CEFINE VARIABLES:) _var_1: boolean
the Conversion Process, when executing specific step, focus only relevant to that step and ignore the BMI required other
Focus:**
_var_2: SomeType
stops. Additionally, when focusing on a particular BNF section, you need to consider the complete syntax of that section, neaning that each part of the syntax must be defined down to "conrds".
[END_ [AR__VATABLES]
WORKER: "Worker description"
the lowest level, such
"AAccuracy in Translation:** Directly convert input
[INPUTS]
into DSL without adding details exponding instructions. user Input descriptions whenever possible, rather than naking up your oun statements (comsndds).
Additional INPUTS necessary
[END INPUTS]
3. *Sttratagic placonant:*s Each plece of user input should be optisally placed into the appropriate DSL section, evaluating the pros and cons of different placesents to deternine best DSL section for it.
[OUTPUTS]
<REF> nawe_a 2 </REF>
Additional OUTPUTS iF necessary
[PNO_OUTPLITS)
"festriction Inference:** Variable types inferred strictly based on esplicit information clear context provided by the user. assumptions interpretations should be rode beyond what is directly supported by the user's input.
[MAIN FLOM)
[SEQUENTIAL BLOCK] E SEQUENTIAL BLOCK example, It's not irean the first block rust be the SEQUENTIAL BLOCK corPAI-x x [COPHAND xoX]
Additional COMMANDS
*1* 4. Example DSL AGENT Structure:
[END_SEQUENTTAL, BLOCK]
Ensure strict adherence to the DSL BNF to generate the following similar structure:
[KKK_BLOCK DESCRIPTION CERRIIITOOLMTTHLRRRE WWWLMITH_RFFERRNEEE # NOT
SEQUINTTAL BI DEK example, Rkk, BLOCK may be a TP __BOCCK
[OEFINE_ AGENT: AGENT_ NOME "Description") )
[GEFINE_ PERSONA:]
[END [END_KK 10000_KK_BLOCC BLOCK]
CORMAND-> [COMMAND 222]
ROLF: DESCRIPTION_NTTEI COATTTOONNNTLLFFFFRRREETE
Additional BLOCK iF necossary, 0.8-,
(END. PERSONA)
OptionalAspecthare DESCRIPTION WITH REFERENCES
SEQUENTIAL BLOCK or IF ELOCK
[END NORKER)
IEND MAIN FLON)
[SND_AGENT]
[DEFINE CONSTRAINTS:
Constrointhame: Lixitation details
optioonlAAppcchirre DESCRIPTIONLITY BEATTOOOONNNLLEEEEEEEEEE REFERENECSS
[ENO_ COMSTRAINTS)
[ore THI TYPES:]
Note that words that are capitalized under1ined are in DSL in above directives (0.g-
Sone Tyne
Attribute_1: str
DESCRIPTION WITH REF ERFEREEEE)) and they can found the DSL
ENN TYPES]
Make sure that DSL you generate confores the structure of DSL
Figure 8: Transformer Agent for NI. to CNL-P Template (Part2)
A.4 DESCRIPTIONS OF CNL-P LINTING TooL
Recall Figure 2, the linting tool mainly consists of two parts: Parser and Node Visitor.Like
A.4.1 PARSER LIKE
Remark Parser Like splits CNL-P into various parts using keyword Like, such as DEFINE PERSONA and END PERSONA. Given the syntactic complexity and character-
15

---

Published as a conference paper at ICLR 2025
isties of each CNL-P part, employ different methods for converting them into Aike
Remark For the Persona and Constraints parts, due to their similar structure and relatively simple syntax, well as the rarity of related keyword in NL, performing lexical analysis, decomposing it into tokens, checking and classifying them one by one, and finally constructing the corresponding AST Like".
method of "first
Remark 3 The WORKER part is the opposite. Constructing its AST Like through similar method is extremely difficult, even infeasible in principle. However, by leveraging the powerful NL processing capabilities of LLMs, cases.
accurately build the AST Like of CNL-P in most
Remark Similarly, for the Type section of CNL-P, with appropriate guidance, LLMs can easily translate the types defined by CNL-P syntax into Python programming language.
Remark 5 These converted types will be marked temporary types and added to the type infor- mation file.
A.4.2 NODEVISITON LIKE
According to the CNL-P grammar, in the JSON structure of CNL-P Aike, the values of the same-named keys have specific patterns. For example, the key command1 must contain the type field. When command1 ("tymandd"type (orppee]== == caal api', , the JSON will contain the paras field, otherwise, it will be absent. Nodevisitor Like extracts values of specific keys and performs series of semantic checks by traversing the JSON of AST Like multiple times.
Remark 6
The first step of semantic checking is to handle the declaration of temporary variables. Remark 7 Search for keys that contain variable declaration information, e.g., response, and re- trieve the names and types of the declared variables.
Remark By analyzing the path of this key in the JSON (e.g.. worker www.n.aain.... www.mmaii....aeeemmmmmmmmmm flow.xxx. www.wxxx.coo.... command1), identify the position of the variable declaration and store this information in a dictionary.
Remark Verify the stored information in combination with global variable information. Remark 10 Record the correct temporary variables in variable description file.
Remark 11 If there is error, log it in the error management file according to predefined struc- ture.
Remark 12 and 13 The second and third steps of semantic checking follow a process similar to the first step, as described in Remark 6.
Remark 14 Type checking is conducted using Pydantic.
A.5 A RUNNING EXAMPLE
In this section, we present an example of developing an intelligent agent using CNL-P for customiz- ing personalized exercise plans and dietary advice. The user initially provides a broad requirement statement, such as:
I need an agent that can customize personalized exercise plans and dietary advice.
Such broad statements fail clearly specify the requirements, making it difficult for large language models (LLMs) to generate the expected agent. Therefore, the requirement needs to be expressed more clearly to better guide LLMs in understanding what kind of agent is needed. An optimized requirement statement might look like this:
You are a fitness and health assistant providing workout routines and diet plans based on user input. You should share only safe and verified health tips, avoiding any content that could harm the user. You can help users
16

---

Published as a conference paper at ICLR 2025
Required external infornation
// CNL-P obtained after transformation NL NL-to-CNLP Agent
SCHEMA INFOMMTION:
Class Pasttier (Tynstict):
[DEPINE FEISONA:]
Begion: Liteeal "DM, 1a0", "uC,, "BNT] // neserkt; The usen's countrtes,,
F: You
fitness and health assistant providing workout
region
routincs and diet plans based usen Input. [EhO_ PSRSONA)
UU5, "CA", TUK', 'DM".
[DEFINE_ C
Class Suertiff(yppsdeet)
only verified health tips, avaid any [EKO_ [OMP_CCSTTAANTTS
position:
that could
dFT Information:
[OFFTNE_ [YPEs:]
pnt_ddet__paatt
#orkouttype ("stronnth", y "cardio", , "Floxbbllty", "balonce") // Aarkoutyyp restricted
"paras":
....
enumerateed valum,, 1.... "strength", "cardio", "flecibility,, "balance".. DietPlan - scals: List[sth), ...] [LNO_ TYPIS)
"usem": "UsarInfo",
"preference": "sttrnng" 13 remaria: the type the string.
parameber preferecee get, prt_ateetpiinn)
[OerIN: VELAALE::]
_chosen workkooo MorkoutType //remaric scchoonnnwnhoosse
1s set data type as Horioutlyse
INFORMATEON
diet plan: DietPlan
[EHO_ [H0_VAAI4BLES]
user info": [ "s klobal": true,
"Korknut
Plarning" fitoesstsatiineerrffiill
volue which violotes the constroint as descrdied in rorand..
"poxitionn:
"regfinnt "Tp" 2/ reeaitts "ap is not defined
[INFUTS]
[1ND_ [RNUSS]
<IlEF> user, user_info: </RLF>
[PAIN FUMM]
(SPQUENTTA:
COMPAND [IMCT "What verkout would you focus
on? (strangth, candio, flez fhexihility, ha balance) wi _chosen yorkout type:
The error message output by the large nodel Enror_path: instruction nainn sequentiol, commond22 Enror_resso:: The variable "_chose__eee jorkout_type" 15 of type "NarkoutType", ; but preference expects type "str",
korkout Type SET)
2 [CALL
get_diet_pia plan
KITH (user:
<RET', over infox//E)>> pre ference::
<RIT>, chown workout type//Err) ste_ plan() wdth _usee__nfoo parsecter preference
clet_plan: DietPlan SETI ressnk
paraneter
The error message outsut by the large model Enror_path: instruction, wotrruttioonnnnnnnnn. fow.ooouuunttttt soquentiol wwt.:bllooocoooookoooooeooeo comranc2 Enror_resson: The usee' paraeeter in pposttion regtonn has an invaltd value 'JP'. Ie should be one of the rollowing: "us', , 'EA', 'AI", 'KK", 'IN".
value _chosin workout type.
(END_ SEQUENTTAL CERTWNATAALLOOO [BLOCK]
LED FLOW)
[NNO [06xra] [FRD] (oo_aarrr
Figure 9: A running example of the full workflow: (i) automatic conversion from NI. to CNL.-P conducted by NL to CNL-P transformer agent described in section 3.2 and (ii) applying linting tool described in section 3.3 to perform a static check.
plan by following these steps: (1) Ask the user what type of workout they would like to focus on (strength, cardio, flexibility, balance) and prompt them for input. (ii) Call the function sat.diet.plar to get diet plan using the user's information and their chosen workout type as parameters, then set the response as the diet plan. (iii) Make reasonable adjustments based on the information provided by the user.
Note that, the requirements are detailed and well-organized. However, ordinary users may struggle to write such precise and well-structured prompts. Transforming initial requirement statement into an optimized version is a complex task that demands significant effort and in-depth analysis, a process known as requirement clarification, which is outside the scope of this paper. Theoretically, we do not require users to articulate their requirements with full precision from the outset. By lever- aging CNL-P as a foundational language, we are developing software engineering tools and agents to assist users in iteratively refining and clarifying their requirements. This approach aligns with established practices in requirements elicitation and analysis, as well as with software requirement patterns in software engineering. For example, to elicit detailed user requirements, we may develop a requirements clarification agent capable of iteratively refining initial vague and incomplete natu- ral language descriptions into detailed CNL-P requirements akin to agile requirements analysis process. This process could produce high-quality prompts similar to the "awesome prompts" used in our experiments. While such agents for refining requirements are still under development, this paper assumes that the user's requirements are already well-defined. Once these agents are fully developed, this assumption can be relaxed.
Once the optimized requirement statement is obtained, the NL to CNL-P transformer agent, de- scribed in section is applied. Figure Jillustaass the results. The agent, leveraging the optimized statement and necessary external information, generates the requirements in CNL-P format. Exter- nal information, such as schema details, is integrated based available system resources, including
17

---

Published as a conference paper at ICLR 2025
API information. Global variable data, like user details, are stored according the application spe- cific requirements, and these predefined configurations are established during system development or user registration, ensuring efficient resource use and customized functionality.
Next, based on the CNL-P requirements, we apply the linting tool from section 3.3 to check for errors. Additionally, we used an LLM to identify potential issues, with the results displayed at the bottom left of Figure | The linting tool accurately detected an error, providing a precise lo- cation and explanation. In contrast, by directly using the LLM, it did not accurately discover the error but regarded a correct setting as an error and gave it. The linting tool accurately identified the error as described at remark 3, where the LLM incorrectly assesses the WorkoutType type to the prefer preference parameter in the get diet plan() API call, as described at remark 6. This mis- take highlights the LLM's limited understanding of CNL.-P syntax, leading to a misinterpretation. Meanwhile, the linting tool offered accurate error location information and detailed explanations.
A.6 ANALYSIS OF RQI
A. EVALUATION CRITERIA OF EXPERIMENT IN RQ1
Adherence to Original Intent Scoring Range: 1-33: Significantly deviates from the original in- tent, with numerous omissions, misunderstandinng,, or misuse of structured keywords that distort the original semantics. 34-66: Mostly aligns with the original intent but contains some inaccura- cies or unnecessary modifications due to structured keywords, causing slight deviations from the intended meaning, or an overemphasis on structure over semantics. 67-100: Completely faithful to the original semantic content, with structured keywords effectively clarifying module boundaries and enhancing understanding without altering or losing any of the original meaning. Examples from the original are adapted appropriately, maintaining the intended message.
Structure is unclear, with structured key- words either absent or misused, leading to ambiguous module boundaries, redundancy, or a lack of clarity; the LLM would struggle to parse and understand the Prompt. 34-66: Structure somewhat present but inconsistently
Readability and Structural Clarity Scoring Range: 1-33:
clear, and structured keywords some ambiguity or minor redundancy; requires more processing effort from the LLM to understand the modular structure. 67-100: Structure is very clear, with structured keywords skillfully applied to define module boundaries, enhance readability, and provide unambiguous guidance, ensuring the LLM can easily interpret and understand the Prompt.
inadequately applied, resulting in
Low level of modularity; the Prompt fails to effectively organize structured knowledge, resulting in poorly defined or highly coupled modules. 34-66: Demonstrates moderate modularity, with reasonable identification of structured knowledge. but some modules show dependencies or coupling. 67-100: Exhibits high level of modularity, accurately identi- fying and organizing structured knowledge into well-defined, independent modules with minimal redundancy and coupling.
Modularity Scoring Range: 1-33:
Extensibility and Maintainability Scoring Range: 1-33: Lacks extensibility and maintainability; unclear rationale for changes, making modification difficult and destabilizing the overall structure. 34-66: Offers some degree of extensibility and maintainability, but modification points are not al- ways clear, requiring additional judgment for adjustments. 67-100: Highly extensible and maintain- able, with well-defined guidance for changes, allowing accurate and seamless modifications without disrupting the overall structure.
Process Rigor Scoring Range: 1-33: Lacks clear process rigor, with unclear instruction iterations, variable management, or input-output handling, leading to confusion in LLM execution. 34-66: Demonstrates a moderately clear process, but certain iterations, variable passing, or input-output elements are not fully defined, requiring the LLM to infer missing steps. 67-100: The process is rigorously defined, with clear iterations, variable passing, and input-output flow, allowing the LLM to execute instructions with precision.
A.6.2 RQ1 STATISTICAL. CHART
The statistical analysis of the LLM evaluation results is presented through five boxplots shown in Figure 10 each corresponding to one of the five evaluation dimensions. In the boxplots, the upper
18

---

Published as a conference paper at ICLR 2025
Adtwnsnce Original Inient
Modiderity
Estemibiilyy and Menismetllyy
Readabiily and Stncchaaal Cartly
Proceess Riger
chip
risen
rodes
0rlp
r1sen
rodes
cnip
risen
190e8
chip
risen
rodes
Ghiip
rieen
rodes
Figure 10: LLM Evaluation Detailed Results
Mart-11
Nar
Nar 207
Parh
Figure 11: Technical Group Evaluation Detailed Result
Adverneee to Org ral Iment
Vosster by
Extere by and Vaints noaityy
Seacoti by and Stucturii Carby
Frecess Siger
Mac
221
210
Figure 12: Non-Technical Group Evaluation Detailed Result
and lower boundaries represent the first (Q1) and third (Q3) quartiles, respectively, while the line in the middle the box indicates the median (Q2). The 'whiskers' extend to the maximum and minimum values that not considered outliers, typically within 1.5 times the interquartile range from the quartiles. Outliers are marked with small circles, indicating data points that deviate from the expected distribution.
And we plotted five sub-bar charts shown in Figure 11 and 12 for the evaluation results of the Technical Group and Tec-necll Group across five scoring dimensions and summarized the findings. Each chart displays three categories: CNL-P, RISEN, and RODES. Below each category, the mean and variance are indicated.
The additional analysis of the special cases of the figure is presented Section A.6.3
A.6.3 ANALYSIS OF SPECIAL CASES IN THE EVALUATION PROCESS OF RQ1
By analyzing the lower outliers for CNL-P in Figure 10 we identified representative case, "DIY Expert" (specific NL and CNL-P prompts provided below[13). In this instance, the scores for Adherence to Original Intent, Extensibility and Maintainability, Readability and Structural Clarity, and Process Rigor were all lower outliers, significantly lower than those of RISEN and RODES.
19

---

Published as a conference paper at ICLR 2025
NL Original Prompt
DIY expert. You will develop the skills necessary complete simple home improvement projects, tutorials and guides for beginners, explain complex concepts in layman's terms using visuals, and work on developing helpful resources that people can use when taking on their own do-it-yourseif project. My first suggestion request "I need help on creating an outdoor seating area for entertaining guests."
want you
CNL-P Prompt
[DEFINE AGENT: DiVExpert "A DIY expert who assists with hame improvement projects"]
[DEFINE_ PERSONA:]
ROLE: DIY expert
Skills: Develop the skills necessary to complete simple home improvement projects
Tutorials: Create tutorials and guides for beginners
Explanation: Explain complex concepts in layman's terms using visuals
Resources: Work on developing helpful resources for DIY projects
[END_ PERSONA]
[DEFINE_ CONSTRAINTS.]
WIN__COSTRRINTSS
[DERINE_ VARIABLES:] [AD__AARIBLES]
[DEFINE_ WORKER: "Assist with DIY projects" DIYWorker)
[INPUTS]
[END_INPUTS]
[OUTPUTS]
[END] [UND_OUTPUTS]
[MAIN FLOW]
[SEQUENTIAL BLOCK]
COMMAND-1 [DISPLAY need creating outdoor seating entertaining
[END_ SEQUENNIAA__LOCK]
[END MAIN_FLOW]
[END WORKER) [END_AGENT]
Figure 13: DIYExpert Prompt
Upon examining the NL prompt and the converted CNL-P, we observed that the original NL text was primarily focused on describing the Agent Persona, with little to no mention of constraints or processes, This lack of detailed descriptions in the NL prompt led to content gaps in the corre- sponding CNL-P modules, resulting in significantly reduced scores. We also analyzed the case of "SVG Designer" (specific NL and CNL-P prompts and RODES, CNL-P demonstrated varying degrees of superior performance across all dimensions, with its for Process Rigor being the only upper outlier among the five box plots for CNL.-P. Upon reviewing the CNL-P, we found that its modules were exceptionally well-structured and com- prehensive. This high-quality outcome can be attributed to the well-crafted and detailed original NL. prompt, highlighting the importance of input quality in achieving superior CNL-P outputs.
provided belov[14p. Compared to RISEN
The technical team argues that the presence of the "Expectation" module in RISEN significantly narrows the scope of the Agent's capabilities, limiting it to the execution of a specific task. A con- crete example provided below Figure 13] This example demonstrates how RISEN customizes an Agent to focus specifically on "a particular task the Agent is designed to address." As result, the broad role of "data visualization expert" reduced that a "visualization expert specializing in atmospheric CO2 level data." Consequently, RISEN and similar Natural Language (NI.) templates received lower scores from the technical team.
A.7 ANALYSIS OF RQ2
A.7.1 TASKS USED THE EXPERIMENT FOR RQ2
We provide the detailed definition of each task in RQ2, as shown in Figure I6. along with specific examples from several tasks in Section A.7.2
20

---

Published as a conference paper at ICLR 2025
NL Original Prompt
would like you to act an SVG designer. will ask you to create images, and you will comee up with SVG code for the image, convert the code to base64 data URL and then give me response that contains only markdown image tag referring to that data URL. Do not put only markdown, so My request give me an image
markdown
CNL-P Prompt
[DEFINE AGENT: SVGDesignerAgent "An agent that generates SVG images and provides them as markdown image tags.
[DEFINE_ PERSONA:]
designer
LEND_PERSONAL
[DEFINE, CONSTRAINTS:]
ResponseFormat: Send only markdown image tag without any additional or code block
[END_CONS) [NN_CONSTRAINTS]
[DEFINE_ VARIABLES:]
imageRequest: text
sygCode: text
base64DataUrl: text
markdownlmage Tag: text
[END_VARE [I[_VARIABLES]
DEFINE_WORKER: WORKER: 'Handles SVG image requests and responses" SVGWorker]
[INPUTS]
@imaaeRRquest
[IND_INPUTS]
[OUTPUTS]
@markdownimageTag
[END OUTPUTS]
[MAIN_FLOW]
[SEQUENTIAL BLOCK]
COMMAND-1
requested image @inageRequest sygCode: text SET]
COMMAND-2 2 [COMMAND Convert @svgCode base64 data URL RESULT base64Dataüri: text SET]
COMMAND-3 [COMMAND Create markdown image tag with @base64DataUni RESULT markdownlmageTan text SET]
COMMAND-4 [DISPLAY @markdownlmageTag]
[END, SEQUENTIAL BLOCK]
[END_ HEND_MAI__RH __ANN__OOO]]
[END_ WORKER]
[END_AGENT]
Figure 14: SVGDesiinerAAent Prompt
NL Original Prompt
want you act as scientific data visualizer. You apply your knowledge data science principles and visualization techniques create compelling visuals that help convey complex information, develop effective graphs and maps for conveying trends over time or across geographies, utilize tools such as Tableau and R to design meaningful interactive dashboards, collaborate with subject matter experts in order to understand key needs and deliver on their requirements. My first suggestion request is need help creating impactful charts from atmospheric CO2 levels collected from research cruises around the world.
RISEN Prompt
Role: Scientific Data Visualizer
Input: need help creating impactful charts from atmospheric CO2 levels collected from research cruises around the world.
Steps:
Gather and analyze the atmospheric CO2 data collected from research cruises globally.
Apply data science principles identify key trends and patterns the
Utilize visualization tools such Tableau and to design compelling and interactive charts.
4. Develop graphs and maps that effectively convey trends over time and across different geographies. Collaborate with subject matter experts to ensure the visuals meet their specific needs and requirements. impactful charts that clearly convey the complex information regarding
Expectation: Create meaningful atmospheric CO2 levels.
Narrowing: Focus using Tableau and R for visualization, and ensure the charts suitable for understanding trends over time and across varlous geographies.
Figure 15: Data Visualizer Prompt
21

---

Published as a conference paper at ICLR 2025
task385 In this task, you're given context passage, question, and three answer options. Your task an incorrect answer option to the question fron choices given. questions, only one of the three answer options
task190: this task, you're given pair of sentences, sentence 1 and sentence 2. job to choose whether two sentonces clearly agree
return
anntailentt//dsagreee (contradiction) with each other, or if this cannot be be in the form of the letters E, C, and N respectively.
deternined (neutral).
incorrect answer options as the output
given question professional law. You are also given options (associated with "A", "#", "c", "D"), out of which only one is orrect. You need to answer the question by selecting correct option. only answer with the choice letter, not
task729
task1424 In this task, need to provide the correct option for given problem probability from the provided options.
should
answer
task1162 In this task, paragraph and title from the research paper, Your task is to classify whether the given title is suitable not for the research paper based on the given paragraph. proper according
given
task1678 Given nntth problen with context and question
answer choices, the task is to provide the correct answer choice chaose
based the problen. the given answer choices by letter: a, b, c, anything else is invalid.
Return "True" paragraph else "False".
Figure 16: Six Tasks Used in the Experiments of RQ2
An instance task 1162
Paragraph: Deubiquitinating enzymes (DUBs) are cysteime proteuse proteins that reverse the ubiquitination by removing ubiquities from the target protein. With over 100 DUBs identified and categorized into at least 7 families, many DUBs interact with one or more cytokines, influencing cellular processes, such as antiviral responses, inflammatory responses, apoptosis, etc. While some DUBs influence cytokine pathway production, cytoline-inducible this article, summarize list of DUBs, their interaction with cytokines, target proteins and mechanisms of action. In Title: Regulatory interplay between deubiquitinating enzymes and cytokines.
An instance of task 1424
Problem: set tansists numbers : 3, 5) possible subsets including the created and subset chosen random, what probability that subset largest number ? Options: /
b)3/B c)3/16 d)5/16 e
An instance of task 1678
Problem: you need to unlock secret code using following clues, can you ? here you have the clues: clue 0:7 numbers correct placed correct position clue 3 nathing correct
0 8 numbers correct but placed its position. clue 9 number correct but not placed at its correct position. clue 5:67 one number correct but not placed at correct position.) Options: 819
918 c. 198
d. 189 e.891
Figure 17: Instance from task 162 task. 1424 task 1678
A.7.2 INSTANCE FROM TASK_1 1162, TASK_142, AND TASK_1672 OF RQ2
We provide three instances, as shown in Figure 17 each from task_1166 task_1424 1424, and task_1672 respectively. The details of the tasks used in the experiment for RQ2 are shown Figure
A.7.3 DETAILED PERFORMANCE OF CNL-P, RISEN, AND RODES IN RQ2
We provide the detailed performance of NL, CNL-P, RISEN, and RODES across all model-task combinations, as shown in Figure 18
A.8 AN INSTANCE OF THE LINTING TOOL IN RQ3
We provide a specific instance examined by the lingting tool in RQ3, which is presented in Figure T9 and Figure 20
22

---

Published as a conference paper at ICLR 2025
task190
spldo
task3
gptdo Lesk729
gptdo lask1162
gpldo Lask1422
task1678
100
22.39
100.0
100.0
32
130.0 100.0
100.0 95.83 25.05 tono
97.56 100.01000
55.35 100.0 57.57 190.0
CHL
RSD
Rones
CHE.P
CHL-
Rac
Rooe
Rore
CRCPP
RoDES
CHL
Ropta
get4omini task190
aptoomini task385
gptdomini task729
gptfomini task1162
gptdomini task1424
aptdomini - task1678
100
95.0
26.0
nAD
56 4.88 96.88
88.0 a6.
32.6892.88 67.8
20.36 58.89
75-
0
CNL
RopES
ORL-P
RODES
CRR
CML
OHL
Arzev
Tlama - task190 90
Temm - Lask385
llama 1esk
llame - lask1162
llama Iask1424
llama - lask
$4.12
$1,82
54 96.37 33.94
91.67 23.3355.85
30.2442.8 57.8
85.67
75
52.44 58.02
CRLP
Rega
CRE
RDEW
ROSEE
CREUP
RODES
CR.
CRE
CR
Rote
gemini
90
gemini task385
gemini - task729
gemini-taskl 162
gemini - task1424
gemini 1ack16778
75-
92.0
50.0
88.0
$4.29
79.17
43,3369311 91.33
82.22
75.33
50
Rem
DRP
RoDes
A2
CRL-
REBR
ROD5
A2
RODES
CRP
ASm
CRLP
nizer
cloude tack190
doude tasiass
dbude - task729
claudo tosk1162
dbude - task1424
daude - task1678
35.0
$0.0
MJ.1
36.21
100.0
34.12
77.78
28.89 92.55
92.6 37 27.62
93.52
30
Roses
DNL
Anidy
RoDES
DHL-
RODES
N
cheu
RizoN
RonEs
Ai
ose
Ause
Rooes
onlo
Resos
RobeSS
Figure 18: Detailed performance of CNL-P, RISEN, and RODES in RQ2
A.9 OVERALL FINDINGS OF THE EXPERIMENT
The main purpose of the three RQs to simulate several key steps of end users when using CNL-P to verify the rationality of our design and explore the opportunities enabled by CNL-P.
Starting from the requirements proposed by users and described in natural language, RQI explores the process of converting the requirements described in NL into CNL-P, RISEN, and RODES. Using LLM and human to score, it compares the effects of CNL-P and template- semplate-based methods such as RISEN and RODES across the evaluation dimensions: Adherence to Original Intent, Readability and Structural Clarity, Modularity, Extensibility and Maintainability, and Process
Rigor.
Due to the introduction of precise syntax and semantics in CNL-P, CNL-P appears more complex in form compared to simple template-based methods. Users may want to know whether this complexity will affect the understanding and execution of prompts expressed in CNL-P by LLMs. RQ2 verifies that without any additional language explanation, few-shot examples, or training the LLMs, the LLMs well understand and execute the prompts described by CNL-P, and the obtained effect is (at least) no worse than that of well-orge well-organized natural language prompts. The results confirm the intuitiveness of CNL-PP syntax and semantics.
23

---

Published as a conference paper at ICLR 2025
[DERNE_ AAEN:: #tmess/tottthAdvvsss
[DE Pt [SONAA]
ROLE: You are fitness and health assistant providing worenut routines and diet plans based on user input.
[END PERSONA]
IDERINE CONSTRAINTS.
FORBID: share only safe and verriid health tiss, avoid any content that could harm the user.
[NN CONSTRAINTS]
[DERNE TYPES:)
Workouthape a I"strength", "cardio"; "flxxiility,, "balance"]
DietPlan a meaks: Lutttaat, total calories: number)
Workoutslon
Lesttest), on_nonnnuuttes number, intensity: text)
feedbackleppnnse "ggeeat. "average",
[NN [YPES]
IDENNE VARIABLES
FitnesUssertnto
_chooen_ workour, Workouthype
cilet_preference text
workout plan: Warkout
diet plan: DittPlan
feedback Fespbackesspnnee
text
ENO VARAABLSS]
[OFFINE WOREF:: "Workout and Det Planning" meVeritittee
(INPUTS)
account,
[END. INPUTS)
[OUTPUTS]
on
[END_ [_TOPUTUSSS
[MAIN_FLOW] FLOW]
[SEQUENTIAL BLOCK
COMMAND-1 JINPUT type would you [strength, cardio, Rexibiity, balancal"
VALUE wwoosnnnnnnoo workout type: WorkoutType SET]
COMMAND [CALL glt_woriout_plan WITH (user: @uusee, wwt.ooottotteeee..ee fitness, type: c.chhooen workoout_yype
RESPONSE workout_plan: WorkoutHlan
Figure 19: An Instance in RQ3, Part
COMMAND-3 (INPUT "Do you have any distary preferences? (+ vegetarian, loweaabb, high-pratein)" VALUF WITH
_Geet profeeefeeeen:cccmmm sir SET]
COMMAND
RESPONSE
[END SEQUENTIALBLOOK
[IF chosen workout "strnggh""]
COMMAND-S IDISPLAY "our
Sworkout,
[ELSEIF @
type "cardio"]
COMMAND DIIPLAYY "Your cardle workout plan includes: @workout_ plan")
[ELSEIF ø e___hhsen__wee workout "Hexbbliyy)
COMMAND-7 [DISPLAY "Your flexibiitt trainigg workout plan includes: @workout_pban")
[Fise p_shosee__wwee Lyye "balance"]
COMMAND-- (DISPAYY balance training workout plan includes: @workout_pkae plan""
JEND IF)
ISEQUENTIAL BLOCK]
COMMAND-9
is: @idiet_plan"] COMMAND-10 [INPUT "How would you rate today's fitress plan? (great, avenage, poorj' VALUE feedbacke
ReddbackResprrse SET
[END. SEQUENTIAL BLOCK]
@feedback "prest"]
COMMAND-11 [COMMAND Set the value of reccommendation to "Nou did an excellent job today! Keep it up!
RESULT recommencation str SET]
[ELSEIF @roedback 'average')
COMMAND-12
_eeoommmnndttio
room mppovement!"
RESULT recommencitioon str SET]
[ELSEIF @leedback "pocr"]
COMMAND-15 [COMMAND Set the value of _eecommnndation to 'Don't be discouraged, Tomorrow another
chance!" RESULT recommendation SETI
[END,IF
[SEUUENTIAL BLOCK]
COMMAND-14 [DISPLAY "Today's feedback: 9 recommendatoo"]
COMMAND-15 [DISPLAY "Your summary: @worrkot, summary]
SEQUENTIAL BLOCK)
[END. MAIN MAIN_FLOW)
[END, WORKER]
[LND_AGENT]
Figure 20: An Instance in RQ3, Part 2
In addition to demonstrating the intuitiveness of CNL-P and its effect on the quality LLM re- sponses, RQ3 further shows the of the linting tool developed for CNL-P conduct syntax and
24

---

Published as a conference paper at ICLR 2025
semantic analysis of the prompts expressed in CNL-P. Linting is a widely adopted static code analy- sis technique for identifying errors in code written in PLs, but this linting capability is impossible to achieve with the existing prompt methods based on natural language. Our CNL-P makes it possible for the first time, and RQ3 also initially demonstrates the technical feasibility of building a series of SE tools around CNL.P.
Through the experimental verification of these three RQs, the rationality and potentials of the CNL- language design has been proved: it not only standardizes the prompt generation process but also opens the door to building a SE4Al infrastructure around controlled natural language, further promoting the progress of the emerging natural language centric programming paradigm by standing on the shoulders of LLMs and for LLMs.
A.10 COMPREHENSIVE COMPARATIVE EVALUATION OF CNL-P VERSUS DSPY, LANGCHAIN, AND SEMANTIC KERNEL
Fundamental Differences: dnldemeentation-rrinteee versus. Requirement- DSPy, Semantic Kemel,, and similar PL-based frameworks are implementation-orientie technologies that extend or augment traditional programming languages with natural language addons. These frame- works rely on programming languages as their foundation, treating natural language as an aug- mented feature. Their core abstractions, such as: (i) LangChain's 's prompt template, pipeline, output parsers; (ii) DSPy's signature and module concepts; (iii) Semantic Kernel's 's semantic functions, na- tive functions, and plans, are still fundamentally programming constructs, designed to encapsulate and simplify programmatic interaction with LLMs and related techniques (e.g., vector stores).
However, these frameworks inherently depend on programming languages and cater to develop- ers with technical expertise (at least to some extent). The technical demands remain high because these tools lack clear separation of concerns between NL and PL. intertwining natural language ex- pressions with programmatic implementation. Consequently, non-technical users or domain experts without programming skills often find these tools inaccessible. For instance: (i) Users still need to understand prompts embedded in or glued by code statements and manage pipelines, parsers, or output templates. (ii) These frameworks do not provide clear natural language syntax or semantics directly to natural language prompts, limiting their usability true natural language tools.
In contrast, CNL-P a natural language-native design, built from the ground up as an executable requirement language for human-Al interaction. Unlike DSPy or Semantic Kernel, CNL-P: (i) Re- quires no reliance on programming languages (e.g., "All-in CNL-P"). (ii) Allows users to express constraints, data, workflows, state management, and security and guardrails (not present in this paper) directly in a structured controlled natural language (CNL). (iii) Functions as a structured, semantically clear requirement document, akin structured technical documentation rather than program code.
By separating what do (requirements definition) from how to do (code implementation), CNL-P enables non-technical users to define agent requirements clearly, leaving implementation details to supporting SE tools (e.g.. The linting tool and the CNL-P compiler discussed in the paper).
User Accessibility and Technical Demand. The key limitation of DSPy and Semantic Kernel lies in their high technical demand: (i) They are designed for developers. requiring significant program- ming expertise. (ii) They lack accessible tools or abstractions for non-technical users, restricting their usability in broader contexts.
CNL-P, on the other hand, is specifically designed for non- non-technical users by nature. Its structure and design are inspired by many widely adopted technical documentation practices, such as: (i) Use Case Descriptions for requirements analysis (ii) User Stories (e.g., Gherkin User Story) for Acceptance Testing and Behavior Driven Development (iii) Standardized Technical Documents (e.g., ATA SPEC 100 in aviation).
These technical documentation practices are tailored for non-technical stakeholders such as business analysts, domain experts, other stakeholders who may lack technical expertise but need to com- municate requirements effectively. CNL-P's natural language-no native nattve design ensures: (i) Ease of use: Syntax and semantics are lightweight and intuitive, reducing the need for training programming experience. (ii) Low technical barrier: CNL-P can be used by non-technical stakeholders who can write structured requirements, making it accessible technical documentation.
25

---

Published as a conference paper at ICLR 2025
In addition, the user accessibility and technical demand of CNL-P can be seen more clearly through the designed RQs of the paper. RQI: Building on the inherent simplicity and ease of understanding of CNL-P's grammar, RQI introduces
NL-to-CNL- conversion agent. This agent further duces the user's workload by automatically transforming natural language descriptions into CNL.-P syntax. RQ2: The simple and intuitive design of CNL--P allows Large Language Models (LLLMs) to understand its syntax and semantics without the need for providing CNL-P language specifications, few-shot examples, additional model training and fine-tuning. This allows LLMs generate out- puts on par with those produced by natural language prompts. RQ3: The linting tool exemplifies CNL-P's broader utility. Functioning as the front-end of a compiler, the linting tool performs syntax and semantic analysis, helping users identify and resolve issues without requiring the user-written CNL-P strictly adhering to CNL.-P syntax. This significantly lowers the technical barrier to entry, ensuring accessibility while maintaining precision and reliability.
These research questions collectively demonstrate CNL.-P's potential to enhance user accessibility and reduce technical demands, thereby advancing the field of software engineering for AI. It is worth noting that these RQs also demonstrate that CNL-P together with the developed NL-to-CNL-P aggnn,, and the linting tool opens the door to achieving the vision of advancing more natural and robust SE paradigm for Al infrastructure to enhance human-Al Interaction. Moreover, our early adoption in lab and with non-technical users (such as high-school teachers, chemists) provide initial evidence on the user accessibility of CNL-PP For example, even the fresh first-year undergraduates with little technical expertise in our lab can effectively use CNL-P to define LLM-based agents. In fact, the first author of this paper is a second-year undergraduate with no significant AI and programming training.
CNL-P as a Complement to Existing PL-based Frameworks. CNL-P is not a competitor to frameworks like DSPy or Semantic Kernel but complementary tool addressing different challenge in agent development. Specifically: (i) CNL-P focuses on what to do: Helping users clearly define, structure, and document requirements at a high level of abstraction. (ii) DSPy and Semantic Kernel focus on how to do: Translating high-level requirements into programmatic implementations and facilitating interactions with LLMs.
The relationship can be likened to an API specification versus API implementations: CNL-P defines a structured, tuaaanreedable guideline for describing agent requirements, while PL-based frame- works like DSPy operationalize it through specific programming abstractions. Furthermore, dis- cussed in Section 5, we are designing a CNL-P compiler that translates user-defined requirements in CNL-P into executable code compatible with frameworks such as DSPy, LangChain, Semantic Kernel, or other emerging Pl--based frameworks.
This separation of requirements in CNL-P to CNL-P Compiler implementations in PLs enables non-technical users to: (i) Focus on high-level requirements without worrying about implementation details. (ii) Rely on compiler to generate corresponding code in the desired programming frame- work, analogous to how high-level programming languages (e.g.. Python, Java) are compiled into machine instructions for execution. (iii) Enjoy the benefits of decoupled requirements and imple- mentations, analogous to "write once, run anywhere" for high-level PLs like Java.
This CNL to compiler to PL architecture inherits the core principles of existing high-level program- ming paradigms (PL to compiler to Machine Instructions), leveraging the capabilities of LLMs to push programming languages closer to natural language. At the same time, it retains the benefits of decoupling high-level programming languages from machine code.
The CNL-P linting tool presented in Section 3.3 a Proof-of-Concept of the front-end of this envi- sioned CNL-P compiler. Our experiments in RQ3 (Section 4.3) demonstrate the feasibility of build- ing the front-end of such CNL--P compiler and its effectiveness in applying robust static analysis to CNL. Connecting this front-end with a compiler backend (c.g., an LLM-based code generator), we will be able to generate agent implementation from the intermediate representation of CNL-P like the one presented in Figure? 2.
Modularity and State Management. A significant advantage of CNL-P is its introduction of mod- ularity and state management at the requirements level, opposed to the programmatic level. For example: (i) Modularity in CNL-P: Users can define data, constraints and instructions in a simple structured and modular syntax, ensuring clarity and reusability. (ii) State Management: CNL-P al- lows users to define state-related releaateereuuieeeeeeeeeeeeeeeeee requirements (c.g., transitions and context dependencies) without
26

---

Published as a conference paper at ICLR 2025
requiring them to manage runtime state explicitly. Instead, the CNL-P compiler and runtime will handle these details automatically.
This approach aligns with high-level programming paradigms: Just as high-level programming lan- guages like Python and Java abstract memory management, CNL-P abstracts away implementation complexities of LLM interactions, enabling users focus defining what states and transitions required, while how to implement and manage state and context will be handled by CNL-P compiler and runtime.
Extensibility and Ecosystem. We recognize that a CNL like CNL-P alone cannot address all the challenges of LLM-based agent development. Just as programming languages rely on compilers and also requires a
an ecosystem of supporting tools-such as linters and testing Trameworks-CNE robust ecosystem of software engincering tools to support the development, testing, maintenance and operation of CNL-P agents. This work, though in its early stage, illustrates that CNL-P paves the way for the development of such a comprehensive SE4AI infrastructure.
We recognize that a controlled natural language like CNL-P alone cannot address all the challenges of I.LM-based agent development. Just as programming languages rely on compilers and an ecosys- tem of supporting tools-such as linters and testing frameworks- CNL-P also requires robust ecosystem of software engineering tools to support the development, testing, maintenance and op- eration of CNL-P agents. This work, though in its early stage, illustrates that CNL-P paves the way for the development of such a comprehensive SE4AI infrastructure.
CNL-P is designed to support evolving ecosystem of SE tools that lower the barriers for adoption and improve user accessibility. In this paper, we presented the proof of concept of two such tools: (i) a Transformer agent which formats a free-form NL prompt into the CNL-P syntax (Section 3.2); (ii) a CNL-P linting tool which validates and optimizes natural language requirements for clarity and correctness through systematic static analysis (Section 3.3).
Furthermore, we discussed the L.M-empowered CNL-P Compiler which can transform high-level requirements in CNL-P into executable code for PL-based frameworks like DSPy or LangChain.
We envision more LLM-assisted software engineering tools to support the entire lifecycle of CNL-P development, for example, (i) A CNL-P tutorial agent that can teach CNL-P, provide relevant ex- amples, and answer CNL-P syntax and semantic questions. (ii) A requirements agent that can itera- tively developing detailed CNL-P requirements from initial vague and incomplete natural language descriptions, similar to the agile requirement analysis process. (iii) A CNL testing framework that can detecting incompleteness, inconsistencies or ambiguities in requirements, similar to Gherkin User Stories and Behavior Driven Development (BDD). (iv) A documentation agent that can trans- form CNL-P requirements into technical documentation (e.g., reStructuredText) to integrate CNL-P into the broader software DevOps context.
Such SE tool ecosystem around CNL-P aligns with software engineering principles and best prac- tices such as carly error detection, which reduces the cost of fixing issues during later stages of development. By identifying errors or inconsistencics at the requirements level, CNL-P prevents costly implementation mistakes. All such tools will constitute CNL-P centrie SE4AI infrastruc- ture, standing on the shoulders of LLMS for more people to leverage AI in their life lives and work.
Quality Assurance and Testing. CNL-P can support improved quality assurance by integrating concepts from behavior-driven development (BDD). (i) Users can define use cases, alternative flows, and exception handling directly within CNL-PP (ii) These structured definitions can be validated against predefined testing scenarios, ensuring correctness before implementation begins. By com- bining BDD principles with LLM capabilities, CNL-P enables robust testing of agents and work- flows, even at the requirements level.
Summary CNL-P represents a paradigm shift from program-centric frameworks like DSPy and Semantic Kernel to natural language-first design. It is tailored for non-technical users, offering a structured, semantically clear medium for defining requirements, which can then be transpiled into executable code using a compiler. This separation of concerns between what to do and how to do reduces technical barriers and ensures broader user accessibility. CNI-P complements exist- ing frameworks by focusing on requirement clarity while leveraging the capabilities of PL-based frameworks for implementation. Its extensibility and ecosystem, inspired by software engineering
27

---

Published as a conference paper at ICLR 2025
principles and best practices, make it a promising tool for advancing more natural and robust SE4AI infrastructure to enhance human-Al Al Interaction.
28


Reviews:
[{"review_id": "CJdSB62j9Z", "reviewer": "Reviewer_32pw", "summary": "- This paper proposes the use of Controlled Natural Language (CNL) by framing prompts as a form of API, which allows users to harness AI model capabilities without needing in-depth technical knowledge.\n- This work applies software engineering (SE) principles such as modularity, abstraction, and encapsulation to Controlled Natural Language (CNL), offering a structure that decouples prompts from code\n- The authors conducted experiments to evaluate how effectively CNL-P adheres to design principles and whether CNL-P or template methods improve the quality of LLM responses.", "questions": "- Considering the setup and the consistency of output generation with single-turn GPT-4-o prompting, what advantages does CNL-P offer over single-turn generation with GPT-4-o?\n- Given the broad goals of this work and the multi-faceted design of CNL-P, what rationale led the authors to focus on these three specific research questions?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Clearly motivated framing of prompting as an API, enabling users to leverage AI model capabilities without extensive technical expertise.\n- Strong connection to first principles in SE, providing a foundation to address challenges in complex NL-PL conversion and prompt-code coupling. This approach is particularly beneficial for language experts and non-technical users by effectively decoupling prompts from code.\n- Dimensions to assess NL-to-CNL-P conversion quality are well-designed, covering diverse quality aspects.", "weaknesses": "- The specific aims of the work remain unclear; while high-level challenges and design considerations are presented, the precise goals are hard to identify.\n- Experiment setup in RQ1 lacks clarity on how the five dimensions are measured and how the 93 prompt instances were chosen. There are also no human validation results presented, even as partial samples.\n- RQ1 primarily assesses design considerations, while RQ2 focuses on accuracy. Given the current setup and task scope in RQ2, the advantages of CNL-P are not fully apparent, as other models also perform well.\n- For a more robust finding that CNL-P is better suited for weaker models, it would be beneficial to include additional experiments with weaker models beyond GPT-4-o mini.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wLTaZnnMvE", "reviewer": "Reviewer_m8X4", "summary": "The paper introduces Controlled Natural Language for Prompt (CNL-P), a novel framework that bridges prompt engineering (PE) and software engineering (SE) principles to enhance the clarity, predictability, and effectiveness of prompts for large language models (LLMs). CNL-P addresses inherent ambiguities in natural language prompts by formalizing grammar structures and semantic norms. The work's primary theoretical contribution is the formalization of prompt engineering through SE principles, supported by a novel static analysis approach for natural language prompts and empirical validation across multiple LLM architectures.", "questions": "Comparative Evaluation: Could you provide a more detailed comparison of CNL-P’s functionality and usability versus DSPy, LangChain, and Semantic Kernel? Specifically, how does CNL-P’s approach to modularity and state management differ in terms of user accessibility and technical demands?\n\nAdvantages and Trade-offs: While CNL-P is designed to decouple prompts from code for accessibility, frameworks like DSPy offer robust control through tight integration with programming language abstractions. Could you discuss specific scenarios where CNL-P might outperform DSPy or vice versa, especially in terms of prompt complexity and user involvement?\n\nNon-Technical Accessibility: CNL-P is described as more accessible to non-technical users than PL-based methods like DSPy and LangChain. Could you elaborate on any studies, tests, or qualitative comparisons you conducted to evaluate this claim? This would clarify the extent to which CNL-P lowers the barrier for non-programmers.\n\nPerformance in Practical Applications: Do you have insights or preliminary results comparing the performance and user experience of CNL-P with DSPy, LangChain, and Semantic Kernel in specific application areas (e.g., dynamic prompt generation or complex workflow management)? Real-world examples could strengthen the practical context of CNL-P’s advantages.\n\nFuture Integration with PL-Based Methods: Given that DSPy and other PL-based frameworks emphasize structured programming benefits, do you foresee potential for CNL-P to integrate with or complement these frameworks? A discussion on interoperability could highlight pathways for combining strengths across approaches.\n\nScoring and Evaluation: Can you elaborate on the specific criteria used to assign scores across the five evaluation dimensions (Adherence to Original Intent, Modularity, Extensibility and Maintainability, Readability and Structural Clarity, and Process Rigor)? Was there a weighting system applied to these dimensions, or were they treated as equally important?\n\nInterpretability of Results: The table of results is challenging to interpret due to its minimal contextual information. Could you provide a more detailed breakdown or rubric that explains how scores were derived, potentially with examples of how different prompt types scored across specific dimensions?\n\nComparative Analysis: Did you consider using statistical measures to compare the performance of CNL-P, RISEN, and RODES across evaluation metrics? This could strengthen the validity of the reported improvements.\n\nPresentation Improvements: The experimental results could benefit from a more visual presentation format, such as radar charts or bar graphs, for easier comparison across dimensions. Would you consider updating the results presentation in a revised version?\n\nCompleteness of Scoring Process: Did you perform any error analysis or additional validation to understand how CNL-P performs in specific scenarios where RISEN or RODES may excel, or vice versa? This would provide insight into potential edge cases for CNL-P.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "Theoretical Innovation:\n- Novel synthesis of SE principles with PE practices\n- Comprehensive formal grammar for controlled natural language\n- Innovative application of static analysis theory to natural language\n\nTechnical Contribution\n- Formal specification of the CNL-P grammar\n- Theoretical framework for prompt verification\n- Rigorous performance analysis across LLM architectures\n- Novel approach to static analysis of natural language\n\nResearch Impact:\n- Opens new theoretical directions in prompt engineering\n- Bridges formal methods and LLM interaction\n- Provides a foundation for analyzing prompt properties\n- Advances understanding of structured approaches to PE", "weaknesses": "Theoretical Limitations:\n- Formal analysis of expressive power could be stronger.\n- Completeness properties of the static analysis need more discussion.\n- Edge cases in the formal grammar require deeper analysis.\n- Theoretical bounds need more rigorous treatment.\n\nMethodological Concerns:\n- Formal comparison with other structured approaches could be deeper.\n- Statistical analysis could be more comprehensive.\n- Theoretical justification for design choices needs elaboration.\n- Formal properties of the conversion process require more analysis.\n\nValidation Gaps:\nLimited formal analysis of grammar properties.\nStatistical significance analysis could be more rigorous.\nTheoretical comparison with other formal methods needed.\nCompleteness of the static analysis approach not fully addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oURz2n1VLI", "reviewer": "Reviewer_QGqE", "summary": "This paper introduces Controlled Natural Language for Prompt (CNL-P), a novel prompt language to elicit high-quality responses from large language models (LLMs). CNL-P combines principles from software engineering with prompt engineering to create structured, accurate natural language prompts for interacting with LLMs. By defining clear syntax and components such as Persona, Constraints, Variables, and Workflow, CNL-P reduces natural language ambiguity and increases response consistency. Its structured, modular format allows for more reliable, maintainable, and interpretable prompts, acting as a natural “API” that makes human-AI interaction accessible and robust. In addition, the author proposed an NL2CNL-P agent to convert natural language prompts into CNL-P format allowing users to write prompts in natural language without expertise in learning syntax of CNL-P. A linting tool is proposed to check the syntactic and semantics of CNL-P. The experiments demonstrate the effectiveness of CNL-P in improving the consistency of LLM responses across various tasks.", "questions": "1. Which Llama model was used in your experiment?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper effectively combines prompt engineering and software engineering to introduce CNL-P as a structured, precise language for prompt design.\n2. CNL-P’s modular design enables independent development, testing, and maintenance.\n3. Its linting tool supports syntactic and semantic checks, which enables static analysis techniques for natural language.", "weaknesses": "I have several concerns regarding the evaluation section:\n1. For RQ1, the authors asked ChatGPT-4o to assess the quality of conversions from natural language prompts to CNL-P or NL style guides based on five criteria. However, the reliability of this evaluation is not properly validated:\n    - The authors did not provide evidence of how the evaluation results correlate with actual human evaluations, which would strengthen their claims. \n    - There is no guideline detailing how the scale for each category is defined, which makes it difficult to interpret the numbers in Table 1.\n2. For RQ2, the experiment lacks comprehensiveness:\n    - Diversity of tasks: Currently, all tasks fall under the classification category. It would be beneficial to include more complex tasks, such as reasoning and coding, to better demonstrate the effectiveness of CNL-P.\n    - The authors should conduct a more thorough evaluation across a broader range of both open-source and closed-source LLMs to validate the effectiveness of CNL-P.\n    - Insufficient error analysis of CNL-P across various LLMs and tasks: The performance of CNL-P varies among different models/tasks. The claim that weaker models benefit more from CNL-P lacks thorough discussion and validation. A comprehensive analysis should include task difficulty, the quality of natural language prompts, the quality of CNL-P, and their relationship to the task performance.\n3. Generalization of CNL-P:  \n    - As all the tasks in the experiment are classification tasks, the natural language prompts should not be overly complex (correct me if I am wrong). Consequently, the paper does not sufficiently address how CNL-P performs with very large and complex prompts. It also fails to clarify whether the linting tool can handle such complex CNL-P.\n4. In lines 789-799, the natural language prompt appears well-organized and detailed. I would like to ask:\n    - For an effective CNL-P prompt, is such a level of detail and organization required from the human input?\n    - How does the organization of the natural language prompt impact the quality of the converted CNL-P prompt?\n    - Does the CNL-P prompt still outperform a well-organized natural language prompt?\n5. The plots and figures for result analysis should be integrated into the relevant pages or paragraphs; otherwise, it is hard to follow the discussion and analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)