Generating Equivalent Representations of Code By A Self-Reflection Approach

## Abstract

Abstract Equivalent Representations (ERs) of code are textual representations that preserve the same semantics as the code itself, e.g., natural language comments and pseudocode. ERs play a critical role in software development and maintenance. However, how to automatically generate ERs of code remains an open challenge. In this paper, we propose a self-reflection approach to generating ERs of code. It enables two Large Language Models (LLMs) to work mutually and produce an ER through a reflection process. Depending on whether constraints on ERs are applied, our approach generates ERs in both open and constrained settings. We conduct a empirical study to generate ERs in two settings and obtain eight findings. ❶ Generating ERs in the open setting. In the open setting, we allow LLMs to represent code without any constraints, analyzing the resulting ERs and uncovering five key findings. These findings shed light on how LLMs comprehend syntactic structures, APIs, and numerical computations in code.
❷ Generating ERs in the constrained setting. In the constrained setting, we impose constraints on ERs, such as natural language comments, pseudocode, and flowcharts. This allows our approach to address a range of software engineering tasks. Based on our experiments, we have three findings demonstrating that our approach can effectively generate ERs that adhere to specific constraints, thus supporting various software engineering tasks.
❸ Future directions. We also discuss potential future research directions, such as deriving intermediate languages for code generation, exploring LLM-friendly requirement descriptions, and further supporting software engineering tasks. We believe that this paper will spark discussions in research communities and inspire many follow-up studies.

## 1 Introduction

Given a code snippet, its Equivalent Representations (ERs) are textual representations that preserve the same semantics as the code. In this paper, ERs include natural language comments, pseudocode, flowcharts, and more. For instance, natural language comments and pseudocode aid developers in efficiently understanding the code and reducing communication barriers (Geng et al., ; Li et al., ), while flowcharts visualize the control flow and data flow, facilitating code optimization. These ERs play a crucial role in the software development and maintenance process. Despite their significance, the automatic generation of ERs remains an open challenge.

In this paper, we propose a self-reflection approach to generating ERs of code. Our approach is built upon a dual framework consisting of two Large Language Models (LLMs). Given an input code, two LLMs work mutually and produce an ER through a reflection process. More implementation details of our approach are in Section 2.
Depending on whether adding constraints on ERs, our approach generates ERs in both open and constrained settings. In the open setting, the form of the ERs is unrestricted, allowing the LLMs to freely use their preferred representations of code. In the constrained setting, we impose specific formats on the ERs (e.g., natural language). Using this approach, we conduct a large-scale empirical study to generate ERs under two settings, leading to eight key findings.

Generating ERs in the open setting. In this setting, LLMs are free to generate ERs without any constraints. As noted, language is the tool of thought (Mercer, ). We believe that the way LLMs represent code reflects how they comprehend it. Therefore, we analyze the generated ERs and summarize four key findings:
❶ LLMs generate diverse forms of ERs in the open setting, including structured ERs (e.g., dictionaries and tables), natural language ERs (e.g., comments and pseudocode), and symbolic ERs (e.g., arithmetic expressions).
❷ From the structured ERs, we find that LLMs treat the code as a structured sequence instead of plain text. LLMs even can produce a plausible syntax tree.
❸ Based on the natural language ERs, we discover that LLMs can reason about the functionality of APIs based on the code text, generating natural language text to explain the APIs.
❹ Through the symbolic ERs, we find that LLMs use mathematical symbols to represent the numerical calculations in code, converting the code into arithmetic expressions.
❺ For different types of code, LLMs flexibly generate different types of ERs. For example, LLMs often generate SQL-style ERs when the code involves searching for elements in a list.
Details of these findings and real cases are in Section 3.1. These findings offer insights into how LLMs understand various types of code, which in turn helps us explore how to better formulate requirements that align with the way LLMs perceive and generate different types of code.

Generating ERs in the constrained setting. By applying constraints to ERs, our approach can support a wide range of software engineering tasks. In our experiments, we impose specific constraints on ERs, including natural language comments, pseudocode, and flowcharts. Based on the experimental results, we obtain three findings. ❻ Our approach effectively generates ERs tailored to software engineering tasks and significantly reduces hallucinations produced by LLMs. ❼ The generated comments provide clear explanations of the code’s purpose and implementation, helping developers quickly grasp the code’s functionality.
❽ The generated pseudocode and flowcharts clearly illustrate the control flow and data flow within the code, aiding developers in optimizing the code. Details of these findings and real cases are in Section 3.2. Looking ahead, researchers can utilize our approach to address various software engineering tasks by applying different constraints.

Finally, we discuss the future directions of our work.

Deriving an Intermediate Language for Code Generation. Directly generating source code from natural language requirements is a challenging task. An effective enhancement strategy is to introduce an intermediate language as a bridge, guiding LLMs to first generate the intermediate language and then the final code (Li et al., ; Paul et al., ). Our proposed approach can be used to automatically derive an intermediate language. Specifically, we use the approach to obtain an ER that lies between natural language and code, with constraints to ensure it adheres to specific syntax rules. For example, we can generate a table that follows a specific pattern. LLMs can first generate this table to clearly represent the entities and relationships in the requirements, and then proceed to generate the code.

Exploring LLM-friendly Requirement Descriptions. Currently, natural language text is primarily used to describe requirements in code generation. However, our findings suggest that in an open setting, LLMs use various forms to represent code. This leads us to consider whether we can express requirements from the perspective of LLMs to facilitate the generation of different types of code. For instance, when a requirement involves extensive numerical calculations, we can directly use mathematical expressions instead of natural language text to describe the requirement. We refer to this approach as “LLM-friendly requirement descriptions.” We believe that such descriptions can unlock LLMs’ reasoning capabilities and improve their accuracy in code generation.

Supporting Software Engineering Tasks. There are many software engineering tasks to generate code-related artifacts, such as code comments and pseudocode. Previous work has primarily focused on specific tasks (Li et al., ; Geng et al., ). We provide a general and unsupervised approach. On one hand, by adding different constraints, our approach can handle different tasks. On the other hand, it does not require labeled data, making it applicable to tasks where data is scarce.

## 2 The Proposed Self-Reflection Approach

This section presents a self-reflection approach to generating ERs of code. We first show an overview of our approach (Section 2.1) and provide detailed descriptions (Section 2.2, 2.3, and 2.4).

### 2.1 An overview of our approach

Figure 1 shows an overview of our approach.
It employs a dual framework consisting of two LLMs, i.e., $M_{CR}$ and $M_{RC}$.
Given an input code $c$, it produces an ER through multiple trials. In each trial, $M_{CR}$ generates a plausible ER based on the code, and $M_{RC}$ generates a new code snippet $\hat{c}$ based on the ER. The similarity between $c$ and $\hat{c}$ serves as an indicator of how much of the original code’s semantics is preserved in the ER. This similarity acts as a supervisory signal to optimize the approach. Besides, our approach allows for external constraints on ERs, which are incorporated into the supervisory signal to guide the process further.

Figure: Figure 1: An overview of our self-reflection approach. Two LLMs ($M_{CR}$ and $M_{RC}$) work mutually and produce an ER through a reflection process. We also can add constraints in ERs.
Refer to caption: x1.png

Because LLMs typically are large-scale or black-box, it is heavy and hard to train them further. To address this problem, we follow the Reflexion framework (Shinn et al., ) and design a self-reflection algorithm to optimize our approach. Specifically, we convert the numerical supervisory signals (i.e., the similarity between $c$ and $\hat{c}$, external constraints) into natural language feedback. The feedback acts as a ‘semantic’ signal by providing a concrete direction to improve representations.
In subsequent trials, the ERs and feedback from previous trials are incorporated as additional context for $M_{CR}$, allowing it to learn from prior mistakes and generate more accurate representations. This mirrors the way humans reflect on past failures to improve their future performance.

### 2.2 Dual Framework

As shown in Figure 1, our approach employs a dual framework consisting of two LLMs, i.e., $M_{CR}$ and $M_{RC}$. Next, we provide a detailed description of two LLMs.

$M_{CR}$ is built upon an LLM capable of following human instructions. Given an input code, we instruct it to transform the code into an ER. The instruction for $M_{CR}$ is shown in Figure 3. If constraints are imposed on the ERs, these constraints are explicitly described within the instruction. As stated in Section 2.1, we expect our approach to learn lessons from past trials. To achieve this goal, we integrate a memory module with $M_{CR}$. The memory stores the outputs of the dual framework in past trials. The memory is added as additional context for $M_{CR}$ in the next trial, helping it generate better ERs.

$M_{RC}$ is also an LLM that can follow human instruction. We instruct it to generate a new code snippet based on the ER produced by $M_{CR}$. The instruction for $M_{RC}$ is shown in Figure 4.

### 2.3 Natural Language Feedback

Given an input code $c$, we leverage the dual framework (Section 2.2) to produce an ER and a reconstructed code snippet $\hat{c}$. Since LLMs may generate hallucinations (Zhang et al., ), the ER may be wrong. Thus, we evaluate the quality of the ER and provide natural language feedback to guide improvements.

Scoring the ERs. An ideal ER should be semantically equivalent to the original code and satisfy external constraints. Thus, we design the following two scoring criteria:

❶ Semantic-equivalent Score. A semantic-equivalent representation can be seamlessly transformed into the original code. Thus, we measure the similarity between the original code $c$ and reconstructed code $\hat{c}$. This similarity reflects how much of the original code’s semantics is preserved in the ER.

How to compute the code similarity remains an open question. Our motivation is that the code not only can be tokenized into a token sequence but also be parsed as a syntax tree. The similarity between code snippets may show in different modalities. As an initial work, we propose a hybrid similarity metric, which combines text similarity $Sim_{text}$ and syntax similarity $Sim_{syntax}$. $Sim_{text}$ measures the rate of overlapping $n$-grams between two code snippets. $Sim_{syntax}$ ignores text surface and computes the rate of overlapping syntax trees between two code snippets. The computations of two similarity metrics are shown in Section A.2.

Finally, we compute the semantic-equivalent score as a weighted sum of $Sim_{text}$ and $Sim_{syntax}$:

$$ $\text{Score}_{semantic}=a*Sim_{text}(c_{o},c_{r})+b*Sim_{syntax}(c_{o},c_{r})$ (1) $$

where $a$ and $b$ are hyper-parameters. We empirically set them to 0.5 and 0.5.

❷ Constraint Score. When additional constraints are imposed on ERs, we further measure whether the ER satisfies the constraints. Since real-world constraints can vary widely, we formally define the constraint score as follows:

$$ $\text{Score}_{constraint}=F(\text{representation};\text{constraints})$ (2) $$

where $F(\cdot)$ is a function returning a numerical score, which indicates the probability that a representation satisfies the constraints.

Generating Natural Language Feedback. We proceed to generate natural language feedback based on the computed scores. As an initial work, we design an empirical feedback template that includes detailed explanations for the scoring criteria and incorporates placeholders for the evaluation scores. This template, illustrated in Figure 5, serves as the foundation for providing comprehensive and coherent feedback. By populating the placeholders with the computed scores, we transform the raw numerical evaluations into well-structured, natural language feedback.

Figure: Algorithm 1 The self-reflection algorithm for optimizing our approach.

### 2.4 The Self-Reflection Algorithm

Our self-reflection approach can be formalized as an iterative optimization process, outlined in Algorithm 1. Given an input code, our approach enters a loop of trials (Lines 4-18 in Algorithm 1). In each trial, an ER is generated (Line 5), followed by the production of natural language feedback (Lines 7 and 8). These outputs are stored in memory and used as additional context in the subsequent trial (Line 9). The loop terminates when either the ER scores exceed a threshold, or the maximum number of trials is reached. Finally, the ER with the highest evaluation score is returned.

## 3 Empirical Study

Based on whether to add constraints on ERs, we design two settings: an open setting and a constrained setting. We explore generating ERs in two settings and analyze the generated ERs.

Generating ERs in the open setting (Section 3.1). We allow our approach to freely generate ERs without constraint. Through a manual inspection of the generated ERs, we have five findings about how LLMs understand the code.

Generating ERs in the constrained setting (Section 3.2). We constrain the ERs to be specific forms (i.e., natural language comments, pseudocode, and flowcharts), applying our approach to specific software engineering tasks. Based on the generated ERs, we have three findings showing the effectiveness and flexibility of our approach in supporting various software engineering tasks.

### 3.1 Generating ERs in the open setting

In the open setting, LLMs can freely represent code using their preferred forms. The language is the tool of thinking. We think that the forms LLMs represent the code can reflect how they understand the code. We analyze the generated ERs and have five findings. These findings reveal how LLMs understand different elements in code, e.g., grammatical rules, APIs, and numerical calculations.

#### 3.1.1 Study Design

Datasets. We select a popular code dataset - CoNaLa (Yin et al., ) as the experimental data. It comprises 500 Python code snippets sourced from the well-known programming Q&A platform, Stack Overflow. We consider these 500 code snippets as input data in our experiments.

Implementation Details. We select the State-Of-The-Art (SOTA) LLM for code - GPT-4o as the LLMs (i.e., $M_{CR}$ and $M_{RC}$) in our approach. Because our study involves time-consuming human inspection, we first study the SOTA LLM and leave other LLMs in future work. To prevent our approach from degenerating into a trivial copy model, we impose no constraints on the form of representations except to ensure that they are non-code. Specifically, we implement the $F(\cdot)$ as a chat with GPT-4o. We instruct GPT-4o to predict a probability score that a representation is non-code. The instruction is shown in Figure 6.

#### 3.1.2 Results and Analyse

We leverage our self-reflection approach to generate 500 ERs in the open setting. We select high-quality ERs whose equivalent score (Equation 5) and constraint score (Equation 2) both exceed 0.9. We manually analyze these ERs and yield the following findings:

(1) Structured ERs consist of the dictionary, flowcharts, tables, and XML. Typically, they decompose the code into different components (e.g., APIs, operators, and constants) and store them in structured forms. We show some examples of structured ERs as follows. The percentage in brackets indicates the proportion of the ERs in the dataset.

(1.1) Dictionary (33.48%), (1.2) Table (7.9%), and (1.3) XML (3.4%). They represent the code using a JSON-style dictionary, a markdown-style table, and an XML-style text. Each component of the code is mapped to a meaningful key, e.g., operation_kind and Atrtibute. Three examples are shown in the following listing.

(1.4) Flowchart (10.26%). An example is shown in the following listing. Each node in the flowchart denotes a variable (e.g., [Variable "my_float"]) or a operation (e.g., [Replace]). The edge (e.g., <-) between two nodes represents the control flow or data flow. For example, the output of [Cast "float"] is transmitted to [Variable "my_float"].

(2) Natural language ERs comprise paraphrased APIs, pseudocode, natural language comments, and SQL. These ERs often use natural language-style phrases or descriptions to represent in code. We show some examples as follows. The percentage in brackets indicates the proportion of the ERs in the dataset.

(2.1) Paraphrased APIs (30.62%). An example is shown in the following listing. It paraphrases the APIs in code using natural languages. The paraphrases can reflect the functionality of APIs. For example, it paraphrases the re.search into PATTERN_MATCH.

(2.2) Pseudocode (4.5%). An example is shown in the following listing. It converts the code into a well-formed pseudocode. It typically consists of an explanation of input-output arguments and a multi-step solving process in natural languages.

(2.3) SQL (2.7%). An example is shown in the following listing. It is a SQL-style query statement and uses natural language phrases to denote operations in code, such as Extract and contains.

(2.4) Natural Language Comment (0.87%). An example is shown in the following listing. It is a natural language text describing the purpose of code. Interestingly, despite the fact that natural language text makes up a significant portion of LLMs’ training data, LLMs rarely use it to describe code. This is an unexpected phenomenon.

(3) Mathematical ERs mainly contain arithmetic expressions. They leverage mathematical notations to represent the elements in code, such as APIs and computations.

(3.1) Arithmetic Expressions (2.2%). An example is shown in the following listing. It leverages mathematical notation and operators to represent numerical operations in code. For example, for i in L $\rightarrow$ $\forall i:i\in L)$, and sum(i) $\rightarrow$ $\sum(i)$.

(4) Others (3.2%). The remaining representations do not fit into the above categories and are therefore classified as “Others”. These representations typically contain uncommon symbols, as shown in the following example.

Or, the forms of representations are hard to understand:

Based on the above ERs, we have some findings about how LLMs understand the code.

LLMs generate many structured ERs (55.04%) in the open setting. In the dictionary and table, LLMs parse the code into different components and reason about their types. In the XML, LLMs even produce a plausible syntax tree. These cases show that LLMs view the code as a structured sequence instead of plain text. Thus, we suspect that LLMs can understand the grammatical structures of code. However, existing LLMs do not explicitly model grammatical structures during training. Therefore, how LLMs learn the structures remains unclear.

For example, in the example of pseudocode, LLMs reason about the API’s purpose (e.g., get current process ID) from its names (e.g., os.getpid). This process is akin to the process of reasoning the meaning of a phrase from its abbreviation. Similar phenomena are found in previous studies (Zhang et al., ). They found that LLMs can reason about the functionality of code based on a corrupted version.

In the example of arithmetic expressions, LLMs transform code into a mathematical expression. We hypothesize that LLMs leverage mathematical knowledge acquired from other data to aid in understanding code. Mathematical expressions serve to unify various numerical operations within the code, facilitating LLMs’ comprehension of underlying data flows.

LLMs generate different types of ERs based on the features of code. LLMs often generate arithmetic expressions when the code contains many numerical computations. When the code involves searching for elements, LLMs are prone to generate SQL-style ERs. This phenomenon reflects that LLMs can flexibly adopt different ways of thinking based on the characteristics of the code.

### 3.2 Generating ERs in the constrained setting

In this setting, we constrain the ERs to be specific forms ( e.g., natural language comments, pseudocode, and flowcharts), applying our approach to specific software engineering tasks (e.g., code comment generation). Through the generated ERs, we obtain three findings, showing the usability of our approach in real-world applications.

#### 3.2.1 Study Design

We reuse the 500 Python code snippets used in Section 3.1. We select GPT-4o as the LLMs in our approach. Besides, we try three constraints on ERs. First, we constrain the ERs to be a natural language comment. Second, we constrain the ERs to be a well-formed pseudocode. Third, we constrain the ERs to be a clear flowchart. To achieve the above constraints, we implement the $F(\cdot)$ as a chat with GPT-4o. We instruct GPT-4o to predict a probability score that an ER satisfies a specific constraint. The instructions for three constraints are shown in Figure 7, 8, and 9.

#### 3.2.2 Results and Analyse

Figure: Figure 2: The distributions of semantic-equivalent and constraint scores in three types of ERs. We show the distribution of averaged constraint scores.
Refer to caption: x2.png

As stated in Section 2.3, our approach scores the ERs in each trial. Figure 2 shows the distributions of final semantic-equivalent and constraint scores in generating three types of ERs. For comparison, we also show the distributions of scores in the first trial. We can see that the ERs in the first trial have lower scores and are often problematic. It may be due to the hallucinations from LLMs, where LLMs generate incorrect ERs though they can understand the code and instructions.

In contrast, the final ERs are substantially improved compared to the ERs in the first trial. We attribute the improvements to our self-reflection algorithm, which enables LLMs to iteratively refine the ERs based on natural language feedback. Figure 10 shows a real case in our experiments, where LLMs continually refine the ERs (i.e., pseudocode) by adding more details.

The following listing shows a natural language comment generated by our approach. The comment accurately describes the code’s purpose (always display warnings) and the implementation details (set the warning filter). It can help developers quickly understand the code and benefit software maintenance.

The following listing shows the generated pseudocode and flowchart. They decompose an intricate Python statement into several steps and clearly show the dependencies between these steps. By reading the pseudocode and flowchart, developers can know the control flow and data flow in code, facilitating code optimization (e.g., deleting dead code).

## 4 Related Work

Reflexion. Technically, our self-reflection approach is inspired by the Reflexion framework (Shinn et al., ). We follow it and leverage natural language feedback to ‘optimize’ LLMs instead of updating parameters.
In contrast to Reflexion, our approach focuses on generating ERs of code and introduces new effective designs (e.g., semantic-equivalent scores and external constraints). Based on the approach, we further conduct a large-scale study and summarize eight valuable findings.

## 5 Conclusion

This paper explores generating Equivalent Representations (ERs) of code by proposing a self-reflection approach. Based on whether to add constraints on ERs, our approach can generate ERs in open and constrained settings. Then, we explore generating ERs in both settings and summarize nine valuable findings. In the open setting, our findings reveal how LLMs understand the different elements in code, e.g., grammatical rules and APIs. In the constrained setting, we find that our approach can effectively support various software engineering tasks. Finally, we discuss the future directions of this work.

## Appendix A Appendix

### A.1 Instruction Templates

Figure 3, 4, 5, 6, 7, 8, and 9 show all instructions used in our approach and empirical study. The content in {…} (in blue) is a placeholder. Users can populate these placeholders with specific items during the inference.

Figure: Figure 3: The instruction for representation generation ($M_{CR}$) in our approach.

Figure: Figure 4: The instruction for code reconstruction ($M_{RC}$) in our approach.

Figure: Figure 5: The template of natural language feedback.

Figure: Figure 6: The instruction for evaluating whether a representation is non-code.

Figure: Figure 7: The instruction for evaluating whether a representation is a natural language comment.

Figure: Figure 8: The instruction for evaluating whether a representation is a pseudocode.

Figure: Figure 9: The instruction for evaluating whether a representation is a flowchart.

### A.2 Implementation Details of Semantic-equivalent Score

As stated in Section 2.3, we compute the similarity between the original code $c$ and reconstructed code $\hat{c}$ as the semantic-equivalent score. Specifically, we employ a hybrid similarity metric, which combines text similarity $Sim_{text}$ and syntax similarity $Sim_{syntax}$. In this section, we describe the computations of two types of similarities.

Text Similarity. Programs with similar functionalities often have similar text surfaces. Researchers have proposed many successful metrics to measure the text similarity between programs Papineni et al. (); Eghbali & Pradel (). Following existing studies Papineni et al. (), we split programs into token sequences and compute the $n$-gram similarity between two code sequences. Specifically,

$$ $\text{Sim}_{\text{text}}(c_{i},c_{j})=\exp\left(\sum_{n=1}^{4}\frac{1}{4}\log \frac{s^{n}_{ij}}{s^{n}_{j}}\right)$ (3) $$

where the maximum of $n$ is empirically set to 4. $s^{n}_{ij}$ is the number of overlapping $n$-grams between $c_{i}$ and $c_{j}$. $s^{n}_{j}$ means the number of $n$-grams in $c_{j}$.

Syntactic Similarity. The code is structured and can be parsed into tree structures, e.g., Concrete Syntax Trees (CSTs). Programs with different text surfaces may have similar syntactic structures. Following the related work Ren et al. (), we calculate the similarity in syntactic structures. Specifically, we parse the programs into CSTs and extract sub-trees from CSTs. Then, we compute the ratio of overlapping sub-trees between two programs. Formally,

$$ $\text{Sim}_{\text{syntax}}(c_{i},c_{j})=\frac{|\text{CST}(c_{i})\cap\text{CST} (c_{j})|}{|\text{CST}(c_{j})|}$ (4) $$

where $\text{CST}(\cdot)$ is a function for extracting all CST sub-trees of a program.

Finally, we compute a weighted sum of two similarity scores as follows:

$$ $\text{Score}_{semantic}=a*Sim_{text}(c_{o},c_{r})+b*Sim_{syntax}(c_{o},c_{r})$ (5) $$

where $a$ and $b$ are hyper-parameters. We empirically set them to 0.5 and 0.5.

### A.3 Examples of ERs

Figure: Figure 10: A real case showing the effectiveness of our self-reflection algorithm.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. For the proposed semantic-similarity score, what is the difference between other famous code similarity evaluation metrics like CodeBLEU and it? Why is it necessary to propose a new metric?
soundness: 2
strengths: 1. The paper is well-written and easy to follow.
2. Discussing ER in cod-related tasks is an interesting topic and remains largely underexplored.
summary: This paper conducts an empirical study to explore the ability of LLMs to generate Equivalent Representations (ER) of code. The authors employ the Reflexion framework to ask LLMs to generate ERs in both open and constrained settings and analyze the generated ERs. Overall, they find that LLMs have the ability to generate ERs for codes, which sheds light on the future exploration of this direction to support code-related tasks.
weaknesses: 1. In this paper, the authors stated that LLMs 'can' generate ERs for codes, which are generally cliches. Instead of studying whether LLMs are able to generate ERs, we are more interested in 'how well' LLMs can generate ERs. However, in this paper, there is no quantitative analysis of the correctness of the generated ERs. The semantic-equivalent score and constrained score can not reflect the correctness. The former only focus on semantic similarity, yet a similarity does not guarantee correctness. And the latter is generated by LLMs, which suffers from hallucination problems. The authors should manually check the correctness of generated ERs.
2. The findings lack in-depth analysis. In section 3, basically, there is only a case and a description of it for each finding.
3. The approach employed in this paper is nothing new compared with Relflexion, except for the designed scores for evaluating semantic similarities.
4. The future direction part lacks details and in-depth discussions. After reading this part, I am still confused about how ERs could actually get involved in the three discussed future directions.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - The paper only analyzes ERs for Python code. Does the analysis of different programming languages ​​show consistency? This may require further experimental evaluation.

- Can the authors elaborate on the impact of their findings on actual software development?
soundness: 2
strengths: 1. The paper is well-written, allowing readers to clearly understand the methodology, experiments, and corresponding discoveries.

2. The analysis on the CoNaLa dataset provides eight findings that offer insights into how LLMs comprehend code.

3. The availability of open-source code facilitates reproducibility. The code is well-structured, providing clear reproduction steps, data, and prompts.
summary: The authors propose a self-reflection approach utilizing iterative collaboration between two LLMs to generate equivalent representations (ERs) of code snippets. They investigate ER generation under both open and constrained settings. Similarity in text and syntax trees is used to measure program equivalence, leading to the identification and analysis of highly similar ERs, resulting in eight findings. The authors have open-sourced their experimental code and prompts.
weaknesses: 1. Novelty: The paper's innovation needs further clarification. The fields of NL2Code and Code2NL have been extensively researched (refer to [1, 2, 3]). What are the core differences between this work and existing studies?

2. Experimental Reliability: Evaluating program equivalence solely through text and syntax tree similarity might be unreliable. The constraint scores are assessed using LLMs; have the authors verified the accuracy and consistency of LLM scoring with human evaluations?

3. The analysis on the CoNaLa dataset yields eight findings, but their significance and guidance for future work are not discussed in the EMPIRICAL STUDY or FUTURE WORK section, suggesting the paper might be incomplete.

```
[1] Hu, Xing, et al. "Deep code comment generation." Proceedings of the 26th conference on program comprehension.2018.
[2] Li, Zheng, et al. "Setransformer: A transformer-based code semantic parser for code comment generation." IEEE Transactions on Reliability 72.1 (2022): 258-273.
[3] Zan, Daoguang, et al. "Large language models meet nl2code: A survey." arXiv preprint arXiv:2212.09420 (2022).
```

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please feel free to answer all of my doubts.
soundness: 2
strengths: + The idea of generating the ERs of code is somewhat interesting.

+ Many vivid examples are provided in the paper.
summary: This paper introduces a dual framework to generate the ERs of input code. Experiments demonstrate the effectiveness of the proposed approach and the authors also discuss several potential applicaitions into the software engineering tasks.
weaknesses: - Some doubts about the definitions. What is an ER? The paper merely lists several possibilities without an explixt definition. Also, I wonder can the pseudocode be considered as natural languages, as argued by the authors in the Abstract?

- Unclear about the state of the art in this direction. Is there any approach that aims at generating ERs for a code snippet? What is their limitation?

- Prompt in Figure 4 is not sound. Why statement? I guess it should be a code snippet.

- As for the semantic score, why not directly adopt existing metrics such as CodeBLEU?

- Extended from the above question, I am not convinced that using a metric is reasonable to assess the similarity between c and c'. I guess a code snippet can also have ERs in the form of code. As such, any metric could have brought bias since in this case, c and c' may have different syntactics. In light of this, why not use LLMs to judge code similarities? Why rely on a fixed metric?

- In the dual process, have you considered the inaccuracy brough by the LLMs, especially in the transforming of ERs into code. To what extent is this approach sound?

- Finding 5 is not well investigated. I wonder what features lead to different ERs? Does it mean code functionality or just syntactic features? Any evidence?

- Findings 7 & 8. IMHO, this is an overclaim to conclude them as findings. They are just single cases.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: - Is the code and ER not included in the prompt for getting NL feedback (Figure 5)?
- What was the average number of trials? What was the maximum number of trials? How was the threshold score determined?
- Is it possible for the type of ER to change across trials (i.e., in Trial 1 it is a comment, in Trial 2 it is pseudocode)? If so, how often does this happen?
soundness: 1
strengths: - While the dual framework for code isn't itself novel (https://arxiv.org/abs/2310.14053, https://arxiv.org/abs/2402.08699), the idea of translating the similarity metric to NL feedback and performing iterative refinement is quite novel and clever. The iterative approach also seems to help improve equivalence scores, based on Figure 2.
- I particularly liked the idea of scoring based on text similarity, syntactic similarity, and external constraint satisfaction. 
- There are many examples provided in the main paper, and the examples in the open setting demonstrate that the LLM is capable of generating many diverse types of ERs based on code.
summary: The authors propose a dual framework for producing Equivalent Representations (ERs) (e.g., NL comments, pseudocode, flowcharts) for code through multiple trials. In each trial, the first LLM call is used to generate an ER based on the input code and then a second LLM call is used to generate a new code snippet based on the ER. Then, the score is computed between the original code and the new code snippet (as a combination of text similarity, syntactic similarity, and external constraint-based satisfaction score derived from the LLM). This score is then translated into natural language feedback that is used in the subsequent trial to improve the ER. The authors perform this procedure for the 500 Python examples in the CoNaLa dataset and then do a qualitative analysis in two settings: open setting (no constraints on the ER that the model can produce), constrained setting (ER must be of a specific type, i.e., comment, pseudocode, flow chart).
weaknesses: - Many of the claims in this paper are not substantiated with empirical evidence. First, in L239-240: "We think that the forms LLMs represent the code can reflect how they understand the code." The authors claim the findings in the open setting "reveal how LLMs understand different elements in code, e.g., grammatical rules, APIs, and numerical calculations." This claim is not supported with any evidence. In fact, just because a model is capable of generating diverse types of ERs doesn't necessarily say anything about whether or not it understands code in that form or that it reasons in that form. For making such a claim, the authors should have run experiments to compare performance when the input or intermediate CoT reasoning chain leverages these different types of ERs. The authors also claim, "From the structured ERs, we find that LLMs treat the code as a structured sequence instead of plain text. LLMs even can produce a plausible syntax tree." However, if the input was something other than code like general text and you still ask it to transform it into some new representation, there's a good chance that it will also be structured. This claim is not well-supported.
- Next, Finding 6 is "Our approach effectively generates ERs for software engineering tasks and reduces hallucinations generated by LLMs." This seems to be based solely on a speculation in L468-470: "We can see that the ERs in the first trial have lower scores and are often problematic. It may be due to the hallucinations from LLMs, where LLMs generate incorrect ERs though they can understand the code instructions." The authors do not provide any evidence of hallucination. In fact, this could also be due to a number of other factors. For instance, it's possible that the LLM didn't capture \textit{all} of the information in the first trial and later added more information. Or that the phrasing or structure of the ER in the first trial was not meaningful to the LLM and it just needed some simple refactoring.
- In Findings 7 and 8, the authors claim that their approach is useful to software developers; however, they do not actually perform any user study to confirm this.
I find that this paper lacks key empirical results. The authors have reported their observations from performing qualitative analysis themselves, but I feel that these are not convincing on their own.
- The authors have only included 1 paper in the Related Work section. I feel that a more thorough literature review is needed. Particularly, there are other papers that use a dual framework for code (https://arxiv.org/abs/2310.14053, https://arxiv.org/abs/2402.08699).

Suggestion:
- The conclusion suggests that future work is presented in detail. However, this is only explained in the introduction. Also, highlighting future work so extensively is not needed in the abstract or introduction. It is better to focus on the approach itself.

Meta-review: This paper presents an empirical study investigating the ability of large language models (LLMs) to generate Equivalent Representations (ERs) of code. Using the Reflexion framework, the authors prompt LLMs to generate ERs in both open and constrained scenarios and analyze the resulting representations. Their findings suggest that LLMs are capable of producing ERs, providing potentials for further exploration of ER generation to support various code-related tasks.

The motivation for focusing on ER generation is not clearly articulated, and while potential implications are discussed, they remain somewhat speculative. Additionally, the evaluation metrics—semantic-equivalent score and constrained score—do not necessarily capture the functional correctness of the generated ERs. This leaves uncertainty about whether the generated representations are truly equivalent in practice.

Decision: Reject