CodeFusion: A Pre-trained Diffusion Model for Code Generation
Authors: Mukul Singh
José Cambronero
Sumit Gulwani
Vu Le
Carina Suzana Negreanu
Gust Verbruggen
Venue: EMNLP 2023 Main (2023)
Keywords: Text-to-code generation
Diffusion models
Program synthesis
Language models
TL;DR: A pre-trained diffusion code generation model that employs an attention-based decoding strategy and a mask-denoising objective to generate code.

Abstract: Imagine a developer who can only change their last line of code—how often would they have to start writing a function from scratch before it is correct? Auto-regressive models for code generation from natural language have a similar limitation: they do not easily allow reconsidering earlier tokens generated. We introduce CodeFusion, a pre-trained diffusion code generation model that addresses this limitation by iteratively denoising a complete program conditioned on the encoded natural language. We evaluate CodeFusion on the task of natural language to code generation for Bash, Python, and Microsoft Excel conditional formatting (CF) rules. Experiments show that CodeFusion (75M parameters) performs on par with state-of-the-art auto-regressive systems (350M-175B parameters) in top-1 accuracy and outperforms them in top-3 and top-5 accuracy due to its better balance in diversity versus quality.

CODEFUSION: A Pre-trained Diffusion Model for Code Generation
Mukul Singh
José Cambronero Sumit Gulwani
Carina Negreanu Microsoft Research Cambridge, UK
Gust Verbruggen Microsoft Keerbergen, Belgium
Microsoft Delhi, India
Vu Le Microsoft Redmond, US
Abstract
approaches then select the vocabulary token with the closest embedding. In the code domain, where there are many syntactic and semantic constraints between tokens, independently projecting embed- dings back to tokens can yield invalid programs.
Imagine developer who can only change their last line of code-how often would they have to start writing function from scratch before it correct? regrreegeesivve models for code generation from natural language have similar limitation: they do not easily allow reconsid- ering earlier tokens generated. We introduce CODEFUSION, pre-trained diffusion code gen- cration model that addresses this limitation by iteratively denoising complete program con- ditioned on the encoded natural language. We evaluate CODEFUSION on task natural language code generation for Bash, Python, and Microsoft Excel conditional formatting (CF) rules. Experiments show that CODEFU- SION (75M parameters) performs par with wate-of-the-ar auto-regressive systems (350M- 175B parameters) in top-1 accuracy and outper- forms them in top-3 and top-5 accuracy. due to better balance diversity versus quality.
We propose CODEFUSION, natural language to code (NL-to-codee model that combines an encoder-decoder architecture (Raffel et al., 2020) with a diffusion process. The encoder maps the NL into continuous representation, which is used by the diffusion model an additional condition for denoising random Gaussian noise input. To generate syntactically correct code, then feed the denoised embeddings transformer decoder, with full self-attention and cross attention with the embedded utterance, to obtain probability distribu- tions over code tokens. Finally, we seleet the token with the highest probability at each index.
To pre-train CODEFUSION for code generation, we extend the continuous paragraph denoising (CPD) task introduced Lin et al. (2023) to the code domain. Specifically, we only apply noise to tokens that correspond to identifiers in code or to built-in keywords in the target language. This denoising task allows the model to learn relations between critical code tokens (like variable names, function names and control flow built-ins).
1 Introduction
Auto-regressive code generation models (Wang et al., 2021; Brown et al., 2020; Scholak et al., 2021; Feng et al., 2020; Fried et al., 2022) can- not easily reconsider tokens generated earlier in the decoding process. This limitation can lead to lower diversity generations (Lin et al., 2023) in the related domain of text. To balance diversity and quality of candidates generated, prior work has explored decoding strategies such as grouped beam search (Vijayakumar al., 2018) or nucleus sampling (Holtzman et al., 2019).
We find that CODEFUSION yields more diverse code (higher n-gram fraction, lower embedding similarity, and higher edit distance) than auto- regressive models (see Table 2). The CPD objec- tive, which biases the model towards learning remove noise in context-aware fashion, paired with a decoder that has access the full denoised representation, jointly lead CODEFUSION to pro- duce 48.5% more syntactically correct generations (averaged over three languages) when compared to GENIE, text diffusion model (Table 3).
Diffusion models, which have shown remark- able performance in image generation (Dhariwal and Nichol, 2021), have recently been extended to generate diverse text (Li et al., 2022; Lin et al., 2023). These approaches use embedding layer convert discrete tokens continuous embed- dings, where Gaussian noise can be added and pre- dicted, to imitate the diffusion process. To map denoised embeddings back to discrete text, these
We evaluate CODFFUSION on NL--to-code for three different languages: Python (Yin et al., 2018) 8), Bash (Lin et al., 201 10, and conditional formatting

---

rules in Microsoft Excel (Singh al., 2022). Our results show that CODEFUSION's (75M parameters) top-1 results are comparable or better than much larger state-of-the-aae systems (350M-175B param- eters). In top-3 and top-5, CODEFUSION performs better than all baselines.
3.1 Architecture
The input to CODEFUSION utterance = {s1, 82, ... .8k] and the output a predicted code snippet y = {ū .2. Us Both input and output are padded to fixed dimension n.. CODEFUSION has three main wansformer-based components (an encoder E, denoiser N, de- coder D) and a classification head H.
natural language
This work makes the following contributions:
1. We propose CODEFUSION, the first diffusion- based NL-to-code model.
The transformer based encoder transforms the tokenized utterance into a vector representation E, = E(s) = {er, E2, ,,,
2. We adapt continuous paragraph denoising (CPD) code and show that substantially improves the results of CODEFUSION.
Conditioned on the encoded utterance E, and the time t, the denoiser (N) predicts and removes noise 6, from the noisy program embedding æf to obtain a predicted denoised program embedding * = N(a,,, (a'; t, F.)) N is transformer block with cross-attention between a' and E, and full self- attention over x!.
3. We compare CODEFUSION to rugo-regres code models and text diffusion models on the NL-to-code task in three languages.
2 Related Work
Before projecting the denoised embeddings back to discrete code tokens, we use a decoder (D), this time applying cross-attention to z and E,, with full self-attention over é to compute final hidden representation Ds {d1, d2.... ,dn} D(x", E.). As opposed prior text diffusion approaches, where tokens are generated indepen- dently, full self-attention allows each hidden di- mension (d,) be generated with full information about the other dimensions.
NL-to-code a popular and challenging sequence to sequence problem. Previous techniques for NL- to-code have used RNNs (Brunner and Stockinger, 2021) and semantic parsers (Lee et al., 2021) I). Attention-based transformer models (Vaswani et al., 2017) have been adapted for the code domain, and include CodeBERT (Feng al., 2020) (en- coder only), T5 (Raffel et al., 2020) (encoder- decoder) and GPT-3 (Brown al., 2020) (decoder only). More recently, large scale transformer-base models like CodeGen (Nijkamp et al.. 2023) and TransCoder (Sun et al., 2023) have shown promis- ing results. Instruction-tuning has also been used further improve performance with models like StarCoder (Li al., 2023a) and WizardCoder (Luo et al., 2023). These models can decode in an ruto-regressive and ren-auvo-regressive fash- ion (Su et al., 2021). Recent surveys have dis- cussed these models in more detail (Xu and Zhu, 2022; Niu et al., 2023; Xu et al., 2022).
Finally, D. is projected to actual code tokens with a classification head H that computes dis- tribution over code tokens p(y|d;). We do not perform search over these tokens and select û arg max, p(y)d:) for each i.
3.2 Training
We train CODEFUSION in two phases: unsupervised pre-training of the denoiser and decoder on code snippets, and supervised fine-tuning of encoder. denoiser and decoder on (utterance, code snippet) pairs. Following prior work on diffusion for text, we use trainable embedding layer L to embed a code snippet into a continuous space where we can add (and remove) noise e, timestep
Diffusion models have been popularly applied unsupervised (Dhariwal and Nichol, 2021) and text conditioned image generation tasks (Saharia al., 2022a,b; Ramesh et al., 2021). More recently, similar approaches have been extended to work in discrete domains such as text (Li et al., 2022; Lin et al., 2023; Li et al., 2023b; Reid et al., 2023).
We take inspiration from prior work on diffusion for text and adapt the loss from GENIE (Lin et al., 2023) to CODEFUSIÓN by incorporating the hidden representation D, from the decoder. At time step f, the loss is computed as
3 Methodology
Figure shows CODEFUSION'' architecture. This section describes each component and our training and inference procedures.
L. 12 ael DD L(v)l log logp(y)
and consists of three parts.

---

x
Decoder (D)
E
Encoder (E)
Denoiser (N)
Classification Head (H)
Query:
ANDCTextEqual (ML ibbary", "Yes"), TextEquals(*Sportt) "Yes"), TextEquals(*Lab* "Yes"))
"You have a 1ist of final year student. HighLight the students who get clear from all departments"
Gaussian Noise (x99
Figure I: Architecture diagram CODEFUSION showing the Encoder (E), Denoiser (N) the Decoder (D) units.
1. We minimize the eтror between the predicted noise & and the actual noise e to train N.
Evaluation Setup
We briefly describe training. baselines, benchmarks and metrics. We provide further details for training and baselines the Appendix.
2. We minimize the error between the D3 and embedded code train D and
4.1 Benchmarks
3. We apply a standard cross-entropy loss over the outputs of the classification head, which produces predicted code tokens given Da, and the ground truth code snippet
We evaluate CODEFUSION on NL-to-code for lan- guages with varying complexity: Python, Bash, and conditional formatting (CF) rules in Microsoft Excel. The CoNaLa dataset (Yin et al., 2018) for Python consists of complex, multi-statement StackOverflow code snippets and associated NL questions. The Bash dataset (Lin et al., 2018) has complex, single-line Bash commands annotated with NL descriptions. The CF dataset (Singh et al., 2022) consists of Excel CF rules, which are single line programs of low complexity, annotated with NL. These benchmarks (and pre-training data- next section) are made publicly available."
The loss function allows train the three main components of our model (denoiser, decoder and classification head) with a diffusion objective.
To pre-train the denoiser (N) and decoder (D) over corpus of code snippets, we use two tasks: unsupervised code generation and our adaptation of continuous paragraph denoising (CPD) (Lin et al.. 2023) for code. This code-specific CPD task only masks tokens associated with identifiers or built-in keywords from the target language. We randomly sample from these two tasks during pre-training.
4.2 Training
For our experiments, we instantiate the encoder (E) as pre-trained CodeT5 encoder (Wang et al..
Both pre-training and fine-tuning tasks use Lt- Because there is no natural language utterance in pre-training, there is no input Fg to the denoiser N. In the unsupervised code generation task, E. is replaced with Gaussian noise sampled every de- noising time step. In the CPD task, E, is computed by passing the masked code y through encoder E.
2021) (embedding dimension is 512), the denoiser (N) as 10 layer transformer block, the decoder (D) 6 transformer decoder layers, and the classi- fication head (II) as single fully connected layer. In the training and pre-training phase, we use square root noise schedule with 120 diffusion steps (Wu et al., 2023). We use the tokenizer and vocabulary from CodeT5 (Wang et al., 2021) and target code length of 128 tokens. We use AdamW optimizer without weight decay (Loshchilov and Hutter, 2019) and learning rate of 5e-4.
3.3 Inference
During inference, we initialize x' with Gaus- sian noise and iteratively remove (scheduler- determined) proportion of the noise over T time steps to obtain - (Ho al., 2020). During this it- erative denoising, we do not use the decoder. After this iterative procedure, the decoder produces the final predicted code ÿ. We post-process ý to select the tokens up to the first pad token.
We pre-train the diffusion and decoder model on code snippets only. For Excel, we use a public corpus 450K conditional formatting rules (Singh et al., 2022). For Python and Bash, we scrape
'https: //gi thub. com/mi http://ccroooo/////oo prose- benchmarks Seaachmarrks/trree/mii///ddddiid

---

Table Comparison of CODEFUSION with baselines on the task of NL to code generation for Python, Bash and CF rules. We report top-1, top-3 and top-5 predictions. Model denotes the underlying base model's checkpoint name. #P denotes the number of model parameters. We note the metric used for each language in parentheses.
System description
Python (CodeBERT)
Bash (template)
CF Rule (exccution)
System
Model
#P
top-1
top-3
top-5
top-1
top-3
top-5
top-1
top-3
top-5
TS
15-large
770M
80.4
82.3
84.8
67.1
68.9
70.3
71.1
73.4
74.6
CodeT5
codet5-large
770M
80.5
83.1
85.0
67.6
69.3
70.5
72.7
75.3
75.8
GPT-3
text-davinci-003
175B
82.5
83.7
85.8
66.9
67.7
68.4
70.3
72.4
72.8
ChatGPT
gpt-3,5-turbo
Unknown
80.6
82.5
83,9
66.1
66.9
67.8
70.8
73.1
74.5
StarCoder
starcoder
15,58
79.2
82.0
84.1
64.5
65.3
66.5
70.6
72.8
74.5
CodeTS+
codet5p-16b 16b
16B
79.6
82.1
84.5
65.7
66.1
67.2
70.5
72.9
74.3
CodeGen
codegen-350m
350M
80.1
81.8
83.7
67.2
69.2
70.3
71.4
73.7
75.0
Diffusion-LM
Custom
50M
70.4
74.3
76.5
59.4
61.6
62.0
62.4
65.5
68.2
GENIE
Custom
93M
73.2
77.1
80.3
60.0
61.5
62.3
62.9
66.8
68.7
CODEFUSION
Custom
75M
80.7
86.3
90.3
66.7
70.2
72.0
72.8
76.7
78.5
GitHub notebooks and StackOverflow posts with tags python, bash and powershell, using regex extractor to detect code (Lin et al., 2018).
string edit distances (Perlitz et al., 2023).
5 Evaluation
4.3 Baselines
We investigate the following questions. Q1. Does CODEFUSION generate correct and diverse code? Q2. How do different design decision impact per- formance? Q3. How does the latent representation evolve during the diffusion steps?
We use combination of transformer and text diffu- sion models baselines: T5 (Raffel et al., 2020), GPT-3 (text-davinei-O0)) (Brown al., 2020),
ChatGPT (gpt-3.5-turbo) (OpenAI, 2023), CodeT5 (Wang ct al., 2021), StarCoder (Li et al., 2023a), CodeTT+ (Wang et al., 2023), CodeGen (Nijkamp et al., 2023), Diffusion-LM (Li et al., 2022), and GENIE (Lin et al., 2023). Out of these GPT-3, ChatGPT, StarCoder, CodcT5+ used in in- context learning setting with five examples dynam- ically selected using BERT (Reimers and Gurevych, 2019) similarity of NL utterance, the others are fine-tuned on our datasets. Diffusion-LM and GENIE are also pre-trained on our corpus.
5.1 Performance and Diversity (Q1)
Table 1 summarizes performance in top-1, top-3 and top-5 settings for CODEFUSION and baselines. In top-1, CODEFUSION performs on par with or even better than (much larger) ruto-regressive models. For Python, only GPT-3 (175B) performs better than CODEFUSION (75M), In top-3 and top- 5, CODEFUSION outperforms all basclines, consis- tent with previous observations that auto-regressive models with high top-1 performance sacrifice di- versity in their generations (Poesia al., 2022).
4.4 Metrics
We evaluate Bash generation using the tem- plate match metric-which performs some basic cummalizaiion-provviii
Table 2 shows diversity results averaged across all benchmark tasks, over the top-5 generations for each model, for CODEFUSION and auto-regressive (T5, CodeT5, StarCoder, CodeGen, GPT-3) base- lines. CODEFUSION produces generations of higher diversity compared to auto-regressive models.
with the dataset. We eval- uate Python using CodeBERTScore (Zhou al., 2023), which has been shown to be high quality non-execution-bane based code matching metric. We evaluate CF using execution match (Singh et al., 2022) by executing a rule on the data column and comparing to the expected output.
Like CODEFUSION, other diffusion methods (Diffusion-LMM and geNIE) improve for top-3 and top-5 relative to top- They fall short of CODEFU- SION result of generating syntactically invalid programs. Table shows the fraction of syntacti- cally valid generations for CODEFUSION and dif- fusion baselines. CODEFUSION generations are more often syntactically valid compared to diffu- sion models not designed for code: 33.8% more
To evaluate generation diversity, measure (1) count of distinct token n-grams in the generated code divided by number of tokens (Vijayakumar al., 2018), (2) summary statistics over pairwise similaritics of CodeBERT encodings (Perlitz et al., 2023), and (3) summary statistics over pairwise

---

Table 2: Comparison of diversity in top-5 code generations for CODEFUSION and baselines for Python, Bash and CF rules. We report fraction of distinct token-level n-grams, pairwise similarities of CodeBERT embedding of outputs, and statistics over pairwise string normalized edit distance of outputs.
Models
N-grams better)
Embedding similarity ( hetter)
Edit distance (f hetter)
2
3
4
Min
Max
Mean
Min
Max
Mean
T5
0.29
0.45
0.54
0.6
0.93
0.99
0.97
0.08
0.58
0.36
T5
0.27
0.41
0.46
0.51
0.94
0.99
0.97
0.08
0.56
0.32
GPT-3
0.24
0.36
0.42
0,48
0.97
0.99
0.99
0.00
0.28
0.21
StarCoder
0.32
0,48
0.56
0.61
0.89
0.98
0.96
0.11
0.61
0.39
CodeGen
0.28
0.41
0.49
0.55
0.94
0.99
0.97
0.09
0.52
0.34
CODEFUSION
0.53
0.65
0.74
0.81
0.83
0.97
0.91
0.21
0.83
0.57
Table 3: % of top-1 generations that syntactically- valid for CODEFUSION and text diffusion based base- lines. CODEFUSION generates more valid candidates.
Normalized Edit Distance
100
Python
CF Rule
70
Bash
System
Python
Bash
CF Rules
30
Diffusion-LMM
19.5
40.4
74.3
GENIE
24.2
54.2
78.6
CODEFUSION 67.6
73.4
94.5
1200 1100 1000
900
A0n
700
600
400
300
200
100
Diffission Timestep
(1)
versus Diffusion-LM and 26.2% more versus GE- NIE averaged across all three languages.
Figure 2: Average normalized edit distance for CODEFU- generations against increasing diffusion timesteps.
5.2
Ablations (Q2)
ML:
Eopy the content of file "file.tst tatt to fill "filed.txt" souttll PP12 'F1262. txt")
Target:
Table-4 shows the results of CODEFUSION with various changes. Removing either pre-training task significantly reduces performance (-10.9% for code generation and -4.6% for CPD on average across the three languages). Results by replacing D and H with grounding (pick closest vocabulary token last denoising step) or clamping (pick closest vocabulary token at each denoising step) highlights the benefit of using a decoder before rounding.
Sa ae betwaee tieffll 164 Losation copyright right pan they destinet
t- 1200
Path caret coree adapter haward diterating éface lipss ded
1000
filefiletxt, filofiletxt quoryResult Screes
500
destucll., wwpyInt//fleeeeettt...... Fillext', "filetxt') return "Filetxe'
destuctl. copyyin fsle. (#12.txt")
copylnt fileutxe', (file.txe")
200
stutil.copy( Fxlee "eilez.txt')
1+0
shutil copy('flle.tx''s 'File2.txx")
Figure 3: Successive stages of denoising by CODEFU-
SION on example from the Python benchmarks.
Table Ablations for CODEFUSION (M) means the component is removed. Full model beats all ablations.
that the edit distance decreases with t. The drop is much faster for CF rules as these are simpler to generate than full programs and bash commands. An example visualizing this is shown in Figure 3.
Model
Python
Bash
CF Rule
Code Generation
70.9
52.3
64.2
CPD Objective
76.7
61.1
68.2
with Grounding
73.5
60.5
63.2
Conclusion
M with Clamping
77.1
63.2
64.4
M (CODEFUSION)
80.7
66.7
72.8
We propose CODEFUSION, the first diffusion nat- ural language to code (NL-to-code) generation model. With decoder and code-focused pre- training, CODEFUSION generates more syntacti- cally correct programs than existing text diffusion models and more diverse programs than existing auto-regressive code models. Our evaluation shows that CODEFUSION competes with state-of-the-a transformer code generation models on Python, Bash and Excel conditional formatting rules.
5.3 Gradual Refinement (Q3)
We study how CODEFUSION gradually reached the final result. For this experiment, we stop the denois- ing timestep
[0, T). and generate code snippet for the current state. We measure the nor- malized string edit distance obtained at cach time step (in increments of 100 steps). Figure shows

---

7 Limitations
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint (rXiv:1904.0977
CODEFUSION is not a global system as we only consider natural language utterances in English. Furthermore, natural language specifications can be provided at varying levels of detail utterances with less detail may result in worse performance. We consider various programming languages, but more complex languages may result in worse per- formance. We also find that CODEFUSION strug- gles when tasked with generating longer code snip- pets or programs that have long-ranged syntactic dependencies. Because diffusion- based models de- noise iteratively, inference latency is substantial, rising quadratically with target generation length.
Celine Lec, Justin Gottschlich, and Dan Roth. 2021. Toward code generation: A survey and lessons from semantic parsing. arXiv preprint arXiv:1005.03317
Raymond Li, Loubna Ben Allal. Yangtian Zi, Niklas Muennighoft, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhski,, Terry Yue Zhuo. Thomas Wang. Olivier Dehaene. Mishig Davaadorj. Joel Lamy-Poirien João Monteiro, Olch Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang. Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov. Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger. Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavit, Dan- ish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis. Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries, 2023a. Starcoder: may the source be with you!
References
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajami, Percy Liang, and Tatsunori Hashimoto. 2022. Diffusion- LM improves controllable text generation. Ad- vances in Newral Information Processing Systems.
Ursin Brunner and Kurt Stockinger. 2021. Valuenet: natural language-to sol system that learns from database information. In 2021 IEEE 37th Inter- national Conference on Data Engineering (ICDE), pages 2177-2182.
Yifan Li, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. 2023b. Diffusion models for non-autoregressive text generation: A survey.
Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer. and Michael D. Ernst. 2018. NI2bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation LREC 2018, Miyazaki (Japan), 7-12 May, 2018
Prafulla Dhariwal and Alexander Quinn Nichol. 2021. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng. Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- BERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536-1547, Online. Association for Computational Linguistics.
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. 2023. Text generation with diffusion language mod- els: pre-training approach with continuous para- graph denoise.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xi-
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. De- noising diffusion probabilistic models. Advances neural information processing systems, 33:6840- 6851.
ubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol- instruct.

---

Erik Nijkamp, Bo Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis.
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau. 2021. PICARD:
Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 9895-9901, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Changan Niu, Chuanyi Li, Vincent Ng. Dongxiao Chen, Jidong Ge, and Bin Luo. 2023. An empirical com- parison of pre-traeiie vre-trained models of source code. In 2023 TEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2136-2148,
Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, Mohammad Raza, and Gust Ver- bruggen. 2022. Cornet: A neurosymbolic approach to learning conditional table formatting rules by ex- ample. arXiv preprint arXiv:2208.00 06032.
OpenAl. 2023, Chat-gpt. Last-accesed: Oct 2023.
Yotam Perlitz, Liat Fin-Dor, Dafna Sheinwald, Noam Slonim, and Michal Shmueli-Scheuer. 2023, Diver- sity enhanced table-to-text generation via type con- trol.
Yixuan Su, Deng Cai, Yan Wang, David Vandyke, Si- mon Baker, Piji Li, and Nigel Collier. 2021. Non- autoregressive text generation with pre-trained lan- guage models.
Gabriel Poesia, Oleksandr Polozov, Le, Ashish Ti- wari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code gen- eration from pre-trained language models. ArXiv, abs/2201.11222
Qiushi Sun, Nuo Chen, Jianing Wang, Xiang Li, and Ming Gao. 2023. Transcoder: Towards unified trans- ferable code representation learning inspired by hu- man skills.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Tlya Sutskever. 2021. Zero-shot text-to-image gener- ation.
Ashwin K Vijayakumar, Michael Cogswell, Ram- prasath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search: Decoding diverse solutions from neural se- quence models.
Machel Reid, Vincent Josua Hellendoomn, and Graham Neubig. 2023. Diffuser: Diffusion via edit-based reconstruction. In The Eleventh International Con- ference Learning Representations.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. Codet5+: Open code large language models for code understanding and generation.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 9982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Yue Wang, Weishi Wang. Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware unified pre- trained encoder-decoder models for code understand- ing and generation. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes. Tim Salimans, Jonathan Ho, David J Flect, and Mohammad Norouzi. 2022a. Pho- torealistic exx-to-image diffusion models with deep language understanding.
Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, Zhongyu Wei, Jian Guo, Nan Duan, and Weizhu Chen. 2023. Ar-diffusion: Auto-regressive diffusion model for text generation.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Scyed Kam- yar Seyed Ghasemipour, Raphael Gontjo-Lopes. Bureu Karagol Ayan, Tim Salimans, Jonathan Ho. David J. Fleet, and Mohammad Norouzi. 2022h. Pho- torealistic text-to-image diffusion models with deep language understanding. In Advances in Neurul In- formation Processing Systems.
Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Jo- sua Hellendoorn. 2022. A systematic evaluation of large language models code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Ma- chine Programming, MAPS 2022, page 1-10, New York, NY. USA. Association for Computing Machin- ery.

---

Yichen Xu and Yangiao Zhu. 2022. A survey on pre- trained language models for neural code intelligence.
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to mine aligned code and natural language pairs from stack overflow. In International Conference on Min- ing Software Repositories, MSR, pages 476-486. ACM.
Shuyan Zhou, Uri Alon, Sumit Agarwal, and Gra- ham Neubig. 2023. Codebertscore: Evaluating code generation with pretrained models of code. arXiv preprint 25X.::3302.0552..

---

A
Implementation
optimizer le 4 learning rate for 100 epochs. We find that CodeT5 performs better than T5. This due to CodeT5 being pre-trained for code gen- eration. These are trained on the train split the datasets and evaluated on the test split. For decod- ing, we use beam search with beam size of 5.
We first describe the environment used to conduct experiments and then describe parameters for each component and training in detail.
A.1 Hardware Specifications
All experiments and studies have been carried out on Python software (version 3.8.7). The system used run the experiments uses an Intel Core 17 processor (base at 1.8 GHz) along with 4 V 100 GPU units, a 64-bit operating system, and 56 GB RAM. CODEFUSION took 8 hours to pre-train and 3 hours to fine-tune average for each dataset.
B.2 GPT-3, ChatGPT, StarCoder and CodeT5+
We use the pre-trained version of GPT-3 (Text DaVinci 003), ChatGPT (gpt-3.5-turbo,, StarCoder (bigcode/starcoodr)) and CodeTT++ (codet5p-18) 6b) models without fine-tuning in few shots setting. We use 5 examples in the prompt. The few-shot examples are selected from the train- ing corpus based on cosine similarity of sentence- BERT (Reimers and Gurevych, 2019) embedding of the query. For GPT-3 and ChatGPT, we experi- ment with temperature values between to with step of 0.1 and report the best result with all other parameters at default value. For StarCoder and CodeT5+ we use beam search with beam size 5.
A.2 Training Parameters
In the training and pre-training phase, square root noise schedule with 1200 diffusion steps (Wu al., 2023). We use the tokenizer and vocabulary from CodeT5 (Wang et al., 2021). We use a learning rate of 5e-4m with batch size of 64 and a target length of 128. Further, we use Adam W optimizer without weight decay (Loshchilov and Hutter, 2019).
B.3 GENIE
A.3 Pre-training Data
We implement GENTE as described in (Lin et al., 2023). We set the diffusion timestep T 1200, embedding dimension 256 and encoding and generation length 128. We choose these parame- ters consistent with CODEFUSION. We also pretrain on the same corpus used to pretrain CODE- FUSION. For sampling top-k candidates, we use the same inference algorithm highlighted by authors in (Lin et al., 2023).
Table shows the breakdown of the number of sam- ples used for pre-training along with their average length terms of number of tokens.
Table 5: Summary dataset used to pre-train CODE- FUSION. We report the number of samples and average length of each code snippet for all languages.
Language
Samples
Avg Length
Python
56K
78.4
B.4 Diffusion-LM
Bash
35K
45.3
Conditional Formatting Rules
100K
23.4
We implement Diffusion-LM as described (Li et al., 2022). We set the diffusion timestep T = 1200, embedding dimension to 256 and encoding and generation length to 128, similar to GeNIE. We use the controlled generation classifier to convert the text-to-code generation to completion task. We simply extend the infilling task, the authors propose for text generation (Li et al., 2022).
B Baselines
We describe imeplementation details of baseline systems.
B.1 T5, CodeT5 and CodeGen
T5 is the base model of CODEFUSION and uses an encoder-decoder architecture. CodeT5 uses T5 as the base model and pre-trains it code genera- tion and understanding tasks. We train both T5 and CodeT5 generate code from the input utter- ance. The input to T5 is same as that of CODEFU- SION fine-tuning (section-3.2), The models are op- timized on standard cross-entrope entsopy loss with Adam
C Further Results
In this section we describe additional results and examples from our corpus.
C.1 Exact Match Results
We also show the exact match accuracy for CODE- FUSION and baselines on the benchmarks. Table 6

---

Table 6: Comparison of CODEFUSION with baselines on the task of text to code generation for Python, Bash and CF rules. We report top-1, top-3 and top-S exact code match of the predictions. "Model" column denotes the underlying base model's checkpoint name. #P denotes the number of model parameters.
System description
Python
Bash
CF Rule
System
Model
#P
top-1
top-3
top-5
top-1
top-3
top-5
top-1
top-3
top-5
TS
15-large
770M
5.2
6.1
6.7
13.5
14.7
15.2
63.1
65.2
67.6
CodeT5
codet5-large
770M
5.5
6.4
7.1
14.1
14.9
15.5
63.2
65.7
67.8
GPT-3
text-davinci-003
175B
7.5
8.2
8.8
12.9
13.7
14.4
60.8
61.4
62.7
ChatGPT
gpt-3,5-turbo
Unknown
5.6
6.2
6,5
12.0
12.6
12.9
62.9
65.0
67.6
StarCoder
starcoder
15,58
4.8
6.0
6.4
12.1
12.5
13.4
62.6
64.8
66.9
CodeTS+
codet5p-16b 16b
16B
4.9
6.1
6.6
12.3
12.8
13.9
62.8
64,9
67.0
CodeGen
codegen-350m
350M
5.0
5.9
6.3
13.6
14.9
15.2
63.3
65.4
67.7
DiffusionL.M
Custom
50M
1.4
2.3
2.8
7,4
8.6
9.0
48.8
50.4
53.1
GENIE
Custom
93M
1.7
2.5
3,0
8.0
9.5
10.3
49.5
52.4
54.6
CODEFUSION
Custom
75M
5.1
7.2
9.0
13.5
15.3
16.4
63.4
67.6
69.1
NL:
Copy the content of file 'file.txt' to file 'file2.txt'
Target:
shutil.copy( ('fileeex't,' 'file2.txt')
Ss between Useful 144 Location copyright pen they destinat Path caret corev adapter NameAnd Giterating Gface 6pass 6FH
t= 1200
t= 1000
(.* for copyInt filefiletxt, filefiletxt queryResult SE Screen
t 800
destutil. filliitt 'filetxt') return 'filetxt'
600
destutil.copyIntt filleeee file.
'fi2.txt')
400
shutil.copyInt txt', 'file.txt')
200
shutil.copy('fiie..tx 'file2.txt')
t=
(ffiille 'file2.txt')
Figure Figure showing the various stages denoising CODEFUSION on an example from the Python benchmarks where CODEFUSION succeeds. CODEFUSION starts from pure noise and gradually denoises to generate the target code.
shows these results. CODEFUSION performs com- parable transformer based models and better than other diffusion based text generation approaches for top-1 accuracy. For top-3 accuracy, CODEFU- SION outperforms all baselines. This is consistent with the results in Table and show that CODEFU- SION produces better and more diverse candidate programs for variety of tasks.
and the reconstructed output from CODEFUSION at timesteps {200, 400, 600, 800, 1000, 1200). We can see how CODEFUSION gradually denoises and generates the correct code. Figure shows a failure case where CODEFUSION unable to generate the correct code. The user asks to re- move the directory tree "foldernname". naaennaaaeee.mmmm CODEFU- SION'S generation is incorrect os. removedir is not valid function, the correct function name is removedirs. Further, this function only moves empty directories while the user wanted remove the directory tree which includes files.
C.2 Visualizing Diffusion
CODEFUSION iteratively denoises the latent repre- sentation to construct the final target. This can be visualized by mapping the representation at each time step discrete tokens. We follow the setup explained in Section-5.3, Figure 4 shows suc- cess example from the Python benchmarks where CODEFUSION able to generate the correct code. The figures shows the input query, the target code
C.3 Effect Diffusion Time Step
The number of diffusion timesteps is directly lated to the generation quality as shown in (Sa- haria et al., 2022a). We explore how CODE- FUSION is affected by the number timesteps

---

NL:
directory tree /ffolder_naie name"
Target:
shotil.mmrree///oddeee.mmm))
1200
Gcidr 66e8e CallwithMethodYype ValueList Ssubdomain 6pow 6(' 6tagName édoct
t= 1000
des.os. (*rent ( ("/foldername'' file. os.del f.read() subdomain
t= 800
des.os. we......tere (*ren# (*/foldername', file.rem(( os. f.read() open
t 600
oshut. removetrt("//ooo.drr "/name").remi(
t 400
as rwwoverre (foodder nane', file)
t 200
os www.eeeddtrtt/foooocccc name")
os (annoveeir(//foldrrrro
Figure 5: Figure showing the various stages of denoising in CODEFUSION benchmarks where CODEFUSION fails. CODEFUSION starts from pure noise and gradually denoises to generate the target code. The final generation incorrect here as the correct function name os. removedi and also this function only removes empty directories while the user wanted to remove directory with files.
example from the Python
the benchmarks found to be 2318 milliseconds. The average GPU memory used was 928 Mega Bytes and the maximum GPU memory used was 1042 Mega Bytes
90
CodeBERT Score
80
70
60
40
D Background
Python
30
D.1 Transformer based Sequence Generation
200
400
600
1000
1200
Diffission Timesteps
Transformer based language models (Vaswani al., 2017) are conditional generative models imple- mented through auto-regressive (AR) decoding. These models predict the likelihood of the target token Ye using the conditional input encoding and previously generated tokens y1, //2,' - The likelihood of the generated sequence is given by:
Figure 6: Plot showing the top-1 I CodeBERT score for CODEFUSION for Python against increasing diffu- sion timesteps. Performance improves with increasing timesteps and stabilizes
1000
for Python. We try different timestep values, T = (200, 400, 600, 800, 1000, 1200} and plot the CodeBERT score corresponding to each variation. against in- creasing timesteps. We that the dependence of quality with timesteps is true for CODEFUSION as well. Adding timesteps has a diminishing gain as we see the plot flatten at 1000. should be noted that adding timesteps also increases the inference latency and memory requirements of the model.
N
P(ylx) II p(u()111132) ==1
(1)
Figure 6 show the CodeBERT
D.2 Diffusion Model
Diffusion processes are discrete-time Markov pro- cess. The process starts with an initial state To at timestep = O, where To is from the original data distribution. The process moves forward by grad- ually adding Gaussian noise mg in accordance to the variance schedule B1, B,...,Brr.. Since the for- ward process only adds noise based on a schedule, at any timestep + 1, Ft+1 can be expressed terms of 24
C.4 Latency and Memory
Diffusion models are known to be complex with millions of parameters, and have higher latency and memory requirements than transformer based models. This is due repeated sampling and se- quential denoising operations. CODEFUSION has 75 Million parameters and requires a disk space of 544 Mega Bytes. The average inference latency on
a( (m.) = (KL41: Bes
(2)
During training, diffusion model learns to per- form the inverse diffusion process, wherein it pre- dicts the noise at the current state, 2y at timestep t.

---

By using the predicted noise, we can generate the previous state, ft-1 by subtracting the noise from 2t and rescaling the mean. Thus, the distribution of #t-1 given 2y is simply a Gaussian with mean 14g and variance of o/-1² where, the parameters of the neural network.
The diffusion model trained by minimizing the mean squared error between the true mean lig (-1 and the predicted mean p1)
D.3 Diffusion Models for Text Generation
Recent works in text generation have started ex- ploring diffusion models. (Li et al., 2022) proposes system which uses embeddings convert dis- crete tokens text to continuous representations which form the target distribution. The diffusion model is then trained to denoise random Gaussian noise gradually to generate a sample from the tar- get distribution. (Lin et al., 2023) extends this to conditional text generation by encoding the input and concatenating it with the diffusion state. (Wu et al., 2023) introduces clamping to force the model follow the target distribution each timestep. improve performance.


Reviews:
[{"review_id": "6SoxVSl1qO", "reviewer": "Reviewer_9CNw", "summary": "", "questions": "* In Table 2, why don't you compare with a stronger baseline such as CodeT5 which is included for comparison in Table 1?\n* It would be better to separate the effects of the diffusion module and CPD pretraing tasks, which are parallel to each other. Sometimes their effects are mixed and coupled in the paper presentation. Can you explain the effects for each of them?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question-1: There are no experimental results for analyzing the better diversity of the generated programs in Table 1**\n\nThank you for making this observation. We will extend the next version of the paper with the table below. This table details the following for CodeFUSION and other models:\n- Count of distinct token-level n-grams in the generated outputs divided by number of tokens [1].\n- Summary statistics over pairwise similarities of CodeBERT-based encoding of outputs [2].\n- Summary statistics over pairwise string edit distance of outputs [2].\n\n|     Models        |     1-Gram    |     2-Grams    |     3-Grams    |     4-Grams    |     Min Embedding Similarity    |     Max Embedding Similarity    |     Mean Embedding Similarity    |     Min Edit Distance    |     Max Edit Distance    |     Mean Edit Distance    |\n|-------------------|---------------|----------------|----------------|----------------|---------------------------------|---------------------------------|----------------------------------|--------------------------|--------------------------|---------------------------|\n|     T5            |     0.29      |     0.45       |     0.54       |     0.6        |     0.93                        |     0.99                        |     0.97                         |     0.08                 |     0.58                 |     0.36                  |\n|     CodeT5        |     0.27      |     0.41       |     0.46       |     0.51       |     0.94                        |     0.99                        |     0.97                         |     0.08                 |     0.56                 |     0.32                  |\n|     CodeFusion    |     0.53      |     0.65       |     0.74       |     0.81       |     0.83                        |     0.97                        |     0.91                         |     0.21                 |     0.83                 |     0.57                  |\n\n \\\nCodeFUSION’s higher n-gram fractions, lower embedding similarity, and higher edit distance on average support the claim that CodeFUSION generates more diverse programs.\n \\\n \\\n \\\n**Question-2: CodeFUSION has a large model size by introducing a denoiser module**\n\nThank you for catching this mistake– Table 1 model size for CodeFUSION is not correctly detailed (this was the result of a copy error).  Please see the corrected model sizes below.\n\n|     Model           |     Size     |\n|---------------------|--------------|\n|     T5              |     770M     |\n|     CodeT5          |     770M     |\n|     GPT-3           |     175B     |\n|     StarCoder       |     15.5B    |\n|     CodeGen         |     350M     |\n|     CodeT5+         |     16B      |\n|     Diffusion-LM    |     50M      |\n|     Genie           |     93M      |\n|     CodeFusion      |     75M      |\n\nCodeT5 is roughly 10 times the size of CodeFUSION.\n \\\n \\\n \\\n**Question-3: This work lacks comparison to other SoTA code LLMs like StarCoder and CodeT5+ and more popular code generation benchmarks such as HumanEval and APPS.**\n\nWe have extended our evaluation to include CodeT5+ [9], StarCoder [8], ChatGPT [10], and CodeGen [7] as suggested by other reviewers—thank you for your suggestion. We will also detail some interesting future work based on this. The table below summarizes their performance. Note that all other models have significantly more parameters. CodeT5+ is 40 times larger, while StarCoder is 200 times larger than CodeFUSION.\n\nWe used two previously published benchmarks for NL to Code in Python and Bash. The Python benchmark, CoNaLa, has been used in [3,4,5,6]. We chose these benchmarks as the average code length (number of tokens) is 15.7 compared to 74.3 in HumanEval/APPs. Extending CodeFUSION to longer code fragments remains future work and we will discuss this limitation in the next version of the paper.\n\nThe updated evaluation table is shown below for Python (CoNaLa) Dataset.\n\n|     System          |     Model      |     Param    |     Top-1    |     Top-3    |     Top-5    |\n|---------------------|----------------|--------------|--------------|--------------|--------------|\n|     T5              |     T5         |     770M     |     80.4     |     82.3     |     84.8     |\n|     CodeT5          |     T5         |     770M     |     80.5     |     83.1     |     85       |\n|     GPT-3           |     GPT 3      |     175B     |     82.5     |     83.7     |     85.8     |\n|     ChatGPT         |     GPT 3.5    |     20B      |     80.6     |     82.5     |     83.9     |\n|     StarCoder       |     GPT-2      |     15.5B    |     79.2     |     82       |     84.1     |\n|     CodeT5+         |     T5         |     16B      |     79.6     |     82.1     |     84.5     |\n|     CodeGen         |     Custom     |     350M     |     80.1     |     81.8     |     83.7     |\n|     Diffusion-LM    |     Custom     |     50M      |     70.4     |     74.3     |     76.5     |\n|     GENIE           |     Custom     |     93M      |     73.2     |     77.1     |     80.3     |\n|     CODEFUSION      |     T5         |     75M      |     80.7     |     86.3     |     90.3     |\n\n \\\n**Questtion-4: In Table 2, why don’t you compare with a stronger baseline such as CodeT5 which is included for comparison in Table 1?**\n\nThe scope of table 2 is to compare CodeFUSION with existing text diffusion models to highlight that adding the decoder helps CodeFUSION in generating syntactically correct code. We will clarify this in the next version.\n\n|     System          |     Python    |     Bash    |     CF Rule    |\n|---------------------|---------------|-------------|----------------|\n|     Diffusion-LM    |     19.5      |     40.4    |     74.3       |\n|     Genie           |     24.2      |     54.2    |     78.6       |\n|     CodeFusion      |     67.6      |     73.4    |     94.5       |\n\n \\\n**Question-5: It would be better to separate the effects of the diffusion module and CPD pretraining tasks.**\n\nWithout the diffusion module, CodeFUSION essentially becomes CodeT5. An ablation on removing CPD can be found in Table 2 (shown as M – CPD objective), as well as CodeFUSION without the decoder (shown as M with Grounding and M with clamping). We will improve the clarity of this table.\n \\\n \\\n \\\n**Reproducibility and Benchmarks**\n\nPlease see the link below for our source code. We include this for review purposes only. We will extend the appendix with more details on training and dataset. In addition, we will be publicly releasing our benchmark tasks and our model’s generations.\n\n[Source Code](https://drive.google.com/drive/folders/1iDFbsH4tYmEJI_UXDh7jSRAD0RGO58y2?usp=sharing)\n \\\n \\\n \\\n**References:**\n\n[1]: Vijayakumar, Ashwin K., et al. \"Diverse beam search: Decoding diverse solutions from neural sequence models.\" arXiv preprint arXiv:1610.02424 (2016).\n\n[2]: Perlitz, Yotam, et al. \"Diversity enhanced table-to-text generation via type control.\" arXiv preprint arXiv:2205.10938 (2022).\n\n[3]: Lomshakov, V., Kovalchuk, S., Omelchenko, M., Nikolenko, S., Aliev, A. (2023). “Fine-Tuning Large Language Models for Answering Programming Questions with Code Snippets.” ICCS 2023.\n\n[4]: Soliman, Ahmed S., Mayada M. Hadhoud, and Samir I. Shaheen. \"MarianCG: A code generation transformer model inspired by machine translation.\" Journal of Engineering and Applied Science 69.1 (2022): 1-23.\n\n[5]: Beau, Nathanaël, and Benoît Crabbé. \"The impact of lexical and grammatical processing on generating code from natural language.\" ACL 2022.\n\n[6]: Norouzi, Sajad, Keyi Tang, and Yanshuai Cao. \"Code Generation from Natural Language with Less Prior and More Monolingual Data.\" arXiv preprint arXiv:2101.00259 (2021).\n\n[7]: Nijkamp, Erik, et al. \"Codegen: An open large language model for code with multi-turn program synthesis.\" arXiv preprint arXiv:2203.13474 (2022).\n\n[8]: Li, Raymond, et al. \"StarCoder: may the source be with you!.\" arXiv preprint arXiv:2305.06161 (2023).\n\n[9]: Wang, Yue, et al. \"Codet5+: Open code large language models for code understanding and generation.\" arXiv preprint arXiv:2305.07922 (2023).\n\n[10]: https://openai.com/blog/chatgpt", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Novel and interesting idea to explore diffusion models for code generation", "reasons_to_reject": "* The motivation of the diffusion model for better diversity is not convincing.  There are no experimental results for analyzing the better diversity of the generated programs In Table 1, showing better top-3 and top-5 results does not imply better diversity. For better diversity, a commonly adopted approach is to use samping decoding with a higher temperature. This work needs to be compared with this to better showcase its benefit of high diverse generations.\n* The experimental results cannot validate the advantages of CodeFusion over other code LLMs. In Table 1, especially for the top-1 results, the performance gains between CodeFusion and CodeT5 are quite limited, even given that CodeFusion has a large model size by introducing a denoiser module consisting of 10 Transformer layers (the Param column should be updated with real total parameter counts for CodeFusion).\n* This work lacks comparison to other SoTA code LLMs such as StarCoder or CodeT5+ and more popular code generation benchmarks such as HumanEval and APPS.", "questions_to_authors": "* In Table 2, why don't you compare with a stronger baseline such as CodeT5 which is included for comparison in Table 1?\n* It would be better to separate the effects of the diffusion module and CPD pretraing tasks, which are parallel to each other. Sometimes their effects are mixed and coupled in the paper presentation. Can you explain the effects for each of them?", "paper_topic_and_main_contributions": "This paper presents CodeFusion, the first diffusion model for code generation tasks. While the idea of exploring diffusion models for code is interesting and novel, this paper does not provide enough evidence for why diffusion models are better than conventional T5 or GPT-style code LLMs. For the design of CodeFusion, the authors insert a denoiser module to connect an encoder and decoder and claim that such a denoiser (acts as a diffusion module) is able to boost the generation diversity and leads to better code generation performance. Besides, this paper introduces another pretraining task of continuous paragraph denoising, which is parallel to the diffusion model design and claimed to improve the syntactic correctness of the generated programs. The experiments show that CodeFusion achieves better results on 3 code generation benchmarks.\n\nHowever, this denoiser module also introduces extra parameters compared to a vanilla encoder-decoder model, making it difficult to determine whether the performance improvement comes from the novel design or just from the large model size. Furthermore, there are no experimental results for analyzing the better diversity of the generated programs (showing better top-3 and top-5 results in Table 1 does not imply better diversity). Worse still, this paper does not compare with SoTA code LLMs such as StarCoder or CodeT5+ and more popular code generation benchmarks such as HumanEval and APPS.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Some recent code LLMs such as CodeT5+ and StarCoder.", "Typos_Grammar_Style_and_Presentation_Improvements": "There is a much room to improve the presentation."}, "extra_text": {}}, {"review_id": "W8nRGq2qVj", "reviewer": "Reviewer_ADn5", "summary": "", "questions": "1. In Figure 1, how is the Gaussian noise added to the encoder? Why is there no connection between the Gaussian noise and the representations?\n\n2. I did not quite understand line 153. How does the formula in line 149 allow each token to be generated with full information about the other tokens?\n\n3. Could you provide a more detailed explanation as to why this design can generate code with correct syntax and diversity?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question-1: The abstract appears somewhat fragmented and lacks motivation**\n\nThank you for bringing this to our attention, there is indeed better motivation for our work. Imagine that you are only allowed to change the last line of code while programming—how often would you need to start programming a function from scratch before it is correct? Auto-regressive models have a similar limitation while generating code: they do not allow the search to reconsider early tokens without a very expensive (grouped) beam search. By leveraging a diffusion process to gradually denoise a program based on a natural language condition, all program tokens can be reconsidered in each iteration. In combination with being conditioned on Gaussian noise, this allows us to generate more diverse candidate programs, while ensuring that all candidates are related to the condition.\n\nWe will update our abstract to more closely reflect this intuition and motivation for applying diffusion to code.\n \\\n \\\n \\\n**Question-2: How does the design of the loss function, as described in line 171, ensure diversity of generated code?**\n\nThe loss function itself is not what drives diversity, but it allows us to train the three main components of our model (denoiser, decoder and classification head) with a diffusion objective. It’s this diffusion objective, where the denoiser is conditioned on Gaussian noise, that drives diversity.\n\nTwo of the three components of this loss were used in [1]. We add a term for the decoder, which we use to improve the syntactical correctness of generated code (see Question-6).\n\nWe will clarify this in the revised version of the paper, as well as show results on the diversity of generated code.\n- Count of distinct token-level n-grams in the generated outputs divided by number of tokens [3].\n- Summary statistics over pairwise similarities of CodeBERT-based encoding of outputs [4].\n- Summary statistics over pairwise string edit distance of outputs [4].\n\n| Models     | 1-Gram | 2-Grams | 3-Grams | 4-Grams | Min Embedding Similarity | Max Embedding Similarity | Mean Embedding Similarity | Min Edit Distance | Max Edit Distance | Mean Edit Distance |\n|------------|--------|---------|---------|---------|--------------------------|--------------------------|---------------------------|-------------------|-------------------|--------------------|\n| T5         | 0.29   | 0.45    | 0.54    | 0.6     | 0.93                     | 0.99                     | 0.97                      | 0.08              | 0.58              | 0.36               |\n| CodeT5     | 0.27   | 0.41    | 0.46    | 0.51    | 0.94                     | 0.99                     | 0.97                      | 0.08              | 0.56              | 0.32               |\n| CodeFusion | 0.53   | 0.65    | 0.74    | 0.81    | 0.83                     | 0.97                     | 0.91                      | 0.21              | 0.83              | 0.57               |\n\nCodeFUSION’s higher n-gram fractions, lower embedding similarity, and higher edit distance on average support the claim that CodeFUSION generates more diverse programs.\n \\\n \\\n \\\n**Question-3: I recommend that you explore more large-scale code models such as CodeGen and CodeX.**\n\nWe compared to text-davinci-003 (listed as GPT-3 in Table 1) and will clarify the exact endpoint in the paper. Text-davinci-003 supersedes CodeX, which has now been deprecated [2]. We have extended our evaluation to include CodeGen, along with three other recent models suggested by other reviewers (StarCoder, CodeT5+, ChatGPT/GPT-3.5). The updated evaluation table is shown below for Python (CoNaLa) Dataset.\n\n| System       | Model   | Param | Top-1 | Top-3 | Top-5 |\n|--------------|---------|-------|-------|-------|-------|\n| T5           | T5      | 770M  | 80.4  | 82.3  | 84.8  |\n| CodeT5       | T5      | 770M  | 80.5  | 83.1  | 85    |\n| GPT-3        | GPT 3   | 175B  | 82.5  | 83.7  | 85.8  |\n| ChatGPT      | GPT 3.5 | 20B   | 80.6  | 82.5  | 83.9  |\n| StarCoder    | GPT-2   | 15.5B | 79.2  | 82    | 84.1  |\n| CodeT5+      | T5      | 3B    | 78.6  | 81.1  | 83.5  |\n| CodeGen      | Custom  | 350M  | 80.1  | 81.8  | 83.7  |\n| Diffusion-LM | Custom  | 50M   | 70.4  | 74.3  | 76.5  |\n| GENIE        | Custom  | 93M   | 73.2  | 77.1  | 80.3  |\n| CODEFUSION   | T5      | 75M   | 80.7  | 86.3  | 90.3  |\n\n \\\n**Question-4: In Figure 1, how is the Gaussian noise added to the encoder? Why is there no connection between the Gaussian noise and the representations?**\n\nThe noise is used by the denoiser, which takes in the embedded utterance (produced by the encoder) to condition the denoising. We will clarify this in the figure.\n \\\n \\\n \\\n**Question-5: I did not quite understand line 153. How does the formula in line 149 allow each token to be generated with full information about the other tokens?**\n\nThe decoder is a transformer where each input token is allowed to attend to all other input tokens. Before mapping each embedded token back to a discrete token (with the classification head) we use this decoder to capture some syntactic properties of programs. For example, the denoiser can emit an embedded token that is similar to the representation for both “(“ and “)” tokens. The classification head can only look at one token and thus cannot reliably decide when the embedded tokens are very similar. The decoder, however, can look at all other tokens (through the full attention) and give more information to the classification head about which token to pick.\n \\\n \\\n \\\n**Question-6: Could you provide a more detailed explanation as to why this design can generate code with correct syntax and diversity?**\n\nThis is partially answered in the previous question: a decoder with full attention prepares the denoised program tokens for being discretized. This decoder can be pre-trained without supervision on code snippets only, which teaches the denoiser, decoder and classification modules to generate syntactically correct programs. Diffusion helps in generating more diverse candidates by (1) being conditioned on Gaussian noise, and (2) allowing each token to be reconsidered during the (reverse) diffusion process.\n\nWe will extend our discussion in the paper on this topic. Specifically, we find that pre-training the decoder on code-only helps generate syntactically valid code (see table below),  while the diffusion process induces diverse generations (see answer 2). \n\n|     System          |     Python    |     Bash    |     CF Rule    |\n|---------------------|---------------|-------------|----------------|\n|     Diffusion-LM    |     19.5      |     40.4    |     74.3       |\n|     Genie           |     24.2      |     54.2    |     78.6       |\n|     CodeFusion      |     67.6      |     73.4    |     94.5       |\n\n \\\n**Question-6: Missing lots of references**\n\nWe will extend our related work discussion to reflect new work . Specifically, we will add [3,4,5,6,7,8,9,10,11] to related works and add [15,16,17,18] as new baselines to the evaluation.\n\nPlease see the link below for our source code. We include this for review purposes only. We will extend the appendix with more details on training and dataset. In addition, we will be publicly releasing our benchmark tasks and our model’s generations.\n\n[Source Code](https://drive.google.com/drive/folders/1iDFbsH4tYmEJI_UXDh7jSRAD0RGO58y2?usp=sharing)\n \\\n \\\n \\\n \\\n**References:**\n\n[1]: Dockhorn, Tim, Arash Vahdat, and Karsten Kreis. \"Genie: Higher-order denoising diffusion solvers.\" Advances in Neural Information Processing Systems 35 (2022): 30150-30166.\n\n[2]: https://platform.openai.com/docs/guides/code\n\n[3]: Vijayakumar, Ashwin K., et al. \"Diverse beam search: Decoding diverse solutions from neural sequence models.\" arXiv preprint arXiv:1610.02424 (2016).\n\n[4]: Perlitz, Yotam, et al. \"Diversity enhanced table-to-text generation via type control.\" arXiv preprint arXiv:2205.10938 (2022).\n\n[5]: Lomshakov, V., Kovalchuk, S., Omelchenko, M., Nikolenko, S., Aliev, A. (2023). Fine-Tuning Large Language Models for Answering Programming Questions with Code Snippets. In: Mikyška, J., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science – ICCS 2023. ICCS 2023. Lecture Notes in Computer Science, vol 14074. Springer, Cham. https://doi.org/10.1007/978-3-031-36021-3_15\n\n[6]: Soliman, Ahmed S., Mayada M. Hadhoud, and Samir I. Shaheen. \"MarianCG: A code generation transformer model inspired by machine translation.\" Journal of Engineering and Applied Science 69.1 (2022): 1-23.\n\n[7]: Beau, Nathanaël, and Benoît Crabbé. \"The impact of lexical and grammatical processing on generating code from natural language.\" arXiv preprint arXiv:2202.13972 (2022).\n\n[8]: Norouzi, Sajad, Keyi Tang, and Yanshuai Cao. \"Code Generation from Natural Language with Less Prior and More Monolingual Data.\" arXiv preprint arXiv:2101.00259 (2021).\n\n[9]: A Systematic Evaluation of Large Language Models of Code https://arxiv.org/abs/2202.13169\n\n[10]: A Survey on Pretrained Language Models for Neural Code Intelligence https://arxiv.org/abs/2212.10079\n\n[11]:\tAn Empirical Comparison of Pre-Trained Models of Source Code https://arxiv.org/abs/2302.04026\n\n[12]: Coder Reviewer Reranking for Code Generation https://arxiv.org/abs/2211.16490\n\n[13]:\tTransCoder: Towards Unified Transferable Code Representation Learning Inspired by Human Skills https://arxiv.org/abs/2306.07285\n\n[14]:\tWizardCoder: Empowering Code Large Language Models with Evol-Instruct https://arxiv.org/abs/2306.08568\n\n[15]:\tNijkamp, Erik, et al. \"Codegen: An open large language model for code with multi-turn program synthesis.\" arXiv preprint arXiv:2203.13474 (2022).\n\n[16]: Li, Raymond, et al. \"StarCoder: may the source be with you!.\" arXiv preprint arXiv:2305.06161 (2023).\n\n[17]:\tWang, Yue, et al. \"Codet5+: Open code large language models for code understanding and generation.\" arXiv preprint arXiv:2305.07922 (2023).\n\n[18]:\thttps://openai.com/blog/chatgpt", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. CODEFUSION is the first diffusion-based NL-to-code model, which brings a novel approach.\n\n2. The authors have clearly defined their formulas.\n\n3. The experimental results are impressive.", "reasons_to_reject": "1. The abstract could benefit from additional clarity to better convey the research motivation and objectives. The current statement, \"Diffusion models have been extended to discrete domains like text, but have not yet been successfully applied to code tasks,\" falls short in providing a clear rationale for the study. It would be helpful to specify the particular question and its importance in code generation, such as \"increase the diversity of code generation\".\nIn the Introduction, you mention that \"Transformer-based NL-to-Code systems can struggle to produce diverse generations.\" While this is a valid point, it should be noted that existing large code models, like in CodeGeex [1], mentioned \"We use budget allocation methods to help improve the diversity of the generated solutions.”, which have shown significant progress in this aspect. If the aim of your work is to improve generation diversity, I suggest elaborating on how your research differs from these existing studies and what unique contributions it is expected to make.\n\n2. How does the design of the loss function, as described in line 171, ensure the diversity of the generated code? Why don't the second and third terms of the Loss function require weight parameters like \\alpha or \\beta? Moreover, the lack of supporting code in this paper does not facilitate the reproducibility of the feasibility of this work.\n\n3. I recommend that you explore more large-scale code models such as CodeGen [2] and CodeX [3]. However, this is a short paper, so this is just a suggestion.\n\n[1] \"Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x.\" arXiv preprint arXiv:2303.17568 (2023).\n\n[2] \"Codegen: An open large language model for code with multi-turn program synthesis.\" arXiv preprint arXiv:2203.13474 (2022).\n\n[3] \"Evaluating large language models trained on code.\" arXiv preprint arXiv:2107.03374 (2021).", "questions_to_authors": "1. In Figure 1, how is the Gaussian noise added to the encoder? Why is there no connection between the Gaussian noise and the representations?\n\n2. I did not quite understand line 153. How does the formula in line 149 allow each token to be generated with full information about the other tokens?\n\n3. Could you provide a more detailed explanation as to why this design can generate code with correct syntax and diversity?", "paper_topic_and_main_contributions": "To enable code generation diversity and \u001dmaintain the validity of generated code, the author introduce CODEFUSION, a novel pre-trained diffusion code generation model that employs an attention-based decoding strategy and a mask-denoising objective to generate code, while attending to contextual token embeddings and sequential dependencies.  The result shows CODEFUSION can compete with or outperform existing transformer and diffusion-based text generation models on the task of text to code generation for Python, Bash and formatting rules.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I hope you can describe the dataset you use, if there is not enough space you can put it in the appendix."}, "extra_text": {"Missing_References": "Missing lots of CodeLLMs appeared in 2022/23, like CodeGen [1] and CodeX [2], etc.\n\n[1] \"Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x.\" arXiv preprint arXiv:2303.17568 (2023).\n\n[2] \"Codegen: An open large language model for code with multi-turn program synthesis.\" arXiv preprint arXiv:2203.13474 (2022)."}}, {"review_id": "9BnePZqW2o", "reviewer": "Reviewer_XjFk", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question-1: The comparisons with other LMs such as text-davinci-003 and CodePTMs like CodeT5 might be slightly “unfair”. The reason is that baselines are general-purpose models, which can support generating different PLs by a single checkpoint (and can to some extent perform zero-shot)** \n\nWe fine-tuned CodeT5 and used text-davinci-003 in a few-shot prompting (as is common in literature). We have now also extended our evaluation to include results with CodeGen [1], StarCoder [2], CodeT5+ [3] and ChatGPT/GPT-3.5 [4]. The extended table is shown below for the Python (CoNaLa) Dataset:\n\n| System       | Model   | Param | Top-1 | Top-3 | Top-5 |\n|--------------|---------|-------|-------|-------|-------|\n| T5           | T5      | 770M  | 80.4  | 82.3  | 84.8  |\n| CodeT5       | T5      | 770M  | 80.5  | 83.1  | 85    |\n| GPT-3        | GPT 3   | 175B  | 82.5  | 83.7  | 85.8  |\n| ChatGPT      | GPT 3.5 | 20B   | 80.6  | 82.5  | 83.9  |\n| StarCoder    | GPT-2   | 15.5B | 79.2  | 82    | 84.1  |\n| CodeT5+      | T5      | 3B    | 78.6  | 81.1  | 83.5  |\n| CodeGen      | Custom  | 350M  | 80.1  | 81.8  | 83.7  |\n| Diffusion-LM | Custom  | 50M   | 70.4  | 74.3  | 76.5  |\n| GENIE        | Custom  | 93M   | 73.2  | 77.1  | 80.3  |\n| CODEFUSION   | T5      | 75M   | 80.7  | 86.3  | 90.3  |\n\n \\\n**Question-2: I suggest adding more 2022/23 papers about models and code generation into references to keep the paper up-to-date** \n\nWe will extend our related work discussion to reflect on new work. Specifically, we will add [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] to the related works and update evaluation to include [1,2,3,4].\\\nThe new evaluation table is attached in Question-1.\n \\\n \\\n \\\n**Question-3: Authors might consider using the additional one page to add more details of the model training and diffusion LMs (like the paragraphs in the Appendix) if accepted.**\n\nPlease see the link below for our source code. We include this for review purposes only. We will extend the appendix with more details on training and dataset. In addition, we will be publicly releasing our benchmark tasks and our model’s generations.\n\n[Source Code](https://figshare.com/s/1ac0b38ebf9ddc6a4a46)\n \\\n \\\n \\\n**References**\n\n[1]: Nijkamp, Erik, et al. \"Codegen: An open large language model for code with multi-turn program synthesis.\" arXiv preprint arXiv:2203.13474 (2022).\n\n[2]: Li, Raymond, et al. \"StarCoder: may the source be with you!\" arXiv preprint arXiv:2305.06161 (2023).\n\n[3]: Wang, Yue, et al. \"Codet5+: Open code large language models for code understanding and generation.\" arXiv preprint arXiv:2305.07922 (2023).\n\n[4]: https://openai.com/blog/chatgpt \n\n[5]: Vijayakumar, Ashwin K., et al. \"Diverse beam search: Decoding diverse solutions from neural sequence models.\" arXiv preprint arXiv:1610.02424 (2016).\n\n[6]: Perlitz, Yotam, et al. \"Diversity enhanced table-to-text generation via type control.\" arXiv preprint arXiv:2205.10938 (2022).\n\n[7]: Lomshakov, V., Kovalchuk, S., Omelchenko, M., Nikolenko, S., Aliev, A. (2023). Fine-Tuning Large Language Models for Answering Programming Questions with Code Snippets. In: Mikyška, J., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science – ICCS 2023. ICCS 2023. Lecture Notes in Computer Science, vol 14074. Springer, Cham. https://doi.org/10.1007/978-3-031-36021-3_15\n\n[8]: Soliman, Ahmed S., Mayada M. Hadhoud, and Samir I. Shaheen. \"MarianCG: A code generation transformer model inspired by machine translation.\" Journal of Engineering and Applied Science 69.1 (2022): 1-23.\n\n[9]: Beau, Nathanaël, and Benoît Crabbé. \"The impact of lexical and grammatical processing on generating code from natural language.\" arXiv preprint arXiv:2202.13972 (2022).\n\n[10]:\tNorouzi, Sajad, Keyi Tang, and Yanshuai Cao. \"Code Generation from Natural Language with Less Prior and More Monolingual Data.\" arXiv preprint arXiv:2101.00259 (2021). \n\n[11]:\tA Systematic Evaluation of Large Language Models of Code https://arxiv.org/abs/2202.13169 \n\n[12]:\tA Survey on Pretrained Language Models for Neural Code Intelligence https://arxiv.org/abs/2212.10079 \n\n[13]:\tAn Empirical Comparison of Pre-Trained Models of Source Code https://arxiv.org/abs/2302.04026 \n\n[14]:\tCoder Reviewer Reranking for Code Generation https://arxiv.org/abs/2211.16490 \n\n[15]:\tTransCoder: Towards Unified Transferable Code Representation Learning Inspired by Human Skills https://arxiv.org/abs/2306.07285 \n\n[16]:\tWizardCoder: Empowering Code Large Language Models with Evol-Instruct https://arxiv.org/abs/2306.08568", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper successfully leverages diffusion LMs to conduct code-related tasks and becomes the first diffusion-based NL2Code model.\n2. Authors empirically demonstrate the effectiveness of CODEFUSION by solid evaluations on NL2Bash, Python code generation, and Excel Conditional Formatting tasks.\n3. As a short paper, the paper includes sufficient analysis and ablations of the results and provides an in-depth discussion of diversity and refinement.\n4. The overall presentation of this paper reaches the standards of an *ACL publication.", "reasons_to_reject": "The comparisons with other LMs such as text-davinci-003 and CodePTMs like CodeT5 might be slightly “unfair”. The reason is that these baselines are general-purpose models, which can support generating different PLs by a single checkpoint (and can to some extent perform zero-shot). However, as is provided in A.3, CODEFUSION is task-specifically trained.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes CODEFUSION that applies diffusion LMs to the domain of neural code intelligence, aiming for diverse and reliable code generation. Specifically, the authors combine transformer architecture and diffusion techniques and then leverage CPD tasks to curb the model from generating syntactically invalid programs. CODEFUSION is evaluated on the tasks of NL2Code generation tasks for Bash, Python, and Excel Conditional Formatting (CF) rules, and can reach comparable performance to mainstream auto-regressive LMs. Moreover, benefiting from the randomness of diffusion LMs, the proposed method reaches higher top-3/5 performance compared to other strong baselines.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Please refer to \"Typos Grammar Style And Presentation Improvements.\""}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Authors might consider using the additional one page to add more details of the model training and diffusion LMs (like the paragraphs in the Appendix) if accepted. \n2. I suggest adding more 2022/23 papers about models and code generation into references to keep the paper up-to-date\n\n[1] A Systematic Evaluation of Large Language Models of Code https://arxiv.org/abs/2202.13169\n\n[2] A Survey on Pretrained Language Models for Neural Code Intelligence https://arxiv.org/abs/2212.10079\n\n[3] An Empirical Comparison of Pre-Trained Models of Source Code https://arxiv.org/abs/2302.04026\n\n[4] Coder Reviewer Reranking for Code Generation https://arxiv.org/abs/2211.16490\n\n[5] TransCoder: Towards Unified Transferable Code Representation Learning Inspired by Human Skills https://arxiv.org/abs/2306.07285\n\n[6] WizardCoder: Empowering Code Large Language Models with Evol-Instruct https://arxiv.org/abs/2306.08568"}}]

Author rebuttal:

Meta-review: This paper introduces a diffusion model for natural language to code generation problems. Code generation is a growing area of research and this paper adds the first diffusion model to improve the diversity of the generated code. Reviewers find that the presentation is clear, the method to be novel and interesting. Reviewers express concern on lack of evidence for why diffusion models work better, for which authors responded with fine-grained analysis of diversity of generated programs. Authors also added the requested baselines such as StartCoder and CodeT5. Authors should add these results along with other clarifications in the revision.
Decision: Accept-Main