Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
Authors: Ziyin Zhang
Chaoyu Chen
Bingchang Liu
Cong Liao
Zi Gong
Hang Yu
Jianguo Li
Rui Wang
Venue: Accepted by TMLR ()

Abstract: In this work we systematically review the recent advancements in software engineering with language models, covering 70+ models, 40+ evaluation tasks, 180+ datasets, and 900 related works. Unlike previous works, we integrate software engineering (SE) with natural language processing (NLP) by discussing the perspectives of both sides: SE applies language models for development automation, while NLP adopts SE tasks for language model evaluation. We break down code processing models into general language models represented by the GPT family and specialized models that are specifically pretrained on code, often with tailored objectives. We discuss the relations and differences between these models, and highlight the historical transition of code modeling from statistical models and RNNs to pretrained Transformers and LLMs, which is exactly the same course that had been taken by NLP. We also go beyond programming and review LLMs' application in other software engineering activities including requirement engineering, testing, deployment, and operations in an endeavor to provide a global view of NLP in SE, and identify key challenges and potential future directions in this domain.

Published in Transactions on Machine Learning Research (09/2024)
Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code
Ziyin Zhang
Shanghai Jiao Tong University And Group
dienerystaggrrynn@gju..ctc. edu.c
Chaoyu Chen And Group
Bingchang Liu And Group
Cong Liao And Group
Zi Gong Ant Group
Hang Yu' And Group
hyw.huug@@tantgrosp.com
Jianguo Li And Group
Nifo.serod@antgroup..com
Rui Wang"
wangguil1@@sjjuuucu.b.om edu.cc
Shanghwi Jia Tong University
Reviewed on OpenReview: https: openreview. net/ forum? id= hkip
Abstract
In this work systematically review the recent adyancements in software engineering with language models, covering 70+ models, 40+ evaluation tasks, 180+ datasets, and 900 related works. Unlike previous works, we integrate software engineering (SE) with natural language processing (NLP) by discussing the perspectives of both sides: SE applies language models for development automation, while NLP adopts SE tasks for language model evaluation. We break down code processing models into general language models represented by the GPT family and specialized models that are specifically pretrained on code, often with tailored objectives. We discuss the relations and differences between these models, and highlight the historical transition of code modeling from statistical models and RNNs to pretrained Transformers and L.Mss, which is exactly the same course that had been taken by NLP. We also go beyond programming and review LLMs' application in other software engineer- ing activitics including requirement enginccring, testing, deployment, and operations an endeavor to provide a global view of NLP in SE, and identify key challenges and poten- tial future directions in this domain. We keep the survey open and updated on Gut at httpp:///github.com/cdooo hatp:eeeaaaaveeeseeeooeeeee---
*Corresponding authors.

---

Published in Transactions on Machine Learning Research (09/2024)
1 Introduction
Language modeling has advanced remarkably in recent years with the advent of pretrained Transform- ers (Vaswani et al.) 2017) such as BERT (Devlin et al.] 2019) and GPT (Radford et al.) 2018 As large language models (L.MM)) scaled to hundreds of billions of parameters and started to display early signs of
artificial general intelligence (Brown et al. 2020: Chowdhery et al. 2023 OpenAT 2023), their applications have also transcended text processing, Pioneered by Codex (Chen et al., 2021b), LLMs have achieved impres- sive results in code processing, giving rise to commercial products such GitHub Copilo and open-source multi-billion code models such as StarCoder (Li et al.
20231) and Code LLaMA (Rozière et al. 2023).
The application of pretrained Transformers in software engineering, however, can be traced back to dates before decoder-only autoregressive models became dominant (Feng et al., 2020 Liu et al.) 2020), and also goes beyond code processing AAroa et al., 2023 Feng et al. 2024; Ma et al. 2024j. However, this domain is yet to witness a comprchensive review. In an attempt to bridge the gap between natural language processing (NLP) community and software engineering (SE) community on the topie of language model applications, undertake panoramic survey of language models for SE in this work, covering 70+ models, 40+ downstream tasks, 180+ datascts, and 900 related works. Putting the emphasis on code language models, we analyze their development, discuss their differences from general language models, and highlight the integration of code-specific features such as abstract syntax trees or data flows, as well as the latest techniques adapted from NLP.
Related to our work, we are aware of several surveys on similar topics, with three works concurrent to us (Hou Zheng et al. 2023b; She et al., 2023 These works, however, focus either on NLP side Zaan
et al.
2023
et al. 2023
Xu & Zhul 2022 or SE side (Niu et al. 2023 Hou et al. 2023 Zheng et al. 2023b; She et al.
2023). and do not cover models, tasks, and challenges from the other side. For example, Zan et al. (2023) focus on L.MSs for text-to-code generation, while giving little discussion of other applications in software enginocring community. Hou et al. (2023) and She et al. (002)), in contrast, comprchensively review works from SE venues such as ASE and ICSE, but cite only a handful of works from deep learning and NLP venues such as ACL, EMNLP, NeurIPS, and ICLR.
Thus, building on these works, we endeavor to unite the perspectives from both communities, and highlight the relationship between NLP and SE: SE applies language models to various tasks for aulomation, ion, while NLP adopts tasks from SE to evaluate language models. We observe that advanced topics from language
modeling have been recently introduced into code processing, including instruction tuning (Honovich et al.
2023
Xu et al. 2024 Luo et al.) 2023), infilling objectives (Tay et al. 2023b Li et al. 2023i Rozière et al.
2023 , recontemplation of scaling laws (Hoffmann et al. 2022: Gunasekar et al. 2023 Li et al., 2023]
architectural improvements (Shazeer 2019; |Su et al. 2024 Dao al.] 2022). and autonomous agents (Qian
et al. 2023 Hong et al., 2023). while in return SE requirements, represented by programming (Chen et al,
2021b), are providing real-world testbeds for these technologies and driving the development of LLMs forward into production and deployment. We believe a systematic review of these advancements would benefit both communities. Furthermore, unlike the existing reviews that focus on programming- the first to explicitly go beyond programming and provide a global view of LLM applications in the full life cycle of software development, covering distinct stages such as requirement engincering, deployment, and operations.
tasks, we are also
The rest of this work is organized following the taxonomy presented in Figure ! In Section 2 we first contextualize the downstream tasks in software engineering, highlighting the focus programming related tasks. Then, in Section 3 we provide the preliminaries of language modeling and Transformer models, and in Section 4 we discuss the plethora of LLMs that have demonstrated coding ability. In Section 5 we review the specialized and often smaller models by their architecture, with special attention on the recent application of infilling objectives, instruction tuning, reinforcement learning, and engineering improvements. Then, in Section G we discuss unique features of code that are not available to natural languages but have been utilized to aid code processing. In Section 7 we review the most recent integration between LLMs and software development, before finally concluding this work in Section R and highlighting the current challenges in code processing.
htttp:://gituub..om//etttttt
2

---

Published in Transactions on Machine Learning Research (09/2024)
LaMDA
[Thppiilan ct al.
2022 PaLM (Chowdhery et al.
2023
GPT-
NeoX
[Back
(2022,
HL.OM (Sao al:
[2024
LJAMA Touvron
et al
20256) GPT [-] (OpenA] 2023
LLakiA
Touvron et al
20230
Raw LM
Phi-
.3
L1 et al. 2023 J. Baichuan Yang ot al.
2022a Ownn Baa
al
2023). Mistral JJangg al. [023a)) Gemini (Amil al. 2023a
DeepSeek DeepSeek-Al cf al. 2021] Mixtral (Jang et a. 2024a
Geumaa Messard
Young
2024
Codex CChen ct al.
2021b)
PaLM
Coder (Chowdhery ct a). 2023
Adapted LM
Minerva (Lewkowyzz
2022).
PaLM 2 (Amil 2023b).
Code LLaMA (Rozière et al.
2023
Code-Q
Qwen wen
Bai et al.
2023
CodeGemma
Team
CuBERT
Kanade al. 2020
CodeBERT FFeng
<t al. 2020). CraphCode
Guo et al.
2021a). Syn-
Encoder
COBERT (Wang al. 1d22dd)
Code-MVP
Wang al.]
2022e
SCodeR (Ll et al. 20227
CodeSage
(Zhang et al.
GPT-C (Syyatkovskiy
2020
CodeG [Lu et all [2021] PolyCoder X1i
al. 2022). CodeGien (Nijkamp
2023b PyCodeGPT (Zan et al. (222)
CLM
PauGit-Code (Christopoulor
al.
CodeGeeX (Zheng et al. [222a) Jam SSu
2023a P [Cunasokar
Code LMs
Code (Di et al.
2023)
Specialized LM
Decoder
InCoder (Fried et al.
2023)
SantaCoder AAl-
[al 2023). StarCoder
20231) >
FIM
DeepSeck Coder (Guo et al.
2024),
S StarCoder
[Lozhkov 2024)
2024). CodeShell
al.
UniLM
CugL.M LLiu et al.
2020 UniXeoder (Guo et al.
2022]
PyM13 Cllement
T5-code (Mas-
tropaolo of all (2021). DOBF
Lachaux et all [2211)
Encoder-
PLBART (Ahmad et al.
2021 CodeTT (Waug et al.
20216 S-'d- [Nau al.
2022
AlphaCode (La
Decoder
2022h J. NatGen (Chalrabarty et al.
2022a). ERNTE
Code Chai al. 2023). CodeT6+
[Wang
2023h
AST 15 [Gong el al. 2021
WizardCoder [Luo et al.
2023 PanGu-Coder2 (Shen
Instruction Finetuning
al.
2023),
OctoCoder Muennighoff al. 2024). In-
struct Coder [Li ct al. 2023h
MFTCoder LLu ct ml.
2023b
WaveCoder YYu
2023c
Astraios ZZuu
Code Finetuning
et al.
2024 CCT (Jiang et al.
2021b
Reinforcement Learning
CompCoder WWang et
2022d
CodeRL [Le
2022). PPOCoder (Shoojaee et
2023 RLTF Liu
2023d
StepCoder (Dou et al. 2024)
Figure 1: Our taxonomy of pretrained language models for code.
2 Downstream Tasks in Software Engineering
Over the past decade, the SE community has found applications of language models in various downstream tasks. CodeXGLUE (Lu et al. 2021 consolidates most of the code-related tasks into single benchmark, while HumanEval Chen et al. (2021b) brought NL-to-code synthesis into the spotlight in the NLP community, which has since become a standard task for evaluating LLMs (Figure However, other tasks, especially those not directly related to coding, have remained understudied.
In this section, we first bricfly introduce each of the traditional SE tasks and the application of pretrained language models in them 2.1 and provide a comprehensive list of related works for each task. Then, review the evaluation metrics in 2.2 and investigate program synthesis in more detail in 2.3 Lastly, we
3

---

Published in Transactions on Machine Learning Research (09/2024)
30
DeegSeek Coder Leeeuse 338
dadeltee 34B GaminTUitra
Witard
70
GPT4
PanGu-Cooer
Wizardooder 168
HumanEval pass@1
Code 348
Phi- 1:38
OctoCoier
Phic
H Mixtral Bx7B
Pat -
PaLM Coder 540B
Instruct MTS 16B
G Owven 148
StarCaater 168
Codox
CodeGen (gno 168
LLMMA 708
PaLM 5408
PanGu Coder 2.68
LLa NA
X 138
208 LaMOX 378
Baichuan 138
SantaCoder 1.18
GPT5 6B
PolyCoder 2.78
New
Jn.
2021
2022
2023
2024
Figure 2: The timeline of code language models' progress on HumanEval.
also discuss the latest trend of repository-level coding in 2.4 In Appendix and respectively, we list benchmarks for each downstream task and language models' performance on them.
2.1 Downstream Tasks in SE
The custom in software engineering is to categorize downstream lasks according to their input/output modal- ity, such as NL-to-PL (also referred to as text-to-code) tasks and PL-to-NL (i.e. code-to-text) tasks (Lu et al. 2021). However, as we are dedicated to the full life cycle of software development, we adopt a different taxonomy this work, and classify downstream tasks according requirement engineering (2.1.1), development (.1.2). testing and analysis (2.1.3), deployment, and opera- tions (2.1.4), and lastly several novel tasks recently proposed for evaluating LLMs (2.15)). We note that this taxonomy is interleaved with the understanding-geneet generation dichotomy in NLP, since each category may contain both understanding and generation tasks, as discussed in 2.1.6
the stages in software development:
2.1.1 Requirement Engineering
Requirement engincering refers to the specification, analysis, and validation of software requirements, soft- ware modeling, and UI/UX design. Related works are listed in Figure 3
Requirement analysis refers to the process of extracting, summarizing, classifying, and validating software requirements. Most early works in this field rely on statistical NLP technologies such as POS (Part-of- Speech) tagging and dependency parsing to extract actions from requirements, while more recent works utilize L.LMs to directly classify, summarize, or extract requirement clauses. Zhao et al. (2020) provide a fine-grained survey on requirement analysis.
UI/UX Design, short for User Interface and User Experience design, is fundamental step in software development. While many works UI design utilize computer vision technologies for layout design (Cheng et al. 2023 Kulkarni et al. 2023 Feng et al. 2023b). we only focus on the intersection of this task and NLP technologies, such as using language models to generate markup languages.
.odel generation aims to generate software models modeling languages such as UML (Unified Modeling Language). Abdelnabi before the rise of LLMss.
(2021a) and Ahmed et al. (2022) provide comprehensive reviews on this task
4

---

Published in Transactions on Machine Learning Research (09/2024)
Nou-neural methodss
Trensformer
Trensformer- methods
motbods
ARSENAL (Ghooh
2016
CAR (Perrai)
2011
Analyais
Requirement
2017
2017
2017
ULL3en
FENL
2017
SNACC
RRTA
NARCIA
SAFE: (Johann et
AUR-BeW [Li A T.lnng
REGICE [Arora
(Bakar m1. 2016
AL.PRTmm (Cmxman
Connng
(Wans mn al. 2013b
LArra
20115
Golarry
2021 W Pitt et
banks at ai K20229
<24X [ Abbubbaa al. 2023
Salnan:
BER de
driguee
CompliXT Arora et 2024
[2023 Azor
(2123
2020
2022 Re
CUItE
2035
Marcacini
RE-
SEMIOS
Deaggonn cl 0l. 2018
Tiwarl
m. 2019
PixciHelp oL al. 2020c
Requirement
Sereen Worns [Wans at 202
JTLMM Aghhajnnaa mt 2022
MarkupLM [L) nu (2022l
Development
pix2cade
Bel-
DOM-LM (Deux et al. 2022b
HTMLBER cf azjcuiio
NLP in SB
Tessing Analysia
C1/UX Design
Crameolt
Agiregin
[tobing
Wans, et
WebForme:
PromptMaker Clang a 22022
[Waug et 22025
20236 (Curr
Razzaran
Deployment Operations
SUGAR
Kumorr & Sanval 2008
grammer
Ping Ni:
Dasptimahant Babar
Evaluacion
Hercht
Weysow
Chasben
Model Generation
Nezrslor
Apora
Abdclaub
Figure NLP applications in the requirement engineering stage SE.
2.1.2 Development
The development phase is the stage where most programming activities take place, and also the stage where LLM applications are most abundant. We further categorize this stage into different activities such as code scarch, code generation, code editing, code suggestion, and code explanation. Related works are listed in Figure and
Code Search
NL-to-code search, also referred to as text-to-code search, aims to retrieve relevant code given natural language queries, or to mine parallel text-code pairs from an unannotated corpus. This task is usually performed by computing a similarity metric between the embedding of query and candidate code, and the contextual embeddings produced by bidirectional language models such as BERT has proven to be extremely helpful. Grazia Pradel (2023) and Xie et al. (2023a) provide comprehensive reviews on this topic.
Code-to-code search is similar task where the input is existing code snippet, often in different programming language from the larget. Code-lo-code search can be reformulated as finding clones of the query in the pool of targets, and is thus equivalent to clone detection to some extent.
API mining refers to the process of finding similar APIs in different librarics, potentially in different programming languages. API mining traditionally tackled by computing similarity metrics between source and target APIs using information retrieval models, but as generative models become ever more capable, it is also worth exploring directly generate the target API as requeence-o-sequeene task. Another closely related task is idiom mining (Allamanis & Sutton 2014), where the objective is to discover commonly used code patterns, which exposes the potential need for new APTs (Sivaraman et al. 2022
Code Generation
Program synthesis aims to generate code (usually a function or method) given a natural language descrip- tion. This task can be viewed updated version of NT-ttocode retrieval using generative models instead
5

---

Published in Transactions on Machine Learning Research (09/2024)
Nou-neural meth--
Non- Transformr noural
Transformor- methods
CODE [lyrr
2016
Deap [50
Requliement
Code
Nl.to- Code Search
2016
Colstow
2022a
Mascher
6l
2015
Code-
Li 00.
Liu 05 m..
2015
0
67 al.
2018
CNN
GraphiScnt ScarehA
NLS
DGMS
TabC8
NJACS
Sachdey
2019
Shaal et h)
2020b
Vin at 1001
UNIF
(Lang at Al
1it #t m..
Xn Ab.
Lambronero
DECe
CARLUS-
Litt
212
2020
2090
202
TranS
CoSoDa
(2022
UodeRattisses [L
20223
Wane
CCT-LB3 Sezokin
(Slat 2012b
[Gu
CDDS (Clar
SAN=Cs Fank
2020e
NIEM
Kuzzo Up
21/33
20231
TranCS
PaCay
Kim
0022
Corlesto- Cade
Arome
2015
Corder
Bui ct 01 2021b
Search
At al 2019
COSA 7 Mathew
Cosco
A: Stoies 2021
Nguyen al.
2016
DespAM
2017
RASH Zhang
JV2C5
(P Panm
API Mining
ct ml 2015
Lu
m)
2217
HaPiM (2ou. al 2029b
TEXPTA HR al
1 Pubayy
[T0 61
2018h
CodeeA MA 202
2020)
BIKRR
[TIAR et
2015
C beb et A1- (222
SAR
B1t st a1 2010
LPN (Linx ct al 2016
NaPS
['Arieatto at 2013
Program Syuthesi
AE cl 2015
Euphoay Lec
Neu Pong cl 2018
2017
65
es a.. 2207
p0:0 2018
Dix 2017
0o 2017
DaepCader Coder (Eaing et
2013
Robust Ti (Devit
ReCodi (Hayati
ASN (Rabloovich
[uggel
NCDS [Kalyan
Yib de Nep-
del e
TreaGen
CODER (Arves
ClarityGPT
Asstldder
Debegging
Jain et di
Sun at
2024
Lahiiii
al.
20/21
2020
2021
(Chon
RED-
Chao-
(Bayml c1 nl.
MANGO
hen a..
RMN Biuch
Huang
Gle
20121
bia
Naturalise
Allmma
2015 Raychey
m..
201n
Completion
Syn Hagrdaaty
Long-
Development: Code Genera-
2016s
Hion
Development: Code Ridising
Code
Development: Sugges-
SQLova [Bwang at al
1101990
Decelopment: Code Explana- Hoon
Seq2SQL
Smadel
Sulu
Tessing Analyaia
Text-to-
9ois Pinegan-1lll lnlas
CoarseäFine (Dong
(Yo
201
202
5050
GT
(Ch
2021 Wang
Deployment Operations
21 20117 TREQS (Wans
20015
ml
20181 CNN [Bogu
20201]
Syntae5QQ Ym
Newr
150L
1021
2022b
2022 Craphiss
Arcacinnoo et al 2022) TKA (Dre
Yang
Picard (Scholakc
UoiexDS DS (lumtter 2022
2021 UnimnEKC RC
(Lr di 20250
2021
Xie
[1. at 20r23 (am
20233
DIN-5OL
[Fourrera 2 Roheel 2023 CCanng Ar
Poolur-Lnnaeer Lunaerr 20243 (Na mu E.7
PaLsi [Sun ot m/ 2125 Gioo
2025
DAIL-SQL (Coo et 2023a
Figure 4: NLP applications in the development stage of SE.
of retrieval models. Statistical machine translation (SMT) and neural machine translation (NMT) models have been widely adopted for this task, often with enhanced decoders that leverage the unique grammatical rules of programming languages (Yin & Neubig 2017 Rabinovich al. 2017). Pretrained language models based Transformer architecture, however, changed the game by directly generating the source code in

---

Published in Transactions on Machine Learning Research (09/2024)
the autoregressive language modeling style, even without task-specific finetuning (Chen al. 2021b). We discuss this task in more detail in Section 2.
Code completion aims to complete a piece of code given its prefix, and remains to date one of the most popular applications of code language models in IDEs. This essentially language modeling applied to code (which we dub "code modeling"), and related technologies have been progressively introduced: n-gram, RNN, and Transformer. However, due the structured nature of programming languages, many early works found grammar-aided statistical models to perform better (Bielik et al. 2016 Hellendoorn & Devanbu 2017 and neural models only became dominant after 2018.
Code infilling is another recently proposed task, after Il-in-thhemiddee pretraining (Bavarian et al. 2022) became popular. It is a generalization of code completion, where not only the left context but also the right context is given.
Text-to-SQL. special case of program synthesis, where the model is tasked generate SQL commands from natural language queries. It has been lopic of special interest in the NLP community (as can be seen from Figure 4 and one hypothesis for this is that SQL's declarative nature and restricted grammar make it casier to optimize comparod with imperative programming languages such as C and Python (Marcus et al. 2019 2020). We refer to Kumar et al. (2022): Deng et al. (2022a): Qin et al. [022a)] Katsogiannis- Meimarakis & Koutrika (2023) for surveys on this topic.
Code Editing
Code translation aims to translate a piece of code (usually a function or method) into another programming language. The relation between code translation and cross-lingual code search is similar to the one between program synthesis and text-to-code retrieval, and SMT/MNT models have also been widely applied to this
task. One of the important applications of code translation is migrating old projects written in obsolete languages. However, we are yet to witness such applications at scale in the LLM era, as the context. window of even the most powerful language models are quite limited in the face of such projects. Malyala et al. (2023) provide a short survey on this lask from the SE perspective.
Program repair, also known as bug fix, aims fix piece of buggy code. Like code translation, is a traditional sequencce-o--scqeeeee generation task, and surveys abundant on this topic (Gazzola et al.
2018 Monperrus 2018 Zhong et al. 2022 Zhang el al. 2023d Huang et al. 2023a
Code Suggestion
Type prediction aims to predict the type of variables in dynamic programming languages such as Python and JavaScript. Tt has been used as a paetingning objective for code language models (Wang et al. 2022e). where it is often simplified as a binary tagging task predict which tokens in the code are identifiers (Wang et al. f).
Identōfier prediction is the lask of predicting identifier names in the code. As these names are deemed to contain important semantic information, this task has been utilized for code summarization (Allamanis let al., 2016b), as well as pretraining code models (Wang et al. 20216 Niu et al. 2022). A special case of identifier prediction method name prediction.
Cloze test a recently proposed task for code understanding, after the risc of BERT-style pretraining. Duc to the unique semantics of programming languages, several keywords are often selected for this test, such min and max (Feng et al. 2020
Code Explanation
Code summarization, comment generation, and code documentation all aim to generate explanations for code to facilitate understanding of the code, but with slightly different emphasis. Code summarization generates a natural language summary of the given code, while comment generation generates comments. These two terms are often used interchangeably in the literature, and their intended audience usually developers. Code documentation, the other hand, is more structured, includes more detailed information
7

---

Published in Transactions on Machine Learning Research (09/2024)
Nou-neural meth--
Non-Tren Transfoofrre noural
Transformor- methods
Requirement
TransCader
TransCnder-S
1pSMT
(gguyn
TransCnder-IB Exatrantec
Development: Code: Seazch
Code
2015
TreeZ Tree
2015
Grammar Deima al...
hen
20/23
3DA-Trans Li
BabelTower WWe
plr2ic
2022
Translation
m1.
2014
Debugxing (Chen
2025
Co-
2015
Tron Jana al 20225
Ziu
Development: Editing
65 ml
2016
21022
Pan
2028
Beplain- Tamb-
Lang
2023b
Macedoo at m]
2724
ak
PP ct cl. 2016 Deep-
Fix
(Gopte eb o.. 2017
CURE (Jang ot 2031a DeepDe- bus
Dexlin of Rooh
Drein e (202) BIFI (Y
Synt xx (1hato et [oo
aunaso & Jang 20/2 1. Recorler 12hu
Code Repait
2019b
Prophat Rinard
Lary
Tear
Lest et
2013
Reinc-
Long &
2022b
202
Codit
palr
nolc ci
Taccnaga ago Llang
Tulano e5
1 Tulano et aL.
[Cleon cl
(203)
Di
2015a
20195
at
2021
2021
Palt
Repair
XXta & Zhong 20121) RING (Jobhi
2021) "TP (lerahi at
20123 Xin et (2024)) Val-
Pan et aL (2022 Alplate
Mouts (Chpkmabsty & Key
PPu al 2022) Codel's
CotoNo (Latel
DLR
nt
Consereational
Ier at 2020
Zaanz
Pall
Regaar
bamppe
Parasaz/mn
2022g
Deep
doorn
Mangat
TypeHert Jeese 2023
TypaWriter
TypeWoaum
Type Prodiction
[Caomae
Gubn 2024]
TypedPy (2ir
HiTyper
[Ten
PYInfer
NLP in SB
Aliemania
Development: Code Suggees toon
GGNN
Allamanni
2013
Codo2Vec [Alo
JSNice al. 2016 Pi-
Kaychey
2019
Cods28eq TAlop
Idonsifier Prediction
2019
2019
2TPR [X
Perbaadez
CTran Kuans nam 2022
GTNM
Liu et a. 202215
2015 HrM Jhnba
2019
Liu cl
Matio ( Woong at 2023c
At ml.
2019
+00
MTV Ngnyen
h7 A/
1070
DeepName
05 m.
20211 20216
DMACO8 Xie
05 02
NamFat
Yook
00
2022
CodaBERT Pang al
2020
Close Teat
2021
Souss Hawee
2021
Comment Generation
2015
Auto AntaComeent
Woog al.
2013
Clo-
Words
A Fard
DempCom
APIComma 2200
20233
2018s] Khahbaxi
DOME
LAMNER SSharmx
GTrran (Kunng
2024
ATESLA Pan
Min
Than33
Wane es
2020c
2020) Cerier (Bat
220211
Tonz
Cade
Ai 2015
Atrancii [Led al.
202210)
Development: Code Exephana-
2019 [el (201H))
Haque (2000 DMA
tion
SUM Woag 2021e]
Bansil
CDS
M1.08
Mi 20121
20V1
70240
LoCo-
20241
McMhlaa
Hnldar
Analysle
LLM Evaluesion
Decument
Translation
Figure 5: NLP applications in the development stage of SE.
about the code's usage and functions, and is usually intended for broader audience. provide survey code sunmarization.
Zhang et al. (2022)
8

---

Published in Transactions on Machine Learning Research (09/2024)
Document translation is the automatic translation of code-related documents. Since models, datasets, and prompting strategies for machine translation are abundant in NL.P (Vaswani et al. 2017 Goyal et al. 2022 He et al. 2023b), we do not go into detail about this task.
2.1.3 Testing and Analysis
The third stage in software development is about testing and analyzing the correctness of programs (as well as other properties, such as security). Related works are listed in Figure
Testing
Unit test generation aims to generate unit tests for a given program. Prior to the rise of Codex and other code LLMs, almost all works in this area employed non-neural methods (see Figure G In the age of LLMs, however, this task is ever more important, LLMs' program synthesis capability may be insufficient (Luu et al. 2023f Wang et al. 2023b provide comprchensive survey on software testing with LLMs.
research has shown that the current unit tests for evaluating
Assertion generation is a subtask unit testing. Given a program and a partial unit test, this task aims to generate assertions (also known as oracles in software engineering) within the unit test. This task has gencrally went unnoticod by the NLP community, as the program synthesis task usod for evaluating LLMs often concern standalone, competition-style methods, for which the simple assertion of the equality bet ween program output and expected answer suffices.
Mutant generation aims to generate mutants of a given program for the purpose of mutation testing, and relates closely to unit test generation and assertion generation. A mulant that is not detected by a given set of unit tests and assertions indicates that either additional test cases or better assertions are required (Fraser
& Arcuri 2011h. Recently, masking out tokens in the source code and sampling them from the output of masked language model has become a common method for this task. Ojdanic et al. 2021 2023) give empirical comparisons between different mutation methods.
Fuzzing is another software testing lask, where the objective is to generate large set of inputs covering many corner cases as possible. While many recent works on fuzzing target deep learning libraries, few have utilized language models to conduct this process (see Figure
Analysis
Defect detection, or vulnerabilily delection, predicts whether the input code is buggy or not, and is a standard single-sentence classification task. Nong et al. (2023); SSeenhookk et al. (2023a): Bi et al. (2023):
Harzevili et al. (023)] Zhou et al. (2024) provide surveys this task.
Malware delection is similar task to defect detection. However, malware differs from other types of vulnerable code in that the bugs therein are malicious- - i.e. they are intentionally injected. We refer to Sahin & Bahtiyar 20200 and Gopinath & Sethuraman (2023) for reviews on con-Transformer based met for this task.
Clone detection predicts whether or not two picces of code are clones of cach other. In software engincering there exist. four types of code clones, and the most challenging type to identify is semantic clones, i.e. syntactically dissimilar code that have the same functionality. As this task can be viewed as two-sentence classification task, BERT-style language models have been widely applied to it. [Svajlenko & Roy (2020) and Zhang & Sakurai 2021 provide comprehensive reviews on non-deep-learning based methods for this task.
Code classification aims to predict the functionality of a piece of code within a predefined set of labels. A very similar task is author identification, which predicts the author of the input codc. Both tasks are standard singlo-sentence classification tasks, and traditional machine learning methods have been widely adopted in them (Kalgutkar al. 2019), while pretrained language models have seen almost no application.
9

---

Published in Transactions on Machine Learning Research (09/2024)
Nou-noural methodss
Trensformer
Trensformer- methods
Tufono
2022a
Geacration
Assection
MaMo Blur al.
2021
Aclas Wasson 2020
TOGA U3lne
PSLM
2022
Yano
20235
Dust 2014
LAVA (Dolan-
Caxis
Tufano al.
Requirement
Mutant Geacration
Hols wild-catpht
Taats Eizewn o1. (2017] Apoc-
alypoc [oo st. 2018
Collrr (Powny
Bug-
Fano
Seweeeo (Polce &
Mtstation 13
[2011lc] Deup-
2020
2022 Khantr
PSLM
ABRIV
[Haret
(Deglovantt & Papariakis
Development
1B1R (K [Khnn/it <1 00-00 12255
lajecto: [aasyyn
2019
Pradel 202
PBM [Bartord
2023
2021a
AthensTart TTtamm all
PSLKI Sarnth
Tessing
TassPilot Scchsler
6023
AsTeut (Angarsamy
2025
TeCo [N15
EvoSuise Froaer & Areuti 2011
2023
CounMosa [Lemnie ux
EvoSuisek ( -hamshhr 2015 Di-
2023
ChaiTester Yuug
Unit Teat Generation
2xMO5A (1 Panichali ot al
2018
202235
ChatUn//arr OXte
LambdaThetar ser ISnlakovicc et Al.
20233
(Tagg et
(20255
2015
TSE Chimnn /e Kahrtal
GP1
Vikzamm ci n.
22023
Baai
2022
Neasie Arteca cl al 20122
05 0.
2025
M. (aache
+ Buc
2025
Zhans et al. 20201
X150
Stcemtocc
20230
SymaFuxs
Che et al 2011))
Guilber mi
Vincene
APLPast
Boome es And (201
LoverUp
1xxorpo Bersep
FaieFuze
Lemucus < 2c11
2024
TELFA
Yang ct
2015
AFLCi Bookie ct:
2017
2018
Angora (Chon & C bom
Tenoor Puss (Ouepa
NEUZZ
Sbe
2015
Aadine (Cho
P20200
TitanPosz DDon es 2023 PoxrAAll
Poszing
LEMON
WWang <t N220530
2023
White:
DorTTer
eln (0220 mppee
k022
ml.
Yang al. 2030
Pusaot
Monns
Deeps
(Dong
NLP In 393
Yang
Wang
LineVal
NXR-
Yesa
0032
PParree
2015
20228
Rtaeal
Boglas Allamanni
(Chakraborty
Uiahh
2022
MloCoLD
(San sl
ELARTVRR
Map
20220
Cun
202
IVDntect
Steenhonlc
20
RoDVD
(Nguyen
2022
Operations
Depleyment
Analysie
Clona Dutection
Pt al 20182)
Ht xl
2020s
SAGA
kawa 05 al. 2025
CCAigge (wang
Deckard Tlang
TXKKopeer Wi
2007
2020)
L at al
al
AL Nakka-
201b
Eaj-
2015
White
2015
2013
2019u
Fank 0t a..
C1.C1'
PA-AST
2022]
2017 Oreg
Despsim Zano Hunng
ASTNN (Zannx
TBCCD (Ya
Pores & Clion
(Wang
(Nal at 20113
20.20
2018 CDLR (Wel
B.. 20200
Valyy 4c Kim
sini
[000
Khajerade et al
SaCD Clocaloy Doy
Dom et al
CCT-L-
Malbert
2023x
20
Xu
500 okm
Lal al.
2022
Yoan
b021b SHER.
2023
20256
(205
LOCK
Evalsasion
Malware Dutection
MSDT
Fire 20229 Mall
BERTv Mabail Akaloun 20123
BocketAl Scanner
Bandaca
Wijyyarthnn 2013
2024
Author
SCAP
Srantzeskou Mab-
2022
2013
Inxt2vwc
DS
ASTRN
Abubamad
Port
20222
2021
Wang
2019
InferCocn (Bmr
2021
Figure 6: NLP applications in the lesting & analysis stage of SE.
2.1.4 Deployment and Operations
After software passes the testing stage, it is deployed into production, and often continuously maintained or updated. For this stage, we summarize the specific tasks and their related works in Figure
Deployment
10

---

Published in Transactions on Machine Learning Research (09/2024)
Non-moural motth-
Non- Transforrrr
Requtrement
Compiler Optimisation
Decp 2017
Decy/Tune Tune Camminn ct ul
Armengal Patapé O
1202
Samplax (omb gi Grubisic
2024b
Development
2024n
Testing Analysis
ВTC Hosoiiii & Dolin Dolen-Cavilt Cavill
Decompilation
NeurDF
Neatzom
LaD 66 4h
Flap Al
4024
2021
A 2023
2022
Nova
Dea Wr Weng
A Armengg -lat sp
Jjans at dm 2lrei
Deployment
DeGuard Sicchal
BEA
Decompil Tinn
al 20
Desbitscation
Tonvin
2015
HT al 20
bi [1n
22
Voer leeou
xl
15m
Artuso al 201
DIRE LLanmmte Al.
2019
LmFa
VarBERT (Baneroc
DIRECT
XB cl al. 2025
Nitio al
2021
Neal TTon cl 0]
2015
NuLog
Nedelkork es
2020a
Logzy
Nedellord of
2020
NLP in SE
Spall
Drain
Deea or
2017
Mens
LoxStump
Neurall og
LegliBRT
L3o 0t
Le Aitanm 1022
Fso
Log Analysia
LogRob
Zuung
LexPPT
Zhans
Scmllo
UniParser
LagGFT
1.t.A
Parser
Mix
2020
Review
Code Review
bwak
2015
Ichandran
Cinange
Ticcodder
al
Tarier 20/2
20
Cluster-
[Barnest
[oong
Crmm-
2018
DCR (Gupta & Sundaresa
CORE
Despitvicy LLt
Slow et 20/20
DACE (lu 2013
N022k AUGER [Li ei 20220
Plfanoo 2022hb
SBERR
Joorn
Reviewer
W0W2b CodeReviens Keieeer
(Yin
Le al 201238
2223 LLaMA
Hellen
H1 m
Operatious
Workflow Automation
Meata
2023
Zhang al.
2024c
F&M Restuation
ChangeScribe (Corte
Lovoln
2017
Jiang
Vitegrea
(20
120
Coreflen Nie al 2021 Coom
Commit
2015 Diacs 6
2019
mi:BERT Dack 2021
RACB Gaa
Gemera- tton
Mchilln (O1T)
CoDišum N
201
2022
FIR (Dong ol
NNtonn (Gu
ATOM LLu ct 20220
2022
CONE Be
20/25
Mt al 10
CC2Vec
Hoang at al 20120
BscroP: (Wang al. 2030
Usanc Huang
CoRec
[Wang ct mlr 2021c
Ht ml 20/20
Figure 7: NLP applications in the deployment & operalions stage of SE.
Compiler optimization is the task of optimizing compiled code to increase its efficiency. Many early works applied reinforcement learning and other machine learning technologies to this task by optimizing task scheduling or pass ordering (Wang & O'Boyle 2018 Leather Cummins! 2020), and is only recently that researchers started to view it as a sequeence-lo-seqqueeec generation task with the help of powerful LLMs (Cummins al. 2023).
Decompilation is the reverse process of compilation, and important topic in reverse enginecring. In this task model takes compiled program represented in assembly code or binary machine code and aims to output the original source program in high-level languages such as C.
Obfuscation refers to the process of renaming identifiers (e.g. variables, methods, and classes), for example generic names like var_1, var 2 or x, y. an important technique virus detection, intellectual property protection, and code size reduction (Collberg Thomborson) 2002 Murad et al., 2010 Vasilescu ee al. 2017). Deobfuscation refers to the reverse process, where meaningful identifier names recovered from obfuscated programs. Obfuscation can be easily achieved statically, but deobfuscation has been
subject of more interest in recent years. It plays significant role in decompiling, and has also been adopted
as a pretraining objective for code language models (Lachaux et al. 2021 Ding et al. 2022a Liu et al.
2022:]
Log analysis aims to analyze the system logs producod by software products, for example parsing logs into structured templates or finding anomalies from raw logs. Zhu et al. (2019) provide survey on traditional
methods for this task up to 2018, and Chen et al. (2021)) give empirical comparison between neural
11

---

Published in Transactions on Machine Learning Research (09/2024)
network based methods. Zhang et al. (023c)) also cover more recent methods for log parsing, while Landauer et al. 2022) survey methods for anomaly detection in logs.
Operations
Code review aims to automate the process of peer code review, and includes many subtasks, such review necessity prediction, review comment generation, code refinement, and review decision prediction.
Commit message generation aims to automatically generate commit messages for code changes. This task takes the code before and after change as input, and output the description for the change. This can be viewed as the dual task of program repair, as many code changes and their accompanying commit messages concern bug fixing. Tao al. 22011) provide a survey methods and datasets for this task up to 2021.
Workflow Automation is the automation of workflows in Git. Due to the complexity involved in this task, it was only recently brought to attention after the advent of powerful LLMs.
2.1.5 LLM Evaluation
Apart from the previously covered stages in SE, we observe that programming-related evaluation of LLMs has been gaining momentum in both the NLP and the SE community since the release of Codex. Thus, we also list several novel tasks that appeared recently for this purpose. We discuss this topic in more detail in iontion 7
Development
Testing Analysis
Code Reasoning
MMLU
C-Eval Hoang
Hendrycks 202
2021b
CodeQA
CMMLU
LLi & Wan 2021 CSIQA LTve al.
CodeApex
al.
NLP SB
CRUXEVA
Pythou10
Deployment Operations
MasksqA-P (Austie et al. 2021 Drosi & Verma 2021
Droti et
2022
Coding For Reasoning
PAI. (Gan et al. 22223 Fol
hen at al. 20233 Ri
Deys
at mh.
22023
GS
Python (Chowdhery ct al. 2024
Wang al. 2023h
CoC
L. Gl.
2023a Think.
Lnd-Exneute (Chae
LLM Reslustion
Vulnerabilitie
Schuster es al.
Perry et al. 2023
2021
Hamer ot al
Asare et al
2023 Sandoval et al.
2023
Correctness
Nadi 2022
2023
Yutistiren
2023
AIGC Analysie
AIGC Detection
TetectGPTTCoad
Yang es al,
2023e CodelP Guan et al.
2024
Understandsblle
Others
2024
Hallucinations
2024
Efficiaccy
2023
Lonk
Waan
Figure 8: Programming-rolated evaluation of LLMss
Code reasoning requires model reason about code or algorithms, and answer related questions which are written in multiple-choice format free-form QA format, which may range from conceptual understanding to numerical calculation and complexity analysis. It often comes subset of composite evaluation benchmarks such MMLU (Hendrycks et al. 2021b).
Coding for reasoning is special case of reasoning where LLMs solve complex reasoning problems by generating code that will be executed by external interpreters. This task abstracts the reasoning process from numerical caleulations, and is thus of special interest evaluating LLMs. More recently rescarch has also shown that asking the LLM itself simulate the execution of generated code can also help its reasoning process.
12

---

Published in Transactions on Machine Learning Research (09/2024)
-AIGC' analysis aims to analyze Al-generated code from various aspects, such as correctness, vulnerabilities, and detection of Al-generated code.
2.1.6 NLP Point-of-View
Unlike software engineering, evaluation tasks in NLP are generally categorized into understanding and gen- eration. The former, represented by GLUE (Wang et al. 2018a) and SuperGLUE (Wang et al. 2019).. emphasizes the comprehension of input text, and is typically formalized as classification, regression, se- quence tagging, or span extraction. The later, on the other hand, involves autoregressive generation of text, such machine translation and summarization.
Among the previously listed tasks, code synthesis, code translation, code repair, deobfuscation, unit test generation, assertion generation, mutant generation, code summarization, code review, identifier prediction, and commit message geneartion are requence-to-sequene
generation tasks. Formally, each instance of these tasks has a source sequence x (e.g. piece of source code) and a target sequence y (e.g. its corresponding summarization), and the language model is tasked to maximize the conditional probability given by ( where 0 can be either decoder-only model concatenated. In the later case, x is processed by the encoder and y is processed by the decoder.
encoder-decoder model. In the former case, x and y are
Code completion and code infilling are also generation lasks, but differ from requence-lo-sequenee lasks where the imput and output are related by different sequences. In these two tasks, the target a contimuation or infill of the input. They correlate closely to the language modcling objectives given Equation (3) and (5) Similarly, cloze test takes the same form as Equation (4) but is usually considered an understanding task, as its output is usually single token and does not involve autoregressive generation.
Defect detection, malware detection, clone detection, code classification, and author identification are se- quence classification tasks. In these tasks, a set of labels y is defined over the input, and each instance is assigned a label y E y (e.g. for defect detection y = {0. 1}, while for author identification a possible y is {Alice, Bob, John, others)). The model is then tasked to maximize
po()))).
(1)
Type prediction is a token classification task, also known as tagging. In this task, each token 2; is assigned a label y/ E v, with an example y being {int, float, string, bool, non-identifier, other]. The model's objective is to maximize
n
II po())))).
(2)
The last two tasks- code retrieval and code search also belong to understanding tasks. In these tasks, each source sequence x is paired with positive target sequence y and a set of negative targets y E (yi, ).s. The model's task is to find similarity metric such that s(x, y) is larger than s(x,
2.1.7 Code LLMs for now-Resource. Low-Level, and Domain-Speciffe Languages
In NLP, human languages are categorized into high-, middle-, and low-resource languages based on the amount of available data in each language. High-resource languages such as English are extensively studied, while low-rcsource languages such as Swahili and Yoruba often rely transfer learning from other languages
to improve performance due to data scarcity (Conneau el al. 2020 Xue el. al. 2021 Zhang et al. 2023g).
target the most popular programming languages such as C, Java, and Python. However, a major difference between NLP and SE is that in NLP, low-resource languages are often spoken by few people or even endangered, while in SE low-resouree languages are often designed for specific domains and purposes, and thus have an active user community, Verilog, a hardware description language, is one such example, for which code modeling research
This phenomenon also exists in code modeling, as most works listed in Figure 3 8
2In the AI community " "AIGO" usually refers to AT-Generated Content. In this work overload this expression with AT-Generated Code.
13

---

2023 Thorat et al, Published in Transactions on Machine Learning Research (09/2024)
is quite abundant (hakur al. 2023a b Lu et al. 2023b Liu et al. 2023g Tsai et al.
2023
Liu et al. 2023h: Nadimi & Zheng. 2024).
From a different perspective, most of these popular languages are imperative languages, while few works have studied NLP and LLM applications in declarative or functional languages, except those concerning SQL and one recent. work on Haskell (van Dam et al., 2024). However, research has shown that declarative languages are more aligned for optimization since they describe the "what" and not "how" in programming (Marcus et al.
2019 2020), thus they worth more attention in the future research.
Similar to NLP, many works in code modeling have studied the transfer of model capability across languages. Chen et al. (2022), for example, investigate the performance of multilingual language models on Ruby (which is one of the six languages in the popular NodeSearch pertaining corpus (Husain et al. 2019) but relatively low-resourced compared with the other five), finding that multilingual models have lower performance-to-tinee ratio compared with monolingual oncs. Cassano ct al. (2023a). on the other hand, propose MutiPL-T, an approach for translating high-resource training data into low-resource languages for fine-tuning. However, such research is yet scarce compared with the NLP community, but with the recent advent of colossal, multilingual datasets such as The Stack v2 Lozhkovv et al. 2024) we expect to see more of it in the future.
2.2 Evaluation Metrics
Of the tasks mentioned in Section 2.1 th understanding tasks are similar in form to natural language understanding tasks (Wang et al. 2018a 2019) and evaluated likewise by metrics such as accuracy, Fl and Mcan Reciprocal Rank (MRR), while short generation tasks such as identifier prediction is also evaluated by accuracy of exact matches. Code-to-text, tasks are evaluated with common metrics for text generation such as BLEU (Papineni et al., 2002).
Evaluation of tasks involving code generation, the other hand, is more complicated. Most early works evaluate syntactical correctness, i.e. the percentage of generations that can be successfully parsed. Chen ct al. (2018) argue against such metries and suggest reference match instead, which is the percentage of generations that are exactly the same as the references. Ren et al.] (2020) propose CodeBLUE, a variant of BLEU that takes code syntax and semantics into account by evaluating the overlap of abstract syntax tree (AST) and data flow.
As code generation models became more capable over the years, however, these metries based on content- overlap have been found to be inadequate (Rozière et al. 2020 Hendrycks et al. 2021a Austin et al. 2021 since functionally equivalent snippets of code can differ dramatically in their lexical forms. Consequently, rescarchers have turned their attention to functional correctness. One popular example of such metrics is passak, proposed by Kulal et al. (2019) and refined by Chen et al. (2021b), which is an unbiased estimator of the model's chance in passing all unit tests of a program with any of & generated samples. This metric can be generalized to passn@k (Li et al.] 2022h), which limits the number of model submissions to n but allows filtering by unit tests given in the input from i samples.
2.3 Program Synthesis
While dozens of evaluation tasks exist software enginecring, they have generally stayed out of the focus of the NLP community until very recently. The only exception is program synthesis, which has become a standard evaluation task for LLMs since the advent of HvmanEval in 2021. Looking back at this task, we identify four changes in program synthesis over the years: shift of coding paradigms (from example-based to intention-based), generalization in languages (from domain-specific languages to general-purpose languages), simplification of model architectures (from grammar-guided decoders to general-purpose language models), and application of execution- based feedback.
Many of the early methods for program synthesis are example-based MMenon et al.] 2013). which means they induce programs from input-output examples, often in domain-specific languages (DSLs) such as Flash-
14

---

Published in Transactions on Machine Learning Research (09/2024)
Fill (Devlin et al. 2017a] and Kare
(unnel et al. 2018 these languages are usually simple in syntax
and structure.
As code generation models became more capable over the years, researchers started to pay attention to program synthesis in general-purpose programming languages as well. Hearthstone (Ling et al. 2016 and CONCODE (Iyer et al. 2018) are two of the early datasets, representing Python and Java respectively. Each example in Hearthstone is the description of card in the game and its corresponding class implementation, while examples in CONCODE are simply Java methods paired with their natural-language documentation crawled from public GitHub repositories. Synthesizing programs from their corresponding natural language descriptions has since then become a standard practice in program synthesis, and has led to some of the most widely used benchmarks, such as HumanEval |Chen et al. 2021b), which has even been translated into multiple languages (Cassano et al., 2023b Zheng et al. 2023a, Muennighoff et al. 2024). Some recent benchmarks use general-purpose languages but focus on specific domains, such as data science
(Bavishi al.,
2019 Lai ct al. (023) or Jupyter notebooks (Agashe et al. 2019), while several math reasoning benchmarks
have a als been converted to programming lasks, including MathQA-Python (Amini al.
2019
Austin et. al.
2021 and GSM8K-Python (Cobbe et al. 2021 Chowdhery et al. 2023 Wang et al. 2023h
Many early works argue that simply treating program synthesis as a text generation task does not utilize the underlying syntax of programming languages, and thus often use syntax-enhanced decoders to inject the target syntax as prior knowledge (Yin & Neubig 2017). LLMs, however, have demonstrated that pretrained language models are capable of generating syutactically correct programs without loss of generality. Under this setting, researchers start ezecute the generated programs and provide feedback to the generation model to inject the prior knowledge of code instead. This has recently led to the popularity of interactive coding, which we discuss in more detail in Section 7.1
2.4 Repository-Level Coding
Most tasks discussed in Section 2.1 are limited to single file or even single function, cross-file code modeling poses challenges that are beyond the capability of most existing language models. Recently, however, position interpolation techniques (Chen et al. 2023d Rozière et al. 2023 Peng et al. 2023a have extended the context window of LLMs to hundreds of thousands of lokens, making it possible to contextualit lize
coding activities within entire repositories. Several works have studied code completion (hrivastaaa et al.
2023b Ding et al. 2022b: Zhang et al. 2023b Shrivastava et al.
2024 Wu et al. 2024b) 2023a Phan et al.
and generation (Liao et al.) 2023 Zan et al. 2024) leveraging repository-level context, and corresponding
benchmarks have been proposed Liu et al. (2023j Ding et al.
(0033)
Li et al. (2024a). Bairi et al. (2023)
investigate the more challenging tasks of repository- level API migration and temporal editing, while Jimenez
ct al. 2023) introduce related benchmark, SWE-bench.
3 Language Modeling Preliminaries
As code is ultimately a subset of natural languages, language models have been extensively used to tackle the tasks listed in Section 2 Before diving into the language models themselves, first briefly review the preliminaries of Traasformerr-bseed language modeling in this section following the common choices of training objectives, and also some implementation designs.
3.1 Causal Language Modeling
Unidirectional language models (also known as causal language model factor the probability of sentence into the product of each token's conditional probability with the chain rule. A piece of input text x =
FFlshhFill used in Microsoft Excel for string transformation. Karel is simple programming language for educational purposc.
"The training objective of such language models Causal Language Modeling (CLM),, but also referred as Next Token
Prediction.
15

---

Published in Transactions on Machine Learning Research (09/2024)
[11 P2,11 ,.nl consisting of tokens is modeled as
P(x) (2z x1s-1);
(3)
where X is a shorthand for tokens before in the input, and 0 is the parameters of the model. With Transformer decoders such as GPT (Radford et al. 2018 2019 Brown et al., 2020) and LLaMA (Touvron
et al. 2023a b), the conditional probability in (3 is modeled by adding an attention mask to the attention matrix of cach Transformer block, ensuring that 22 can only attend to previous tokens. During training, the cross entropy loss on all tokens the input is calculated in parallel, while at inference time each new token is generated antoregressively, For further details about the Transformer architecture we refer to Vaswani
ct al. 2017))
3.2 Masked Language Modeling
Unlike cansal language models, bidirectional language models are trained to acquire a better contextual representation of text. rather than generating text autoregressively. Tn the vanilla Transformer, the en- coder part is allowed jet al.) 2219)) takes one step further and pretrains only Transformer encoder. A set M of randomly cho- sen tokens in the imput replaced by a special token [MASK] to obtain noisy input &, for example [[CLS] 21, [MASK], 23, [MASK], ,Ts, [EDS] y and the model is trained to recover the original tokens by maxi- mizing
attend to a token's left as well as right context for this purpose. BERT (Devlin)
II MEM
(o(m 8).
(4)
While this objective requires the model to have a deep understanding of the input text to reconstruct it, it suffers from low training efficiency, since only a small set of tokens (usually 15%) are masked (and thus "trained on"). To address this issue, Clark et al. 20200) propose ELECTRA, which is trained to discriminate whether not each token in the input has been replaced by BERT-like model instead, thereby computing loss on all input tokens.
3.3 Denoising Objectives
GPT-style causal LM and BERT-style bidiroctional LM each has its own strengths and weaknesses. While GPT can be used for autoregressive generation, it lacks bidirectional representation of input text, and is thus unsuitable for sequence- cocetcceett-seeupeee te-euueeee (seq2seq) generation tasks such translation and summarization. BERT, on the other hand, can produce bidirectional representations, but is pretrained only for mask filling, not generation.
The vanilla Transformer encoder-decoder architecture combines the respective advantages of GPT and BERT. T5 [Raffll et al. 2020) is such a model pretrained with span corruption, which can be regarded as a variant of MLM. During pretraining, spans of text in the input are replaced with sentinel tokens, which play the same role [MASK] in BERT. The noisy input is first processed by the encoder with bidirectional attention. and the masked spans are then generated autoregressively by the decoder. Formally, if k spans are sampled for corruption in input x, the noisy input & is then constructed by replacing each span with a special token <extra_id_ ____ddi____________________ for i = 1,2,--. ... k, and the target is constructed by concatenating all spans prepended with corresponding sentinols: bextra_id_1x span1, "., <extra_id k>, spang]. The model is then trained with a standard seq2seq objective, by maximizing
po(y II po po(u)(2
(5)
Both (CLS) and [EDS) are artificial tokens added the input text. [CLS] is added the boginning and its representation used for sentence classification, while [EOS] indicates end of sentence. The original BERT also another special taken [SEP], which is not in common use LLMs, and refer Devlin al. (2019) for details.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Block Architecture
Position Embedding and Interpolation
Pre-normalization Parallel Attention
1
2
Cosine Learnable Relative RoPE/ALiBi RoPE Interpolation
Mixed Precision Training
Attention Optimization
4
3
FP16 BF16
MQA, GQA FlashAttention
Figure 9: Major implementation changes LLM over the past few years.
Lester et al. (2021) show that models pretrained with such objectives can be adapted for autoregressive language modeling with extra pretraining using the prefix language modeling objective, i.c. splitting the text into two parts, processing the first part with encoder and generating the second part with decoder.
Tay et al. 2023b) argue that span corruption is also closely related to CLM, since one can mask out the whole input text as a single span and train the decoder to generate it antorogressively. Inspired by this relation, they propose UL2, which the combination of many span corruption objectives that differ in corruption rate and span length. Applying it to both encoder-decoder models and decoder-only models, they find that encoder- decoder models perform better under the same computation budget constraint. Other researches have also found that such encoder-decoder models generally perform better than causal decoder-onke models (Wang
et al. 2022c Soltan et al. 2022
3.4 Auxiliary Objectives
Language modeling objectives, such as previously discussed CLM and MLM, mainly train the model to cap- ture token-level information and are ineffective at modeling document structures. Thus, auxiliary objectives are often added to help the models learn such global information. BERT pretrained with next. sentence prediction (NSP) along with MLM, which is formulated segments in the input are neighboring in the original corpus. Lan al. (2000) propose more challenging sentence-order prediction (SOP) task, where the negative samples constructed by swapping the order of two neighboring sentences instead of sampling random sentence from other documents.
binary classification task to predict whether
Relatedly, Raffel et al. 2020) mix supervised downstream samples such as GLUE (Wang et al. 2018a) into T5's pretraining dataset to conduct multi-task pretraining. However, it is worth noting that since they unify all tasks into text-to-text format, the training objective the same for their self-supervised pretraining and supervised downstream fasks, iee. conditional generation of the target sequence.
3.5 Implementation Design
While most researches on pretraining language models have focused on designing novel training objectives, low-level implementation of the Transformer architecture itself is also being continuously improved over the years in pursuit of stability, performance, and efficiency, shown in Figure g
The original Transformer block proposed by Vaswani et al. 2017 is formulated as
LN(ALtention(s) +), LN(FFN(H) + h).
(6)
(7)
where x is the layer's input; y is the layer's output: " "Attention" is the self-attention sublayer; "FFN" is the feed-forward sublayer, and *LN" is layer normalization (Ba et al. 2016).
17

---

Published in Transactions on Machine Learning Research (09/2024)
GPT-2 (Radford et al. 2019 moves layer normalization to the input of each Transformer sub-block to stabilize training:
h = Attention(INN(x)) +
(8)
= FFN(LN(A) +
(9)
and such pre-norm has since become standard practice in Transformer decoders.
GPT-J [Wang & Komatsuzaki, 2021 modifies the Transformer block to compute FFN sub-layer and self- attention sub-layer in parallel to increase computation throughput:
y= x (F)(LN((:)) (LL)n)ion(LN(()));
(10)
and Chowdhery et al. (2023) observes limited performance degradation when applying this design to larger models.
As self-attention alone cannot capture the position information of each input token, the vanilla Transformer adds a non-learnable vector that's dependent on the absolute position in the input sequence to the embedding of each token, which is known as cosine position embedding. Later works such as GPT and BERT use learnable embeddings instead, while T5 adopts relative position embedding (Shaw et al. 2018), where the information of relative position between query and key is injected in the computation of self-attention instead. A more recent scheme RoPE (Su et al.,
2024) multiplics the keys and queries by position-dependen rotation matrix, and is later shown to enable position interpolation for processing of longer sequences (Chen et al. 2023d Rozière et al. 2023 Peng et al. 2023a). Alternatively, Press et al. (2022) propose ALiBi, which directly attemuates the attention scores according to the relative position between key and query. RoPE is popularized by its application in PaLM (Chowdhery al. 2023) and GPT-NeoX (Black et al. 2022), while
ALiBi adopted by BLOOM (Scao et al., 20022. We refer Zhao al. 2023 for a survey on position embedding and interpolation in Transformers.
Apart from position embeddings, another issue in Transformer that has long troubled researchers is the fact that the complexity of self-attention scales quadratically with the imput sequence length. Some works such as Reformer (Kitacv et al. 2020), Linformer (Wang et al.) 2020ch, Performer (Choromanski et al., 2021 and cosFormer (Qin et al. (022bb use approximate attention to reduce this complexity, but they mostly come at the cost of degraded performance (Tay et al. 2023a Lin et al., 2022). Other works tackle this issue from an engineering point-of-view. MQA (Shazeer 2019) shares the same of keys and values across all attention heads to optimize nemoorytoo-rrtthmeti ratio and significantly improves inference speed at small costs of model performance. Its variant Grouped-Query Attention (GQA, Ainslie et al.] 2023) takes a middle-ground approach by dividing attention heads into groups and sharing the same set of keys/ haassvvaues within each group. Orthogonally, Dao et al. 2022) introduce FlashAttention, which an exact but improved implementation of self-attention that optimizes 10 operations on the accelerating device via tiling to improve memory efficiency.
Another important technique for improving LLMs' training efficiency mixed precision training (Micikevi- cius et al. 2018), which stores model weights and activations in lower precision to save memory consumption. Early works use FP16 format for this low precision, while BF16 has now become more popular (Chowdhery'
ct al. 2023 Scao et al. 2022), as it can represent the same range of floating point numbers as FP32, thus preventing numerical overflowing and underflowing during training.
4 General Language Models for Code
Since language models scaled to hundreds of billions of parameters (Brown et al. 2020 Chowdhery et al. 2023), many of them have demonstrated non-trivial coding capability, even they are not specifically de- signed or trained for code. Pioneered by Codex, researchers have also found continual pretraining on code significantly benefit. language models' performance on code
While some works refer to this process "finctuning on code", it is still self-supervised in nature. Thus we choosc to adopt the term aaataa//ddiiiona//ontinna pretraining" this work to avoid confusion with supervised in-task finetuning or instruction finetuning.
18

---

Published in Transactions on Machine Learning Research (09/2024)
4.1 Off-the-Shee Language Models
Large language models often pretrained on tril- lions of tokens following the scaling laws (Kaplan et al. 2020 Hollmann et. al. 2022), and such an amount of text data is often a diverse com- posite with a non-negligible part of code. The Pile (Gao et al. 2021), for example, includes 95GB of code crawled from GitHub out of its 800GB raw dataset, while the multilingual pretraining dataset ROOTS [Laurengon et al., 2022) also contains 163GB of code spanning 13 programming languages in its 1.6TB compound. As two of the largest open- source pretraining datasets, they have supported many language models with coding ability. GPT-
Table 1: Pass@k performance of raw language mod- els (lop) and language models with extra training on code (bottom) on HumanEval (0-shot) and MBPP (3- shot), ordered chronologically. For Phi-1.5 we consider
Phi-1.5-web version, and for Code LLaMA we consider
its Python version. CChen et al.
b2021b)
Chowdh-
ery et al. 2023 ;; 3 Austin et al.
(021
Scao et al.
2022 Touvron et al. (2023a
Open (0228)
Bubeck et al. (2023): Touvron al. al.
(2023b)
(2023j); Yang al.
(2023a):
Bai
al.
2023
Jiang al.
(2023a);
Anil al.
2023a
[Wann & Komatsuzaki 2021), for example, ro-
DeepSeek-At Al et al.
2024
Jiang et al.
2024a
ported by Chen al. (2021b) to demonstrate non- trivial performance on HumanEval, while (Scao al. 2022) report similar results for GPT-NeoX (Black ct al. 2022 and BLOOM. LLaMA (Touvron et al., 2023a whose pretraining dalaset includes 328GB code from GitHub, achieves 23.7 pass@1 perfor- mance on HumanEval, and its successor LLaMA (Tovrron al., 2023b). achieves an even higher
16
Dai ct al. (2024a)
Anil et al.
(2023b) 18
Rozière
et al.
[2023].
Size
HumanEval (0)
MBPP (3)
k=1
100
k=1
k-80
GPT-J'
6B
11.6
27.7
LaMDA
137B
14.0
47.3
14.8
62.4
PaLM
540B
26.2
76.2
36.8
GPT-NeoX*
20B
15.4
41.2
BLOOM
176B
15.5
55.5
score of 29.9.
LLaMA
65B
23.7
79.3
37.7
76.8
GPT-4
67.0"
/827
Closed-source models, on the other hand, perform generally better. LaMDA (Thoppilan et al.) 2022 and PaIM (Chowdhery et al.] 2023), whose pre-
LLaMA 2
70B
29.9
89.0
45.0
81.5
Phi-1.5
1.3B
41.4
43.5
Baichuan 2
13B
17.1
30.2
training dataset contains 12.5% and 5% code respec-
Qwen
14B
32.3
40.8
tively, achieve 14.0 and 26.2 pass@1 performance on HumanEval, while GPT-4 (OpenAl 2023) set staggering record of 67.0 (and an carly version re- ported by Bubeck et al. 20233 to be 82) that until recently has remained higher than any specialized models pretrained or instruction- inneructioo-fieeuueed for code.
Mistral12
7B
30.5
47.5
Gemini 18
Ultra
74.7
DeepSeek
67B
42.7
57.
Mixtral15
Sx7B
40.2
60.7
DeepSockMoE'6
16B
26.8
39.2
Codex
12B
28.8
72.3
PaLM-Coder?
540B
36.0
88.4
47.0
80.8
PaLM 2
$
37.6
88.4
50.0
86.8
More recently, the general trend has been Lo train smaller models with larger datasets, following the re- vised scaling law (Hoffmann al.) 2022). Baichuan 2 (Yang al. 2023a),, for example, is 13B model trained on 2.6T tokens, while Qwen (Bai et al. 2023 is 14B model trained on 3T tokens. They achieve 17.1 and 32.3 passo1 on HumanEval, respectively. Li et al. 2202jjp. however, demonstrate that models as small as 1.3B can acquire coding capability that's comparable to much larger models while also maintaining a reasonable performance on general text processing and even manifesting some emergent abilities (Wei et al. 2022c such as chain-of-though reasoning (Wei et al. 2022d). Their model, Phi-1.5, trained on 21B tokens of textbook data generated by ChatGPT, and 100B tokens of filtered web data from Stack Overflow and Refined Web [Penedo et al. 2023 and attains 41.4 pass@1 performance on HumanEval.
Code LLaMA18
34B
53.7
94.7
56.2
Code- Tede-Qwen!!
14B
45.1
51.4
Another rising trend is mixture-of-expert models [Lepikhin et al. 2021 Fedus et al. 2022 Du et al. 2022 Notably, Jiang et al. (2024a) recently introduce Mixtral 8x7B, where only 13B parameters are activated for each token during inference, but achieving 40.2 on HumanEval and surpassing much larger dense models such as LLaMA 2. Similarly, Dai et al.) (2024a) present DeepSeekMoE, where only 2.8B parameters are activated in a 16B model, scoring 26.8 on HumanEval.
The exact performance of these models are presented in Table
19

---

Published in Transactions on Machine Learning Research (09/2024)
4.2 Language Models with Additional Pretraining Code
Along with the seminal benchmark HumanEval, Chen (2021b) kick-started the age of LLM for code with Codex, which are GPT-3 checkpoints pretrained on 100B additional code tokens and one of the earliest multi-billion models for code. Following their work, other researchers have also specialized their LLMs on code with additional pretraining. [Chowdhery et al. (2023) train PaLM on 7.8B additional code tokens to obtain PaLM-Coder, setting new state-of-the-ard on HumanEEvl and MBPP (Tabbl 1 that are only broken later by its successor PaLM 2-S*, the smallest version of PaLM 2 (Anil et al. 2023b) further trained on an undisclosed amount of code. Similarly, [Lewkwwcz papers and mathematical content, while Rozière et al. (2023) train LLaMA (Touvron et al. 2023b) on more than 500B code tokens to acquire Code LLaMA, whose performance on HumanEval surpasses all previous LMs except GPT-4 (Table []. Liu et al. (2033b) further train Code LLaMA with multi-task finetuning (MFT) to introduce CodeFuse- ModdeFuseCoodeLJ..o MaMe achieving 74.4 pass@1 on HumanEval and surpassing even the performance of GPT-4 published in OpenAl (2023).
al. (2022) train PaLM on 38.5B tokens of arXiv
While almost all of these models are Transformer decoders pretrained with CLM, several architectural modifications have been introduced along the way, as we noted in Section 3.5 All these models use pre-norm, and GPT-J introduces parallel attention, which is later adopted by PaLM, GPT-NeoX, and Phi-1.5. PaLM introduces MQA and RoPE into LLMs, and RoPE is now employed by most language models, including GPT-NeoX, two generations of LLaMA, Qwen, and the 7B version of Baichuan 2. BLOOM and the 13B version of Baichuan 2, however, use ALiBi for position embeddings, while LLaMA 2 and Code LLaMA adopt GQA instead of MHA or MQA. In Section 5, we show that specialized models pretrained exclusively on code have also followed these advancements closely.
5 Specialized Language Models for Code
As pretrained Transformers such GPT and BERT achieved remarkable success in natural language pro- cossing, such model architectures, learning paradigms, and training objectives were soon adopted by the software engineering community to produce specialized models for code understanding and generation. In this section, we first review common datasets used for pretraining code language models (Section 5.1 and then dive into the complex family of code LMs by their model architccture: encoder-only models (Sec- tion 5.2). encoder-decoder models (Section 5.3), decoder-only models (Section 5.4), UniLM (Section 5.5
and diffusion models (Section 5.6 Lastly, in Sec- tion 5.7 we also illustrate the current trend of ap- plying more recent techniques in NLP, such as in- struction tuning (Wei et al., 2022b Sanh el al.) 2022 Chung et al. 2022) and reinforcement learn- ing Ouyang et al.
Table 2: Statistics of several pretraining datasets for code models: size in bytes, number of files, and mum- ber of programming languages. In CodeSeerchhNl each file is function. For Pile and ROOTS only consider their code composite. Husain et al. (009)); h Gao et al. (2021): e Biderman et al. (2022) https: htt////ggggggccceeeeeee ctl.aacc...////aaaaaaeeeeeeaeaae http//daaatts/cccdoeeees
2022 to code processing. An overview of these pretrained models are provided in Table
github- code: e Kocetkov ct al. (2023);
Laurençon
5.1 Training Dataset for Code
el
[202)];
L Lozhkov et al. (0044).
While text data for pretraining language models are often crawled from the web and must undergo meticulous and often aggressive preprocessing (Raf- fel et al. 2020), code data come naturally whole documents from public GitHub repositories. Even better, they come with readily available quality in- dicators such as the count of stars or forks (although
Dalaset
Size (GB)
Files (M)
# PL
CodeSearchNet The Pi
20
6.5
6
95
19
CodeParrot
1K
115
30
The Stack"
3136
317
30
ROOTS!
163
15
The Stack v29
32K
3K
619
Allal et al. (2023) suggest that star count correlates poorly with downstream performance). As result, many large-scale code pretraining datasets have been introduced, including NodeSearchhet (Husain
20

---

Published in Transactions on Machine Learning Research (09/2024)
al. 2019), CodeParrot (Tunstall al 2022 and the Stack (Kocekov et al. 2023 totaling 20GB, 50GB and 3TB of code documents respectively (Table 2h
While these datasets are meant for training code models, it should be noted that code is ultimately a special form of natural language, as the vocabulary most programming languages is a small subset of English. Besides, high-quality code often interleaved with natural language comments or documentations, which also enables models to acquire certain knowledge of general text representation. In fact, of the 6.5M functions in CodeSearchNet,, 2.3M are paired with natural language documentation, allowing models to train explicitly such bimodal data.
Compared with natural language, another byproduct of scraping code from GitHub is commit histories, which consist of code before commit, code after commit, and a short message describing the commit, which can loosely serve as an instruction for language models, Muennighoff et al. (2024) utilize this feature and construct a 2GB dataset CommitPackFT containing 742K samples of instruction data for code, obviating the need of extensive human labor that's required to construct natural language instructions (Sanh et al. 2022
Wang al. 2022gh.
Apart from bimodal training and instruction finetuning, another recent trend in constructing code dataset is synthesizing data with powerful models such as ChatGPT. While this method is originally proposed for generating instruction data in natural language (Wang et al. 2023g Honovich et al. 2023 Gunasekar et al. (2023) take one step further and synthesize 1B tokens of Python textbooks and coding exercises to pretrain 1.3B model, achieving state-of- the-rrr-rrt results on HumanEval that's comparable much larger models trained significantly larger datasets.
21

---

Published in Transactionss on Machine Learning Research (09 (2024)
Table 3: An overview of pretrained code language models' architecture and training details: their base architecture, model size, vocabulary, context length, position embedding, training precision, attention type (MHA, MQA, or GQA), layer normalization type (post-norm or pre-norm), usage of FlashAttention, training initialization, objectives, dataset size (either in disk size, measured by GB/TB, or in number of tokens, measured by B/T), tokens seen during training, supported number of programming languages, and institute. We note that the number of training tokens does not count the training tokens of the model used for initialization, if any. The training objectives are: MLM (Masked Language Modeling), NSP (Next Sentence Prediction), RTD (Replaced Token Detection), IP (Identifier Prediction), CL (Contrastive Learning), SC (Span Corruption), DAE (Denoising Auto- Encoding), Text++Code (text-to-code generation and code-to-text generation), MT (Machine Translation). Missing information (such as AlphaCode's position embodding type) is left as blank.
Date
Modid
Anch.
Sire
Voesh
Contect
sion
Asten. Type
Paralle] Atten.
Narm
Atten.
Init, from
Objjctioon
Dataoet
Thaining
Inst.
2019-12
CUBERT
BERT
350M
30K
1024
absolute
fp52
MITA
MLM NSP
9:3B
0SB
Google
2020
CodeBERT
ROBERTA
125M
50K
absolute
Ep32
MILA
RoBERTI
MLM RID
2:GB
105B
Microsoft:
Grap&Code
2020-99
BERT
ReRERTA
125M
30K
640
absolute
FU32
MHA
CodeRERT
MEM Rdge Prediction + Node Alignment
20GB
131B
Microsof:
2021 08
SYAC&BERT
ROBERTA
120M
30K
512
abcalute
5p82
MILA
CodeBERT
MIL3E IP AST Elge Prediction CL
20GB
7B
6
[luaee
2021-10
DISCO
BERT
100M
21K
alaslute
MHA
MLM Node Type MLM
1,80B
Calumlia IIM
2022-11
Cade-MVV
RoBERIA
125M
30K
absolute
Sp82
MHA
GomglCodeBERt
MLM Tyse Infermoee
2GB
Huswei
2022-10
SColeR
RoBERTA
125M
51K
1024
absplute
fp32
MHA
UniXnoder
CL.
2GBB
Microsof:
2020 05
GPT C
GPT 2
374M
60K
1024
abrolute
MILA
CLM
270B
Microsoft:
2012-112
CodeGPT
GPT-2
124M
M0K
1122
absolute
4p32
MHA
GPT-2
CLM
2GB
1
Miczosofi
2022-02
Codler
GPT-NeoX
100M-278
50K
2048
RoPE
Ep32
MHA
CLM
2540BB
30B
CMU
22
2022 05
Matt(MMooo
CodeGan-
GPT-J
330M1
16.1B
50K
2048
RoPE
5p16
MIIA
CLM
1611(1881 B3/
S06D()7773)
IT(1)TT)
6(3)
Salesforos
2022-01
InCoder
50K
2048
fp52
MIIA
Caural Macking
20GBB
2022 05
PyCodeGPT
GPT Nco
110M
32K
1024
absolute
5p22
MIA
CLM
90GB
100B
Milrosoft
PanGa
2022-07
Coder
PanGree
317M-2.6
42K
1024
abwolute
MHA
CL.M
147GB
230R
Huswi
2023-01
SantaCoderr
GPT-2
1.18
48K
2048
abselute
fp32
MQJA
FIM
28R
296B
3
BigCode
2023-01
CodeGecX
PanClu-x
1413
52K
2048
absolute
5p16
MHA
CLM
158B
8500
23
Tringhun
2021-05
StarCader
GPT-2
16.68
48K
8152
alasdute
Ep32
MQJA
8160B
HigCldle
2023-05
Jum
GPT-2
350M
50K
256
absolute
6/16
MHA
CLM
960 2018
9R
1
1. Notro Dame
2023
GPT-3
51K
2048
RoFE
fp16
MILA
Microsoff
2023 10
CodePuse
GPT-J
35081
203
101K
RoPE
Ep16
MIAA
CLM
1.01B 11
40+
Ant Group
202--10
CodeSheil
GPT-2
78
70K
8192
RoPE
GQA
1008
600B
Poking
2020-10
P.MEO
GPT22
374M
SUK
10244 11124
absolute
4p16
MHA
SC
27GB
1
Micsosofi
Mastcopaolo al. DOBF
2021-02
TS
60M
32k
512+512
T5
5716
MITA
SC
LGB
UST
2021-02
250M
SUK
5124-512
absolute
4p26
MHA
MILM Duulducttiaa
45GB
2
Mett
2021-03
PLBART
BART
140M
50K
10244-1024
absolute
fp32
MHA
DAE
665GB
210B
UCLA Calumbie
2011-119
Code11
15
60M--220M
5124/256
15
fp16
MHA
SC + IP + Madded IP + ToatesCode
~220B
Salesdoroo
2022-01
SPT-Code
BART
202M
BDK
5124-512
abwolute
fp16
MHA
NSP + 80 - MMthed Name Prodiction
2033B
6
Narjing U.
2022
AlphaCode
300M
1538++768
bd16
MQA
MLM CLM
715GB
967B
Deephind
2022-05
NatGen
TS
220M
39K
512++256
T5
fp16
MHA
CodeiTS
Natoralistion
~26GB
148
8
Cohumbis UCD
2022-12
ERNIK-
580M
10344+109
b/16
MHA
+ Teste+Gade
197B
Reisho
2023 05
CodeT3+
T//GT
22081 1633
50K
20488-20048
abcolute
5p16
MIIA
CodeGen mono
SC CLM CL o
32
Salesforos
2020-12
Cugl.M
BERT
50K
absolute
f232
MHA
MIM NSP
1.28
Pakkng Microsol:
2022-03
UniXosa
ROBERIA
125M
51K
1024
absolute
fp32
MHA
MLM CLM SC Codi2Ted
20GB
839B

---

Published in Transactions on Machine Learning Research (09/2024)
5.2 Encoders
Pretrained Transformer encoders such as BERT (Devlin et al. 2019 RoBERT's (Liu et al. 2019d), and
ELECTRA (Clark et al.] 2020) have attained impressive results on natural language understanding tasks, and these methods were soon introduced into code processing after their advent. Kanade et al. (2020)
replicate the training procodure of BERT on a code corpus to produce CuBERT, showcasing its superior performance over LSTM (Hochreiter & Schmidhuber 1997) and non-pretrained Transformers. Feng et al.) 22000, the other hand, train CodeBERT with MLM and ELECTRA's RTD CodeSearchNet. They also utilize the explicit text-code pairs in CodeSearchNet, and use them respectively as the first and second segment in BERT's imput. When using CodeBERT to initialize the encoder part of a vanilla Transformer for sequence-lo-sequence generation tasks such as code summarization, they observe a moderate performance gain over non preeraardd baselines.
Apart from these standard training objectives, many auxiliary objectives specifically designod for code have also been introduced. GrRP (Guo al.) 2021a) and SynCoBERT (Wang al. 2021d both extract graphs from the source code (data flow graph and abstract syntax tree, respectively) and train the models to predict the typological relations between the nodes, while SynCoBERT and Code-MVP (Wang et al. 2022cb also add type inference to their pretraining stage in the form of tagging. Another common objective is contrastive learning: SynCoBERT and Code-MVP contrast between different views of the input (such as code, comment, AST, and transformed code), while DISCO (Ding et al. 2022a) constructs positive sample pairs by semantic-preserving transformations such as obfuscation, and negative pairs by injecting artificial bugs.
5.3
-eccodes
In NLP,, pretrained Transformer encoder-deroders such as T5 (Raffel et al. 2020) and BART (Lewis et al. 2020) have also left notable mark in the past few years' advancement in language modeling. T5, for example, unifics all textual tasks into sequence to sequence format and sets new records on GLUE (Wang) et al.] 2018 Sal and SuperGLUE (Wang et al. 2019). Compared with encoder-only models, encoder-decoders are naturally more powerful as they can be used for conditional text generation, while their encoder part can always be taken out to perform tasks that require an encoder-only architecture, such as regression (Tay
ct al. 2023b).
Inspired by these advantages of encoder-docoder architecture, many such models have been proposed for code processing. PyMT5 (Clement et al. 2020) and Mastropaolo et al. (2021) replicate the pretraining and multi-task finetuning process of T5 code corpus, while Ahmad et al. 2021 introduce PLBART, BART
pretrained on 655GB combined data of Java, Pyhton, and natural language. Lachaux et al. (2021 argue that MLM could be too easy a task for programming languages as identifier names often occur multiple times in a single context window, and propose a deobfuscation ion pretraining objective, where the model is trained to convert obfuscated code back to its original form. Related to this method, we note that meaningful variable names have also boen found to have a positive impact on the code generation process of large language models (Chen al. 2023e
Building these early works, Wang al. 2221f propose CodeT5, which is pretrained alternatively with 1) T5's original span corruption; 2) identifier tagging (where each token in the code input is tagged as either identifier or non-identifier): 3) masked identifier prediction (a special form of span corruption where all identifiers are masked); and 4) text-to-code & code-to-text generation. Its successor, CodeT5+ (Wang let al. 2023h), take inspiration from UL2 (Tay et al. 2023b) and introduce causal language modeling (CLM) into pretraining, along with additional contrastive objectives based on text-code matching.
AlphaCode (Li et al. 2022h is also trained with multiple objectives, where the encoder is trained with MLM and the decoder is trained with CLM, with architecture modifications such as shallow-encoder & deep- decoder, multi-query attention (Shazeer 2019). and being much larger than CodeT5 (up to 41B parameters). NatGen [Chakraborty et al. 2022a), on the other hand, is pretrained with a 'naturalization' objective similar to dcobfuscation: semantically equivalent but unnatural code is generated by predefined operations such as loop transformation, dead code injection, and variable renaminng and the model is pretrained to translate
23

---

Published in Transactions on Machine Learning Research (09/2024)
these unnatural code back to its original form. We note that some of these models are built. on previ- ous works. For example, NatGen is initialized with CodeT5, while the largest version of CodcT5+ is initialized from a decoder-only model, CodeGen (Ni- jkamp et al. 2023b).
Table 4: Pass@1 performance of pretrained code mod- els (top), instruction finetuned code models (middle), in comparison with some of the best general language models (bottom), with models in each category or- dered chronologically. The sources of these figures can
be found in Section 5.3 Section 5.4 and Table 1
Apart from these general pretraining objectives, sev- eral works have also trained Transformer encoder- decoders with focus on code translation, which is a natural application of Transformer models in code as the Transformer architecture was originally pro- posed by Vaswani et al. (2017) for machine trans- lation (MT). However, unlike natural languages, where parallel corpus across two or more human languages exist in abundance, there is little par- allel data for code. To tackle this issue, Rozière et al. (2020) propose Transcoder, which first pre- trains an encoder with XLM (Conneau & Lample 2019), and then initializes a vanilla Transformer with this encoder and continue to pretrain with Denoising Auto-Facodiin (DAE, Lewis et al. 2020) and back translation (Sennrich ct al. 2016), while its follow-up work (Szafraniec et al. 2023) also utilize language-independeat intermediate representations to enhance this process, which we discuss in more detail in Section
Model
Size
HumanEval
MBPP
PolyCoder CodeGen-Mono
2.7B
5.6
16.1B
29.3
35.3
InCoder
6.7B
15.2
19.4
PyCodeGPT
110M
8.3
Vangu-Codes
2.6B
23.0
SantaCoder
1.1B
14.0
35.0
CodeGeeX
13B
22.9
24.4
StarCoder
15.5B
33.6
52.7
CodeTT++
16B
30.9
Phi-1
1.3B
50.6
55.5
CodeFuse
13B
24.8
DeepSeek Coder
33B
56.1
66.0
InstructCodeT5+
16B
35.0
WizardCoder
15.5B
57.3
51.8
Bangu-Coder
15.5B
61.6
OctoCoder
15.5B
46.2
CodeFuse
34B
74.4
DeepSeek Coder-Instruct
33B
79.3
70.0
GPT-4
67.0/82
PaLM 2
S
37.6
50.0
Code LLaMA
34B
53.7
56.2
Apart from training data and objectives, these mod- els mostly keep to the original architectures pro- posed by the NLP community, as shown in Ta- ble 3 Models based on BART, for example, use post-normalization and learnable absolute position embeddings, while those based on T5 use its simplified relative position embeddings and pro-normalization.
Phi-1.5
1.3B
43.5
5.4
Decoders
After the momumental debut of GPT-3 Brrwn et al. 2020 and the discovery of in-context learning, decoder-
only Transformer models have become dominant in language modeling |Rac et al., 2021 Hoffmann et al.
2022 Touvron et al. 2023a b. inter alia). Many models similarly pretrained with CLM have also emerged in code processing, such as GPT-C (Svyatkovskiy et al. 2020) CodeGPT (Lu et al. 2021), PolyCoder (Xu et al. 2022), CodeGen (Nijkamp et al. 2023b), PyCodeGPT (Zan ct al. )022), Pangu-Coder (Christopoulou et al.] 2022), CodeGeeX (Zheng et al. 2023a)). Jam (Su et al. 2023a Phi-1 (Gunasekar et. al.) 2023), and CodeFuse (Di et. al.
2022
Chowdhery et al. 2023 Scao et al.
2023). Of these models, several alternative training objectives have been experimented with, such as MLM and Masked CLM in Pangu-
but are found to underperform compared with CLM-only training, Zan et al. (2022) also propose continual training on sketches, where the model learns to first generate sketch of program and then the actual code. Notably, Gunasekar et al. (2023) present Phi-1, a 1.3B small model trained on a dataset of only 7B tokens consisting of 6B tokens from StackOverflow and 1B synthetic data generated by ChatGPT but achieving 50.6 pass@l on HumanEva and 55.5 pass@1 on MBPP, comparable to much larger (both in model size and training data size) models such as Code LLaMA or PaLM
In their paper, MLM is conducted by replacing tokens the input with <mask> and predicting from only the left context, while Masked CLM performed by adding <mask> the input and predicting next token from it. Both tasks do not change the attention mask patterns the model.
24

---

Published in Transactions on Machine Learning Research (09/2024)
(2022) report denoising objectives underperform in decoder-only models, there have been other works that successfully combine denoising or multi-task pretraining with decoder architecture. Incoder (Fried et al. 2023). SantaCoder (Allal et al.) 2023), StarCoder (Li et al. 2023i)
Although Christopoulou et al.
DeepSeck Coder (Guo et al.) 2024). and CodeShell (Xie et al.) 2024 are trained with fil-in-the-middle (FIM)
objective, also referred to as causal masking by FFrie et al. (2023) which is essentially span corruption (Rafee)
et al. 2020) adopted to decoder-only architecture. One of the visible advantages of these infilling objectives is that they inject the models with the ability to fill in blanks in the middle of input code at inference time, while CLM allows only for autoregressive generation. As Table shows, however, these objectives also lead to higher performance on downstream tasks when compared with CL.M-nlly models such as CodeGen, although the exact benefits of infilling training remain controversial NNjkkamp et al. 2023a).
Observing Table 3 it is clear that decoder-only models for code have generally followed the practices in NLP more closely, when compared with other model architectures. All these models use pre-normalization, while MQA, RoPE, and parallel attention have also been adopted by several models. Notably, the three most recent models- StarCoder, Phi-1, and CodeFuse- also employ FlashAutenioon to improve model throughput.
5.5 UniLMs
Following UniLM (Dong et al. 2019 in NLP, several works in code processing have also pretrained this fourth family of Transformer models on code. CugLM (Liu et al. 2020) is trained with both CLM and MLM + NSP via alternating attention masks, while UniXcoder is trained with CLM, MLM, Span Corruption (in Prefix LM style) along with auxiliary objectives including contrastive learning and text-code mutual generation. Both two models, however, are relatively small in size, and whether or not this architecture is suitable for code processing is yet to be explored at scale.
5.6 Diffusion Models
Currently the Transformer architecture dominates text generation, but several works (Li et al. 2022d Lin
ett al.] 2023) have also adopted Diffusion Models (Ho et al. 2020) from computer vision for text generation. Recently CodeFusion (Singh et al.) 2023) also introduces diffusion models into code modeling, and demon- strates that a 75M diffusion model can outperform StarCoder, CodeT5+, and GPT-3 on three code synthesis datasets.
5.7 Instruction Finetuning and Reinforcement Learning for Code
In natural language processing, training models on a diverse set of tasks with instruction prefix, known as instruction finctuning, has been shown to unlock the ability of cross-task generalization (Ouyang ot al. 2022 Chung et al. 2022; Iyer et al. 2022). At first, these instruction data samples are manually compiled or crowd-sourced @Wei et al. 2022b
Sanh et al.) 2022), but later researches find I.I.M-gnneraed instructions Honovich al. 2023).
to be sufficient (Wang et al. 2023g
Following these works in natural language, researchers from the code community have applied instruction tuning to their models as well. Wang et al. (2023h) finetune CodeT55+ with 20K instruction data generated by InstructGPT (Ouyang et al. 2022) to obtain InstructCode13++ WizardCoder (Luo et al. 2023) follows the methods of WizardLM (Xu et al. 2024 to evolve 20K code Alpaca (Taori et al. 2023) samples into a 78K dataset and uses it to finetune StarCoder. Pangu-Coder 2 (Shen et al.] 2023) also uses WizardLM'ss Evol-Instruct to generate 68K instruction samples from 20K code Alpaca, but also introduces reinforccment learning via Rank Responses to align Test & Teacher Feedback (RRTF). OctoCoder (Muennighoff et al., 2024), on the other hand, takes a different path and uses Git commit histories instruction data to finetune StarCoder and CodeGeeX2. More recently, CodeFuse (Liu et al.) 2023b) also employs and explicitly introduces multiple downstream tasks into their instruction data. The performance of these instruction finetuned code models can also be found in Table
In NLP, another technology closcly related to instruction finetuning reinforcement learning from human feedback (RLHF), which has played a significant role in aligning LLMs with human values (Ouyang et al., 2022 Bai et. al.] 2022 The merit of reinforcement learning is that it can incorporate non-differentiable
25

---

Published in Transactions on Machine Learning Research (09/2024)
reward signals into training, such BLEU (Bahdanau et al. 2017) and human preference (Crissiaano al, 2017], but the human feedback required in aligning LLMs often involves extensive labor on annotation. Tn comparison, applying reinforcement learning to code models has a natural advantage, as compilers can be used for automatically generating foedback for code samples produced by language models.
CodeRI. (Le et al., 2022 is one such model, which defines four levels of rewards for each generated program (viz. compile error, runtime error, unit test failure, pass) as well as fine-grained token-level reward estimated by critic model. The actor model, which is an extension of CodeT5, is then trained with REINFORCE
algorithm (Williams 1992). Similarly, CompCoder (Wang et al. 2022dh and PPOCoder (Shojace et al. 2023) train CodeGPT and CodcT5 respectively with proximal policy optimization (Schulman et al. 2017): while RLTF (Liu et al.) 2023db proposes fine-grained feedback based on the error information and location provided by the compiler, as well as adaptive feedback that takes the ratio of passed test cases into account.
6 Code Features for Language Models
A major difference between programming languages and natural languages is that the former is artificially defined to be precise and unambiguous, and need to be compiled (or interpreted) without orror before execution. This allows for a much larger flexibility in designing pretraining objectives on code, beside lexical manipulations such as CLM, MLM, and Span Corruption. A similar trend can be observed in the last years before neural networks were introduced into mainstream NLP literature (Sutskever ct al. 2014 Bahdanau et al. 2015 when researchers in the MT community utilized alternative views of lext such as synlactic features to improve the performance of SMT systems (Galley et al., 2006] Chiang 2007). These features, however, are not universally applicable or even agreed upon, and often result in highly complicated systems (for example, the size of English part-of-speech tagging's s label set may range from dozens to hundrods).
Programming languages, however, fare much better in these aspects. Each mainstream programming lan- guage, such as C, Python, and Java, comes with readily available compiler toolkits that allow for easy and accurate extraction of semantic information such Abstract Syntax Tree (AST), language-independen Intermediate Representation (TR), and auxiliary information such as type of each token and control flow graph (CFG/DFG). Thus, in the context of Transformer-bseed language modcling for code, many works have incorporated these features into their training procedure.
6.1 Abstract Syntax Tree and Intermediate Representation
AST is one of the most common intermediate results of the compiling process, where program is parsed into a tree of operations and their operands. Before the popularization of Transformer in the code processing community, there had been works such as InferCode (Bui et al. 2021a) that processes these representa- tions with special network architectures like Tree-Based CNN and conducts self-supervised pretraining by predicting subtrees.
TreeBERT (Jiang et al. 2021b is one of the first attempts to take AST into the Transformer-base based cettraining-finetuiin framework. It's a Transformer encoder-decoder pretrained with Tree MLM and Node Order Prediction, where the encoder takes a set of constituent paths in the AST as input (with each token being a path, which is the concatenation of its nodes' representations) while the decoder takes the code as input. Tree MLM is then performed by masking certain nodes in a path representation and its corresponding code tokens in the decoder input, while Node Order Prediction accomplished by swapping nodes in a path and predicting it with a [CLS] token similar to BERT.
The method used by TreeBERT, however, is complicated and does not scale well. Later works mostly opt to first process AST into text sequence and treat it like normal part of the input. Wang et al. (202dd), for example, process AST with depth-first traversal and concatenate it with code and comment, and then train SynCoBERT (which, unlike TreeBERT, is actually a BERT-like encoder-only model) with four objectives: 1) MLM; 2) identifier tagging; 3) AST edge prediction (predicting whether there exists an edge between two AST nodes from the dot product of these nodes' representations); and 4) contrastive learning over i) code and AST pairs, as well as ii) text and code-AST pairs. Similarly, SPT- (Niu et al. (2022). a Transformer encoder-decoder, takes the concatenation of code, sequentialized AST, and text input, and
26

---

Published in Transactions on Machine Learning Research (09/2024)
is pretrained with 1) span corruption; 2) code-AST prediction (NSP with one segment being code and one segment being AST); and 3) method name generation, a special form of span corruption where method name is masked. Different from other works, however, they do not take the docstrings as the text segment in their input, but instead concatenate all method names appearing in the code as a succinct natural language description. Likewise, UniXcoder (Guo et al. 2022 Lakes flattened AST instead of source code as its input during training.
In the compiling pipeline, AST is usually followed by language-independent intermediate representations, such as LLVM IR (Lattner & Adve, 2004). Such features' independence from specific programming languages makes them suitable candidatos for translation pivots, as is English in machine translation of low-rosource natural languages (Leng et al. 2019). Szafraniec et al. (203)) take advantage of this characteristic and extend Transcoder (Rozière et al. 2020) with translation language modeling (Conneau & Lample 2019) over code and IR, as well as TR generation from code. They also investigate other objectives such as TR decompilation (i.c. generating code from IR) and IR pivot (i.c. directly generating code in one language from the IR. of another language), both showing promising results.
6.2 Control Flow and Data Flow
While AST and IR have proved to be useful information in certain tasks such code translation, they are static by nature, just like the source code, and may fail to capture semantic properties of code that are only revealed at runtime (Wang& & Su 2020). Such semantics, however, are contained in dynamic features such as control flow and data flow. Similar to AST, specialized networks were used to process such information before the rise of pretrained Transformers, such as Message Passing Neural Network used by ProGraML (Cummins/ et al. 202Tp. Unlike AST, however, even after pretrained Transformers became dominant. few works have looked in this direction.
GraphCodeBERT (Guo et al. 2021a is one of such works, which creates special tokens and position embed- dings for variables in the flow graph, and concatenates the variable sequence after text and source code to construct model imput, with tailored attention masks on the code and variable segments: tokens from code segment and variable segment can allend to each other if and only if the variable is identified from the code token, and for tokens within the variable segment, 2/2 is allowed to attend to By if there a direct edge from "y to i in the dataflow. The model then protrained with MLM in combination with edge prediction and node aligument, both of which are accomplished by binary classification from the dot product of two tokens' representations (one from code segment and one from variable segment for node alignment, and both from variable segment for edge prediction).
6.3 Type
Apart from AST, IR, and data flow, type information has also been used to aid language models in processing code. CugLM (Luu et al. 2020), for example, uses type information during finetuning to aid in the prediction of tokens for unidirectional MLM (i.e. MLM with unidirectional attention mask): the type of a masked token is first predicted from the final Transformer layer's representation, and then the token itself predicted based on both the hidden representation and predicted type. In contrast, both CodeT5 (Wang et al., (021f)) and SynCoBERT (Wang et al.) 2021d] include identifier tagging in their pretraining objectives, which can be viewed as coarse-grained type prediction.
6.4 Program Transformation
As we have shown in Section 5.3 function- function-preservv preserving program transformations have proven to be important techniques in pretraining code language models. Obfuscation is one instance of program transformation, and others include loop transformation (for-while), condition transformation (if-switch), dead code injection (e.g. if True: pass), and statement swapping. DOBF (Lachaux et al. 2021) and NatGen (Chakraborty et al. 2022a) are two code language models pretrained to recover the original program from such transformations, while Wang et al. (2022a) also apply program transformations during the finetuning stage of language models to make them more robust to transformed test. samples.
27

---

Published in Transactions on Machine Learning Research (09/2024)
Notably, Wang al. (2022e) integrate many of the aforementioned features into Code-MVP: source code, docstrings, AST, CFG, and transformed source code via identifier renaming, loop exchange, and dead code insertion. The model, initialized from CrapH is then trained with MLM, fine-grained type prodiction, and contrastive learning across different views, such as text vs. code, code vs. AST, and code vs. CFG.
7 LLMs in Software Development
As language models set new records on software engineering benchmarks, software engineering technologies are also expanding the boundarics of language models in return, and have subsequently led them into real- world development cycles.
7.1 LLMs Extended with Coding Tools
Research in the NLP community has shown that LLMs can learn to use external tools such as calculators, MT systems, and search engines (Thoppilan et al., 2022 Schick et al.] 2023). As such, interpreter has been used to augment LLMs in complex reasoning tasks. PAL (Gao al., 2023b) and PoT (Chen et al. 2023eb both extend Codex with Python interpreters for numerical calculations, while ViperGPT (Surís et al. 2023 extends it further by calling vision APIs to extract information from visual input and answer related questions.
However, LLMs do not always produce code that can be interpreted and executed, and there has also been a line of works that explore the possibility of emulating the interpreter with LLMs themselves. Nye el al.) (2021) trains LLMs to emulate the execution of programs by outputting program states at each step. More rocently, Li et al. (2023a) propose Chain-of- Codde, where real interpreter exccutes the generated code until error occurs, w whereupon the LLM takes over to simulate execution. Chae et al. (2024) similarly propose Think-and exxexct framework, where an LLM generates pseudo code and then simulates its execution to solve reasoning tasks.
Apart from alleviating the burden of numerical calculation in abstract reasoning tasks, interpreter (together with unit tests) also provides feedback on the process of code generation itself, allowing for interactive generation and refinement of code. Such works include Self-Edii (Zhang et al. 2023c), Le'll (Wang et al. 2023f), OpenCodeenneerpreeer (Zheng et al., 2024), ProCoder (Bi et al. 2024h, Cycle (Ding et al. 2024), and SOAP (Huang et al.) 2024), which run model -eneee-eeeeeated code against unit Lests to provide feedback for further refinement. Alternatively, CodeT (Chen et al. 2023a), TiCoder (Barciß et al. 2022), and CONLINE (He let al. 2024) also utilize the LLM itself to generate unit tests, while Self- Refine MMdaan et al. 2023 sends the generated code to an LLM instead of an interpreter for feedback. Zhoo et al. (2033a) show that OpenAl's
interpreter plugir Soagirgaallon allows GPT-4 to self-debug, while InterCode (Yang et al.] 2023d) provides a benchmark for evaluating interactive coding. In Section 5.7 we have also shown that the execution results on unit tests serve as natural supervision signals for reinforcement learning code.
A topic closely related to tool using in LLM research is planning as intelligent agents, which has been shown to enhance LLMs' capability both theoretically and empirically (Feng et al. 2023a). Ruan et al. 2023 find that LLMs can plan to solve complex tasks using external SQL generators and Python generators, while CodePlan (Bairi et al. 2023 demonstrates they can perform repository-Iewe coding via adaptive planning.
Another stream of works use LLMs to create multi-agent systems for code generation, such as self- collaboration (Dong et al. 2023), ChatDev (Qian al. 2023), MetaGPT (Hong et al., 2023). ; LCG {Lin) et al.) 2024), MAGIS (Tao et al.) 2024], and SoA (Ishibashi & Nishimura] 2024]. In these frameworks, multiple LLMs are prompted to play distinct roles such as programmer, reviewer, and manager. These roles interact with each other, breakdown code generation into different phases (e.g. designing, coding, testing, and documenting), and collaborate to complete complex tasks.
28

---

Published in Transactions on Machine Learning Research (09/2024)
7.2 LLMs Integrated into Software Development
With the increase in LLMs' interactive coding capability, researchers have also started to integrate them into each and every process of software development.
Auto code completion is one of the carliest applications of language models in software development, as they require only the ability to predict the next loken. Even before language models scaled to billions of parameters, there had been integration of completion systems such as Pythia (Svyatkovskiy et al. 2019 and IntelliCode (Svyatkovskiy al. 2020 into popular IDEs.
Recently, however, the application of code language models have transcended simple code completion. GitHub Copilot is arguably one of the most popular AI code assistants, with diverse features including code generation, vulnerability detection, and license managemen o while CodelFuse (Di et al.) 2023) also integrates code generation, code translation, code commenting, and testcase generation into a single IDE extension. As code language models become larger, however, their client-side deployment and real-time performance also raise new challenges.
As LLMs contimue to advance, building applications on top of them is also evolving into consequential task itsclf. Many open-source frameworks for such applications have been released, including LangChain [D AutoGP and WorkGPT These frameworks provide abstractions over language models for developers, and are actively revolutionizing the entire process of software development even as this survey is being finalized.
7.3
Analysis of LLM-Generated Code
As AI code assistants become prevalent, many recent works have also focused on examining AT-generated code from different aspects, including correctness (Nguyen Nadi
2022
Yetistiren et al. 2023
bugs (Jesse
et al. 2023 Tambon al. 2024), vulnerabilities (Asare et al. 2023
Sandoval et al.) 2023
Perry et al. al. 2024). and
2023
Hamer et al. 2024], syntactic robustness (Sarker et al.
2024).
efficiency (Niu
2023)'s i' studies show that Al code assistants do not introduce extra security risks compared with human programmers, and Hamer et al. (2024) also find that ChatGPT's responses to security-related questions contain less vulnerabilities than answers from StackOverflow. Contrarily, Perry et al. (2023) find that users write significantly less secure code when assisted by AI. In terms of code complexity, Nguyen & Nadi (2022) find GitHub Copilot to generate low- complexity code with no statistically significant. differences between languages, whereas Liu et al.) (2023n) find ChatGPT to generate the most complex code in C and the least complex code in Python. Overall, Al code assistants are still in their nascent stage and constantly evolving, and their implications software development are yet. to be invest systematically.
hallucinations (Liu et al. 2024). Asare et al. (2023 and (Sandoval et al.
8 Conclusion and Challenges
In this work, we systematically reviewed the history of pretrained Transformer language models in code processing and other software engineering tasks. The advancement in code modeling generally follows the history course NLP, evolving from SMT models, to NMT models, and then to finetuning pretrained Trans- formers and lastly to few-shot application of LLMs and even autonomous agents in real-world production. Unlike natural languages, the nature of code makes it easy to extract auxiliary information from alternative views, and to utilize interpreter and unit tests for automatic feedback.
With these in mind, we identify several challenges in the current development of code modeling.
More comprehensive and challenging benchmarks. The widely used HumanEval benchmark plays a key role in the evolution of Code LLMs. However, it is relatively small and its scoreboard has been manip- ulated to near perfect, and the community is eager for a new standard to evaluate LLMs, HumanEval and
httpp://kwww hwwp://wwwwncchii.co
bttps://github... wtt.://wwo..oniiinnanaaiooaaaaee Graaifaccaaatttrattttttttttaa
https://github... cot/ tea/t/tannnoee wpapeaaaooppppooorrraaa
29

---

Published in Transactions on Machine Learning Research (09/2024)
other similar benchmarks focus generating standalone Python functions from well-structured docstrings, which do not reflect real-world user behaviors. In real-world scenarios, software requirements are seldom condensed into a single docstring, and LLMs must learn to commumicate effectively for clarifications when uncertain about the requirements (Wu & Fard 2024). Also, standalone functions are hardly used in pro- duction, where soflware has complex dependencies within repository. Thus, next-generation benchmarks should also take such cross-file context into account, and recent benchmarks such as SWE-bench (Jimenez) jet al.) 2023) represent promising efforts in this direction. Contrary to the models' perfect scores on Hu- manEval, these repository-level benchmarks expose LLMs' limits in real-world applications, with the most recent SWE-agent resolving only 12.5% real-world GitHub issues in SWE-bench (Yang et al. 2024b).
Evaluation and application of LLMs beyond traditional code generation. As we have pointed out in Section [ current evaluation of LLMs' coding capability in the NLP community is focused on code generation, while overlooking other activities in software engineering, such as software modeling and testing. From an application point-of-view, the most widespread application of LLMs in software engineering is IDE plugins, which provide developers with code suggestions, while other stages in software development that we mentioned in Section 2 are also largely overlooked. However, it should be also noted that LLMs trained on code are highly specialized to reasoning-head -eavy tasks such as programming and math, and non-coding tasks in SE such as requirement elicitation, product management, and marketing are probably better suited for general-domain models. Beyond the previously mentioned text-based evaluation and applications, the recent advancement of multimodel LLMs also created new opportunities, especially for UI design and other activities that involve visual input, leading to novel tasks such as visually grounded code generation (Li et al. 2024b Wu et al.; 2024a), webpage reverse engineering Si et al. 2024 Laurençon et al. 2024 and
layout design (Feng et al.) 2023b
Acquissiin of high-quality data. With Gunasekar et al. (0223 achieving SOTA performance with a 1.3B model trained on textbook data, we believe the selection of training data and utilization of synthetic data will be ever more prominent in the near future, for both self-supervised protraining and supervisod finetuning.
Integration of code features into language models. As we noted in Section 6.2 CFG and DFG are yet to be employed at scale in code language modeling. The few works that do employ data flow make changes to the models' attention masks, which severely limits their cross-Lask generalization and scaling ability. We believe the seamless integration of such features into textual input is worth researching in the future.
Beyond the imperative programming paradigm. In Section 2.1.7 we have shown that most of the current research on code LLM focus on popular imperative programming languages such as C and Python, while neglecting declarative and functional languages except SQL. However, the paradigm of declarative and functional languages makes them more aligned to natural languages, and thus is worth more research attention in the future.
Alternative model architectures and training objectives. In Table 3 we have shown that many code language models are pretrained with auxiiaary objectives specific to code, but these models all belong to the encoder-only or encoder-decoder family, while decoder-only models are yet to be augmented with alternative objectives. Also, as pioncered by [Singh et al. (2023), we believe diffusion models will find its ground in code modeling in the future.
Safety and ethics issues related to code LLMs As language models grow in might, they also raise safety concerns including but not limited to data contamination, taxic or biased generation, personal information leak, and hallucinations. In software development, these models should be deployed with extra caution, as their generated code may contain security risks leading to catastrophic results. Pretraining data is also becoming a sensitive topic of ethics, and Kocetkov et al. (2023) take a meaningful step towards this issue by allowing developers to remove their code from the Stack. As synthetic training data becomes widespread, researchers should also proceed with caution about such practice, the consequence of training Al models with Al generated data is yet to be investigated at scale.
30

---

Published in Transactions on Machine Learning Research (09/2024)
With the presentation of this survey, hope to provide a global view of language models' application in software engineering and connect the research from the two communities. We believe the current surge of LLMs will be ultimately transformed into real-world applications, and lead humanity into a brighter future.
Acknowledgments
This work is supported by Ant Group. Ziyin Zhang and Rui Wang are partially supported by the National Natural Science Foundation of China (62176153) and the Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102, as the MoE Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University).
References
Esra A. Abdelnabi, Abdelsalam M. Maatuk, Tawfig M. Abdelaziz, and Salwa M. Elakeili. Generating uml class diagram using nlp techniques and heuristic rules. In 2020 20th International Conference on Sciences and Techniques of Automatic Control and Computer Engineering (STA), pp. 277-282, 2020. doi: /T1TSTAA55067222009922222
Esra A. Abdelnabi, Abdelsalam M. Maatuk, and Mohammed Hagal. Generating unl class diagram from natural language requirements: A survey of approaches and techniques. In 2021 IEEE 1st International Maghreb Meeting of the Conference on Sciences and Techniques of Automatic Control and Computer Engineering MI-STA, pp. 288-293, 2021a. doi: D01109/MI-STA52233
Esra A. Abdelnabi, Abdelsalam M. Maatuk, and Mohammed Hagal. Generating uml class diagram from natural language requirements: A survey of approaches and techniques. In 2021 IEEE 1st International Maghreb Meeling of the Conference on Sciences and Techniques of Aulomatic Control and Compuler Engineering MI-STA, pp. 288 293, 2021b. doi:
Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amil Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javahcripi, Xin Jin, Picro Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/ 22/22244441122 2024. doi: 10.48550/ / 220...2444.22211 URL https:://oot http::///oooooo//1/// org/10. 00te//0..8555000 2005500aaXiv.2000 24044.4299
Sallam Abualhaija, Marcello Ceci, and Lionel C. Briand. Legal requirements analysis. CoRR, abs/2311 123711117111 2023. doi: 10.48550/AA 11R.XII..311.13...
URL http::///oa..ooo////..555//22221
Mohammed Abubamad, Tamer AbuHmed, Aziz Mohaisen, and DaeHun Nyang. Large-scale and language- oblivious code authorship identification. In David Lie, Mohammad Mannan, Michael Backes, and Xi- aoFeng Wang (eds.), Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communica- tions Securily, CCS 2018, Toronto, ON, Canada, Oclober 15-19, 2018, pp. 101-114. ACM, 2018. doi: 10.1145/22449334.3223333 URL httpps://abi.orr/0..11155555 http:/...114///2334...3833
Rajas Agashe, Srinivasan Iyer, and Luke Zettlemoyer. Juice: large scale distantly supervised dataset for open domain context-based code generation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNI.P-IJCNI.P 2019, Hong
31

---

Published in Transactions on Machine Learning Research (09/2024)
Kong, China, November 3-7, 2019, pp. 5435 5445. Association for Computational Linguisties, 2019. doi: 10. (0.1663/VV1DD19-55// URL https://doi.
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettle- moyer. HTLM: hyper-text pre-training and prompting of language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nee 2022. URL http::///pppameooreeeee//foooooo
Lakshya A Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K. Lahiri, and Sriram K. Rajamani. Guiding language models of code with global context using monitors. CoRR, abs 2306//2336.0763, 2023, doi: 10.48550/ TEXIVV2330.100
URL https://do1..rgg 48550/ arxiv. http:/aa............1 10763
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. A transformer-based approach for souree code summarization. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Associalion for Compulational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 4998-5007. Association for Computational Linguistics, 2020. doi: URL https: /tops://doi..oo org/10. 10. http:/3/112220.a111111
10553//1/222.acl-mmg
449
Wasi Uddin Ahmad,, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang, Unified pre-training for program understanding and generation. In Kristina Toutanova, Anna Rumshisky, Luke Zeutlemoyer, Dilek Hakkani- Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguislics: Human Language Technologies, NAACL-HLI 2021, Online, June 6-11, 2021, pp. 2655- 2668. Association for Computational Linguistics, 2021. doi: 10.18653/v1 /at/11//2211aaaa/ ma02...aaclmmmin.2. URL https://doi. org/ 10. 10..008665//1112200 http://2221..aacmmaaaa... www.caaaaamnnn... 211
Wasi Uddin Ahmad, Md Golan Rahman Tushar, Saikat Chakraborty, and Kai- Wei Chang, AVATAR: A parallel corpus for jhon program translation. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2029, pp. 2268-2281. Association for Computational Linguistics, 2023. doi: 10. 10.18653/V1 /V/2223. FINDINGS- ACL.143. URL httpp:://dii.rrg////
http://2222..fndnggeeee wwwdinnnnnnnnl....... 143
Sharif Ahmed, Arif Ahmed, and Nasir U. Eisty. Automatic transformation of natural to unified modeling language: A systematic review. In Juyeon Jo, Yeong-Tae Song, Lin Deng, and Junghwan John Rhee (eds.), 20th IEEE/ACIS International Conference on Software Engineering Research, Monagement and Applications, SERA 2022, Las Vegas, NV, USA, May 25-27, 2022, pp. 112 119. IEEE, 2022. doi: 10. 1109/SERA54SSS 202.. 98066783 URL https://doo. org/ 10. 10099/EEA554885 2022. 9806783
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sunit Sang- hai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali (cds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singupore, December 6-10, 2023, pp. 4895 4901. Association www.lloogyorrg///e/e
for Computational Linguistics, 2023. URL https://accaanhhologg. org/2023. htt//2223.emmppppp.. emnnl-mmiie 298
Saranya Alagarsamy, Chakkrit. Tantithamthavorn, and Aldeida Aleti. A3test: Assertiion-aunmentee au- tomated test case generation. CoRR, abs/2302.10352, 2023. doi: 10.48550/ARXIV ARXIV 230.2.22232. URL
Attps://dois wtt.://g./....////r///oo../ 2302. 10352
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Muñoz Fer- randis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Car- olyn Jane Anderson, Yangtian Zi, Joel Lamy-Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Ab- ulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zoeca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Scan Hughes, Daniel Fried, Arjun Guha, Harm de Vries, and Leandro von Werra. San- tacoder: don't reach for the stars! CoRR, abs/ 2301.03888, 2023. doi: 10.48550/ 485500/aXiiv...... at..aa.3vivv....... 2301.03888. URL https://doi. 48550/
w arxiv. 2398.
32

---

Published in Transactions on Machine Learning Research (09/2024)
Miltiadis Allamanis and Charles Sutton. Mining source code repositories at massive scale using language modeling. In Thomas Zimmermann, Massimiliano Di Penta, and Sunghun Kim (eds.): Proceedings of the 10th Working Conference on Mining Software Repositories, MSR '13, San Francisco, CA, USA, May 18-19, 2013, pp. 207-216. IEEE Computer Society, 2013. doi: 10.1109/ (t.1109/MSR.22 / 0013.6624029. URL https: 10. 20000/MMRRRRR 6624029
Miltiadis Allamanis and Charles Sutton. Mining idioms from source codc. In Shing-Chi Cheung, Alessandro Orso, and Margaret-Anne D. Storey (eds.), Proceedings of the 22nd ACM SIGSOFT International Sym- posium on Foundations of Software Engineering, (FSB-22), Hong Kong, China, November 16- 22, 2014, URL https://doi. otg/oo. 1145/2638/6688
pp. 472 483. ACM, 2014. doi: 2635901
Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. Learning natural coding conventions. In Shing-Ch Chi Cheung, Alessandro Orso, and Margaret-Anne D. Storey (eds.), Proceedings of the 22nd ACM SIGSOFT International Sympostum on Foundations of Software Engineering, (FSE-22), Hong Kong. China, November 16 22, 2014 pp. 281-293 ACM, 2014. doi: aaloi.or//00 10. 11111111422538 2635883
22222663336663668 URI. https:
Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. Suggesting accurate method and class names. In Elisabetta Di Nitto, Mark Harman, and Patrick Heymans (eds.), Proceedings of the 2015 10th Joint Meeting on Poundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30- September 4. 2015, pp. 38-49. ACM, 2013. doi: 10.1145/77888052778884,, URL https: 01w.1111111/7778858.1 2786849
Miltiadis Allamanis, Earl T. Barr, René Just, and Charles Sutton. Tailored mutants fit bugs better. CoRR, Abs/1161.02516, 2016a. URL http://arxiv.orggg. wttp/os////////////////////// wta.///1................ 025116
Miltiadis Allamanis, Hao Peng, and Charles Sutton. A convolutional attention network for extreme sum- marization of source codc. In Maria-Florina Balcan and Kilian Q. Weinberger (cds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 2091-2100, 2100. JMLR.org, 2016b. URL http://proceedings. mlr. wwt.:mmwr...pmm.eemmmmmmmmmmee.e
wwwpppeesssss.eee v48/ hwtp///aaaaaa.....mmmm html
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- - May 8, 2018, Conference Track Proceedings. OpenReview.net net, 2018. URL https: /ooperreiie wecciiiww.ee//eeeee aet/fooummte Bd=
Miltiadis Allamanis, Earl T. Barr, Soline Ducousso, and Zheng Gao. Typilus: neural type hints. In Alastair F. Donaldson and Emina Torlak (eds.), Proceedings of the 41st ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2020, London, UK, June 15-20, 2020, pp. 105, ACM, 2020. doi:
URL luttps://dol org/10. 1145/3385412.33888
Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. Self-supervised bug detection and repair. In
Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (ods.), Advances in Newral Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 27865 27876, 2021. URL httpps:/prroceeingg... http://dggre.eerrss../////////// ea96ef SaS0fff000aaaeee111100occCCoooo0
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. A general path-based representation for predicting program properties. In Jeffrey S. Foster and Dan Grossman (eds.), Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PL.DI 2018, Philadelphia, PA, USA, June 18-22, 2018, pp. 404-419. ACM, 2018. doi: 1145/3192366 3192412
URL attps://doi.org//
33

---

doi: 10:006/0A4S0C:20001000 Published in Transactions on Machine Learning Research (09/2024)
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured representations ions of code. In 7th International Conference on Learning Represennalions, TCLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019a. URL httpps:/pppemrrrie...ee////oaaet
HigXYo09tX
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: learning distributed representations of code. Proc. ACM Program. Lang., 3(POPL)):01114::2) 2019b. doi: 10.1145/3290388 URL https: //doi. org/10..//0 10. 1145/3290053
Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Evend, volume 119 of Proceedings of Machine Learning Research, pp. 245 256. PMLR, 2020. URL http://
proceedings. mlr. httpe//v11/aaloo2aa.me
Hussein Alrubaye, Mohamed Wiem Mkaouer, Igor Khokhlov, Leon Reznik, Ali Ouni, and Jason Mcgoff. Learning to recommend third-party library migration opportunities at the API level. Appl. Soft Comput., 90:106140, 2020.
URL https://doi.oor aaooo..s..s...............o..a.. asoc 2020.
106140
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Kedcio-Kediiorsii Yejin Choi, and Hannaneh Ha- jishirzi. Mathqa: Towards interpretable math word problem solving with pperation-based formalisms. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 2357 2367. Association for Computational Linguistics, 2019. doi: 10.18653/v1/nI1 1245. URL https://doi. org/ 10. 11t.11111/11111111111
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katic Millican, David Silver, Slav Petrov, Melvin Johnson, Ioamnis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat,, James Molloy, Michael Tsard, Paul Ronald Barham, Tom Hennigan, Ben- jamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Karcem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Tain Barr, Nikolay Savinov, Tvo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mchran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Syg- nowski, and et al. Gemini: A family of highly capable multinodal models. CoRR, abs/ 2212..1880,, 2023a. doi: 10. 48.48550/ ARR50//AXIV ARXIV 2882, URL https: //doi otto//..rr/11..1 48550/ arxiv 11805
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robin- son, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Ábrego, Junwhan Ahm, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette- Choo, Aakanksha Chowd- hery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebas- tian Gehrmann, Lucas Gonzalez, and et al. Palm 2 technical report. CoRR, abs//205..nn 10403,, 2023b. doi: 10.48550/ard aiv. 2tw...235...4444
URL htttps://do https://doi.org//// 100 485511...85/4481 48550//801iv arxiv. 2305. 10403
Samuel Arcadinho, David Aparício, Hugo Veiga, and António Alegria T5QL: taming language models for SQL generation. CoRR, 0t////229.1222 2022. doi: 2t0p://AAAXIV2220.11222 URL https:
7/doi. 4wt/:///a../0///........ 2209. 10254.
Jordi Estango--Estapé and Michael F. P. O'Boyle. Learning C to x86 translation: An experiment in neural compilation. CoRR, abs/2108.07639, 2021. URL httpp://arrzw.orr/abbsss 07639
34

---

Published in Transactions on Machine Learning Research (09/2024)
Jordi Armengol-Estapé, Jackson Woodruff, Chris Cummins, and Michael F. P. O'Boyle. Slade: A portable small language model decompiler for optimized assembler. CoRR, abs/22055. 12520, 2023. doi: 10.48550/ ARXIV 2205.12200. URL httpp:://oii org/10.....//... www./.....//////a/...//// 088504455Xiii 200550//XXiv.3055 12520
Chetan Arora, Mehrdad Sabetzadeh, Lionel C. Briand, and Frank Zimmer. Automated checking of confor- mance to requirements templates using natural language processing. IEEE Trans. Software Eng., 41 (10): 944 968, 2015a. doi: (0.1109/TSE.2215.22. 2428779. URL https://doi..ooo org/ 10. 1109/TSE. 2000 2428709
Chetan Arora, Mehrdad Sabetzadeh, Arda Goknil, Lionel C. Briand, and Frank Zimmer. Change impact analysis for natural language requirements: An NLP approach. In Didar Zowghi, Vincenzo Gervasi, and Daniel Amyot (eds.), 23rd IEFE International Requirements Engineering Conference, RE 2015, Ottawa, ON, Canada, August 24-28, 2015, pp. 6 15. LEEE Computer Society, 2015b. doi: 10.1109/RE.2000 7320403. URL httpss//doi. org/10.11 1109/RE RE. 2015.7320400
Chetan Arora, Mchrdad Sabetzadch, Lionel C. Briand, and Frank Zimmer. Extracting domain models from natural-language requirements: approach and industrial evaluation. In Benoit Baudry and Benoît Combemale (eds.), Proceedings of the ACM/IEEE 19th International Conference on Model Driven Engi- neering Languages and Systems, Saint-Male Prance, October 2-7, 2016, pp. 250 260. ACM, 2016. URL
http://dl.. acm. www.aat......................... http:ciaatiinn...o wtwpcd ott.::icaaa.oo.ccfddd77777777771
Chetan Arora, Mehrdad Sabetzadeh, Lionel C. Briand, and Frank Zimmer. Automated extraction and clustering of requirements glossary terms. IEEE Trans. Soflware Eng., (30) 945, 2017. doi: 10.
URL https://doi..c.co http://ooooorrg////// 111911...111.11.1111111111 2016. 2635134 206.13351134
Chetan Arora, John Grundy, and Mohamed Abdehrazek.. Advancing requirements engincering through gen- erative AI: assessing the role of 1lms. CoRR, abs/2210.13976, 2023. doi: 11.48550AARXTI 2210V.2310. 13976.
URL bettps//ddo..org org/10. (8550/arXiv 2310 13976
Chetan Arora, John Grundy, Louise Puli, and Natasha Layton. Towards standards-compliant assistive technology product specifications via llms. CoRR, abs/2404.00 00s/2444.0312, 2024. doi: (0.48550/ARXIV.20 2404.
03122. URL https://doi org/ /0/ 48540/55///// arviv. 2404. 2404.03122
Ellen Arteca, Sebastian Harner, Michael Pradel, and Frank Tip. Nessie: Automatically testing javascript apis with asynchronous callbacks. In 44th JEEE/A ACM 44th International Conference on Software Engincering, TCSF 2022, Pillsburgh, PA, USA, May 25-27, 2022, pp. 1594-1505. ACM, 2022. doi: 10.1145/3510003
3510106. URL https://doi. org/10. 1145/3510003 3510016
Fiorella Artuso, Giuseppe Antonio Di Luna, Luca Massarelli, and Leonardo Querzoni. In nomine function: Naming functions in stripped binaries with neural networks, 2021.
Owura Asare, Meiyappan Nagappan, and N. Asokan. Is github's copilot bad as humans at introducing vulnerabilities in code? Empir. Softw. Eng., 28(6):129, 2023. doi: 0111//111664--2--100000 URL
https://doi.org. 10.
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Ud- din Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Kr- ishna Ramanathan, and Ramosh Nallapati. Multi-lingual evaluation of code generation models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Open 2023. URL
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL
Batuhan Agroglu, Büşta Rümeysa Mete, Eyyüp Yıldzz, Yağz Nalçakan, Alper Sezen, Mustafa Dağtekin, and Tolga Ensari. Automatic html code generation from mock-up images using machine learning techniques. In 2019 Scientific Meeting on Elletrooncssssssroniss & Biomedicol Engineering and Computer Science (FBBT), pp. 1 1-4, 2019. doi: 10.1109/EBBBT 201
35

---

Published in Transactions on Machine Learning Research (09/2024)
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.064500
2016. http:://aaxxvv.rgabbbbee 064450
Hannah McLean Babe, Sydney Nguyen, Yanglian Zi, Arjun Gula, Molly Q. Feldman, and Carolyn Jane Anderson. Studenteval: A benchmark of student- prompts for large language models of code. doi: 10.48550/ ARXIV .2306.04556. URL httts:///doi.orr/11...
CoRR, /2///2200005555 2023,
arXiv. 2306.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to aligu and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
URL http://arxiv... http:://aaxxvvoor/aaaaba /tg/abs//a 1409.0477
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htttp::/ wttp:///oooounidddSaaaaaoo
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Kemning Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. CoRR, abs/2309.16600 2023.
doi: 10.48550/ar http//aa.............ee 2309.16609. URL https: :
//doi. org/10 100 48550/arXiv 2309. 16609
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Doddss, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovilt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforeement lcarning from human feedback. CoRR, 0es/2204.05862. 2022. doi: 10.48550/AR11 ARXIV. 025622... URL https: //doi. www...../..////........ 48850/arXiv.... 2204. 220..05862.
Ramakrishna Bairi, Athary Sonwane, Aditya Kanade, Vageosh D. C, Arun Iyer, Suresh Parthasarathy, Sriram K. Rajamani, Balasubramanyan Ashok, and Shashank Shet. Codeplan: Repository-level coding using ilms and planning. CoRR, abs /t////209119999 2023. doi: 00.48550/ARX// (2OXX, 2309,12499,
URL https:
7/doi. org/ 10. 48550/arXiv 2309.020900 12499.
Noor Hasrina Bakar, Zarinah Mohd Kasirun, Norsaremah Salleh, and Hamid Abdullah Jalab. Extracting features from online software reviews to aid requirements reuse. Appl. Soft Comput., 49:1297-1315, 2016. doi: 10. 1001//.AAO...01...0..0
URL htttp:://doi...oo http:///......//...11.10...////. wtt.:////.................. asoc. 2016. 07.
Vipin Balachandran. Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. Tn David Notkin, Betty H. C. Cheng, and Klaus Pohl (eds.), 35th International Conference on Software Engineering, ICSE '13, Son Francisco, CA, USA, May 18-26, 2013, pp. 931-940. IEEE Computer Society, 2013. doi: 10.1109/ICSE.2011 266666666......66666 URL https: //doi.org/10. 11tp:////...110010...00000 26166666
6606642.
Matej Balog, Alexander L. Gaunt, Mare Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Tearning to write programs. In 5th International Conference on Learning Representations, TCLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:
Upul Bandara and Gamini Wijayarathna. Deep neural networks for source code author identification. In
Minho Lee, Akira Hirose, Zeng- GoaggGuang Hou, and Rhee Man Kil (ods.), Neural Information Processing 20th International Conference, ICONIP 2013, Daegu, Korea, Nowember 3-7, 2013. Proceedings, Part II, volume
36

---

Published in Transactions on Machine Learning Research (09/2024)
8227 of Lecture Notes in Computer Science, pp. 368- 375. Springer, 2013. doi: 10.1007 111007/978-3-662-420222 46. URL.
Pratyay Banerjee, Kuntal Kumar Pal, Fish Wang, and Chitta Baral. Variable name recovery in decompiled binary code using constrained masked langnage modeling. CoRR, abs/2103.12800 12801, 2021. URL https:
//arxiv.org/ wtt../r///bbbaag.//a abs/2103. 12801
Aakash Bansal, Sakib Haque, and Collin McMillan. Project-level encoding for neural source code summa- rization of subroutines. In 29th JEEE/ACM International Conference on Program Comprehension, ICPC 2021, Madrid, Spain, May 20-21, 2021, pp. 253 264. IEEE, 2021. doi: URL https://doi htop:::/doi.orr//..... 1109//IPCC22811.... 00..000200000
2002100002.
Patrick BarciB, Beatriz Souza, Marcelo d' d'Amorim, and Michael Pradel. Code generation tools (almost) for free? A study of few-shot, pre-trained language models on code. CoRR, abs/2206.01335, 2022. doi: 10.48550/arK00 www.220660133355 URL hittps://dol http::///ooo.o////////////////// www..o......////..... 48550/arXiv. 2206. 01335
Mike Barnett, Christian Bird, João Brunet, and Shuvendu K. Lahiri. Helping developers help themselves: Automatic docomposition of code review changesots. In Antonia Bertolino, Gerardo Canfora, and Sc- bastian G. Elbaum (eds.), 37th IEEE/ACM International Conference on Software Engineering, ICSE 2015, Florence, Italy, May 16-24, 2015, Volume 1, pp. 134-144. IEEE Computer Society, 2015. doi: DAL1109/ICSE.201..300 URL
1ttp:/10.119//ICSS 2015.33
Antonio Valerio Miceli Barone and Rico Sennrich. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. In Greg Kondrak and Taro Watanabe (eds.). Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27 December 1, 2017, Volume 2: Short Papers, pp- 314 319. Asian Federation of Natural Language Processing, 2017. URL hatp::://acaaatthol.ee wwt//o////////////////////.oo 1005///
Ezio Bartocci, Leonardo Mariani, Dejan Nickovic, and Drishti Yadav. Property-based mutation testing. In IEEE Conference on Software Testing, Verification and Validation, ICST 2023, Dublin, Ireland, April 16-20, 2023, pp. 222 233. TEFE, 2023. doi: 10. 10.1109/CCST5711222288888 00029. L109/ICST57152.200 00029
URL httpp:://dol httpp:://doooor/10...
Nauman Bashir, Muhammad Bilal, Misbah Liaqat, Mohsen Marjani, Nadia Malik, and Mohsin Ali. Modeling class diagram using nlp in object-oriented designing. In 2021 National Computing Colleges Conference (NCCC), pp. 1-6, 2021. doi: 10. 10.1109/ /NCCCCC00030022219995555
Mohammad Bavarian, Hoewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. CoRR, abbs/2207.1122 2022. doi: 10.48550/ arvi 220.i...27..... 220...125..... URL https://doi. http:/oo..orrrr/ org/oorrr/o 48550/arXii .207 14255
Rohan Bavishi, Caroline Lemieux, Roy Fox, Koushik Sen, and lon Stoica. Autopandas: neural-backed generators for program synthesis. Proc. ACM Program. Lang., 3(000PSL4)::1:11118227, 2019. doi:
10. 10.1145/3360594. URL https://dd/: /ttp:::://ooi.or//1.... 0144/33336554
Tony Beltramelli. pix2code: Generating code from graphical user interface screenshot. In Proceedings of the ACM SIGCHI Symposiun on Engineering Interactive Computing Systems, EICS 2018, Paris, France, June 19-22, 2018, pp. 3:1-3:6. ACM, 2018. doi: 10. 10.1145/32200144332200 URL https:///oi.orgg: http///oooorr//.... 11 0t05//2222134..2222112
Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hocfler. Neural code comprehension: A learnable representation of code semantics. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi. and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 9-8, 2018, Montréal, Canada, pp. 3589-3601, 2018. URL http:///ppoodddnggg.ooror....cpp
T7a23feec21557000aaaaaaaaaao0 -bs.ttttttt--btbtrtttt...... html
37

---

Published in Transactions on Machine Learning Research (09/2024)
Berkay Berabi, Jingxuan He, Veselin Raychev, and Martin T. Vechev. Thi:: Learning to fix coding errors with a text-to-text transformer. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, JCML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 780-791. PMLR, 2021. URL http::/proceedinng/ mlr https/einnnsmml....psss////9/ press/v13//
berabi2la.htm
Guru Prasad Bhandari, Amara Naseer, and Leon Moonen. Cvefixes: automated collection of vulnerabilities and their fixes from open-source software. In Shanc McIntosh, Xin Xia, and Sousuke Amasaki (ods.), PROMISE '11 17th International Conference on Prediclive Models and Dala Analylics in Software Engi- neering, Athens Greece, August 19-20, 2021, pp. 30 39. ACM, 2021. doi:
URL
https://doi.orge: ww.//o.///......
Sahil Bhatia, Pushmeet Kohli, and Rishabh Singh. Neuro-symbollcc program corrector for introductory programming assignments. In Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (eds.), Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenbuy,, Sweden, May 27- June 03, 2018, pp. 60-70. ACM, 2018. doi: 10.1145/318015551180222 URL https: 7/doi. org/ 10. 100.111//3111155333111
Yingzhou Bi, Jiangtao Huang, Penghui Liu, and Lianmei Wang- Benchmarking software vulnerability de- tection techniques: A survey, CoRR,
2023. doi: 10.48550/ARXIV.22000 2603..666.... URL
https://doi.. http///oo..ro////// 10/ 48550/855/a/0 2at000//rivv... 2303. 16362
Zhangqian Bi, Yao Wan, Zheng Wang, Hongyu Zhang, Batu Guan, Fangxin Lu, Zili Zhang, Yulei Sui, Xuanhua Shi, and Hai Jin. Iterative refinement of project-level code context for precise code generation with compiler feedback. CoRR, a2433 16792, 2024. doi: 10.48550/ ARXIV 1403.16792. URL https: 7/doi org/10. 48550/arXiv.. 2403 2409.16792.
Pan Bian, Bin Liang Wenchang Shi, Jianjun Huang, and Yan Cai. Nar-miner: discovering negative as- sociation rules from code for bug detection. In Gary T. Leavens, Alessandro Garcia, and Corina S. Pasarcanu (eds.), Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Con- [erence and Symposium on the Foundalions of Software Engineering, ESEC//SISOFT FSE 2018, Lake Buena Visto, FL, USA, November 04-09 2018, pp. 411 422. ACM, 2018. doi: 10.145//2330224.233002. URL http::///...org//0011111333 11t.//...11133221110 3236032 3236032
Benjamin Bichsel, Veselin Raychev, Petar Tsankov, and Martin T. Vechev. Statistical deobfuscation of an- droid applications. In Edgar R. Weippl, Stefan Katzenbeisser, Christopher Kruegel, Andrew C. Myers, and Shai Halevi (ods.), Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Securily, Vienna, Austria, October 24-28, 2016, pp. 343 355. ACM, 2016. doi: 10.115//297674922977822 URL /tttp:://a1i.or/11..11111 10t../0.111//9767/414111111 2978422
Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the pile. CoRR, abs/2201.07311, 2022. URL
https://arxis: wttpvvv.org//ab/221..0702
Pavol Bielik, Veselin Raychev, and Martin T. Vechev. PHOG: probabilistic model for code. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 39nd International Conference on Machine Learning, 1CML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 2933-2942. JMLR.org, 2016. URL http://procceeingg. mlr press/v48, bielik16.hta.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-201B An open-source autoregressive language model. In Angela Fan, Suzana Ilic, Thomas Wolf, and Matthias Gallé (eds.): Proceedings of BigSciince Episode #5 - Workshop on Challenges & Perspectivess in Creating Lorge Lan- guage Models, pp. 95-136, 10. 10.18653/v /2t////12222.//// wwgg.2..igcienncee111 1.9. URL httpt:tt////ett htatpp://aclanthologyy... wtg/2022 http:////22bbiigicceeeeeeeeeeeee bigscience-1.9
May 2022. Association for Computational Linguistics. doi:
38

---

Published in Transactions on Machine Learning Research (09/2024)
Arianna Blasi, Alessandra Gorla, Michael D. Ernst, Mauro Pezzè, and Antonio Carzaniga. Memo: Automat- ically identifying metamorphic relations in javadoc comments for test automation. J. Syst. Softus, 181: 111041, 2021. doi: j0.1016/..ss..221..111
URL https:///oi.orr//100 1016//j....111......... ja..jjs.22..11004..
Ben Bogin, Jonathan Berant, and Matt Gardner. Representing schema structure with graph neural networks for text-to-sql parsing. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 4560-4565. Association for Computational Linguistics, 2019. doi: 10.18653/v1 111/11--1448 URL hattps://dot http:/0.11665/11111111555
Marcol Böhme, Van- Thuan Pham, Manh- Nguyen, and Abhik Roychoudhury. Directed greybox fuzzing. In Bhavani Thuraisingham, David Evans, Tal Malkin, and Dongyan Xu (eds.), Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, October 30- - November 03, 2017, pp. 2329 2344. ACM, 2017. doi: 10.1145/3133356.3134444 https://dos /tw. org/ 10. 1001111//1339663133300 3134020
URL
Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. Coverage-based greybox fuzzing as markov chain. IEEE Trons. Software Eng., 45(5))489-500 2019. doi: 00.1109/SSE.2017.25555 URL https:
//doi.org// 1ttg//1..119/TSE... 2017 2785841
Casper Boone, Niels de Bruin, Arjan Langerak, and Fabian Stelmach. Dltpy: Deep learning type inference of python function signatures using natural language context. CoRR, abs/19 hass/992.00680, 2019. URL http:
7/arxiv.org
00680
David Bingham Brown, Michael Vaughn, Ben Liblit, and Thomas W. Reps. The care and feeding of wild- caught. mutants. In Fric Bodden, Wilhelm Schäfer, Arie van Deursen, and Andrea Zisman (eds.), Proceed- ings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Pader- born, Germany, Seplember 4-8, 2017, pp. 511 522. ACM, 2017. doi: 10.144//106223.3100288 URL https://doi org/10..11.... 10. 11t.//111111....... 31062880
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Pro- cessing Systems 39: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL Abstract html
Marcel Bruch, Martin Monperrus, and Mira Mezini. Learning from examples to improve code completion systems. In Hans van Vliet and Valérie Issarny (eds.). Proceedings of the 7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2009, Amsterdam, The Netherlands, August 24-28, 2009, pp. 113-222. ACM, 2009. doi: 10.1145/1506666.1111 1595699611955725 URL https:://oi..orrr. 11tp:///...14//1555566666 1595728.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, 2023. doi: 10.48550/ 1148850///Xivv.22/ wtt.:////X.....3..12222 12712.. URL htttps://coi..rg//// wtt..//.///......0.... arXiv www.iv.303.......... 12712
Nghi Bui, Yue Wang, and Steven C. H. Hoi. Detect-localize-repair: A unified framework for learning to debug with codet5. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. $12-823. Association for Computational Linguistics, 2022. doi: (..1653//11/200 SENOPP.IININNGSEEAE 57. URL htttp:://oo..rrg//00B6663// 10tp//000186533/112222222222 9m.
39

---

Published in Transactions on Machine Learning Research (09/2024)
Nghi D. Q. Bui, Yijun Yu, and Lingxiao Jiang. SAR: learning cross-language API mappings with little knowledge. In Marlon Dumas, Dietmar Pfahl, Sven Apel, and Alessandra Russo (eds.), Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFE FSE 2019, Tallinn, Estonia, August 26-30, 2019, pp. 796-806. ACM, 2019. doi: 2..144/3338906.333
URL https: wtt.://....rr/11111111113331
3338924
Nghi D. Q. Bui, Yijun Yu, and Lingxiao Jiang. Infercode: Self-supervised learning of code representations by predicting subtrees. In 43rd TEEE/ACK ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pp. 1186 1197. IEEE, 2021a. doi: 10.11090 /1SSEE43992.2020000000 URL
https://doi. org/ 10. 111111199IIIET IAt///ICSEEEEE...E 2021 00109
Nghi D. Q. Bui, Yijun Yu, and Lingxiao Jiang. Self-supervised contrastive learning for code retrieval and summarization via semantic- preeerrvipr transformations. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.), STGIR '21: The 44th International ACM STGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11- 15, 2021, pp. 511 521. ACM, 2021b. doi: 10.1465/300435:34622800 URL https://dol http::///ooooo///.111111 1145/ 3404835 3462840)
Rudy Bunel, Matthew J. Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmeet Kohli. Leveraging grammar and reinforcement learning for neural program synthesis. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.nel, 2018. URL https::/opperreiiee www.:neeeoooummidiiiiiiiio
Javier Cámara, Javier Troya, Lola Burgueño, and Antonio Vallecillo. On the assessment of generative Al in modeling tasks: experience report with chatgpt and UML Softw. Syst. Model., 22(3):781-793, 2023. doi: 10.1007 LL1007/112200--22 0230000333000 01105-5. URL https://doi. 10. 1007/ 0tt1://111200002-011000
José Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, and Satish Chandra. When deep learning met code search. In Marlon Dumas, Dietmar Pfahl, Sven Apel, and Alessandra Russo (eds.), Proccedings of the ACM Joint Meeting Europeun Software Engineering Conference and Symposium the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2019, Tallinn, Eslonia, August 26-30, 2019, pp. 964 974. 3340458
ACM, 2019. doi: 10.1145 40444//333390633400000
URL httpp:://oot org/ 10.
Jialun Cao, Meiziniu Li, Ming Wen, and thing-Chi Cheung. A study prompt design, advantages and limitations of chatgpt for deep learning program repair. CoRR, 20040090, 2023. doi: 10.48550/
URL https:://oi. org/10..//000/ 48550/85551i www.aawwwww....aa...e.......e..e
Ruisheng Cao, Lu Chen, Zhi Chen, Yanbin Zhao, Su Zhu, and Kai Yu. LGESQL: line graph enhanced text- to-sql model with mixed local and non-local relations. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, (CL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 2541 Association for Computational Linguistics, 2021. doi:
10.18653/v1 aall///lalloooooooggggo 198. URL httpp:://ddoiooooee 1tt////..1115//1/2/2/222 acl-ooono.......oooo.o.... 198
Ying Cao, Ruigang Liang, Kai Chen, and Peiwei Hu. Boosting neural networks to decompile opti- mized binaries. In Annual Computer Security Applications Conference, ACSAC 2022, Austin, TX, USA, December 5-9, 2022, pp. 508 518. ACM, 2022. doi: 10.1155/856662333577988 URL https:
aidoi.or//ia 10//10.1100 11555/556462..5577777
Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Carolyn Jane Anderson, Michael Greenberg, Abhinav Jangda, and Arjun Guha. Knowledge transfer from high-resource to low-resource programming languages for code 1ms. CoRR, wwws/3008.000.0 09895, 2023a. doi: 10.48550/ ARXIVV 22..2.280899 09895. URL https://doi. org/10 1005008850//aaa/ arviv. 2308 09895
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-c: a scalable and polyglot approach to benchmarking neural code generation. IEFE Transactions on Software Engineering. 2023b.
40

---

Published in Transactions on Machine Learning Research (09/2024)
Federico Cassano, Ming-Ho Yee, Noah Shimn, Arjum Guha, and Steven Holtzen. Type prediction with pro- gram decomposition and fill-in-thetyppe training. CoRR, abs 222223351111 2023c. doi: 10.48550/ 118550/AAXXV.... ARXIV. 2305.17145. URL https://doi. org/10. 48550/arXix arxiv. 2305 17145
Sang Kil Cha, Maverick Woo, and David Brumley. Program-adaptive mutational fuzzing. In 2015 IEEE Symposium on Security and Privacy, SP 2015, San Jose, CA, USA, May 17-21, 2015, pp. 725 741. LEEE
Computer Society, 2013. doi: 10.1109/SP.2015.556 URL https://t/ooooo org/10 1109/SP 2015 50
Meriem Ben Chaaben, Lola Burgueño, and Houari A. Sahraoui. Towards using few-shot prompt learning for automating model completion. In 45th IEEE/ACM International Conference on Software Engineering:
New Ideas and Emerying Resulls, NIER@ICSE, Melbourne, Australia, May 14-20, 2023, pp. 7-12. TEEE, 2023. doi: 10.1109/ICSB /IECIISSSENEESSSTT... 200.0088..7222.000000 URL https://doi..oo org/ 10. A1110///CS-NNIIEE 2023. 00008
Hyungjoo Chae, Yeonghyeon Kim, Seungone Kim, Kai Tzu iumn Ong, Beong woo Kwak, Moohyeon Kim, Sconghwan Kim, Tacyoon Kwon, Jiwan Chung, Youngjac Yu, and Jinyoung Yoo. Language models compilers: Simulating pseudocode execution improves algorithmic reasoning in language models. ColR, abs/ abs/2440.00257,, 2024.
doi: 10.48550 (18550/ARXXV..O AAX.V.240....25.55 2404.02555. URL https://doi: org/10 10. 00///00.4850/rr// 48550/arXiv.
Yekun Chai, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, and Hua Wu. Ernie-code: Beyond english- centric cross-lingual pretraining for programming languages. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.); Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 10628-10650. Association for Computational Linguisties, 2023. doi: 10. (8653/V1/2022 WWT/222.FFIIDINGGS. ACL.666..
http://a....rrr......6/v//////3/ wtw.::gs...innnnnaac..... URL hattts://aoi.orrg/11 10....1163//11////
Yitian Chai, Hongyu Zhang, Beijun Shen, and Xiaodong Gu. Cross-domain deep code search with meta learning. In 44th TEFE/ACH 44th International Conference on Software Engineering, ICSE 2022, Pitts- burgh, PA, USA, May 25-27, 2022, pp. 487-498. ACM, 2022. doi: 10.1145/3510008.35500000 3510125
URL
https://doi.orgg 10.
Saikat Chakraborty and Baishakhi Ray. On multi-modal learning of editing source code. In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021, pp. 443 455. IEEE, 2021. doi: 1AL1009/ASE515222222....99 2021 9678559.
URL https://doi. org/
10. 10.1109/ASE5
Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar T. Devanbu, and Baishakhi Ray. Nat- gen: generative pre-training by 'naturalizing' source code. In Abhik Roychoudhury, Cristian Cadar, and Miryung Kim (eds.), Proceedingg of the 30th ACM Joint European Sofbware Engineering Confer- ence and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Sin- gapore, November 14-18, 2022, pp. 18-30, ACM, 2022a. doi: 10. 10.11455/5540250.5549111 aeloi.org//00. 1145/3540200.33665 3549162.
URL https:
Saikat Chakraborty, Yangruibo Ding, Miltiadis Allamanis, and Baishakhi Ray. CODIT: code editing with tree-based neural models. IEEE Trans. Software Eng., 48(4):1385 1399, 2022b. doi: 10. 10.1109/TSE.2000.
3020502. URL https://doii org/10 10. 1109/TSE. 2020 3020502
Saikat Chakraborty, Rahul Krishna, Yangruibo Ding, and Baishakhi Ray. Deep learning based vulnerability detection: Are we there yet? IEEE Trans. Software Eng., (28)333803222))) 2022c. doi: 10.1109/TTI LSEE 0027.30874 URL attpss///oi....
10. 1119//1E..2222 2021. 3087402
Aaron Chan, Anant Kharkar, Roshanak Zilouchian Moghaddam, Yevhen Mohylevskyy, Alec Helyar, Eslam Kamal, Mohamed Flkamhawy, and Neel Sundaresan. Transformer-based vulnerability detection in code at edittime: Zero-shot, few-shot, or fine-tuning? CoRR, abs/2306.01154, 2023. doi: 0048550/ARXII
2306.01754. URL hhttps:///oll. wwt.:o..///...//....////./////// arXiv. 23306.07544
Shubham Chandel, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. Training and evaluating a jupyter notebook data science assistant. CoRR, abs/2201.1290 2022. URL htttps://arxt htt////axvvvor/aaabbaa org/abs/ 2201 12901
41

---

Published in Transactions on Machine Learning Research (09/2024)
Shuaichen Chang and Eric Fosler-Lussier. How to prompt Ilms for text-to-sql: A study in zero-shot, single- domain, and cross-domain settings. CoRR, abs abs/2300 /2055 11853, 2023. doi: 0055500/RXTV.2366 URL
https://aoi.orr/100
/t/p:///0...0//11..485555/////// 1...0..855//a/X15 2305. 11853
Bei Chen, Fongji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a. URL
http:ss///pperreieeeeeeeteee http://wwww.eeeddddddddddtdeoeee ?id=ktr668ttugg
Boqi Chen, Kua Chen, Shabnam Hassani, Yujing Yang, Daniel Amyot, Lysanne Lessard, Gunter Muss- bacher, Mchrdad Sabetzadch, and Dániel Varró. On the use of GPT-4 for creating goal modcls: An exploratory study. In Kurt Schneider, Fabiano Dalpiaz, and Jennifer Horkoff (eds.), 31st IEEE Interna- tional Requirements Engineering Conference, RE 2023 Workshops, Hannover, Germany, Seplember 4-5, 2023, pp. 262 271. IEEE, 2023b. doi: REW57809. 2023. 00052
2023..002222 URL http:://///o..//..1111111
Chunyang Chen, Zhenchang Xing, Yang Liu, and Kent Ong Long Xiong. Mining likely analogical apis across third-party libraries via large-scale unsupervised API semantics embedding. IEEE Trans. Software Eng., 47(3):432 447, 2021a. doi: 200////S..219.222..2
URL https::/doi. wrt.o/oooorg//..o 1rg//..1119///TE........
2019.289611
Fuxiang Chen, Fatemeh H. Fard, David Lo, and Timofcy Bryksin. On the transforability of pre-trained language models for low-resource programming languages. In Ayushi Rastogi, Rosalia Tufano, Gabriele Bavota, Venera Arnaoudova, and Sonia Haiduc (eds.), Proceedings of the 30th TEEF/ACK International Conference on Program Comprehension, ICPC 2022, Virtual Event, May 16- 17, 2022, pp. 401 412. ACM, 2022. doi: 10.1145/ 011144/35246100 01524610.3527991
URL https://doi 10.//0...///...... 10000...4555 1145/ 35271
Kua Chen, Yujing Yang, Boqi Chen, José Antonio Hernández López, Gunter Mussbacher, and Dániel Varró. Automated domain modeling with large language models: A comparative study. In 26th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, MODELS 2023, Västerás, Sweden, October 1-6, 2023, pp. 162 172. IEEE, 2023c. doi: D0.108//ODDELSS88522000000 https://doi.. org/ 10. 11100//00EEE8015..... 2013.
URL
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Ka- plan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodoi, Sam McCandlish, Ilya Sutskever, and Woj- ciech Zaremba. Evalusting large language models trained on code. CoRR, abs//2107.0337, 2021b. URL https://arxiv.. wrw..rrggbbba... abs/ 2207. 03374
Peng Chen and Hao Chen. Angora: Efficient fuzzing by principled search. In 2018 IEFE Symposium Security and Privacy, SP 2018, Proceedings, 21-23 May 2018, San Francisco, California, USA, pp. 711 725. IEEE Computer Society, 2018. doi: St009//PP..00.0000000000 URL 00046)
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. CoRR, //////220..5555 2023d. doi: 10. 10.48550/ARXC .2066. 15595. URL https: 2.0..48850//a1111
15595
Wenhu Chen, Xucguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Dis- entangling computation from reasoning for numorical reasoning tasks. Transactions on Machine Learning Research, 2023e. ISSN 2835-8856. URL httpp://gaaaaaarer..eeee////aaat
42

---

Published in Transactions on Machine Learning Research (09/2024)
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation. In Sany Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Resa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Confer- ence on Neural Information Processing Systems 2018, NewrIPS 2018, December 9-8, 2018, Mon- Iréal, Canada, pp. 2552 2562, 2018. URL ptttps:/procceeding. wwt.nnps..c.paaeeeeeee.e.ee wc/paper//018 8/hash/
Xinyun Chen, Linyuan Gong, Alvin Cheung, and Dawn Song. Plotcoder: Hierarchical decoding for syuthe- sizing visualization code in programmatic context. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Compulational Linguistics and the 11th International Joint Conference Natural Language Processing, ACL/IJCNIL 2021, (Vol- ume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 2169-2181. Association for Computational Linguisties, 2021c, doi:
URL httph::/to/o:t http://o....rr11..118///////////
acl acl-long... 169
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self- debug. CoRR, abs abs/2304.05128, 2023f. doi: 10.488550/aXXiv.2300 23..000.....22001222... URL https://doi.org// 48550/arXiv. 2304 05128
Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. Comments as natural logic pivots: Improve code generation via comment perspective. CoRR, abs/2404.07549,, 2024. doi: 10.48550/
URL https://doi. http:///o....g//..... www.o//./////////////////// http//aavvvvv....... 07549
Yizheng Chen, Zhoujic Ding, Lamya Alowain, Xinyun Chen, and David A. Wagner. Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection. In Proceedings of the 26th International Symposium Research in Attacks, Intrusions and Defenses, RAID 2023, Hong Kong, China, October 18, 2023, pp. 654-668. ACM, 2023g. doi: 10.1145/300719993300722 URL. https: 3607242
aedoi.org/10..
Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, and Michael R. Lyu. Experience report: Deep learning- based system log analysis for anomaly detection. CoRR, abs/2107.05900 2021d. URL https:
05908
Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noë Noël Pouchet, Denys Poshyvanyk, and Martin Mon- perrus. Sequencer: Sequence-to-sequene learning for end-to-end program repair. IEEE Trons. Software Eng., 1799))1943-1955, 2021c. doi: 299009990.199.990770... URL https://dot: org/10. /0/.//1..10 1109/TSE.
2019.2940177
Chin-Yi Yi Cheng, Forrest Huang, Gang Li, and Yang Li. Play: Parametrically conditioned layout generation using latent diffusion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (cds.), International Conference on Machine Learning, ICML 2023, 23- 29 July 2023, Honolulu, Hawwii, USA, volume 202 of Proceedings of Muchine Leurning Research, pp. 5449 5471. PMLR, 2023. URL https::/prooceeding/ mlr wwt.emmer....eeeeeeeeee 2200///2022/ cheng23b. html
David Chiang. Hicrarchical phrasc-based translation. Comput. Linguistics, 33(2):201-22) 2012/coi.2007..333 20.2 URL https: http:///660...r1010116//////////
2007. doi: 10.
10. 110.1112/ccl... 22011///oo...07....... 2007.33.
Muslim Chochlov, Gul Aftab Ahmed, James Vincent. Patten, Guoxian Lu, Wei Hou, David Gregg, and Jim Buckley. Using a nearest-neighhour, bert-based approach for scalable clone detection. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2022, Limassol, Cyprus, Oclober 3-7, 2022, pp. 582 591. IEEE, 2022. doi: 10.108//CCMM55501:22220002 //doi orgo 10. 1119/111S0/510011 2022.
URL https:
DongHyun Choi, Mycongcheol Shin, EungGyun Kim, and Dong Rycol Shin. RYANSQL: recursively applying sxetch-based slot fillings for complex text-to-sql in cross-domain databases. Comput. Linguisties, 47(2): 309 002, 2021. doi: 10. 10.1162/coli,, a) 00403. URI. https:://aoi.org//0 11t.2/001.112/1111111 _a0__l___00000330
43

---

Published in Transactions on Machine Learning Research (09/2024)
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 8-7, 2021. OpenReview.ncc. nect 2021. URL
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyaashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Frica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Helllseen Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1 240: 113, 2023. URL http://jmlr. otg.papers/v2A/22---122
1144. html
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Urike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advonces in Newral Information Processing Systems 30: Annual Conference Newral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4299 2017. URL htttps://proceeediii neurips.. Abstracct.bbttt html
cc/paper/200
Fenia Christopoulou, Gerasimos Lampouras, Milan Gritta, Guchun Zhang, Yinpeng Guo, Zhongqi Li, Qi Zhang, Meng Xiao, Bo Shen, Lin Li, Hao Yu, Li Yan, Pingyi Zhou, Xin Wang, Yuchi Ma, Ignacio lacobacci, Yasheng Wang, Guangtai Liang, Jiansheng Wei, Xin Jiang, Qianxiang Wang, and Qun Liu. Pangu-coder: Program synthesis with function-level language modeling. CoRR, abs/2207.11280 2022. doi: 1195500arXiv.207..
URL https:://oi...... org/ 10. 18550/arXiv 11280
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Fd H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling fintuuction--intuunee language models. CoRR, Abs//2200.1116 2022. doi: 10. 10.48550/ aiv 201..2200.1144. URL https://dol org/10 10. 48550/arXix 2210.
11416
Kevin Clark, Minh- Fiah-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openrevioor net wewoorormiirriiiiii IIMMIBBV
Colin B. Clement, Dawn Drain, Jonathan Timcheck, Alexcy Svyatkovskiy, and Neel Sundaresan. Pymt5: multi-mode translation of natural language and python code with transformers. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 9052 9065. Association for Computational Linguistics, 2020. doi: 10.18653/v1 /tt///220.emmmpppp 728. URL attps://doi.org
10. 20.18653/v1/202.. 1ttpss//i122220.eemmm-mmaaaaaa emnlp-main. 728
Colin B. Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain, Nan Duan, Neel Sundaresan, and Alexey Svyatkovskiy. Long-range modeling of source code files with ewash: Extended window access by syntax hicrarchy. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (ods.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp- 4713 4722. Association for
44

---

Published in Transactions on Machine Learning Research (09/2024)
Computational Linguisties, 2021. doi: 10. 11653/v1/20211 ewt.092..mmmmmmma.333e3 387
1T11//2222EENN--MAI...... URL https://doi.org/
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reicchrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2 2021. URL
https://arxil: /ttp:://arxiiv.org/bb//2/22 httpa//211....4 14168
Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Authony Ventresque. PIT: a practical mutation testing tool for java (demo). In Andreas Zeller and Abhik Roychoudhury (eds.): Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saar- brücken, Germany, July 18-20, 2016, pp. 459. ACM, 2016. doi: 10.1145/2931000 299.2999.......777.. URL
ittps://doi... .tw..org/gio 10. ..0.0....... 1 145/2931037 22955/99991137797777777 2948707
Christian S. Collberg and Clark Thomborson. Watermarking, tamper-proofing, and obfuscation tools for software protection. IEEE Transactions on Software Engineering, 28(8))735-77 August 2002. ISSN 0098- 5589. doi: 20100/TTF.0002.100000 and insightful comments of two anonymous referees. This material is based upon work supported by the US National Science Foundation under Grant No. 0073483.
Funding Information: The authors are grateful for the extensive
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Ro- man Garnelt (eds.), Advances in Neural Information Processing Systems 32: Annvual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 7057 7067, 2019. URL https::/prooceednng (4le1962222742bt557744433322
-wterrbbbstttttee.. html
Alexis Conneau, Kartikay Khandelval, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (ods.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 8440 8451. Association for Computational Linguistics. 2020. doi: (0.18653/v1// acl.0200...llain.... 747. URL attps://doi.
18633..1111111100 attp:///2200c11111110011a acl- meaan..477 747
Luis Fernando Cortes-Coy Mario Linares Vásquez, Jairo Aponte, and Denys Poshyvanyk. On automatically generating commit messages via summarization of source code changes. In 14th IEEE International Working Conference on Source Code Analysis and Manipulation, SCAM 2014, Victoria, BC, Canada, September 28-29, 2014, pp. 284. IEEE Computer Socicty, 2014. doi: 10.1109/SCAM.2014.1 URL https://doi.orgg 10. wtt..//////.1.../...... 1ww./SCMM./MMMMM 2014.14 14
Viktor Csuvik and László Vidáes. Fixjs: A dataset of bug-fixing javascript commits. In 19th IEEE/ACM International Conference on Mining Software Repositories, MSR 2022, Piltsburgh, PA, USA, May 23- 24, 2022, pp. 712-716. ACM, 2022. doi: (0.1145/824881..5228888 URL https: oeciii.rr/11... 1145/ 3524842. 3528480
Haotian Cui, Chenglong Wang, Junjie Huang, Jeevana Priya Inala, Todd Mytkowicz, Bo Wang, Jianfeng Gao, and Nan Duan. Codeexp: Explanatory code document generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (cds.), Findings of the Association for Computational Linguisties: EMNLP 2022, Abu Dhabi, United Arab Emirales, December 7-11, 2022, pp. 2342 2354. Association for Computational Länguistics, 2022. doi: 10.18653/V1 /2002.0FF22DNGGG 1.7..P.174. URL. httts:://oii.rgg//e v1 v1/2022. fennee.ffniinns-emn.n. 174
Siwei Cui, Gang Zhao, Zeyu Dai, Luochao Wang, Ruihong Huang, and Jeff Huang. Pyinfer: Deep learning semantic type inference for python variables. CoRR, abs/2106. 14316, 2021. URL https://arxiv..or org/
abs/2106. 14316
Chris Cummins, Pavlos Petoumenos, Zheng Wang, and Hugh Leather. End-to-end deep learning of optimiza- tion heuristics. In 26th International Conference on Parallel Architectures and Compilation Techniques, PACT 2017, Portland, OR, USA, September 131 2017, 7, pp. 219-232 IEEE Computer Society, 2017. doi: 10.1109/PACT.2000 2017.24. URL hTR.pptt::: httph:ttt////ttt /tt///ti.oogg../////// 1109/PACT 2017.22
45

---

Published in Transactions on Machine Learning Research (09/2024)
Chris Cummins, Zacharias V. Fisches, Tal Ben-Nun, Torsten Hoefter, Michael F P O'Boyle, and Hugh Leather. Programl: A graph-based program representation for data flow analysis and compiler opti- mizations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2244-2253. PMLR, 18 24 Jul 2021. URL
mlr prwss/ressssviee cummins21a..... html
Chris Cummins, Volker Seeker, Dejan Grubisic, Mostafa Elhoushi, Youwei Liang, Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Kim M. Hazelwood, Gabriel Synnaeve, and Hugh Leather. Large language models for compiler optimization. CoRR, abs/ 0ass/2309.00000 2023, doi: 10.48550/ARMN 2309.07062. URL https://doi. org/ 10. 48w50/arXiv......//aa arXiv. 2309. 07062
Deborah A. Dahl, Madeleine Bates, Michael Brown, William M. Fisher, Kate Hunicke-Smith, David S. Pallett, Christine Pao. Alexander I. Rudnicky, and Elizabeth Shriberg. Expanding the scope of the ATIS task: The ATTS-3 corpus. In Human Language Technology, Proceedings of a Workshop held al Plainsboro, New Jerey, USA, March 8-11, 1994. Morgan Kaufmann, 1994. URLptttpe http:/tt/ccaaaatttoooee //aaaaalaannoooy..... org/H94-1010/
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang- Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. CoRR, 0bs/2401.06066 2024a. doi: 10.48550 A..4550//ARXV..241.... 48550/ar 2401 060666
URL https://dol.org///
Hetong Dai, Heng Li, Che-Shao Chen, Weiyi Shang, and Tse-Hsun Chen. Logram: Efficient log parsing using SnSn-gram dictionaries. IEEE Thans. Software Eng., 48(3):879-88) 2022. doi: 10110//TSE.20203007755.. URL stttpa///doi org/10. 1109 2020. 3007554
Jianbo Dai, Jianqiao Lu, Yunlong Feng, Rongju Ruan, Ming Cheng, Haochen Tan, and Zhijiang Guo. Mhpp: Exploring the capabilities and limitations of language models beyond basic code generation. 2024b. URL
h org/ 10. 48550/ ha1550//rxiv.... w40...2450..0 11430
Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Des- marais. Effective test generation using pre-trained large language models and mutation testing. CoRR, 2023. doi: 10.48550/ARXIV.. 2..55/AAAAII..20.6665.. URL https://doi. 4055//0/r..55//..v
2308. 16557|
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awarencss. In NeurIPS, 2022. URL http://papers. http://paprr..ips...... www.miiss.spppaaaeeee cc/paper_
Filles/paper/222//aahh// 67chaahh/775777 htt.aa/ccaaraaa2222e2eeeeeeeeeee
html
Adailton Ferreira de Araújo and Ricardo Marcondes Marcacini. RE-BERT: automatic extraction of software requirements from app reviews using BERT language model. In Chil-Cheng Hung, Jiman Hong, Alessio Bechini, and Eunjee Song (eds.), SAC' '21: The 36th ACM/SIGAP! Symposium Applied Computing, Virtual Event, Republic of Korea, March 22-26, 2021, pp. 1321-1327. ACM, 2021. doi: 10.1145/3412841. 3442006. URL https://doi. httt::://doi.or/11... 1114/1444414114444220
Nelson Tavares de Sousa and Wilhelm Hasselbring. Javabert: Training transformer-based model for the java programming language. In 36th IEEE/ACM International Conference on Aulomated Soflware Engineering, ASE 2021- - Workshops, Melbourne, Australia, November 15-19, 2021, pp. 90-93 95. IEEE, 2021. 1109/ A...0///SE5555222 2021. 2021.00028
doi: 10. ASEWASSS22222222 2002200028. URL hattps:////o...oo
DeepSeek-A1, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi
46

---

Published in Transactions on Machine Learning Research (09/2024)
Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek 01:: Scaling open-source language models with longtermism. CoRR, abs/2401 2401002954, 2024. doi: 10.48550/ ARXIR TEXXIV.2401.00 2401.0294... URL htttps://oi.... org/10. 411.11..45555/005/ 2401. 02954
Deva Kumar Deeptimahanti and Muhammad Ali Babar. An automated tool for generating UML models from natural language requirements. In ASE 2009, 24th JEFE/ACM International Conference on Automated Software Engineering, Auckland, New Zealand, November 16-20, 2009, pp. 680 682. IEEE Computer Society, 2009. doi: 10.1109/ 00.1109/ASE.2000.4. 200.209..8.
URL Inttps://do... org/10 1109/ASEE.... 2009. 48
Renzo Degiovanni and Mike Papadakis. pbert: Mutation testing using pre-trained language models. In 15th IEEE International Conference on Software Testing, Verification and Validation Workshops ICST Workshops 2022, Valencia, Spain, April 4-13, 2022, pp. 160 169. IEEE, 2022. doi: 10.1109/ICSTW55330 2022.00039. URL httts::////ao cecaoi.orge org/10 10. 11199/CSTT55335 2022. 00039.
Naihao Deng, Yulong Chen, and Yue Zhang. Recent advances in text-to-sql: A survey of what we have and what we expect. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kuro- hashi, Patrizia Paggio, Nianwen Xue, Scokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12- 17, 2022, pp. 2166 2187. International Committee Computational Linguistics, 2022a. URL https: //aclanthology org wtw.orgg////..... 2wwpg//o22.....onnooo coling- 190
Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structuro-groundde protraining for text-to-sql. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 1337-1350. Association for Computational Linguistics, 2021. doi: 10.18653/VV /V1/2021. NAACI-MAIN.10S URL https: Pattps///doi.. http///oo..o///////// 1tt.://0rr/r00111///////000000 naacl-main. 105
Xiang Deng, Prashant Shiralkar, Colin Lockard, Binxuan Huang, and Huan Sun. DOM-LM: Iearn-
ing generalizable representations for HTML documents. CoRR, abs/2201. 2022b. URL https:
//axxiv. org/ httg/ab//2/0 2hbb://201..000.
Yinlin Deng, Chenyuan Yang, Anjiang Wei, and Lingming Zhang. Fuzzing deep-learning libraries via au- tomated relational API inference. In Abhik Roychoudhuy,, Cristian Cadar, and Miryung Kim (eds.), Proceedings of the 30th ACM Joint Furopean Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, pp. 44 56. ACM, 2022c. doi: 2..1145/3540250.354999 URL https://doi org/10. 1 1145 /145//540025 3549085
Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In René Just and Gordon Frascr (eds.), Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, Seatile, WA, USA, July 17-21, 2023, pp. 423 233-435.4 ACM, 2023. doi: 011114///55972663555588 URL https:///oii http::://doo.or/1000// 10. 10t.o.111////999..11 3598066
Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy 1/O. In Doina Precup and Yee Whye Teh (ods.); Proceedings of the 94th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australio, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 990-999. PMLR, mlr. pttss///o/dewwwi///// html
2017a. URL http:/pppeccanaaa...pprrrrerrrrp
47

---

Published in Transactions on Machine Learning Research (09/2024)
Jacob Devlin, Jonathan Uesato, Rishabh Singh, and Pushmeet Kohli. Semantic code repair using neuro- symbolic transformation networks. CoRR, abs/1710.11054, 2017b. URL http:///aaxxiv..o/abbbbao//
11054
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
pre-training of deep bidi- rectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.); Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June Association for Computational Linguistics,
2-7, 2019, Volume 1 (Long and Short Papers), pp. httpp////oaooorrg/....///////// 18653/v1/n19-146 2019. doi: 10.18653/v1/nll/ 1423. URL attpp:///ooi org/10.
Peng Di, Jianguo Li, Hang Yu, Wei Jiang, Wenting Cai, Yang Cao, Chaoyu Chen, Dajun Chen, Hongwei Chen, Liang Chen, Gang Fan, Jie Gong, Zi Gong, Wen Hu, Tingting Guo, Zhichao Lei, Ting Li, Zheng Li, Ming Liang, Cong Liao, Bingchang Liu, Jiachen Liu, Zhiwei Liu, Shaojun Lu, Min Shen, Guangpei Wang, Huan Wang, Zhi Wang, Zhaogui Xu, Jiawei Yang, Qing Ye, Gehao Zhang, Yu Zhang, Zelin Zhao, Xunjin Zheng, Hailian Zhou, Lifu Zhu, and Xianying Zhu. Codefuse- 133:: A pretrained multi-lingual code large language model. CoRR, abs/2310. 2210.06266, 2023. doi: 10.48550/ 448550/AARIIV ARXIV 2310.06226.
URL https:
7/doi org/10.000 111.1..55//arait 2310 06266
Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. TOGA: A neural method for test oracle generation. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsbuuggh, PA, USA, May 25-27, 2022, pp. 2130-2141 ACM, 2022. doi: 10.1145/85100033550111.. URL https:://dii.rrggeeee htt.://...rrr.....1////11 1115/31110011 0511///555000033111111 3510141
Yangruibo Ding, Luca Buratti, Saurabh Pujar, Alessandro Morari, Baishakhi Ray, and Saikat Chakraborty. Towards learning (dis)-simillritty of source code from program contrasts. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 6300-6312 Association for Computational Linguistics. 2022a. doi: 10.18633/v1/200 ww2/0/2..cllooooo....a.. URL https://doi.org 1trg/10..0 10. 18653/v1/2022. http:///222222..1111..m acl-long. 436
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Cocomic: Code completion by jointly modeling in- file and cross-file context. CoRR, abs/2222.10000 2022b. doi: (1.5550/ARXV..22 2000. URL
https://doi.orge 0tt.o///////...55///aat.....
10007
Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. CoRR, abs/2310.112 2023. doi: 10.48550/ ARXIV DEXIVV2310.111 13100.1148.. URL
485404455533.. arxiv ww..xi..310... 11248
Yangruibo Ding, Marcus J. Min, Gail E. Kaiser, and Baishakhi Ray. CYCLE: learning to self-refine the code generation. CoRR, abs/2403. 18746, 2024. doi: 11.45500/ARXI..240..0 10. 48550/8rXiv 2403 18746
URL hhttps://dof: org/
Brendan Dolan-Gavitt, Patrick Hulin, Engin Kirda, Tim Leek, Andrea Mambretti, William K. Robertson, Frederick Ulrich, and Ryan Whelan. LAVA: large-scale automated vulnerability addition. In IEEE Sym- posium on Securily and Privacy, SP 2016, San Jose, CA, USA, May 22-20 26, 2016, pp. 110-122 121. TREE Computer Society, 2016. doi: 10. SS..1........ URL https:///oii org/ 10. 1109/SP 2016.15
Jinhao Dong, Yiling Lou, Qihao Zhu, Zeyu Sun, Zhilin Li, Wenjie Zhang, and Dan Hao. FIRA: fine- grained graph- code change representation for automated commit message generation. In 44th ACEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 970-981. ACM, 2022. doi: 00.115//31100883550006 URL http::://doi.or//11 1145/3510003.1 3510069
48

---

Published in Transactions on Machine Learning Research (09/2024)
Li Dong and Mirella Lapata. Coarse-to-fine decoding for neural semantic parsing. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeling of the Associalion for Compula- tional Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 731-742. Association for Computational Linguistics, 2018. doi: 10.18688/v1/P11-1000 URL https: Taadlatttooogy.or///111---..
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianseng Gao, Ming Zhou, and Alsiao-Wuen Hon. Unified language model pre-training for natural language understanding and gen- eration. In Hanna M. Wallach, Hugo Larochelle, Alina. Beygekzier, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Con- ference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancou- ver, BC, Camada, pp. 13042 13054, 2019. URL Absasaaaa wweaaabbsstrttttaeaeeeaaa html
Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. CoRR, has//30040775990 2023. 2304.00559
doi: (188550/ARXV.220000
URT. https://doi. http://o....r/111144446///////
Shihan Dou, Junjie Shan, Haoxiang Jia, Wenhao Deng, Zhiheng Xi, Wei He, Yueming Wu, Tao Gui, Yang Liu, and Xuanjing Huang. Towards understanding the capability of large language models on code clone detection: A survey, CoRR, abs/ /2308.01191, 2023. doi: n0.48550/arXiv. 2300.1110... URL httpp:://oi. 2rg/10.44 org/10.48550//an arXiv. 2308.01191
Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Xuanjing Huang, and Tao Gui. Stepcoder: Improve code generation with reinforcement learning from compiler feedback. CoRR, abs/2402.011
2024.
doi: (0.48550/ARXIV.22444 013.0. URL htttp::///oi. http:////00..//11..4443.43 org/10.48550/
arXiv 210..24402.019911
Dawn Drain, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. Deepdebug: Fixing python bugs using stack traces, backtranslation, and code skeletons. CoRR, abs/2105.09352, 2021. URL https:
Mchdi Drissi, Olivia Watkins, Aditya Khant, Vivaswat Ojha, Pedro Sandoval Segura, Rakia Segev, Eric Weiner, and Robert Keller. Program language translation using a gramnar-driven tree-to-tree model. CoRR, abs/1807.01784 2018. URL http://arxiv org/ abs//abb./////11//1////////
Iddo Drori and Nakul Verma. Solving linear algebra by program synthesis. CoRR, abs/ a21/2111.08111, 2021.
URL https://aaxit: //tpvvv..ivvv//////// wwrgg/ab//111e abss2111..8811
Iddo Drori, Sarah Zhang, Recce Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Läu, Linda Chen, Sunny Tran, Newman Cheng, Roman Wang, Nikhil Singh, Taylor L. Patti, Jayson Lynch, Avi Shporer, Nakul Verma, Eugene Wu, and Gilbert Strang. A neural network solves, explains, and generates university math problems by program synthesis and sew-shot learning at human level. Proceedings of the National Academy of Sciences (PNAS), 119(32), 2022.
Lun Du Xiaozhou Shi, Yanlin Wang, Ensheng Shi, Shi Han, and Dongmei Zhang. Is a single model enough? mucos: A multi-model ensemble learning approach for semantic code search. In Gianluca De- martini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (eds.), CIKM 221: The 30th ACM International Conference Information and Knowledge Management, Virtual Event, Queens- land, Australia, November 1 5, 2021, pp. 2994 2998. ACM, 2021. doi:
URL
orgooo/////1 10. /111//1////333771 34447//245563333882227 3482127
Min Du and Feifei Li. Spell: Streaming parsing of system event logs. In Francesco Bonchi, Josep Domingo- Ferrer, Ricardo Bacza-Yates, Zhi-Hua Zhou, and Xindong Wu (eds.). JEEE 16th International Conference on Data Mining, ICDM 2016, December 12-15, 2016, Barcelona, Spain, pp. 859 864. IEEE Computer 11t.:/00.1119111/10001 0103
Society, 2016. doi: 10. 10.1109/TCDM.22166000
URL
49

---

Published in Transactions on Machine Learning Research (09/2024)
Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Bhavani Thuraisingham, David Evans, Tal Malkin, and Dongyan Xu (eds.), Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS 2017, Dallas, TX, USA, October 30- November 03, 2017, pp. 1285-1298. ACM, 2017. doi: 10.1145/
3133956.3134015. URL https::/ddi: org/ 10. 1145/3133956 3134015
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang. Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoe V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 5547 5569. PMLR, 2022. URL https::/pprccedd wwwpeeeedinnnmm.........m.m.. mlr. wwe.s.press//ee.. www.pprrsssse... v162/du22c. html
Shubhang Shekhar Dvivedi, Vyshnav Vijay, Sai Leela Rahul Pujari, Shoumik Lodh, and Dhruv Kumar. A comparative analysis of large language models for code documentation generation. CoRR, abs/ 2312.10349, 2023. doi: 10.48550/ ARXIV ERRXIV.312..11.... 1311.110349 URL ltttps://do/ org/ 1155010..4555///./. 2312 10349
Meryem Flallaoui, Khalid Nafil, and Raja Touahni. Automatic transformation of user stories into UML, use case diagrams using NLP techniques. In Elhadi M. Shakshuki and Ansar-Ul-Haque Yasar (eds.); The 9th International Conference on Ambient Systems, Networks and Technologies (ANT 2018) / The 8th Indernational Conference on Sustainable Energy Information Technology (SEIT 2018) / Affiliated
Workshops, May 8-11, 2018, Porto, Portugal, volume 130 of Procedia Computer Science, pp. 42 49. Elsevier, 2018. doi: /T01://.PROCS..188...... 04.010
URL https://doi. http::://oo..orr///0000///// proci.
Aleksandra Eliseeva, Yaroslav Sokolov, Egor Bogomolov, Yaroslav Golubev, Danny Dig, and Timoley Bryksin. From commit message generation to history-aware commit message completion. In 38th JEEE/ACM International Conference on Automated Software Engincering, ASE 2023, Luzembowy, September 11-15, 2023, pp. 723 735. IEEE, 2023. doi: 10.1109/ASE56229. 2023.00078. URL https: 7/doi. orgooog//00 10. 0119//ASE5222
0007880088
Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. A C/C++ code vulncrability dataset with code changes and CVE summaries. In Sunghun Kim, Georgios Gousios, Sarah Nadi, and Joseph Hejderup (eds.), MSR '20 17th International Conference on Möning Software Repositories, Scoul, Republic of Korea, 29-30 June, 2020, pp. 512- ACM, 2020. doi: 011145/3377597.3333 8379597.3388 3387501
URL https:://ol. http::////oi.oo///...o
Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. Automated repair of programs from large language models. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 1469 1469-1481. IEFE, 2023. doi: 10.
1109/ICSE48619. 201.://92E88619.222.....
URL https:/ //oii. org/ 10. 1111119IIIIEE900 2w0.3....22.... 2023.
Chunrong Fang, Zixi Liu, Yangyang Shi, Jeff Huang, and Qingkai Shi. Functional code clone detection with syntax and semantics fusion learning. In Sarfraz Khurshid and Corina S. Pasareanu (eds.), 1SSTA '20: 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020, pp. 1145/3395363.1 3397362
ACM, 2020. doi: 10.1145/3395363.8880
URL https://doi. org/10.
Sen Fang, Youshuai Tan, Tao Zhang, and Yepang Liu. Self-attention networks for code search. Inf. Softw. Technol., 134:106542, 2021. doi: 1001://IIFSO...22..00000 URL infsof 2021. 106542
William Fodus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1-120:39 2022. URL http:///mir.org org/ pappers/v23211-0988 html
50

---

Published in Transactions on Machine Learning Research (09/2024)
Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: theoretical perspective. CoRR, abs/2305. 15408, 2023a. doi: 10.48550/ ARXXIV2330..15 URL http::///aii.rg/1.......5. wwt.://///.....20000 2305. 15408
K. J. Kevin Feng, Q. Vera Liao, Ziang Xiao, Jennifer Wortman Vaughan, Amy X. Zhang, and David W. McDonald. Canvil: Designerly adaptation for llm-powered user experiences. CoRR, abs/2400 00//22401.00005 2024. doi: 1.48550AARIVV 2401.09051. URL https://áoi. 1tt.:/..0000......8///////////
(eExxxi..41.... 2401.09051
Weixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun R. Akula, Xuchai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpl: Compositional visual planning and genera- tion with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Newral Information Processing Systems 36: An- nual Conference on Newral Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b. URL, httt:://pppp nips pttps.ccppppprrr files/ patpee///ppr/22////aaa/ 84758668855007225777000000000010
html
Yu Feng, Ruben Martins, Osbert Bastani, and Isil Dillig. Program synthesis using conflict-driven learning. In Jeffrey S. Foster and Dan Grossman (eds.), Proceedings of the 39th ACM SIGPLAN Conference on Pro- gramming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018, pp. 420-435. ACM, 2018. doi: 01144/3119336631922322 URI. https://doi hrg///o...r//// org/ 114..11911//331...
3192882
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 1536- 1547. Association for Computational Linguisties, 2020. doi: 10.18653/vv h/1l/v/vv//
URL
1ttg///0...00///////0000.11 findings-emmlpp 139
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. Structured neural summarization. In 7th International Conference on Learning Represenlaliins, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.nee 2019. URL
Alessio Ferrari, Felice Orlettt,tta, Giorgio Oronzo Spagnolo, and Stefania Gnesi. Measuring and improving the completeness of natural language requirements. In Camille Salinesi and Inge van de Weerd (eds.), Requirements Engineering: Foundation for Software Quality 20th International Working Conference, REFSQ 2014, Essen, Germany,, April 7-10, 2014. Proceedings, volume 8396 of Lecture Noles in Compuler 0110//9978--319-5888
Science, pp. 23 38. Springer, 2014. doi: 1007/978-3-311 1---119-584-6-3
3. URL https:///ot org/10. 10.
Alessio Ferrari, Sallam Abualhaija, and Chetan Arora. Model generation from requirements with llms: an exploratory study. CoRR, abs/2400.06371,, 2024. doi:
URL https:
2404.404400 06371
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir R. Radev. Improving text-to-sql evaluation methoollggy Tn Tryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 351 360. Association for Computational Linguistics, 2018. doi: 10.1863//1/P16-1033. URL https: //aclanthologyy.om ort.:/oooo.or//-1113333333333333
Georgia Frantzeskou, Stephen G. MacDonell, Efstathios Stamatatos, Stelios Georgiou, and Stefanos Gritzalis. The significance of user-defined identifiers in java source code authorship identification. Comput. Syst,
Sci. Eng., 26(2), 2011.
Gordon Frascr and Andrea Arcuri. Evosuite: automatic test suite generation for object-oriented software. In Tibor Gyimóthy and Andreas Zeller (eds.), SIGSOFT/FSE'11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSF-19) and ESEC'11: 13th European Software Engineering
51

---

Published in Transactions on Machine Learning Research (09/2024)
Conference (ESEC-13), Szeged, Hungary, September 5-9, 2011, pp. 416-419. ACM, 2011. doi: 10.1145/ 2025113.2025179 URL
11.....1111222221133 202220211111120227999 2025179
Gordon Fraser and Andrea Arcuri. A large-scale evaluation of automated unit test generation using evosuite. ACM Trans. Softw. Eng. Methodol., 24(2):8:1 8:42, 2014. doi: (0.1145/26850) URL https://doi...... org/
10. 1145/2686666
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Represenlations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.ner net, 2023. URL https:/ hwtp://iwww.eee//dd/ddd--o----/o 1bMMEL
Lingyue Fu, Huacan Chai, Shuang Luo, Kounianhua Du, Weiming Zhang, Longteng Fan, Jiayi Lei, Renting Rui, Jianghao Lin, Yuchen Fang, Yifan Liu, Jingkuan Wang, Siyuan Qi, Kangning Zhang, Weinan Zhang, and Yong Yu. Codcapex: A bilingual programming evaluation benchmark for large language models. CoRR, /ass//.99.09440 2023. doi: 10.48550/aaa /rvi 20...20099..99 2309. 01940]
URL/tttps: :/dos. org/10.48550//r/// 48550/arXiv.
Michael Fu and Chakkrit Tantithamthavorn, Linevul: A transformer-basce line-level vulncrability prediction. In 19th IEEE/ACM International Conference on Mining Software Reposilories, MSR 2022, Pillsburgh, PA, USA, May 23-24. 2022, pp. 608 620. ACM, 2022.
doi: 10.1145/3624442235228848 URL https:
7/doi org/ 10. 1145/114555 35244/222444222222 3528452.
Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Van Nguyen, and Dinh Q. Phung. Vulrepair: t5-based automated software vulnerability repair. In Abhik Roychoudhury, Cristian Cadar, and Miryung Kin (eds.), Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC//SS 2022, Singapore, Singapore, November 14-18, 2022. ftng/10..10 11455//5544//
pp. 935 947. ACM, 2022. doi: 10.1145 10.1145/3540230.3549995 3549098
URL https: org/
Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. Scalable inference and training of context-rich syntactic translation models. In Nicoletta Calzolari, Claire Cardie, and Pierre Isabelle (eds.), ACL 2006, 21st International Conference on Computational Linguistics and 44th Annual Mecting of the Association for Computational Länguistics, Proceedings of the Conference, Sydney, Australia, 17-21 July 2006. The Association for Computer Linguisties, 2006. doi: 11111//22211..12222222 URL https: httpps:/acclantholooe www.at/oowooooooooooooooo 1121/
Yujian Gan, Xinyun Chen, Qiuping Huang, Matthew Purver, John R. Woodward, Jinxia Xie, and Peng- sheng Huang. Towards robustness of text-to-sql models against synonym substitution. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI//JCNT.P 2021, (Volume 1: Long Papers), Virtual Evend, August 1-6, 2021, pp. 2505- 2515. Association for Computational Linguistics, 2021a. doi: (1865//V1/021.1CCLA/2 https://ddi.org/11 10. 000.0/00000.....1111//////00.01. 200.:/// 195
URL
Yujian Gan, Xinyun Chen, and Matthow Purver. Exploring underexplored limitations of cross-domain text-to-sql generalization. In Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Longuage Processing, FMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 0931- Association for Computational Linguisties, 2021b. doi: https://doi.org: cefooi.orrg/10 10g/10.:10 11653/ http:///ww1222....mppmaaaaaaa 702
URL
Yujian Gan, Xinyun Chen, Qiuping Huang, and Matthew Purver. Mcasuring and improving compositional generalization in text-to-sql via component alignment. In Marine Carpual, Marie-Catherine de Marnelle, and Iván Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 831- 141-833. Association for Computational Linguis- tics, 2022. doi: STA.:///V//22..IIIIISSSSSS findings-naacl.
URL https:///oi. http:///oiooor/10.10000 11t//1...663/11/22020
52

---

Published in Transactions on Machine Learning Research (09/2024)
Chang Gao, Bowen Li, Wenxuan Zhang, Wai Lam, Binhua Li, Fei Huang, Luo Si, and Yongbin Li. To- wards generalizable and robust text-to-sql parsing. In Yoav Goldberg, Zornitsa Kozareva, and Yue
Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, Pp. 2113-21222 2125. Association for Computational Linguis- tics, 2022. doi: (0.18653/V1/2092 /122///I/// F.N.IINNNNN----............ URL findings-emnlp.15
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to- sql empowered by large language models: A benchmark evaluation. CoRR, abs/2308. 15363, 2023a, doi: 10.48550/ BAB.0///RIV.... 2308. 15363. URL
48550/arx1v 2308 15363.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101. .00027, 2021. URL htttss://arx:t http:///axxxvv...aaaa/ org/abs//// /
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: program-aided language models. In Andreas Krause, Emıa Brunskill, Kyunghyun Cho, Bar- bara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learn- ing, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10764 10799. PMLR, 2023b. URL
amliiinnn..........m......m... mlr mal..preee press/v202/ https///22//gga233/ gao23f html
Shuzheng Gao, Cuiyun Gao, Yulan He, Jichuan Zeng, Lunyiu Nic, Xin Xia, and Michael R. Lyu. Code structure-guided transformer for source code summarization. ACM Trans. Softw. Eng. Methodol., 32(1): 23:1-23:32, 2023c. doi: 10.1145/3522674. URL https::/dol..rrg/ 1tt.:/1..11111// 1145/3522674
Yuexiu Gao and Chen Lyu. M2TS: multi-scale multi-modal approach based on transformer for source code summarization. In Ayushi Rastogi, Rosalia Tufano, Gabriele Bavota, Venera Arnaoudova, and Sonia Haiduc (eds.), Proceedings of the 30th TEEE/CM International Conference on Program Comprehension, ICPC 2022, Virtual Event, May 16-17, 2022, pp. 24 35. ACM, 2022. doi: 10.1455/55266103337799 URL https://doi. org/ 10. 1t///00..111/3355 1111/114////11111 3527907
Zeyu Gao, Hao Wang, Yuchen Zhou, Wenyu Zhu, and Chao Zhang. How far have we gone in vulnerability detection using large language models. CoRR,
2023d. doi: 10. 23T...2331..244444
URL https://dot: ptttp://ddo..org/00 48550/
Luca Gazzola, Daniela Micucci, and Leonardo Mariani. Automatic software repair: a survey. In Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (ods.), Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, pp. 1219. 3182526.
ACM, 2018.
doi: 10. 0..114531180555315555
URL https::///t...... wttp://..0//111111111111113
Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning. In Proceedings of the 46th TEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pp. 39:1-39:13 ACM, 2024. doi: 10.1155/5597503336888444 URL https://doi //do1. org/10. 1145/3597503 3555//33557503333814444
Shalini Ghosh, Daniel Elenius, Wenchao Li, Patrick Lincoln, Natarajan Shankar, and Wilfried Steiner. AR- SENAL: automatic requirements specification extraction from natural language. In Sanjai Rayadurgam and Oksana Tkachuk (eds.), NASA Formal Methods 8th International Symposium, NFM 2016, Min- neapolis, MN, USA, June 7-9, 2016, Proceedings, volume 9690 of Lecture Notes in Computer Science, pp. 41 46. Springer, 2016. doi:
4. URL lttts:://doo wooei.orr/11110...70
978-3 978-3-319-4068--0
Linyuan Gong, Mostafa Elhoushi, and Alvin Cheung. AST-T5: structure-aware pretraining for code gen- eration and understanding. CoRR, abs/2401.032 .03003, 2024. doi: 00.45550/ARXT ARXIV.2401.03000 2201..00003. URL
https://doi. 4055///0...050//....v.... 2401
53

---

Published in Transactions on Machine Learning Research (09/2024)
M. Gopinath and Sibi Chakkaravarthy Sethuraman. A comprehensive survey on deep learning based malware detection techniques. Compul. Sei. Rev., 47:100529, 2023. doi: 10.101/...C3SRRN22222200000 100229 URL
/tt.///ii..rg//.111///1111 cttp://...oree.22....... 2022. 100529
Claire Le Goues, Neal J. Holtschulte, Edward K. Smith, Yuriy Brun, Premkumar T. Devanbu, Stephanie Forrest, and Westley Weimer. The manybugs and introclass benchmarks for automated repair of C programs. IEEE Thans. Software Eng., (1(1)):123611283 2015. doi: 00.109/TSE.2015.2555511 URL https:///oi..on 10. 1109/TSE. 2015. 2454513
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- Ranzato, Francisco Guzmán, and Angela Fan. The flores-101 evaluation benchmark for low-resource and multilinguaa machine Iranslation. Trans. Assoc. Compud. Linguistics, 10:522 538, URL https://doi. org/10 1162/ttcc_a_000000
ishnan,
2022. doi: (0.1162/TACL) A 00474.
Luca Di Grazia and Michael Pradel. Code search: A survey of techniques for finding code. ACM Comput. Surv., (5()):22 220:31, 2023. doi: 10.1145/3563971 URL https://doi.org/ 1tt.:/i..rrr/11111/////1111 1145/3565971
Dejan Grubisic Chris Cummins, Volker Seeker, and Hugh Leather. Compiler generated feedback for large language models. CoRR, abs/2403.14414 14714, 2024a. doi: org/10. 01..4855/aaiii 2403. 14714
2ttp://AAXXI.220..17777 URL https://doi.
Dejan Grubisic, Volker Seeker, Gabriel Synnaeve, Hugh Leather, John M. Mellor- Crummey, and Chris Cummins. Priority sampling of large language models for compilers. In Proceedings of the 4th Workshop on Machine Learning and Systems, Euro 2024, Athens, Greece, 22 April 2024, pp. 91 97. ACM, 2024b. doi: 10.1145 //33//227773635868 URL https://doi. org/10. 1145/3642970 3655831
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. CoRR, abs/2240.033065 2024. doi: 10.48550/ 148550/ARXTVV ARXIV .2401.03065. URL hattps://doi..o //tp://///aa.rr///004444//////// 2401 030665
Jian Gu, Zimin Chen, and Martin Monperrus. Multimodal representation for neural code search. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2021, Luzembourg. September 27 October 1, 2021, pp. 483 494. IEEE, 2021. doi: (0.1100/CCSMM5551022200000000 URL https:
//doi org/ 10. 1109 II11099/ICMM2210.... 2021.00043
Jiazhen Gu, Xuchuan Luo, Yangfan Zhou, and Xin Wang. Muffin: Testing deep learning libraries via neural architecture fuzzing. In 44th IEEE/ACE AAE 44th International Conference on Software Engineering, 1CSE 2022, Pitttsburgh, PA, USA, May 25-27, 2022, pp. 1418-1430. ACM, 2022. doi:
URL https::/doii org/ 10. 1145/3510003 3510092
Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. Deepam: Migrate apis with multi-modal sequence to sequence learning. In Carles Sierra (od.), Proceedings of the Twenty-Sirkh International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp. 3675 3681. ijeai.org, 2017. doi: 10.24963/ 11011//11/111211 /544. URL httpss:::///oo org/10. 24963 /ijasiiijai..//// http://jjaa.21//1111/11/1 2017/514
Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. Deep code scarch. In Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (eds.). Proceedings of the 40th International Conference Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 June 08, 2018, pp. 933-944. ACM, 2018. doi: 011144/318115531001100 URL https:///oi.org: 11111111111155.1110110
Batu Guan, Yao Wan, Zhangqian Bi, Zheng Wang, Hongyu Zhang, Yulei Sui, Pan Zhou, and Lichao Sun. Codeip: A grammar-guided multi-bit watermark for large language models of code. CoRR, abs/ /244.. 15639, 2024. doi: 10.48550/ 1..8855/AARIV... AAw.I/ 2..4 URL fttps::/ddoi.rgee 48550/
15639
Vitor Guilherme and Auri Vincenzi. An initial investigation of chatgpt unit test generation capability. In Awdren L. Fontão, Débora M. B. Paiva, Hudson Borges, Maria Istela Cagnin, Patrícia Gomes Fernandes, Vanessa Borges, Silvana M. Melo, Vinicius H. S. Durelli, and Edna Dias Canedo (eds.), 8th Brazilian Symposium Systematic and Aulomated Software Testing, SAST 2023, Compo Grande, MS, Brazil,
54

---

Published in Transactions on Machine Learning Research (09/2024)
September 25-29, 2023, pp. 15-24. ACM, 2023. doi: 10.1145 0,.1146//622432.36222000 URL https://doi.or/ 10. 12. 1145/3624032..... 3624035
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harki- rat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. CoRR, abss/2306.111 2023. doi: URL https://doi org/10 48550/arXi: 2306.11644
/a1/500//ri..222 2306. 11644.
Tugge Günes and Fatma Basak Aydemir. Automated goal model extraction from user stories using NLP, In Travis D. Breaux, Andrea Zisman, Samuel Fricker, and Martin Glinz (cds.), 28th IEEE International
Requirements Engineering Conference, RE 2020, Zurich, Switzerland, August 31 Seplember 4, 2020, URL httph:t//pp.oiii.. 1ttg//00..109///8521111
pp. 382 387. TEFE, 2020. doi: (41109/RF48511.222 2020. 00052)
Chunxi Guo, Zhiliang Tian, Jintao Tang, Shasha Li, Zhihua Wen, Kaixnan Wang, and Ting Wang. Retrieval- augmnented gpt-3.5-based text-to-sql framework with sample-aware prompting and dynamic revision chain. In Biao Luo, Long Cheng, Sheng-Guang Wu, Hongyi Li, and Chaojie Li (eds.), Neural Information Pro- cessing 30th International Conference, ICONIP 2023, Changsha, China, November 20-23, 2023, Pro- ceedingg, Part VI, volume 14452 of Lecture Notes in Computer Science, pp. 341 356. Springer, 2023a. doi: $_25
1110077978-981-11 -
25. URL https://doo:
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svy- atkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. Graphcodebert: Pre-training code representations with data flow, In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL. htttt::// http:/eeeeetttommmmmmdddoooooeee
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. Unixcoder: Unified cross- modal pre-training for code representation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Compulational Linguisties (Volume Association for Compu-
1: Long Pupers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 499. URL fttps://ddo.orr///1 186533/v1
tational Linguisties, 2022. doi:
2022. ac1-1ong. 499
Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian J. McAuley. Longcoder: long-range pre-trained language model for code completion. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt,, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 12107. PMLR, 2023b. URL
ppwss/v22/ /00oss//222/guo233 23 .tml wttp://iinn......pprrr/////////p
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence. CoRR, //t/:/440111111 2024. doi: 00.48550/ARXII
2401.14196 URL /tttss:/ddo..oom
200..88500//r// arxiv www.:xiii............ 14196.
Haixuan Guo, Shuhan Yuan, and Xintao Wu. Logbert: Log anomaly detection via BERT. In International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021, pp. 1-8. IEFE, 2021b. doi: W0.1109/JCNNN2237..22 .9531113. URI. 9534113
11111110//9111111 2021.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. To- wards complex text-to-sql in cross-domain database with intermediate representation. In Anna Korhonen, David R. Traum, and Lluis Márquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume Long Pa- pers, pp. 4524 4535. Association for Computational Linguistics, 2019. doi: 10. (..18533/v/plll 1444. URL httpps//doi..oe http://////...00//////1111111000
55

---

Published in Transactions on Machine Learning Research (09/2024)
Qianyu Guo, Xiaofei Xie, Yi Li, Xiaoyu Zhang, Yang Liu, Xiaohong Li, and Chao Shen. Audee: Automated testing for deep learning frameworks. In 35th TEEE/ FEEE/ACM ACM International Conference on Aulomated Software Engineering, ASE 2020, Melbourne, Australia, September 21-25, 2020, pp. 486 498. IEEE, 2020. doi:
00.1145/88224884.4111 341677... URL https:///oi. 11111114111133348444111 3416571
Anshul Gupta and Neel Sundaresan. Intelligent code reviews using deep learning, 2018.
Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish K. Shevade. Deepfix: Fixing common C language errors by deep learning. In Satinder Singh and Shaul Markovitch (eds.), Proceedings of the Thirty-
AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp.
1345-1351. AAAI Press, 2017. doi: 10. 10.1609/aaaii http//aai..v311.1000.... URL Atttps:/dooi wtt. /600/////00000111/////// 1000..609/aaaa aaai. v0741
Izzeddin Gur, Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sha- ran Narang, Noah Fiedel, and Aleksandra Faust. Understanding HTML with large language mod- cls. In Houda Bouamor, Juan Pino, and Kalika. Bali (ods.). Findings of the Association for Com- pullational Linguislics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 2803 2821. Association for Computational Linguistics, 2023. doi: 10. 10.18653/V "1/2023.FINDLNGG //doi.org//l 10 18653/v1/2023 http://222.....nnnn..eeg findings- emnlp. 185
URL https:
Emitza Guzman, Mohamed Ibrahim, and Martin Glinz. A little bird told me: Mining tweets for requirements and software evolution. In Ana Moreira, João Araújo, Jane Hayes, and Barbara Pacch (cds.), 25th TEEE International Requirements Engineering Conference, RE 2017, Lisbon, Portugal, September 4-8, 2017, pp. 11-20. IEEE Computer Society, 2017. doi: 10.1109 10.1109/RE.2017 R0...2....... URL htttp:://don org/o0///// 10. 10./ 2017.88
Péter Gyimesi, Béla Vanesics, Andrea Stocco, Davood Mazinanian, Árpád Beszédes, Rudolf Ferenc, and Ali Mesbah. Bugsjs: benchmark of javascript bugs. In 12th IEFE Conference Software Testing, Validation and Verification, ICST 2019, Xi'an, China, April 22-27, 2019, pp. 90-101. IEEE, 2019. doi: 10.1109/ICST.200 (0T.C.T.....001999 URL /tttps:///oo.orgg//..o. 1109/ICCT 2019.00019
Rajarshi Haldar and Julia Hockenmaier. Analyzing the performance of large language models code summarization. CoRR, abs/2444.00000 2024. doi: 10.48550/ 00.550/AXXV..44444
URL https://doi.
org/10. 1//50/04x555/rrxiv 2404. 08018
Patrick Haller, Jonas Golde, and Alan Akbik. PECC: problem extraction and coding challenges. In Nicoletta Calzolari, Min-Yen Kan, Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.); Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Raly, pp. 12690 12699. ELRA and ICCL, 2024. CRL https://aclantthllgggg wwg/////////// 2024. Aireemmain..
Sivana Hamer, Marcelo d'Amorim, and Laurie Williams. Just another copy and paste? comparing the security vulnerabilities of chatgpt generated code and stackoverflow answers. CoRR, /te//2443.156000 2024. doi: 10.48550/ ARXTV 15600. URL https:///ol: htt///ti..oorr////// org/10. 48550/arX11 2403. 15600
Quinn Hanam, Fernando Santos De Matios Brito, and Ali Mesbah. Discovering bug patterns in javascript. In Thomas Zimmermann, Jane Cleland-Huang and Zhendong Su (eds.), Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, Seattle, WA, USA. November 13-18, 2016, pp. 144-156. ACM, 2016. doi: 011144//29502902255033 URL https: 7/doi. org. 10. 00111/1152222000 290029000200202 2950308
Hazim Hanif and Sergio Maffeis. Vulberta: Simplified source code pre-training for vulnerability detection. In International Joint Conference on Neural Networks, IJCNN 2022, Padua, Italy, July 18-23, 2022, pp. 1- 8. IEEE, 2022. doi: LLLLO//JCCNN550442222222222 98992880. URL https: //ooi/ http:::///ooooo/////// 110.11111///CI154...... 2022. 9892280
Yiyang Hao, Ge Li, Yongqiang Liu, Xiaowei Miao, He Zong, Siyuan Jiang, Yang Liu, and He Wei. Aixbench: A code generation benchmark dataset. CoRR, htss/2006.131777 2022. doi: 10.48550//r2ixv22266 www..xvv..22..111111 131999
URL htttps://do..rg www..//////...///a.a.ee/// arr.v.///vvv.... 2tt.XXiv.22666.22
56

---

Published in Transactions on Machine Learning Research (09/2024)
Sakib Haque, Alexander LeClair, Lingfei Wu, and Collin McMillan. Improved automatic summarization of subroutines via attention to file context. In Sunghun Kim, Georgios Gousios, Sarah Nadi, and Joseph Hejderup (eds.), MSR '20: 17th International Conference on Mining Software Repositories, Seoul, Republic of Korea, 29-30 June, 2020, pp. 300-310. ACM, 2020. doi: 10.1145/ 10.1145/3379591 018877977338774777
URL https:
//doi.org// 10. 01145/33759973332 3387449
Nima Shiri Harzevili, Alvine Boaye Belle, Junjie Wang, Song Wang, Zhen Ming Jiang, and Nachiappan Nagappan. A survey on automated software vulnerability delection using machine learning and deep learning. CoRR, abs/2230.11167, 2023. doi: 10.48550/ARXIV 2306..1663. URL https://doi https://doi.orr/100.
48550/arK arxiv. 2306. 11673
Mostafa Hassan, Caterina Urban, Marco Eilers, and Peter Müller. Maxsmt-based type inference for python 3. In Hana Chockler and Georg Weissenbacher (cds.), Computer Aided Verification 30th International Conference, CAV 2018, Held as Parl of the Pederaled Logic Conference, FloC' 2018, Oxford, UK, July 14- 17, 2018, Proceedings, Part II, volume 10982 of Lecture Notes in Computer Science, pp. 12-19. Springer, 2018. doi: (10007/977-33319-6611222
2. URL https://aoi.org//
Shirley Anugrah Hayati, Raphaël Olivier, Pravalika Avvaru, Pengcheng Yin, Anthony Tomasic, and Graham Neubig. Retrieval-basod neural code generation. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.): Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, Oclober 31- November 4, 2018, pp. 925 930. Association for Computational Linguistics, 2018. doi: N..18633/VI/DLL 1111, URL https: http://:://ii...//./..//// wttp://///00.1//////////////111
Moshe Hazoom, Vibhor Malik, and Ben Bogin. Text-to-sql in the wild: A naturally dataset based on stack exchange data. CoRR, abs/2106 2210600500,, 2021. URL hattps//aaraa...og//b///00.....
Jingxuan He, Pesho Ivanov, Petar Tsankov, Veselin Raychev, and Martin T. Vechev. Debin: Predicting debug information in stripped binaries. In David Lie, Mohammad Mannan, Michael Backes, and XiaoFeng Wang (cds.), Proccedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS 2018, Toronlo, ON, Canada, Oclober 15-19, 2018, pp. 1667 1680. ACM, 2018. doi: 10. 10.1145/3243734. 3243866. URL https://doi. http::///ooooog//..../ 1145/3243734 324333
Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R. Lyu. Drain: An online log parsing approach with fixed depth tree. In 1lkay Altintas and Shiping Chen (eds.), 2017 IEEE International Conference on Web Services, ICWS 2017, Honolulu, HI, USA June 25-30, 2017, pp. 33-40. IEEE, 2017. doi: 10.1109/ICWS.
2017.13. URL https: //doi org/10. 1109/111S 2017.13
Shilin He, Jieming Zhu, Pinjia He, and Michael R. Lyu. Loghub: A large collection of system log datasets towards automated log analytics. CoRR, abs/2008.0644 2020. URL https: httppp:://xxxv../////// http:///arxii..o//a///////// org/abs/200e. 06448
Xinyi Hc, Jiaru Zou, Yun Lin, Mengyu Zhou, Shi Han, Zejian Yuan, and Dongmei Zhang. CONLINE: com- plex code generation and refinement with online searching and correctness lesting. CoRR, abs/2403 /403. 13583, 2024. doi: 10.48550/ AAR.0//RRR...... 21..2244333555 URL huttps://dot org/10 10/g./1.../ 48550/ 235500/rX1iv arxiv 2403. 13583
Yichen He, Liran Wang, Kaiyi Wang, Yupeng Zhang, Hang Zhang, and Zhoujun Li. COME: commit message generation with modification embedding. In René Just and Gordon Fraser (eds.), Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis ISSTA 2023, Seattle, WA, USA, July 17-21, 2023, pp. 792-803. ACM, 2023a. doi: 10.1145 /t////359992...99000 URL 3598096.
https: //oii org/ 10.
Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. Exploring human-like translation strategy with large language models. CoRR, Abs/2335.00413 2023b. doi: 10.48550/ ARXIV.2330.0411.. .2005.04118. URL https:://oii http://ttooo/....//////////////t 48550.0/..5000......... 2305.04113
Joy Hc. Kucya, Gabriel Poesia, Rose E. Wang, and Noah D. Goodman. Solving math word problems by combining language models with symbolic solvers. CoRR, 2304.09102. URL htll:https::/// 488500..00///////////..// arXiv 2304 09102
2023. doi: 10.48550/ ARXXV.
57

---

Published in Transactions on Machine Learning Research (09/2024)
Vincent J. Hellendoorn and Premkumar T. Devanbu. Are deep neural networks the best choice for modeling source code? In Eric Bodden, Wilhelm Schäfer, Arie van Deursen, and Andrea Zisman (eds.), Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017, pp. 763-773. ACM, 2017. doi: 10.1145/310622 0115//1100037.1102900. URL https: //doi.org/10. wtt.ll11///1117....00 /111//331662777112290 3106290
Vincent J. Hellendoorn, Christian Bird, Earl T. Barr, and Miltiadis Allamanis. Deep learning lype inference. In Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareanu (eds.), Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL. USA, November 04-09, 2018, pp. 152 162. ACM, 2018. doi: 0..144/3223002 3236055. URL. lattps:///oli .opooi.oooo org/10. wtw.g//...11//32211 11w.111311/334444111 3236051
Vincent J. Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevicw.net, 2020. URL httts::/ http::/pppeeeevieeennertttrorooo
B1 B11nbRNtwr]
Vincent J. Hellendoorn, Jason Tsay, Manisha Mukherjee, and Martin Hirzel. Towards automating code review at scale. In Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (ods.), ESEC/FSE '21 29th ACM Joint European Software Engincering Conference and Symposium on the Poundations of Software Engineering, Alhens, Greece, August 23-28, 2021, pp. 1479 1482. ACM, 2021. org/ 10. 1145//4661/446444111 347333444444333133 3473134
doi: 10.1145/3 3466264 URL https: /ttss://dooi htt.//iio/////////////eeeeooeeee
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, USA, June 24-27, 1990. Morgan Kaufmann, 1990. URL https://acllatthologgg hww.e/hooooggoooogg/oog oTg/H90-102//
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace Hc, Dawn Song, and Jacob Steinhardt. Mcasuring coding challenge competence with APPS. In Joaquin Vanschoren and Sai-Kit. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, De- cember 2021, virtual, 2021a. URL httpp:/s//aatssstt----oocaa-a-ap proceedings wwwpodiiinnnnnn.inniineeee meurips.cc/pap wtt.cc/cppppp... paper/
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.ned nct, 2021b. URL http:///ppppooooiieeee/rrrraaaaa
Hatem Herchi and Wahiba Ben Abdessalem. From user requirements to UML class diagram. CoRR, abs/1211.0713, 2012. URL
Geert Heyman and Tom Van Cutsem. Neural code search revisited: Enhancing code snippet retrieval through natural language intent. CoRR, als/2008.12193, 2020. URT. https://arxiv httpi/o.ogg/bba////a/eae 12193
Abram Hindle, Earl T. Barr, Zhendong Su, Mark Gabel, and Premkumar T. Devanbu. On the naturalness of software. In Martin Glinz, Gail C. Murphy, and Mauro Pezzè (eds.), 34th International Conference on Software Engineering, ICSE 2012, June 2-9, 2012, Zurich, Switzerland, pp. 837-847 IEEE Computer /ttp:/CCEE.2.....00.
Sociely, 2012. doi:
URL
22126227333
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NewrIPS 2020, December 6-1 2020, virtual, 2020. URL http:::/pproceeiigg.eeee.. www.ips.cc/pappprr......e h000:hhaan/ceecccccsssssoooo1110 htt.aa/000.0..00000000ooa---aaa.
Thong Hoang, Hong Jin Kang, David Lo, and Julia Lawall. Cc2vec: distributed representations of code changes. In Gregg Rothermel and Doo-Hwan Bae (eds.), ICSE '20: 42nd International Conference on
58

---

Published in Transactions on Machine Learning Research (09/2024)
Software Engineering, Seoul, South Korea, 27 June 19 July, 2020, pp. 518-529. ACM, 2020. doi: 10.
htttt:::////:.orrrre 1144/337781111111 138883333788113333831
Sepp Hochreiter and Jürgen Schnidhuber. Long short-term memory, Neural Comput., (8)) 1997. doi: ww.1162/neco.199.998.110
1733
Jordan Hoffmann, Scbastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compate-optimal large language model training. In Sanmi Koyejo, S. Mohamod, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, http///222//a/aa// November 28 December 9, 2022, 2022. /ttptt/:://paeerr... nips wttps/ccccpppppeeeeeeeeeeeeeeeee feoaperfflle//ee E10701110558877703111111666 ott. Constract-Confereee html
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyn Ran, Lingfeng Xiao, and Chenglin Wu. Metagpt: Meta programming for multi-agent collaborative framework. CoRR, hbs/2308.00352, 2023. doi: 10.48550/
ARXIV 20E.V.230..00352. URL
48550//r/..55 2008.
Yang Hong, Chakkrit Tantithamthavorn, Patanamon Thongtamunam, and Aldeida Aleti Commentfinder: simpler, faster, more accurate code review comments recommendation. In Abhik Roychoudhury, Cristian Cadar, and Miryung Kim (eds.), Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, pp. 507-519. ACM, 2022, doi: 10.1145/354002503354999 URL https:
7/do1. otw.l.org/111111 1ttp://...111/5455500050333 3549119
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACL. 2023, Toronto, Canada, July 9-14, 2023, pp. 14409-14428. Association for Com- putational Linguistics, 2023. doi: 10.18653/v1 acl-lon.. 806. URL https://doi.org//// 18653//11 2023. acl-long. 806]
Iman Hosseini and Brendan Dolan-Gavitt Beyond the C: retargetable decompilation using neural machine translation. CoRR, /bss/212..0ess 089500 2022. doi: 005500/ARXIV.2222.08855 URL https://doi. org/ 10. 00.48550/ark/ wwwaa.ivvvv.... 221.22222222222 08950
Xinyi Hou, Yanjic Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John C. Grundy, and Haoyu Wang. Large language models for software engineering: systematic literature review. CoRR, abs/2300..066 10620, 2023. doi: 10.48550/ARM/ 1062. URL https://doi. org/10. /0/0 48550/arXiv.
2308. 10620
Gang Hu, Min Peng, Yihan Zhang, Qianqian Xie, and Mengting Yuan. Neural joint attention code scarch over structure embeddings for software q&aa sites. J. Syst. Softw., 170:110773, 2020. doi: 10.1016/J.JSS. 1100.1 URL https://doi. org/10...111... 1016/ jss. 2020. 110773
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep code comment generation. In Foutse Khomh, Chanchal K. Roy, and Janet Sicgmund (ods.), Proceedings of the 26th Conference on Program Com- prehension, ICPC 2018, Gothenbury, Sweden, May 27-28, 2018, pp. 200 210. ACM, 2018a. doi: 011444///99621.3111
URL httphttss//i..rrgg/ggo/ 1///100.11455 /115/1196321 3196333
Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. Summarizing source code with transferred API knowledge. In Jérôme Lang (ed.), Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pp. 2269 2275. ijcai.org, 2018b. doi: 0.224663/jeaai.20383
URL https:///dot org/10. 24963/1jcai .211/3311
59

---

Published in Transactions on Machine Learning Research (09/2024)
Yang Hu, Umair Z. Ahmed, Sergey Mechtaev, Ben Leong, and Abhik Roychoudhury. Re-factoring based program repair applied to programming assigmmnns.. In 34th IEEE/ACM International Conference Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019, pp. 388 398. URL littps://ddot org/10. 11111...11AAA.... 2019
IEEE, 2019. doi: 10.1109/ A00.//....221..000.
Di Huang, Rui Zhang, Xing Hu, Xishan Zhang, Pengwei Jin, Nan Li, Zidong Du, Qi Guo, and Yunji Chen. Neural program synthesis with query. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nng net, 2022a. URL htttss// net/
forum Dormmiid=NyJJKII8811
Dong Huang, Jianbo Dai, Han Weng, Puzhen Wu, Yuhao Qing, Jie M Zhang, Heming Cui, and Zhijiang Guo. Soap: Enhancing efficiency of generated code via sell-optimivation. 2024. URL. https://doi. org/
10. 40.50550//rxvv 2405.15189
Junjic Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. Cosqa: 20, 000+ web queries for code search and question answering. In Chengging Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing, IIIII 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 5690- 5700. Association for Computational Linguistios, 2021. doi: 10.18653/v1 (0653/v1/20/ /2221 lall-long.42.. URL https::/doi... orgil.or//110 18653/11/20211 http:////22211111111111aa acl-long. 442
Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin B. Clement, Nan Duan, and Jianfeng Gao. Execution-based evaluation for data science code generation models. CoRR, abs/2211 2211//22111991.1 2022b. doi: AR.IIV
URL https://doi.org: http:////ooorg/1....
48550/arXiv arwiv..iiivvvviiaaaeaaaaavvaaela 221122222222112 2211.09374
Kai Huang, Zhengzi Xu, Su Yang, Hongyu Sun, Xuejun Li, Zheng Yan, and Yuqing Zhang. A survey on automated program repair techniques. CoRR, abs/ 22003.181888 2023a. doi: A0.48350/ARRIV 2303.18184. URL https::/doi. org/10 100 48550/arXiv. 2303 2308.18184
Qiao Huang, Xin Xia, Zhenchang Xing, David Lo, and Xinyu Wang. API method recommendation without worrying about the task-api knowledge gap. In Marianne Huchard, Christian Kästner, and Gordon Fraser (eds.), Proceedings of the 33rd ACM/IEEE International Conference on Aulomated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018, pp. 293-300 304. ACM, 2018. doi: 10.1145/32388 3238191. URL lhttt::://aat /tttt::://ao..rg/11... 1145/3238147 32388191
Yuan Huang, Nan Jia, Hao-Jie Zhou, Xiangping Chen, Zibin Zheng, and Mingdong Tang. Learning human- written commit messages to document code changes. J. Comput. Sei. Technol., 355)) 2020. doi: 10100//11190002/1
URL
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi- level multi-discipline chinese evaluation suite for foundation models. CoRR, abs/2305.08322 2023b. doi: 10.48550/ aiv. 20w.23305.8883. 08322. URL https://doii org/10/ 100 48550/ arro0./axxv..... 2305. 2305.08322
Faria Huq, Masum Hasan, Md. Mahim Anjum Haque, Sazan Mahbub, Anindya Tqbal, and Toufique Ahmed. Review4repair Code review aided automatic program repairing. Inf. Softw. Technol,, 143:106765, 2022.
doi: 10.1016/J.INFSCE 10621. URL https:///oi. http///oooorgg///// 1etg//0..016//111 infsof
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet
challenge: Evaluating the state of semantic code search. CoRR, abs/ hbs/1199.09446, 2019. URL http:
//arxiv.org// www.:////g//..o....//. 09436
Wonscok Hwang, Jinyeung Yim, Sounghyun Park, and Minjoon Soo. A comprehensive exploration on wikisq] with table-aware word contextualization. CoRR, abs/ abs/1192.00069, 2019. URL http::///axxv..rr/abbb 1902. 01069
60

---

Published in Transactions on Machine Learning Research (09/2024)
Mohd Ibrahim and Rodina Ahmad. Class diagram extraction from textual requirements using natural language processing (nlp) techniques. In 2010 Second International Conference on Computer Research and Development, pp. 200 204, 2010. doi: D..1009/CCCDD.2000000
Yoichi Ishibashi and Yoshimasa Nishimura. Self-organized agents: A Ilm multi-agent framework toward ultra largo-scale code generation and optimization. CoRR, 0tss/2444.0022 2024. doi: 10.48550/ARXIV ARD./ 2404. 02183. URL httpss://dii.rrg: 10. 48550/arXii 240. 02183
Srinivasan Iyer, Ioannis Konstas, Alvin Choung, and Luke Zettlemoyer. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association URL https://doi. http:://boi.oor/11...88 18653/v1/
for Computer Linguistics, 2016. doi: 10.18633/11pp16-1195
p16-1195
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. Learning a neural semantic parser from user feedback. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 163--733 Association for Computational Linguisties, 2017. doi: 10. 10.18653/V1/ /V/1///1//7110 URL http:///aaooooo/.000///////11111 18653//11/P17-1089
Srinivasan Iyer, Toannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in pro- grammatic context. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31- November 4, 2018, pp. 1643-1652. Association for Computational Linguistics, 2018. doi: 10.18653/v1 //11/-111T 1192-1192. URT. httt:://ooa.oog///....1///////// 11553//11/188-11993
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Percyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. OPT-IML: scaling language model instruction meta learning through the lens generalization. CoRR, abs/ abs/2212.120 2022. doi: 10.48550/ http///.i.............. 2212.12017. URL atttps://ooi https://doi.org/11 100/10. 48550/arXiv. 2212.12017
Chirag Jain, Preethu Rose Anish, Amrita Singh, and Smita Ghaisas. A transformer-basee approach for abstractive summarization of requirements from obligations in software engineering contracts. In Kurt Schneider, Fabiano Dalpiaz, and Jemnifer Horkoff (eds.), 31st IEEE International Requirements Engineer- ing Conference, RE 2023, Hannover, Germany, September 4-8, 2023, pp. 169- 1799. IEEE, 2023. doi: http://2a...rr///1.011100///////
10.1109 10.1109/RE57729 /RE///555782280 20177227..222.000 2022.0002. URL
2023. 2023.00025 00025
Naman Jain, Skanda Vaidyanath, Arun Shankar Tyer, Nagarajan Natarajan, Suresh Parthasarathy, Sri- ram K. Rajamani, and Rahul Sharma. Jigsaw: Large language models meet program synthesis. In 44th SEEE/ACM 44th International Conference on Software Engincering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 3510203 //doi.org/Bi 10. 1144/151113000 3510003000000000 151444/3250003331122333
ACM, 2022. doi: (0.1145/351100033551222
URL https:
Prithwish Jana, Piyush Jha, Haoyang Ju, Gautham Kishore, Aryan Mahajan, and Vijay Ganesh. Attention, compilation, and solver-based symbolic analysis are all you neod. CoRR, abs/2306.00m 06755, 2023. doi: 10.48550/ARD ARX.V. T.t.22366000 06755. URL
2306 06755
Abhinav Jangda and Gaurav Anand. Predicting lypes dyyamiicill lyped programming languages. CoRR, abs/ 1111119111.01111 2019. URL http://arriv..rr//be///11....1
Kevin Jesse and Premkumar T. Devanbu. Manyttyees4typpecrriit: A comprehensive typescript dataset for sequence-based type inference. In 19th TEEE/ACM International Conference Mining Software Repositories, MSR 2022, Pittsburgh, PA, USA, May 23-24, 2022, pp. 294 298. ACM, 2022. doi: 00.1145//524812.3325577 URL https://doi org/10 10. 1145/3524842.56665 35285077
Kevin Jesse, Premkumar T. Devanbu, and Toufique Ahmed. Learning type annotation: is big data enough? In Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (eds.), ESEC/PSR
61

---

Published in Transactions on Machine Learning Research (09/2024)
211 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Soflware Engineering, Athens, Greece, August. 23-28, 28, 2021, pp. 1483-1486 ACM, 2021. doi: 10.1145/ 3468264.3473135. URL https://doi...o.. org/10 111...11/3/43466....1 34414//336626644333333 3473135
Kevin Jesse, Toufique Ahmed, Premkumar T. Devanbu, and Emily Morgan. Large language models and simple, stupid bugs. In 20th IEEE/ACM International Conference on Mining Software Repositories, MSR 2023, Melbourne, Australia, May 15-16, 2023, pp. 563 575. TEEE, 2023. doi: 10. (0.1109/MSS55973.2022 00082. URL https://doi. 11..:...1119/// 1109/MS5599995 022.000882..
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnicr, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/ abs/2310.0 /2310.33002223 2023a. doi: 10.48550/ (18550/ARXVV ARXIV 2310.06882.. URL
https://don org/ 10.g//10.440 408.44485//iiiv... 2310. 06825
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guil- laume Bour, Guillaume Lample, Lélio Remard Lavaud, Lucile Saulnier, Marie-Ama Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. CoRR, abs/2401..0 04088, 2024a. doi: 10.48550/ARAA ARRII/ 2401.04088. URL aattps://aoi.org: 48550/ arwiv. 2401 04088
Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J. Cai. Promptmaker: Prompt-based prototyping with large language models. In Simone D. J. Barbosa, Cliff Lampe, Caroline Appert, and David A. Shamma (eds.), CHI '22 CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 5 May 2022, Extended Abstracts, pp. 35:1- 35:8. ACM, 2022. doi: 10.1145/3491100 1013353564... URL https::/t/oot http:////ooo.rg//./.. /00 /145/3491//111111 3453339
Lin Jiang, Hui Liu, and He Jiang. Machine learning based recommendation of method names: How far are we. In 34th IBEE/ACM International Conference on Aulomated Software Engineering, ASE 2019, San Diego CA, USA, November 11 11-15, 2019, pp. 602-614. IEEE, 2019. doi: B..1099AAE.2019.000 https: //to///ooooooooooo org/oogg/// 10. 1119/110/SSS ASE. 2019. 00062
URL
Lingxiao Jiang, Ghassan Misherghi, Zhendong Su, and Stéphane Glondu. DECKARD: scalable and accurate tree-based detection of code clones. In 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007, pp. 105. IEEE Computer Society, 2007. doi: 10. 1109/ICSE.2007.30 30. URL hhttps:///oot org/10.gg.o.
2007 .3
Nan Jiang, Thibaud Lutellier, and Lin Tan. CURE: code-aware neural machine translation for automatic program repair. In 43rd IEEE/A ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pp. 1161-1173 TEEE, 2021a. doi: https://doi.org 10. 1109 11w.11199IIISS30 IC////SSE44332 2ww.:/990.22022..00 00107
URL
Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. Impact of code language models on automated program repair. In 45th IEEE/A AAC International Conference on Software Engincering, ICSE 2023, Melbourne, Austrulia, May 14-20, 2023, pp. 1430 1442. IEEE, 2023b. doi: 10.1109/CCSE44611.2223 2023. 001255
URL :/ttps:///i... 0rg//0.0/0 1tt.:11199///5880
Nan Jiang, Chengxiao Wang, Kevin Liu, Xiangzhe Xu, Lin Tan, and Xiangyu Zhang. Nova Generative language models for binaries. CoRR, abbs/2311.13// 2023c. doi: 10.48550/ 118650/ARXIIV2231111122 URL
https::/ddoi.org//1 48550//rri55 www.:xwwi...i......m.mmm.1 2311 13721
Siyuan Jiang and Collin McMillan. Towards automatic generation of short summaries of commits. In Giuseppe Scanniello, David Lo, and Alexander Serebrenik (eds.), Proceedings of the 25th International Conference on Program Comprehension, ICPC 2017, Buenos Aires, Argentina, May 22-23, 2017, pp. 220-323. IEEE Computer Society, 2017. doi: 10.1109/ICPC..0111112 URL https:///oi.orrg ICPC. 2017.1 12.
62

---

Published in Transactions on Machine Learning Research (09/2024)
Siyuan Jiang, Ameer Armaly, and Collin McMillan. Automatically generating commit messages from diffs using neural machine translation. In Grigore Rosu, Massimiliano Di Penta, and Tien N. Nguyen (eds.); Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, Urbana, IL, USA, October 30 November 03, 2017, pp. 335-146. IEEE Computer Society, 2017. doi: 10.1109/ 10.1109/ASE.2017 207. URL hattps://don org 10. 1109 ASE. 2017. 88115626
Xue Jiang, Zhuoran Zheng, Chen Lyu, Liang Li, and Lei Lyu. Treebert: A tree-based pre-trained model for programming language. In Cassio P. de Campos, Marloes H. Maathuis, and Erik Quaeghebeur (eds.), Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, UAI 2021, Virtual Event, 27-30 July 2021, volume 161 of Proceedings of Machine Leurning Research, pp. 54 63. AUAI Press, 2021b. URL https::///// tttt:p::/proceeeiigg
mlr wwwps//vee///ppppppp http:///vvvv//anne.e ianggla..haa .html
Yufan Jiang, Qiaozhi He, Xiaomin Zhuang, and Zhihua Wu. Code comparison tuning for code large language models. CoRR, abs/ 120111912,, 2024b. doi: 10.48550/ARXTT) AAX URL. https:///oi..on org/10.
48550/arXiv. 2401. 19121
Zhihan Jiang, Jinyang Liu, Zhuangbin Chen, Yichen Li, Junjie Huang, Yintong Huo, Pinjia He, Jiazhen Gu, and Michael R. Lyu. Lilac: Log parsing using Iims with adaptive parsing cache. CoRR, abss/331000177 2023d. doi: 10.48550/ ARXIV 231001177... URL https://dof: org/10. 408500 aar5vv/ 017966
Zhihan Jiang, Jinyang Liu, Junjie Huang, Yichen Li, Yintong Huo, Jiazhen Gu, Zhuangbin Chen, Jieming Zhu, and Michael R. Lyu. A large-scale benchmark for log parsing. CoRR, abs/ /208 10828, 2023e, doi: 10.48550/ ARW.V 2308. 2308.10828. URL /ttp:::://oi.org/100.. 48550/45555333 2308 10828
Mingshong Jiao, Tingrui Yu, Xuan Li, Guanjic Qiu, Xiaodong Gu, and Beijun Shen. On the evaluation of neural code translation: Taxonomy and benchmark. CoRR, abs/230s 02308,08961, 2023. doi: 10.48550/ AAXIVV. 2308.08961. URL https://doi.ooo org/10. 10. 48550/ http//aavivv.... 2306108961,
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? CoRR, abs/2310.00 06770, 2023. doi: 10.48550/ CAB::///RXVV.. D2IXXI..331000770, URL https://doii: http::///ooi.oo///..../ 000.44550///// (at/s00/aaxi..... 2310.0677000
Timo Johann, Christoph Stanik, Alireza M. Alizadeh B., and Walid Maalej. SAFE: A simple approach for feature extraction from app descriptions and app reviews. In Ana Morcira, João Araújo, Jane Hayes, and Barbara Paech (eds.). 25th TEFE International Requirements Engineering Conference, RE 2017, Lisbon, Portugal, September 4-8, 2017, pp. 21- 30. IEEE Computer Society, 2017. doi: (0.1109/RE.2017.77 URL
https://doi. org/ 10.
Harshit Joshi, José Pablo Cambronero Sánchez, Sumit Gulwani, Vu Lc, Gust Verbruggen, and Ivan Radicok. Repair nearly generation: Multilingual program repair with llss. In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), Shirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty- Fifth Conference on Innovatéve Applications of Artificial Intelligence, TAAI 2023, Therteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, URL httpp://ddot org/10.
2023, pp. 5131 51400 AAAT Press, 2023. doi: 2.1609/AAALV3374..220 1609/aaai httaa/aaaiv.3..... v37.4.256.444
Tae-Hwan Jung. Commitbert: Commit message generation using pre-trained programming language model. CoRR, 0bss21155142444 2021. URL
wttp://wwww...raa////.././. 14242
René Just. The major mutation framework: efficient and scalable mutation analysis for java. In Corina Pasarcanu and Darko Marinov (ods.), International Symposium on Software Testing and Analysis, ISSTA '14, San Jose, CA, USA July 21- 26, 2014, pp. 433-436 ACM, 2014. doi: 00.1145/261038822228033
URL https://doi. org/ /145/2610384.... 2628053
René Just, Darioush Jalali, and Michael D. Ernst. Defects4j: a database of existing faults to enable controlled testing studies for java programs. In Corina S. Pasarcanu and Darko Marinov (eds.), International Sympo- sium on Software Testing and Analysis, ISSTA '14, Son Jose, CA, USA July 21 26, 2014, pp. 437-440. ACM, 2014. doi: 10.1145 2255/610336 22610384.622055, URL http:////oo.oo//.///// 1. http:///....111/1110200200200000 2628055
63

---

Published in Transactions on Machine Learning Research (09/2024)
Vaibhavi Kalgutkar, Ratinder Kaur, Hugo Gonzalez, Natalia Stakhanova, and Alina Matyukhina. Code authorship attribution: Methods and challenges. ACM Comput. Surn., 52(1))3:11385) -:33:,,, 2019. doi: 10. 1145/3292577. URL https://doi.orrr org/10. 0tt../0......1//993377777777 1145/3292577
Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gulwani. Neural-guided deductive scarch for real-time program synthesis from examples. In 6th International Con- ference on Learning Represenlations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Con- .nt. wwt.//orumidddrryDDD... -g
ference Track Proceedings. OpenReview.nee net, 2018. URL [ttLL:tt////pperrviiii
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volune 119 of Proceedings of Machine Learning Research, pp.
PMLR, 2020. URL http://proceediings mlr wwwps//v19//aaaaaeeeeee kww/deaaaaaddaaaaaeeaaa 2mmmm
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL
Svetoslav Karaivanov, Veselin Raychev, and Martin T. Vechev. Phraso-based statistical translation of programming languages. In Andrew P. Black, Shriram Krishnamurthi, Bernd Bruegge, and Joseph N. Ruskiewicz (eds.), Owwwrrd! 2014, Proceedings of the 2014 ACM International Symposium on New Ideas, New Parodigms, and Reflections on Programming & Software, part of SPLASH '14, Portland, OR, USA, October 20-24, 2014, pp. 173-184. ACM, 2014. doi: 10.1145/26611362661111 2661141188..6.888 URL https://doi... oww.ooo//////... 11.....111///6111... 2661148
Mafhel-Michael Karampatsis and Charles Sutton. How often do single-statement bugs occur?: The manysstubs4j dataset. In Sunghun Kim, Georgios Gousios, Sarah Nadi, and Joseph Hejderup (eds.), MSR 20: 1 17th International Conference on Mining Software Repositories, Scoul, Republic of Korea, 20- 30 June, 2020, pp. 573-577. ACM, 2020. doi: 10.1145/3399597.338748 URL https://doi. org/10.
1145/3379597.3387744
Rafhel-Michacl Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. Big code != -big vocabulary: open-vocabulary models for source code. In Gregg Rothermel and Owan Bae (cds.). ICSE '20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June 19 July, 2020, pp. 1073 1085. ACM, 2020. doi: 10.1145/33378111 33380344.. URL https://doi org/10. 1145/3377811 3358/33337811..333422 3380342.
Amir Hossein Kargaran, Nafiseh Nikeghbal, Abbas Heydarnoori, and Hinrich Schütze. Menucraft: Interactive menu system design with large language models. CoRR, abs/2303. 0ass/2333.044990 04496, 2023. doi: 10.48550/ 00.4855/AAXTIV. ARXIV. 2303.04496. URL /tttp::/ddo..org/10. 42g/// arXiv 2303 04496.
Vinceth Kashyap, Jason Ruchti, Lucja Kot, Emma Turetsky, Rebecca Swords, Shih An Pan, Julien Henry, David Melski, and Eric M. Schulte. Automated customized bug-benchmark generation. In 19th Inter- national Working Conference on Source Code Analysis and Manipulation, SCAM 2019, Cleveland, OH, USA, September 30- October 1, 2019, pp. 103 114. IEEE, 2019. doi: AALL00/SCAAM.019.00000 URL
00020
George Katsogianiis-Meimaaaaiis and Georgia Koutrika. A survey on deep learning approaches for text-to- sql. VLDB J., 32(4) 936, 2023. doi: 10.1007/50077--022-00777 URL 8807788-02-00776-88
Amol Kelkar, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, and Peter Relan. Bertrand-dr: Improving text-to-sql using discriminative re-ranker. CoRR, abs/2002. 00557, 2020. URL htttps://arxiv.orr
abs/2002.00557
Mohamad Khajezade, Fatemeh Hendijani Fard, and Mohamed S. Shehata. Evaluating few shot and con- trastive learning methods for code clone detection. CoRR, abs/2204.07501 2022. doi: 00.48550/ARXI1
2204.07501. URL httttt://doi..oooo 48550/ arXiv 2204 2204..75011111 07501
64

---

Published in Transactions on Machine Learning Research (09/2024)
Junaed Younus Khan and Gias Uddin. Automatic code documentation generation using GPT-3. In 37th TEFE/ACM International Conference on Aulomated Softwwore Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022, pp. 174:1 174:6. ACM, 2022. doi: 00.1455/3551344.35598 URL https: 11111114455551 /ESS//55514999 3559548
Mohammad Abdullah Matin Khan, M. Saiful Bari, Xuan Long Do, Weishi Wang, Md. Rizwan Parvez, and Shafiq R. Joty. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. CoRR, alss/2303.00m 03004, 2023. doi: (148550/ARXI) 2300.030044
URL https://do..orgg org/ arxvv. 2ww.x.i...0...... 2303 03004
Ahmed Khanfir, Renzo Degiovanni, Mike Papadakis, and Yves Le Traon. Efficient mutation testing via pre-trained language models. CoRR, abs abs/2301..000 003443, 2023a. doi: (0.48550/arXiv.222 0tt.r.231..3344.. URL
https://doi.oom org/ 10///10.4888 48550/855/aaa /ar000////v.v... 2301 03543
Ahmed Khanfir, Anil Koyuneu, Mike Papadakis, Maxime Cordy, Tegawendé Rissyandé, Jacques Klein, and Yves Le Traon. ibir: -ugrreport-driven fault injection. ACM Trons. Softw. Eng. Methodol., 32(2): 33:1-33:31, 2023b.
doi: 10.1145/3542946. URL hettps://doi.rrg/1/0 1145/3542946
Kisub Kim, Dongsun Kim, Tegawendé F. Bissyandé, Eunjong Choi, Li Li, Jacques Klein, and Yves Le Traon, Facoy: code-to-code search engine. In Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (eds.), Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 June 03, 2018, pp. 946-957. ACM, 2018. doi: URL https://doi. org/ 10. 1145/3180155 31801877
Nikita Kitacv, Lukasz Kaiscr, and Anselm Levskaya. Reformer: The efficient transformer. In 8th Inter- national Conference on Learning Represenlations, ICLR 2020, Addis Ababo, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpps:/ heet:/foruniidrrgkWtttttt
Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harm de Vries. The stack: TB of permissively licensed source code. Trunsactions on Machine Learning Research, 2023. ISSN 2835-8856. URL http::///pppaaeeeee....e////rrrp
Kuang, Cong Zhou, and Xiaoxian Yang. Code comment generation based on graph neural net- work enhanced transformer model for code understanding in open-source software ecosystems. Au- tom. Softu. Eng., 29(2):43, 2022. doi: 10.1007/$106115-022000111--- URL https://doi. httt::://doo.org/1.... org/10.1000)
610515-022-00341-
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang, Spoc: Search-basee pseudocode to code. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' A'Achh-Buu,, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, Decem- ber 84, 2019, Vancouver, BC, Canada, pp. 11 11883 -1899411894 2019. URL httth:tt///// ptttps:://proceeingss neurips..
Chinmay Kulkarni, Stefania Druga, Minsuk Chang, Alex Fiannaca, Carrie J. Cai, and Michael Terry. A word is worth a thousand pictures: Prompts as AI design material. CoRR, abs/2303.1264 2023. doi: 10.48550/ARD /ttp//RXIV///// 200VV2230.1111 12647. URL httes:://ao..rg//// wtw.///.....r./....////////////
htt.aa.wwww..........e. 2303. 12647
Ayush Kumar, Parth Nagarkar, Prabhav Nalhc, and Sanjeev Vijayakumar. Deep learning driven natural languages lext to SQL query conversion: survey. CoRR, abs/2208 04405, 2022. doi: 10.48550/ARDA ARXIV.
URL https://doi. wrw.al..ggg....o www.....///.////////////////.../ arwii. 2208 2208.004151
Deeptimahanti Deva Kumar and Ratna Sanyal. Static uml model generator from analysis of requirements (sugar). In 2008 Advanced Software Engineering and Its Applications, pp. 77-84, 2008. doi: 10.1109/ 250...2....2.
65

---

Published in Transactions on Machine Learning Research (09/2024)
Marie-Anne Lachaux, Baptiste Rozière, Mare Szafraniec, and Guillaume Lample. DOBF: A deobfuscation pre-training objective for programming languages. Tn Marc' Aurelio Ranzato, Alina Beygekzime, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 94: Annual Conference on Newral Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 14967-149799 2021. URL httpp::/prooceedings http://eeeee....ppee/////2////2
Jeremy Lacomis, Pengeheng Yin, Edward J. Schwartz, Miltiadis Allamanis, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. DIRE: A neural approach to decompiled identifier naming. In 34th TEEE/AC3 CM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November 11-15, 2019, pp. 628-639. IEEE, 2019. doi: 10.1109/ASE.2013 00064. ASE. 2019. 00064
URL lntpps://do1. org/10. 1109/
Shuvendu K. Lahiri, Aaditya Naik, Goorgios Sakkas, Piali Choudhury, Curtis von Veh, Madanlal Musuvathi, Jeevana Priya Inala, Chenglong Wang, and Jianfeng Gao. Interactive code generation via test-driven user-intent formalization. CoRR, abs/2200 009500, 2022. doi: 10.48550/ar 220.v..2200.006. URL https: 2208 05950
//doi.org//0/
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida I. Wang, and Tao Yu. DS-1000, A natural and reliable benchmark for data science code generation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honobulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 18319- 18345. PMLR, 2023. URL https::/ http:://pooeeddig......rrrr/////
mmlr wtt....preee//222211222222 html
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for supersupe learning of language representations. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020. URL
Max Landauer, Sebastian Onder, Florian Skopik, and Markus Wurzenberger. Deep learning for anomaly detection in log data: A survey. CoRR, 22000020000 2022. doi: 10.48550/ (18550 22..2222022020 URL https://doi. org/ 10. 48550/ wrt.aaaaiivvv. 2207.2227..3333
Chris Lattner and Vikram S. Adve, LLVM: A compilation framework for lifelong program analysis & transformation. In 2nd IEFE/ / ACM International Symposium on Code Generation and Optimization (CGO 2004), 20-24 March 2004, San Jose, CA, USA, pp. 75 88. IEEE Computer Society, 2004. doi: 10.1109/CGO.2000 200..02041110000 665. URL hhttps://doi org/ 10. 11w.//C0...... 2004. 1281665
Hugo Laurengon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Seao, Leandro von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Sasko, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Bi- derman, Anna Rogers, Loubna Ben Allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, So- maieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepereq, Suzana Tlic, Margaret, Mitchell, Alexan- dra Sasha Luccioni, and Yacine Jernite. The bigscience ROOTS corpus: A 1.6tb composite multi- lingual dataset. In NewrIPS, 2022. URL httpp://appers... wwwpp.ppppppppp....e aecel2aal22222133333777800o---22
wwtpccppppaeeeeeeeeeeeee httpl/ppprffeep/ppellller https/ppaprr2222//aaae
and Benchmarks html
Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web sereenshots into HTML. code with the websight dataset. CORR, abs abs/2403 /2/22.24902999 2024. doi: 10.48550/ (18550//AXXVV ARX.V.2400.000000000000 090.09...... URL httpp:://bi..rgg////
2403 09029
66

---

Published in Transactions on Machine Learning Research (09/2024)
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learn- ing. In NewrIPS, 2022. URL http://papers: nips. https/cccpppeeaaeeeeeeeeeee Filees/papper2222///ehl 1G56419dealla@$tbo55fc2222 702a4 Mebsssract-Coofffee
html
Van-Hoang Le and Hongyu Zhang. Log-based anomaly detection without log parsing. In 36th TEEE/A AABE/AAM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021, pp. 492-504. IEEE, 2021. doi: 21110//ASEE116222221..00 10. N0.1109/ASE51522 2021 9678773
URL https:///oi.co org/
Van-Hoame Le and Hongyu Zhang. Log parsing with prompt- based few-shot learning. In 45th IEEE/ACM International Conference Software Engineering, ICSE 2023, Melhowrne, Australia, May 14-20, 2023, 20w.:/661..20200022222 00204.. URL https://doi org/10. 1109/
pp. 2438 2449. IEEE, 2023a. doi: ICSE48619 2023.
Van-Hoang Le and Hongyu Zhang. Log parsing: How far can chatgpt go? In 38th IEEE/ACM Interna- tional Conference on Automated Software Engineering, ASE 2023, Luzembourg, September 11-15, 2023, pp.
IEEE, 2023b. doi: AA1009/ASE56229..222.... 002006. URL https: /ttp:::///oooor/11...o ASE56229 00206)
Hugh Leather and Chris Cummins. Machine learning in compilers: Past, present and future. In Forum for Specification and Design Languages, FDL 2020, Kiel, Germany, September 15-17, 2020, pp. 1-8. IEEE, 2020. doi: 10.1109/PDL5000 2FD.://0111...200922222.. URL ttttps://doi..or/10... wtt.://.rr//1.111///0556665 1wt.111199FFF001 2020. 9232934.
Alexander LeClair, Siyuan Jiang, and Collin McMillan. A neural model for generating natural language summaries of program subroutines. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (ods.), Proccedings of the 41st International Conference Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 795-800 IEEE/ / ACM, 2019. doi: D0.109//CSEE2211.00085 URL https:///oi..no org/
10. 10.1109/ICSE, 2019 00087
Changyoon Lee, Yeon Seonwoo, and Alice Oh. CSIQA: A dataset for assisting code-based question answering in an introductory programming course. In Marine Carpuat, Vladimir Meza Ruíz (ods.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL 2022, Seattle, WA, United Stales, July 10-15, 2022, pp. 2026 2040. Association for Computational Linguistics, 2022. doi: 148
de Marneffe, and Iván
10.18653/ W1..022NNAACL-MMANN..
URL https: /ttps://doi..oo org/10. 111111111115331/2/222222
Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. Kaggledbqa: Realistic evaluation of text- to-sql parsers. In Chengqing Zong, Fci Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume 1: Long Papers), Vir- tual Event, August 1-6, 2021, pp. 2261 2273. Association for Computational Linguistics, 2021. doi: URL https: wtt.a/i...rr11............. 1863310.11161311222222 acl- 176
Woosuk Lee, Kihong Heo, Rajeev Alur, and Mayur Naik. Accelerating search-based program synthesis using learned probabilistic models. In Jeffrey S. Foster and Dan Grossman (eds.), Proceedings of the 39th ACM SIGPLAN Conference Programming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018, pp. 436-449 ACM, 2018. doi: 10.1145/319226663192222 URL attps://do1 org/ 10. /111/111923666311011
Wengiang Lei, Weixin Wang, Zhixin Ma, Tian Gan, Wei Lu, Min-Yen Kan, and Tat-Seng Chua. Re- examining the role of schema linking in text-lo-sql. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2020, Online, November 16-20, 2020, pp. 6943 6954. Association for Computational 2112220..MMLPPN.O
Linguistics, 2020. doi: 2020. 1020.emnlp-... emnlp-main 564
URL https: //doi. wroeii.org/...o org/110.8655///111 18653/v1/
67

---

Published in Transactions on Machine Learning Research (09/2024)
Caroline Lemieux and Koushik Sen. Fairfuzz: targeted mutation strategy for increasing greybox fuzz testing coverage. In Marianne Huchard, Christian Kästner, and Gordon Fraser (eds.), Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018, pp. 475-485. ACM, 2018. doi: 041145/8238147.32222 3238176
URL https:
Caroline Lemicux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. Codamosa: Escaping cov- erage plateaus in test generation with pre-trained large language models. In 45th LEEE/ACM. International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 139-931. 2023.
IEEE, 2023. doi: N0.1009/CSSE486192203. 00085
URL http::://ooo.or/1....11.
Yichong Leng, Xu Tan, Tao Qin, Xiang-Yaun Li, and Tie-Yan Liu. Unsupervised pivot translation for distant languages. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedinggs of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 175-183. Association for Computational Linguistics, 2019. URL https: org/10. 111611//1//111110/11
doi: 2..18653/v1/p99-1
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.neg net, 2021. URL @rwe7XHTmYb
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.). Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNIP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 3045-3059) Association for Computational Linguistics, 2021. doi: 10.18653/ emnlp emnlp-main. main. 243
www.le-mainnnnnmm main.243..
URL https://doi. http://ddo..rr//1111116666///1/
Keletso Letsholo, Liping Zhao, and Valeriu Chioasca. TRAM: A tool for transforming textual re- quirements into analysis models. In Ewen Denney, Tevfik Bultan, and Andreas Zeller (eds.). 2013 28th IEEE/ACM International Conference on Automated Software Engineering, ASE 2013, Silicon Valley, CA. USA, November 11-15, 2013, pp. 738-741. IEEE, 2013. doi:
AAE.AAEE..1..61144440 URL
https://doi. org/ 10. 11111/09AASEE 6693146
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-ssqquence pre-training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Anrual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7871-7880. Association for Computational Linguistics, 2020. doi: 10. (0.18653/v1/ /2tp:///000...mmm......... URL https://dot orgo 10///11.110 106138811311222 http://3///112222..mmmamm.. acl-main. 703
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wa, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models. In NewrIPS, 2022. URL http://papers. cttppssccppppeepppeeeepppp filess/paper/222///a html
Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei- Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-auggmented code emulator. CoRR, abs/ 22110.2444,, 2023a. doi: 10.48550/ARXT/ AAT.AARRXXV..1..47..... URL hbtps://aao..or//1...433
arXiv
Fei Li and H. V. Jagadish. Constructing an interactive natural language interface for relational databasos. Proc. VLDB Endow., 8(1):73-84 84, 2014. doi: (114777/273546.2222 URL http:://wt: vldb. org/
pvldh/rol8/pp3-11i.tt
68

---

Published in Transactions on Machine Learning Research (09/2024)
Guanhua Li, Yijian Wu, Chanchal K. Roy, Jun Sun, Xin Peng, Nanjie Zhan, Bin Hu, and Jingyi Ma. SAGA: efficient and large-scale detection of near-miss clones with GPU acceleration. In Kostas Kontogiannis, Foutse Khomh, Alexander Chatzigeorgiou, Marios Serrio--leetherios Fokaefs, and Minghui Zhou (eds.), 27th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2020, Lon- don, ON, Canada, February 18-21, 2020, pp. 272 283. IEEE, 2020a. doi: 10.1000/SANERR422555222 9054832. URL hhtpps///ot or/p::///doo.or/100... 10. S009SAAAR448255... 2020. 9054832
Haochen Li, Chunyan Miao, Cyril Leung, Yanxian Huang, Yuan Huang, Hongyu Zhang, and Yanlin Wang. Exploring representation-level augmentation for code search. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 4924- 4936. Association for Computational Linguisties, 2022a. doi: (2/6633//1/22..mmm/./ mww.p.2..mmmmmmmmmm32..... 327. URL https://doi.org: 18653/v1/20222 emnlp-main .27
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: measuring massive multitask language understanding in chinese. CoRR, //////336.092220 2023b. 488550/5555aii 2306. 09211
doi: 10.48550/ 11.8855//AXIIV ARXIV 09211..... URL https://doi.
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. RESDSQL: decoupling schema linking and skeleton parsing for text-to-sql. In Brian Williams, Yiling Chen, and Jemnifer Neville (eds.), Thirty-Seventh AAAT Conference Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference Innovative Applications of Arlificial Intelligence, IAAI 2023, Thirleenth Symposium on Educational Advances in Artificial Intelli- gence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pp. -3067-13075 AAAI Press, 2023c. doi: 10.1609/ 10.1609/AATT, A.A.A.AA..1IIII V37511.26535. URL https: org/100 10. 1609/ aaai
Heng- Yi Li, Shu-Ting Shi, Ferdian Thung, Xuan Huo, Bowen Xu, Ming Li, and David Lo. Deep- review: Automatic code review using deep multi-instance learning. In Qiang Yang, Zhi-Hua Zhou, Zhiguo Gong, Min-Ling Zhang, and Sheng-Jun Huang (eds.), Advances in Knowledge Discovery and Data Mining - 23rd Pacific-Asii Conference, PAKDD 2019, Macau, China, April 14-17, 2019, Proceed- ings, Part II, volume 11440 of Lecture Notes in Computer Science, pp. 318 330. Springer, 2019a. doi: 10.1007/978- 3-030-0006---00 16145-61 25. URL https:///oi. http::://ol..rgg/...10.0. 10t//00.1.077970-333-0000000 -30----5-----1--------
Hongyu Li, Scohyun Kim, and Satish Chandra. Neural code scarch evaluation dataset. abs/190s 1908,09804, 2019b. URL http::/
CoRR,
httg/ab//1908.000
Jia Li, Chongyang Tao, Zhi Jin, Fang Liu, Jia Allen Li, and Ge Li. ZC3: zero-shot cross- code clone detection. CoRR, abss2300813777 2023d. doi: 10.48550/ ARXIV 2308.13754, URL https::/doi. org/10. hrg/10.48588 (00550//rXiv... 2308 13754
Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. Acecoder: Utilizing existing code to enhance code generation, 2023e.
Jia Li, Ge Li, Xuamming Zhang, Yihong Dong, and Zhi Jin. Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. CoRR, ARXIV 2404.00599. URL https://doi. org/10 10. 48550//rX....20000
2024a. doi: 10.48550/
Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. Code completion with neural attention and pointer networks. In Jérôme Lang (ed.), Proceedings of the Twenty-Seventhh International Joinl Conference Artificial Intelligence, IJCAI 2018, July 18-19, 2018, Stockholm, Sweden, pp. 4165. ijcai.org, 2018a. doi: 10.24963 htts:/iiccii...... 2018/578. URL https://ddoi org/10. 20t.o.29933/]jc/ 2018/178
Jinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin, Chenhao Ma, Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing. In Brian Williams, Yiling Chen, and Jemnifer Neville (eds.), Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAT 2023, Washington, DC, USA, February 7-14, 2023, pp.
AAAI Press, 2023f. doi: 10.1609/ AAAL 1.656.. URL http::://a....rg//...oooo 1609/ waw.//00.aa....... 103711...5566
69

---

Published in Transactions on Machine Learning Research (09/2024)
Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang. Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chen-Chuan Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can LLM already serve as A database interface? A big bench for large- scale databasc grounded text-to-sqls. CoRR, abs/ 02////3550011111 2023g. doi: 10.48550/AAB /AR5IV 2305.03111. URL https://doi. org/10 10. 48550/arXiv. 2305. 03111
Junlong Li, Yiheng Xu, Lei Cui, and Furu Wei. Markuplm: Pre-training of text and markup language for visually rich docunent understanding. In Smaranda Muresan, Preslav Nakov, and Aline Villav- icencio (cds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 6078 6087. Asso- ciation for Computational Linguisties, 2022b. doi:
URL https:
18653/v1/2022 http://////2222..1111..... acl-long.42222 .420
Kaixin Li, Qisheng Hu, Xu Zhao, Hui Chen, Yuxi Xie, Tiedong Liu, Qizhe Xie, and Junxian He. Instruct- coder: Instruction tuning large language models for code editing. CoRR, abs/ abs/2310 2at./2330.2022,, 2023h. doi: 10.48550/ARXI 20..2..0020000 20329. URL
Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, and Jing Ma. Mmeode: Evaluating multi-modal code large language models with visually rich programming problems. CoRR, abs/2404.09486, 2024b. doi:
10.48550/ /A////XXIIV.2/ 2404.09886.. URL https: htt.///oo.orrr//// 10/ 48550/arXXa 2404.09486
Lingwei Li, Li Yang, Huaxi Jiang, Jun Yan, Tiejian Luo, Zihan Hua, Geng Liang, and Chun Zuo. AUGER: automatically generating review comments with pre-training models. In Abhik Roychoudhury, Cristian Cadar, and Miryung Kim (eds.), Proceedings of the 30th ACM Joinl. European Sofbware Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, pp. 1009 1021. ACM, 2022c. doi: 10.1145/3540233. URL https://doi.coo orgg..rg/111 1eeg//0..115//333 111114//554250.555554
Raymond Li, Loubna Ben allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia LI, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Joel Lamy-Poirier, Joao Monteiro, Nicolas Gontier, Ming-Ho Yee, Lo- gesh Kumar Umapathi, Jian Zhu, Ben Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason T Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Urvashi Bhattacharyya, Wenhao Yu, Sasha Luccioni, Paulo Villegas, Fedor Zhdanov, Tony Lec, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Carolyn Jane Anderson, Brendan Dovan,Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Scan Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. Starcoder: may the source be with you! Transactions Mu- chine Learning Research, 2023i. ISSN 2835-8856. URL / http://ppneeeeeeeeeeeeeorro//ooa het//formm?dd-ooPggll Reproducibility Certification.
Ruitong Li, Gang Hu, and Min Peng. Hierarchical embedding for code search in software q&a sites. In 2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020, pp. 1-10. IEFE, 2020b. doi: 2000://UCNNN400..200000.20 10. 0..1109/IJCNN48600 2020.92071011
URL htttps://dol org/
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-Im im- proves controllalle text generation. In NewrIPS, 2022d. URL httpp:/papeess wwtpppprrr.iipss....m httpssccc/ppperr wttpppppafalleee///// http:////sss/eecccooeeoeeeeeeeee Saw.aaaaaaabssaaat--ooooeeeeeeee
html
Xiaonan Li, Yeyun Gong, Yclong Shen, Xipeng Qiu, Hang Zhang, Bolun Yao, Weizhen Qi, Daxin Jiang, Weizhu Chen, and Nan Duan. Coderetriever: A large scale contrastive pre-training method for code search. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, FMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 2898 2898-2910. Association for Computational Linguistics, 2022c. doi: 10.18653/ v1 21/2022. eempp---mmmmmmain 187. URL //tp:////ooorg///0.0////////2220 emnlp-main. 187
70

---

Published in Transactions on Machine Learning Research (09/2024)
Xiaonan Li, Daya Guo, Yeyun Gong, Yum Lin, Yelong Shen, Xipeng Qiu, Daxin Jiang, Weizhu Chen, and Nan Duan. Soft-labeled contrastive pre-training for function-level code representation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistios: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 118-129. Association for URL :ttps://doi.org/10.. org/10.
Computational Linguistics, 20221. doi: 10. 10.18653/v1 www.:222..inngseeeeee
18653/v1/2022 Landdings-emnl....
Xuan Li, Zerui Wang, Qianxiang Wang, Shoumeng Yan, Tao Xie, and Hong Mei. Relationship-aware code search for javascript frameworks. Tn Thomas Zimmermann, Jane Cleland-Huang, and Zhendong Su (eds.): Proceedings of the 24th ACM SIGSOFT International Symposium Foundations of Software Engineering, FSE 2016, Seattle, WA, USA, November 13-18, 2016, pp. 690-701. ACM, 2016. doi: 10. 19.1145/950090. 2950341. URL. http::ttt///dtt http://tt..oooo/o//////o//////// 0011111/5/929999 29903333090999231111
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (ods.): Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 8198 8210. Association for Computational Linguistics, 2020c. doi: 10. 100.6/3/1120200AL--MMA
URL htttps://doi.rrg/110 wrgg//...866//1/22222222 acl-main. 729
Li, Shaohua Wang, Tien N. Nguyen, and Son Van Nguyen. Improving bug detection via context- code representation ion learning and boteetion-based neural networks. Proc. ACM Program. Lang,, ((OOSSL)): 162:1-162:30, 2019c. doi: 10.1145/3360588, /3665588. URL httpp::///oi...o
10. 1targ//1..114//33688333
Yi Li, Shaohua Wang, and Tien N. Nguyen. Dlfix: context-based code transformation learning for automated program repair. In Gregg Rothermel and Doo-Hwan Bae (eds.). ICSE '20: 42nd International Conference
on Software Engineering, Scoul, South Korea, 27 June 19 July, 2020, pp. 602-614. ACM, 2020d. doi:
(0.1145/3377811.23
URL htttp::/ddo..rgg/// wtt..o////...1/////......1 3380334
Yi Li, Shaohua Wang, and Tien N. Nguyen. Vulnerability detection with fine- fine-grained interpretations. In Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimíliano Di Penta (eds.), ESEC/PSE '21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, pp. 292-303. ACM, 2021a. doi: 10.1145/ 3468264.3468597 URL https:://do: org/10. 10. 1145/14661666 3468597
Yi Li, Shaohua Wang, and Tien N. Nguyen. A context-based automated approach for method name consis- tency checking and suggestion. In 43rd IEEE/ ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pp. 574-586. IEEE, 2021b. doi: 00060. URL httpss//doi.onr /ttp::////oo.o//////// 10.
Yi Li, Shaohua Wang, and Tien N. Nguyen. DEAR: A novel deep learning-based approach for automated program repair. In 44th (GEE/ACM 44th International Conference Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 27, 2022, pp. 511-523. ACM, 2022g. doi: 10.1145/3510000.33001111 URL https://doi.org 10.g/1..///0 /045115511113.00000 3510177
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Text- books are all you need II: phi-1.5 technical report. CoRR, Abs//2009.0/ 03099,0543,, 2023j. doi: 10.48550/ark arrviv 2309.05463. URL https: //doi: org/10. 10. 48550/arXX1 arxiv. 2309 05463.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowilz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-Ievel code generation with alphacode. Science, 378 378(6624) 2022h. doi: 00.1166/sciencc.abg] doi/abs/10..
URL httpp:://ww.sctt .org/ org/
Zehan Li, Jianfei Zhang, Chuantao Yin, Yuanxin Ouyang, and Wenge Rong. Procqa: A largo-scale community- programming question answering dataset for code search. CORR, abs 16700 2024c. doi: 10.48550/AA (188550/ARXIV-2 wRw./ 1670. URL https::/doi. org/10. 40t.///r0..55////a11...
71

---

Published in Transactions on Machine Learning Research (09/2024)
Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. Vuldeepecker: A deep learning-based system for vulnerability detection. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018, Son Diego California, USA, February 18-21, 2018. The Internet Society, 2018b. URL https://wwat www.nnddssspppp symposimm wwtp:///wp..ooottttaaaaaaaaaa//p
ndss2018 100088_03-2__21 Sai.iiipperpppppp .dff
Zhen Li, Deqing Zou, Shouhuai Xu, Zhaoxuan Chen, Yawei Zhu, and Hai Jin. Vuldeelocator: A deep learning-based fino-grained vulncrability detector. IEEE Trans. Dependable Secur. Comput., 19(4):2821- 2837, 2022i. doi: 1LL109/TTS..2021300077
URL https: http:///ooo..og///... 20w111000/SS02......
Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawci Zhu, and Zhaoxuan Chen. Sysevr: A framework for using deep learning to detect sore vulnerabilities. JEEE Trans. Dependable Secur. Compul., 2022j. doi: 10.1109 200....0..........11..
URL https: /ttp::://doiiooo wtt.://./0..1111.///11101 1119//1SCCCCCC 2021.
Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. Automating code review activities by large-scale pre-training, In Abhik Roychoudhury, Cristian Cadar, and Miryung Kim (eds.), Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022 pp. 1035-1047. ACM, 2022k. doi: 10. 10. 041444/55525005440000 URL https:: http:::///oioor/1///// 1144/35403000 35490811
Ruigang Liang, Ying Cao, Peiwei Hu, and Kai Chen. Neutron: an attention-based neural decom- piler. Cybersecur, 4(1):,, 2021. doi: 00.1186//4240000210000777
URL https:///oi.org// htgg/10.1865
842400-021-00070000
Dianshu Liao, Shidong Pan, Xiaoyu Sun, Xiaoxue Ren, Qing Huang, Zhenchang Xing, Huan Jin, and Qinying Li. A -odddgn::: A repository-level code generation framework for code reuse with local-aware, global-aware, and third/partyy-lbraayyawwaer CoRR, abs/ 2abs/2211.100 2312.05777
2023. doi: 10.48550 ARX1:/XRRRXX.312.572222.. 2312.05772. URL
10. 200550/arXivv 231.202222777
Derrick Lin, James Koppel, Angela Chen, and Armando
Quixbugs: a multi-lingual program repair benchmark set. based on the quixey challenge. In Gail C. Murphy (ed.), Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanily, SPLASH 2017, Vancouver, BC, Canada, October 23 - 27, 2017, pp. 55 56. ACM, 2017. doi: 10.1145/38 00.1145/3135932 /33333333332233 111355332.1139941 URL https://doi: httt::::/doi.orr/1...o /144/31359322 3135941,
Feng Lin, Dong Jae Kim, and Tse-Husn Chen. When Ilm-based code generation meets the software de- velopment process. CoRR, abs/2403.15852, 2024. doi: 10.48550 /AB.://ARXVV... 1203.1.2200.. URL https:
//doi.org//i 48550/arXiy 2403. 15852
Guanjun Lin, Jun Zhang, Wei Luo, Lei Pan, Yang Xiang, Olivier Y. de Vel, and Paul Montague. Cross-project transfer representation learning for vulnerable function discovery. IEEE Trans. Ind. Informatics, 14(7): 3289-3297, 2018a. doi: 10.1109/TII.ww TII./0III.11...... 2011.2821768. URL https://doi. 1rg/10..109/TIII
1109/TII, 2018. 2821768.
Guanjun Lin, Wei Xiao, Jun Zhang, and Yang Xiang. Deep learning-basel vulnerable function detection: A benchmark. In Jianying Zhou, Xiapu Luo, Qingni Shen, and Zhen Xu (eds.), Information and Commuu- nications Security - 21st International Conference, ICICS 2019, Beijing, China, December 15-17, 2019, Revised Selected Popers, volune 11999 of Lecture Notes in Computer Science, pp. 219 232. Springer, 2019. doi: 10.1007 0111//9977-----000
13. URL https:///ol. org. 10001079777---000-1155----------
Guanjun Lin, Jun Zhang, Wei Luo, Lei Pan, Olivier Y. de Vel, Paul Montague, and Yang Xiang. Software vulnerability discovery via learning multi-domain knowledge bases. IEEE Trans. Dependable Secur. Com- put., (((5):2469229) 2021. doi: 10.1109/TDSC.20 201.//C.S..222....2.22 29540995..8..... URL httt:s://bb..orr//...0/////// 2019. 2954088
www.ooo////1//o 1109/TDSC.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipong Qiu. A survey of transformers. AI Open, 3:111-132, 2022. doi: (0.1016/. 10.0000 URL
wwwppp...e..........om.oooo 2022.10.000
72

---

Published in Transactions on Machine Learning Research (09/2024)
Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. N12bash: A corpus and semantic parser for natural language interface to the linux operating system. Tn Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Köiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariami, Hélène Mazo, Asunción Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (cds.), Proceedings of the Eleventh International Conference on Language Resources and Bvaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA), 2018b. URL http://www.2. www...rooooooff..oo org/ httg://roeeddiggg/ee200///////// http://20000//mmmaaae///1//121/ html
Zhenghao Lin, Yeyun Gong, Yelong Shen, Tong Wu, Zhihao Fan, Chen Lin, Nan Duan, and Weizhu Chen. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise. In Andreas Kranse, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 21051-21064. PMLR, 2023. URL httppe//prroceedings mlr, www.sr.essssseeeee 1indddd html
Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, Fumin Wang, and Andrew W. Senior. Latent predictor networks for code generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10.18653/V1/P16 1057. URL https://doi.cooo http://do...rg////// org/10. 18653/v1/p16-105/
Xiang Ling, Lingfei Wu, Saizhuo Wang, Gaoning Pan, Tengfei Ma, Fangli Xu, Alex X. Liu, Chunming Wu, and Shouling Ji. Deep graph matching and searching for semantic code retrieval. ACM Trans. Knowl. Discov. Data, 15(5):88:1 88:21, 2021. doi: 10.1145/3447571. URL https:///oi....... org/10 10. 1145/3447571
Aiwei Läu, Xuming Hu, Lijie Wen, and Philip S. Yu. A comprehensive evaluation of chatgpt's zero-shot text-to-sql capability. CoRR, Abs/2300.13354 2023a. doi: 10.48550/ 11.48550/ARXIV ARI.A 1303.13777. URL https:
aoloi.org/10 48550/arXiv 2303 13547
Bingchang Liu, Chaoyu Chen, Cong Liao, Zi Gong, Huan Wang, Zhichao Lei, Ming Liang, Dajum Chen, Min Shen, Hailian Zhou, Hang Yu, and Jianguo Li. Mftcoder: Boosting code Ilms with multitask fine-tuning, CoRR, abs 0ass/2231.0220 2023b. doi: 10.48550/ ARX.V.X3X1..00
URL https://doi. org/10.48550/
arXiv 023003
Chao Liu, Xin Xia, David Lo, Zhiwei Liu, Ahmed E. Hassan, and Shanping Li. Codematcher: Searching code based on sequential semantics of important query words. ACM Trans. Softw. Eng. Methodol., 31((): 12:1-12:37, 2022a. doi:
URL https://aol.orr/// 1ttpr//....115/34411 1145/3465403
Chenxiao Liu and Xiaojun Wan. Codeqa: A question answering dataset for source code comprehension. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (cds.), Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Association for Computational Linguistics, 2021. doi: 10. 1861100100111 http://221.....nii/
Republic, 16-20 November, 2021, pp. MANNDINGSEEMNLP..0 223. URL
223
Fang Liu, Ge Li, Yunfei Zhao, and Zhi Jin. Multi-task learning based pre-trained language model for code complction. In 35th IEFE/ACC CEEE/ACM International Conference on Automated Software Engineering, ASE 2020, Melbourne, Australia, Seplember 21-25, 2020, pp. 473 485. IEEE, 2020. doi: 10.1145/33248884.341666 URL https://dot: /ttp::////oo.o//////// 11 1145/3324882 3416591
Fang Liu, Ge Li, Zhiyi Fu, Shuai Lu, Yiyang Hao, and Zhi Jin. Learning to recommend method names with global context. In 44th IEEE/ A AM 44th International Conference on Software Engineering,, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 2294-13066 ACM, 2022b. doi: 10.1145/3510003.355 attps:///ol org/ 10. 1....1.1. /141//501003....11 3510154
URL
Fang Liu, Jia Li, and Li Zhang. Syntax and domain aware model for unsupervised program translation. In 45th TEEE/ACM, International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 755 767. IEEE, 2023c. doi: 10.100//CSEE4661922222 00072.. URL https://doi. org/ 19//10.11000
1ww..1109/IISEEBBSS ICSE48619.444. 2023.0007272
73

---

Published in Transactions on Machine Learning Research (09/2024)
Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, and Li Zhang. Exploring and evaluating hallucinations in lIm-powered code generation. CoRR, als abs//444400077,, 2024. doi: 10.48550/
URL https://doi.
http:///oo...g//..... wtt.ll...////////rr..////../
Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. RETF: reinforcement learning from unit test leedback. CoRR, abs/ /t////337..3333 04349, 2023d. doi: 10.48550/ARXIT 200....300.444499 URL
https://aoi. org/10. 48550/55531131 2307. 04349
Jiawei Liu, Jinkun Lin, Fabian Ruffy, Cheng Tan, Jinyang Li, Aurojit Panda, and Lingming Zhang. Nn- smith: Generating diverse and valid test cases for deep learning compilers. In Tor M. Aamodt, Natalie D. Enright Jerger, and Michael M. Swift (eds.), Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2023, Vancouver, BC, Canada, March 25-29, 2023, pp. 330-543. ACM, 2023e. doi: (0.1145//575599333557777 URL https://doi. orgg wtt.:ooog//.//........... 111...1111/33798.333555555 3575707
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. CoRR, mtsp/2305.01211 2023f. doi: 10.48550/arXiv 2305.00100000000000 URL https:///oi. http::://dooorr/11/// 2...0.8855//rrra 2305. 01210
Kui Liu, Dongsun Kim, Tegawendé F. Bissyandé, Tae-young Kim, Kisub Kim, Anil Koyuncu, Suntae Kim, and Yves Le Traon. Learning to spot and refactor inconsistent method names. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (eds.), Proceedings of the 1st International Conference on Sofwware Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 12. IEEE / ACM, 2019a. doi: URL https: :/tpii///do.coo ///doi..rg//ue org/oogg//10 10/ 1101 ICSE. 00009
10.1109 (0000///CSEE.29..00000
Kui Liu, Anil Koyuncu, Dongsun Kim, and Tegawendé F. Bissyandé. Thar: revisiting template-basel automated program repair. In Dongmei Zhang and Anders Møller (eds.), Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2019, Beijing, China, July 15-19, 2019, pp. 31-42. ACM, 2019b. doi: 10.1145 0293882233305173 URL https://dol: oroeii.org/1... /rgg//.1145/ 0193882.3330577
Mingjie Liu, Nathaniel Ross Pinekney, Brucek Khailany, and Haoxing Ren. Verilogeval: Evaluating large language models for verilog code generation. CoRR, abs/2301 23099075444 2023g. doi: 10.48550/ARNS ARO//.XXXXVV. 07544. URL https://doi.orr 10. 485550//rrx.. 22099 07544
Qin Liu, Zihe Liu, Hongming Zhu, Hongfei Fan, Bowen Du, and Yu Qian. Generating commit messages from diffs using pointer-generator (eds.), Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019, 26- 27 May 2019, Montreal, Canada, pp. 399-309. IEEE / ACM, 2019c. doi: https: orgg////ooooo//////// 10. 1109/MSR..S.... 2019.00056
generator network. In Margaret-Anne D. Storey, Bram Adams, and Sonia Haidue
URL
Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongee Zhang, and Zhiyao Xie. Rtlcoder: Outperform- ing GPT-3.5 in design RTL generation with our open-source datasct and lightweight solution. CoRR, abs/ //////2222..21.. 2023h. doi: 10.48550/ 10.48550/ARXIV ARIII 23112.06617. URL hnttps://don wtt/:....////....... 48550/arXiv. 2312. 08617
Shangqing Liu, Cuiyun Gao, Sen Chen, Lun Yiu Nie, and Yang Liu. ATOM: commit message generation based on abstract syntax tree and hybrid ranking. IEEE Trans. Software Eng., 48(5):1800 18100 2022c. doi: 10.1109 11110099TTT/EE ASSE.2020.303882
URL httts:://doi.... orgo/.o.rgg000 10. 1109/TSE 2020. 3038681
Shangqing Liu, Xiaofei Xie, Jing Kai Siow, Lei Ma, Guozhu Meng, and Yang Liu. Graphsearchnet: Enhancing gnns via capturing global dependencies for semantic code search. IEEE Trans. Software Eng., 49(4):2833 2855, 2023i. doi: N0.1109/TTSB. 20S...2022.2333900 URL httpp::/doi.oor 1123310000000002222223311111
1109/TSE..E.. 202.//2..2222..... 3233901
Tianyang Liu, Canwen Xu, and Julian J. McAulcy. Repobench: Benchmarking repository-level code auto- completion systems. CoRR, abs/ 0330600309,, 2023j. doi: 00.48550/ARXI11 22066.03091. URL https:
4ttp////r.455/////x/ 2306 03091
03091
74

---

Published in Transactions on Machine Learning Research (09/2024)
Xiaoyu Liu, Jinu Jang, Neel Sundaresan, Miltiadis Allamanis, and Alexey Svyatkovskiy. Adaptivepaste: Code adaptation through learning semantics-aware variable usage representations. CoRR, abs/ abs/2205. 11023, 2022d. doi: a0.48500/arXiv
URL
2tt.i...2.....000 11023
Yilun Liu, Shimin Tao, Weibin Meng, Jingyu Wang, Wenbing Ma, Yanqing Zhao, Yuhang Chen, Hao Yang, Yanfei Jiang, and Xun Chen. Logprompt: Prompt engineering towards zero-shot and interpretable log analysis. CoRR, //////23000022 2023k. doi: 10.48550/ARXTY 2308.07610. URL https:///ot.
48550/arXiv 23.8.02002222
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1190.11692, 20.9d. URL http://arxiv http://aaxii..oogabbb/////e/
11692
Yudong Liu, Xu Zhang, Shilin He, Hongyu Zhang, Liqun Li, Yu Kang, Yong Xu, Minghua Ma, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. Uniparser: A unified log parser for heterogeneous log data. In Frédérique Laforest, Raphaäl Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (eds.), WWW '22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 29, 2022, pp. 1893-1901, ACM, 2022c. doi: 341145/448447.35//
URL https: http:///o...rg//////// 10. 1145/ 3511993
Yuliang Liu, Xiangru Tang, Zefan Cai, Junjic Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, Ruijun Huang, Shuzheng Si, Sheng Chen, Haozhe Zhao, Zhengliang Li, Liang Chen, Yiming Zong, Yan Wang, Tianyu Liu, Zhiwei Jiang, Baobao Chang, Yujia Qin, Wangchunshu Zhou, Yilun Zhao, Arman Cohan, and Mark Gerstein. MI-bench: Large language models leverage open-source libraries for machine learning tasks. CoRR, abs/2311 109835, 20231. doi: 10.48550/ 10.48550/ARXI ARXIV 2311.09883.
URL httttttttt//o....//. org/ 10. 48550/arXiv. 2311.09835
Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang. and Qing Wang. Chatting with GPT-3 for zero-shot human-like mobile automated GUI testing. CoRR, hls/2305.00433 2023m. doi: 10.48550/ARXTV/ 248500/ARRIV.22000004444. URL http::::/doo..rrgg/// 1r1..///0/./.....5////....
Zhijie Liu, Yutian Tang, Xiapu Luo, Yuming Zhou, and Liang Feng Zhang. No need to lift a finger anymore? assessing the quality of code generation by chatgpt. CoRR, abs/ /////2208...888 04838, 2023n. doi: 10.48550/ 48XI0//AXXIV. ARXIV. 2308.04 04888. URL https://dott org/10. 48550/ arXiv 2308.. 04838
Zhongxin Liu, Xin Xia, Ahmed E. Hassan David Lo, Zhenchang Xing, and Xinyu Wang. Neural-machine translation-based commit message generation: how far are we? In Marianne Huchard, Christian Kästner, and Gordon Fraser (eds.), Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018 pp. 373-384. ACM, 2018. URL https://doi.orrg http://.///111111328888388 114./32313333177 3238819
doi: 021145/3238147:33333
Fan Long and Martin C. Rinard. Automatic patch generation by learning correct code. In Rastislav Bodík and Rupak Majumdar (ods.), Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2016, St. Pelersbury, FL, USA, January 20 22, 2016, pp. 298-312. 312. ACM, 2016. doi: 10.1145/2833614422337 2837617
URL httpp:://dll org/10, 1145/28361//
Pablo Loyola, Edison Marrese- Taylor, and Yutaka Matsuo. A neural architecture for generating natural language descriptions from source code changes. In Regina Barzilay and Min-Yen Kan (ods.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancowver, Canada, July 30- August 4, Volume 2: Short Papers, pp. doi: 10.18653/ 10.18633/V1/P17- /V11117-220 URL hntp::tt///// http:///oo...go....././///////// 00t.11666/1//11-2000
Association for Compulational Linguistios, 2017.
Pablo Loyola, Edison Marrese-Tay Jorge A. Balazs, Yutaka Matsuo, and Fumiko Satoh. Content aware source code change description generation. In Emiel Krahmer, Albert Gatt, and Martijn Goudbeek (eds.), Proceedings of the 11th International Conference on Natural Language Generation, Tilburg University, The Netherlands, November 5-8, 2018, pp. 119-128, Association for Computational Linguistics, 2018. doi: 10. 8653/V1/W1186666/ URL http:/pss//o....og//o.oooooa wtt.///..0//.......o.......
75

---

Published in Transactions on Machine Learning Research (09/2024)
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhiar, Jiawei Tiu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen- Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Ni Osae Osac Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abali, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian J. McAuley, Han Hu, Torsten Scholak, Sébastien Paquet, Jen- nifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, and et al. Starcoder 2 and the stack v2: The next generation. CoRR, abs/2402.19173, 2024. doi: 10.48550/ARXIV 220.1........1...... URL https://dof org/ 10. 48550/arXiv. 2402. 19173
Junyi Lu, Lei Yu, Xiaojia Li, Li Yang, and Chun Zuo. Llama-reviewer: Advancing code review automation with large language models through parancter-offiient finc-tuning (practical experience report). CoRR. 0b88/2308.1111 2023a. doi: 118550/arXiv.... 20..2230111444 URL httttt://///ii..... http:////o.or///1... 411111...88///// 48550/arXiy arxiv. 2308 11148
Meili Lu, Xiaobing Sun, Shaowei Wang, David Lo, and Yucong Duan. Query expansion via wordnet for effective code search. In Yann- Gaël Guéhéneuc, Bram Adams, and Alexander Serebrenik (eds.), 22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2015, Montreal, QC, Canada, March 2-6, 2015, pp. 545-549. IEEE Computer Society, 2015. doi: 7081874. URL lhtts:://do/l http:::///oi..r//1.... 1109/ SANER. 7081874
Mengmeng Lu and Peng Liang. Automatic classification of non-functional requirements from augmented app user reviews. In Emilia Mendes, Steve Counsell, and Kai Petersen (eds.), Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE 2017, Karlskrona, Sweden, June 15- 15-16, 2017, pp. 444-553. ACM, 2017. doi: 011144/30042266308444 URL. hhttps://ooi. org/10. 1111101115/30842200 3084241
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyalkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tu- fano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundarcsan, Shao Kun Dong, Shengyu Fu, and Shu- jie Liu. Codexglue: A machine learning benchmark dataset for code understanding and genera- tion. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neurol Information Process- ing Systems Truck on Datasets and Benchmarks 1, NewrIPS Datasets and Benchmarks 2021, Decem- ber 2021, virtual, 2021. URL
wwwpodiinnnnnuuuis..... wwuurips.ccpppprr///e. aabssracctroundeee htmm
Siyang Lu, Xiang Wei, Yandong Li, and Liqiang Wang. Detecting anomaly in big data system logs using convolutional neural network. In 2018 IEEE 16th Inll Conf on Dependuble, Aulonomic and Secure Com- puting, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Dato Intelligence and Computing and Cyber Science and Technology Congress, 2018, Athens, Greece, August 12-15, 2018, pp. 151 158. IEEE Computer Society, 2018. doi: 10. 000037. URL httthhtts::///e org/10. wrt.://...r//.....//AAAS... 1109/DASC/
htt////CCoDDaaCooo DataCom/CyyberSciTTee /CyberSciTech
(Ttp://A///POO///AAAAAC/////ETTT PiCom/DataCom htlc://DDtacccmm/yyyeesSeeeeeeee 2018 hat.://mmmeeyeeesseetce..00....0
Yangyang Lu, Ge Li, Zelong Zhao, Linfeng Wen, and Zhi Jin. Learning to infer API mappings from API docu- ments. In Gang Tı, Yong Ge, Zili Zhang, Zhi Jin, and Michael Blumenstein (eds.), Knowledge Science, En- gineering and Management- 10th International Conference, KSEM 2017, Melbourne, VIC, Australia, Au- gust 19-20, 2017, Proceedings, volume 10412 of Lecture Noles in Compuler Science, pp. 237 248. Springer, 2017. doi: 10.1007 00007/977 /978-99-8-33 319-63555--3 20. URL attps:://oo/ wtt.://i.rrr1..............0...0 1ttp://10.100/97---33111------ 03559996355--33 3_05688-32000 20 httppp///oo....oo//o/00000011333
Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xic. RTLLM: an open-source benchmark for design RTL generation with large language model. CoRR, abs/2308. .034450555 2023b. doi: 00.48550/ARXI WWR.XI..23..05...00 22008.055...
URL httpts::tt.i.orrgpgeeoo 48550/ arxiv. 2308 2308.05345
76

---

Published in Transactions on Machine Learning Research (09/2024)
Yuwen Lu, Ziang Tong, Qinyi Zhao, Chengzhi Zhang, and Toby Jia-Jun Li. UI layout generation with Ilss guided by UI grammar. CoRR, abs/2310.116 15455, 2023c. doi: /ttps://aoi..or//10..
URL
2310. 15455
Sifci Luan, Di Yang. Celeste Barnaby, Koushik Scn, and Satish Chandra. Aroma: code recommendation via structural code search. Proc. ACM Program. Lang., 3(00PSLA)::5211 2019. doi: 10.1145/
3360578. URL htttps:/doi..rrg wtt..o.......1.0..0.771711 1145/3360778
Francesca Lucehetti and Arjun Guha. Activation stecring for robust type prediction in codellms. CoRR. abs/2444.01103,, 2024.
doi: 10.48500/ARXIV22401..0 2404.01903. URL http:::/ddo.orr//1// btgg/0..48550/rkk6.
2404.01903
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. CoRR, abs//2306005668, 2023. doi: 10.48550/arXin 230..0000.000..88.. URL httts://ddi.oor 0855000000000//////// arvvv 2306. 08568
Thibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li, Moshi Wei, and Lin Tan. Coconut: com- bining context-aware neural translation models using ensemble for program repair. In Sarfraz Khur- shid and Corina S. Pasareanu (eds.). ISSTA '20: 29th ACM SIGSOFT International Symposium Software Testing and Analysis, Virtual Event, USA, July 18-22, 2020, pp. 101 114. ACM, 2020. doi: 01545//335553333399
URL https:///oi. org/ 10. 1145/3395366. 3397369
Fei Lv, Hongyu Zhang, Jian-Guang Lou, Shaowei Wang, Dongmei Zhang, and Jianjun Zhao. Codehow: Ff- fective code search based on API understanding and extended boolean model (E). In Myra B. Cohen, Lars Grunske, and Michael Whalen (eds.), 30th IEEE/ ACM International Conference on Automated Software Engineering, ASE 2015, Lincoln, NE, USA, November 9-13, 2015, pp. 260 270. IEEE Computer Society, 10. 1109/ASE..... 2015.
2015. doi: 10.1109/ 201. URL
Qin Lyu, Kaushik Chakrabarti, Shobhit Hathi, Souvik Kundu, Jianwen Zhang, and Zheng Chen. Hybrid ranking network for text-to-sql. CoRR, abs hls/2008.04759, 2020. URL https://arxi: org/ abs//08.. 04759
Zeyang Ma, An Ran Chen, Dong Jac Kim, Tsc-Hsun Chen, and Shaowei Wang. Llmparser: An exploratory study using large language models for log parsing. In Proceedings of the 46th TEEE/ACM, International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pp. 99:1-991 99:13. URL httth::////o/ wwt/oo../../........ 10/010..1444 100.111533597503.. 3639150
ACM, 2024. doi: 10. 1155/3597503.36630
Marcos Macedo, Yuan Tian, Filipe Roseiro Cogo, and Bram Adams. Exploring the impact of the output format on the evaluation of large language models for code translation. CoRR, abs/ 22444.172111 2024. doi: 10.48550/ ARXIV 17214. URL attpps:/doi..or
00w.l/48550//// arwiv. 2403. 17214
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhunoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Ma- jumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iter- ative refinement with self-feedback. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: An- nual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http://papers nips ctwppper www.f/ileepppe 2.lle//paper/2223/mee 2lafffor2222bb5saaaaeeeefoooo wtw.://eeeeoeee-aaaaaaaa--oooooo Centtract-Conffrreee html
Fernanda Madeiral, Simon Urli, Marcelo de Almeida Maia, and Martin Monperrus. BEARS: java. bug benchmark for automatic program repair studies. In Xinyu Wang, David Lo, and Emad Shihab (ods.): 26th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2019, Hangzhou, China, February 24-27, 2019, pp. 468-478. IEEE, 2019. doi: 10.1109/SANER.2009
an extensible
8667991. URL https:///oii org/oo..//// 11tp://...000///11111 1111////E......... 2019.8667991
77

---

Published in Transactions on Machine Learning Research (09/2024)
Parvez Mahbub, Naz Zarreen Oishie, and S. M. Rafizul Haque. Authorship identification of source code segments written by multiple authors using stacking ensemble method. CoRR, abs/ hes/2222.06661) 2022. doi:
10.48550/arXiv. aivi wtw.i.2222.0561000 URL https:///oi..... org/ /0/ 48550/8rXiv. www.xiii.ii2......
Yubo Mai, Zhipeng Gao, Xing Hu, Lingfeng Bao, Yu Liu, and Jianling Sun. Are human rules necessary?
generating reusable apis with cot reasoning and in-context learning. 2024. URL attps://doi.orr/11
48550/arXiv. 2405.44550 03509
Rabee Sohail Malik, Jibesh Patra, and Michael Pradel. N12type: inferring javascript function types from natural language information. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (eds.), Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 304 315. IEEE / ACM, 2019. doi: 10.1099ICSSE:2209.00005 URL https://doiiorg/ 10. 1109/ICSE, 2019.000455
Aniketh Malyala, Katelyn Zhou, Baishakhi Ray, and Saikat Chakraborty. On ml- nal-based program translation: Perils and promises. In 45th TEFF/ACM International Conference on Software Engineering: New Ideas and Emerging Results, NIER NICS,, Melbourne, Australia, May 14-20, 2023, pp. 60 65. IEEE, 2023, doi: 2023.
N0.1199/CSE-NEER36688... NIEER58667.2223300000011 URL
2023.00017 00017
Zhenyu Mao, Jialong Li, Munan Li, and Kenji Tei. Multi-role consensus through Ilms discussions for vul- nerability detection. CoRR, abs/2403. 14274, 2024. doi: 01055///AARXXV.24..1142 URL https:
//doi.org/1i 48550/arXiv. 2403 14274
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaem- manouil, and Nesime Tatbul. Nco: learned query optimizer. Proc. VLDB Endow., 12(11):1005-1718, 2019. doi: 044777/3334223333222223 URL http:///wer www.. http://wwww.llbbborr///ooobbb//a http:////pr///ooooo////111000000
pdf
Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Alizadeh, and Tim Kraska. Bao: Learning to steer query optimizers. CoRR, abs/2004.03814, 2020. URL https://arxiy http//axxiivoooor///e abs/2044.
03814
Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Vader-Palacis Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. Studying the usage of text-to-text transfer transformer to support code- related tasks. In 43rd IEEE/ACM International Conference on Software Engineering, ICSE 2021, Madrid, Spain, 22-30 May 2021, pp. 336 347. IEEE, 2021. doi: N0.100//CCEE44302222.000000 1111:/011000//1139900000002 //doi.org//0......ooooo wtt.:////111.10//15883 htt.//SISE44392.220 20211 20041
URL https:
George Mathew and Kathryn T. Stolee. Cross-language code search using static and dynamic analyses. In Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (eds.), ESEC/FSE '11: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundalions of Software Engineering, Athens, Greece, August 23-28, 2021, pp. 217-217. ACM, 2021. doi: 10.1145/ 3468264.3465336 URL https:://oo....... org/10. 115....1115//46224444688380
Deep Mehta, Kartik Rawool, Subodh Gujar, and Bowen Xu. Automated devops pipeline generation for code repositories using large language models. CoRR, abs/ abs/2312. 13225 2023. doi: 10.48550/ ARXIV .2312.13225.
URL https:://ooi org/ 10. 48550/ 2312. 13225
Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, and Rong Zhou. Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs. In Sarit Kraus (ed.), Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 4739 4745. ijcai.org, 2019. doi: 10.24963/IJCA.. HTB.://ILAAAL119//////
URL https://doi.org// http///.....////aaaaaaooooo 2201 2019/668
Aditya Krishna Menon, Oner Tamuz, Sumit Gulwani, Butler W. Lampson, and Adam Kalai. A machine learning framework for programming by example. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, volume 28 of JMLR Workshop and Conference Proceedings, pp. 187-195. JMLR.org, 2013. URL http:://prooeedngs..lll. mlr. press/228/ menon13. .htll.
78

---

Published in Transactions on Machine Learning Research (09/2024)
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Mor- gane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christo- pher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchalskaya, Eric Ni, Eric Noland, Gong Yan, George Tucker, Ccorge-Chrrstian Muraru, Grigory Rozhdestvenskly, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean Baan-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, and et al. Gemma: Open models based on gemini research and technology. CoRR, abs/ 22//2244400299 2024. doi: 10.48550/ ARXIV M....2403.0000 08229. URL https:////o/ org/ http//10..855/// 115550/aXiiv 20w..4403..8800
Manel Mezghani, Juyeon Kang, and Florence Sèdes. Industrial requirements classification for redundancy and inconsistency detection in SEMIOS. In Guenther Ruhe, Walid Maalej, and Daniel Amyot (eds.), 26th IEEE International Requirements Engineering Conference, RE 2018, Banff, AB, Canada, August 20-24, 2018, pp. RE. 20t..211..0000 00037
IEEE Computer Society, 2018. doi: 10.1109 00.1109/RE.0010 RE.2001.00003, URL https:
//doi.org//i 1109/RE
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.nee net, 2018. URL /top::///opeerviee..
Amir M. Mir, Evaldas Latoskinas, and Georgios Gousios. Manytypes4py: A benchmark python dataset for machine learning- based type inference. In 18th IEEE/ACK International Conference Mining Soflware Repositories, MSR 2021, Madrid, Spain, May 17-19, 2021, pp. 585-589, IEEE, 2021. doi: 10.1109/ MSR52588.2022 2021-00079. URL https::///otl http:::///oo.or//1.... 1109 191109/2955888.... 2021.
Amir M. Mir, Evaldas Latoskinas, Sebastian Proksch, and Georgios Gousios. Type4py: Practical deep similarity carning-based type inference for python. In 44th TEEE/A IEEE/ACM 44th International Conference Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 2241-2252. ACM, 2022. doi:
URL https://doi org/ 10.
3510124
Facundo Molina, Marcelo d' Amorim, and Nazareno Aguirre. Fuzzing class specifications. In 44th TEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 1008-1020. ACM, 2022. doi: 10.1145/2510003.3511120 3510012
URL https:///oi. httpp:://oi.oor/10.. 1145/
Martin Monperrus. Automatic software repair: A bibliography. ACM Comput. Sure., 51(1):17:1 17:24, 2018 doi: 10.1145/3105906. URL http:///60...g/1110044144//31131
Martin Monperrus, Matias Martinez, He Ye, Fernanda Madeiral, Thomas Durieux, and Zhongxing Yu. Megadiff: A dataset of 600k java source code changes categorized by diff size. CoRR, abs/2108.00661. 04631, 2021. URL httt/http::/aat wttpaa/wwwwwgg//////////..o
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree structures for programming language processing. In Dale Schurmans and Michael P. Wellman (eds.), Proceedings of the
Thirtieth AAAT Conference on Artificial Intelligence, February 12-17, 2016, Phoenis, Arizona, USA, pp. 1287 1293. AAAI Press, 2016. doi: 10.1609 aaaaaaaavvvii. v0011.10139. URL https: /tops:///oi..oo ort/ 1609 /aaai. v3011. 10139
Fangwen Mu, Xiao Chen, Lin Shi, Song Wang, and Qing Wang. Developer-intent driven code comment generation. In 45th TEEE/ACM International Conference on Software Engineering, ICSE 2023, Mel- bourne, Australia, May 14-20, 2023, pp. 768-780. IEEE, 2023a. doi: 10.1109/CCSEA4899203..3 URL https://doi..or
79

---

Published in Transactions on Machine Learning Research (09/2024)
Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, Chenxue Wang. Shichao Liu, and Qing Wang. Clarifygpt: Empowering llm-based code generation with intention clarification. CoRR.
abs/2310..JB 10996, 2023b. doi:
22100..999.0.0099 URL https://doi. 02g/100.4850//xxix
10996)
Priyanka Mudgal and Rita H. Wouhaybi. An assessment of chatgpt on log data. CoRR, abs/ 000s/2309.07933 2023. doi: 10.48550/ARAN ARXIV 2309.07938. URL Inttps://dl..orr org/10. 48550/arXiv aarii..ivvvv.... 23.. 07938
Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Lougpre. Octopack: Instruction tuning code large language models. In The Twelfth International Conference Learning Representations, 2024. URL
https:///oeenrevie net/forumm witer/mmmmmmmmwwmm PWWSWZZ
Khurram Murad, Syed Noor-ul- Hassan Shirazi, Yousaf Bin Zikria, and Nassar Ikram. Evading virus detection using code obfuscation. In Tai-Hoon Kim, Young-Hoon Lee, Byeong Ho Kang, and Dominik Slezak (eds.), Future Generation Information Technology Second International Conference, FGIT 2010, Jeju Island, Korea, December 13-15, 2010. Proceedings, volume 6485 of Lecture Notes in Computer Science, pp. 394-401, Springer, 2010. doi: (0.10079978-3-442 17569-77769-521 39. URL https://0o1. http::////oo.rrg....111 otg/10.1007. 978-3-642-11569-53
Bardia Nadimi and Hao Zheng. A multi-cxpeert large language model architecture for verilog code generation. CoRR, abs/2404.000 24400000,9, 2024.
doi: 10.48550/ A (XTVV2204.00002. URL.
arX1v.2404.0
Kawser Wazed Nañi, Tonny Shekha Kar, Banani Roy, Chanchal K. Roy, and Kevin A. Schneider. CLCDSA: cross language code clone detection using syntactical features and APT documentation. In 34th TEEE/ACM International Conference on Automated Software Engineering, ASE 2019, San Diego, CA, USA, November /nttss//:///oo org/10. 1109/
11-15, 2019, pp. 1026-1037. IEEE, 2019. doi: AS..09//SE..19....0 2SE9.019.00 00099
Tasuku Nakagawa, Yoshiki Higo, and Shinji Kusumoto. NIL: large-scale detection of large-variance clones. In Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (eds.), ESEC/FSE '11 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, pp. 830 841. ACM, 2021. doi: 10.1145/ 3468264.3468664. URL https: http::////oo.or/1...../ /144/ 3468564
Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. Enhancing few-shot text-to-sql capabilities of large language models: A study on prompt design strategies. CoRR, abs/ //////235..5555 2023. doi: 10.48550/ aarvii Stt.i..305.125555 URL https://doi. oarei/oor//11 10.
48550/arKi a005v./rxxiv.... 2305. 12586
Sasho Nedelkoski, Jasmin Bogatinovski, Alexander Acker, Jorge Cardoso, and Odej Kao. Self-supervised log parsing. In Yuxiao Dong, Dunja Mladenic, and Craig Saunders (eds.), Machine Learning and Knowledge Discovery in Databases: Applied Data Science Track European Conference, ECML PKDD 2020, Ghenl, Belgium, Seplember 14-18, 2020, Proceedings, Parl IV, volume 12460 of Lecture Noles in Computer Science, pp. 122 138. Springer, 2020a. doi: 10.1007/978. 1-10007977---000 000000060-7--55 8. URL https: //doi.org// 10. 10771178------00-6755555
Sasho Nedelkoski, Jasmin Bogatinovski, Alexander Acker, Jorge Cardoso, and Odej Kao. Self-attentive classification-baseed anomaly detection in unstructured logs. In Claudia Plant, Haixun Wang, Al- fredo Cuzzocrea, Carlo Zaniolo, and Xindong Wu (cds.), 20th IEEE International Conference on Data Mining, ICDM 2020, Sorrento, Tlaly, November 17-20, 2020, pp. 1196 1201. IEEE, 2020b. doi: 10.1109/CCDM50000...O W0W.:/50100.220.0011111 2020.00148. URL httph::////o/ org/10. 10. 1119/11I00000000000 200.://..00000...00.....2 2020. 00148
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N. Nguyen. Lexical statistical machine translation for language migration. In Bertrand Meyer, Luciano Baresi, and Mira Mezini (eds.). Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE'13, Saint Petersburg, Russian Federation, August 26, 2013, pp. 651 654. ACM, 2013. doi: 10. 10.1145/2491411 2494584. URL hsttss///obi..rrr//..114///4442 24944444
80

---

Trong Due Nguyen, Anh Tuan Nguyen, and Tien N. Nguyen. Mapping API elements for code migration with vector representations. In Laura K. Dillon, Willem Visser, and Laurie A. Williams (eds.), Proceedings of the 38th International Conference on Software Engineering, 1CSE 2016, Austin, TX, USA, May 14- 22, 2016 Companion Volume, pp. 756-758. ACM, 2016. doi: 0111452288116028922222 Published in Transactions on Machine Learning Research (09/2024)
Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N. Nguyen. Divide-and-conquer approach for multi-phase statistical migration for source code (T). In Myra B. Cohen, Lars Grunske, and Michael Whalen (eds.); 30th IEEE/ACM CM International Conference on Automated Software Engineering, ASE 2015, Lincoln, NE, USA, November 9-13, 2015, pp. 585-596. IEEE Computer Society, 2015. doi: 10.1109/ 111109/AEE2011..777 URL https://doi. org/ 10. 1109 ASE. 2015.74
Hoan Anh Nguyen, Tien N. Nguyen, Danny Dig, Son Nguyen, Hieu Tran, and Michael Hilton. Graph-based mining of in-the-wild, fine-grained, semantic code change patterns. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (eds.), Proceedings of the 41st International Conference on Software Engineering. ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 819 830. IEFF/ / ACM, 2019. doi: 10. N0.1009/CCSF.2219000085 URL attps://ddo org/10. 1101 (Ww.0/IICSE.200 2019
Nhan Nguyen and Sarah Nadi. An empirical evaluation of github copilot's code suggestions. In 19th TEFE/ACH / ACM International Conference on Mining Software Reposilories, MSR 2022, Pillsburyh, PA, USA, May 23-24, 2022, pp. 1-5. ACM, 2022. doi: 10.1145 2..145/3522844.35288888 URL https://dol.. 11445// 3524842 3528477
Son Nguyen, Hung Phan, Trinh Le, and Tien N. Nguyen. Suggesting natural method names to check name consistencies. In Gregg Rothermel and Doo-Hwan Bae (eds.), ICSE '20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June - 19 July, 2020, pp. 1372 1384. ACM, 2020, doi: URL https:///oi. org//o..///./.o/ 10. 1145/3377811 3338000 3380926
10.1145/33778.1 01.1145//877811.338995 3338096...
Thieu Nguyen, Satoru Kobayashi, and Kensuke Fukuda. Logdtl: Network log template generation with deep transfer learning. In Toufik Ahmed, Olivier Festor, Yacine Ghamri-Doudane, Joon-Myung Kang, Alberto E. Schaeffer Filho, Abdelkader Lahmadi, and Edmundo R. M. Madeira (eds.), 17th IFIP IFIP/IEEE International Symposium on Integnated Network Monagement, IM 2021, Bordeaur, France, May 17-21, 2021, pp. 348-853. TEEE, 2021. URL
leee. wwggddccument/944406
URL https:
7/doi 1////00.114/28889160 2892661
Van-Anh Nguyen, Dai Quoe Nguyen, Van Nguyen, Trung Le, Quan Hung Tran, and Dinh Phung. Regvd: Revisiting graph neural networks for vulnerability detection. In 44th TEEE/ACM International Conference on Software Engineering: Companion Proceedings, ICSE Companion 2022, Pittsburgh, PA, USA, May 22- URL https://dooooor//1//
24, 2022, pp. 178 182. ACM/IEEF, 2022. doi: 0411145/31104443333 11 1145/3510454 3516865
Daniel Nichols, Joshua Hoke Davis, Zhaojun Xie, Arjun Rajaram, and Abhinav Bhatele. Can large language models write parallel code? CoRR, abs/ /2////2111.155.. 2024. doi: D0.4850/ARXIV..401..20 URL
http::://aii.org11...44558
12554
Lun Yiu Nie, Cuiyun Gao, Zhicong Zhong, Wai Lam, Yang Liu, and Zenglin Xu. Coregen: Contextualized code representation learning for commit message generation. Neurocomputing, 459:97-107, 2021. doi: 10.1016/L.NEUUCMM.2P22 20..000000021000...... URL https::/doii ort./0.1006//3 neucom. 2021. 05.039
Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J. Mooney, and Milos Gligoric. Learning deep semantics for test completion. In 45th TEEE/ACM International Conference on Software Engincering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 2111 2123. IEEE, 2023. doi: 10.1109/ICSE48619. 1109 1109/ICSE48619 2023. 00178
2023.00178. URL hattps://doi..oo
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savareso, and Yingbo Zhou. Codegen2: Lessons for training Iims on programming and natural languages. CoRR, abs/ 22222333000000 2023a. doi: 10.48550/ ARXTV TRXTV.2305.022 0230.0 URL htttps://oo..rg/// 48550//r0..455 wtte:xiv..305.222 02309
81

---

Published in Transactions on Machine Learning Research (09/2024)
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, JCLR 2023, Kigali, Rwenda, May 1-5, 2023. OpenReview.net, 2023b. URL httpt:t//popprreeiieeee
Georgios Nikilopoulos, Konstantina Dritsa, Panos Louridas, and Dimitris Mitropoulos. Crossvul: a cross- language vulnerability dataset with commit data. In Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (eds.), ESEC/FSE '21 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, URL. https://doi. ottg//..1///000
2021, pp. 1565-1500 1569. ACM, 2021. doi: (0.1145/3408264..11 0118264.347312
Vikram Nitin, Anthony Saieva, Baishakhi Ray, and Gail Kaiser. DIRECT A transformer- model for decompiled identifier renaming. In Royi Lachny, Ziyu Yao, Greg Durrett, Milos Gligoric, Junyi Jessy Li, Ray Mooney, Graham Neubig, Yu Su, Huan Sun, and Reut Tsarfaty (eds.), Proceedings of the 1st
Workshop on Natural Language Processing for Programeingg (NLP4Prog 2021), pp. 48 57, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ /0////////222111// nlp4prog- mlpppoo 1.6. URL https:// n1p4prog-1.6
Changan Niu, Chuanyi Li, Vincent Ng, Jidong Ge, Liguo Huang, and Bin Luo. Spt-code: Sequence-to- sequence pre-training for learning source code representations. In 44th IEEE/ACEE AAMM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 1-13. ACM, URL htttp::///o// hrg:///tt..or/////// 10. 1144/3511000000 3510096.
2022. doi: 10.1145/ 141145/3510003 11111000333110000
Changan Niu, Chuanyi Li, Vincent Ng, Dongxiao Chen, Jidong Ge, and Bin Luo. An empirical comparison of pre-trained models of source code. In 45th IEEE/A ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 2023.00180. URL https: org/10. 10. 11w.////00//119....
IEEE, 2023. doi: 10. 10.1109/TCSE486119
00180
Changan Niu, Ting Zhang, Chuanyi Li, Bin Luo, and Vincent Ng. On evaluating the efficiency of source code generated by lims. CoRR, abs/ 2204..0044, 2024. doi: 10.48550/ /0800//RRVV.... 24040044 URL https:
//doi.org/11 org/10 10. 38850//axiv, 2404.0440000 06041
Yu Nong, Rainy Sharma, Abdelwahab Hamou-Lhadj. Xiapu Luo, and Haipeng Cai. Open science in software engineering: A study on deep learning-barte based vulnerability detection. IEEE Trans. Software Eng., 49(4), 1983-2005, 2023.
doi: (0.1109/TSE..202...0 320714....... URL BtTTLhhtps::/// http::///ooo...///././o/// wttg//..11//////.. 2022. 149.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Ailor Lewkowycz, Maarlen Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. CoRR, abs/2112.00114, 2021. URL http:::/aarxiv.or///s///1122111
Augustus Odena, Catherine Olsson, David G. Andersen, and Ian J. Goodfellow. Tensorfuzz: Debugging neural networks with coverage- guided fuzzing. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, TCML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 4901-4911, PMLR, 2019. URL htttp:://proceedingss .ml.eeedddinnme...ee........ mlr, wwweepressvvveee http://////dddaaaa/ee html
Saeyoon Oh and Shin Yoo. Csa-trans: Code structure aware transformer for ast. CoRR, abs/2444.05567, 2024. doi: N0.48550/ARXK0 WWA.RIII....4.0766660 2404.05767. URL https://doi..rgg http:///rr//.....///rrr../// a005550/arXiv.. 2404. 05767
Wonseok Oh and Hakjoo Oh. Pyter: ellective program repair for python type errors. In Abhik Roychoud- hury, Cristian Cadar, and Miryung Kim (eds.), Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, pp. 922 922-934. ACM, 2022. doi: 10.1145/3544250... URL httpps://aai.orr//001111//////// 354110000/0
82

---

Published in Transactions on Machine Learning Research (09/2024)
Milos Ojdanic, Aayush Garg, Ahmed Khanfir, Renzo Degiovanni, Mike Papadakis, and Yves Le Traon. Syn- tactic vs. semantic similarity of artificial and real faults in mutation testing stuies. CoRR, abs/21 21 12. 14508, 2021. URL http://////riiioooaaa///////////
14508
Milos Ojdanic, Ahmed Khanfir, Aayush Garg, Renzo Degiovanni, Mike Papadakis, and Yves Le Traon. On comparing mutation testing tools through learning- bbasig mulant selection. In IEEE/A / International Conference on Automation of Software Test, AST 2023, Melbourne, Australia, May 15- 16, 2023, pp. 35- 46. TEEE, 2023. doi: 10.1109 A0011//AST189952223..000
URL httpp:://aol.orrg/...10///////1/
00008
OpenAL. GPT-4 technical report. CoRR, abs abs/2333.08874, 2023. doi: 10.48550//r211:22333322
URL
https://doi... org/ 10/ 48550/85555A3 arliv. http://vvvvv......... 2303.08774
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with hu- nips. cctpp.cc//aprrr.... f1les/pppeer/222/bbaaee html
man feedback. In NewrIPS, 2022. URL blerasbbedd4777394488666660
Rangect Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. Understanding the effectiveness of large language models in code translation. CoRR, //////23.0000000 2023. doi: 10.48550/
ARXIV 2308..390.. URL htaps://ddi.org////
48850/aaXiv.... arviv. ww0.............. 2008 03109
Xinglu Pan, Chenxiao Liu, Yanzhen Zou, Tao Xie, and Bing Xie. MESIA: understanding and leveraging sup- plementary nature of method-level comments for automatic comment generation. CoRR, abs 0oos/2403.1170 17357, 2024. doi: 10.48550/ ARXIV S.A.KIV..403.110 2403.17357. URL https: /ttt:::///oo.orr/10... wttp://../.......///./ 48550/arXiv.. 2403.17355
Irene Vlassi Pandi, Earl T. Barr, Andrew D. Gordon, and Charles Sutton. Opttyper: Probabilistic type inference by optimising logical and natural constraints. CoRR, abs/2004.00348, 2020. URL https:
wwwpeiivorg/abb///// 00348
Annibale Panichella, Fitsum Meshesha Kifetew, and Paolo Tonella. Automated test generation a many-objective optimisation problem with dynamic selection of the targets. IEEE Trans. Software
Eng., 44(2):122 158, 2018. doi: 10.1109/TSE.200 201///S...2212222..2 26664363... URL httpt:://oo:iii http://oo..ogg///// 1tte//0..111///11 110.0..../TTTE.20 1000://0///.....1TT....... 2017. 2663435
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311-318. ACL, 2002. doi: 10.3115/1073083
1073135. URL https:// org/po2-1000))
Nikhil Parasaram, Hujjie Yan, Boyu Yang, Zineb Flahy, Abriele Qudsi, Damian Ziaber, Earl Barr, and Sergey Mechtacv. The fact selection problem in Ilm-based program repair. CoRR, fabbs//444005555 2024.
doi: 10.48550/ARRA ARXIV 2404.05520. URL https: //do: wwt..o..//////////////. 48550/arXiv 2404.05520
Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli, Neuro-symbolic program synthesis. Tn 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenKeview..ne net, 2017. URL
https://openreviiet wttpr//reeeeeeeeeeoooorrrrttrde.
Md. Rizwan Parvez, Saikat Chakraborly, Baishakhi Ray, and Kai-Wei Chang. Building language models for text with named entities. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 2373 2383. Association for Computational Linguistics, 2018. doi: 10. N0.16633/1/PP8-122 URL http::://cclattholloyy http:/athhhooooooooe--ee---11 org//18-111111
83

---

Published in Transactions on Machine Learning Research (09/2024)
Md. Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Wei Chang. Retrieval augmented code generation and summarization. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pp. 2719-2734. Association for Computational Linguistics, 2021. doi: 10. 10.18653/V1 /11//221...IDIIGG 233
URL https://doi.
Jibesh Paira and Michael Pradel. Semantic bug seeding: a learning- approach for creating realistic bugs. In Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (eds.), ESEC/FSE 21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, pp. 906-918. ACM, 2021. doi: 10.1145/
3468264.3466623, URL htttps://dol org/10. 10. 101111144444888 3468264.8666444. 3466644442244443 3468623
Rishov Paul, Md. Mohib Hossain, Masum Hasan, and Anindya lqbal. Automated program repair based on code review: How do pre-trained transformer models perform? CoRR, abs/230 mns/2304.07840, 2023. doi: 10.48550/ ARXIV 2304.07840. URL htttps://doi.orgg/ 488.504555///aix 230. 200.2378077000
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the keyboard? assossing the security of github copilot's ' code contributions. In 43rd IEEE Symposium on Security and Privacy, SP 2022, Son Francisco, CA, USA, May 22-26, 2022, pp. 754-768. IEEE, 2022. doi: 22110//SP4424..2222..o02 283337777 URL httpss///di. . org/10. 1109//9469911 2022. 9833571
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon LLM: outperforming curated corpora, with web data, and web data only, CoRR, 0bss/2306.011/1 2023. doi:
116. URL htttps://doi.rr/// w0w0///....55//r///// aar.v//rrvvvv... 2306.001101
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Fffficient context window extension of large language models. CoRR, abs/2309.000 22///2339000700 2023a. doi: 10.48550/ARXIV.22505 230000000.....0000...... URL https:
48550/4555iii arviv.iiiiiiivi. 2309 00071
Yun Peng, Cuiyun Gao, Zongjie Li, Bowei Gao, David Lo, Qirun Zhang, and Michael R. Lyu. Static inference meets deep learning: A hybrid type inference approach for python. In 44th JEEE/A ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 2019 2030. URL https:://oo...... org/ 10. 1145/3510003 3510038
ACM, 2022. doi: 0..1145/3510008.111
Yun Peng, Shuzheng Gao, Cuiyun Gao, Yintong Huo, and Michael R. Lyu. Domain knowledge matters: Improving prompts with fix templates for repairing python type errors. CoRR, abs/ abs/2306. 23000013944 2023b.
doi: 00.4850/AARXI 2306.01394. URL https: /tttpp::/doo..org/10.. org/10. httg//..8550////// 48550/arXiv arxiv 2306 01394
Yun Peng, Chaozheng Wang, Wenxuan Wang, Cuiyun Gao, and Michael R. Lyu. Generative type inference for python. CoRR, abs/ /alp//237.000 09166,,,,......... 2023c. doi: 10.48550/ARXTV ARXIV RERXIVV2307.0000
URL https://doi. org/
10. 485500arXiii 2307-09163
Daniel Porez and Shigeru Chiba. Cross-languaae clone detection by learning over abstract syntax trees. In Margaret-Anne D. Storey, Bram Adams, and Sonia Haiduc (eds.), Proceedings of the 16th International Conference Mining Software Repositories, MSR 2019, 26-27 May 2019, Montreal, Canada, pp. 518 528, IEEE/ ACM, 2019. doi:
/ttp://SS...1..007.7.7. URL htttp:://dst orgoai..rr//t/o htrg/10.110 1...9.19/MMRR 2019. 2019.00078 00078
Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. Do users write more insecure code with AI assistants? In Weizhi Meng. Christian Damsgaard Jensen, Cas Cremers, and Engin Kirda (cds.), Proceedings of the 2023 ACM SIGSAC Conference on Compuler and Commmniccatinns Security, CCS 2023, Copenhagen, Denmark, November 26-30, 2023, pp. 2785-778220 2799. ACM, 2023. doi: 10.145/3556611533223177 URL httph:://ddot org/10..000 10. 0114445/5776915362222
Jannik Pewny and Thorsten Holz. Evilcoder: automated bug insertion. In Stephen Schwab, William K. Robertson, and Davide Balzarotti (ods.), Proceedings of the 32nd Annual Conference on Computer Security Applications, ACSAC 2016, Los Angeles, CA, USA, December 5-9, 2016, pp. 214-225. ACM, 2016. URL
http://dl. acm. http://itatt.o...mmaaa2991111111
84

---

Published in Transactions on Machine Learning Research (09/2024)
Hung Dang Phan, Anh Tuan Nguyen, Trong Duc Nguyen, and Tien N. Nguyen. Statistical migration of API usages. In Sebastián Uchitel, Alessandro Orso, and Martin P. Robillard (eds.), Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017 - Companion Volume, pp. 47-50. IEEE Computer Society, 2017. doi: 10.1109/TCSE-C22017.. URL
https://doi. 10. 1119//1/E-/-C200000 2017. .17
Huy N. Phan, Hoang N. Phan, Tien N. Nguyen, and Nghi D. Q. Bui. Repohyper: Better context retricval is all you need for repository-level code completion. CoRR, abs 0bs//240330000 2024. doi: 10.48550/ARB ARXIV.
2403.06095. URL stttps://dol. org/10. 48550/855k5 arxiv 06095)
Juan Altmayer Pizzorno and Emery D. Berger. Coverup: Coverage-guided Ilm-based test generation. CoRR, abs 1122221311221,, 2024.
doi: 10.48550/ARAA ARXIV 1401..6218.
URL https: / //ooi org/10, 10/ 48550/arXiv.
2403. 16218
Serena Elisa Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi, and Cédric Dangremont. A manually- curated dataset of fixes to vulnerabilities of open-source software. In Margarct-Anne D. Storey, Bram Adams, and Sonia Haidue (eds.), Proceedings of the 16th International Conference on Mining Soflware Repositories, MSR 2019, 26-27 May 2019, Montreal, Canada, pp. 383-387. IEEE / ACM, 2019. doi: 200.:09/MMS..1999.00000 URL https: wt...o.....//......... 201..0000000 00064
Mohammadreza Pourreza and Davood Rafiei. DIN-SQL: decomposed in-context learning of text-to-sql with self-correction. CoRR, /tt//2304.1101,,, 2023. doi: 10.4850/AARXIV22200.100 URL http::ttt///ee org/ 10. 10.48550/arXiv. 2304.
Michael Pradel and Koushik Sen. Deepbugs: a learning approach to name-based bug detection. Proc. ACM Program. Lang., 2(00PSLA))147:1 147:25, 2018. doi: 10.1145/3276517. URL https://doi..on org/10.
1145/3276517
Michael Pradel, Parker Schuh, and Koushik Sen. Typedevil: Dynamic type inconsistency analysis for javascript. In Antonia Bertolino, Gerardo Canfora, and Sebastian G. Elbaum (eds.), 37th IEEE/ACM International Conference on Software Engineering, ICSE 2015, Florence, Raly, May 16-24, 2015, Volume 1, pp. 314 324. IEEE Computer Society, 2015. doi: (0.1109/CCSE:201555. URL https://doi. org///00...r//.... 1109/ICSE, 20
Michael Pradel, Georgios Gousios, Jason Liu, and Satish Chandra. Typewriter: neural type prediction with search-based validation. In Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (eds.), ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, pp. 209 220. ACM, 2020. doi: 10.1145/ 3368089.3409715 URL https::/dooi http:///oo.ooo//..../ 1145/3368089..... 3409715
Julian Aron Prenner and Romain Robbes. Runbugrun - an executable dataset for automated program repair. CORR, abs/ /304..0000 2023. doi: 10.48550/ 1..8555//AXII1.0 W.R.IV.2304.01101 URL attps://doi.org/:
48550/arXiv 2304 01102
Ofir Press, Noah A. Smith, and Mike Lowis. Train short., test long: Attention with lincar biases enables input length extrapolation. In The Tenth International Conference on Learning Representations, JCLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nee net, 2022. URL
net/forum?
id=R8sQPPGCvO
Yewen Pu, Karthik Narasimhan, Armando Solar-Lezama, and Regina Barzilay, sk p: a neural program corrector for mooes. In Eclco Visser (cd.), Companion Proceedings of the 2016 ACM SIGPLAN In- lernational Conference on Systems, Programming, Languages and Applications: Software for Humanily, SPLASH 2016, Amsterdam, Netherlands, October 30 November 4, 2016, pp. 39-40. ACM, 2016. doi: 00.1145/2984043.2988888
URL https://doi. org/ooo.///oo 10.1 www.i1/////44..2222 2989222
Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolo- tov, Julian Dolby, Jie Chen, Mihir R. Choudhury, Lindscy Docker, Veronika Thost, Luca Bu- ratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Co- denet: A large-scale AT for code dataset for learning diversity of coding tasks. In Joaquin
85

---

Published in Transactions on Machine Learning Research (09/2024)
Vanschoren and Sai-Kit Young (eds.), Proceedings of the Newral Information Processing Systems Track on Dalasets and Benchmarks 1, NewrIPS Dalasels and Benchmarks 2021, December 2021, virtual, 2021. CRL https:/ benchmarks-proceeee Abstrsad----tttttttoao-aa-e Sbbrrao0755.aaaa055..4446666-663
html
Jiaxing Qi, Shaohan Huang, Zhongzhi Luan, Carol J. Fung, Hailong Yang, and Depei Qian. Loggpt: Ex- ploring chatgpt for log-based based anomaly detection. CoRR, abs/2309) 0t//2239.000 2309.00189, 2023. doi: 2309.01188 URL httpt:://ts..ooooooo 3rg/10.48550////// arXiv. 2309.01189)
Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. CoRR, abs/ 0t//2307.079922 2023. doi: 10.48550/ 148550/AAXIV.. ARXIV. 2307.07924. URL https://doi.org wtt.......0////...... 2307 2307.07922 .79244
Bowen Qin, Binyuan Hui, Lihan Wang, Min Yang, Jinyang Li, Binhua Li, Ruiying Geng, Rongyu Cao, Jian Sun, Luo Si, Fei Huang, and Yongbin Li. A survey on text-to-sql parsing: Concepts, methods, and future directions. CoRR, abs/ 13629, 2022a. doi: 10.48550/ARXIV www. 13629. URL https:
48550/arXiv. 2208 13629
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong, cosformer: Rethinking softmax in attention. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022b. URL
Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract syntax networks for code generation and semantic parsing. In Regina Barzilay and Min-Yen Kan (eds.), , Proceedings of the 55th Anrvual Meeling of the Association for Computational Linguistics, ACI 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Pupers, pp. 1139 1149. Association for Computational Linguistics, 2017. doi: 10.18653/ V1/P17-1105. URL https:://oo: 10/5//0..188///1//11110000
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAT 7 blog, 1(8):9, 2019.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Holfmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Trina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Frich Fllen, Siddhant M. Jayaku- mar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Lau- rent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autune, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, lason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukeuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/2112. 11446, 2021. URL https://arxxvv.orr/bb///// 11446
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. 2020. URL http:://mil...orggppppss///////p
J. Mach. Learn. Res.,
Abir Rahali and Moulay A. Akhloufi. Malbertv2: Code aware bert-basod model for malware identification. Big Data Cogn. Comput., 7(2):60, 2023. doi: BDCC300/DCC0000000. URL bdcc7020060
86

---

Published in Transactions on Machine Learning Research (09/2024)
Md Mahbubur Rahman, Ira Ceka, Chengzhi Mao, Saikat Chakraborty, Baishakhi Ray, and Wei Le. Towards causal deep learning for vulnerability detection. CoRR, httss/2310007955 2023. doi: 10.48550/ ARXTV.2310. 07958. URL https://doi..... http///o..or///////// org/10. 48550/arXiv....a///// 2ttp:/.......2..7055113 2310.07958
Baishakhi Ray, Vincent J. Hellendoorn, Saheel Godhane, Zhaopeng Tu, Alberto Bacchelli, and Premkumar T. Devanbu. On the 'naturalness" of buggy code. In Laura K. Dillon, Willem Visser, and Laurie A. Williams (ods.), Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin,
TX, USA, May 14-22, 2016, pp. 428 439. ACM, 2016. doi: 10.1145/2884781..88888 28844888.
URL https:
//doi //doi.org/10 00111/22887888 2884848
Veselin Raychev, Martin T. Vechev, and Eran Yahav. Code completion with statistical language models. In Michael F. P. O'Boyle and Keshav Pingali (eds.), ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '14, Edinburgh, United Kingdom June 09- 11, 2014, pp. 419-428. ACM, 2014. doi: 10.1145, 201141/2592291.0 2599432.. URL https://doi. org ///g//11111 1t///10.111555 1t..111//2594999 254449922559999999331111 2594321
Veselin Raychev, Martin T. Vechev, and Andreas Krause. Predicting program properties from "big code". In Sriram K. Rajamani and David Walker (eds.), Proceedings of the 42nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2015, Mumbai, India, January 15-17, 2015, pp. 111-124. ACM, 2015. doi: 10.1145 011145/267672626677000 URL https:///bl..rr/11..1111 111/////..14///7777... 2677009
Veselin Raychev, Pavol Bielik, and Martin T. Vechev. Probabilistic model for code with decision trees. Eelco Visser and Yannis Smaragdakis (eds.), Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, part of SPLASH 2016, Amsterdam, The Netherlands, October 30- November 4, 2016, pp. 731 747. ACM, 2016a. doi: 10.1145 10.1145/2983990. 2983999029844411. 2984041. URL https::/doi.orgg: 00w..118399911...110 2984041
Veselin Raychev, Pavol Bielik, Martin T. Vechev, and Andreas Krause. Learning programs from noisy data. In Rastislav Bodík and Rupak Majumdar (eds.), Proccedings of the 43rd Annual ACM SIGPLAN-SIGACT
Symposium on Principles of Progrumming Languages, POPL 2016, St. Pelersburg, FL, USA, Janwary 20 -22, 2016, pp. 2837614 2837671
ACM, 2016b. doi: 10.1455/937614423377711. URL https: pettp://aco.oor/100 /trg//..115/
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Am- brosio Blanco, and Shuai Ma. Codebleu: a mothod for automatic evaluation of code synthesis. CoRR, abs/2009.1022 2020. URL
10297
Cedric Richter and Heike Wehrheim. TSSB-3M: mining single statement bugs at massive scale. In 19th IEEE/ACM International Conference on Mining Software Repositories, MSR 2022, Pillsburyh, PA, USA, May 23-24, 2022, pp. 418 422. ACM, 2022. doi: 10.1145/352484223558888 URL https://doi..rg/// 11 1145/3524842 3528505
Niklas Risse and Marcel Böhme. Limits of machine learning for automatic vulnerability detection. CoRR, abs/2306.17190 17193, 2023. doi: 00.8550/ARRXTI Rt5.://AAAXX...0..7799909 URL https://doi.org// 40t.:///...05////xv.v..... 2306.17193
Marcel Robeer, Garm Lucassen, Jan Martijn E. M. van der Werf, Fabiano Dalpiaz, and Sjaak Brinkkemper. Automated extraction of conceptual models from user stories via NLP. In 24th IEEE International Re- quirements Engineering Conference, RE 2016, Beijing, China, September 12-16, 2016, pp. 196-205. IEEE Computer Society, 2016. doi: F..10//FF.216..... 40. URL httth:://///// orgo 10. 1109/R0..R...1 2016.40
Alex Robinson. Sketch2code: Generating a website from a paper mockup. CoRR, abs/ 1111119001111 2019.
URL http://araiv.rrr//////00....13 13750
Alberto D. Rodriguez, Katherine R. Dearstyne, and Jane Cleland-Huang. Prompts matter: Insights and strategies for prompt engineering in automated software traceability. In Kurt Schneider, Fabiano Dalpiaz, and Jennifer Horkoff (ods.), 31st IEEE International Requirements Engineering Conference, RE 2023- Workshops, Hannover, Germany, September 4-5, 2023, pp. 455 455-464. 464. IEEE, 2023. doi: 2023.00087. Utt.:ttte////aaa..o////0/11/1/11
87

---

Published in Transactions on Machine Learning Research (09/2024)
Krishna Ronanki, Beatriz Cabrero Daniel, Jennifer Horkoff, and Christian Berger. Requirements engineering using generative AI: prompts and prompting patterns. CoRR, abs/2311 221///311.033,, 2023. doi: 10.48550/ ARXIV 2AXIVV2311.038322 URL
wtt.::i...rg/...855500. 2ttp:///...ii............ 2311 03832
Subhajit Roy, Awanish Pandey, Brendan Dolan-Gavitl, and Yu Hu. Bug synthesis: challenging bug-finding tools with deep faults. In Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareamu (eds.), Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Visla, FL, USA, November 04-09, 2018, pp. 224 234.. ACM, 2018. doi: 3236024. 3236084
3236084 URL httph:://p/oo http:///oo..o/////// 10. 1145/
Bapliste Rozière, Marie-Anne Lachaux, Lowik Chanussol, and Guillaume Lample. Unsupervised translation of programming languages. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (ods.), Advances in Newral Information Processing Systems 33: Annual Conference on Neurul Information Processing Systems 2020, NearIPS 2020, December 6-12, 2020, virtual, 2020. URL Abstract. html
hwwpeediiinnnnuruipsp.ee cwtpppppecccppper/////eee /aahh/
Baptiste Rozière, Jie Zhang, François Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. Leveraging automated unit tests for unsupervised code translation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://oopenrevieet http://rww.....eeeooooo///ttttte
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Ttai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. CoRR, abs/2308 12950, 2023. doi: (115550/rrXiv2255 URL attpp:///oli org/10. 48550/arXiv. 2308 12950
Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Xingyu Zeng, and Rui Zhao. TPTU: task planning and tool usage of large language modellbased Al doi: 106550/ARXI.23000
agents. CoRR, abs/ (bss/230.000 2308.03427, 2023.
URL http:::/doii.or//1//
48550/arXis 2308. 03427
Ohad Rubin and Jonathan Bcrant. Smbop: Semi-autoregressive bottom-up somantic parsing- In Kristina Toutanova, Anna Rumshisky, Luke Zeitlemoyer, Dilek Hakkani-Tüt, Iv Beltagy, Sleven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Humon Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 311-324. Association for Computational Linguistics, URL https://doi. wtrr//...1653/////
2021. doi: 10.18653/v1 www.......llmmminn...
maacl-main
29
Rebecca L. Russell, Louis Y. Kim, Lei H. Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul M. Ellingwood, and Mare W. McConley. Automated vulnerability detection in source code using deep representation learning. In M. Arif Wani, Mehmed M. Kantardzic, Moamar Sayed Mouchaweh, João Gama, and Edwin Lughofer (ods.), 17th IEEE International Conference on Machine Learning and Ap- plications, TCMLA 2018, Orlando, FL, USA, December 17-20, 2018, pp. 757-762. TEFF, 2018. doi: 2018.00120.
10L109/IIMM..000 2011.. .01220. URL https:///ot http////iooog//...//// 11w.11109IMMMA 2018
Saksham Sachdev, Hongyu Li, Sifei Luan, Seohyun Kim, Koushik Sen, and Satish Chandra. Retrieval on source code: a neural code search. In Justin Gottschlich and Alvin Cheung (eds.), Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL@PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018, pp. 31-41. ACM, 2018. doi: 10.1145/ URL TRRLhtttt::: htttss://doi org/10 11 3211353
3211346.321 1353.
88

---

Published in Transactions on Machine Learning Research (09/2024)
Caitlin Sadowski, Jeffrey van Gogh, Ciera Jaspan, Emma Söderberg, and Collin Winter. Tricorder: Building a program analysis ecosystem. In Antonia Bertolino, Gerardo Canfora, and Sebastian G. Elbaum (eds.); 37th IEEE/ACAE ACM International Conference on Software Engineering, ICSE 2015, Florence, Italy May 16-24. 2015, Volume 1, pp. 598-608. IEEE Computer Society, 2015. doi: 10.1109/ICSE.2015.2 URL 10. 1w19/199IISSSS 2015.76
https://ddo org/
Ripon K. Saha, Yingjun Lyu, Wing Lam, Hiroaki Yoshida, and Mukul R. Prasad. Bugs.jar: large- scale, diverse dataset of real-world java bugs. In Andy Zaidman, Yasutaka Kamei, and Emily Hill (eds.), Proceedings of the 15th International Conference on Mining Software Repositories, MSR 2018, Gothenbury, Sweden, May 28-29, 2018, pp. 10 13. ACM, 2018. doi: (0.1145/319633 31994473.. 3196473
URL https://doi.
Muhammet Sahin and Serif Bahtiyar. A survey on malware detection with deep learning. In Siddika Berna Örs and Atilla Elgi (eds.), SIN 2020: 13th International Conference on Security of Information and Networks, Virtual Event / Istanbul, Turkey, November 4-6, 2020, pp. 34:1-34:6. ACM, 2020. doi: 10. 1145/34431744.44328
URL https::/ddii http::::/dooooo//////// 10. 11...01114//3337...3333 3433609
Anthony Saieva, Saikat Chakraborty, and Gail E. Kaiser. On contrastive learning of semantic similarity forcode to code search. CoRR, abs/23035 2as//255503843, 2023. doi: 10.48550/ARXX 2305.0383.. URL https:
//doi.org/10. 40550//r0..855//rxx...v. 2305 03843
Abhishek Sainani, Precthu Rose Anish, Vivek Joshi, and Smita Ghaisas. Extracting and classifying require- ments from software engineering contracts. In Travis D. Breaux, Andrea Zisman, Samuel Fricker, and Martin Glinz (eds.), 28th TEFF, International Requirements Engineering Conference, RE 2020, Zurich, Switzerlondd, August 31- September 4, 2020, pp. 147 157. IEEE, 2020. doi: 10.1109/RE4488 2tt.//RRE8222222..000000 URL sttp::///li..or/10..1111 00t...0/19////111110 200..
Vaibhav Saini, Farima Farmahinifarahani, Yadong Lu, Pierre Baldi, and Cristina V. Lopes. Oreo: detection clones in the twilight zone. In Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareamu (eds.), Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018, pp. 364.365.44 ACM, 2018. doi: 10. 3236622 URL httph:tt./:tttt /tttps:/ddo.org org/ 10. 0011/32366624444. 323662
Hitesh Sajnani, Vaibhav Saini, Jeffrey Svajlenko, Chanchal K. Roy, and Cristina V. Lopes. Sourcererce: scaling code clone detection to big-code. In Laura K. Dillon, Willem Visser, and Laurie A. Williams (eds.). Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016, pp. 1157-1168. ACM, 2016. doi:
28848777777 URL https:
//doi org/10. /1115/11887811228888448
Pasquale Salza, Christoph Schwizer, Jian Gu, and Harald C. Gall. On the effectiveness of transfer learning for code search. IEEE Trans. Software Eng., 49(4):1804 1822 2023. doi:
URL https://doi.org:
1109/TSE. 3192755
Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, and Brendan Dolan-Gavitt. Lost at C: A user study on the security implications of large language model code assistants. In Joseph A. Calandrino and Carmela Troncoso (eds.), 32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA, August 9-11, 2023, pp. 2205-2222. USENIX Association, 2023. URL https://www. usenix useeixxorg/ http:///ww..eeeeeeeeeeeeeeeeeaee http://ccccctttteeeeeeeeeaaaaaaa saadovaa
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaf- fin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoli Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Necraj, Jos Rozen, Abhcesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In The Tenth
89

---

Published in Transactions on Machine Learning Research (09/2024)
International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nel, 2022. URL https::/opennrevisw wwtp://www....ooooooooooooottaaa
Laboni Sarker, Mara Downing, Achintya Desai, and Tevfik Bultan. Syntactic robustness for Ilm-based code generation. CoRR, abs/ 0155444331,3 2024. doi: (0.45550/ARXIV.24000
URL httpss//doi.oo org/
10. 2055v0/arXiv..m 2404. 01535
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana 1lic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Amnanamanchi, Thomas Wang. Benoît Sagot, Niklas Muen- nighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan- Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezuc, Christopher Klamm, Colin Leong, Danicl van Strien, David Ifeoluwa Adclani, and el al. BLOOM: 176b-parameter open-access multilingual language model. CoRR, abs/22 2222222121100 2022. doi: 10.48550 htt///aaaiiiii..1.
URL https://doi. org/ 10. 48540/
arxiv. www.ii..21...0.... 2211.05100
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Thirty-seventth Conference on Newral Information Processing Systems, 2023. URL
https://openreviie wwweeerrevie....m.. nct.forruf//dd/aaa22331
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: parsing incrementally for constrained auto-regressive decoding from language models. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / PPunta Cana, Dominican Republic, 7-11 November, 2021, pp. 779. URL https://doi.
Association for Computational Linguisties, 2021. doi: 10.18653/v) 12212/22.eeeeee -emnlp-main. 10w.181163vvv000 emnlp-main. 779
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. CoRR, abs/ hals/1707.03344 2017. URL httt::///rxxii/ http:://wawwwrrraaaeae// wwtpg//bb/1777... .63447
Roci Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. You autocomplete me: Poisoning vulnerabilities in neural code completion. In Michael D. Bailey and Rachel Greenstadt (eds.), 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, pp. 1559 1575. USENIX Association, 2021. URL URLttttss:/// htttp:://www uswnix.
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation, 2023.
Marija Selakovic, Michael Pradel, Rezwana Karim, and Frank Tip. Test generation for higher-order functions in dynamic languages. Proc. ACM Program. Lang., 2000 2018. doi: 10.1145/3276532 URL https:://oi.orgg: 11e./3276511111
Sachith Seneviratne, Ridwan Shariffdeen, Sanka Rasnayaka, and Nuran Kasthuriarachchi. Self-supervised vision transformers for malware detection. IEEE Access 10:103121 103135, 2022. doi: 10.1109/ACCESS. 2022.3206442 URL http::::/ddo..rgg////
TAA.L////ACSS.... 3206445
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguis- tics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Popers. The Association for Com- puter Linguistics, 2016. doi: 10. /0/5////11116100000 URL https::/doiiooooo org/ 10. 18653//11pp16-1009
Ramin Shahbazi and Fatemeh Hendijani Fard. Apicontext2com: Code comment generation by incor- porating pre-defined API documentation. In 31st IEFE/ACM International Conference on Program Comprehension, ICPC 2023, Melbourne, Australia, May 15-16, 2023, pp. 13-24. IEEE, 2023. doi: 10.110//CCPC5590022222
wtw.:/8999.002...00 00012. URL hatllhhtts:///dt http///.....//...////// wttp:///......///////0000000 http:////000.0000000000101
90

---

Published in Transactions on Machine Learning Research (09/2024)
Sina Shamshiri. Automated unit test generation for evolving software. In Elisabetta Di Nitto, Mark Harman, and Patrick Heymans (eds.), Proceedings of the 2015 10th Joönt. Meeling on Foundations of Software Engineering, ESEC/FSE 2015, Bergamo, Italy, August 30 - September 4, 2015, pp. 1038 1041. ACM, URL https://doi. wttpooo..///////.....
2015. doi: 10.1145 (1555/2766805288003
2803196
Rishab Sharma, Fuxiang Chen, and Fatemeh H. Fard. LAMNER: code comment generation using character language model and named cntity recognition. In Ayushi Rastogi, Rosalia Tufano, Gabricle Bavota, Venera Arnaoudova, and Sonia Haidue (eds.), Proceedings of the 30th IEEE/ACM. International Conference on Program Comprehension, TCPC 2022, Virtual Event, May 16-17, 2022, pp. 48-59. ACM, 2022. doi: 10. 114553524610.3865
URL https://doli http://doo.oogg/l//// 10. 1144//552/3411111 255
Peter Shaw, Jakob Usækoreit, and Ashish Vaswani. Self-attention with relative position representations. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguislics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Short Popers), pp. 464-468. Association for Computational Linguistics, 2018. doi:
URL httpp:://oo..... org/
10. 10.18653/v11/18--1
Peter Shaw, Ming-We Chang, Panupong Pasupat, and Kristina Toutanova. Compositional generalization and natural language variation: Can semantic parsing approach handle both? In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Associalion for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/ ACI/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 922-938. Association for Computational Linguistics, 2021. doi: (0.18653/v1/2000 acc--long. 75. URL https://doi.orr http///0oorr//0.../// 00553//v12021.ac111000
Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/ (bs/1991.02210,, 2019.
URL http:://arx....rrggaa/////11 02150
Dongdong She, Kexin Pei, Dave Epstein, Junfeng Yang, Baishakhi Ray, and Suman Jana. NEUZZ: efficient fuzzing with neural program smnothing. In 2019 IEEE Symposium on Security and Privucy, SP 2019, San Francisco, CA, USA, May 19-23, 2019, pp. 803-817. IEEE, 2019. doi: https: ortoooog//// 10. 1109/SP. 2019.00052
URL
Dongdong She, Rahul Krishna, Lu Yan, Suman Jana, and Baishakhi Ray. Mtfuzz: fuzzing with a multi- task neural network. In Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (eds.), ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, pp. 737 749. ACM, 2020. doi: 10.1145/
9868089.33409723 URL https://doi. org/10 10. 1145/3368089 3409723
Xinyu She, Yue Liu, Yanjie Zhao, Yiling He, Li Li, Chakkrit Tantithamthavonn Zhan Qin, and Haoyu Wang. Pitfalls in language models for code intelligence: A taxonomy and survey, CoRR,
2023.
doi: 10.48550/ (1850//AXIVV ARXIV 22310.19903 URL https://doi..rg:e http://.././///r//////////////// aatovvrrivv.... 2310. 17903
Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, Yuenan Guo, and Qianxiang Wang. Pangu-coder2: Boosting large language models for code with ranking fcodback. CoRR, abs/2307. /bs//3377.4936, 2023. doi: 10.48550/ 2tt...2007.19998 URL https://doi. org/ 101 www.0///0ii....../ 2307 14936
Chaochen Shi, Borui Cai, Yao Zhao, Longxiang Gao, Keshav Sood, and Yong Xiang. Coss: Leveraging statement semantics for code summarization. IEEE Trans. Software Eng., (46))3372-34488 2023a. doi: 2000///SEE.2023355252
URL hittps://dof org/10. 1199/TTTE 2023. 3256362
Ensheng Shi, Yanlin Wang, Wei Tao, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. RACE: retrieval-augmennted commit message generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 5520-5530. Association for Computational Linguistics, 2022. doi: 10. 10.18533vv//022 ennlp-main.37 372
211/2222..EELLPPP 372.. URL https: attps://doi.org:
91

---

Published in Transactions on Machine Learning Research (09/2024)
Ensheng Shi, Yanlin Wang, Wenchao Gu, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. Cocosoda: Effective contrastive learning for code search. In 45th FEEE/ACM International Confer- ence on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 2198 2210. LEEE, 2023b. doi: D0.1099ICSSEAS00 0tt.:/611...2..001100 2023.00185. URL https: / otg/////oo.oo///////// 1109/ 11009/9SE48619.... 2023 00185
Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cícero Nogueira dos San- tos, and Bing Xiang. Learning contextual representations for semantic parsing with pre-training. In Thirty-Fifll, A AAAT Conference on Artificial Intelligence, AAAT 2021, Thirty- Third Confer- ence on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educa- tional Advonces in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 13806-13814. AAAI Press, 2021. doi: 10.1609/ A.../AAALL3311111 17627
URL https://doi... org 10. 1609/ /aaii.aaa....... v35115.
Shu-Ting Shi, Ming Li, David Lo, Ferdian Thung, and Xuan Huo. Automatic code review by learning the revision of source code. In The Thirty-Thite Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-Pirst Pirst Innovalive Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth
AAAI Sympostum on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- - February 1, 2019, pp. 4910-4917. AAAT Press, 2019. doi: 10.1609/ama /aaaaaaaiiia.aa 1a3i01
w33.01.331111010330 .30014910.
URL https://dot org/10. 10. 1609/ /0009//aaa.... v33101. 33014490
Tianze Shi, Chen Zhao, Jordan L. BoydeGraber, Hal Daumé III, and Lillian Lec. On the potential of lexico-logical aliguments for semantic parsing to SQL queries. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pp. 1849-1864. Association for Computational Linguistics, 2020. doi: 10.18653/VV /015/3/////// 1112220..FFF. FINDIIGG-EEMNAAANNN
URL
1t/2220..indiings-eeee 167
Samiha Shimmi and Mona Rahimi. Leveraging code-test co-evolution patterns for automated test case recommendation. In JEEE/ACM International Conference on Automation of Software Test, AST@ICSE 2022, Pittsburgh, PA, USA, May 21-22, 2022, pp. 65-76. ACM/TEEE, 2022. doi: 10.1145/352446 3527222. URL htttss://oi. org/10. 10.
Jiho Shin, Sepchr Hashtroudi, Hadi Hemmati, and Song Wang. Domain adaptation for deep unit test case generation. CoRR, abs/2308. 08033, 2023. doi: 10.48550/ ARXIV 2308.08033. URL htttps://oi.. org/ 10. 10.48550/arXia www.rxwwwvv.......m 230.8. 08033
Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. Execution-based code generation using deep reinforcement learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL fttpp:://pparrevee..nett//
Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, and Torsten Scholak. Repofusion: Training code models to understand your repository. CoRR, abs/ bbb6/233.... 10099 2023a. doi: 10.48550/ TEXXVV2306.10980 URL hattps://aii.rg//1 485500 http:/a............... 10998
Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. Repository-level prompt generation for large lan- guage models of code. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, 1CML 2023, 23- 29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 31693 31715. PMLR, 2023b. URL httph::t/p htttpa://proceeding.. mlr.
wwtp//////aaaaaaaaaaaaaaaaaa html
Jianhang Shuai, Ling Xu, Chao Liu, Meng Yan, Xin Xia, and Yan Lei. Improving code search with co- attentive representation learning. In ICPC '20: 28th International Conference on Program Comprehension, Seou,, Republic of Korea, July 13-15, 2020, pp. 196 207. ACM, 2020. doi: 10. /115//333379413392666 URL https:///oi.. org/ 10. 1et.111453338900 338969
Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far are we from automating front-end engincering? CoRR, abs /2//220303333333 2024. doi: 1148550/ARXIT 240.002200.0 URL htttps://doi.org//// www.oo/////....////a.a arriv. 2403.031033
92

---

Published in Transactions on Machine Learning Research (09/2024)
Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, and Gust Verbruggen. Codefusion: A pre-trained diffusion model for code generation, 2023.
Jing Kai Slow, Cuiyun Gao, Lingling Fan, Sen Chen, and Yang Liu. CORE: automating review recom- mendation for code changes. In Kostas Kontogiannis, Foutse Khomh, Alexander Chatzigeorgion, Marios- Eleftherios Fokaefss, and Minghui Zhou (eds.), 27th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2020, London, ON, Canada, February 18-21, 21, 2020, pp. 284 295. 1.009//NNER8827555
TEEE, 2020. doi: 2020. 9054794
200. URL bttts::/ddoi.rg///
Aishwarya Sivaraman, Rui Abreu, Andrew Scott, Tobi Akomolede, and Satish Chandra. Mining idioms in the wild. In 44th IEEE/ACM, International Conference on Software Engineering: Software Engineer- ing in Practice, ICSE (SEIP) 2022, Pittsburgh, PA, USA, May 22-24, 2022, pp. 187-196. IEEE, 2022, doi: 10. N0.110//CSE--EEP55383.228 202..30000022229904093 URL hnttps://doi.rrge: 11t.1111--SSSSISS0000
1109/10SSSSS SEIP55303 2022.
9794062
Salch Soltan, Shankar Ananthakrishnan, Jack FitzGorald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gökhan Tür, and Prem Natarajan. Alexatmn 20b: Few-shot learning using large-scale multilingual seq2seq model. CoRR, abs/ hss/2208.004488, 2022. doi: 10.48550/ (18550//rXiv.2200 01448. URL https:://oi..ooo org/ 48550/.855//88.
2208.01448
Nikita Sorokin, Dmitry Abulkhanov, Sergey I. Nikolenko, and Valentin Malykh. Cct-code: Cross-consistency training for multilingual clone detection and code search. CoRR, abs abbs/2300 220/2 11626, 2023. doi: 10.48550/
ARXIV 2306.11666. URL httpp:://aii.org/// 2tt.:/100..555///xxxxxxx wat.5o//rxxi.v.. 2305. 11626
Davit Soselia, Khalid Saifullah, and Tianyi Zhou Learning ui-to-code reverse generator using visual critic without rendering. CoRR, abs/ 1405, 2023. doi: 10.48550/ARXIV.220000 23..5XXXXV..3..4467... URL httpp:///oi.
org/10.48550//rr/ 2202 14637
Benjamin Steenhoek, Md Mahbubur Rahman, Richard Jiles, and Wei Le. An empirical study of deep learning models for vulnerability detection. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 2023. 00188 URL https://doi org/10. 1///111.11100
IEEE, 2023a. doi:
Benjamin Steenhock, Michele Tufano, Neel Sundaresan, and Alexey Svyatkovskiy. Reinforcement learning from automatic feedback for high-quality unit. test. generation. CoRR, abs 2ab1//03300233 2023b. doi: 10.
48550/ARXIV 221.000223..00.26... URL https:///oi..... org/10 w0w..//80/a...v...../....o 2310. 2310.02368
Benjamin Steenhoek, Md Mahbubur Rahman, Monoshi Kumar Roy, Mirza Sanjida Alam, Earl T. Barr, and Wei Le. A comprehensive study of the capabilities of large language models for vulnerability detection. CoRR, abs/2403, 17218, 2024. doi: 10.48550/ARXII 1728.. URL https://ddo..or//10..o
arXiv. 2403 17218
Chia-Y Yi Su and Collin McMillan. Semantic similarity loss for neural source code summarization. CoRR, abs/2308.00 07449, 2023a. doi: 10.48550/ARXIV 0749.. URL https:: htt../........0...05/////... 2308.07422
48550/arXiv.
Chia-Yt Yi Su and Collin McMillan. Distilled GPT for source code summarization. CoRR, abs/2308, 147331, 2023b. doi: T1.4550/AARXII 2308.14771... URL https://doi. wtt.oo......5//...ii.......
48550/arXiv 2308 14731
Chia-Yi Su, Aakash Bansal, Vijayanta Jain, Sepideh Ghanavati, and Collin McMillan. A language model of java methods with train teet deduplication. CoRR, 08286, 2023a. doi: 10. 10.48550/ AR..0//RXIV.... .2055. 08286. URL httpp:://oi...... htt.///oo.oooo////////o///oo 10. 48550/555000
wat.i//rrivv... www.aiwiw............ 08286
Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing. 568:127063, 2024. doi:
10..01//INFUUCOM.200..... URL https://doi. 1016// .nw.om.. 2023. 127063
93

---

Published in Transactions on Machine Learning Research (09/2024)
Yiming Su, Chengcheng Wan, Utsav Sethi, Shan Lu, Madan Musuvathi, and Suman Nath. Hotgpt: How to make software documentation more useful with a large language model? In Malte Schwarskopf, Andrew Baumann, and Natacha Crooks (eds.), Proceedings of the 19th Workshop on Hot Topics in Operating Systems, HOTOS 2023, Providence, RI, USA, June 24, 2023, pp. 87-93. ACM, 2023b. doi: 10.1145/ 3593856.3595910. URL httpss///dii. orgg 100 1145/3593856. 3595910
Alane Suhr, Srinivasan Iyer, and Yoav Artzi. Learning to map context-dependent sentences to executable formal queries. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Långuistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Pa- pers), pp. 2238 2249. Association for Computational Linguistics, 2018. doi: 10.18653/ attpps///ooi org/10 10. 1wt..1116///11-111111
10.1185//v1//88-/1/ ///////1/////////// URL
Ruoxi Sun, Sercan Ö. Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. Sql-palm: Improved large language model adaptation for text-to-sql. CoRR, abs /2//222300033333 2023a. doi: 10.48550/
URL httpp:::s//// 40tg:/000..555///11.v wttp://///..v.......... www.23336.00733
Weisong Sun, Chumrong Fang, Yuchen Chen, Guanhong Tao, Tingxu Han, and Quanjun Zhang. Code search based on context-aware code translation. In 44th JEEE/ACN ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25- 25-27, 2022, pp. 388-400. ACM, 2022. doi: 10.11455/550008.33511111 URL https://doi. 0r1.://...0.00..1111111000000000
Weisong Sun, Chunrong Fang, Yudu You, Yun Miao, Yi Liu, Yuekang Ti, Gelei Deng, Shenghan Huang, Yuchen Chen, Quanjun Zhang, Hanwei Qian, Yang Liu, and Zhenyu Chen. Automatic code summarization via chatgpt: How far are we? CoRR, abs/2305. 12865, 2023b. doi: (18555/ARXIV.... 1285. URL
attps://doi org. 10. 48550/arXiv. 2305. 12865
Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Miaolei Shi, and Yang Liu. Llm4vul: A unified evaluation framework for decoupling and enhancing 1lms' vulnerability reasoning. CoRR, abs/ abs/2401.16185, 2024. doi: 10.48550/ARXII 4485006/AXX..2200166665.. 1401.16185 URL http::///ooi.og/11....4
arXiv
Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, Lili Mou, and Lu Zhang. Treegen: tree-based transformer Thirty-Fourth architecture for code generation. In The Thirty- Fourth AAAT Conference on Artificial Intelligence, AAAT
2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8984 8991. AAAI Press, 2020. doi: 10. V1.1600/AAALLV3410..... V34105.6430. URL https://doi.. orgo 10. 1609/ v34105. 6430
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. 2wW.IV...330012222 URL https://doi wttp///i..rg//...88/....
CoRR, abs/2303, 28, 2023. doi:
arXiv arXiv.2303.00 220..3333.0811 08128
llya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to sequence learning with neural networks, In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems 27: Annual Conference on Neural Information Process- ing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3104-3112, 2014. URL https:// sonceedinggs.nuuip.... proceedings neurips. www..pp...cppppee///////// 2ww//004//hhaee
Abstract ceastract.htm]]
Jeffrey Svajlenko and Chanchal K. Roy. A survey the evaluation of clone detection performance and benchunrkingg. CoRR, abs/2006.155 15682, 2020. URL httpss//arxiv.orrgaa///00...
Jeffrey Svajlenko, Judith F. Islam, Iman Keivanloo, Chanchal Kumar Roy, and Mohammad Mamun Mia. Towards a big data curated benchmark of initen-rrooo project code clones. In 30th IEEE International Conference on Software Maintenance and Evolution, Victoria, BC, Canada, September 29 October 3, 2014, pp. 476-
480. IEEE Computer Society, 2014. doi: 11110///CMME.201..77 URL https://doi. wtt.ddi..og//...111 1109/
ICSME. 2014.77
94

---

Published in Transactions on Machine Learning Research (09/2024)
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, and Neel Sundaresan. Pythia: Ai-assisted code completion system. In Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (eds.), Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pp. 2727-2735. ACM, 2019. doi: 10.1145/ 3292500.3330699. URL httpss//ddii org/10. 10. /141//2920///./.// 3330699
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose: code genera- tion using transformer. In Prem Devaubu, Myra B. Cohen, and Thomas Zimmermann (eds.), ESEC/FSH '20: 28th ACM Joint European Software Engineering Conference and Sympostum on the Foundations of Software Engineering, Virtual Event, USA, November 8- 8-13, 2020, pp- 1433 1433-1443. ACM, 2020. doi: URL https://doi.ors wtt.oo/orrg/11.111//3333331 1145/336813338 3417055
10. 01..145/3336089.34417777
Mare Szafraniec, Baptiste Rozière, Hugh Leather, Patrick Labatut, François Charton, and Gabriel Synnaeve. Code translation with compiler representations. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.nef, 2023. URL https://
Florian Tambon, Arghavan Moradi Dakhel, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Giu- liano Antoniol. Bugs in large language models generated code: An empirical study. CoRR, abs/ abs/2403.0063 2203308937, 2024. doi: 10.48550/ ARXIV 2403.08937. URL ftttps//doi..oo htt.://...........////////..///0 arxvv. 2403 08937
Hanzhuo Tan, Qi Luo, Jing Li, and Yuqun Zhang. Lim4decompile: Decompiling binary code with large language models. CoRR, 00s//2403.0522 2024. doi: 10.48550 ARXIV ARA...22....2.222222222 2403.05228. URL httpps://oi.
4///10..855//rr// 2403.4433000 05286)
Shin Hwei Tan, Jooyong Yi, Yulis, Sergey Mechtaev, and Abhik Roychoudhury. Codeflaws: a pro- gramming competition benchmark for evaluating automated program repair tools. In Sebastián Uchi- tel, Alessandro Orso, and Martin P. Robillard (eds.), Proceedings of the 39th International Confer- ence Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017 Companion Volume, pp. 180 IEEE Computer Sociely, 2017. doi: 10.1109 10100//CSE-C.211.... 2017.76
URL https:
//doi org/10. wwt.:///////////..........
Lappoon R. Tang and Raymond J. Mooney. Automated construction of databasc interfaces: Intergrating statistical and relational learning for semantic parsing. In Hinrich Schütze and Keh-Yih Su (eds.), Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, EMNLP 2000, Hong Kong, October 7-8, 2000, pp. 133-141. Association for Computational Linguistics, 2000. doi: 10.3115/1117794.111
URL
Yutian Tang, Zhijie Liu, Zhichao Zhou, and Xiapu Luo. Chatgpt vs SBST: comparative assessment of unit test suite generation. CoRR, abs/230 23070005508 2023a. doi: 10.48550/ARRA ARXIV 22..2700000088 URL https://doi.org: 48550/00.48.///88 arw.v..xiiv... 2307 00588
Ze Tang, Xiaoyu Shen, Chuanyi Li, Jidong Ge, Liguo Huang, Zheling Zhu, and Bin Luo. Ast-trans: Code summarization with efficient tree-structured attention. In 44th IEEE/ACE /AC 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 150-162. ACM, 2022. URL https: org/ wtt.:///gg//............. 1145/3510003 R01.0000003.33244444.444430 3510224
doi: 10. 10.1145/35100003351000
Zilu Tang, Mayank Agarwal, Alex Shypula, Bailin Wang, Derry Wijaya, Jie Chen, and Yoon Kim. Explain- then-translate: An analysis on improving program translation with self-generated explanations. CoRR, abs abs/2311.07070, 2023b.
doi: 10.48550/ARXIV 2311000007. URL httpps:///al /ttp::////oooor/10.... htt/10..8550//a2ii..... 48550/arXiv.
231 17070
Shimin Tao, Weibin Meng, Yimeng Chen, Yichen Zhu, Ying Liu, Chunning Du, Tao Han, Yongpeng Zhao, Xiangguang Wang, and Hao Yang. Logstamp: Automatic online log parsing basod on sequence labelling. SIGMETRICS Perform. Evaluation Rev., 49(4):93-99 98, 2022. doi: 10.1145/3543146.335 46.33116688. URL https:
95

---

Published in Transactions on Machine Learning Research (09/2024)
Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang. On the evaluation of commit message generation models: An experimental study. In IEEE International Conference on Software Maintenance and Evolution, ICSME 2021, Luzembourg, September 27 October 1, 2021, pp. 126-136. IEEE, 2021. doi: 2..109/TCSMES2211.222...000 URL https: Vidoi.org//l 1.0011109/ICSME521000 2021.00018
Wei Tao, Yucheng Zhou, Wenqiang Zhang, and Yu Cheng, MAGIS: Ilm- based multi-agent framework for github issue resolution. CoRR, abs/ (bb//2403.1722 2024. doi: 10.48550/ ARXIV 2403.17927. URL https:
//doi.org/10 48550/arXiv 2403. 17927
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsuoori B. Hashimoto. Stanford alpaca: An instruction-folloiing llama model. https://github. com/
Batsu-lab/stanfort _lpacca 2023.
Yi Tay, Mostafa. Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Compul. Surv., 55(6):109:1110988)) 2023a. doi: 10.1145/3530811. URL htttps://aoi.org/// 1115///30111////1111111
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. UL2: unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenKeview.net, net, 2023b. URL https:://pperrevie..
pdf 6rUVLB72MCC
CodeGemma Team. Codegemma: Open code models based on gemma, 2024. URL Sttps://storaas
googleapis wwwpapppp...a.aaaaaaaaaaaaaaeaee wwt.com///eee/e -edeanindddmddaadaaa httpa//aeecoeeeeeeeeeeeeeaae pdf
Shailja Thakur, Baleegh Ahmad, Zhenxing Fan, Hammond Pearce, Benjamin Tan, Ramesh Karri, Brendan Gavint and Siddharth Garg. Benchmarking large language models for automated verilog RTL code generation. Tn Design, Automation & Test in Europe Conference & Erhibition, DATE 2023, Antwerp, Belgium, April 17-19, 2023, pp. 6. IEEE, 2023a. doi: 112299///AATE3557522222.. 1013700 URL https://doi.. orgg 10///10.210 23919/ 2023. 10137086
Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri, and Siddharth Garg. Verigen: A large language model for verilog code generation. CoRR, /ttp/2308.00000 2023b. doi: 10.48550/ ARXIV 2308.00708. URL https://doi. ore/10..r111...0 48550/arXiv wt1.5//aXxiv.208... 2308 00708
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Kathleen S. Meier-Hellsseern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Lc. Lamda: Language models for dialog applications. CoRR, hts/2201..0233) 2022. URL org/ abs/ htt///bb//2200000 2tt///221..02229
Kiran Thorat, Jiahui Zhao, Yaotian Liu, Hongwu Peng, Xi Xie, Bin Lei, Jeff Zhang, and Caiwen Ding, Advanced large language model (Ilm)-drieen verilog development: Enhancing power, performance, and area optimization in code synthesis. CoRR, abs/ 2b1/201200 2023. doi: 10.48550/ARM3 2312.01022.
URL https:///oi.org http////oo.or/////////// 48550/arXiv. 2012.
Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Zhiyuan Liu, and Maosong Sun. Debugbench: Evaluating debugging capability of large language models. CoRR, abs/2401. 04621, 2024. doi: 1188550/ARXI.2400 04621.. URT https://ddii..... org/10 48550//r.... artiv...iii.. 2401.
96

---

Published in Transactions on Machine Learning Research (09/2024)
Saurabh Tiwari, Deepti Ameta, and Asim Banerjec. An approach to identify use case scenarios from textual requirements specification. In Ravindra Naik, Santonu Sarkar, Thomas T. Hildebrandt, Atul Kumar, and Richa Sharma (eds.), Proceedings of the 12th Innovations on Software Engineering Conference (formerly known as India Software Engineering Conference), ISEC 2019, Pune, India, February 14-16, 2019, pp. 5:1- orgg 100 /294//////99911./ 32299774
5:11. ACM, 2019. doi: 0114/////99711.33
URL https:://doo http://oi.og//////////
David A. Tomassi, Naji Dmeiri, Yichen Wang, Antara Bhowmick, Yen-Chuan Liu, Premkumar T. Devanbu, Bogdan Vasilescu, and Cindy Rubio-González. Bugswarm: mining and continuously growing a dataset of reproducible failures and fixes. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (eds.), Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 339 349. IEEE/ / ACM, 2019. doi: 11110///CSE.211..00000 URL https:///oi..... 10. 1119/ICSE.... 2019. 00048
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 135//230.13971, 2023a. doi: 10.48550/ aaii/ (t..ii..302.11977.. URL httph:://p/oo /rgp///00o.or//..///// 11eeg//...855//// 18550/arXiv. serXiv.2322 13971
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Sub- ramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Au- rélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307 220000900878... 09288, 2023b, doi: 10. 10.48550/ar 2...v.2307.00988. URL https://doi. org/ 10. 48550/ Deexiv.23001 09288
Hieu Tran, Ngoe M. Tran, Son Nguyen, Hoan Nguyen, and Tien N. Nguyen. Recovering variable names for minifiod code with usage contexts. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (cds.), Proceedings of the 41 International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 1165-1174 TEEE/ / ACM, 2019. doi: 10.1109/ICSE.2019.001 URL https://aoi.org: 10. 10.1109/ICSE. 2019.00111
Immanuel Trummer. Codexdb: Synthesizing code for query processing from natural language instructions using GPT-3 codex. Proc. VLDB Endow., 15(11):2921 29222292221,111212211122192112 2022. URL https://tww:/ vldb. wwwpb.orgppp............o. org/pvldb/
vol15/p2921 111921-trumne trummer.... .pdf
Yun-Da Tsai, Mingjie Liu, and Haoxing Ren. Rtlfixer: Automatically fixing RTL syntax errors with large language models. CoRR, 2//.//1111..11.. 2023. doi: (0448550/ARXIV2231111 165433. URL httpps://ooi
2///100.455//aaaa 2311.165433
Chen Tsfaty and Michael Fire. Malicious source code detection using transformer. CoRR, abs/ Abs/2209.00957) 2022. doi: 10.48550/ ARXIV 2209.07957.. URL http:::/ddo..org/// 48550/arXiv 220...2299000 07957
Zhaopeng Tu, Zhendong Su, and Premkumar T. Devanbu. On the localness of software. In Shing- Chi Cheung, Alessandro Orso, and Margaret-Anne D. Storey (eds.), Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22), Hong Kong, China, November 16 22, 2014, pp. https://doi.org 2ttp///0..1//263586665 2665572773356888287755
ACM, 2014. doi: 10.1145/2635868 2633585.. URL
97

---

Published in Transactions on Machine Learning Research (09/2024)
Michele Tufano, Jevgenija Pantiuchina, Cody Watson, Gabricle Bavota, and Denys Poshyvanyk. On learning meaningful code changes via neural machine translation. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (eds.), Proceedings of the 41st International Conference on Software Engineering, ICSE' 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 25-36. IEEE/ / ACM, 2019a. doi: 10.1109/ICSEE2019.0022 URL https://doi. org/ 1109/ICSEE
Michele Tufano, Cody Watson, Gabriele Bavola, Massimiliano Di Penta, Martin White, and Denys Poshy- vanyk. An empirical study on learning bug-fixing patches in the wild via neural machine transla- tion. ACM Trans. Softw. Eng. Methodol., 2844),9::-, 2019b. doi: 10.1145/334055
URL
https:/ /tt::://doooooooo org...rg/1... 101111135335444
Michele Tufano, Cody Watson, Gabriele Bavola, Massimiliano Di Penta, Martin White, and Denys Poshy- vanyk. Learning how to mutate source code from bug-fixes. In 2019 IEEE International Conference on Software Maintenance and Evolution, ICSME 2019, Cleveland, OH, USA, September 29 - October 4, 2019. URL https: htops://doi.. http///ii.ogg/g///// wttp///...//////...../ 1109/ICSME.
pp. 301-312. IEEE, 2019c. doi: 0
Michele Tufano, Jason Kimko, Shiya Wang, Cody Watson, Gabricle Bavota, Massimiliano Di Penta, and Denys Poshyvanyk. Deepmutation: a neural mutation tool. In Gregg Rothermel and Doo-Hwan Bae (ods.), ICSE '20: 42nd International Conference on Software Engineering, Companion Volume, Seoul, South Korea, 27 June - 19 July, 2020, pp. 29 32. ACM, 2020. doi: 10.1155/3778812.332148 URL https://doi.. orgg 10. 01111/1//3711222 3382146
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan, Unit test case generation with transformers and focal context, 2021a.
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, and Neel Sundaresan. Generating accurate assert statements for unit test cases using pretrained transformers. In JEEE/ACM International Conference on Automation of Software Test, AST@ICSE 2022, Pittsburgh, PA, USA, May 21 21-22, 2022, pp. 54-64. ACM/IEEE, 2022a. doi: 10.1145/3524 351444111..27722.0 URL hhtpps://doi org/10. 10411144//555 01444///222481352222222
Rosalia Tufano, Luca Pascarella, Michele Tufano, Denys Poshyvanyk, and Gabriele Bavota. Towards au- tomating code review activities. In 43rd TEFE/A A CM International Conference on Software Engineering, ICSE 2021, Madrid, Spoin, 22-30 May 2021, pp. 163-174. IEEE, 2021b. doi: 10. D0.1009/CCSE4300222211 00027. URL https:///ti...oo// / 10. N0..1109/ICSE4300000 2021.00027
Rosalia Tufano, Simone Masicro, Antonio Mastropaolo, Luca Pascarella, Denys Poshyvanyk, and Gabriele Bavota. Using pre-trained models to boost code review automation. In 44th IEEE/ACM ACM 44th International Conference on Software Engineering, TCSE 2022, Pittshurgh, PA, USA, May 25-27, 2022, pp. 2291-2302. 1144/35100000 URL https://doi. 000.:/26.....r.10...116666/.0.0. 30000000000333302111111 3510621
ACM, 2022b. doi: 10.1145/3510003.351000
Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Natural Language Processing with Transformers: Building Language Applications with Hugging Face. O'Reilly Media, Incorporated, 2022. ISBN 1098103246.
URL http://oww...ooeeeee/////////aaa
Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse K. Coskun, and Gianhuca Stringhini. Can large language models identify and reason about security vulnerabilities? not yet. CoRR, abs 22///231222221 12575, 2023. doi: 10.48550/ /A////RXII//2 22312.12775 URL https://doft http:////oooro/1////// 4855444555///. http:/avavvvvv........ 2312. 12575
Tim van Dam, Frank van der Heijden, Philippe de Bekker, Berend Nieuwschepen, Mare Otten, and Maliheh Izadi Investigating the performance of language models for completing code functional programming languages: a haskell case study. CoRR, abbs/2403.15ke 2024. doi: 10.48550 000550/ARXIV..00 A......2200..... URL 48.50000/aaaa /00/550//XXiv.../ arwiv. 2403 15185
https://doi.com wtw.ooo.rrr111..881....
Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh Singh. Neural program repair by jointly learning to localize and repair. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview net, 2019. URL
met//forummiideBBlloo//22gg
98

---

Published in Transactions on Machine Learning Research (09/2024)
Bogdan Vasilescu, Casey Casalmuovo, and Premkumar T. Devanbu. Recovering clear, natural identifiers from obfuscated JS names. In Eric Bodden, Wilhelm Schäfer, Arie van Deursen, and Andrea Zisman (eds.); Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017, pp. 683-693. ACM, 2017. doi: attpss//doi... /ttp///..111/1117733333333 310628
3106228.. URL
Mario Linares Vásquez, Luis Fernando Cortes-Coy, Jairo Aponte, and Denys Poshyvanyk. Changescribe: A tool for aulomatically generating commit messages. In Antonia Bertolino, Gerardo Canfora, and Se- bastian G. Elbaum (eds.), 37th TERE/ACM International Conference on Sofbware Engineering, ICSE 2015, Florence, Italy May 16-24, 2015, Volume 2, pp. 709 712. IEEE Computer Society, 2015. doi: URL https://doi. http::://do.orr/10.... org/10. wttp://..//.....//...... IICO0//CCE.0000 229
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advunces in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL https::/pproceedings sngprocceedinggs.euuip..... neurips. http:////...aaaeaae222226/666111 absttaaaaaaaaaaaaaaattat htmll
Vasudev Vikram, Caroline Lemieux, and Rohan Padhyy, Can large language models write good property- 230.ARXXI.220..032200 04346. URL ltttps://doi....
based tests? CoRR, /2//2/2377333333 2023. doi: 10. 10.48550/arXiv. wttp:Xvvvv.......
Johannes Villmow, Jonas Depoix, and Adrian Ulges. ConTest: A unit test completion benchmark featuring context. In Royi Lachmy, Ziyu Yao, Greg Durrett, Milos Gligoric, Junyi Jessy Li, Ray Mooney, Gra- ham Neubig, Yu Su, Huan Sun, and Reut Tsarfaty (eds.), Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), pp. 17-25, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ http/2221..lppppoo/ 1.2. /ttl:ttt/////aattttooo .rgg 2021 n m1ppprog-1.2 1.2
Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. Improving automatic source code summarization via deep reinforcement learning. In Marianne Huchard, Christian Kästner, and Gordon Fraser (eds.), Proceedings of the 33rd ACM/IEEE International Conference Aulomaled Softwuree Engineering, ASE 2018, Monlpellier, France, Seplember 3-7, 2018, pp. 397-407. ACM, 2018. doi: 10. 44445//333114733322200 URL WtRLhhtps:://oo org/10. 1145/32331111 3238323
Yao Wan, Guanghua Wan, Shijie Zhang, Hongyu Zhang, Yulei Sui, Pan Zhou, Hai Jin, and Lichao Sun. Does your neural code completion model use my code? membership inference approach. CoRR, abs/2 0lbs/2404.1220 2024.
doi: 10.48550/ A08550/ARXXVV ARXIV 24404.1299.
URL https://dol: httpp://ddo..orr/10.. 10g/110.....55////2.. 48550/arXiv.
2404. 14296
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi (eds.), Proceedings of the Workshop: Analyzing and Interpreling Newral Networks for NLP, BleckbozNLLPEEMNND 2018, Brussels, Belgium, November 1, 2018, pp. 353-355 Association for Computational Linguistics, 2018a. doi: 10./5/33/1/1185//// URL attps://doi.org:
10. 10.18653/v1/v18- 5446
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Supergluc: A stickier benchmark for general-purpose language understanding systems. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Con- ference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancou- /hash/
ver, BC, Canada, pp. 3261 3275, 2019. URL 7096ff/44aeeTTabb6f60444444aaaa htw.::/aseeeaeeaaaaaaaeeaaeeaet. Abstrrctttrt.t.er.ee html
99

---

Published in Transactions on Machine Learning Research (09/2024)
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. RAT-SQL: relation- aware schema encoding and linking for text-to-sql parsers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 7567-7578. Association for Computational Linguistics, 2020a. doi: 10.18653/v /v11 222000acl-mminn...
URL http:::t///// http:///................/////0.. acl-maincmmm .677
Bailin Wang, Wenpeng Yin, Xi Victoria Lin, and Caiming Xiong. Learning to synthesize data for semantic parsing. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Tz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), ), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 7760-2766. Association for URL https:://ol. org/
Computational Linguistics, 2021a. doi: 212G////11/211.AAAIA.A. 10. 10.18653/v1/2021 maacl-main 220
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. May 2021.
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile UI summarization with multimodal learning. In Jeffrey Nichols, Ranjitha Kumar, and Michael Nebeling (eds.), USTT '21: The 34th Annual ACM Symposium on User Interface Software and Technology, Virtual Event, USA, October 10-14, 2021, pp. 498-510, ACM, 2021b. doi: 10.1145/ 10.1145/3472749.34777777 /44//344474449477... URL https://doi.org orggoo////// 1trg/10..30 01111144/3477444 3474765
Bryan Wang, Gang Li, and Yang Li. Enabling conversational interaction with mobile UI using large language models. In Albrecht Schmidt, Kaisa Väänänen, Tesh Goyal, Per Ola Kristensson, Anicia Peters, Stefanie Mueller, Julie R. Williamson, and Max L. Wilson (eds.), Proceedings of the 2023 CHI Conference Human Factors in Computing Systems, CHI 2023, Hambury, Germany, April 23-28, 2023, pp. 4 432:17. ACM, 2023a. doi: 05444/555444448358885585 URL httptttt:://oooorr///// 1001/1144/33455 3580895,
Chunhui Wang, Fabrizio Pastore, and Lionel C. Briand. Automated generation of constraints from use case specifications to support system testing. In 11th IEEE International Conference Soflwore Testing, Verification and Validation, ICST 2018, Västerås, Sweden, April 9-13, 2018, pp. 23-33. IEEE Computer Socicty, 2018b. doi: 10.1109/ICST.2000 201.....01.00013 20..20180001111
http://od.....eecmmutrrrrrcoceo httpaccccppttersseceeeerrrr/.//a org/10. 1109/
Deze Wang, Zhouyang Jia, Shanshan Li, Yue Yu, Yun Xiong, Wei Dong, and Xiangke Liao. Bridging pre- trained models and downstream tasks for source code understanding. In 44th IEEE/ACM 44th Interna- tional Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 287- URL stttps://do...o
298. ACM, 2022a. doi: 00.1145/3510003.310000
Haoye Wang, Xin Xia, David Lo, Qiang He, Xinyu Wang, and John Grundy. Context-aware retrieval- based deep commit message gencration. ACM Trans. Softw. Eng. Methodol., :064):66:1-36330 2021c. doi: 10. 10.1145/3464639. URL htttps://doi.org// 10. 1145/3464689
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. Software testing with large language models: Survey, landscape, and vision. CoRR, abs/2307 222223077222211 2023b. doi: 10.48550/ ARXIV 2307.07221. URL https://doi..rg/// 48550/arx10. http:/r.vi..........
Ke Wang and Zhendong Su. Blended, precise semantic program embeddings. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2020, pp. 121 134, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450376136. doi:
10.1145/33854112333377 33.5519...3999 URL https:://doiii org/10. 10. 1e5.111113341.23555 3385999
Liran Wang, Xunzhu Tang, Yichen He, Changyu Ren, Shuhua Shi, Chaoran Yan, and Zhoujun Li. Delv- ing into commit-issue correlation to enhance commit message generation models. In 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luzembourg, September 11-15, 2023, pp. 710-722. IEEE, 2023c. doi: 10.1109 ALL109/ASE52999.20 2023.00020 ASE56229 2023.00055
URL Inttp::// http:///....or/..11111 org/10..109/
100

---

Published in Transactions on Machine Learning Research (09/2024)
Pengeheng Wang, Jeffrey Svajlenko, Yanzhao Wu, Yun Xu, and Chanchal K. Roy. Ccaligner: a token based lapge-gap clone detector. In Michel Chaudron, Ivica Crnkovic, Marsha Chechik, and Mark Harman (eds.), Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenbury, Sweden, May 27 - June 03, 2018, pp. 1066-1077. ACM, 2018c. doi: https://doi.... org. 10.
3188119...... URL
3180179
Ping Wang, Tian Shi, and Chandan K. Reddy. Text-to-sql generation for question answering on electronic medical records. In Yennun Huang, Irwin King, Tic-Yan Liu, and Maarten van Steen (eds.), WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pp. 350 361. ACM/ / IW3C2, 2020b. doi: 10. 01114//3366423333801111 URL https:///oi.coo
1145/33664333.... 3380120
Qiaosi Wang, Michael Madaio, Shaun K. Kane, Shivani Kapania, Michael Terry, and Lauren Wilcox. Design- ing responsible Al: adaptations of UX practice to meet responsible Al challenges. In Albrecht Schmidt, Kaisa Väänänen, Tesh Goyal, Per Ola Kristensson, Anicia Peters, Stefanie Mueller, Julie R. Williamson, and Max L. Wilson (eds.), Proceedings of the 2023 CHT Conference on Human Factors in Comput- ing Systems, CHI 2023, Hamburg, Germany, April 23-28, 2023, pp. 249:1-249:16. ACM, 2023d. doi: 10.1145/3544548..0 358554444357777 URL https://ddi. https////dooooo/////e/e 10. 11w.o.1441/33448.... 35844447888881278
Qifan Wang, Yi Fang, Anirudh Ravula, Fuli Feng, Xiaojun Quan, and Dongfang Liu. Webformer: The web-page transformer for structure information extraction. In Frédérique Laforest, Raphaël Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (eds.), WWW '22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 29, 2022, pp. 3124-3138 ACM, 2022b. doi: 10.1145/34865447:3522200 2032. URL https://doi org/ 10. 11 1145/3485447 455144557777313332222
Shangwen Wang, Ming Wen, Bo Lin, Yepang Liu, Tegawendé F. Bissyandé, and Xiaoguang Mao. Pre- implementation method name prediction for object-oriented programming. ACM Trans. Softw. Eng. Methodol, 32)),, sep 2023e. ISSN 1049-331X. doi: 10.1145/359720 URT, https://doii ftttp::/ddi..org/00.. 1145/ 3597203
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. CoRR,
2020c. URL /tttps://axxiv. http://arxiv..rg/bbbb////s// org/ 04768
Song Wang, Devin Chollak, Dana Movshovitz- Attias, and Lin Tan, Bugram: bug detection with n-gram language models. In David Lo, Sven Apel, and Sarfraz Khurshid (eds.), Proceedings of the 31st IEEE/A AAC International Conference on Aulomated Software Engineering, ASE 2016, Singopore, Seplember 3-7, 2016, pp. 708 019- ACM, 2016a. doi: 090444//9777766.99999 2970341
URL https://doi. org/10. 1145/2970276.
Song Wang, Taiyue Liu, and Lin Tan. Automatically lcarning semantic features for defect prediction. In Laura K. Dillon, Willem Visser, and Laurie A. Williams (eds.), Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA May 14-22, 2016, pp. 297-308. ACM, 2016b. doi: 10.1145 /11145/2844881111 2884471.. URL attpps///oot org/10../// 10. 1141 2284//88888.... 2884804
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Ballimore, Maryland, USA, volume 162 of Proceedings of Muchine Learning Research, pp. 22964- 22984. PMLR, 2022c. URL sttps://proceeddnng mlr. www.mmrr.esss.. wwtppp/eeee/aaaeeepeeeeeeee
Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. Detecting code clones with graph neural net- work and llow-augmented abstract synlax Iree. In Kostas Kontogiannis, Foutse Khomh, Alexander Chatzigeorgion, Marios Eleftherios Fokaefs, and Minghui Zhou (eds.), 27th IEEE International Confer- ence on Software Analysis, Evolution and Reengineering, SANER 2020, London, ON, Canada, Febru- ary 18-21, 2020, pp. 261-271. IEEE, 2020d. doi: 10.1109/SANER440 2.2.....000000555557. URL https: 7/doi. www.1111/0//992//// 2020. 99548577
101

---

Published in Transactions on Machine Learning Research (09/2024)
Wenhua Wang, Yuqun Zhang, Zhengran Zeng, and Guandong Xu. Trans 3: A waasformer-based framework for unifying code summarization and code search. CoRR, abs/2003.0322 2020e. URL https: //arxiv.
org/abs/2003.co 03238
Xin Wang, Yasheng Wang, Fei Mi, Pingyi Zhou, Yao Wan, Xiao Liu, Li Li, Hao Wu, Jin Liu, and Xin Jiang. Syncobert: Syntax-guided multi-modal contrastive pre-training for code representation, 2021d.
Xin Wang, Yasheng Wang, Yao Wan, Fei Mi, Yitong Ti, Pingyi Zhou, Jin Liu, Hao Wu, Xin Jiang, and Qun Liu. Compilable neural code generation with compiler feedback. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Compulational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 9-19. Association for Computational Linguistics, 2022d. doi: URL htttss://oo. www.:/i...rr..... 10/.//....66/vvvv 11111//66//222222 (tt.22222....222ni findings-acl.
Xin Wang, Yasheng Wang, Yao Wan, Jiawei Wang, Pingyi Zhou, Li Li, Hao Wu, and Jin Liu. CODE- MVP: learning to represent source code from multiple views with contrastive pre-training. In Marine Carpuat, Mario- Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Findings of the Association for Compulational Linguislics: NAACL 2022, Seallle, WA, Uniled States, July 10-15, 2022, pp. 1066- 1077. Association for Computational Linguisties, 2022e. doi: 10.18653/v1 (2/03//v12222...// //20022.inndiggg--eee..aaa naacI.80. URL https://doi. .oo.oogg///// 10. 18..3//102/ f findings - naacl. .0
Xingyao Wang, Hao Peng, Reyhaneh Jabbarvand, and Heng Ji. Leti: Learning to generate from textual interactions. CoRR, 0bs//2355.11311 2023f. doi: 10.48550/ AAR00//RRR..... A 1031.1 URL /tttp::/doi.org/ 10. 1.8550/arXiv.2555 10314
Xuheng Wang, Xu Zhang, Liqun Li, Shilin He, Hongyu Zhang, Yudong Liu, Lingling Zheng, Yu Kang, Qing- wei Lin, Yingnong Dang, Saravanakumar Rajmohan, and Dongmei Zhang. SPINE: a scalable log parser with feedback guidance. In Abhik Roychoudhury, Cristian Cadar, and Miryung Kim (eds.), Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022. Singapore, Singapore, November 14-18, 2022, pp. 1198-1208 ACM, 2022f. doi: 10.1145/3540250.5555
URL https:///oi..org 00w.l..1111335555.00833 3549176
Yanlin Wang, Eusheng Shi, Lun Du, Xiaodi Yang, Yuxuan Hu, Shi Han, Hongyu Zhang, and Dongmei Zhang. Cocosum: Contextual code summarization with multi-relational graph neural network. CoRR, abs/2107.019333 2021c. URL htttgs://aaxxx..or//ab//////// 01933
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjum Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Gi- annis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Mailreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Puro- hit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Sa- van Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Superr-naturalinsttrcdde
Generalization via declarative instructions on 1600- NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirotes, December 7-11, 2022, pp. 5085-5109, Association for Computational Linguisties, 2022g. URL https: haala///hotnnnoooog/..om org/ 2022.02222222222222 emnlp-main. 340
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jor- dan L. Boyd-Graber, and Naoaki Okazaki (cds.), Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), ACL 2023, Toronlo, Canada, July 9-14, 2023, pp. 13484-1228 13508. Association for Computational Linguistics. 2023g. doi: 10. 111683/111202255 acl-long, 754. URL httts:://oi..oo org/ 10. 18653/v1/2022 http:////222..1111111e acl-lcog.775 754
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-awwr aware unified pre-trained encoder-dccoder models for code understanding and generation. In Maric-Francin Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (ods.), Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, FMNLP 2021, Virtual Event / Punta Cana, Dominican
102

---

/111//22..FNDINNGCCC Published in Transactions on Machine Learning Research (09/2024)
Republic, 7-11 November, 2021, pp. 8696-8708. Association for Computational Linguistics, 2021f. doi:
10. 1653/v1 men.pp--.nnl--..66.... URL https:///// htttop////oo..rrr///0111331//111 ean21..mmnpppp. emnlp-maing 685
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. +odet5+: Open code large language models for code understanding and generation. CoRR, abs Abss/2305.rr /205.07922, 2023h. doi: /8.48550/aal arxiv. 2305.07922. URL https:://oi..rr::: http://...00../10..04444//////1 2305. 07922.
Zan Wang, Ming Yan, Junjie Chen, Shuang Liu, and Dongdi Zhang. Deep learning library testing via effective model generation. In Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (eds.), ESEC/FSI 20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, pp. 788 799. ACM, 2020f. doi: 10.1145/3368083 3409761. URL https://doi http:///oooorgg///// 00w...1111/33381...1. 3409777
Zheng Wang and Michael F. P. O'Boyle. Machine learning in compiler optimisation. CoRR, abs/ 1805.03441, 2018. URL httl:tttp:/aat http:p//aaxvv....o www.aawwwwww.../////a/aeeee/ee/e 03441
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution-based evaluation for open- domain code generation. CoRR, 2221222.220488, 2022h. doi: D.44650//RXXV..2122.11 URL https: //doi. 40t5/////..55///xx....... 221.2212222222 10481
Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F. Xu, and Graham Neubig. Mconala: A benchmark for code generation from multiple natural languages. In Andreas Vlachos and Isabelle Augenstein (eds.), Find- ings of the Association for Computational Linguistics: EACL 2023, Dubrounik, Croatia, May 2-6, 2023, pp. 265-22 273. Association for Computational Linguistios, 2023i. doi: 10.
URL https:://ol.. ttttps://aoi.oor//0..
2006://///2233.ffndng----- findings-eacl... 20
Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. On learning mean- ingful assert statements for unit test cases. In Gregg Rothermel and Doo-Hwan Bae (eds.), ICSE '20: 42nd International Conference on Software Engineering, Seoul, South Korea, 27 June- - 19 July, 2020, pp. 1398 1409. ACM, 2020. doi: 10.1145/3377 01////377811133380455 URL htttp:://doi.rre/ 10. 1145/333771111 3380429
Anjiang Wei, Yinlin Deng, Chenyuan Yang, and Lingming Zhang. Free lunch for testing: Fuzzing deep- learning libraries from open source. In 44th IEEE/A ACM CM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 995-1007. ACM, 2022a. doi: URL https://doi org/10.. 1.....145/300 100.111//511003.3335 35100441
1..1145/3510003.3510000
Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu,, and Zhi Jin. Code generation a dual task of code summariza- tion. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bue, Emily B. Fox, and Roman Garnett (eds.), Advances in Newral Information Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 6559 6569, 2019. URL htttps://proceeee 652x069177511199944444707 www.erdreeeccc-saaaat.tt........ html
Huihui Wei and Ming Li. Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in souree code. In Carles Sicrra (ed.), Proceedings of the Twenty-Siath International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp. 3034 30400 ijcai.org, 2017, doi: (0.24963//jccai.200) 423.. URL https://doi. http::///ooooor/100/ 10. 00./10...33/111 24963/ ijcai 2017/423
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zcro-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virlual Event, April 25-29, 2022. OpenReview.ned 2022b. URL
http:pprreevee..eeffforrf..m nww.forummmmetem. ? 1i::id=EEeaa -dddEEgrrozzzgg...
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fodus. Emergent abilitics of large language models. Transactions on Ma- chine Learning Research, 2022c. ISSN 2835-8856. URL Survey Certification.
nee//forurr/dwyyzkssszahd
103

---

Published in Transactions on Machine Learning Research (09/2024)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thougtt prompting elicits reasoning in large lan- guage models. In NewrlPS, 2022d. URL http::/paapers.. mips. cc/ppcr httppppprr..illeeppppe////////// hash/
001600661520o0444141107777711111 http:///saacaaa-staaaa---------a Abstract- Conbrract-Conference html
Jiayi Wei, Maruth Goyal, Greg Durrett, and Isil Dillig. Lambdanet: Probabilistic type inference using graph neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL.
Hkx6hANtwHl
Jiayi Wei, Greg Durrett, and isil Dillig. Typet5: Seq2seq type inference using static analysis. In The Eleventh International Conference on Learning Representations, JCLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL http:///opppooooeeooooooo/adaa//
Yuanbo Wen, Qi Guo, Qiang Fu, Xiaqing Li, Jianxing Xu, Yanlin Tang, Yongwei Zhao, Xing Hu, Zidong Du, Ling Li, Chao Wang, Xuehai Zhou, and Yunji Chen, Babeltower: Learning to auto-paralleized program translation. In Kamalika Chaudhuri, Stefanie Jegelka, Te Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, 1CML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 23685 23700. PMLR, 2022. URL https::/prroeeddings .ml. ppress/v162/oe www.d//wee2lbb.eee html
Martin Weyssow, Houari A. Sahraoui, and Eugene Syriani. Recommending metamodel concepts during modeling activities with pre-trained language models. Softw. Syst. Model, 1371 2022. doi: (20000//110070002200099777
URL https://doi. /ttp:::///iii.r/11....... 1000 http:/00011000000000000000000000
Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C. Schmidt. Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. CoRR, org/10. 48550/arXiv.
0bss/2333007833 2023. doi: 10.48550/ARXII 2303.07839. URL 2303.012 07839
Martin White, Christopher Vendome, Mario Linares Vásquez, and Denys Poshyvanyk. Toward deep learning software repositories. In Massimíliano Di Penta, Martin Pinzger, and Romain Robbes (ods.), 12th (BEE/ACM, Working Conference on Mining Software Repositories, MSR 2015, Florence, Italy, May 16-17, 2015, pp. 334-345. IEEE Computer Society, 2015. doi: 10.1109/MSR.2015..33 URL https://dot org/ 10. 1109/MSR. 2015.38
Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. Deep learning code frag- ments for code clone detection. In David Lo, Sven Apel, and Sarfraz Khurshid (ods.), Proceedings of the 31st IEEE/ACM International Conference Automated Software Engineering, ASE 2016, Sin- gapore, September 3-7, 2016, pp. 87-98. ACM, 2016. doi: 00.144//990276..970088. URL https: //doi.orr/// wtt.:o////....////..... 1141 1145/297027766666 2970326
Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yich, Brian Goh, Ferdian Thung, Hong Jin Kang, Thong Hoang, David Lo, and Eng Lieh Ouh. Bugsinpy: database of existing bugs in python programs to enable controlled testing and debugging studies. In Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (eds.), ESEC/FSE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, pp. 1556-1560. ACM, 2020. doi: 10.1145/ 33368089.3417999 URL https://dooiorrg// 111.///0011113388889999999999 34179433
Ronald J. Williams. Simple statistical gradient algorithms for wwn.necciooiisteeiiineeeeeee Mach. Learn., 8:229 256, 1992. doi: 10. FBF000/900009926666. URL http///aps///aa...rr///////////0
learning.
Edmund Wong, Jinqiu Yang, and Lin Tan. Autocomment: Mining question and answer sites for au- tomatic comment generation. In Ewen Denney, Tevfik Bultan, and Andreas Zeller (eds.), 2013 28th IEEE/ACM International Conference on Automated Software Engineering, ASE 2013, Silicon Valley, CA, USA, November 11-15, 2013, pp. 562-567. IEEE, 2013. doi: ALL109/ASE.2113.69990 https://doi. wrw.ll.rg/goo.. 1109/ ASE. 6693113
URL
104

---

Published in Transactions on Machine Learning Research (09/2024)
Edmund Wong, Taiyue Liu, and Lin Tan. Clocom: Mining existing source code for automatic comment gener- ation. In Yann-Gak Guéhéneuc, Bram Adams, and Alexander Serebrenik (eds.), 22nd IEFE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2015, Montreal, QC, Canada, March 2-6, 2015, pp. 380. IEEE Computer Society, 2015. doi: 10.1109/SANEBI 201.. 7011888 URL httpss//dooi org/ 10. 1109/ 2015. 7081848
Wai Kin Wong, Huaijin Wang. Zongjie Li, Zhibo Liu, Shuai Wang, Qiyi Tang, Sen Nie, and Shi Wu. Refining decompiled C code with large language models. CoRR, abs abbs/231.006530 2023. doi: 10.48550/ ARXIV .3100. 06530. URL https://doi. org/ 10. 148550/arXivv www.aawwww......... 06530
Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. Plot2code: A comprchensive benchmark for evaluating multi-modal large language models in code generation from scientific plots. 2024a. URL btttps://doi.org/// 48550/arXiv 0400500799)
Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. Repoformer: Selective retrieval for repository-level code completion. CoRR, abs/2403.10055 2024b. doi: 10.48550/ ARXIV.2403.1005. URL https:://ao..org/100
2403. 10059
Hongqiu Wu, Hai Zhao, and Min Zhang. Code summarization with irruttur-induued transformer. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Com- putational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pp. 1078-1090. Association for Computational Linguisties, 2021. doi: 10. 10.1 00053/v1/202/ findings-acl.9. URL httts::///aet http:://oooo.o/1.1.o11/ 1t10//11..163/11222...1...1. fiiiiiigsnndiiciiiii
Jie JW Wu and Fatemch H Fard. Benchmarking the communication competence of code gencration for 1lms and Ilm agent. arXiv preprint 000:2406.0021,, 2024. URL 00215
118650/arXiv... arXiv.2406.
Ming Wu, Pengcheng Wang, Kangqi Yin, Haoyu Cheng, Yun Xu, and Chanchal K. Roy. Lvmapper: A largo-variance clone detector using sequencing alignment approach. IEEE Access, (877986277997 2020.
doi: 10. 10.1109/AM 10//0//CEESS.... AC2CECCS20..001TTT.. 2971.297 URL https://aoi.org/ 1109/ACCESS. 2020 2971545
Mingyuan Wu, Ling Jiang, Jiahong Xiang, Yuqun Zhang, Guowei Yang, Huixin Ma, Sen Nie, Shi Wu, Heming Cui, and Lingming Zhang. Evaluating and improving neural program-smoothhngg--ee based fuzzing, In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp. 847-858. ACM, 2022. doi: 00.145/351000033510000 URL. https://doi. org/ 10.
3510089
Chunqiu Steven Xia and Lingming Zhang, Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Abhik Roychoudhur, Cristian Cadar, and Miryung Kim (eds.), Proceed- ings of the 30th ACM Joint European Software Engineering Conference and Symposium the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, pp. 959-971. ACM, 2022. doi: 01144//554250.35445
URL https:///oi.org wtt.oo/.../////...../ 0001111155355555 35491001
Chunqiu Steven Xia and Lingming Zhang. Conversational automated program repair. CoRR, Abs/2231.13344/ 2023. doi: (18550/ARRXV.2301111 2301.13245
URL https://doi. http://///oooo//1...../ org//0.485500000. 48550/arXiv.
Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. Fuzz4all: Univer- sal fuzzing with large language models. CoRR, abs abs/2308 2200..00444 2023a. doi: 10.48550/ 10.48350/ARXIT /ARXIV 2308.04748. URL https://doi.org: 10. 48550/arXiv. 2308 2308.04748
Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era large pre- trained language models. In 45th TEFE/ACK International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 1482-1494. IEEE, 2023b. doi: 10.1109/ICSEA8619. 2023.00129. URL http::://oi..rrg///// /t///oi..rrg//0// 1ttp://...10//1//481186111111111 2023.00129
105

---

Published in Transactions on Machine Learning Research (09/2024)
Chenhao Xie, Wenhao Huang, Jiaqing Liang, Chengsong Huang, and Yanghua Xiao. Webke: Knowledge extraction from semi-structured web with pre-trained markup language model. In Gianluca Demar- tini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (eds.), CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 5, 2021, pp. 12200. ACM, 2021a. doi: 10.1145/3459037.34482222 3482449.. URL https: org/ 10. 1111/115595331 3482491
Danning Xie, Yitong Li, Mijung Kim, Hung Viet Pham, Lin Tan, Xiangyu Zhang, and Michael W. Godfrey, Docter: documentatiion-guidee fuzzing for testing deep learning API functions. In Sukyoung Ryu and Yannis Smaragdakis (eds.), ISSTA '22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, South Korea, July 18 22, 2022, pp. 176-188. ACM, 2022a. doi: 10.1145/ 3533767 3534220. URL https: http:////oo.rr///.... 1145/3533766 3534220
Rui Xie, Wei Ye, Jinan Sun, and Shikun Zhang. Exploiting method names to improve code summarization: A deliberation multi-task learning approach. In 29th IEEE/ACE International Conference on Program Comprehension, ICPC 2021, Madrid, Spain, May 20-21, 2021, pp. 138-148. TEEE, 2021b. doi: 10.1109/ 20022000022
SLPC5228812021.00002 URL
Rui Xie, Zhengran Zeng, Zhuohao Yu, Chang Gao, Shikun Zhang, and Wei Ye. Codeshell technical report. CoRR, abs/ 2244311577, 2024. doi: 10.48550/ARXTV.244000 2403..577.7 URI. https://doi.orr 48550/
arXiv 15747
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida 1. Wang. Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radov, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zelulemoyer, and Tao Yu. Unifiedskg: Unifying and multi-lasking structured knowledge grounding with text-to-text language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 602-631. Association for Computational Linguistics, 2022b. doi: 10. 10.18853/vv, 20/533/112022.em/// http:/2222eeemmmmmme mainn..9.ma.......mm URL https://::://at http:///ooooooog.....1o.. 11551/000.16663111222222 emnlp-main. 39
Yutao Xie, Jiayi Lin, Hande Dong, Lei Zhang, and Zhonghai Wu. A survey of deep code search. CoRR, fos//205..05500 2023a. doi: 10.48550/arXin 2305000000 URL /tttps:://oi.org//00.. 40555//rri.55// 2305. 05959
Zhuokui Xie, Yinghao Chen, Chen Zhi, Shuiguang Deng, and Jianwei Yin. Chatunitest: a chatgpt-based automated unit test generation tool. CoRR, abs/2305, //////355006644 2023b. doi: 10.48350/ARXIV .2305.04744
URL https://doi.
48550/arXiv 2305 04764
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024. URL https:://openreviee het/forrumiid=CFhheeee
Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large
language models of code. In Swarat Chaudhuri and Charles Sutton (eds.), MAPS@PLDI 2022: 6th ACM SIGPLAN International Symposium on Muchine Programming, San Diego, CA, USA, 13 June 2022, pp. 1 1-10. ACM, 2022. doi: (0.1145/35203111 ///33/5222233445666 URL https://doi. org/10. 1 /144/3520000 3534862
Ling Xu, Huanhuan Yang, Chao Liu, Jianhang Shuai, Meng Yan, Yan Lei, and Zhou Xu. Two-stage altention- based model for code search with textual and structural features. In 28th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2021, Honolulu, HI, USA, March 9-12, 2021, pp. 342 353. IEEE, 2021a. doi: 10.1109/SANER55067..2 2021.00033 URL https:///oi. wtt.d/i..or/...1000.000 SANER50967 2021 00039.
106

---

Published in Transactions on Machine Learning Research (09/2024)
Shengbin Xu, Yuan Yao, Feng Xu, Tianxiao Gu, Hanghang Tong, and Jian Lu. Commit message generation International Joint
for source code changes. In Sarit Kraus (ed.), Proceedings of the Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 3975 3981. ijcai.org, 2019a. doi: 10.24963 10.24963/IJCA12220 /IICAAIIII.119/555. URL https:://oi.org/ 10. 20922/22443322 hica//ijaaiia// 2019/552
Sihan Xu, Sen Zhang, Weijing Wang, Xinya Cao, Chenkai Guo, and Jing Xu. Method name suggestion with hicrarchical attention networks. In Manuel V. Hermenegildo and Atsushi Igarashi (eds.), Proceedings of the 2019 ACM SIGPLAN Workshop on Purtial Evaluation and Program Maniipuaation, PEPM@POPL 2019, Cascais, Portugal, Janwary 14-15, 2019, pp. 10 21. ACM, 2019b. doi: (11114//399032.32200 URL https: org/10 1145 /ttp//32243322299999
Xiangzhe Xu, Zhuo Zhang, Shiwei Feng, Yapeng Ye, Zian Su, Nan Jiang, Siyuan Cheng, Lin Tan, and Xiangyu Zhang. Lmpa: Improving decompilation by synergy of large language model and program analysis. CoRR, abs/2306..r 02546, 2023. doi: 10.48550/ ARXIV 2306.02246.. URL :ttt::://doi.org11...4555
arXiv 02546)
Xiaojun Xu, Chang Liu, and Dawn Song. Sqinet: Generating structured queries from natural language without reinforcement learning. CoRR,
2017. URL htttp::/aaxxi..rgg/bbbbs. 21t1///0//11..044666 04436
Yichen Xu and Yanqiao Zhu. A survey on pretrained language models for neural code intelligence. CoRR, (bs//2212.1079, 2022. doi: 10. 208550/arXi.:2222.10000 URL https:///oi.orr//00 48550/ (03350//rXiv... arXiv.2212.
10079
Zhaogui Xu, Xiangyu Zhang, Lin Chen, Kexin Pei, and Baowen Xu. Python probabilistic type inference with natural language support. In Thomas Zimmermann, Jane Cleland-Huang, and Zhendong Su (eds.), Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Sofbware Engineering, FSE 2016, Seattle, WA, USA, November 13-18, 2016, pp. 607-618. ACM, 2016. doi: 00.1145/2950000 2950343. URL https:://do:::
Zhifeng Xu, Xianjin Fang, and Gaoming Yang. Malbert: A novel pre-training method for malware detection. Comput. Secur., 111:102458, 2021b. doi: 10. /006//J.COSF.2201110242555 URI.httss:///ao http::////.o..o//...ooe./ orgg/...111...
J.cose.208 102458
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 483 498. Association for Computational Linguistics, 2021. doi: 10. 18653/v1 12/0//11222111// maaec-mmai..4... URL https:://oii http://oooorrg/1// or./ 100...1111////12211.. aaaacc-main...
Navid Yaghmazadch, Yuepong Wang, Isil Dillig, and Thomas Dillig. Sqlizer: query synthesis from natural language. Proc. ACM Program. Lang., ((0OPSLA))::: 63:26, 2017. doi: 10. 00.1145/313888 URL https:
//doi.org 10. 1145/3133887
Mohammad A. Yahya and Dae-Kyoo Kim. Cross-language source code clone detection using deep learning with infercode. CoRR, abs/2200 22//22255049913 2022. doi: 10.48550/ar 2205.044900 URL https://doi.co /rrg////o/.o//////////////
10.48550/arXiv 2205. 04913.
Shuhan Yan, Hang Yu, Yuting Chen, Beijun Shen, and Lingxiao Jiang. Are the code snippets what we are scarching for? A benchmark and an empirical study on code scarch with natural-language queries. In Kostas Kontogiannis, Foutse Khomh, Alexander Chatzigeorgiou, Marios-Eleftherics Fokaefs, and Minghui Zhou (eds.), 27th TEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2020, London, ON, Canada, February 18-21, 2020, pp. 344 354. IEEE, 2020. doi: 10.1109/ STWTERR4275520209955424
URL /attss://aoi.org/ /01 11199/ANERA8225.... 2020. 90548400
Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. Codetransoccan: A comprehensive multilingual benchmark for code translation. CORR, abs/2310.04951, 2023. doi: 10.48550/ ARXIV .3100 04951. URL https://doi.org:: org/ (8850//rXivv. wtt.xii..231.009955
107

---

Published in Transactions on Machine Learning Research (09/2024)
Aiyua Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nic, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuchai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. Baichuan 2: Open large- scale language models. CoRR, abs /2309. 10309, 2023a. doi: 10.48550/ /ar500///ivv..0 2309.10305. 7/doi org/10 48550/arX11 2309. 10305
URL https: :
Chen Yang, Junjie Chen, Bin Lin, Jianyi Zhou, and Ziqi Wang. Enhancing Imm-based test generation for hard- to-cover branches via program analysis. CoRR, abs/2404. 2404.09966 2024a. doi: 10.48550/AA ARXIV 2404.04966. URL btttps://doi.org 10. 48550/ (08550/arXiv... arriv. 2404. 04966
Chenyuan Yang, Yinlin Deng, Runyu Lu, Jiayi Yao, Jiawei Liu, Reyhanch Jabbarvand, and Lingming Zhang. White-box compiler fuzzing empowered by large language models. CoRR, abs/2310. 2023b. doi: 10.48550/ ARX.V 1599. URL https://doii. 3Tgg/0..485500 aar.://rXi.v... http//aaavvvvv........ 15991
Chenyuan Yang, Yinlin Deng, Jiayi Yao, Yuxing Tu, Hanchi Li, and Lingming Zhang. Fuzzing auto- matic differentiation in deep-learning libraries. In 45th IEEE/ACM International Conference on Soft- ware Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp. 1174 1186. IEEE, 2023c. doi: URL https:://ol..ooo org/ 10tg//1...10 1.00.1100 1111 IIW.//ICSEE669.... 2ww.:/6661..2........ 00105
1LL109//CCBE48812200233.3
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. In Alice Oh, Tristan Naunann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advonces in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023d. URL httt::/ aww.addddndnaaaeee.aaaaaa
faceeer_file/ppeeeee wwtps///pppp///22///aaaaaa hash//
html
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engincering. CoRR, /t///2405.15799, 2024b. doi:
48550/arXiv. URL https::/doi.orgg/ wrt.dd..orrr....455//rr.....
Wei Yang, Peng Xu, and Yanshuai Cao. Hierarchical neural data synthesis for semantic parsing. CoRR, abs/2112.02212, 2021. URL http:///aaxiv..o/aabaa///sa/ 02212
Xianjun Yang, Kexin Zhang, Haifeng Chen, Linda R. Petzold, William Yang Wang, and Wei Cheng. Zero- shot detection of machine-gencrated codes. CoRR, abs/ 2310.00100, 2023c. doi: 10.48550/ARB ARRIV 23100. 05103. URL https: http::///ooooog//1/// 10. 133500/aXii.... www.riv..ii1.....m 2310.05103
Yanping Yang, Ling Xu, Meng Yan, Zhou Xu, and Zhongyang Deng. A naming pattern based approach for method name recommendation. In IEEE 33rd International Symposium Software Reliability En- gineering, ISSRE 2022, Charlotte, NC, USA, October 31 Nov. 3, 2022, pp. 344-354. IEEE, 2022. doi: 10.1109//SSRE55666 0ISSSREE55669.222200000 2022.00011. URL htttps://dan :tttps://doi.oor/1100// 10. 101111999/SS55999 00041
Ziyu Yao, Daniel S. Weld, Wei-Peng Chen, and Huan Sun. Stage: A systematically mined question-code dataset from stack overflow. In Pierre- Antoine Champin, Fabien Gandon, Mounia Lalmas, and Panagio- tis G. Ipcirotis (ods.), Proceedings of the 2018 World Wide Web Conference World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018, pp. 1693 1703. ACM, 2018. doi: 10. 111144/317888663186008 URL 10. 111.1114/117887663333333 3186081
Michihiro Yasunaga and Percy Liang. Graph- self-supervised program repair from diagnostic feedback. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 19-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 10799-10800 PMLR, 2020. URL http:://tooceeddnnnnt//// mlr. htt.a/rrreee//11/yaaaaagggge html
108

---

Published in Transactions on Machine Learning Research (09/2024)
Michihiro Yasunaga and Percy Liang. Break-it-fix-it: Unsupervised learning for program repair. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, html
pp. 11941-11952. PMLR, 2021. URL http://proccedings..aa mmlrr mlr.
Ming-Ho Yee and Arjun Guha. Do machine learning models produce typescript types that type check? In Karim Ali and Guido Salvaneschi (eds.), 37th European Conference on Object-Oriented Oriented Programming, ECOOP 2023, July 17-21, 2023, Seattle, Washington, United States, volume 263 of LIPIcs, pp. 37:1- 37:28. Schloss Dagstuhl Leibnniz-Zenrrm für Informatik, 2023. doi: (.4230/LIPIcs.FCOOPP... URL //ttpa org/10. 4230 ECOOP 2023.37
Burak Yetistiren, Isik Özsay, Miray Ayerdem, and Fray Tüzün. Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt. CoRR, abs/2304.107773 2023. doi: 10.48550/ARX5 2074..7800.. 10778
URL https://doi. org/10. wtw.l....///////////..//////..// 48550/arXiv.
Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL, 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 440 450. Association for Computational Linguistics, 2017. doi: 10.18653/v1/PI7--004 URL https://doi.. 1ttp://0.00053////111000000
Pengeheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. Learning to mine aligned code and natural language pairs from stack overflow. In Andy Zaidman, Yasutaka Kamei, and Emily Hill (eds.). Proceedings of the 15th International Conference on Mining Software Repositories, MSR 2018, Gothenburg, Sweden, May 28-29, 2018, pp. 476 176-486. ACM, 2018. doi: 10. 11144/1196388839666666 URL https://dom oro.oorgg//10/ 11t.11191/331999888 3196408
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint un- derstanding of textual and tabular data. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Anneal Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 8413-8426. Association for Computational Linguistics, 2020. doi: URL httppp://oo..rrg///0./66//////// http:/2222....mmaaaaa......aam acl-main. 745
10.18653/ 10.18653/V1/2022 /11/2020.AAAIMAATTTTT
Ying Yin, Yuhai Zhao, Yiming Sun, and Chen Chen. Automatic code review by learning the structure information of code graph. Sensors, 23(5):2551, 2023.
doi: 10.3390/:230525511 URL httpp:///oi.... org/
10. 00.3390/s23052257)
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by Ol.ii. CoRR, 0ee//2433.04665 2024. doi: 10.48550/ARMM ARXSI 2403.04652. URL https://doi.org: 48550/ aarsvv./rxiv.... 2403. 2403.04652
Hao Yu, Wing Lam, Long Chen, Ge Li, Tao Xie, and Qianxiang Wang. Neural detection of semantic code clones via treo-basod convolution. In Yann-Gaäl Guéhéncue, Foutse Khomh, and Foderica Sarro (ods.). Proceedings of the 27th International Conference on Program Comprehension, ICPC 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 70 80. TEFE/ / ACM, 2019a. doi: 10.118//ICPC.2019.000 //doi. o:c/ioorg111.0 M111/19IIPPPP 2019 00021
URL https:
Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Tao Xie, and Qianxiang Wang. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. CoRR, abs/ has/2302.0002 00288, 2023a. doi: 10.48550/arXiv.28000 48550/arXiv. 2302.00288 00288
URL attps://doi.org//
Siyu Yu, Ningjiang Chen, Yifan Wu, and Wensheng Dou. Self-supervised log parsing using semantic contribution difference. J. Syst. Softw., 200:111646, 2023b. doi: 10.1016/J.Jss. 202..02...... 111646. URL
https://bbi..or//1....uu
2023. 111646
109

---

Published in Transactions on Machine Learning Research (09/2024)
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir R. Radev. Typesql: Knowledge-basd based type-aware neural text-to-sql generation. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLI New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pp. 588 594. Association for Computational Linguistics, 2018a. doi: 2..18653/v1n1855 2093. URL https://doi. org/ 10. 11153///1/111///02
Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir R. Radev. Syntaxsqinet: Syntax tree networks for complex and cross-domain text-to-sql task. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsuji (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, Oclober 31 NNvember 4, 2018, pp. 1653 1663. Association for Computational Linguistics, 2018b, doi: 10.18653/vvlb 111/653//11/18-19933 URL https://doi. wwt.:/116//111---11
Tao Yu, Rui Zhang, Kai Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-lo-sql task. In Ellen RilofT, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 3911-39211 Association for Computational Linguistics, 2018c. doi: 10.18653/v1 1425. URL 10. 18051881//1111111123
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander R. Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter S. Lasccki, and Dragomir R. Radev. Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases. In Kentaro Tnui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, SMNLP-IJCNLL 2019, Hong Kong, China, November 3-7, 2019. pp. 1979. Association for Computational Linguistios, 2019b. doi: 10.18653/v1/D19) /D111-1204 URL https://doi.oon htttes///ooo.orr///000006//////0
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir R. Radev. Sparc: Cross-domain semantic parsing in context. In Anna Korhonen, David R. Traum, and Tluís Mârquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August Association for Computational Linguistics, 2019c. doi:
2, 2019, Volume 1: Long Papers, pp. 10. N0.1653//v1/ple 1443. URL https://doi org/10. 18653/v1/p19-1493
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang. Yi Chern Tan, Xinyi Yang. Dragomir R. Radev, Richard Socher, and Caiming Xiong- Grappa: Grammar-angmentted pre-training for table semantic pars- ing. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL /ttps://openreviet http:///ppppooooooooeooooooooooo
Xiaojing Yu, Tianlong Chen, Zhengjic Yu, Huiyu Li, Yang Yang, Xiaoqian Jiang, and Anxiao Jiang. Datasel and euhanced model for eligibility criteria-lo-sql semantic parsing. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isa- hara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Mar- seille, France, May 11-16, 2020, pp. https://aclanthooogy..oo www.gyorrg//200/0 1ttp//2222.1ee-...11 1rec-1.711/
Furopean Language Resources Association, 2020. URT.
Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation. CoRR, abs/23112 /ts//2211.4187, 2023c. doi: 10.48550/ARXII A2X.5//./AR 1412. arXiv 2312. 141817
URL https://doi. www.oo...ggo...000 48550/
110

---

Published in Transactions on Machine Learning Research (09/2024)
Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, and Yiling Lou. Evaluating instruction- tuned large language models on code comprehension and generation. CoRR, abs/ /2/2222000020000 2023a. doi: 10.48550/arka www..........1..... 2308.0124.. URL httpp:///oi..orgg// 1.00.48550/rr/// 40w.40///X1..200..1112223 2308..1240
Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. No more manual tests? evaluating and improving chatgpt for unit test generation. CoRR, abs/ / 2305.04207, 2023b.
doi: 10.48550/ 1..4850//ARIIV ARXIV 2305.04207. URL https://doi wttpe/....111.. 00t5///0..855//000 (88500/arX11 2305 04207
Nusrat Zahan, Philipp Burckhardt, Mikola Lysenko, Feross Aboukhadijeh, and Laurie A. Williams. Shifting the lens: Detecting malware in npm ecosystem with large language models. CoRR, (abs//243.12209 2024.
doi: 10.48550/ ARXTV 24403121960 URL httpp:://ddd org/://///o..oo/////. www..../////////a....////////// arrvvv 2403. 12196
Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Hlan-Guane Lou. CERT: continual pre-training on sketches for library-oriented code generation. In Luc De Raedt (ed.), Proceedings of the Cuirty-First International Joint Conference on Artíficial Intelligence, IJ- CAI 2022, Vienna, Austria, 23-29 July 2022, pp. 2369-2377 ijeai.org, 2022. doi: 10.24963 1..24463//jcai.... (ic.ai.202////.... URL attps:///dos org/10 10. 24963/2939939 /itp://ijaaaaaa 2022/329
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Yongji Wang, and Jian-Guang Lou. Large language models meet nl2code: A survey. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 7443-7464 Association for Computational Linguistics, 2023. doi: 10.18653/v1 http///1/2223.//// acllogg.... URL https://doi. org/ 10.
18653/v1/2023 http:///1/2222...1111110 acl-long.... 411
Daoguang Zan, Ailun Yu, Wei Liu, Dong Chen, Bo Shen, Wei Li, Yafen Yao, Yongshun Gong, Xiaolin Chen, Bei Guan, Zhiguang Yang, Yongji Wang, Qianxiang Wang, and Lizhen Cui. Codes: Natural language code repository via multi-layer sketch. CoRR, 22////243....44 2024. doi: (18550/ARXIV/ 1643.3 URL https:///oo..rrg:
48550 wat.:./riii.... 2403 16443
John M. Zelle and Raymond J. Mooney. Learning to parse database queries using inductive logic program- ming. In William J. Clancey and Daniel S. Weld (eds.), Proceedings of the Thirteenth National Conference on Artificial Intelligence and Eighth Innovative Applications of Artificial Intelligence Conference, AAAI 96, IAAI 96, Portland, Oregon, USA, August 4-8, 1996, Volume 2, pp. 1050- 1055. AAAI Press / The MIT Press, 1996. URL hattp://ww. aaai. org/ wtt/ooorrrrrrlle Www.bbrry/AAAA//// 1990////99///// awwiaa-aa2662 156. 15p.
Jichuan Zeng, Xi Victoria Lin, Steven C. H. Hoi, Richard Socher, Caiming Xiong, Michael R. Lyu, and Irwin King, Photon: robust cross-domain text-to-sql system. In Asli Celikyilmaz and Tsung Hsien Wen (eds.), Proceedings of the 58th Annual Meeting of the Associalion for Compulational Linguistios: System Demonstrations, ACL 2020, Online, July 5-10, 2020, pp. 204 214. Association for Computational URL htttp:://ao..or//10..8888 wtt.oo/1.....1//////2222.
Linguistics, 2020. doi: /111666//V1/2000AA---1 acl-demos. 24
Chenyuan Zhang, Hao Liu, Jiutian Zeng, Kejing Yang, Yuhong Li, and Hui Li. Prompt-enhanced software vulnerability detection using chatgpt. CoRR, abs/ abs/2308. 12697, 2023a. doi: 10.48550/ ARXIV 12697. URL https://doi. org/10. 38550/arXiv> 2308 12697
Chunyan Zhang, Junchao Wang, Qinglei Zhou, Ting Xu, Ke Tang, Hairen Gui, and Fudong Liu. A survey of automatic source code summarization. Symmetry, 14(3):471, 2022. doi: (0.3390/SYM140304777 URL
Dejiao Zhang, Wasi Cddin Ahmad, Ming Tan, Hantian Ding, Ramesh Nallapati, Dan Roth, Xiaofei Ma, and Bing Xiang. Code representation learning at scale. CoRR, abs/2402.01 213,01193., 2024a. doi: 10.48550/ARD ARXIV. 2402.01935. URL /tttps://do..ooo
48550/ arXiv. 9402.019355
Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retricval and gencration. CoRR, (bbs/230..120 2023b. doi: 10.48550/
10.48550/ARXIV ARRIIV 2302. 12570. URL https: attts:///ol.s ://doi, org/10. 48555///r..////
2303. 12570
111

---

Published in Transactions on Machine Learning Research (09/2024)
Haibo Zhang and Kouichi Sakurai. A survey of software clone detection from security perspective. 1EEE 201.306887.... URL https://doi. ott//0..109/..
Access, 0:48157-48177 2021. doi:
2021.
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixnan Wang, and Xudong Liu. A novel neural source code representation based on abstract synlax tree. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle (eds.), Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pp. 783 794. IEEE / ACM, 2019a. doi: 10.1109/ICSE:2219.0000 URL https:///oi. org/ 10. 1119/1109SSSS 2019.00088
Jiansong Zhang and Nora M. Fl-Gohary. Integrating semantic alp and logic reasoning into a unified system for fully-automaated code checking. Automation in Construction, 73:45 57, 2017. ISSN 0926-5805. doi: https:// www.auutto.........0.0. S09265805166008199
Jingxuan Zhang, He Jiang, Zhilei Ren, and Xin Chen. Recommending apis for API related questions in stack overflow. IEEE Access, 6:6205 6219, 2018. doi: 10.1099AACCES8.217..77777 URL https:
//doi. 10. 1109/ 2009//ACESs..o 2017 2777845
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. Self-edit: Fault-aware code editor for code generation. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 769-787. Association for Computational Linguisties, 2023c. doi: 10. 111633//1/2022 ACI-LONG.44. URL https:::: wttp:://///................
Quanjun Zhang, Chunrong Fang, Yuxiang Ma, Weisong Sun, and Zhenyu Chen. survey of learning- based automated program repair. CoRR, bbs/2301.03270, 2023d. doi: 10.48550 444550/arriv...... 2301103277. URL
https://doi.. org. 10. 48550/ arrvvv. 2301 03270
Rui Zhang, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir R. Radev. Editing-based SQL query generation for cross-domain context- dependent questions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Ith International Joint Conference on Naturul Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 5337-5348, Association for Computational Linguistics, 2019b. doi: 10.18653/v1/D19- 1537. URL https://doi.. org/ 10. 11153//1//111-11011
Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user prompts. 2024b. URL htttss://aoi.rrg// 41w5://///../..////////// wttp:5//xxxiv...v0 2405.
Tianzhu Zhang, Han Qiu, Gabricle Castellano, Myriana Rifai, Chung Shue Chen, and Fabio Pianese. System log parsing: A survey. IEEE Trans. Knowl. Dala Eng., 35(8):8596 8614, 2023e. doi: 10.1109/TKDE:202 3222417. URL https:///oi.oooo org/10. 1109/ T022 3222447
Xinyu Zhang, Siddharth Muralee, Sourag Cherupattamoolayii, and Aravind Machiry. On the effectiveness of large language models for github workflows. CoRR, abs/2403. 12446, 2024c. doi: 10. 10.48550/ ARXIV 12446. URL httpttttt///....oo/ org/ 10. w.8550/arXiv....
12446
Xu Zhang, Yong Xu, Qingwei Lin, Bo Qiao, Hongyu Zhang, Yingnong Dang, Chunyu Xie, Xinsheng Yang, Qian Cheng, Ze Li, Junjic Chen, Xiaoting Hc, Randolph Yao, Jian-Guang Lou, Murali Chintalapati, Furao Shen, and Dongmei Zhang. Robust. log-based anomaly detection on unstable log data. In Marlon Dumas, Dietmar Pfahl, Sven Apel, and Alessandra Russo (eds.), Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSI FSE 2019, Tallinn, Estonia, August 26-30, 2019, pp. 807-817. ACM, 2019c. doi: 10. //////33390033333888333 URL TRRLhttps:: htttss://ooiiii wwt.://...........1. 33389311
112

---

Published in Transactions on Machine Learning Research (09/2024)
Ying Zhang, Wenjia Song, Zhengjie Ji, Danfeng Yao, and Na Meng. How well does LLM generate security tests? CoRR, 2b3100300000 2023f. doi: 001.:///AAXXV23111.00000 URL https://doi.org// http///rxvv.221..000
Ziyin Zhang, Yikang Liu, Weifang Huang, Junyu Mao, Rui Wang, and Hai Hu. MELA: multilingual evalu- ation of linguistic acceptability. CoRR, /211 1.9003,, 2023g. doi: 10.4550/ARXXV.2211.0000. URL https://doi.org// 48550/ /ar/0/////vvv... 2311.09033
Ziyin Zhang, Lizhen Xu, Zhaokun Jiang, Hongkun Hao, and Rui Wang. Multiple-choice questions are efficient and robust Ilm evaluators. 2024d. URL https://doi.org:: w00////.....///2111...... 2405 11966
Gang Zhao and Jeff Huang. Deepsim: deep learning code functional similarity. In Gary T. Leavens, Alessandro Garcia, and Corina S. Pasareanu (eds.), Proceedings of the 2018 ACM Joint Meeling on Eu- ropean Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, Nouember 04-09, 2018, pp. 141-151. ACM, 2018. doi: 02L144/3336024332233
URL https:///doi.org/110/ 1ttp///111111133333110 3236066
Liang Zhao, Hexin Cao, and Yunsong Zhao. GP: context-free grammarr pre-training for text-to-sql parsers. CoRR, abs/2101.09900 2021. URL
Liang Zhao, Xinocheng Feng, Xiachong Feng, Bing Qin, and Ting Liu. Length extrapolation of transformers: A survey from the perspective of position encoding. CoRR, abss/232..7704 2023. doi: 10.48550/ ARXIV. 2312.17044. URL
000///0.8855//// arXiv 2312.17044
Liping Zhao, Waad Alhoshan, Alessio Ferrari, Keletso J. Letsholo, Muideen A. Ajagbe, Erol- Valeriu Chioasca, and Riza Theresa Batista-Navarro. Natural language processing (NLP) for requirements en- gineering: A systematic mapping study. CoRR, abs/2004.01099, 2020. URL 2004.01099
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufci Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, pp. 5673 5684, New York, NY, USA, 2023a. Association for Computing Machinery. ISBN 9798400701030. doi: 111144/555000053599997777 URL attps::/doo org/ 10. N0.1145/358035..556
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinneeppreee::: Integrating code generation with execution and refinement. CoRR, abs/ 22//22402..663 14658, 2024. doi: 10.48550/ARXI1 ww........2........
URL https:://oi. org/ 10. 4854048555(0. 2402 14658
Yunhui Zheng, Saurabh Pujar, Burn L. Lewis, Luca. Buratti, Edward A. Epstein, Bo Yang, Jim Laredo, Alessandro Morari, and Zhong Su. D2A: A dataset built for ai-based vulnerability detection methods using differential analysis. In 48rd IEEE/AI /AC International Conference on Software Engineering: Software En- gineering in Practice, ICSF (SEIP) 2021, Madrid, Spain, May 25-28, 2021, pp. 111 IEFF, 2021. doi: TwW.IICSEESSIIPPPP 2021. 00020
10.1109/1C53 (010099CSE--EIIEISS0 S00.15500002222.0000020
URL battp::/ddo..org
Zibin Zheng, Kaiwen Ning, Jiachi Chen, Yanlin Wang, Wenqing Chen, Lianghong Guo, and Weicheng Wang. Towards an understanding of large language models in software engincering tasks. CoRR, abs/ //////211.111113 2023b. doi: 10.48550/ ARXIV wtt.2228.13330 URL htttps://doi.or// 48850//55Xiiy 2308. 11396
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs//1799.00003 2017. 00103
org/ wwwp//////a///.o//////////
Victor Zhong, Mike Lewis, Sida I. Wang, and Luke Zettlemoyer. Grounded adaptation for zero-shot executable semantic parsing. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 6869-6882. Association for Computational Linguistics, 2020. doi: N011655/V1/2000 /222020022222 WWW..EENN.P-MA...... URL http:///ooooooo.......
113

---

Published in Transactions on Machine Learning Research (09/2024)
Wenkang Zhong, Chuanyi Li, Jidong Ge, and Bin Luo. Neural program repair Systems, challenges and solutions. In Internetware 2022: 13th Asia-Pacific Symposium on Internelware, Hohhot, China, June 11 - -12, 2022, pp. 96 106. ACM, 2022. doi: 10. 10.1145/3545258.35455 3545258. 3545268
URL https://doi.orgge 2xg///..115/
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjic Zhan, and Hongshong Li. Solving challenging math word problems using GPT-4 code interpreter with based self veriffctiii... CoRR, abs/2308. 009921, 2023a. doi: 10.48550/ARXI0 2308.07921. URL https://doi. orgg 10. 48550/arXXit 2308.07922
Bingzhe Zhou, Xinying Wang, Shengbin Xu, Yuan Yao, Minxue Pan, Feng Xu, and Xiaoxing Ma. Hybrid API migration: A marriage of small API mapping models and large language models. In Hong Mei, Jian Lv, Zhi Jin, Xuandong Li, Xiaohu Yang, and Xin Xia (eds.), Proceedings of the 14th Asia-Pacifie Symposium on Internetware, Internetware 2023, Hangzhou, China, August 4-6, 2023, pp. 12 21. ACM, 3609466
2023b. doi: 10.1145/3609437, 16609437.3609466 URL attps:///ofi org/10. 1111111441988/ 36094377 36609444
Xin Zhou, Sicong Cao, Xiaobing Sun, and David Lo. Large language model for vulnerability detection and repair: Literature review and the road ahead. CoRR, abs/ 0/////244..22555 2024. 02525. URL. https://doi..ooo orgg////i.ooo////////// 10.
doi: 10.48550/ARXIV 2404.
http://axvvv........ 02525
Yaqin Zhou, Shangqing Liu, Jing Kai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulner- ability identification by learning comprehensive program semantics via graph neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 10197-10207, 2019. URL https: sattps://proceeding: peeurips.cc/paper//ee 3926622446c335f4ee766666ee622 www.:altff--tttttt.tttaatt
Ziyi Zhou, Huiqun Yu, Guisheng Fan, Zijie Huang, and Kang Yang. Towards retrieval-based neural code summarization: A meta-learning approach. IEEE Trans. Software Eng., 49(4):3008-300 2023c. doi: 021109//TEE.22233333333 3238166.. URL https://doi. org/10. 1109/TSE.. 2000///...223.......... .2238163
Jieming Zhu, Shilin He, Jinyang Liu, Pinjia He, Qi Xie, Zibin Zheng, and Michael R. Lyu. Tools and benchmarks for automated log parsing. In Helen Sharp and Mike Whalen (eds.), Proceedings of the 41st International Conference on Software Engincering: Software Engineering in Practice, ICSE (SEIP) 2019, Montreal, QC, Canado, May 25-31, 2019, pp. 121-130. IEEE/ / ACM, 2019. doi: 10.1109/1CSE-- SEIP. 2019.00021. URL https::/doo..oo
1109/ICSEEEEEEA SEIP. 2019.00021.
Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K. Reddy, Xlcost: A benchmark dataset for cross-lingual code intelligence. CoRR, abs/2206.08877 2022a. doi: URL
10.48550/
http:/10..8555/axiiiv 2206.2206..20 220.608477444
Ming Zhu, Karthik Suresh, and Chandan K. Reddy. Multilingual code snippets training for program trans- lation. In Thirty-Sizth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovalive Applications of Artificial Intelligence, IAAT 2022, The Twelveih Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22- March 1, 2022, pp. 11783 11790. AAAI Press, 2022b. doi: 1.1609/AAAAVV331110.. 236110.10 21434
URL https: /ttp::/dooooooooo org/oogg///oo 10. 1609/aa0aa aaai.
Qihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang, Kang Yuan, Yingfei Xiong, and Lu Zhang. A syntax- guided edit decoder for neural program repair. In Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (eds.), ESEC/FSE '21: 29th ACM Joint. European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23- 28, 2021, pp. 341-353. ACM, 2021. doi: 14.144//3468264.44664444 URL https::/dol: /sttp:///oo..or/10.. 0r4//0.1145/ 3468264. 3468544
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Poiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepsock-coder-v22 Breaking the barrier of slosed-source models in code intelligence. arXiv preprint ar Xic:2240 1 19911, 2024. URL http::////oi..rrr/////.4//////..
48550/ wttp://000aaxv......... 11931
114

---

Published in Transactions on Machine Learning Research (09/2024)
Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. Astraios: Parameder-efficient instruction tuning code large language models. CoRR, /t////2411000000
2024. doi: 000.:50//AAII.2400...0.00 URL http::///d0i.org/114488855
arXiv. 2401.00788
Deqing Zou, Sujuan Wang. Shouhuai Xu, Zhen Li, and Hai Jin. mmpuyyuudeeeepce A deep lcarning-basee system for multiclass vulnerability detection. IEEE Trans. Dependable Secur. Compuul, 18(5):2224 2236, 2021. doi:
URL https: //doi. org/10. 111110...1111.11111/ 2942990
115

---

Published in Transactions on Machine Learning Research (09/2024)
A
Benchmarks for Downstrem Tasks
Table
10
list benchmark datasets for code downstream tasks. For the size of these datascts, we
use sell-re sell-reported number whenever available.
Table 5: Benchmarks for text-to-SQL generation.
Task
Date
Benchmark
Source
Size
Hemphill et al. et al.
(1900);
Dahl
1990
ATIS
1 11508
1994)
1996
GeoQuery
Zelle & Mooney
1996
877
2000
Restaurants
Tang Mooney
20000
378
2014-09
MAS
Jagadish [2014
196
2017-02
Yelp
Yaghmazadeh et al.
2017
128
2017-02
IMDb
Yaghmazadch et al.
2017
131
2017-04
Scholar
Lyer et al.
(017)
816
201 7-08
WikiSQL
Zhong ct al.
(2017 7
80654
2018-06
Advising
Finegan-
Do lak et al. 20018)
4570
2018-09
Spider
Yu et al.
2018c
10181
2019-06
SParC
Yu et al.
2019c
12726
Text-to-SQL
2019-07
MIMICSQL
Wang
(020bb
10000
2019-09
CoSQL
Yu et al.
201 2019b
15598
2020-05
Criteria-to-S0
Yu et al.
2020
2003
2020-10
Squall
Shi et al.
20200
11276
2020-10
Spider-Realisee Realistic
Deng et al.
(2021
508
2021-06
Spider-Syn
Gan et al.
2021 aa
8034
2021-06
SEDE
Hazoom et
al.
(2021
12023
2021-06
KaggleDBQA
Lee
2021
400
2021-09
Spider-DK
Gan al.
202 b)
535
2022-05
Spider-SS
Gan al.
2022
8034
2022-05
Spider-CG
Gan et al.
2022
45599
2023-05
BIRD
Li et al.
2023g
12751
116

---

Published in Transactions on Machine Learning Research (09/2024)
Table 6: Benchmarks for program synthesis. JS is short for JavaScript. "Automatically mined/ mined/human hhuman- annotated. 'hese 1749 prompts for 48 problems. +10538 prompts for 1420 methods. Machine-
generated/human-amneet prompts.
Task
Date
Benchmark
Source
Size
Language
2018-02
NL2Bash
et al.
2018b
9305
Bash
2018-08
CONCODE
Iyer et al.
22018
104K
Java
2019-10 10
JuICe
Agashe al.
2019
*1.5M/3725
Python
2021-05
APPS
Hendrycks
2021a
10000
Python
2021-07
HumanEval
Chen et al.
202
164
Python
2021-08
MBPP
Austin al.
2021
974
Python
2021-08
MathQA-Python
Austin al.
202
23914
Python
2021-08
PlotCoder
Chen et al.
202
40797
Python
2022-01
DSP
Chandel al.
2022
1119
Python
C++, Python,
2022-02
CodeContests
Li ct al.
2022h
13610
Java
2022-03
MCoNaLa
Wang et al.
2003i
896
Python
Program
2022-06
AixBench
Hao al. [2022
Java
Synthesis
2022-10
MBXP
Athiwaratiuu et al.
2023
12.4K
13
2022-10
Multilingual HvmanEval
Athiwaratiun al.
2023
1.9K
12
2022-10
MathQA-X
Athiwaratkun
2023
5.6K
Python, Java, JS
2022-11
ExeDS
Inang et al. 2202b
534
Python
2022-11
DS-1000
Lsi et al. 20
1000
Python
2022-12
ODEX
Wang a. 2022h
945
Python
2023-02
CoderEval
Yu et al.
123a
460
Python, Java
2023-03
xCodeEval
Khan et al.
2023
5.5M
11
Python, C++,
2023-03
Hvmlln
Zheng et al.
2023a
820
Java, JS, Go
2023-06
Eval
Babe et al.
2023
11749
Python
2023-06
DotPrompts
Agrawal al.
2023
110538
Java
2023-09
CodeApex
Fu et al.
2023
476
C++
2023-09
VerilogEval
Liu al.
202.
9864151556
Verilog
2023-11 11
ML-Beneh
al.
10040
Bash
C++, 11IP.
2024-01
ParEval
Nichols al.
2024
420
CUDA
2024-04
MMCode
Li ct 22024b
3548
Python
2024-04
PECC
Haller et al.
2024
2396
Python
2024-05
NoddrahCode
Zhang et 2024bb
402
Python, Javs
2024-05
Plot2Code
Wu ct 202-aa
132
Python
2024-05
MHPP
Dai ot a..
2024b
140
Python
117

---

Published in Transactions on Machine Learning Research (09/2024)
Table 7: Benchmarks for code translation and program repair. JS short for JavaScript. "These are pairwise sample counts. For example, HumanEval-X includes 164 programs, each implemented 5 languages, totaling 164 x (5 / 2) 1640 translation pairs. Thhese are code change datasets, and only a subset therein
concern Mug-fixing
Task
Date
Benchmark
Source
Size
Language
C++. Java,
2020-06
GeekstorGecks
Rozière et al.
2020
1.4K
Python Java, C# Java, Python
2021-02
CodeTrans
Lu et al.
2021
11.8K
2021-08
Avatar
A Ahmad et al.
2023
9515
C++, Java,
2022-06
CoST
Zhu et al.
2022b
*132K
Python, C#, JS,
PHP, C
C++, Java,
Code
2022-06
XLCoST
Zhu et al.
2022a
*567K
Python, C#, JS,
Translation
PHP, C
2023-03
xCodeEval
Khan et al.
2023
*5.6M
11
Python, CII,
2023-03
HumanBval-X
Zheng al.
2023a
*6640
Java, JS, Go
C+ t. Java, C#.
2023-08
G-TransEval
Jiao
2023
4000
JS, Python
2023-10
CodeTransOcean
Yan et al.
2023
270.5K
45
2014-07
Defects4.J
Just et al.
2014
357
Java
2015-12
ManyBugs
Goues et al.
2015
C
2015-12
IntroClass
Goues al.
2015
998
C
2016-11
BugAID
Hanam ct a..
20016
105K
JS
2017-02
DeepFix
Gupta
2001
6971
C
2017-05
Codellaws
Tan et al.
2017
3902
C
2017-10
QuixBugs
Lin et 2017
80
Java, Python
2018-05
Bugs.jar
Saha al
22118
1158
Java
2018-12
BFP
Chakraborty et al.
(202bb
124K
Java
2019-01
Bears
Madeiral et 2019
251
Java
2019-01
unnamed
Tulano et al.
2019aa
121.8K
Java
2019-04
BugsJS
Gyimesi et al.
2019
453
2019-05
BugSwarm
Tomassi et al.
2019
1127/1264
Java PPython
2019-05
CPatMiner
Nguyen et al.
2019
17KK
Java
2019-05
ManySStuBe4]
Karampatsis
Sutton
2002
154K
Java
Program
2019-11
Refactory
Hu ct al. 2019
1783
Python
Repair
Java, JS, C,
2020-07
CoCoNut
Lutellier al.
2020
24M
Python
2020-10
Review&RRepiir
Huq et al. 2022
58021
Java
2020-11
BugsInPy
Widyasari al.
2020
493
Python
2021-07
TFix
Berabi et al. 2022
105K
JS
2021-08
Megadiff
Monperrus al.
(2021
1663K
Java
2022-01
SSB/TSSB
Richter & Wehrheim
2022
9M/3M
Python
2022-10
FixJS
Csuvik & Vidáes
2022
324K
JS
2022-11
TypeBugs
Oh Oh 2022
.93
Python
2023-03
xCodeEval
Khan et al. 2023
4.7M
11
C, C++, Java, Python, JS,
2023-04
RunBugRun
Prenner & Robbes
2023
450K
Ruby, Go, PHP Python, JS, Go, Java, C+ Rust C++. Java, Python
2023-08
HumanEvalPack
Muennighoff et al.
2024
984
2024-01
DebugBench
Tian al. 2024
4253
118

---

Published in Transactions on Machine Learning Research (09/2024)
Table 8: Benchmarks for code summarization, code completion, commit message generation, and de- fect/ veeeevllerability detection. JS is short for JavaScript. "The task of code completion be evaluated any source code corpus, we only list few widely used benchmarks here. For cross-file code completion please refer to Table 10 WWtth/withon verb-direct object filter.
Task
Date
Benchmark
Source
Size
Language
2016-08
CODE-NN
Iyer et
2016
66K/32K
C#/SQU
2017-07
unnamed
Barone &
Sennrich
2017
150K
Python
2018-05
DeepCom
Hu et
2018a
588K
Java
2018-07
TI-CodeSum
Hu et al.
2018bb
411K
Java
Code
2018-11
unnamed
Wan
20018
109K
Python
Summarization
2019-02
unnamed
LeClair et al.
2019
2.1M
Java
Go, JS, Python,
2019-09
NodeSearchNet
Husain et al.
2019
2.3M
PHP, Java, Ruby
Python, JS, Go, Java, C++, Rust
2023-08
FxalPck
Muennighoff et al.
2024
984
2013-05
GitHub Java Corpus
Allamanis & Sutton
2013
2.1M
Java
'Code
2016-10
Py150
Raychev al.
2016a
150K
Python
Completion
2016-10
JS150
Raychev et al.
2016a
150K
JS
Python, Java,
2023-06
LCC
Guo et al.
2023b
360K
C#
2017-03
unnamed
Jiang & McMillan 2017
509K
Java
Python, C++,
2017-04
CommitGen
Loyola et al. 2017
153K
Java, JS
2017-08
CommitGen
Jiang et al. 2017
T3K 32KK/7KK
/7KK
Java
2018-09
NNGen
Liu et al.
2018
27K
Java
2019-05
PtrGNCMsg
Liu et al.
2019c
64.9K
Java
Commit
2019-08
CoDiSum
Xu et al.
2019a
90.7K
Java
Message
2019-12
ATOM
Lau et al.
2022c
160K
Java
Generation
Python, PHP.
2021-05
CommitBERT
Jung
2021
346K
Go, Java, JS.
Ruby
Java, C#. C++.
2021-07
MCMD
Tao et al.
2021
2.25M
Python, JS
2021-07
CoRee
Wang et al. 2021c
107K
Java
2023-07
ExCroFi
Wang et al.
2023c
19263
Java
2023-08
CommilCCroniee
Eliseeva el
2023
10.7M
20
2018-01
CGD
Li et al.
2018b
62K
C, C++
2018-07
Draper VDISC
Russell
2018
12.8M
C, C++
2018-07
SySeVR
Li et al.
2022
15591
C, C++
2018-04
unnamed
Iann
2018a
32988
C, C++
2019-02
unnamdd
Ponta et al. 2019
2019
624
Java
2019-09
Devign
/hou
48687
C
2019-11
unnamed
Linn et al.
2021
170K
C, C+H
2019-12
GREAT
Hellendoomn al.
2020
2.8M
Python
Defect
2020-01
MVD
Zou et. al. (222
182K
C, C++
(Vulnerability)
2020-02
unnamed
Lin et 2019
1471
C
Detection
2020-09
ReVeal
Chakraborly et al.
2022c
18K
C
2020-09
Big-Vul
Fan
(2220
265K
C, C++
2021-02
D2A
Zheng el al. (2021
1.3M
C, C++
2021-05
PyPIBugs
Allamanis al.
2021
2374
Python
2021-07
CVEfixes
Bhandari et a..
2021
5495
27
2021-08
CuossVul
Nikitopoulos el al.
2221
27476
40
2023-04
DiverseVul
Chen et al. 2023
349K
C, C++
2023-06
PulnPatchPairs
Risse Bohme
2023
26K
C
2023-11 11
VulBench
Gao et al.
2023d
455
C
119

---

Published in Transactions on Machine Learning Research (09/2023)
Table 9: Benchmarks for code retrieval, code reasoning, type inference, and clone detection search,, JS is short for JavaScript. "These benchmarks include large number of automatically constructed samples, and a small set of human-annotated samples. 'These are general-domain reasoning benchmarks, and only subset therein concern programming, algorithms, and other topics related to computer science. TThese are project counts (or, in the case of Cassano et al. (2023c), file counts). Yee & Guha (2023) propose to measure project-level type check rate instead of type prediction accuracy for TypeScript. These are
21K/24K Java/Python Python methods for 576 programming problems.
Task
Date
Benchmark
Source
Size
Language
2018-03
StaQC
Yao et al.
2018
268K
Python, SQL.
2018-05
DeepCS
Gu al.
2018
16M
Java
2018-05
CoNaLa
Yin al.
2018
*600K/2. /2.9K
Python
2019-08
unnamed
1, et al.
20T9h
287
Java
Go. JS, Python,
2019-09
CodeSearchNet
Husain et al.
2019
"2.3M/99
PHP, Java, Ruby
Code Retrieval
2020-02
CosBench
Yan et al.
2020
52
Java
2020-08
SO-DS
Heyman &
Cutsem
2020
2.2K
Python
2020-10
FB-Java
Ling al.
2021
249K
Java
2021-02
AdyTest
La et al.
2021
280K
Python
2021-02
WebQueryTest
2021
1K
Python
2021-05
CoSQA
Huang et al.
2022
21K
Python
C, C++, Java,
Python, Ruby,
2024-03
ProCQA
Li et al.
2024c
5.2M
Lisp, JS, C##
Go, Rust, PHP
2020-09
MMLU
Hendryckss et al.
2021b
115908
2021-09
CodeQA
Liu & Wan 202
120K/70K
Aoaa/yyhoon
Code
2022-10
CSIQA
Lee
2022
9237
Reasoning
2023-05
C-Eval
Huang et al.
2023b
13948
2023-06
CMMLU
Li al. 2023b
115228
2023-09
CodeApex
et 2023
C++
2024-01
CRUXEval
Gu et al. 2024
800
Python
2024-05
PythonIO
Zhang al.
2024d
2650
Python
2019-12
TypeWriter OSS
Pradel et al.
2020
208K
Python
2020-04
Typilus
Allamanis
2020
252K
Python
2020-04
LambdaNet
Wei et al. 2020
1300
TypeScript
2021-04
Mir ct al. 2021
869K
Python
Type Inference
2022-10
ManyTypessTyppeSeript
Jesse &z Devanbu
2022
9.1M
TypeScript
2023-02
Type Weaver
You & Guha
2023
1513
TypeScript
2023-03
BetterTypes/ee
Wei et al. 2023
608K
Python
2023-03
Typs4Py
Wei al.
4.6K
Python
2023-05
OpenTau
Cassano et al.
2023c
1744
TypeSeript
2014-09
BigCloneBench
Svajlenko
2014
6M
Java
Clone
2014-09
POJ-104
Mou et al. 2016
52K
C, C++
Detection /
2019-05
unnamed
Perez Chiba
2019
221K/24K
PottPython
Code Search
Java, C#
2019-11
CLCDSA
Nafi et al. 2019
78K
Python
120

---

Published in Transactions on Machine Learning Research (09/2024)
Table 10: Benchmarks for unit test/assertion generation, log parsing, and repository level coding. "TogHub
(2023) is an annotated subset of LogHub (2018). 'Line Completion/API Invocation Completion/Fundet /Function Completion. "Relrieal/Coompleoon Coppletion/Pipeliiii *File count. Migrattion EnigratioonTTemprall Temporal Edit.
Task
Date
Benchmark
Source
Size
Language
Unit Test /
2014-12
SF110
Fraser & Arcuri 2014
24K
Java
Assertion
2020-09
Method2Test
Tulano et
2021a
781K
Java
Generation
2021-08
CouTest
Villmow el a.
2021
365K
Java
Zhu et al.
2019
He et al.
2018-11
LogHub (2018)
379M
Log Parsing
2020
2023-08
LogHub (2023)
lang et al.
2023c
*50.4M
2023-03
RepoEval
Zhang et
(2023b)
/1600/1600/377 1600/373
Python
2023-06
RepoBench
Liu et al.
2023
/890K/9M/444 /93/
Python, Java
2023-06
PragmaticCode
Agrawal al.
2023
*880
Java
Repository-
2023-06
Stack-Repo
Shrivastava al.
2023a
816K
Java
Level Coding
2023-09
CodePlan
Hairi al. 2023
645/21
OC#
(C#/Pyhhon
2023-10
SWF-Bench
Jimenez et al.
2123
2294
Python
Python, Java,
2023-10
CrossCodeEval
Ding et al.
2023
9928
TypeScript, C##
2024-03
EyoCodeBench
Li 2024a
275
Python
121

---

Published in Transactions on Machine Learning Research (09/2024)
B Results on Downstream Benchmarks
For the benchmarks listed in Appendix A of the most widely used ones, including:
wc also list code language models' reported performance on some
NL-code search (Table CodeSearchNet (Husain et al.
2019 CosQA (Huang al. 2021 and 2021 all measured by Mean Reciprocal Rank (MRR);
AdvTest (Lu et al.
Clone detection (Table
BigCloneBench (Svajlenko el al. 2014], measured by F1, and POJ- measured by Mean Average Precision; the train/test splits of both
104 (Mou et al. datascts are provided by
2016
Lu ct al. (021);
Defect detection (Table
13 Devign Zhou et al. 2019 measured by accuracy; the train/test. split
provided by Lu 2021
Code summarization (Table 14h:
CodeScarchNet (Husain et al. 2019), measured by BLEU; the
train/test split is provided by Lu et al. 2021))
measured by BLEU and CodeBLEU, and Transcoder [Rozèrre al. 2020), measured by pass@l;
Code translation (Table 15
16
CodeTrans (Lu et al. 2021
Code reasoning (Table 17 c CRUXEval GGu et al.
2024
measured by pass@1, and PythonIO,
measured by accuracy.
Table 11: Mean Reciprocal Rank of NL-code search performance on CodeScarchNet, AdvTest, and CosQA.
Ruby
JSS
Go
Python
Java
PHP
Average
AdvTest
CosQA
RoBERTa
58.7
51.7
85.0
58.7
59.9
56.0
61.7
18.3
60.3
Lu et al.
2021
CodeBERT
67.9
62.0
88.2
67.2
67.6
62.8
69.3
27.2
65.7
GraRhCodeBERT
70.3
64.4
89.7
69.2
69.1
64.9
71.3
35.2
68.4
Guo al.
2021a
SynCoBERT
72.2
67.7
91.3
72.4
72.3
67.8
74.0
69.6
Wang
2021d
SPT-Code
70.1
64.1
89.5
69.9
70.0
65.1
71.5
Niu et al.
2022
CodeRetriever
77.1
71.9
92.4
75.8
76.5
70.8
77.4
46.9
75.4
Li et sl.
122e
UniXcoder
74.0
68.4
91.5
72.0
72.6
67.6
74.4
41.3
70.1
Guo
2022
PLBART
67.5
61.6
88.7
66.3
66.3
61.1
68.5
34.7
65.0
T
71.9
65.5
88.8
69.8
68.6
64.5
71.5
39.3
67.8
CoCoSoDa
81.8
76.4
92.1
75.7
76.3
70.3
78.8
Shi ct al
2023b
Codc-MVP
40.4
72.1
Wang
2022e
SCodeR.
77.5
72.0
92.7
74.2
74.8
69.2
76.7
74.5
20221
Code +targs
78.0
71.3
92.7
75.8
76.2
70.1
77.4
44.7
74.0
Wang ct al.
2023h
CodeGen-mutt: 350M
66.0
62.2
90.0
68.6
70.1
63.9
70.1
34.8
64.8
122

---

Published in Transactions on Machine Learning Research (09/2024)
Table 12: Clone detection performance BigCloneBench (F1) and POJ-104 (Mean Average Precision).
Works in the two blocks use different. implementations and report different. baseline scores.
BigCloneBench (F1)
POJ-104 (MAP)
source
RoBERTa
94.9
79.96
Lu et al.
(2021
CodeBERT
96.5
84.29
DOBF
96.5
Lachaux et al.
2021
PLBART
97.2
Ahmad et al.
2021
GrapECodeBERT
97.1
85.16
Wang et al.
2021d
SynCoBERT
97.4
88.24
CodeT5 basse
97.2
Wang et al.
(2021
RoBERTa
91.3
76.67
Guo et al. 2022
CodeBERT
94.1
82.67
Guo el al.
2022 la
GRRRRCodeBERT
95.0
85.16
DISCO
94.4
83.32
Ding et al.
2022a
PLBART
93.6
86.27
Guo et al.
2022
CodeTT Tbase
95.0
88.65
UniXcoder
95.2
90.52
SCodeR
95.3
92.45
Li et al.
20221
Table 13: Accuracy of defect detection Devign. pt: pretrained. ft: finetuned.
Accuracy
pt
ft
prompt source
Transformer
61.6
Ahmad et al.
2021
RoBERTa
61.0
/
Lu et al. 2021 4
CodeBERT
62.1
PLBART
63.2
Ahmad et al.
2021
GraRhCodeBERR
63.2
Wang et al.
2021d
SynCoBERT
64.5
CodeT5
65.8
Wang et al.
2021f
DISCO
64.4
Ding et al.
2022a
VulBERTa
64.8
Hanif & Ma ffeis (2222)
Enstrutt-CodeGi 16B
47.8
0-shot
Yuan al.
2023a
CodeAlpaca 7B
51.9
0-shot
Vicuna 7B
54.0
0-shot
WizardCoder 15B
54.4
0-shot
123

---

Published in Transactions on Machine Learning Research (09/2024)
Table 14: BLEU scores of code summarization on CodeSearchNNet, using the CodeXGLUE split (top) and the original split. (bottom). pt: pretrained. ft: finetuned.
Ruby
Python
Java
PHIP
Average
pt
prompt
souree
Transformer
11.18
11.59
16.38
15.81
16.26
22.12
15.56
La et al.
2021
RoBERTa
11.17
11.90
17.72
18.14
16.47
24.02
16.57
CodeBERT
12.16
14.90
18.07
19.06
17.65
25.16
17.83
PLBART
14.11
15.56
18.91
19.30
18.45
23.58
18.32
Ahmad
2021
CodeTTbaaee
15.69
16.24
19.76
20.36
20.46
26.09
19.77
Wang ct a
2021F
UniXeoder
14.87
15.85
19.07
19.13
20.31
26.54
19.30
Guo et al.
2022
InCoder
18.27
O-shot
Fried et al.
2023
NatGen
15.38
16,00
19.43
20.09
20.38
26,00
19.55
Chikraborty
ct
2022a
CodeT5-ttarge
15.63
20.47
20.83
26.39
20.15
Wang
2023h
StarCoder
21.99
O-shot
Cf al.
2023
ChatGPT
10.28
D-shot
Sun
2023b
PaLM
19.23
few-shot
Haldar Hockenmaier
2024
LLaMA 2 70B
22.41
few-shot
Mastropaolo al.
7.8
9.0
15.3
9.7
13.5
18.4
12.3
et 2022
CugLM
7.4
10.0
17.0
11.4
13.2
17.8
12.8
TreeBRRT
7.4
10.3
18.0
12.9
CodeBERT
8.3
10.0
16.0
10.7
13.6
20.1
13.1
CraphCodeBERT
8.3
10.9
16.5
11.0
14.5
20.0
13.5
SPT-Code
8.4
12.8
18.8
12.8
16.8
20.4
15.0
Table 15: BLEU and CodeBLEU scores code translation on CodeTTanss
Java-
C## -Jva
BLEU
CodeBLEU
BLEU
CodeBLEU
source
Transformer
55.8
63.7
50.5
61.6
Lu et al.
2021
CodeBERT
79.9
85.1
72.1
79.4
GraphCodeBERT
80.6
72.6
Guo et al. (022aa)
PLBART
83.0
87.9
78.3
85.3
Ahmad et al. 22021
SynCoBERT
80.8
84.8
76.5
82.2
Wang et al.
(202
CodeT5o T5hase
84.0
87.8
79.9
84.4
Wang et al.
(2021])
NatGen
88.1
85.2
Chakraborty et al.
2022a
Table 16: Pass@1 scores of code translation on TransCoder evaluation set.
C++ Java
C++
Java C++
Java Py
Py C++
Py
Java
TransCoder
60.9
44.5
35.0
32.2
Raztère
DOBF
40.6
16.6
Lachaux
212
TransCoder-STT
66.7
61.1
84.1
67.8
52.2
56.7
Hozière
1129
TransCoder-IR
62.9
74.5
Szatremec
A723
LaMDA
30.2
Chowlherry et al
212
PaLM
51.8
PaLM Coder
55.1
StarCoder (self-debug)
(76.6)
Chen al. 20236
Codex (self debug)
(92.5)
GPT-3.5 (selE-debug)
(92.7)
GPT-4 (self- (sbugg)
77.3
(90.4)
124

---

Published in Transactions on Machine Learning Research (09/2024)
Table 17: Pass@1 scores of code reasoning CRUXEval input and output prediction. DeepSeek-Coder-v2 is a Mixture-of-Expee model with total of 236B parameters, where 21B are activated for each token.
size
input
output
source
Phi-1
1.3B
13.9
23.3
Gu et al 2024
Phi-1.5
1.3B
24.1
27.1
Mistral
7B
36.0
31.7
StarCoder Base
16B
31.6
33.3
CodeLLaMA (CoT)
34B
46.5
(50.4)
41.1 (46.0)
DeepSeck-Coderraaatt Instruct
33B
47.4
44.0
GPT-3.5 (CoT)
49.2 (49.1)
50.0 (63.3)
GPT-4 (CoT)
67.1 (74.8)
63.4 (81.9)
StarCoder2
16B
48.1
47.1
Lozhkov et al. (024
Codestral + CoT
22B
48.0
60.6
Zhu et al. [024
pnstttt + CoT
70B
61.1
64.3
DeepSeek- SaeneeekkCCoe---- Vnnntrttct + CoT
236B
(21B)
70.0
75.1
Gemini-1.5- + CoT
67.0
77.5
Claude-3-Opie CoT
73.4
82.0
GPT-4o CoT
77.4
88.7
Table 18: Accuracy of code reasoning performance on PythonIO, extracted from Zhang el. al.
2024d
size
accuracy
Phi-2
2.7B
29.8
Phi-3
3.8B
38.2
ChatGLM3 Base
6B
27.7
Gemma
7B
30.5
Mistral
7B
31.7
Qwen1.5
32.8
LLaMA-3/netean
8B
39.0
Flan-T5
11B
29.3
LLaMA-2
13B
26.6
Qwen1.5
14B
40.9
LLaMA-2
70B
38.2
LLaMA-3
70B
70.1
Qwen1.5
72B
50.4
125


Reviews:
[{"review_id": "U4hdrIfNRJ", "reviewer": "Reviewer_sQxn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \n- The paper is an impressive work requiring lots of effort reading articles and summarizing, taxonomizing their contents. I found the paper easy to read and understanding; the structure is clear. \n\n- The taxonomy is intuitive and reasonable. \n\n- Figure 2 is very helpful to capture the progress of code-language models. \n\n## Weaknesses\n- This is a good paper yet it seems focus mainly on text-to-code's performance with lots of comparisons (figure 2, tab 1,2,4). However, it's difficult to see the progress in other tasks, and also real applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a comprehensive literature review about the current research on text-code-related problems (e.g. text-to-code generation, code retrieval). The review covers an impressive number of things: 50+ models, 30+ evaluation tasks, 170+ datasets, and 700+ related works. The paper introduces a task-based taxonomy, covering a wide range of tasks requiring text and code.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see \n- the progress of other task rather than only text-to-code. \n- deeper discussion about the impacts of text-code tasks, especially about which tasks we can use LLMs with high confidence, which tasks can be automated, which tasks are still challenging, etc.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "n1JbYKp93o", "reviewer": "Reviewer_uFxT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The paper presents an interesting survey that aims at closing a gap between efforts to use/evaluate language models from a software engineering perspective and the same efforts from an NLP/modelling perspective. The tasks surveyed in section 2 provide a useful framework to analyse language models from a perspective that could be more useful to software engineering. \n- The paper examines a vast amount of research from the two perspectives, which is according to what's expected for a survey, and adds experimental results to at least one of these perspectives which helps strenghten the contribution. \n\nWeaknesses:\n- Although I appreciate how systematically the paper presents existing work, and the effort on trying to bridge the gap between SE and NLP perspectives on language models for coding, the organization of the paper seems a bit disconnected from this intention. It seems like most of the content is related to the NLP/modelling perspective, while the SE perspective is only tangentially touched in section 2 by listing methods that perform each of these tasks. I would have expected that, after each perspective is presented (as in section 2), the existing language models would be analysed to the lens of both perspectives (in sections 4 and 5). Or, at least, a more clear \"unifying perspective\", as claimed in the title of the paper.\n- Related to the previous item: considering the thorough analyses regarding the performance of general and specialized language models presented in sections 4 and 5, which I assume cover the NLP/modelling perspective, it feels incomplete that the same type of experimental results are not presented for section 2, where specific types of tasks coming from the SE are mentioned. It would be useful to see actual experimental results for at least some of these SE tasks. \n- A minor point: readability of figures in section 2.1.1 - 2.1.5 could be improved by adding titles to the list of references presented in the last three columns of these diagrams.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a survey of code processing methods using large language models. The paper explores related work in this field by surveying a range of models, datasets and tasks. As opposed to previous surveys, the paper proposes to bridge the gap between tasks coming from NLP and tasks proposed from a software engineering perspective, and surveys tasks and approaches from both perspectives. First, it surveys tasks coming from software engineering (sections 2.1.1 - 2.1.5). Then, it surveys tasks in NLP (section 2.1.6). In sections 4 and 5, the paper surveys general and specialized language models for coding. In section 6 the paper surveys some additional features added to language models for coding tasks. Finally, in section 7 the paper surveys advances of large language models in the software engineering process. Along these sections, the paper presents analyses of existing datasets, metrics and performance of state-of-the-art models for coding.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical changes that would impact recommendation for acceptance: \n1. Add experimental results for all or some of the SE tasks presented in 2.1.1 - 2.1.5, or clarify which of the experiments presented in sections 4 and 5 would cover these. \n\nChanges that would strengthen the work:\n1. Make the 'unifying perspective' of SE and NLP/modelling perspectives of language models for coding clearer. \n2. Improve readability of figures in sections 2.1.1 - 2.1.5.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1Sl6WKNJzU", "reviewer": "Reviewer_ntKZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I like this paper. I think it is on its way to being a publishable work. However, I do have some large concerns that may require the authors to rethink some core principles of the paper. Based on the work that the authors have already done, I have confidence that these requests are not too far afield for them to push this paper across the publication-worthy threshold.\n\nStrengths:\n- The surveyed work is notable and fairly comprehensive. The authors have clearly spent a reasonable amount of time conducting it and trying to make sense of the many works they analyze.\n- I think the trend analysis graphs (Figure 2 and Table 3 (pg 19)) are quite insightful in the important data they provide.\n- The “code” tree system the authors use throughout the paper (Figure 1 and then many variants of Figure 3) are extremely helpful to visually digest a large amount of information in a fairly organized way.\n- I think the authors provide a reasonable snapshot of the “LLM for code” landscape and its evolution over the last few years.\n\n\nWeaknesses:\n- I am confused about the authors’ use of “software engineering” throughout this paper. To me, it seems that the paper mostly discusses tasks that go from natural language (NL) to programming language (PL). NL <-> PL. Software engineering (SE) has many activities that are entirely outside the scope of PL. For example, Agile Software Development has nothing to do with PL, but has a lot to do with SE practices. The same could be said about software requirements creation; it is a fundamental step of SE but rarely (and usually should not) includes code (PL). One could argue that the process of doing performance profiling has more to do with proper use of perf. tooling, sampling, and heterogenous hardware evaluation than code. The development of UI/UX systems which are fundamental to SE rarely consider code in their design. And so forth … As such, it feels like many aspects of the paper are confused because of the perhaps misappropriation of the term SE where instead the target is PL (principally around “code”).\n- I’m not sure I agree with some of the categorization of ideas in the paper (e.g., text-to-SQL being considered a “one-off” – I explain more below, but I believe this is a declarative/intentional programming language thing not a one-off (e.g., Halide has similar SQL-like properties))\n- In Section 2, I think there are at least three fundamental aspects that should be resolved. First, I think it’s a misnomer to call these text-to-code and code-to-text categories, as “all code is text, but not all text is code.” So how does one do “code-to-text” translation as code is already text? I think the authors mean to say PL-to-NL; this goes back to the core conflation of PL with SE, in my opinion.  Second, there is an important nuanced space between NL and PL which is the space of “software specifications”, which are neither normal NL and not PL, yet it exists all throughout software. It has the appearance of being NL, but it’s not as it’s more rigorous (constrained) than typical NL, yet it is intentionally not written in PL because then the “spec” would be an “implementation” which it is precisely the opposite its goal as a spec. Third, the downstream items seem pretty disorganized to me. It felt more like a potpourri and an arbitrary selection of topics rather than an organized, rigorous framework for tasks. (For example, the code-to-text and text-to-code is more rigorous because we can see the exhaustiveness emerge, but with the downstream tasks that’s absent). Because of the lack of systematic organization / framework, it’s rather hard to know things like: (1) is this an exhaustive list and (2) is this a proper way to reason about these systems? More on this below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors provide a fairly deep survey of the use of transformer-based neural networks (mostly large language models (LLMs) in the context of software code. The authors also visually illustrate the trends of advancement using LLMs for code-related tasks. The paper considers over 50 models, 30 evaluation tasks, 170 datasets and 700 related works (the references alone are ~75 pages of the 100 page manuscript).\n\nIn addition to the survey, the authors provide some insight on emerging research and engineering directions, including how well LLMs are performing in certain tasks and a categorization (i.e., taxonomy) of natural language and programming language analyses.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There are two main areas where I think the paper needs improvement before I can give it an accept rating for publication. After reading the paper in its entirety, I have a strong level of confidence the authors will be capable of handling these two items (although they may be non-trivial). The two items are listed below.\n\n1. I think this paper is conflating SE with PL. In almost all cases, the paper is focusing on PL (i.e., analysis or synthesis of code). Yet, the paper is dressing the PL up like it’s SE. However, what is really happening is the authors are documenting the transition of using LLMs from NL to PL. Therefore, the focus is on PL and only the concepts in SE that are *related* to PL (or NL). The paper is not (to my knowledge) focusing on the broader space of SE, but right now there is a lot of confusion in the paper indicating otherwise. \n\nI think this is extremely important to get right. The reason for this is that the field of SE is much, much bigger than just PL. As such, when the authors are saying it’s “NL to SE” many software engineering tasks will need to be covered otherwise the work will miss critical aspects that are pertinent to SE. But I don’t think that’s what the authors are actually *trying* to say. I think what the authors are really intending to say is the following: “this paper is about the use of LLMs for PL, where the PL usage has some potential value in SE.” This is what I think the paper is really about, but the authors appear to be saying “NL to SE” which created a lot of confusion for me. Let me explain.\n\nPL is really about the study and use of programming languages. Like NLP before it, I think this paper is principally about how LLMs, which were initially targeted for NLP, are faring in PLP. SE is really about how to engineer software. Many of those tasks have little or nothing to do with code: requirements, team organization, hiring, scheduling, software dev methods, version control systems, defect tracking, integration testing, software supply chain, dependency analysis, software deployment, etc. As such, after reading the paper, it seems in almost all cases where the authors are talking about SE they are talking principally about how PL can be used for software, not how LLMs cover all aspects of SE (otherwise there are large holes of aspects in SE that are absent).\n\nI think the authors should try to reframe this so it’s clear that the emphasis is on evolution of LLM application for PL in addition to NL. If the authors continue to try to draw the corollary of “NL -> SE” I think we will ultimately be creating more confusion in the community because NLP/NLU is about natural language processing and understanding and software engineering is about a bunch of things that are not related to PL. But NL <-> PL, while there are still major differences, is at least more of an apples to apples comparison, in my opinion (moreover, it appears this is what is actually being discussed in the paper).\n\n2. Some of the organization of the ideas appear to miss perhaps minor, but critical, details that I think may confuse readers if they remain absent or the paper isn’t changed to minimally inform the reader on the broader scope of work. \n\nThere are at least three examples of this in the paper: (1) the “code-to-text” categories, (2) “text-to-SQL” being a special case (I’m not convinced that it is) and (3) the organization of the downstream tasks appears less systematic and organized than other aspects. Let’s dissect each one.\n\n“Code-to-text”:\n\nFirst, I think this gets back to the larger issue of not using the right language and therefore a bit (or a lot) of the message is lost. For example, all code is text, but not all text is code. So if you already have code, then you already have text. So then what are we doing if we translate from “code-to-text”? Code *is* text.\n\nWell, what the paper is *really* trying to say is we are translating PL-to-NL (but because it doesn't say this, unless you know this already, you may become very confused). Moreover, in the other cases we are doing NL-to-PL. Sometimes it’s PL-to-PL and NL-to-NL. But words matter and I think the words “code” and “text” are not the right ones here. Moreover, once we see it’s a NL-to-PL thing it sheds even more light on larger issues: which NLs are going to which PLs and which ones aren’t? But if we use “text-to-code” suddenly it pluralizes the discussion and the hidden details may be lost.\n\nMoreover, by using NL and PL, this opens up the possibility to fix the next major issue which is the lack of mention of specification in the space of software engineering. Specs, unlike generally ambiguous NL are more precise and attempt to describe the “what the software will do” or “what the PL constructs are intended to do.” It is imperative that specs do not describe “how it will be done”, which is why specs (let’s call them specification languages, or SLs) are not written casually in NL nor are they written precisely in PL. In fact, I would argue there is a third middle ground here of NL <> SL <> PL. And once we correct the language mistakes, it becomes more clear that SL fits in quite nicely with the correct nomenclature.\n\n“Text-to-SQL”:\n\nI’m not sure this is a one-off as the paper suggests. I actually think it’s more about SQL being the closest to an intentional programming language (except for perhaps Halide) than any other language. In short, SQL is declarative and declarative languages generally only specify the “what” not the “how” thus it’s far easier to translate NL to declarative/intentional languages than NL to imperative languages (like C, C++). Moreover, the grammar is significantly more restricted in SQL than other languages, like C, C++. Therefore the search is computationally more efficient. I think what is really being described here is: “NL-to-intentional PL” and again, using the right language helps us see the right picture that the PL we’re translating to matters, but because of the type of language and its constraints. \n\nRecommend reading JRK’s research at MIT on Halide for a deeper examination on this to understand this phenomenon does not relate only to SQL. In mine (and others opinion) it’s really about opportunities using intentional languages (“Three Pillars of MP”) as Ryan Marcus et al. discuss in their Bao and Neo SQL research (https://arxiv.org/abs/2004.03814, https://arxiv.org/abs/1904.03711) and why there are so many advances to be made in their inventiveness and adaptation for optimization. It's really about SQL's declarative nature that makes this so well aligned to optimization. The same I believe is true about NL-to-SQL, it's more about \"NL-to-intentional PLs\".\n\nDownstream tasks: \n\nI liked the list of the downstream tasks for this work, but I couldn’t understand why these were selected nor if they were “exhaustive” in some sense. I think it would help quite a bit to organize the downstream tasks in some kind of framework. Right now, it appears like a long list of things (all of which are important in their own right), but I don’t know how to categorize them in my head.\n\nSadly, in being transparent, I don’t know how to organize these items (unlike my suggestion of changing “text-to-code” to “NL-to-PL”). My hope is that the authors can understand the concern and work to improve it. I have a strong degree of confidence based on their treatment of the other visual diagrams in the paper that they’ll be able to rectify this elegantly.", "broader_impact_concerns": "Getting the language right on NL to PL is important, in my opinion, because I think getting it wrong could confuse many readers into thinking LLMs are handling all SE tasks, including those that have nothing to do with code/text. I think not only would that be a disservice to the larger community, but also to the authors and the strong work they’ve put together here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found the survey to be a comprehensive resource for a rapidly evolving space. The authors' desire to connect the differing NLP and SE perspectives to this area is especially laudable. While the original submission had a few weaknesses, these issues were addressed in the revisions.
Decision: Accept (poster)