Exploring and Evaluating Personalized Models for Code Generation
Authors: Andrei Zlotchevski
Dawn Drain
Alexey Svyatkovskiy
Colin Clement
Neel Sundaresan
Michele Tufano
Venue: ICLR 2022 Submitted (2022)
Keywords: code generation
custom models
NLP

Abstract: Large Transformer models achieved the state-of-the-art status for Natural Language Understanding and are increasingly the baseline architecture for source code generation models. Transformers are usually pre-trained on a large unsupervised corpus, learning token representations and transformations relevant to modeling generally available text, and then fine-tuned on a particular task of interest. While fine-tuning is a tried-and-true method for adapting a model to a new domain, for example question-answering on a given topic or a source code generation model, generalization remains an on-going challenge. Here we explore the ability of various levels of model fine-tuning to improve generalization by personalized fine-tuning. In the context of generating unit tests for Java methods, here we evaluate learning to personalize to a specific project using several methods to personalize transformer models for unit test generation for a specific Java project. We consider three fine-tuning approaches: (i) custom fine-tuning, which allows all the model parameters to be tuned; (ii) lightweight fine-tuning, which freezes most of the model's parameters, allowing a tuning of the token embeddings and softmax layer or the final layer alone; (iii) prefix tuning, which keeps language model parameters frozen, but optimizes a small project-specific prefix vector. Each of these techniques offers a different trade-off in total compute cost and prediction performance, which we evaluate by code and task-specific metrics, training time, and total computational operations. We compare these fine-tuning strategies for code generation and discuss the potential generalization and cost benefits of each in deployment scenarios.

Under review as a conference paper at ICLR 2022
EXPLORING AND EVALUATING PERSONALIZED MODELS FOR CODE GENERATION
Anonymous authors Paper under double-bling review
ABSTRACT
Large Transformer models achieved the state-of-the-ar status for Natural Lan- guage Understanding and are increasingly the baseline architecture for source code generation models. Transformers are usually pre-trained on a large unsupervised corpus, learning token representations and transformations relevant to modeling generally available text, and then fine-tuned on particular task of interest. While fine-tuning is a tried-and- and--ruee method for adapting a model to a new domain, for example prestion-answering
given topic or source code generation model, generalization remains an on on-going challenge. Here we explore the ability of various levels of model fine-tuning improve generalization by personalized fine-tuning. In the context of generating unit tests for Java methods, here we evaluate learning personalize to specific project using several methods personalize transformer models for unit test generation for a specific Java project. We consider three fine-tuning approaches: (i) custom fine-tuning, which allows all the model parameters to be tuned; (ii) lightweight fine-tuning, which freezes most of the model's 's parameters, allowing a tuning of the token embeddings and softmax layer or the final layer alone; (iii) prefix tuning, which keeps language model parameters frozen, but optimizes small project-specific prefix vector. Each of these techniques offers a different trade-off in total compute cost and prediction performance, which we evaluate by code and task-specific metrics, training time, and total computational operations. We compare these fine-tuning strategies for code generation and discuss the potential generalization and cost benefits of each in deployment scenarios.
INTRODUCTION
It is well-known that even the best models can fail to generalize properly to new domains, and even to new users of said models. For example, a model trained to answer questions in general may not answer StackOverflow questions well the questions in the training domain, software developer in an Enterprise environment with private code may have libraries and attribute name which differ from public source code used train code synthesis model.
The current dominant paradigm in Natural Language Processing (NLP) modeling is to pre-train a large transformer model (Vaswani al.) 2017a) on large corpus and then fine-tune on a particular task of interest. For example, question-answering (Q&A) model is generally first pre-trained on large corpus of textual data for the specific language (e.g., Wikipedia, and news articles in English), then fine-tuned on task-specific dataset of paired questions and corresponding answers. The pre-training process aims learning semantic vector representation the language and words, while the fine-tuning process specializes the model for a specific domain.
Transformer models are also increasingly the baseline architecture used for code generation tasks, such as writing methods from natural language description (Clement et al. 2020 Austin et al. 2021 Chen et al. 2021), or generating test cases from the focal method under test (Tufano et al.) 2021) Similarly for NLP tasks these models are pre-trained on large corpus of natural text and publicly available source code and then fine-tuned on a specific code-related task. Further, these models also may not generalize new domains of interest, and can benefit from task or user-specific fine-tuning, here called customization or personalization. Customization is particularly relevant for code generation models since it provides several benefits:

---

Under review as a conference paper at ICLR 2022
allows fine-tuning source code data that may not be available when training base model (e.g.. private repositories or internal codebases), enabling improved overall performances on codebases with proprictary dependencies and code styles;
the opportunity to improve data privacy by considering private or sensitive data only during the customization process on the client side;
the opportunity to reduce deployment cost customized models offer better user performance without increasing model size.
Custom models provide clear benefits to users and model providers. We envision serving tens hundreds of thousands of custom models, but doing so presents several logistical hurdles, including the costs of training, storing, and loading these models into GPU memory for inference. Worse, memory costs will only be exacerbated when working with ever larger and more powerful models.
For these reasons, investigate several customization approaches, some of which dramati- cally reduce the memory footprint and amortized computational cost introduced by custom models. Specifically, we consider three fine-tuning approaches: (i) custom fine-tuning, which allows all the model parameters to be tuned; (ii) lightweight fine-tuning, which only optimizes the token embedding representations or the final softmax layer, (ii) prefix tuning. which keeps language model parameters frozen, but optimizes small project-specific vector prefix.
In our extensive empirical evaluation we found that all the customization strategies lead to significant model improvements on target project in terms of both intrinsic and task-specific metrics. While there is no unambiguous winner among the customization strategies, each approach can provide spe- cific benefits in particular deployment scenarios. This paper provides insights on these customization strategies, their benefits and drawbacks, as well as providing guidelines and suggestions on which one to use based on the training cost, memory and storage, number of users, and deployment scenarios.
2 APPROACH
This section describes the proposed customization approach for code generation models. We begin by formally defining the customization process, then we provide details for each of the fine-tuning strategies.
2.1 CUSTOMIZATION PROCESS
We use the term customization to refer to the process of fine-tuning a model m, previously trained on generic dataset for a task t, with the goal of improving its performance The performance of a machine learning model m on a dataset p is measured by one or more evaluation functions f(m.p) where f can be either a maximization (e.g.. BLEU, top-k accuracy) or minimization (e.g., perplexity) function. The customization process is designed to modify the trainable parameters of the model m, obtaining the model m', such that the performance of m p is better than what was attained by m. Specifically, f(m', .p) > f(m, p) for maximization functions, or fin', p) fmm,p) for minimization functions.
specific dataset
In this work, m is an encoder-decoder transformer model, / is a code generation task, and p is a target software project to which we intend customize m.
2.2 CUSTOM FINE-TUNING
Custom fine-tuning is the most straightforward customization approach. The model to be customized taken as and trained on a selected project. All parameters are trainable during this process. Figure Ta shows the model during fine-tuning, where all the parameters from the encoder and decoder blocks, as well as embeddings and output layers can be modified.
2.3 LIGHTWEIGHT FINE-TUNING EMBEDDINGS AND OUTPUT LAYER (L-EO)
Fully fine-tuning model for every project user may be prohibitive in terms of storage and memory costs. As a result, we explore ways to mitigate these costs by reducing the number of parameters that vary from one custom model to another. In our lightweight fine-tuning experiments, achieve this
2

---

Under review as a conference paper at ICLR 2022
Linear Softmax
Linear Softmax
Lineer Softmax
Linear Softmar
ENcopot
Decoout
ERIDUDER
Disopee
ENCODER
DECOpen
PREFR
ENCODER
PREFEt
Dioooo
Dccoom
ENCOUsit
SRIDOER
DICOOSE
DECOUGR
PREFI
ENCosen
ENCCdoo
DECOO0t
ERCCODER
DOCODER
ENCoReR
DECOpOR
PRERX
ENCOXER
PREFEE
DECODUR
Decoio
BFCOFF
FNCOOER
BFCOOOR
FREFIX
ENCODER
PROFEE
DECOnN9
ENizzoo1
DOOODER
DEEDDOn
DEc000n
DEC0GEs
ENCOOCA
BEEOOCn
PRZFK
ENCOsER
PRSSEI
DECOO6A
DeCOess
ENCODER
DECONOR
ENCOzER
Moris
DECOUUN
PASRI
ENCOSER
DiCaoon
ENCCOER
DOCOBER
ENCODER
DEcODeR
PREFIX
ENCOZER
PREFEE
DEcooGn
Embaddings
Embeddings
Embeddings
Embeddings
(a) Custom
(b) L-EO
(c) L-LDB
(d) Prefix
Figure Overview of the Customization Approaches Transformer models during fine-tuning, where the frozen parts of the model (not trainable) are displayed in gray: (a) Custom fine-tuning modifies all parameters during training: (b) L-EO trains only the embedding and output layers; (c) L-LDB allows to train only the parameters of the last decoder block; (d) Prefix tuning adds a trainable prefix to the encoder and decoder blocks.
by freezing most parameters in the baseline model, and only keeping small subset trainable. Figure Tb shows the Lightweight fine-tuning Embeddings and Output Layer (L-EO) design, where most of the model parameters are frozen (displayed in gray), and we allow only the embedding and output layers parameters to be fine-tuned, following the approach in et (202I).
2.4 LIGHTWEIGHT FINE-TUNING LAST DECODER BLOCK (L-LDB)
In this lightweight fine-tuning strategy, shown in Figure Ic (L-LDB), most of the model's parameters kept frozen, while only the parameters in the last decoder block trainable, this includes: self-attention, encoder-decoder attention, layernorm and feedforward layers. This design decision of training only the last decoder block is motivated by experimental results analyzing the model's parameter changes during custom fine-tuning. Figure reports the average absolute changes, during fine-tuning, in the parameters belonging different Encoder and Decoder blocks for BART model. We observe that, as we go through the transformer model, the average change in parameter values tends to increase, with the last decoder block showing the highest changes in parameter values. As result, we hypothesize that it could be sufficient to tune the last decoder block and obtain performance improvements similar to the fully custom fine-tuned model.
2.5 PREFIX TUNING
Prefix tuning was first introduced by Li and Liang (021). with the goal of fine-tuni minettnnng a general model for different tasks. The technique concatenates sequence (prefix) of virtual tokens (trainable parameters) to the front the input of every encoder and decoder block. In our context, the intuition behind this approach is that the prefix embeds the properties of specific project, which allows the model to generate customized responses for that repository. Practically, we set the prefix length to 200 tokens, and thus with an embedding size of 1024, this gives total of 1024 200 24 2 10M trainable parameters. The prefix is initialized to the most frequent words in the repository for which the model is customized.
2.6 TRAINABLE PARAMETERS DURING FINE-TUNING
Table provides an overview of the number of total and trainable parameters involved in each customization process, in the case of BART Transformer model with 406M parameters. Custom fine-tuning allows to train 100% the 406M available parameters in the model. During L-EO finetuining, instead, only 13% (53M) parameters are trained. The L-IDB fientuning reduces the
3

---

Under review as a conference paper at ICLR 2022
0,000275
Averuge Aasolnee Chinge
0,000250
0.000225
0,000200
0,000175
0.000150
0.000125
0,000100
(Decosier 0)
(Decosrr 0)
Decosar 7)
(Decosier 8)
(Ducosarr 9)
(Dccoder 10)
(Decodey, 13)
(Dther
[Encosar,
(Encosar
(Encoder
(Encodee
Itrcceter
(Encoder
(Enooder
(occosar
Transoomer
Figure 2:
This figure shows the total average parameter change after fine-tuning to new project
domain, showing that the largest parameter changes occur in deeper parts of the model. This motivates our choice to try only fine-tuning the later layers of the model.
number of trainable parameters to 4.2% (17M). Finally, Prefix tuning has the lowest number of trainable parameters, only 2.4% (10M) of the total, but these are additional parameters added to the model, which reaches a total of 416M.
Parameters
Customization Process
Total
Trained
Custom
406M
406M (100%)
L-EO
406M
53M (13%)
L-LDB
406M
17M (4.2%)
Prefix
416M
10M (2.4%)
Table 1: Comparing the number of trainable parameters in each fine-tuning method.
EXPERIMENTAL DESIGN
The goal of our experimental design is to investigate whether custom models outperform the baseline model, leading performance improvements in terms of intrinsic metrics (RQ1), well extrinsic tase-fpecific metrics (RQ2). Next, we analyze and compare the different customization approaches in terms of training and compute costs (RQs) well model size and required storage for deployment.
In our case study, we chose Unit Test Case generation as our code generation task t, and AthenaTest baseline model m. which is BART transformer model pre- trained
by Tufano et al. (2221) on source code and English, and fine-tuned on Java unit test generation. The task is modeled as a translation task, where the input is focal method (i.e., method under test), and the output is test case which tests the focal method's correctness. We randomly sample 20 projects from the test set, each of those representing the dataset p on which custom model is fine-tuned. Specifically, for each project we start from the baseline model m and fine-tune four difffrentt custom models according the four proposed fine-tuming fine-tuning strategies. For each project and fine-tur fine-tuning strategy (e.g.. L-EO), we fine-tune and evaluate the models using 4-fold cross-validation. The models are trained until the best validation loss is reached, independently for every fold, every repository, and every customization approach. In total, we fine-tune and evaluate 20(projecls) x 4(approuches) x 4(folds) 320 models.
3.1 DATASET
Table reports information about the 20 GitHub repositories sampled from the test set, which will be used customize models. The table shows (i) the Project ID,, which will be used in the paper
4

---

Under review as a conference paper at ICLR 2022
reference specific project; (ii) the project name; (iii) the project size in terms of disk usage; (iv) the popularity of the project in terms of number of stars obtained on GitHub; (v) and the dataset size, which corresponds to the number of data points for the unit test generation task (i.e., pair of focal method and test case). The list of projects represent a diverse set of repositories with different size, domain, and popularity. They span from small personal projects (e.g., Tutoria aal with 6 stars), open source projects developed by large organizations such as Apache and Google.
Project ID
Name
Project Size (MB)
Stars
Dataset Size
26644682
Talend Data Prep
68.8
56
651
40735368
GeoTools
62.4
653
107330274
Titus Control Plane
36.0
302
660
52972024
Smart Actors
57.8
22
704
9714608
Arakhně Foundation Classes
17.9
13
753
60701247
Android Plugin for IntelliJ IDEA
1026.7
716
754
14550159
EverRest
5.3
24
761
9278888
Brave
18.8
2084
787
66940520
DHIS 2
118.1
211
862
33645537
Tutorials
34.4
878
62253355
Mobi
62.6
35
986
155883728
OakPAL
15.0
1005
4710920
Apache Dubbo
36.1
36231
1058
29603649
Wilma
6.7
40
1074
42949039
Herd
227.2
127
1249
1381 1381673
Drools
176.7
3908
1394
1244027
ModeShape
131.1
212
1550
73948366
AthenZ
38.8
639
1920
660443
Chemistry Development Kit (CDK)
214.8
305
2591
87849739
Eclipse DM Project
52.5
311
2842
Table Dataset- Projects used for customization
3.2 RQJ: INTRINSIC EVALUATION METRICS
RQ:: Do custom models obtain better performances intrinsic metries, such as BLEU and perplexity, w.r.t. the baseline? To begin, we investigate how the different model customization approaches described in Sec. score on intrinsic metries such as BLEU and perplexity. All approaches entail fine-tuning the baseline model to the dataset of a specific project, with the choice of parameters being tuned depending on the approach taken. The four variants are trained independently until the best validation loss is achieved. We report the BLEUU4 score and the mean perplexity per token on the test fold, for all the 20 projects. Next, we perform statistical tests to investigate whether the observed differences between the baseline and custom models are significant, as well as differences among the customization approaches. Specifically, we rely on the Kruskal-Wall test, a non-parametric statistical test.
3.3
RQ2: TASK-SPECIFIC PERFORMANCES
Do custom models improve on performance metrics specific to unit test generation? We want to investigate how the different customization approaches compare with respect to the downstream task of generating unit tests. Beyond BLEU score and perplexity, would like if custom models can produce the correct target code, how closely their unit tests mimic the repository style, if they perfectly match the desired output.
RQ2:
Perfect Matches: We compare the model's output string with the target developer- written unit test. If the two strings identical, this is considered perfect match. We do not take spacing and indentation into account as we are using a Java dataset (where indentation not required). We report the proportion of perfect matches among the top 5 model predictions.
5

---

Under review as a conference paper at ICLR 2022
Abstracted Code Matches: We pass the model output and target output through the src2abs tool (Tufano 2018), to obtain an abstracted version, masking variable names, method names, etc. We also do not distinguish between different objects of the same type.
Coding Style: For each project's custom model, we would like to determine how closely the model learns the developer's 8 personal programming style and preferences. To this end, we extract the collection of all identifiers (i.e., variables and functions' names) from the unit tests written by the developer as well those generated by the models. We then pass these text outputs through tf-idf vectorizer and compute the cosine similarity between them. This allows to compare the developer's and the models' word usage. We examine the similarity between the developer's 's unit tests and the baseline and custom models generated tests. This scores the vocabulary similarity of the unit tests with the model generated code.
3.4 R RQ3: TRAINING COST COMPARISON
RQ:: Given the same amount of compute, which custom models achieve the biggest perfor- mance improvement? Since our four training regimes tune different number of parameters, simply comparing the training time or number of optimization steps to reach the best validation loss may not be appropriate. For model with N parameters, we approximate the computation cost of forward pass to be C' R 2NV floating point operations per training token, with an additional correction for embedding layers. The backward pass takes roughly twice the amount of compute, but it is unnecessary for layers that are frozen. For additional details, we refer to Table in Kaplan et al. (2020). We report the resulting compute in petaFLOPS
4 RESULTS
4.1
RQ1: INTRINSIC EVALUATION METRICS
Table 3 presents the results of custom models in terms of the intrinsic metrics: BLEU and perplexity. Specifically, for each project, we report the average BLEU and perplexity over the four folds, achieved on the test set by the different customization strategy, well as the baseline model. We observe notable improvements both metrics for every project w.r.t. the baseline, with BLEU going from 16.1 achieved by the baseline model to 36-38 by custom models.
The statistical tests reported in Table Table4conff confirm that the improvement observed by the four customiza- tion techniques statistically significant w.r.t. the baseline < 1e-7). However, do not observe statistical significance in the differences among the customization strategies, meaning that, in terms of intrinsic metrics performances, the differences within margin of error.
4.2
RQ2: TASK-SPECIFE PERFORMANCES
The results in terms of task -seeific performances are presented in Figure 3 The plot Ba shows the top-k accuracy for perfect matches (solid line) and abstracted matches (dotted line), aggregated over the 20 projects. The baseline model outputs the same code structure (abstracted) in roughly 3% of all cases, and virtually never produces the exact target output (c)%). Moreover, its performance does not improve as we consider more predictions. All customization processes show significant improvement compared the baseline. Specifically, these improvements are observed for every single project (full results will be available on our online appendix). Customized models produce the correct code structure as their top prediction in 1-3--14% of instances, and perfect match in ~4-6% of cases. They also tend to improve as we consider their top 5 predictions. Between the different customization processes, Custom consistently performs the best, closely followed by Prefix and L.-LDB. When considering abstracted code matches, these three approaches are nearly identical. L-EO, however, performs slightly worse than the others.
Plot 3b shows the distribution of tf-idf cosine similarity computed between identifiers used in the developers' written tests and the models' generated outputs. We observe that the distribution for custom models is skewed towards the higher values of cosine similarity. This result demonstrates that custom models tend to use variable and function names that are more similar to what developers used in their own tests.

---

Under review as a conference paper at ICLR 2022
BLEU4
Perplexity
Project
Base
Cust.
L-FO
LALD
Prefix
Base
Cust.
L-FO
L-L.BB
Prefix
26644682
14.1
32.9
31.6
31.9
34.0
1.275
1,212
1.208
1.238
1.197
40735368
18.5
30.7
29.0
29.4
29.1
1.276
1.186
1.197
1.186
1.194
107330274
14.8
38.0
35.0
35.9
35.7
1.273
1.160
1.168
1.164
1.175
52972024
10.2
31.8
33.2
1.271
1.142
1.146
1.135
1.135
9714608
14.7
41.0
38.1
40.4
40.2
1.263
1.155
1.145
1.150
1.138
60701 247
10.8
28.9
24.4
25.9
26.6
1.267
1.187
1.190
1.172
1.176
14550159
20.0
49.5
47.2
46.6
46.4
1.245
1.121
1.122
1.116
1.124
9278888
17.3
46.8
44.5
47.2
47.8
1.272
1.137
1.152
1.138
1.140
66940520
17.4
37.9
33.9
37.7
1.264
1.154
1.163
1.154
1.150
33645537
17.0
30.4
31.2
32.0
31.0
1.264
1,231
1.200
1.192
1,211
62253355
14.7
48.0
45.7
47.3
48.0
1.292
1.113
1.114
1.114
1.116
155883728
13.7
41.3
37.5
39.3
39.5
1.238
1.132
1.148
1.146
1.140
4710920
39.1
38.1
38.6
1,218
1.161
1.162
1.167
1.160
29603649
19.1
58.4
54.9
56.6
56.8
1.266
1.096
1.110
1.099
1.098
42949039
17.0
38.2
37.7
37.5
37.3
1.238
1.154
1.152
1.154
1.148
1381673
14.3
33.3
29.3
30.9
30.8
1.261
1.133
1.152
1.138
1.138
1244027
19.6
30.1
29.7
30.0
30.0
1.244
1.142
1.160
1.142
1.150
73948366
12.0
33.1
31.8
33.6
1.267
1.161
1.157
1.159
1.164
660443
15.0
34.0
37.2
36.5
34.3
1.281
1,180
1.170
1.169
1.177
87849739
13.4
45.1
47.0
48.9
46.8
1.259
1.138
1.136
1.124
1.144
Average
16.1
38.4
36.9
37.8
37.7
1.262
1.153
1.158
1.153
1.154
Table 3: The BLEU score and perplexity for the customization methods evaluated on the 20 projects in our test set.
BLEU4 I.FOO
Perplexity L-FO
Base
Cust.
L-LDB
Prefix
Base
Cust.
L-LDB
Prefix
Base
3e-08
3e-08
3e-08
3e-08
3e-08
3e-08
3e-08
3e-08
Cust.
3e-08
0.4
0.7
0.7
3e-08
0.5
0.9
0.9
EO
3e-08
0.4
0.5
0.7
3e-08
0.5
0.5
0.5
LDB
3e-08
0.7
0.5
0.9
3e-08
0.9
0.5
0.8
Prefix
3e-08
0.7
0.7
0.9
3e-08
0.9
0.5
0.8
Table 4: Kruskal-Walli Test p-values testing the significance of the pairwise hypothesis that one customization method is superior than another. Custom strategies are significantly better than baseline.
Tap-K Accuracy
Similarity output target output
20
Bazeine
Basslina
Custom
LE0
L-LDS
14.00
Profix
Norch
Pamoorr
5-
dimilartty
(a) Exact (solid) and abstracted (dotted) match rate
(b) Coding style identifier similarity
Figure 3: Task-specific metrics (a) custom models outperform the baseline in terms of perfect matches (solid line) and abstract matches (dotted line); (b) custom models generate code that uses identifiers (i.e., variable and function names) that are more similar to the project codebase.
7

---

Under review as a conference paper at ICLR 2022
4.3 RQ3: TRAINING COST COMPARISON
For each customization process, we plot validation loss as function of compute, as defined in section 3.4 The results are presented in Figure where the light lines represent the validation loss curve for each individual project and fold, while the bold line represents the average for each custom strategy. First the fact that its validation loss starts much lower than L.-OO and L--DBB. Custom also outperforms other customization processes when given limited amount of compute. However, we observe that beyond certain amount of compute, Custom and L-LDB tend to achieve similar performances. In contrast, L-BO starts at the same validation loss as L-LDB but converges much slower to the best loss, requiring 2-3 times much compute.
that Custom achieves very large gains during the first epoch, as evidenced by
Since the prefix parameters suffer from poor initialization, Prefix is the most expensive customization process. To overcome this problem, it is possible to first train the prefix large generic dataset. Then, given proper hyperparameter tuning, is possible to substantially cut down compute cost for customizing the prefix.
Custom
4.0
L-EO
L-LDB
3.5
Prefix
3.0
Cross entropy loss
2.5
2.0
1.5
1.0
0.5
10
10
10
102
103
Compute
Figure 4: Validation Loss Compute (PF-seconds) Light lines represent the validation loss curve for each individual project and fold, while the bold line represents the average for each custom strategy. Custom the most efficient, lightweight approaches require slightly more compute reach comparable validation loss, while prefix is the least efficient, suffering from poor initialization.
DISCUSSION & LESSONS LEARNED
The four customization strategies considered in this work are effective improving a code generation model's performances given software project. Specifically, all custom models significantly outperform the baseline in terms of intrinsic metrics (i.e.. BLEU and perplexity) as well as task- specific metrics (i.e., abstract and raw matches). While the differences among the customization approaches are not significant (no clear winner), each strategy offers specific advantages in different circumstances and deployment scenarios.
8

---

Under review as a conference paper at ICLR 2022
Custom fine-tuning achieves the overall best performances and the customization process is relatively fast and efficient. This is somewhat expected, since this customization strategy allows all the model's parameters to be tuned on the specific project. This characteristic also leads to the major disadvantage of this approach: each custom model is an entire copy of the original model. Storage and inference costs could become prohibitive when serving many users with personalized custom models.
Lightweight fine-tuning achieves good results while training fewer parameters. This allows to serve potentially many users with custom models which can be stored and loaded efficiently. Specifically, L-LDB trains fewer parameters than L-EO, however the latter could allow to deploy the embedding and output layers on the user side, with
focus.
Prefix fine-tuning trains the lowest number of parameters (only 2.4% for a BART model), while improving over the baseline. However, increases the total number of parameters of the model (prefixes are additional virtual tokens) and requires more compute time to achieve good performances, mostly due to the prefix initialization problem. On the bright side, this strategy allows to batch together requests from different users (with different prefixes), which can be processed by a single model, generating personalized outputs.
6 RELATED WORK
This work is related to two areas of the existing literature: neural source code generation and model personalization. Neural code generation has generated an intense recent interest in NLP, using Transformer models Vaswani et al. (2017b) in particular for code completion Svyatkovskiy
al.
(2020 (019):[Clemees et al. (2020); Raychev et al. (2014); Bruch et al. (2009); Brockschmidt et al.
(2018 code synthesis from examples Chen et al. (2018), natural language to code Clement et al.
(2020) Chen et (201 Austin et al. 2021), code feature summarization Liu et al. (202) 1) Moreno ct al. (2013); Scalabrino et al. (2017); Wan et al. (218)); Alln et al. 2018); Moreno et al. (2014),
code search Husain et al. (2019 Feng et al. (2020), unit test generation Tufano al. (2021) and even
bug fixing Drain et al. (2021) and detection Zhai et al. (2020). This paper naturally is an extension and
evaluation of personal ized unit test generations as studied by Tufano al. (2021. and important contribution to the understanding optimization in a deployment scenario.
Much of the previous literature personalized models focuses client-side training to keep data on device Shor et al. (201)); Popov et al.] (2018), and most work is in the domain of search query completion Jaech and Ostendorf (2018), 8), natural language completion Poppov et al. (2018), or even automated speech recognition Shor et al. (2019). Naturally this work extends the domain of evaluation beyond natural language tasks and into the software engineering domain. This paper does not evaluate methods for client side training with restricted resources, however, as the most powerful large language models which enable state of the art code synthesis have 10-100 million parameters. At the time of writing such large models cannot be executed in a reasonable amount of time on most consumer laptops. We leave to future work extending these studies to models which have been pruned, quantized, distilled, and optimized to be ran in limited resource environments.
7 CONCLUSION
In this paper we explored different ways to customize code generation model for a given codebase, with the goal of improving its performances on target project. We described and analyzed four customization strategies and applied them on 20 different software projects for the task of generating unit test cases. Specifically, we considered the following strategies: (i) custom fine-tuning, which allows all the model parameters to be tuned on the target project; (ii) I-EOO fine-tuning, lightweight training which freezes most of the model's parameters, tuning only embedding and output layers; (iii) L-LDB fine-tuning, lightweight training which only tunes the last decoder block; (iv) prefix tuning, which keeps language model parameters frozen, but optimizes a small project-spectee vector (prefix).
In our extensive empirical evaluation we found that all the customization strategies lead to significant model's 's improvements on a target project, in terms of both intrinsic and task-specific metrics, with the custom models adapting to the coding style of the target project. While there is clear winner among the customization strategies, each approach can provide specific benefits in particular deployment scenarios.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured representations of code. arXiv preprint 1808: 01400, 2018.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 100.00722.. 2021.
Marc Brockschmidt, Miltiadis Allamanis, Alexander L Gaunt, and Oleksandr Polozov. Generative code modeling with graphs. In International Conference on Learning Representations, 2018.
Marcel Bruch, Martin Monperrus, and Mira Mezini. Learning from examples to improve code completion systems. In Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, pages 213-222, 2009.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107 100.032107..3374 2021.
Xinyun Chen, Chang Liu, and Dawn Song. Towards synthesizing complex programs from input- output examples. In International Conference on Learning Representations, 2018. URL https
7/openreveew net. www.:neeett/niiiieooeoooooto f1o.orrmmm--sppLeexX
Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. Pymt5: Multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9052-9065, 2020.
Dawn Drain, Chen Wu, Alexey Svyatkovskiy, and Neel Sundaresan. Generating bug-fixes using pretrained transformers. arXiv preprint arXiv:2 104007896 2021.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1536-1547, 2020.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. Code- searchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.000 1e.../9999999,,, 2019.
Aaron Jaech and Mari Ostendorf. Personalized language model for query auto-completion. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 700-705, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, 0bs/2001.08366 2020. WtLt::tss://arr
wwwp/oa////////////////../...aet /bb//////////// / /2001201101 08361
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint ww..vx2201.00/11 2021.
Shangging Liu, Yu Chen, Xiaofei Xie, Jing Kai Siow, and Yang Liu. Retrieval- generation for code summarization via hybrid (gnn). In International Conference on Learning Representations (ICLR), 2021.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. CoRR, 0a./22103.0524,, 2021. URL
05247
Laura Moreno, Jairo Aponte, Giriprasad Sridhara, Andrian Marcus, Lori Pollock, and K Vijay- Shanker. Automatic generation of natural language summaries for java classes. In 2013 21st International Conference Program Comprehension (ICPC), pages 23-32. IEFE, 2013.
10

---

Under review as a conference paper at ICLR 2022
Laura Moreno, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrian Marcus, and Gerardo Canfora. Automatic generation of release notes. In Proceedings of the 22nd ACM SIGSOFT International Symposium Foundations of Software Engineering, pages 484-495, 2014.
Vadim Popov, Mikhail Kudinov, Irina Piontkovskaya, Petr Vytovtov, and Alex Nevidomsky. Dis- tributed fine-tuning of language models on private data. In International Conference on Learning Representations, 2018.
Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementaatinn, pages 419-428, 2014.
Simone Scalabrino, Gabriele Bavota, Christopher Vendome, Mario Linares-Vásquez. Vásquez, Denys Poshy- vanyk, and Rocco Oliveto. Automatically assessing code understandability: How far are we? In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 417-427. IEEE, 2017.
Joel Shor, Dotan Emanuel, Oran Lang, Omry Tuval, Michael Brenner, Julie Cattiau, Fernando Vieira, Maeve McNally, Taylor Charbonneau, Melissa Nollstadt, et al. Personalizing asr for dysarthric and accented speech with limited data. arXiv preprint 1907.1990..13511 2019.
Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, and Neel Sundaresan. Pythia: Ai-assisted code completion system. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2727-2735, 2019.
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. Intellicode compose: code generation using transformer. In Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann, editors, ESEC//SE '20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Viriual Event, USA, November 8-13, 2020, pages 1433- 1443. ACM, 2020. doi: 10.1145/3338089.4417033 URL httph: /doi: /ttp::://ddii.rr/11... 1145/ 0188089.3417000
Michele Tufano. src2abs. http:p///gggtaa...oooo/.caaaaaaa
2018.
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. Unit test case generation with transformers and focal context. arXiv preprint arXiv:2009.05617 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, Abs//170.00376, 2017a. URI.
http:// /ttp:///avvvv//aa///a/ org/ abs/ 176 03762
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008. 20176 7b.
Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying. Jian Wu, and Philip S Yu. Improving automatic source code summarization via deep reinforcement learning. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pages 397-407, 2018.
Juan Zhai, Xiangzhe Xu, Yu Shi, Guanhong Tao, Minxue Pan, Shiqing Ma, Lei Xu, Weifeng Zhang, Lin Tan, and Xiangyu Zhang. Cpc: Automatically classifying and propagating natural language comments via program analysis. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pages 1359-13711 2020.
11


Reviews:
[{"review_id": "fOBU1wV6enr", "reviewer": "Reviewer_v2C4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Welcome empirical study of fine-tuning strategies specifically for source code applications.\n+ Good use of statistical analysis helps correct for small dataset.\n\nWeaknesses:\n- Empirical analysis is too narrow, focusing on a single task and dataset, as well as very small models.\n- Technical contributions, such as the fine-tuning of a single decoder layer, appear very ad-hoc.\n- Overall results provide no actionable new insight into the use of fine-tuning for software engineering.\n\nFine-tuning and other domain adaptation methods are rightfully becoming important subjects in machine learning driven software engineering research, so an empirical study of this process is relevant. However, the approach in this work is too narrow to provide much in the way of actionable insights. The analysis is based on a single paper and dataset, on which it considers mostly long-established fine-tuning methods. The results are exactly as expected; complete-model fine-tuning is somewhat slower, but effective; fine-tuning just one or two layers is a bit faster and slightly less accurate (depending on compute), etc. Indeed, much of the motivation is very generic to the motivation for, and problems with, fine-tuning in general. The use of statistics to avoid drawing unwarranted conclusions is a welcome addition and further reinforces that no new results are found.\n\nIn terms of the original motivation, which identifies key factors such as privacy and memory/compute footprint, the results do not provide a particularly satisfying conclusion. The analysis entirely considers other open-source projects, so it shines little light on fine-tuning behavior within corporate code-bases, which may well have much more different characteristics to the training data than this test set. And the compute results in Figure 4 suggests that simply fine-tuning the full model is already the best solution at around 1 PF-second, which translates to much less than a minute on virtually every commonly used GPU. That seems like a _very_ small cost for adapting to a new project, which somewhat undercuts the need for any other analysis here. On the other hand, arguable more product-relevant issues such as working memory usage and inference time/cost are not addressed by this work.\n\nThe main technical novelty in this work stems from the addition of fine-tuning a single decoder layer. This is based on an analysis that shows that higher layers are updated (rather) slightly more in terms of parameter change when fine-tuning the full model. This feels very ad-hoc; it falls well short of more grounded analyses of fine-tuning different layers such as in Yosinski et al. (NeurIPS 2014), and its motivation suffers from the clear problem that parameter changes in full-model fine-tuning are not necessarily indicative of which layer \"needs\" to be fine-tuned the most. Updates naturally cascade across layers, so the behavior in Figure 2 seems natural when updating an entire model. The results indeed do not show a substantial difference between this and updating another layer (embedding/output), except perhaps at very specific compute values, for which no significance test was applied.\n\nDetailed Comments:\n- Introduction: please add references to substantiate the claims in the first two paragraphs\n- P2, top paragraph: \"performances\" -> \"performance\"\n- P2: the first two bullet points in the introduction paint a very similar picture, mostly because of the second half of the first, which also essentially talks about privacy.\n- The use of the word \"Custom\" throughout the paper is rather odd. For one, it is often used interchangably with \"customized\", but those evoke rather different ideas -- \"custom\" suggests a specially designed model for a new task, which does not resonate the contributions here. \"Customized\" should really just read \"fine-tuned\"; I see no reason for another term. The variant called \"Custom fine-tuning\" (2.2) would be better called something like \"Full model fine-tuning\".\n- P3, last paragraph: \"finetuining\" -> \"finetuning\" (twice)\n- Table 2/3: why use such long IDs, instead of simply numbering the projects for this paper?\n- What motivated fine-tuning both embeddings and the output layer, besides following Lu et al. -- are these weights tied in this model?\n- Figure 2 is rather hard to read, primarily in terms of both axes' labels. Please also explain \"(Other, 1)\"\n- Comparing matching rates on abstracted code mostly makes sense in terms of ignoring identifiers, but not distinguishing between different objects of the same type means this metric does not capturing semantic correctness. Please clarify to what extent this impacts the results; ideally ablate this latter decision to better isolate the effect of just naming conventions.", "summary_of_the_paper": "Software projects tend to differ strongly from one another in terms of coding style, which significantly reduces the efficacy of even large pretrained models in new project contexts. This work explores the practical performance of several fine-tuning strategies of a model trained for a software engineering task (test-case generation) to new project contexts. The results suggest that fine-tuning can substantially improve model performance, but the specific choice of fine-tuning strategy makes very little difference.", "summary_of_the_review": "The contribution is very narrow, targeting a single (far-from broadly used) task and dataset, and involving a relatively small set of projects that are not representative of the type of proprietary code the work is motivated as supporting. The results provide no new information beyond what is generally understood about fine-tuning, and lack the kind of systematic rigor seen in related work. Technical novelty is low, consisting of a single new fine-tuning approach that echoes traditional analyses, the motivation for which is also rather questionable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Xzeao9MU-zX", "reviewer": "Reviewer_BJx7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper compares the standard strategy of fine-tuning of all weights against two lightweight strategies and the prefix tuning strategy of (Li and Liang 2021). The lightweight strategies include fine-tuning only the embedding and output layers (Lu et al. 2021) and fine-tuning the last decoder layer.\n\nThese strategies are evaluated on the task of generating Java unit tests on 20 different projects, where the personalized models outperform the baseline model on BLUE4, perplexity and task-specific measures. The performance of various personalization strategies is mixed across the projects, with the standard strategy of fine-tuning all weights being the best overall. Thus, the paper does not offer any new dominant strategy within this space. It is unclear whether these observations hold in general for code generation as the evaluation is performed only on a single task.\n\nThe motivation of the paper is clear, but the proposed method of project-wise fine-tuning is expensive and difficult to scale in practice. Alternative methods such as few-shot learning have been shown to be successful for customizing large pre-trained models in the literature. These methods do not incur the cost of re-training. The paper does not evaluate the proposed strategies against them.\n\nThe assumption of having several hundred to thousand labeled examples for each project is also not practical. In contrast, few-shot learning usually requires only a handful of examples (something like 1-5).", "summary_of_the_paper": "The paper proposes personalization of a baseline model to a project by fine-tuning it on the project-specific labeled examples. It demonstrates that personalization yields performance benefits over the baseline model. The baseline model (Tufano et al. 2021) is obtained from a pre-trained Java BART model after fine-tuning for the task of generating Java unit tests. The paper evaluates four fine-tuning strategies that select different subsets of weights to be fine-tuned.", "summary_of_the_review": "The paper has limited technical novelty and empirical contributions. The project-wise fine-tuning strategy is not practical. There is no comparison against the few-shot learning techniques which can customize a model with a smaller number of examples and without re-training.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "FbPHR3QhBJd", "reviewer": "Reviewer_1jM4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n- The paper studies a relevant problem of customizing models for code towards individual/group/project preferences.\n- The domain of personalized unit test case generation is novel, and could be useful for the broader code generation research community.\n- Insights from the paper have, in my opinion, applications to settings beyond the specified server-side customization. Insights from RQ3, for example, can be useful for clients fine-tuning the model on their local devices to optimize for compute time and cost.\n\n**Weaknesses:**\n- This paper is mainly an empirical study of the effectiveness of various model fine-tuning strategies on the performance of the fine-tuned model. As an empirical study, I found the following things lacking:\n\n    - **Evaluation:** This work utilizes BLEU score, Perplexity score, and Exact Match (EM) scores to evaluate the models. My concern here is that BLEU score is designed primarily for natural language evaluation and has been shown to not be a good measure for code evaluation [1]. Exact Match is not a good measure either as it simply checks for string equivalence while a piece of code can be written multiple ways and achieve the exact same goal. Why did the authors not choose to evaluate the generated test cases through execution, or the ability to compile as [2] does, or for their code coverage metrics? The paper does not touch upon these methods of evaluation, which I would argue are more important to judge the quality of generated test cases from an algorithmic and user point of view.\n\n   - **Conclusion:** The conclusion of the study are: \n       - All fine-tuning approaches improve the baseline model - this follows naturally from the definition of fine-tuning. Modifying the model weights over specific data distribution will improve the performance over a similar distribution during test time\n      - Different customization strategies have different pros and cons: This is a trivial observation from how the customization strategies are defined. Modifying fewer weights will require fewer different sets of weights to be stored for customization.\n     - All customization strategies achieve the same performance albeit with different levels of compute: This is the only insight that I found somewhat interesting, and something that could be useful to practitioners.\n\n**Other clarifications:**\n\n- Figure 2: There is no information on how this figure is created. It is mentioned as a motivation for L-LDB (Section 2.4), but does not describe what dataset/project that particular model is trained on, for how many iterations, and what the resulting performance is. Having experimental details of the procedure behind Figure 2 are important to understand the conclusions.\n\n- Section 3.1 (Dataset): The authors state that they sample 20 project from the test set to fine-tune the model. They do not state what test set are these 20 projects subsampled from? Is it preexisting test set they utilize for the task, or they create a new dataset from GitHub? How are the parallel dataset of function <--> unit test created? Can the authors please provide more details on this.\n\n- How is the *Abstracted Code Matches* metric computed? The paper simply mentions the conversion of code to an abstract form using *src2abs* tool, but does not discuss how the abstracted forms are matched thereafter.\n\n- Section 3.4: The authors state that *\"Backward pass takes roughly twice the amount of compute, but it is unnecessary for frozen layers\"*. This is only true for L-LDB setting where once the final Decoder layer is fine-tuned, the backward propagation can be terminated. For all other cases, backward propagation needs to be done and the computation graph needs to be computed until the last trainable parameter, which is generally the entire network.\n\n- The metric for compute (PF-seconds) unit is not defined or prior work utilizing/defining this metric cited anywhere.\n\n[1] Hendrycks, Dan, et al. \"Measuring Coding Challenge Competence With APPS.\" arXiv preprint arXiv:2105.09938 (2021).\n\n[2] Tufano, Michele, et al. \"Unit Test Case Generation with Transformers.\" arXiv preprint arXiv:2009.05617 (2020).", "summary_of_the_paper": "- This work studies customizing models for code towards specific projects/coding standards/preferences for unit test case generation task. It specifically studies this in the context of server-side customization, where one entity would need to maintain multiple customized models, as opposed to client-side customization where each client can store its own customized model.\n- It is an empirical study of how 4 customization strategies -- a) modify entire model weighs, b,c) modify a chosen subset of model weights, and d) prefix tuning -- affects the performance of the fine-tuned model on the test case generation task.\n- This work finds no substantial difference in the final fine-tuned performance of any of the customization strategies, but argues that each strategy has its advantages and disadvantages, therefore allowing the user/implementor of this approach to make a more informed choice.", "summary_of_the_review": "Overall, I found the problem well motivated but the empirical study and conclusions a little underwhelming. The paper could use A) better evaluation, B) More concrete conclusions and more discussion on when a particular technique might be useful over others (maybe experimental design imitating a particular situation to demonstrate the effectiveness of one approach over another could help), C) More qualitative and quantitative analysis into the test cases generated from various models, and D) Adding text to address concern I raised in the main review (dataset origins and collection procedure, compute metric description, etc.)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7bin_tpv32c", "reviewer": "Reviewer_bTU6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\n- The paper is easy to follow in most parts\n\n*Weaknesses*\n\n - No novelty in the approach: The paper just analyzes already existing and well known approaches on a pre-existing model, dataset and task. The metrics used in the paper are pretty standard as well and there is no novelty at that end.\n\n- No interesting insights: I find no interesting conclusions drawn from the analysis experiments. There was nothing discovered in the paper that is already not well known ( atleast for NLP). Just translating the same experimental settings and methods for one specific task and one model in code generation and not resulting in any interesting insights makes a very weak case for acceptance for me, keeping in view the ICLR standards.\n\n- Lacking thorough evaluation: If empirical analysis is the only contribution, then I expect authors to be more thorough and extensive in terms of the following: \n i) considering more tasks like bug repair or code autocompletion\nii) considering more datasets and programming languages per task\niii) considering different models of source code especially the large language models like Codex\n\n- Other questions/ concerns:\ni) Is tf-idf similarity the best way to measure coding styles? I will suggest something that uses the parse trees and measure node-wise similarity. I am also not sure that measuring the tf-idf based on the identifiers from the unit tests previously written by the developers and the ones that are used in the test project say much about coding styles. The identifiers may be named purely based on the project for which the unit test case is written rather than the style of the developer.\nii) \"On the bright side, this strategy allows to batch together requests from different users (with different prefixes), which can be processed by a single model, generating personalized outputs.\" I am not sure how exactly will this work.\niii) Even though it is stated in the abstract, introduction and conclusions that the authors specify guidelines as to when to use a particular strategy they don't discuss this explicitly anywhere in the paper. I was expecting a section dedicated to just discussing the specific use cases where a particular finetuning strategy can be used.\niv) I feel the details about the dataset like Table 2 can be moved as part of the supplementary material. I'd give more space to the analysis rather than specific implementation details.", "summary_of_the_paper": "The paper analyses three types of fine-tuning approaches: (a) custom, (b) lightweight and (c) prefix on AthenaTest model for the task of unit test case generation in Java. The paper report results for 20 test projects in teens of BLUE, perplexity, task-specific performance as well as training cost computation.", "summary_of_the_review": "No significant insight and not thorough experiments from an analysis paper point of view.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an empirical study of different strategies for fine tuning a large language model for the task of generating Java Unit tests *for a specific project*.

As several reviewers pointed out, the setup itself is fairly impractical, requiring fine-tuning on an individual project, thus making it applicable only to the very tail-end of very large projects where the investment of doing this would make sense and where one could reasonably collect sufficient data for that project. 

On top of that, the paper contributes relatively little in terms of novel techniques. This in itself would be OK if the paper presented some extremely important empirical evidence. However, reviewers also raised some important concerns with the empirical evaluation itself. For example, as reviewer 1jM4 pointed out, there is prior research explicitly showing that the BLEU score is not a good measure for code evaluation. 

Overall, the meta-reviewer agrees with the reviewers that this paper is below the bar for publication.
Decision: Reject