On-the-Fly Adapting Code Summarization on Trainable Cost-Effective Language Models
Authors: Yufan Cai
Yun Lin
Chenyan Liu
Jinglian Wu
Yifan Zhang
Yiming Liu
Yeyun Gong
Jin Song Dong
Venue: NeurIPS 2023 poster (2023)
Keywords: Code Summarization
Adaptation
Language Model
TL;DR: We design Adacom to re-training the model on the fly to strengthen the helpful samples and unlearn the harmful samples for each test sample.

Abstract: Deep learning models are emerging to summarize source code to comment,
  facilitating tasks of code documentation and program comprehension.
  Scaled-up large language models trained on large open corpus have achieved good performance in such tasks.
  However, in practice, the subject code in one certain project can be specific,
  which may not align with the overall training corpus.
  Some code samples from other projects may be contradictory and introduce inconsistencies when the models try to fit all the samples.

  In this work, we introduce a novel approach, Adacom, to improve the performance of comment generators by on-the-fly model adaptation.
  This research is motivated by the observation that deep comment generators
  often need to strike a balance as they need to fit all the training samples.
  Specifically, for one certain target code $c$,
  some training samples $S_p$ could have made more contributions while other samples $S_o$ could have counter effects.
  However, the traditional fine-tuned models need to fit both $S_p$ and $S_o$ from a global perspective, 
  leading to compromised performance for one certain target code $c$.
  In this context, we design Adacom to
  (1) detect whether the model might have a compromised performance on a target code $c$ and
  (2) retrieve a few helpful training samples $S_p$ that have contradictory samples in the training dataset and,
  (3) adapt the model on the fly by re-training the $S_p$ to strengthen the helpful samples and unlearn the harmful samples.
  Our extensive experiments on 7 comment generators and 4 public datasets show that
  (1) can significantly boost the performance of comment generation (BLEU4 score by on average 14.9\%, METEOR by 12.2\%, and ROUGE-L by 7.4\%),
  (2) the adaptation on one code sample is cost-effective and acceptable as an on-the-fly solution, and
  (3) can adapt well on out-of-distribution code samples.

On-the-Fly Adapting Code Summarization on Trainable Cost-Effective Language Models
Yufan Cai
Yun Lin"
Chenyan Liu National University of Singapore chenyan@u. wws. edu
Shanghai Jiao Tong University National University of Singapore cai yufanuu. nus. edu
Shanghai Jiao Tong University lin_ Lin_yundsjta tuuudu.ccoud
Jinglian Wu
Yifan Zhang
Yiming Liu
National University of Singapore j jinglian_ wudu nus edu
National University of Singapore
National University of Singapore e09457948u. nus edu
yfzhangênus edu. sg
Yeyun Gong
Jín Song Dong
Microsoft
National University of Singapore desdjs@nus edu. sg
yegong@microosft..com
Abstract
Deep learning models are emerging to summarize source code to comment for code documentation and program comprehension. can achieve good performance by training the model large training corpus. However, in practice, the code samples from different projects can have contradictory training signal for learning a deep comment generator, making the model struggled fit all the training samples.
In this work, introduce novel approach, AdaCom, to improve the performance of comment generators by on-the-fly model adaptation. This research is motivated by the observation that deep comment generators often need to strike a balance as they need to fit all the training samples. Specifically, for one certain target code c, some training samples Sp could have made more contributions while other samples S, could have counter effects. However, the traditional fine-tuned models need both Sp and S, from a global perspective, leading compromised performance for one certain target code In this context, we design AdaCom to (1) detect whether the model might have a compromised performance on a target code and (2) retrieve few helpful training samples Sp that have contradictory samples the training dataset and, (3) adapt the model on the fly by re-training the Sp to strengthen the helpful samples and unlearn the harmful samples. Our extensive experiments on comment generators and public datasets show that (1) AdaCom significantly boost the performance of comment generation (BLEU4 score by on average 14.9%, METEOR by 12.2%, and ROUGE-L by 7.4%), (2) the adaptation one code sample is cost-effective and acceptable on-the-fly solution, and (3) AdaCom can adapt well on out-of-distribution code samples.
Introduction
Deep learning models are increasingly emerging to generate comments from the source code, facilitat- ing programmers' tasks such code documentation (27). program comprehension [18), and reverse- engineering 17] Regarding the code-to-comment generation problem as a language translation or
*Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

summarization problem, many deep language models, such as CodeBERT [TI, GraphCodeBERT [5]. CodeTS [37], and CodeGPT [4, are designed as sfate-of-the-arr code comment generators. The forthcoming ChatGPT 13 also demonstrates its effectiveness in emulating human-like interpretation of the target code.
On one hand, to improve the model performance, deep language models have evolved to scale up their size. (Ultra-)lagee language models such as GPT3 (28), CodeX 77. and InstructGPT [29] have shown their effectiveness. However, they usually consist of millions of neurons, which results in substantial training and maintenance costs for individuals.
On the other hand, the relatively small models such as CodeBERT and CodeT5 are more economical and maintainable, but they can struggle to compromise their performance different subsets of training samples. Our empirical study with the CodeBERT model on the CodeSearchNett 19] dataset shows that on average 49.51% samples the model only fits some consistent training samples for one certain test case, it will show much better performance. Overall, we observe that the conflicting effects are prevalent, regardless of the types of models and the types of training corpus.
or less "conflicting" with each other. In other words,
For the sake of a cost-effective comment generator, we propose AdaCom, on-the-fly model adaptation solution to boost the performance of deep comment generators. We design AdaCom to fulfill three research goals:
Compromise Detection: Given a source code, AdaCom aims to detect whether a comment generator may have a compromised performance the source code.
Samples Searching: Given a source code and a dataset, AdaCom aims to search for a few samples in the training dataset that can contribute to mitigating the model bias.
On-the-Fly Model Tuning: Given source code and few labeled samples, AdaCom aims to retrain the model on the fly to boost its performance over the source code.
AdaCom takes input from target code c, comment generator 9. and a dataset D. AdaCom adapts g on the fly into g' so that g' can have a better performance on generating the comment of c. Our rationale lies in that there could be small subset of samples D, C D which is more helpful for the comment generator to learn to summarize e. Technically, we first build an influential graph Gin on the dataset D where each node is a sample and the edge between two nodes indicates the helpful or harmful relation between the two training samples. Secondly, given a target code e, we use a nooell-representation based measurement to find the potentially semantic helpful samples of in the training dataset. Then, we enrich the set of helpful samples D based on the G.nf. Those samples chosen to fine-tune the model in a lightweight way that the model can (I) better learn helpful samples and (2) unlearn harmful samples. Finally, we retrain the model to g' with those samples on the fly for the target code
The solution of AdaCom is orthogonal to many deep- seep-learning-bast based comment generators including improving model architectures 25 model training [5), and code representation [22].
We evaluate AdaCom on boosting 7 comment generators over datasets. The experimental results show that (1) AdaCom can significantly boost the performance of comment generation (the BLEU4 score by on average 14.9%, METEOR by 12.2% and ROUGE-L by 7.4%), (2) the whole adaptation
on an individual code sample takes a very small run-time overhead, and (3) AdaCom can generalize
well towards out-of-distribution code samples.
2 Motivating Example
Table
shows an example of how AdaCom applies on-the-fly adaptation to adapt the comment generator to improve its performance. In the table, the fourth row shows the comment after model fine-tuning, the fifth shows the generated comment after model trained by the helpful training samples, and the last row shows the generated comment after model trained by the harmful training samples.
Step 1. Influence Construction. In this example, AdaCom first establishes the influential relationship among all the training samples. For clarity, only show the influence between Cherm and Chelp
2

---

Table 1: The motivating example to illustrate the compromise of the comment generator. The difference is shown on the last three rows (i.e., model prediction after trained by both Charm and Chelp: after retrained by Charm after retrained by Chelp- Sample
Target Test Sample, C
Harmiul Training Sample, Charm
Helpiul Traning Sumple, Cheip
private Date longTolata val, int sqlDataType) (sqlDataType)
public static Java.
public static java.
sql. Tinestamp internalTeTinestang
Date internalloDate int v) { final long MILLIS_PER_DA, return new java. sal. (ate(
case Types. DATE:
long
return new aql Date (val()
Code
return new Java. aql Timestamp
Types. TIME:
return new java.sql. Time(val);
(v LOCAL_TZ getoffaat(v));
t LOCAL, LLOCAL_T2. T2. getDffsat (t));
Types. PIMESTAMP:
return new java aql. Tig..Tinettapp val);
Ground
Converts the internal represenlation Parse the long valued timestamp into the approgri atc SQI, type
Cunverts the internal representation of
of SQL. TIMESTAMP (long) tn
SQL DATE the Java type used for
Truth
the Java type used for UDF
UDF parameters (@ link java lal.sgg..... Date)
Origin
Converts internal
link long I to link TIMESTAMP
representation
Alter
long valued TIMESTAMP a link TIMESTAMP represenlation
Harmful
After
Converts the internal representation of a SQL TIMESTAMP ( long ) to the java type used for UDF
Helpful
Table 2: Influence and estimated training contribution for examples in Table I
Estimated Influence
Estimated Contribution
Sample
Cheip
Charm
Chelp
Ct
0.23
0.67
Charm
40.6
Chelp
0.76
Generally, we can see that Charm and Chelp have an estimated influence score of -0.76. indicating that the deep-learning model has made compromise fit the pair. We will illustrate the calculation of the influence score in Section 3 In this stage, we define and cache all the "contradictory" pairs in the training dataset.
in Table 2
Step 2. Training Contribution Construction. Next, we estimate the potential training contribution of each training sample to the target code ot and select Chelp as one of the semantic contributing training samples. Intuitively, G and Chelp shares similar representation within the deep comment generator, which serves as clue for us select Cheip- We further enhance the Chely to set of code samples using the influence score. More details can be found in Section 3.2 In this stage, we construct the possible helpful samples from the training dataset.
Step 3. On-the-fly Retraining. Finally, helpful code samples like (Chelp] are used to retrain the comment generator g to g'. In Table 1 both Charm and Chelp are well fitted by the model. However, we can observe the changes of generated comments in the last three rows of Table I when retraining happens. The origin comment generator generates more general and inaccurate comment but after re-training with the helpful sample, the model generates a more precise comment for Gt.
Overall, AdaCom is designed for finding possible compromises and adapting the model under specific scenarios. Note that, both g and g' make a compromise. The original comment generator g inclines to favor fitting both Cherm and Cheip- which optimizes g from global perspective. In contrast, g inclines to favor fitting Chelp over Charm. which optimizes g locally for improving the comments for the target code et. As shown in experiment (see Sec the adaptation boosts the performance of -fat--of-the-art comment generators with acceptable run-time overhead.
3 Approach
shows an overview of the design AdaCom, which consists of an offline stage and an online stage. In the offline stage, a comment generator y is assumed to be trained from the training dataset D. Based on D and g, we define the influence estimation and construct the influence graph among the training samples in by their pairwise mutual influence. We cache both the representation of the training samples and the influence scores for online retrieval.
Figure
3

---

Stage
Stage
SOTA Comment Generation
Trahinn
0
Retraining
Generator,
Comment
Code Comment
Contributing Training Samoles
Influence Construction
Training Contributing
Sample Influence
Reletion
Target
Figure 1: Overview of AdaCom. We assume sfate-of-thh-ar approach has been adopted to train the comment generator on the dataset D. Influence Construction the offline stage for characterizing the helpful and harmful samples and training contribution estimation in the online stage is for retrieving the semantic helpful samples.
In the online stage, we retrain the comment generator to g' with the retrieved contributing training samples and then generate the new code comment by g'.
3.1 Influence Construction
We introduce gradient-based estimation for efficiently calculating model-dependent mutual influence of any pairs of training samples.
Influence Estimation We denote comment generator as 9. its training dataset as D = (di:d2,... d2,...,,d, dn.) where d; (c,, com;); C; is the code and com, is its comment. Let ((g(G),com, be the loss of d, and the parameters the trained comment generator g as 0, we calculate the empirical mutual influence mul_ine snl_infe of di and d2 as:
grad(d1) gra)((d2)
mal_inSe(dd,de
Igrad(d1) [grad(d22)
grad(d)) ac(g" (c)).
In another word, the loss on de (and vice versa) on the comment generator g". Specifically, muu mul_infe (d1 (d1,d)) being close to I indicates that dj and de are mutually helpful, that being close to -1 indicates that dj and d2 are mutually harmful and that being 0 indicates that d1 and d2 are independent.
empirically investigate how likely
reduce the loss of dj by reducing
We further build the influence graph by caleulating the mutual influence between every pair of the training samples. Generally, the resultant influence graph serves as a cache to improve the retrieval efficiency of unseen target code.
Influence-based Graph
3.2 Training Contribution Construction
In this section, given a target code c, with unknown comment, a training sample Ctro with labelled comment comtrra and comment generator g. estimate how likely dera (Ceras comtra)) can be used retrain to improve the generated comment for et.
Semantic Similarity Our rationale lies in that, from the perspective of the model g, if the training samples have a more similar internal representation with that of ct, they are more likely to share the similar distribution of c, thus having more semantic informative to be contributing to predict cc.
Representation Assumption In this work, we assume that the deep-learning-base generators have an internal representation of each code sample. We denote code sample as <= (41, (1,,,,,.., L,) and the comment generator as g. Then, g can generate an internal representation when parsing t,, denoted rą h(t,), where h(.) is representation function inside g. This
comment

---

assumption generally holds for most neural network-bass approaches. For example, all one- directional or bi-directional recurrent neural networks and transformer-base model architectures conform to the assumption.
Contribution Estimation Given the code sample c (f1, f2..... (n). we can derive its sequence of internal representation as 2 hh(t),, h(t2)...., h(t,,) (r1, (2,. Fn). Since each code will be rep- resented by a sequence of high-dimensional vectors, we compare the target code c c= (ri, 72: ..., r') with all training samples. In this work, score training contributions by calculating the cosine similarity between the vectors of two code samples' internal representations.
3.3 On-the-fly Model Retraining
Given a target code and two thresholds th1, the, AdaCom first retrieves few samples M {m1, m.... from all the training samples whose training contribution score is higher than the user-defined th If training samples can be retrieved, we do not retrain the model. Otherwise, AdaCom retrieves some new samples N (M1, n2..... from the influential graph related to nodes in M (see Section The estimated influence score between the samples in M and the samples N should be higher than the user-defined the. The threshold th is designed to filter out the semantic helpful training samples and the threshold the is to find out the samples that could help the model unlearn the mutually harmful samples during the learning of some mutually helpful samples. If no training samples are mutually harmful to the samples in and based on the influence graph, AdaCom will remove these samples for further retraining. Finally, 9 is retrained to g' by all the retrieved samples and then generates the new comment as g((c..
For the sake of run- nin-time overhead, we freeze the majority of the neural network and retrain only the last few layers. We also adopt a dropout training strategy and early stop mechanism to remedy the over-fitting and run-time overhead problem.
4
Evaluation
We evaluate AdaCom with the following research questions:
RQI:
Whether AdaCom can boost the performance of diverse comment generators?
RQ2: Whether AdaCom can boost the performance of comment generation on diverse datasets?
RQ3: Whether AdaCom can outperform the retrieval-aus -aggeetdd neuron- based approaches?
RQ4: How generalized AdaCom can boost the performance on the -uitt--distribution samples?
RQS: What is the runtime overhead of AdaCom to boost the performance?
RQ6: How each component of AdaCom can contribute to its boosting performance?
Experiment Setup Our experiments conducted on 2 Ubuntu 20.04 servers equipped with AMD RyzenTM TM 9 5950X 16-Core CPU, 128 GB memory, and Nvidia RTX TM A4000 GPU cards. All the experiment details and replication details can be referred to our website (6).
In this experiment, we adopt four public datasets including CodeSearchNet [19] (CSN), CodeKG [9]. FunCom [2I] and CosBench [41]. We select the datasets for their diversity and representativeness in terms of cross-language features (CodeSearchNett, available project information (CodeKG), and scalability (FunCom and CosBench). Table 3 shows the details.
We choose seven popular language models including Roberta 26]. CodeBERT(smaal and base version) [1I, GraphCodeBERRT (15), and Code'T5 (small, based, and large version). Those models are chosen for their diversity their pretrained corpus (e.g., Roberta on natural language and CodeBERT on code), architectures, and scalability.
Following existing literature 4. and ROUGE-L 23 to evaluate the performance of code comment generation.
21
39 20 2 5)
use smoothing BLEU4 30) 24), METEOR
Experiment Design To answer RQ1 (cross-model evaluation), we fine-tune all the language models on the CodeKG dataset. We select the CodeKG dataset for its diversity in the selected projects,
5

---

Table 3: Experiment Datasets
Dataset
Train
Valid
Test
FunCom
1,954,807
104,273
90,908
CosBench
296,425
42,348
84,694
CodcKG
161,857
20,282
40,512
CSN-Python
251,820
13,914
14,918
CSN-PHI
241,241
12,982
14,014
CSN-Go
167,288
7,325
8,122
CSN-Java
164.923
5,183
10,955
CSN- JavaSSrttr
58,025
3,885
3,291
CSN Re-Ruuby
24,927
1,400
1,261
and explicit project and code relation information for further analysis [9]. Given the randomness introduced by the dropout strategy in the on-the-fly retraining, we repeat the experiment five times to evaluate the consistency of AdaCom.
To answer RQ2 datoset evaluation), evaluate how AdaCom boost CodeT5-small on all four datasets. We fine-tune the code'T5-small model for each dataset and then apply the AdaCom to all four test datasets.
To answer RQ3 (retrieval-metiee method Retro [16] and the method that uses [CLS] with cosine similarity for helpful examples retrieval. Compared to Retro, AdaCom does not need to retrieve and concatenate similar samples in the training stage but retrieves some samples for each test sample in the test stage.
comparison), we compare AdaCom with the retrieval-augmmentdd
To answer RQ4 (generalization), we follow the setting in the and create a new dataset split into source and target. The source and target dataset have different rypes of training and testing samples. We evaluate the performance of AdaCom on the model on the target testing dataset by finding helpful samples in the target training dataset. Note that we do not fine-tune the model on the target domain but only cache the estimated influence and representation of the target training dataset in the offline stage. In this case, we can evaluate how AdaCom adapts the model to new domains by only on-the-fl fly training the helpful samples without the need to fine-tune the whole dataset. We evaluate the generalizability of AdaCom by defining the split-type regarding three different granularity:
We apply AdaCom to T5 natural language corpus (as source training dataset) generate code comments on the CodeKG dataset. We evaluate the boosting performance of AdaCom on the CodeKG as the target testing dataset by on-the-fly retraining few helpful samples in the training dataset of CodeKG.
Cross Languages:
and Roberta model that are pre-trained only on
Cross Program Languages: We apply the AdaCom to enforce the Roberta model trained on CSN-Java to generate code comments the Python, PHP, Go, Javascript, and Ruby programming languages on the CSN dataset. We train the Roberta model on training data of CSN-Java (source training dataset) and then evaluate the boosting performance of AdaCom the testing dataset of other programming languages by finding the helpful subset in their training dataset.
Finally, based the code project information reported in CodeKG, we split the code corpus by projects, denoted as Dpj and Dpiz- Similarly, we apply AdaCom to enforce the Roberta model trained D pm to generate code comments Dpjz.
Cross Projects:
To answer RQ5 (runtime overhead evaluation), we instrument the runtime overhead on the experiment for RQ2. We conducted two settings: utilizing the RTX 3080 GPU on a Windows platform to emulate a programmer's working environment, and another using the A4000 GPU on Ubuntu within the lab setting. We record the average boosting of smoothing BLEU-4 score over all datasets and the consuming time for the whole testing stage. Then we use the boosting smoothing BLEU-4 score per second the metric to measure the cost performance of different methods.
To answer RQ6 (ablation study), we compare the AdaCom performance with three settings. The first setting just uses the comment of the most helpful sample as the prediction, and the second setting uses the standalone model (i.e., adaptation disabled) to predict the comment. Last, we compare the larger model Code (36] with the CodeT5-smal and base model equipped with AdaCom.

---

Table 4: Cross-model evaluation for the boosting performance of AdaCom.
BLEU4
METEOR
ROUGE-L
Model
Scale Parameter
before
after
bst (%)
before
after
bst (%)
before
after
bst (%)
code smmall
60M
34.8920,26
49.0520.38
40.6
43.7420.00 56. 36.7320.08
29.7
50,9820.00 61,2320.09
20.1
small
CodeBERTTsmall
84M
40.83+0.31
48.84 18.8420.72
19.6
48.84=0.32
57.07=0.50
16.9
54,82+0.28
60 60.40=0.47
10.2
RoBERTa
173M
44.73+0.08
48.71
8.9
$2.67+0.17
57.23+0.60
8.7
57.7840.15
60. 00.28=0.59 5.9
4.3
Cod:BERTTbase
173M
44.30=0.34
48.35=0.43
9.1
52.35=0.38 56.74a0.28 26
57.71+0.39
59,81=0.26
3.6
GraphCCddeERR
base
173M
45.5120.48
49.4020.53
8.5
53.6820,58
57.8220.46
7,7
58.8940.60
61.1320.48
3.8
uodeTS
223M
45.53=0.00
49.79=0.03
9.4
54.19:0.00
57.6340.15
58.48+0.00
61.85a0.09
5.8
codeTS-Jarge
large
738M
45.9920.65
49.8720.66
8.4
54.0920.69 58.2720.47
7.7
59.2940.57 61.7320.46
4.1
Table 5: Cross- evaluation for the boosting performance of AdaCom.
BLEU4
METEOR
ROUGE-L
Dataset
before
after
bst (%)
before
after
bst (%)
hefore
after
bst (%)
CodeKG
34,89
49.05
40.6
43.74
56.73
29.7
50.98
61.23
20.1
Cosbench
29.22
35.88
37.07
37.23
37.48
0.67
FunCom
33.32
33.76
1.32
41.71
42.04
0.79
49.23
49.53
0.61
CSN java
19.17
32.09
38.28
CSN-js
15.15
17:06
12.61
22.84
24.40
6.83
30.38
31,18
2.63
CSN-python
19.71
19.92
1.07
30.57
30.68
0,36
37.23
37.48
0.67
CSN-go
18.88
19.15
1.43
33.95
34.10
0.44
41.21
41.40
0.46
CSN-php
24.70
25.76
4.29
36.34
36.96
1.71
44,53
45.40
1.95
CSN--rby
14.78
25.11
31.77
31.94
RQ1: Cross-model Evaluation Table 4] shows that AdaCom can improve the state-of-the-art comment generators with decent significance and consistency. Overall, AdaCom can boost the smoothing BLEU-4 score by on average 14.9%, the METEOR score by 12.2%. and the ROUGE-L score by 7.4%. Compared to the boosted performance in large models, the improvement of AdaCom is more significant the smaller models, which aligns with the expectation that smaller models can be more "struggled" over conflicting subsets of code samples. In addition, AdaCom performs in a consistent manner in the experiment. The deviation generally ranges between 0 and 0.72 in BLEU4, 0.08 and 0.5 in METEOR, and 0.09 and 0.59 in ROUGE-L.
Cross-dataset Evaluation Table shows that AdaCom generally works well on different code datasets based on CodeT5-small, which improves average 8.3% in BLEU4 score, 5.0% in METEOR, and 3.2% in ROUGE-L. We first notice that the performance not consistent over all dataset. The AdaCom shows much better boosting performance dataset. Our investigation on these datasets shows that the number of helpful training samples in some dataset such as CSN iSN-python are limited, which lead to minor improvement performance. Sccondly, as we mainly adjust our apper-prrameters for optimizing BLEU score, the BLEU4 scores increase much higher than other two measurements. We observe that the choice of a stricter threshold of helpful sample selection is a useful and practical mitigation. Given a higher threshold, AdaCom may include less irrelevant training samples to force the on- on-the-fr fly retraining. More experiment results on other sizes of models can be referred in our website 6
RQ2:
the CodeKG, Cosbench, CSN-js
the model
RQ3: Retrieval-methods Comparison Table 6/sows that AdaCom works generally better than the retrieval-augmentie augmented methods. We find that for traditional retrieval-augmeente methods, they can not adapt the model to one certain sample simply using the similar code comment. In the training stage, their model still suffer the bias of the contradictory samples and simply concatenate some similar training samples will not avoid the compromise. the testing stage, AdaCom can adapt the model to re- re-learn the helpful samples and unlearn the harmful samples to gain better performance for one certain test sample.
RQ4: Generalization Evaluation Table shows the generalizability of AdaCom, which indicates that its boosting performance for cross-language cross-PL, and cross-project is significant. For adapting from natural language to programming language, both T5 and Roberta model have much better performance even approaching the performance of traditional fine tuning model. In AdaCom, the large language model can skip the time-consuming fine tuning stage but only search few helpful samples in the target dataset and use AdaCom to fir for one certain test sample. For adapting from one
7

---

Table 6: The performance of -agmmeteed method Retro and using CLS embedding with cosine similarity based on CodeTT5-base model.
Retro
CLS-Cosine
Dataset
223M
223M
223M
CSN-java
20.05
20.14
20.85
CSN-js
CSN python
19.67
19.58
20.46
CSN-go
19:46
19.12
19.61
CSN-php
24.91
26.45
26.90
rSy-ruby
14.91
14.45
15.39
overall
19.19
Table 7: Generalizability of AdaCom cross-language, cross programming-language and cross-
project evaluation.
Generalization Type
Option
BLEU4
METEOR
ROUGE-L
before
after
bst (%)
before
after
bst (%)
before
after
bst (%)
TS-base
10.44
41.97
302.01
24.91
57.27
129.91
18.99
50.72
167.09
Cross language
Roberta-base
37.51
484.27
42.85
318.46
38.77
238
CSN-JS
7.00
15.50
121.43
14.37
23.48
63.40
7.90
18.16
129.87
CSN- Python
7.42
12.91
73.99
16.34
22.61
38.37
8.54
16.71
95.67
Cross PT.
CSN-Go
4.20
11.32
169.52
21.99
121.01
4.91
17.14
249.08
CSN-PHP
7.56
16.71
121.03
16.39
28.61
74.56
8.66
22.38
158.43
CSN-Ruby
9.95
29.05
24.72
13.69
Cross project
CodeKG
11.52
35.19
205.47
20.82
45.57
118.88
28.03
48.64
73.53
programming language to another, AdaCom also shows effectiveness on cross programming language generalizability. For adapting across projects, AdaCom still shows quite competitive performance. The model improves a lot on the out-of-distribution data the code from unseen projects. Generally, AdaCom has shown its effectiveness address the distribution-shift problem diverse scenarios.
RQ5: Runtime Overhead Evaluation Table 8) shows the detailed runtime overhead over boosting of smoothing BLEU-4 score using CodeT5-base model as the baseline model. We can see that on Windows, both AdaCom-small and base outperform the Code T5+ and on Ubuntu, the AdaCom- base shows better performance than CodeTT5+. Based on this experiment, we deem that AdaCom can be well applied for the practical performance boost for the cost-effective smaller models.
RQ6: Ablation study Table sshwws that simply utilizing helpful comments or models will decrease the AdaCom performance. In contrast, AdaCom (the last two column) can combine them and adapt the model by the helpful samples to increase the model performance. AdaCom can also adapt the base model to outperform the CodeTS+ large model.
Related Work
Large Language Model Generally, the neural network model architecture follows encoder- decoder structure. Regarding the code text as sequential data, CNN (2]. LSTM 20,
38), or GRU
21 used to
backbone model. With the advance of model, attention mechanism [d and
transformer [1 8 are adopted to further improve the performance. CodeBERT [IT] is a bimodal pre-trained model for multiple programming languages (PLs) and natural language (NLs), it supports
Table 8: Run-time Overhead over BLEU-4 score boosting compared to CodeT5 model.
Model Name
CodeT5 223M
CodeT5+
AdaCom-small
Adacom-base
Parameter
Time (secood)
1.21
4.06
2.34
4.81
Windows
Time per sample (sccond)
0
2.85
1.13
3.60
Average boost
0.63
0.45
Bousi BLEU scure per secoad
22.11%
39,82%
35.56%
Time (sccood)
0.38
1.91
1,46
3.16
Ubuntu
Time sample (second)
0
1.53
1.08
2.78
Average boost BLEU score
0
0.63
0.45
1.28
Boost score per second
41.18%
46.04%

---

Table 9: Ablation Study on AdaCom
Retrieved Comment
Model Only
Larger Model
AdaCom-small
AdaCom-base
223M
223M
770M
60M
223M
CSN-java
13.18
19.17
20.83
20.06
20.85
CSN-js
13.96
15.15
17.93
17.06
18.81
CSN-python
11.70
19.71
19.92
20.46
CSN-go
11.17
18.88
19.64
19.15
19.61
CSN-php
17:47
24.70
26.39
25.76
26.90
CSN-ruby
8.91
14.78
15.63
15.03
15,39
overall
12.73
18.73
20.15
19.50
20.34
downstream NL-PL. tasks like code search and code comment generation. Compared with CodeBERT, the input of GraphCodeBERII includes both code tokens and a code data flow graph among the vari- ables. Compared to CodeBERT and GraphCodeBERT, CodeT5 [37] aims at a generic representation of both PLs and NLs. Following the original T5 model [3], the architecture of CodeT5 is faithful to the original Transformer.
Retrieval Based Comment Generator As code duplication common large-scale repositories. setrieval-basee techniques like vector space model [77 and code clone detection 40 are used to find the most similar code comment pairs from the database in the early stage. By manually designing a set of rules, they can filter out the most similar pair and copy its comment as the output. Recent works combine the retrieval system and neural network model to generate comments. Researchers use neural network as semantic feature extractor to retrieve the comment of the most similar training sample. CCGIR 42 adopts CodeBERT and BERT-whitening operation [32] for retrieval. Re²Com [39] uses BM25 to find the most semantically similar sample and utilizes four encoders to process target code, similar code, similar code's s comment, and code abstract syntax tree. Through the attention mechanism, the four encoders' outputs are fused and fed into the decoder to get the final comment. Similarly, in [4], a fusion layer is designed to combine the CodeBERT output of both target code and retrieved code.
Different from the previous works that retrieval during the training stage, we search for the helpful samples in the testing stage to reduce the bias of the model.
Test-time Adaptation Adaptation means the generalization from one distribution to another distri- bution. Transfer learning 10 4K]
a technique to transfer the parameters of pre-trained model to new model on the target dataset. Through transfer learning, we can share the learned parameters (also the knowledge learned by the model) with the new model speed up the learning procedure. The model is first pre-traine pre-trained on the source dataset and then trained on the training data of the target dataset. Then can be used for predicting the test data of the target. Domain adaptation techniques [12 aim to improve the performance of the model in another domain (target domain) to approach the effect of the original domain (source domain). Further, the test-time training technique 33 updates the model during testing with unlabeled data in a self-supervised way. Compared to test-time training, Wang et al. propose [35] an adaptation technique independent of the training data and training loss. The model is trained by test entropy minimization during the test stage and adapts itself to feedback from its predictions.
To the best of our knowledge, we are the first to introduce test-time adaptation for comment generation. Different from the aforementioned works, AdaCom selects the helpful samples from the existing training samples and then adapts the model for each target code sample.
6 Limitation
Scalability AdaCom requires more time to adapt the model, especially for billion-size model. To mitigate this, freezing techniques are necessary, specifically freezing the parameters in the encoder and only training part of the parameters of the decoder.
Construct Validity AdaCom assumes that it can find helpful examples in the training dataset based on the two metrics to enhance the performance of the model. However, there exist test cases with no helpful examples or possible helpful examples that are ignored. It also remains question whether proposed methods would introduce some harmful samples.

---

Table 10: An example showing the potentially over-fitting problem of AdaCom
Sample
Target
Sampl,,
Harmtul Training Sample, Charm
Helptul Training Sample, Chelp
ReceiptVient PecespeViexModel purchase
(...)
RecesptViesModel purchase
Db. User
Db. gotInstance
String receiveRequest( Object paranaters) throws DoUhavailabldekccppi...
(...)
Ob.duotct iten Db get
#indUsarByUsarrlnn
() (idd
(userNane); (user null)
if (iten " null) {
var id generateld
}
Code
var req new PaynentRequest (id
Receipt receipt ...2 if (transaction mull)
Db. Account sccount findAccount (user)
(float)
paranatera [o]);
return purchase (user, account, itenNana )
return updateDb (reg);
return raceipt;
public method which will receive request
domain purchase with user, account and item- Name, with validation whether product out of stock and whether user has insufficient funds in account
Ground
purchase with userName item- Name, with validation for userName
from @link com. iluwatar. commander
Truth
Commander
Origin
(BLEU 16.78) register purchase with
AdaCom
(BLEU FEU 66.43) domain purchase with
and item with validation for whether enabled
Epoch8
AdaCom
84.84) domain purchase user name name validation account
Epoch11
AdaCom
(BLEU 47.79) domain purchase with user account and item name. with validation for whether user is enabled or not
Epoch15
Internal Validity We further analyze the cases where the AdaCom shows limited performance. We find that AdaCom sometimes overfits the retrained training samples. In Table 10] despite that AdaCom can successfully select useful reference Chelp. it cannot stop at correct point. In this example, AdaCom retrains the deep comment generator for 15 epochs. However, the best stopping condition happens at epoch 11, which means that AdaCom should have stopped earlier to gain higher accuracy and lower run time. We think AdaCom can leave these candidates the users to choose and use the helpful samples to interpret the model behavior.
7 Conclusion
In summary, we propose novel system AdaCom which boosts the performance of comment generators on-the-fly. Extensive experiments has shown that AdaCom can effectively improve the performance the code comment generation on diverse datasets, programming languages, and different deep language models. Further, AdaCom also shows its decent generalizability on cross- language, cross gegraammin-languuaan and cross-projed settings. Overall, our solution shows promising results in enhancing model performance and potentially efficiency.
Acknowledgments and Disclosure of Funding
This research is supported in part by the Minister of Education, Singapore (12HP20120-0019 MOET32020-0004),, NUS-NCS Joint Laboratory for Cyber Security, Singapore, the National Re- search Foundation, Singapore, and Cyber Security Agency of Singapore under its National Cyber- security Research and Development Programme (Award No. NRF-NCR TAU TAU_2021-000 A*TAR CISCO Systems (USA) Pte. Ltd and National University of Singapore under its Cisco- NUS Accelerated Digital Economy Corporate Laboratory (Award 121001 National Research Foundation, Singapore, and the Cyber Security Agency under its National Cybersecurity R&D and National Natural Science Foundation of China (62172099, 237990203011,, Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Cyber Security Agency of Singapore.
and
Programme NNCRP25-PO4-TAICEN
10

---

References
[1] Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. A transformer-ben approach for source code summarization. ACL (2020).
[2] Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A convolutional attention network for extreme summarization of source code. In International conference on machine learning. PMLR, 2091-2100.
[3] Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019. code2seq: Generating sequences from structured representations of code. ICLR (2019).
[4] Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65-72.
[5] Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. 2021. Self-supervised contrastive learning for code retrieval and summarization via semantic-presse preserving transformations. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 511-521.
[6] Yufan Cai. 2023. AdaCom website.
home
[7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 03374 (2021).
[8] Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. 2020. PyMT5: multi-mode translation of natural language and Python code with transformers. arXiv preprint arXiv:2010.03/50 (2020).
[9] CodeKG. 2022. CodeKG.
Aattps://sitee gww.se.oooog......... com wwtpe/ee//.eeeeeeeeeeeeeeeeeeeee homee
[10] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffmann, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recogni- tion. In Proceedings the 31st International Conference on Machine Learning (Proceedings of Machine Leurning Research, Vol. 32), Eric P. Xing and Tony Jebara (Eds.). PMLR, Bejing, ngs. mlr. Ttress/v32/donee
the
China, 647-655.
14. html
[11] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. EMNLP (2020).
[12] Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised Domain Adaptation by Backpropa- gation. Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37 (Lille, France) (ICML'15) 15). JMLR.org, 1180-1189.
[13] Roberto Gozalo-Brizuce and Eduardo C Merriao-Merehan. 2023. ChatGPT is not all you need. State of the Art Review of large Generative Al models. arXiv preprint arXiv:2301.0465 (2023).
[14] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. arXiv preprint arXiv:2203.0388 (2022).
[15] Daya Guo. Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Grapheodebert: Pre-training code representations with data flow. ICLR (2020).
[16] Vivek Gupta, Akshat Shrivastava, Adithya Sagar, Armen Aghajanyan, and Denis Savenkov. 2022. RetroNIUU: Retrieval Augmented Task-Oriented Semantic Parsing. In Proceedings of the 4th Workshop on for Conversational AI. Association for Computational Linguisties, Dublin, Ireland, 184-196. https::/doi. org/ 181..011111131121220022 http://222.nnncooone.eo m1p4convai-1...

---

[17] Sonia Haiduc, Jairo Aponte, Laura Moreno, and Andrian Marcus. 2010. On the use of automated text summarization techniques for summarizing souree code. In 2010 17th Working Conference on Reverse Engineering. IEEE, 35-44.
[18] Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment generation. In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC). IEEE, 200-200100
[19] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09400 (2019).
[20] Srinivasan Iyer, loannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2073-2083.
[21] Alexander LeClair, Siyuan Jiang, and Collin McMillan. 2019. A neural model for generating natural language summaries of program subroutines. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 795-806.
[22] Jia Li, Yongmin Li, Ge Li, Xing Hu, Xin Xia, and Zhi Jin. 2021. Editsum: A retrieve-and-edit framework for source code summarization. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 155-166.
[23] Chin- Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74-81.
[24] Chin Yew Lin and Franz Josef Och. 2004. ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics. COLING, Geneva, Switzerland, 501- 507, atttss:/aaclantholog. prg/C04-107
[25] Shangqing Liu, Yu Chen, Xiaofei Xie, Jingkai Siow, and Yang Liu, 2020. Retrieval-augmented generation for code summarization via hybrid gnn. arXiv preprint arXiv:2006.05400 (2020).
[26] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv: 1692 [cs.CL]
[27] Paul W McBurney and Collin McMillan. 2014. Automatic documentation generation via source code summarization of method context. In Proceedings of the 22nd International Conference on Program Comprehension. 279-290.
[28] OpenAL. 2020. OpenAl GPT-3 Model. https:://openai.com/ gpt- 3/
[29] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.0211 (2022).
[30] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 3-1-318.
[31] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res. 21, 140 (2020), 1-67.
[32] Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316 (2021).
12

---

[33]
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. 2020. Test-Time Training with Self Supervision for Generalization under Distribution Shifts. In Pro- ceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119.. Hal Daumé III and Aarti Singh (Eds.). PMLR, 9229-9248. https://proceediigg mlr. press/ wwtp///n1........o.mm html
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems 30 (2017).
[35] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. 2021. Tent: Fully Test-Time Adaptation by Entropy Minimization. In International Conference on Learning Representations. Nittps://openrevaai net/forum? widiiid3bukrkree
[36] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Jumnan Li, and Steven C. H. Hoi. 2023. CodeTT+:: Open Code Large Language Models for Code Understanding and Generation, arXiv:2305.0792 [cs.CL.]
[37] Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint 20Xi002109.00859 (2021).
[38] Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019. Code generation as a dual task of code summarization. NearlPS (2019).
[39] Bolin Wei, Yongmin Li, Ge Li, Xin Xia, and Zhi Jin. 2020. Retrieve and refine: exemplar- based neural comment generation. In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 349-360.
[40] Edmund Wong, Jinqiu Yang, and Lin Tan. 2013. Autocomment: Mining question and answer sites for automatic comment generation. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 562-567.
[41] Shuhan Yan, Hang Yu, Yuting Chen, Beijum Shen, and Lingxiao Jiang. 2020. Are the code snippets what searching for? A benchmark and empirical study code search with natural-language queries. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 344-354.
[42] Guang Yang, Ke Liu, Xiang Chen, Yanlin Zhou, Chi Yu, and Hao Lin. 2022. CCGIR: Informa- tion retrieval-basee code comment generation method for smart contracts. Knowledge-Bard Bassed Systems 237 (2022), 107858.
[43] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How Transferable Are Features in Deep Neural Networks?. In Proceedings of the 27th International Conference Neural Information Prucessing Systems - Volume 2 (Montreal, Canada) (NIPS' 14). MIT Press, Cambridge, MA, USA, 3320-3328.
[44] Chi Yu, Guang Yang. Xiang Chen, Ke Liu, and Yanlin Zhou. 2022. BashExplainer: Retrieval- Augmented Bash Code Comment Generation based on Fine-tuned CodeBERT. arXiv preprint 2r06v:2206.1332 (2022).
13


Reviews:
[{"review_id": "7A3Y5kBZAw", "reviewer": "Reviewer_4N9q", "summary": "This work proposes AdaCom for boosting code comment generation by on-the-fly model adaptation. The authors started with the motivation that existing generators are heavily affected by conflicts within the dataset (CodeSearchNet). The proposed AdaCom method is divided into two stages: clustering the training samples in the offline stage, identifying similar training samples and their corresponding clusters in the online stage, and retraining the model using these samples. The effectiveness of AdaCom is verified by cross-model and cross-language experiments.", "questions": "It is recommended that the work improve the expression of the figures.\n", "limitations": "The authors discuss the limitations of their work and give ideas for improvement.\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The experimental results given are good and can support their claims. ", "weaknesses": "1. According to Table 3, the work is still valid for datasets that are inherently conflicting.\n2. According to Table 2, the boost of AdaCom becomes smaller when the model size increases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our work. In response to your concern about the limited improvement on larger models, we would like to highlight the following points:\n\n### Relative Improvement: \nLarger models already exhibit relatively better performance compared to smaller models. Consequently, even a modest boost percentage represents a substantial advancement, given the already higher baseline performance.\n\n### Adacom Design: \nAdacom is primarily tailored for relatively small models. Smaller models often encounter challenges related to data limitations, whereas larger models possess a greater number of parameters and the capacity to effectively fit intricate training datasets.\n\nThese factors collectively contribute to the impact of Adacom on larger models compared to its pronounced effect on smaller ones.\n\nCertainly, we'd be glad to improve our work. Could you please provide more specific details or context about the figures you're referring to? Are there particular aspects or elements within these figures that you believe might need clarification or improvement? Thank you for your kindness and support for this work."}, {"review_id": "rE2MEvQDGc", "reviewer": "Reviewer_to4c", "summary": "AdaCom is a system meant to adapt a trained model, on-the-fly, when making a prediction (specifically targeting code summarization models). It does this by identifying training examples that seem similar to the target example, lightly re-training the model to adapt better to those similar examples, and then making a prediction on the target. This re-training takes on the order of a few seconds.\n\nAll this is done by clustering training examples, ahead of time, by their mutual influence. Intuitively, two examples are mutually influential if their loss changes in the same way when the model parameters change.\n\nWhen a target example comes, it is embedded (using the existing model), and then examples with proximal embeddings are retrieved from the training dataset, along with their influence clusters. The union of all such influence clusters are then used to (lightly) retrain the existing model, possibly only a few layers, and the resulting model is used to make a prediction.  \n\nThe system is evaluated on a number of code summarization datasets and with several pre-trained code models. It shows that a model's performance can improve using this on-the-fly adaptation for each test example; it shows that a model trained on one dataset can adapt well to another; it shows that generalization across projects and languages is also improved; and that all features are usefully contributing to the performance. ", "questions": "1. Where does the training contribution score come from? Is it the mutual influence? \n2. The approach seems very much general. Why focus on code summarization?", "limitations": "No concerns", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Improving on out-of-domain samples seems very important for code tasks, and this work seems to make a great improvement in existing benchmarks.\n\n2. The presentation is fairly systematic and well-structured (with some exceptions, see below) and makes for an informative paper that not only describes its design and results, but also motivates its design well (with some exceptions, again)!\n\n3. The formulation of the problem is somewhat original and different from how similar adaptation formulations are done, but I worry it might not be that different from retrieval-augmented problem formulations.", "weaknesses": "1. I don't quite understand how broad and significant the difference is between something like AdaCom and other retrieval-augmented approaches such as Retro, Memorizing Transformers, or or other Retrieval Augmented LMs (e.g., https://arxiv.org/abs/2302.00083). They all do retrieval in slightly different ways, but they all intend to condition a model on relevant documents before making a prediction. Here AdaCom retrains instead of somehow using the retrieved embeddings, but it's not clear how big this difference is. I would recommend that the authors compare against other retrieval-augmented baselines to tease out how their adaptation technique differs and where it is stronger.\n\n2. Although the paper is structured well, in parts it is lacking clarity and seems to be rife with typos or changed notations. For instance\n     1. I don't know what the difference is between $th_etc$ and $th_tcs$\n     2. How is $C_inf$ both a cluster set and a function around lines 135--140\n     3. Also models like GPT3 have hundreds of billions, not millions of parameters (line 35)\n     4. Shouldn't line 76 be $g'$ instead of $g$? Otherwise the formulation doesn't make sense.\n     5. Shouldn't it be $mut_inf$ instead of $mul_inf$?\n\n3. Some design aspects of the work seem to be elaborating what might not be that unusual. For instance I didn't understand the need for the extensive description of how to obtain a contextual embedding of a code snippet in section 3.2, as a trajectory of the contextual embeddings of individual tokens. Isn't that what all contextual embeddings since the downfall of word2vec really do? Why is special way to compare token-by-token embeddings necessary, rather than taking a cosine similarity over the full sample embedding (e.g., the [CLS] token in BERT-based formulations)? \n\n\nAlthough #2 and #3 are easy to fix through improved writing, I'm concerned about #1 and would like to understand how AdaCom compares to strong retrieval-augmented baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the appreciation and support for our work. We address your concerns as follows:\n\n# Q1: Where does the training contribution score come from? Is it the mutual influence?\n\nThe training contribution score is defined between a training sample *s_i* and a testing sample *s_j*, indicating how likely retrain the model on *s_i* can boost its performance on *s_j*. \nNote that, the model does not have the idea of the label of the testing sample \n\nIn contrast, the mutual influence is defined on two training sample *s_i* and *s’_j*, indicating indicating whether the training *s_i* can help reduce the training loss of *s’_j*. \nNote that, the model know the label of both *s_i* and *s’_i*.\n\n# Q2: The approach seems very much general. Why focus on code summarization?\n\nWe agree that the approach can be further generalised to other NLP tasks or CV tasks. \nNevertheless, we think that AdaCom is more needed in LLM-based settings such as code summarization. Here’s why:\n1. Resource Constraints: Larger language models typically face more stringent resource limitations, making efficient and focused training methods like AdaCom particularly valuable. On the other hand, smaller models encounter greater challenges with respect to information capacity.\n2. Structural and Semantic Information: Code data inherently possesses rich structural information, including class hierarchies, function calls, and specific features like variable names. These attributes facilitate a more straightforward identification of \"similar\" code segments, rendering AdaCom especially advantageous.\n\nIn the future, we aim to generalise AdaCom to other tasks such as code generation, which requires us to consider code context information (e.g., class hierarchy, caller and callee relation) in a more sophisticated way. \n\n\n# Q3: The difference between AdaCom and Retrieval-based Approach (RBA)\n\nA vivid comparison can be referred to Figure 1. A comparison with Retro has been made in Table 2.\nDespite both AdaCom and retrieval-based approach (RBA) need to search for samples to enhance the model summarization performance, we highlight the difference as follows:\n\n1. Different Search Target:\nAdaCom searches for under-represented training samples; while RBA searches for “similar” samples in possible external dataset.\n\n2. Different Rationale\nAdaCom tries to exploit existing training samples largely under-utilised during the training; while RBA tries to explore similar samples and concatenate them into one input and make the prediction.\n\n3. Different Adoption\nAdaCom can adapt the model with a few targeted training samples on-the-fly; while RBA includes retrieved code as additional input for the model to make a prediction.\n\n# Q4-5: Typos, justification, and confusion\n\nWe genuinely appreciate the reviewer's attention to the typos, and we are committed to addressing and rectifying them in our revision.\n\nComputing cosine similarity over the [CLS] token embeddings was our initial approach. The outcomes of this method are presented in Table 2. Notably, in the JavaScript and PHP datasets, where similar samples are prevalent, this technique proved effective in enhancing the model's performance. However, in other datasets, a simple utilization of the [CLS] embedding resulted in a decrease in performance. In that case, we utilize a more complex way and use the our influence estimation to enhance the retrieval."}, {"review_id": "Pmtp0tz3Ia", "reviewer": "Reviewer_EdLg", "summary": "Language models trained on code tend to suffer in performance when applied to unseen code bases. This work investigates adapting smaller LMs to a new project context based on a brief fine-tuning phase that is invoked immediately before the model predicts a comment for a given method. Fine-tuning examples are selected according to their embedding similarity to the given sample. Experimental results suggest that this improves performance by a few percent on several benchmarks.", "questions": "- Is there any connection between the empirical influence estimation and the practical one used in the implementation, in terms of either theoretical guarantees or practical proof that the latter realizes the former?\n- What is the algorithm for computing the minimum edit distance between two series of embeddings?\n- What are the hyper-parameters used for fine-tuning?\n- Given the minimal performance improvements on benchmarks other than CodeKG and the fine-tuning cost involved (not just in terms of time but also memory and GPU resources), what would motivate a developer adopt this approach?", "limitations": "The limitation section should be expanded to discuss the lack of baselines from other published work. While it may be true that no other work has considered this exact task, many methods have been proposed for adapting trained models to new data distributions.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "This work tackles the important challenge of adapting, especially smaller, language models to programs from unseen projects. The approach is structured by considering a reasonable series of formulations, starting with the high-level goal and working towards a practical implementation. Along the way, it introduces several new metrics for measuring similarity between samples. The results indicate that the proposed retraining approach can improve performance, which highlights that the models may indeed fail to precisely store relevant samples from their training data.", "weaknesses": "The work suffers substantially in terms of clarity and soundness and the performance gains reported mostly rather weak.\n\n**Clarity & Soundness:** I group these two together because there are several places in which the paper cannot be checked for soundness due to missing or very limited reporting. This includes:\n\n- The minimum edit distance between the sequences of embeddings of two samples, introduced in L170. The method for calculating this metric is never described.\n- The connection between the empirical and \"practical\" influence estimation in Section 3.1 is unclear. The former does not appear to play any role in the creation of the latter; there are no theoretical guarantees connecting the two, nor practical results that show that the \"practical\" algorithm realizes the empirical desiderata.\n- Many hyperparameters are unstated. This especially includes several important thresholds, mentioned in 3.2, as well as the number of layers that are unfrozen and the number of samples that they are retrained with.\n\nIn addition, there are many places in which the paper makes unclear or incorrect claims, including:\n\n- L4: \"the subject code in a project can be specific\" is not a meaningful statement.\n- L6: this claim explicitly motivates the work by the cost of retraining large language models, yet the paper focuses exclusively on far smaller models (and does not offer an approach that would scale to LLMs).\n- L37: \"normal-sized models\" is not a meaningful descriptor, especially given that models of the size considered in this paper are far smaller than those typically used in production environments.\n- Equation 1 is missing the element being arg-min'ed over\nThe work also includes numerous ungrammatical sentences and should be revised substantially to improve the writing.\n\n**Results:** the performance gains are mostly small and skew very heavily towards the CodeKG benchmark. Performance gains on other benchmarks is often small or non-existent (Tab. 3, Tab. 5). This suggests that the approach may be overfitting to the CodeKG dataset.\n\nThe various cross- settings reported in Tab. 4 could use additional baselines. The \"before\" performance reported here apparently consists of directly querying models that are not trained on such datasets. Another important data point is the performance of models that were trained on these datasets from scratch, or finetuned on their training portion before any inference queries are made.\n\nMore generally, the work does not include comparisons with any other approaches for adapting to new datasets, such as more conventional neural retrieval approaches. This casts substantial doubt on the multiple newly proposed metrics for identifying similar samples. This is particularly problematic because it proposes two such metrics that seem to serve the same purpose, one based on gradients and one on pair-wise comparisons of the embeddings. It is unclear why there are two and no ablations are offered that substantiate the need for both.\n\nOverall, the work provides insufficient motivation for incurring the added runtime overhead of selecting and fine-tuning on examples detected with its method.\n\nUpdate: following extensive discussion with the authors, both the clarity of the work has improved and the performance (esp. in terms of runtime overhead) was established as being worth the tradeoff compared to larger models used in a simpler inference setup, at least on the benchmark considered. While the work still suffers from several limitations including the lack of motivation for the newly introduced metrics and somewhat limited breadth of evaluation, I no longer strongly favor rejecting it and am raising my score to a borderline accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s comments for improving our work. We understand the importance of providing accurate and helpful statement and we will do our best to ensure that our responses are clear. It seems that your concerns on the soundness derive from the seemingly confused interpretation. We will make every effort to address these points and provide clarification in our revised manuscript. \n\nIf your confusion is addressed, we request your kind support by changing your stance. Your stance means a lot to us. \n\n# Q1: Why are the performance gains mostly small or non-exist and skew very heavily towards the CodeKG benchmark?\n\nPlease check some related work [1][2][3][4].\nAs CodeSearchNet is a widely adopted dataset for code summarization, compared with other models like CodeBert (with an average improvement of 2.84%), PLBART (2.73%), CodeGen (3.07%), and CodeT5+ (3.36%) as reported in their original papers, we respectfully disagree that our boost is small or non-exist.\n\nOnly codeT5-small models skew towards the CodeKG while other models show average 8.86% boost percent (Table 2 in origin paper). We believe this is a average boost percent.\n\nThe reasoning behind CodeT5-small's superior performance on CodeKG aligns precisely with the discussion presented in our paper. Small models encounter challenges with data compromise, especially in CodeKG, where code snippets stem from various code projects. During training, small models are compelled to accommodate distinct code formats and token distributions, often requiring substantial effort. Conversely, during the testing phase, individual samples may pertain to specific classes, leading to a significant deviation between the original model and the potentially best model. This is where Adacom's role becomes evident. Adacom is designed to identify and select the optimal models for each individual test sample, addressing the inherent shift between the model's training diversity and the distinct requirements of specific test cases.\n\n# Q2: The linkage between the empirical influence estimation and the practical one used in the implementation?\n\nWe organise the logic lines as follow:\n\nStep 1: Given a test sample (i.e., a piece of code), *c*, and a model *m*, we would like to know whether *m* has compromised performance on *c*.\n\nStep 2: To estimate the compromised performance, we first look for training samples similar to *c*, where we use minimum editing embedding distance. We denote the set of similar training samples as *S*.\n\nStep 3: If *S* is not empty, we check whether *m* has compromised performance on *S* to estimate the test sample *c*, by our proposed empirical influence estimation. Intuitively, if *m* can decrease the loss of *S* at the cost of the loss of many other samples, we deem *m* has compromised performance on *S*. The cluster is used for speeding up the retrieval of those training samples.\n\nIn addition, *S* is then selected as retraining samples to update *m*.\n\n# Q3: What is the algorithm for computing the minimum edit distance between two series of embeddings?\n\nThe algorithm for minimum editing distance is exactly similar to the classical LCS (longest common subsequence) algorithm.\n\nNote that, if the sequences are token sequences such as <A, T, C, G, …,>, the editing distance is the Levenshtein distance between two sequences. In the classical *hard* settings, we define the cost of replacing, adding, and inserting a token as 1.\n\nIn our *soft* settings, we define the cost of replacing a token *t_1* with another token *t_2* by (1 - cos(*t_1*, *t_2*)). Then, we just follow the dynamic programming algorithm used in LCS to calculate their distance.\n\n# Q4: What are the hyper-parameters used for fine-tuning?\nWe follow the hyper-parameters of the baseline papers. The models will be trained for 40 epoches with early stopping mechanism. All the models are convergent. We select the model with the best bleu score in the valid dataset in further testing. \nAll models and results can be found on our anonymous website.\n\n# Q5: Given the minimal performance improvements on benchmarks other than CodeKG and the fine-tuning cost involved (not just in terms of time but also memory and GPU resources), what would motivate a developer to adopt this approach?\n\nWe respectfully disagree that our approach provides minimal performance. See our argument in Q1.\n\nGiven existing biilion-level LLM service, the request and response takes additional financial and network latency cost. Therefore, a small model allows us to do tasks on most devices in an offline way, addressing the need of most software developers. \n\n# Q6: Difference between AdaCom and retrieval-based approach and more baselines.\nA vivid comparison can be referred to Figure 1 and Table 1 and 2 in the rebuttal pdf. We have compared Adacom with the latest CodeT5+, CodeGen and retrieval based method Retro (reviewer to4c).\n\nDespite both AdaCom and retrieval-based approach (RBA) need to search for samples to enhance the model summarization performance, we highlight the difference as follows:\n\n1. Different Search Target:\nAdaCom searches for under-represented training samples; while RBA searches for samples in external dataset, indicating that AdaCom has less assumption than RBA.\n\n2. Different Rationale\nAdaCom tries to exploit existing training samples largely under-utilized during the training; while RBA tries to explore new similar samples and merge them into the prediction.\n\n3. Different Adoption\nAdaCom can adapt the model with few targeted training samples on-the-fly; while RBA includes retrieved code as additional input for the model to make a prediction.\n\n[1] CodeT5+: Open Code Large Language Models for Code Understanding and Generation\n[2] CodeBERT: A Pre-Trained Model for Programming and Natural Languages\n[3] Unified Pre-training for Program Understanding and Generation\n[4] CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis"}, {"review_id": "wCVG81499W", "reviewer": "Reviewer_Ugvn", "summary": "This paper proposes an AdaCom approach to improve the code comment generation performance by on-the-fly model adaptation. For one code sample at test time, AdaCom first detects whether the model might have a compromised summarization performance and if yes,  re-adapt the model on-the-fly by training the most contributing training samples to improve its performance for this sample. By experimenting with 7 deep comment generators and 4 public datasets, results show that AdaCom can effectively improve the performance of the code comment generation on diverse datasets, programming languages (PLs), and code language models (LMs). However, AdaCom achieves the performance improvements at a cost of adding an on-the-fly retraining stage in test time, incurring considerable runtime overhead. Its method seems to be over-complicated compared to the popular retrieval-augmented generation techniques, and there is no comparison with these techniques. This work also lacks a comparison with many state-of-the-art (SoTA) code LLMs such as CodeGen and Incoder.", "questions": "See the above weakness section for more details.", "limitations": "The authors discuss the limitations lightly.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed AdaCom is technically sound and back with convincing performance gains after adopting AdaCom on multiple datasets covering various PLs and code LMs.\n* The paper is well structured and easy to read.", "weaknesses": "* While the proposed method boosts the performance of code comment generation, it achieves this at a huge cost of runtime overhead during inference. Though the paper uses some techniques such as model parameter freezing to optimize the runtime cost due to the on-the-fly retraining, it still takes 5.54 seconds for large-size models (below 1B) to generate a comment on average. Such a latency is not that acceptable for real-world applications such as in an integrated development environment, where developers typically expect less than 1 second latency. \n* Additionally, the proposed method seems to be over-complicated compared to other widely adopted retrieval-augmented generation techniques [1] that similarly improve performance by referring to some related training examples. There is no comparison with these techniques in terms of both performance and runtime overhead. Though retrieval-augmented generation techniques might require an extra stage of retrieval during training, they seem to be much more efficient during inference which often has a high requirement for low latency. Particularly, in the era of LLMs such as ChatGPT, retrieval-augmented generation becomes a more popular approach to provide retrieved examples for LLMs via in-context learning without any retraining. It might be difficult to justify the benefits of AdaCom in both performance and runtime overhead.\n* This work does not compare with state-of-the-art code LLMs such as CodeGen, Incoder and the recently released CodeT5+ and StarCoder. While the authors claim that AdaCom is a technique which can be well applied for the deployed smaller models, 1B to 16B parameter LLMs have been widely deployed and should be also considered. I would like to see how AdaCom can help improve performance for these larger LLMs. Suppose that if AdaCom can only significantly boost the performance for small-sized LMs, and larger LMs actually achieve better performance than AdaCom+small LMs, it is difficult to justify the use of AdaCom. Note that for large LMs from 1B to 6B, the inference time can be reduced to less than 1 second by adopting some acceleration tricks (based on my experience with LLM inference).\n\n[1] “Retrieval Augmented Code Generation and Summarization”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer’s appreciation of our work. We observe that your fundamental concern lies in the runtime overhead of our approach. In this rebuttal, we (1) clarify that improving smaller models is far more valuable and (2) show the inference trick for AdaCom to work on larger models.\n\nWe sincerely hope that your concern is addressed. If it does, we request your kind support by changing your stance. We have updated this work with a few revisions. Your stance means a lot to us.\n\n\n# Q1: AdaCom works well on smaller model but the latency of 5s in larger model is too much\n\nIn the era of LLM, a *cost-effective* solution to exploit the full potential of LLM is important for everybody to use LLM in an offline manner. We design AdaCom to this end.\n\nThe smaller the model, the more devices it can be explored, but the less capacity it can encode the whole training dataset. AdaCom provides a solution for a model to re-favour some training samples on-the-fly for a better performance. On the other hand, the larger model (10b) has larger capacity, suffering less from the limit. In this regard, we deem that we have achieved our research goal despite that we show our limitation on larger models from a research perspective. \n\nIn this rebuttal, we further show that, with certain tricks, the inference time of AdaCom can be reduced to 1s on larger models (748M). Please see more in Q3.\n\n\n# Q2:Over-complicated  compared to Retrieval-Based Approach (RBA)\n\nA vivid comparison can be referred to Figure 1. A comparison with Retro(recommend by to4c) and REDCODER (as you recommend) has been made.\n\nREDCODER[1] integrates an extensive external database, encompassing millions of entries acquired from sources like Stack Overflow and GitHub. This diverges from the notion you mentioned of \"referring to some related training examples.\"\nIt is hard to compare the complexity of Adacom and REDCODER as we don’t know how much time it takes in crawling the data, cleaning the data and extracting the useful code summary. In our Adacom, we only utilize the same training dataset with the baselines. \n\nWe opted to employ the retrieval-augmented approach RetroNLU (as shown in Table 2) for direct comparison with Adacom. This method aligns with the concept you mentioned earlier, involving \"referring to some related training examples.\" However, it's notable that the performance of this method is relatively subpar compared to the original model. This observation might shed light on the absence of similar adaptations from Retro to Code-related tasks within the CodeSearchNet dataset.\n\n Relying solely on the straightforward cosine similarity of the [CLS] embedding yields nonsignificant results, as demonstrated in Table 2. Given this context, we find it imperative to adopt a more sophisticated approach, inspired by the inference function, to effectively eliminate noise and amplify model enhancement. The discrepancy in results highlights the complexity of achieving optimal performance through \"referring to related training examples\" and emphasizes the significance of approaches like Adacom, which is designed to address specific challenges and achieve notable improvements in performance.\n\n# Q3: This work does not compare with state-of-the-art code LLMs  \nWe have compared CodeGen, CodeT5+ with Adacom in the rebuttal pdf Table 1. \n\n# Q4: 1B to 16B parameter LLMs have been widely deployed and should be also considered  \n1. For our individual users, it is hard to afford 1B on our own computers. While seeking assistance from established entities like OpenAI is an option, the preference for retaining sensitive data like code on one's own system is understandable.\n2. Furthermore, employing smaller models holds commercial advantages, as they are often more cost-effective and resource-efficient. This can be particularly relevant in scenarios where resource constraints and data privacy considerations come into play.\n3. We believe for a billion-size model, prompts are a better way to instruct the model on one certain test sample.However, for smaller models, prompts might not yield optimal results due to limitations in their capacity.  In this context, Adacom's primary value proposition becomes apparent. Adacom offers an adaptable solution specifically tailored to small models, enabling them to effectively adapt and enhance their performance for individual test samples. This underscores the significance of Adacom as an efficient approach for small models to cater to specific use cases and optimize results.\n4. We currently find that the 220M CodeT5 model + Adacom will be better than the performance of 770M CodeT5+ mode. \n\nTraining and utilizing billion-size large models like a 1B model such as LLAMA can be cost-prohibitive for many research labs and organizations (including us). We will add this part to our limitations.\n\n# Q5: For large LMs 1B+, the inference time can be reduced to less than 1 second. Is there any trick to reduce the time of Adacom to 1s?\n\nFor devices that can afford 1B to 6B LMs with 1 second inference, we also believe they can train a small 220M LM with only 5 training samples in 10 epochs in 1 second by adopting some acceleration tricks.\n\nA possible trick to reduce the inference time of larger models is to cache the embedding of each training sample and only train the some layer of the encoder and the decoder (like 100M in each layer) and fix other layers. By this means, we can save the efforts of both forward and backward propagation. In this case, the large model only needs 0.97s on CodeSearchNet dataset.\n\n# Demonstrate the potential for small models\nOverall, we wish to underscore that Adacom primarily embodies the concept of tailoring one model to a single sample. While our focus has not been extensively directed towards optimizing training or inference times, our aim is to demonstrate the potential for small models to not only approach but even surpass the performance of larger models through retraining.\n\n"}]

Author rebuttal:
We thank the reviewers' comments for improving our work. We observe that the reviewers' concerns lie in (1) required new baselines, (2) confused technical differences with retrieval-based baselines, and (3) general presentation problems. In this rebuttal, we have fixed all the concerns and request reviewers with negative scores to change your kind stance. Your support means a lot to us. We will fix all the comments in our revision.




# New experiments compared to CodeT5+, CodeGen (reviewer UgVn, reviewer EdLg)
We run new experiments on the publicly available dataset CodeSearchNet (CSN) and the fine-tuned model parameters shared by CodeT5-base[1] on Huggingface, by measuring the classical smoothing BLEU4 scores. 

Table 1 shows that Adacom (built upon CodeT5-base) still leads the performance of code summarization (boosting 4.31% over the rest 1.23%, 3.36%, and -5.97%). The score of CodeT5+[2] 220m, 770M and CodeGen-multi can be found in the original paper of CodeT5+. 
Overall, Adacom's boosting performance can be well generalized.  

Also, CSN is a widely adopted dataset for code summarization, compared with other models like CodeBert[3] (with an average improvement of 2.84%), PLBART[4] (2.73%), CodeGen[5] (3.07%), and CodeT5+ (3.36%) as reported in their original papers, we respectfully disagree that our boost is small or non-exist (reviewer EdLg).  

# Performance compared to retrieval system RetroNLU[6] (reviewer to4c, reviewer UgVn, reviewer EdLg)
Figure 1 shows the fundamental difference between Adacom and the current retrieval systems, including REDCODER[7] and RetroNLU. Especially,  
(1) Adacom does not need extra database as REDCODER  
(2) Adacom does not need to learn similar samples in training stage as RetroNLU  
(3) Adacom can adapt a new model on-the-fly for different test samples.  

Due to time constraints, our comparison will focus solely on RetroNLU(recommend by to4c) for two reasons:
(1) REDCODER, which incorporates a substantial external dataset, introduces potential information that might not ensure fairness in comparison.
(2) The process of embedding the extensive extra data in the million-level range and subsequently searching for similar samples is notably time-consuming.


As per the original RetroNLU paper's methodology, we adhere to the process of initially constructing an index of the training dataset, retrieving the nearest neighbors, augmenting the code with the retrieved samples, and subsequently training the CodeT5-base model. In Table 2, we show the performance of RetroNLU compared to the original CodeT5-base. However, the experimental results indicate that the Retro method does not yield a performance boost.


# Novelty and Strengthen of Adacom (all reviewers)
###  Adapting a new model given a test sample
The current model is trained in a generalized manner to achieve overall performance on the test dataset. However, for specific individual samples, a more tailored model might be better suited to accurately predict the outcome. For large language models like GPT-4, using well-crafted prompts could serve as an effective means of instructing the model. On the other hand, for smaller models, Adacom could provide a simple approach to instruct the model for specific individual samples.

###  Use influence functions to remove noise
To train the model for a specific test sample, it is necessary to identify an appropriate small training dataset. One direct approach is to locate "similar" samples within the training data, such as using cosine similarity with the [CLS] embedding of the encoder. Nevertheless, our experimentation has shown that this method tends to introduce a significant amount of noise particularly when the most "similar" samples are still not similar.  

Motivated Inference function [8], we conducted an experiment where we selectively removed certain samples from the training dataset and observed the impact on predictions. Our findings indicate that in the context of code data, a single test code may heavily rely on several training codes. Removing these specific training codes can significantly disrupt the prediction accuracy for certain test code. 
 
The original inference function requires test sample labels and is time-consuming to calculate influence. As a result, we have developed a simplified yet highly effective variant of the inference function to assess each training sample. Consequently, we eliminate the retrieved samples with low influence scores. If no training samples remain, retraining is not performed.  

###  Difference from the SOTA
The SOTA transformer-based models predominantly emphasize strategies such as augmenting the training dataset[2], introducing supplementary information with prompts[6][7], or devising alternative pre-training methods[1][3]. We hold a commendable regard for these endeavours.  

However, it is equally valuable to contemplate how we can maximize the utilization of the original training dataset and the initial fine-tuned model to enhance overall model performance. Adacom offers a straightforward yet highly effective approach to enhance model performance, demonstrating the potential to surpass even larger models in terms of performance.  

Note that all the related new code and results have been published on our anonymous website.  
[1] CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation  
[2] CodeT5+: Open Code Large Language Models for Code Understanding and Generation  
[3] CodeBERT: A Pre-Trained Model for Programming and Natural Languages  
[4] Unified Pre-training for Program Understanding and Generation  
[5] CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis  
[6] RetroNLU: Retrieval Augmented Task-Oriented Semantic Parsing  
[7] Retrieval Augmented Code Generation and Summarization  
[8] Understanding Black-box Predictions via Influence Functions  


Meta-review: The paper introduces AdaCom, a method for adapting smaller language models to improve code-comment generation, supported by metrics and experiments across multiple datasets.

The strengths are mostly aligned around the strong experimental results, systematic presentation, and the paper's potential contribution to adapting smaller models for unseen projects. 

The weaknesses center on a lack of clarity in the paper (to4c, EdLg). Specific concerns about missing hyperparameters and metrics (EdLg).  Missing comparisons (to4c and EdLg). Ugvn and 4N9q focus on the issue of performance gains diminishing as model size increases, with Ugvn adding concerns about computational costs. 

After considering the rebuttal and discussions, it seems like the major concerns are resolved.
Decision: Accept (poster)