SlowGuess/NeurIPS_2023_OCR huggingface dataset:
RQ1: Cross-model Evaluation Table 4 shows that AdaCom can improve the state-of-the-art comment generators with decent significance and consistency. Overall, AdaCom can boost the smoothing BLEU-4 score by on average  $14.9\%$ , the METEOR score by  $12.2\%$ , and the ROUGE-L score by  $7.4\%$ . Compared to the boosted performance in large models, the improvement of AdaCom is more significant on the smaller models, which aligns with the expectation that smaller models can be more "struggled" over conflicting subsets of code samples. In addition, AdaCom performs in a consistent manner in the experiment. The deviation generally ranges between 0 and 0.72 in BLEU4, 0.08 and 0.5 in METEOR, and 0.09 and 0.59 in ROUGE-L.
RQ2: Cross-dataset Evaluation Table 5 shows that AdaCom generally works well on different code datasets based on CodeT5-small, which improves on average  $8.3\%$  in BLEU4 score,  $5.0\%$  in METEOR, and  $3.2\%$  in ROUGE-L. We first notice that the performance is not consistent over all dataset. The AdaCom shows much better boosting performance on the CodeKG, Cosbench, CSN-js dataset. Our investigation on these datasets shows that the number of helpful training samples in some dataset such as CSN-python are limited, which lead to minor improvement on the model performance. Secondly, as we mainly adjust our hyper-parameters for optimizing BLEU score, the BLEU4 scores increase much higher than other two measurements. We observe that the choice of a stricter threshold of helpful sample selection is a useful and practical mitigation. Given a higher threshold, AdaCom may include less irrelevant training samples to force the on-the-fly retraining. More experiment results on other sizes of models can be referred in our website [6].
RQ3: Retrieval-methods Comparison Table 6 shows that AdaCom works generally better than the retrieval-augmented methods. We find that for traditional retrieval-augmented methods, they can not adapt the model to one certain sample simply using the similar code comment. In the training stage, their model still suffers the bias of the contradictory samples and simply concatenate some similar training samples will not avoid the compromise. In the testing stage, AdaCom can adapt the model to re-learn the helpful samples and unlearn the harmful samples to gain a better performance for one certain test sample.
RQ4: Generalization Evaluation Table 7 shows the generalizability of AdaCom, which indicates that its boosting performance for cross-language, cross-PL, and cross-project is significant. For adapting from natural language to programming language, both T5 and Roberta model have much better performance even approaching the performance of traditional fine tuning model. In AdaCom, the large language model can skip the time-consuming fine tuning stage but only search few helpful samples in the target dataset and use AdaCom to fir for one certain test sample. For adapting from one
Table 6: The performance of retrieval-augmented method Retro and using CLS embedding with cosine similarity based on CodeT5-base model.
Dataset	Retro 223M	CLS-Cosine 223M	AdaCom 223M
CSN-java	20.05	20.14	20.85
CSN-js	16.15	17.35	18.81
CSNpython	19.67	19.58	20.46
CSN-go	19.46	19.12	19.61
CSN-php	24.91	26.45	26.90
CSN-ruby	14.91	14.45	15.39
overall	19.19	19.52	20.34
Table 7: Generalizability of AdaCom on cross-language, cross programming-language, and cross-project evaluation.
Generalization Type	Option	BLEU4	METEOR	ROUGE-L
before	after	bst (%)	before	after	bst (%)	before	after	bst (%)
Cross language	T5-base	10.44	41.97	302.01	24.91	57.27	129.91	18.99	50.72	167.09
Roberta-base	6.42	37.51	484.27	10.24	42.85	318.46	11.45	38.77	238.60
Cross PL	CSN-JS	7.00	15.50	121.43	14.37	23.48	63.40	7.90	18.16	129.87
CSN-Python	7.42	12.91	73.99	16.34	22.61	38.37	8.54	16.71	95.67
CSN-Go	4.20	11.32	169.52	9.95	21.99	121.01	4.91	17.14	249.08
CSN-PHP	7.56	16.71	121.03	16.39	28.61	74.56	8.66	22.38	158.43
CSN-Ruby	7.71	9.95	29.05	16.18	20.18	24.72	8.63	13.69	58.63
Cross project	CodeKG	11.52	35.19	205.47	20.82	45.57	118.88	28.03	48.64	73.53
programming language to another, AdaCom also shows effectiveness on cross programming language generalizability. For adapting across projects, AdaCom still shows quite competitive performance. The model improves a lot on the out-of-distribution data - the code from unseen projects. Generally, AdaCom has shown its effectiveness to address the distribution-shift problem on diverse scenarios.
RQ5: Runtime Overhead Evaluation Table 8 shows the detailed runtime overhead over boosting of smoothing BLEU-4 score using CodeT5-base model as the baseline model. We can see that on Windows, both AdaCom-small and base outperform the  $\mathrm{CodeT5 + }$  and on Ubuntu, the AdaCom-base shows better performance than  $\mathrm{CodeT5 + }$ . Based on this experiment, we deem that AdaCom can be well applied for the practical performance boost for the cost-effective smaller models.
RQ6: Ablation study Table 9 shows that simply utilizing helpful comments or models will decrease the AdaCom performance. In contrast, AdaCom (the last two column) can combine them and adapt the model by the helpful samples to increase the model performance. AdaCom can also adapt the base model to outperform the CodeT5+ large model.
Large Language Model Generally, the neural network model architecture follows an encoder-decoder structure. Regarding the code text as sequential data, CNN [2], LSTM [20, 38], or GRU [21] are used to serve as backbone model. With the advance of model, attention mechanism [34] and transformer [1, 8] are adopted to further improve the performance. CodeBERT [11] is a bimodal pre-trained model for multiple programming languages (PLs) and natural language (NLs), it supports
Table 8: Run-time Overhead over BLEU-4 score boosting compared to CodeT5 model.
Model Name	CodeT5	CodeT5+	AdaCom-small	Adacom-base
Parameter Size	223M	770M	60M	223M
Windows	Time (second)	1.21	4.06	2.34	4.81
Time per sample (second)	0	2.85	1.13	3.60
Average boost BLEU score	0	0.63	0.45	1.28
Boost BLEU score per second	-	22.11%	39.82%	35.56%
Ubuntu	Time (second)	0.38	1.91	1.46	3.16
Time per sample (second)	0	1.53	1.08	2.78
Average boost BLEU score	0	0.63	0.45	1.28
Boost BLEU score per second	-	41.18%	41.67%	46.04%
Table 9: Ablation Study on AdaCom
Retrieved Comment 223M	Model Only 223M	Larger Model 770M	AdaCom-small 60M	AdaCom-base 223M
CSN-java	13.18	19.17	20.83	20.06	20.85
CSN-js	13.96	15.15	17.93	17.06	18.81
CSN-cs	11.70	19.71	20.47	19.92	20.46
CSN-go	11.17	18.88	19.64	19.15	19.61
CSN-xml	17.47	24.70	26.39	25.76	26.90
CSN-ruby	8.91	14.78	15.63	15.03	15.39
overall	12.73	18.73	20.15	19.50	20.34
downstream NL-PL tasks like code search and code comment generation. Compared with CodeBERT, the input of GraphCodeBERT includes both code tokens and a code data flow graph among the variables. Compared to CodeBERT and GraphCodeBERT, CodeT5 [37] aims at a generic representation of both PLs and NLs. Following the original T5 model [31], the architecture of CodeT5 is faithful to the original Transformer.
Retrieval Based Comment Generator As code duplication is common in large-scale repositories, retrieval-based techniques like vector space model [17] and code clone detection [40] are used to find the most similar code comment pairs from the database in the early stage. By manually designing a set of rules, they can filter out the most similar pair and copy its comment as the output. Recent works combine the retrieval system and neural network model to generate comments. Researchers use neural network as semantic feature extractor to retrieve the comment of the most similar training sample. CCGIR [42] adopts CodeBERT and BERT-whitening operation [32] for retrieval.  $\mathrm{Re}^2\mathrm{Com}$  [39] uses BM25 to find the most semantically similar sample and utilizes four encoders to process target code, similar code, similar code's comment, and code abstract syntax tree. Through the attention mechanism, the four encoders' outputs are fused and fed into the decoder to get the final comment. Similarly, in [44], a fusion layer is designed to combine the CodeBERT output of both target code and retrieved code.
Different from the previous works that retrieval during the training stage, we search for the helpful samples in the testing stage to reduce the bias of the model.
Test-time Adaptation Adaptation means the generalization from one distribution to another distribution. Transfer learning [10, 43] is a technique to transfer the parameters of a pre-trained model to a new model on the target dataset. Through transfer learning, we can share the learned parameters (also the knowledge learned by the model) with the new model to speed up the learning procedure. The model is first pre-trained on the source dataset and then trained on the training data of the target dataset. Then it can be used for predicting the test data of the target. Domain adaptation techniques [12] aim to improve the performance of the model in another domain (target domain) to approach the effect of the original domain (source domain). Further, the test-time training technique [33] updates the model during testing with unlabeled data in a self-supervised way. Compared to test-time training, Wang et al. propose [35] an adaptation technique independent of the training data and training loss. The model is trained by test entropy minimization during the test stage and adapts itself to feedback from its predictions.
To the best of our knowledge, we are the first to introduce test-time adaptation for comment generation. Different from the aforementioned works, AdaCom selects the helpful samples from the existing training samples and then adapts the model for each target code sample.
Scalability AdaCom requires more time to adapt the model, especially for a billion-size model. To mitigate this, freezing techniques are necessary, specifically freezing the parameters in the encoder and only training part of the parameters of the decoder.
Construct Validity AdaCom assumes that it can find helpful examples in the training dataset based on the two metrics to enhance the performance of the model. However, there exist test cases with no helpful examples or possible helpful examples that are ignored. It also remains a question whether our proposed methods would introduce some harmful samples.
Table 10: An example showing the potentially over-fitting problem of AdaCom
Sample	Target Test Sample, ct	Harmful Training Sample, charm	Helpful Training Sample, chelp
Code	ReceiptViewModel purchase (...) { Db.User user = Db.getInstance().findUserIdByName(userName); if (user == null) { ... } Db.Account account = findAccount(user) ; return purchase(user, account, itemName) ;}	String receiveRequest(Object... parameters) throws DbUnavailableException { var id = getId(); var req = new PaymentRequest(id, (float) parameters[0]); return updateDb(request);}	ReceiptViewModel purchase (...) { Db.Product item = Db.get().find(itemName); if (item == null) { ... } Receipt receipt = ... ; if (transaction == null) { ... } return receipt;}
Ground Truth	domain purchase with userName and item-Name, with validation for userName	public method which will receive request from @link com. iluwatar. commander.Commander	domain purchase with user, account and item-Name, with validation for whether product is out of stock and whether user has insufficient funds in the account
Origin	(BLEU 16.78) register purchase with userName
AdaCom Epoch8	(BLEU 66.43) domain purchase with userName and item name, with validation for whether user is enabled or not
AdaCom Epoch11	(BLEU 84.84) domain purchase with userName and item name, with validation for the account
AdaCom Epoch15	(BLEU 47.79) domain purchase with user, account and item name, with validation for whether user is enabled or not
Internal Validity We further analyze the cases where the AdaCom shows limited performance. We find that AdaCom sometimes overfits the retrained training samples. In Table 10, despite that AdaCom can successfully select a useful reference  $c_{help}$ , it cannot stop at a correct point. In this example, AdaCom retrans the deep comment generator for 15 epochs. However, the best stopping condition happens at epoch 11, which means that AdaCom should have stopped earlier to gain higher accuracy and lower run time. We think AdaCom can leave these candidates to the users to choose and use the helpful samples to interpret the model behavior.
In summary, we propose a novel system - AdaCom which boosts the performance of comment generators on-the-fly. Extensive experiments has shown that AdaCom can effectively improve the performance of the code comment generation on diverse datasets, programming languages, and different deep language models. Further, AdaCom also shows its decent generalizability on cross-language, cross programming-language and cross-project settings. Overall, our solution shows promising results in enhancing model performance and potentially efficiency.
This research is supported in part by the Minister of Education, Singapore (T2EP20120-0019, MOET32020-0004), NUS-NCS Joint Laboratory for Cyber Security, Singapore, the National Research Foundation, Singapore, and Cyber Security Agency of Singapore under its National Cybersecurity Research and Development Programme (Award No. NRF-NCR_TAU_2021-0002) and A*STAR, CISCO Systems (USA) Pte. Ltd and National University of Singapore under its CiscONUS Accelerated Digital Economy Corporate Laboratory (Award I21001E0002), National Research Foundation, Singapore, and the Cyber Security Agency under its National Cybersecurity R&D Programme (NCRP25-P04-TAICeN), and National Natural Science Foundation of China (62172099, 23Z990203011). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Cyber Security Agency of Singapore.
[33] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. 2020. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119), Hal Daumé III and Aarti Singh (Eds.). PMLR, 9229-9248. https://proceedings.mlr.press/v119/sun20b.html
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[35] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. 2021. Tent: Fully Test-Time Adaptation by Entropy Minimization. In International Conference on Learning Representations. https://openreview.net/forum?id=uX13bZLkr3c
[36] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. 2023. CodeT5+: Open Code Large Language Models for Code Understanding and Generation. arXiv:2305.07922 [cs.CL]
[37] Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021).
[38] Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, and Zhi Jin. 2019. Code generation as a dual task of code summarization. NeurIPS (2019).
[39] Bolin Wei, Yongmin Li, Ge Li, Xin Xia, and Zhi Jin. 2020. Retrieve and refine: exemplar-based neural comment generation. In 2020 35th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 349-360.
[40] Edmund Wong, Jinqiu Yang, and Lin Tan. 2013. Autocomment: Mining question and answer sites for automatic comment generation. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 562-567.
[41] Shuhan Yan, Hang Yu, Yuting Chen, Beijun Shen, and Lingxiao Jiang. 2020. Are the code snippets what we are searching for? A benchmark and an empirical study on code search with natural-language queries. In 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 344-354.
[42] Guang Yang, Ke Liu, Xiang Chen, Yanlin Zhou, Chi Yu, and Hao Lin. 2022. CCGIR: Information retrieval-based code comment generation method for smart contracts. Knowledge-Based Systems 237 (2022), 107858.
[43] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How Transferable Are Features in Deep Neural Networks?. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (Montreal, Canada) (NIPS'14). MIT Press, Cambridge, MA, USA, 3320-3328.
[44] Chi Yu, Guang Yang, Xiang Chen, Ke Liu, and Yanlin Zhou. 2022. BashExplainer: Retrieval-Augmented Bash Code Comment Generation based on Fine-tuned CodeBERT. arXiv preprint arXiv:2206.13325 (2022).