Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study
Authors: Razan Baltaji
Saurabh Pujar
Martin Hirzel
Louis Mandel
Luca Buratti
Lav R. Varshney
Venue: Accepted by TMLR ()

Abstract: Large language models (LLMs) have achieved state-of-the-art performance in various software engineering tasks, including error detection, clone detection, and code translation, primarily leveraging high-resource programming languages like Python and Java. However, many critical languages, such as COBOL, as well as emerging languages, such as Rust and Swift, remain low-resource due to limited openly available code. This scarcity hampers the training and effectiveness of LLMs for these languages, increasing software maintenance costs and stifling innovation. Addressing this gap, we investigate the potential of transfer learning to enhance LLM performance on low-resource programming languages by leveraging data from high-resource counterparts. Our extensive empirical study evaluates transferability across 10 to 41 programming languages and five key tasks: code generation, clone detection, code repair, solution domain classification, and error detection. Additionally, we develop a performance prediction model to guess the best source languages for a given target and task, and analyze the features that influence transfer performance. We further replicate a representative subset of experiments with a larger model to test the generalizability of our conclusions to contemporary large‑scale LLMs. Our findings demonstrate that cross-lingual transfer significantly outperforms zero-shot learning, with effectiveness varying based on both source and target languages. Languages such as Java and Go emerge as the best targets, while Kotlin and JavaScript are excellent sources. Furthermore, our model reliably predicts successful transfer sources by considering linguistic and dataset-specific features, offering practical guidance for data acquisition and model training. This work contributes to the development of LLM-driven tools for low-resource programming languages and provides insights into the characteristics that facilitate transfer across language pairs.

Published in Transactions on Machine Learning Research (06/2025)
Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study
Razan Baltaji
balteji@illnooiseed edu
Deportment of Electrical and Computer Engineering University of Illmois Urbana- Chempaign
Saurabh Pujar
purrabh.pujja@iiiie com Imandel@ru..io
Louis Mandel
Martin Hirzel
hirsel@us. us. ibm. com luca. bwratfi2@bbm com
Luca Buratti
IBM Reseorch
Lav R. Varshney
varshhneyddillimoos...r edu
Department of Electrical and Computer Engineering University of Illmois Urbona- Chompaign
Reviewed on OpenReview: https: openreview. net/ forum? id= 1VV
Abstract
Large language models (LLMs) have achieved state-of-the-art performance in various soft- ware engineering lasks, including error detection, clone detection, and code translation, primarily leveraging high-resource programming languages like Python and Java. However, many critical languages, such as COBOL, as well as emerging languages, such as Rust and Swift, remain low-resource due to limited openly available code. This scarcity hampers the training and effectiveness L.MMs for these languages, increasing ware maintenance costs and stifling innovation. Addressing this gap, we investigate the potential of transfer learning to enhance LLM performance low-resource programming languages by leveraging data from high-resource counterparts. Our exlensive empirical study evaluates transferability across 10 to 41 programming languages and five key tasks: code generation, clone detection, code repair, solution domain classification, and error detection. Additionally, we develop a performance prediction model to guess the best source languages for a given target and task, and analyze the features that influence transfer performance. We further replicate a representative subset of experiments with a larger model test the generalizability of our conclusions to contemporary large-scale LLMs. Our findings demonstrate that linsull transfer significantly outperforms zero-shot learning, with effectiveness varying based both source and target languages. Languages such as Java and Go emerge as the best tar- gets, while Kotlin and JavaScript are excellent sources. Furthermore, our model reliably predicts successful transfer sources by considering linguistic and dataset-specific features, offering practical guidance for data acquisition and model training. This work contributes to the development of LLM-driven tools for lov-resource programming languages and provides insights into the characteristics that facilitate transfer across language pairs.
1 Introduction
Large language models (LLMs) leverage the fact that software code is often readable and repetitive (much like natural language (Hindle et al., 2016)) to achieve statc-of-the-art performance on many software engineering tasks such as error detection, clone detection, and code translation (Lu et al., 2021). However, LLMs have thus far been trained and evaluated mainly high-resource programming languages such Python and

---

Published in Transactions on Machine Learning Research (06/2025)
Java with vast amounts of openly available code. Yet there are many low-resource programming languages that lack enough openly available code to train LLMs.
COBOL, which supports many critical financial applications and may have over 775 billion lines of code overall (MicroFocus, 2022), is low-resource due to lack of open availability. Notwithstanding their popular- ity (Zhou et al., 2022; Coursera, 2024), new languages such as Rust and Swift start out low-resource, by definition. Other languages such as R are low-resource because most code in that language has a license that inhibits use for LLM training (Li et al., 2023). Besides being underrepresented in training of LLMs, organizations often spend significant resources training software developers to work with unfamiliar low- resource languages. This adds to the cost of software maintenance and may also stifle innovation (Edwards & King, 2021). There is a need for state-of-the-art AI tools to enhance developer productivity for low-resource languages.
Recent works have shown the possibility of transfer learning: leveraging data from one programming language to compensate for the lack of data in another (Ahmed & Devanbu, 2022; Yuan el al., 2022; Pian et al., 2023; Cassano et al., 2024). Another recent work has developed several similarity metrics to decide which high- resource programming language dataset can be used to augment fine- tuning data for target language (Chen et al., 2022). Prior work, however, has used at most 6 programming languages, none of which are truly low- resource. Further, the notion of similarity among programming languages is underexamined: more research is needed to make reliable claims.
To overcome these shortcomings, we examine transferability from source to target language combinations for tasks and for many more languages than prior work (10-41 depending on the task). We start by using training data for a task in a source language to fino-tune small LLM (for 4 tasks) to perform in-context learning using& larger LLM (for the 5th lask). Then, feed the respective LLM disjoint test data for the same task on a target language. Using this methodology. we evaluate the results across all source languages, target languages, and tasks. The source languages are, by definition, high-resource languages since we have enough data to train a model on them. Target languages include both high-resource and low-resource ones. The finetuning experiments focus on the four tasks clone detection, code repair, solution domain classification, and error detection. The in-context learning experiments use fifth task, code generation.
Figure 1 summarizes the methodology for the finetuming part of our empirical study, which builds upon the work of de Vries et al. (2022) by exploring programming languages and multiple tasks. On the left, we start with a pretrained LLM that has seen unlabeled code in several programming languages. Next, for each of four tasks, for each source language with a suitable amount of labeled training data for that task, we finetune the LLM. For each target language with a suitable amount of labeled test data for the task, we evaluate the finctuned LLM. For example, Figure 1 illustrates this with a solution domain classification task, using C as the source language and Rust as the target language, which yields an Fl score of 0.48. Doing this for four different tasks and all their source and target language combinations yields four different heat maps. The methodology using in-context learning on the fifth task is analogous and yields a fifth heat map.
This paper addresses three research questions (RQ1 RQ3).
RQ1: How well does cross-lingual transfer work for a given task across different language pairs? We answer this question directly by exploring the heat maps.
RQ2: Given lask and largel language, how should we pick the source language for best performance? We answer this question by training a performance prediction model from the language pair features and the ground-truth target labels from the heat maps, which predicts the ranking of source languages. For example, in Figure 1, the performance prediction model for the solution domain classification task might rank C behind Kotlin as a source language for transfer to Rust. Given the cost of training an LLM, the performance prediction model is useful in directing data acquisition efforts and deciding how to spend compute resources.
RQ3: Which characteristics of language pair are predictive of transfer performance, and how does that depend the given task? We answer this question by measuring the importance of language pair features in the prediction model from the previous experiment.
2

---

Published in Transactions on Machine Learning Research (06/2025)
for each task (e.g. Solution Domain Classification)
Seen Languages
for each source language (c.g. C)
Pairwise Features
for each target language
(e.g. Rust)
Unlabeled Examples
Finetuned
Labeled Examples Rust code, sag
LLM
pointsiiiaio, wacaarwrrs
coul Dello, voctdnart
CENPDA "mall, MOrLOT
( vecad""):
prtas("Bello, wocld!!!
RQ1
Performance Prediction Model
Features Importance
RQ2
Pretrained LLM
Labeled Examples C code, tag
RQ3
Figure 1: Overview. Consider an LLM pretrained unlabeled code in multiple seen languages. Finetune target language.
task-specific labeled samples from source language. For RQL, test performance For RQ2, train model that ranks transfer performance given features of a language pair. For RQ3, measure how important the features of language pairs are. Repeat for several languages and four lasks.
This paper makes the following contributions:
Evaluating the pairwise transferability for several programming languages (including low-resource ones) and five tasks using language models.
Developing a method predict the best language to transfer from, for different target languages and tasks.
Characterizing the features of programming language pairs that are predictive of transferability for given tasks.
One goal of this paper is to ofler practical guidance to engineers who build LLM-driven software engineering tools towards more informed choices for data acquisition and modeling. A second goal of this paper is to use the lens of transfer learning to shed light on programming language characteristics. Some highlights of our results are as follows:
Learning transfers well for all tasks and cross-lingual learning transfers better than zero-shot.
Transfer learning depends on the target language. Java and Go the best target languages whereas C++ and Python the worst.
Transfer learning also depends on source language used for finetuning. Kotlin and JavaScript are the best source languages, C++ is the worst.
3

---

Published in Transactions on Machine Learning Research (06/2025)
We can reliably predict which source languages will perform well given task by considering linguistic, synthetic, dataset-specifie, and model-specific features.
Different tasks rely on different features; for some, keywords and names are most important.
Assuming soflware engineers increasingly benefit. from LLM support, we believe this paper will help democ- ratize such support into low-resource settings and help inspire more rescarch in this arca.
2
Related Work
Cross-lingual transfer for natural languages. Lin et al. (2019) use tasks to study transfer among up to 60 natural languages and explore how important features of language pairs are for predicting how well learning transfers between them, finding that feature importances vary a lot across tasks. Lauscher et al. (2020) use 5 tasks to study transfer among up to 15 languages and argue that while zero-shot performance can work for low-level tasks, higher-level tasks benefit. from at least few target-language shots. de Vries et al. (2022) use task (POS tagging) with 65 source and 105 target languages and study the effects of language families and writing systems, with Romanian emerging as a particularly good source language. Ahuja et al. (2022) use 11 tasks with source and varying numbers of target languages, confirming earlier findings that feature importances vary lot across tasks. Our paper takes inspiration from these works but dilfers by studying programming languages instead of natural languages, which have different lasks and different features affecting transferability.
lioss-lingual transfer for programming languages. Zhou et al. (2022) call for studying transfer, motivating with task (code completion) and languages (from Hack to Rust). Chen et al. (2022) use tasks (code summarization and search) and languages to study transfer to Ruby, and propose picking the source language based on language similarity. Ahmed & Devanbu (2022) use 3 tasks to study transfer among 6 languages, demonstrating that due to the nature of their tasks, signals from identifiers are highly important for transferability. Yuan et al. (2022) use 1 task (automated program repair) to study transfer among 5 languages, sequentially fine-tuning on multiple languages with innovative tricks to prevent catastrophic forgetting. Pian et al. (2023) use 2 tasks (code summarization and completion) to study transfer among languages, using meta-learning to improve a base learner. Cassano et al. (2024) introduce MultiPL-T, a new semi-synthetic training dataset, and show that it improves transfer for task (NL code completion) across 19 languages. Our paper also focuses on programming languages, but considers more tasks and more languages to explore conditions for effective transfer.
There is also work on learning across multiple programming languages that has been instrumental in making studies like ours possible, but it differs in that it does not focus on cross-lingual transfer. Unsupervised pretraining on multiple languages has become common, but transferability been thoroughly studied; our paper addresses that gap. One could also imagine exploring transfer from multiple source languages a low-resource target; we leave this to future work.
supervised tasks has not yet
3 Experimental Setup
Most of our experiments are based on finetuning a small model for four tasks as illustrated in Figure I; in addition, there are also some experiments based on few-shot prompting a larger model without finetuning. For most experiments, given task, our experimental approach first finetunes the CodeT5 (Wang et al., 2021) model for each source language individually and then tests each finetuned model on all target languages.
We applicd this approach to four tasks where the number of source languages varies from 6 to 22, Icading to 58 fineluned models in total. For each source language, we finetune the model using one A100 GPU for to 30 hours depending on the task. Each model was then evaluated on 11 to 43 target languages producing 1, 8,808 experiments. All results are presented in Figures (da)-(d) and analyzed in Soction For the few-shot prompting experiments, we generated responses with the LLAMA 3.3 70B-Instruct model via Together 2025 using temperature of 0.8. Those results are presented in Figure
4

---

Published in Transactions on Machine Learning Research (06/2025)
9.6 10 9x100
88x10"
-7x10"
6=100
0.51
0.85
0.85
0.89
0.96
0.87
0.87
0.85
0.89
0.87
0.89
0.88
0.89
0.89
0.92
9.93
0.88
0.87
0.87
0.86
0.93
0.92
Dart
0.5
0.78
0.76
0.88
0.33
0.84
0.82
0.96
0.88
0.88
0.89
0.91
0.9
0.93
0.92
0.88
0.89
0.87
0.93
0.89
0.92
0.91
Go
0.44
0.65
0.56
0.85
0.66
0.67
0.85
0.95
0.69
0.88
0.91
0.92
0.92
Java
0.79
0.81
0.84
0 86
0.89
0.66
0.85
0.86
0.69
0.87
0.69
0.85
0.86
0.68
0.92
0.88
0.86
Pascal
0.5
0.76
0.86
0.84 84
0.78
0.81
0 83
0.81
0.82
0.82
0.83
0.62
0.84
0.84
0.84
0.8
0.86
0.66
0 84
0.82
0.85
0.85
Awk
0.51
0.77
0.77
0.86
0:83 0.83
0.83
0.86
0.89
0.87
8.32
0.86
0.83
0.88
0.85
0.84
0.3
0.83
0.89
0.84
0.88
0.84
Elixir
0.51
0.77
0.76
0.82 0.83 0.79
0.8 0.77 0.79
0.84
0.82 0.82
0.86
0.85
0.83
0.83
0.86
0.85
0.87 0.86
0.82
0.88
Lua
0.48
0.81
0.76 0.67
0.81
0.62
0.82 0.68
0.86
0.87
0.85
0.64
0.86
0.9
Javascript
0.74
0.77 0.63
0:83
0.83
0.62
0.81
0.86
0.85
0.86 0.84
0.86
0.87
0.65
0:911
0.88 0.86 0.85
0.84
0.86
0.86
Scala
0.49
0.73
0.76 0:84
0.83
0.85
0.8
0.82
0.83
0.83
0.83 0.88 0.84
0.87
0.9 0.88
0.87
O. 86 0.82
0.83
0.88
0.87
Typescript
0.45
0.8
0.76
0.62
9.96
0.82
0
0.78
0.81
0.82
0.83 0.88 0.85 0.87
0.86
0.85
0.85
0.83 0:97
0.9
0.88
0.8
C
0.49
0.88
0.72
0.8 0.84
0.8
0.51
0.78
0.85
0.84
0.83
0.85 0.86
0.88
0.84
0.85
0.86
0.84
0.85
0.85
0.88
0.89
Julia
0.5
0.94
0.76
0.62
0.82
0.81
0.63
0.64
0.82 0.68
0.65
0.86
0.86
0.68
0.92
Pylhon
0.8
0.78
0.67 0.86
0.82
0.61
0.84
0.63 5:52
0.87 0.65
0.81
0.9 0.91
0.86
0.88
0.62
0.87 0.86
0.89
0.89
0.52
0.84
0.8
0.88
0
0.81 0.82 0 85 0.86 0.91
0 88 0.88
0.88
0.88
0.91 0.9
0.9 0.86
0 84 0.87
0.9
0.9
Swift
0.49
0.8
0.76
0.84 9.96
0.82 0.81 0.81
0.82 0.87
0.84 0.88 0.88
0.9
0.88
0.91
0.34
0.89
0 86 0.84
0.89
0.88
Kotlin
0.48
0.85
0.78
0.8 0.83 0.82 0.82
0.79
0.87 0.86
0.84 0.89 0.87
0.9 0.88
0.88
0.9
O.
0.89 0.86
0.96
0.92
C##
0.49
0.77
0.64 0.82
0.85
0.83
0.88
0.87
0.67
0.93
Crystal
0.71
0.72
0.75 0.75
0.78 0.79
0.74 0.78 0.79 0.75
0.74 0.75
0.8 0.75
0.77 0.75
0.82
0.78
0.72
0.78
Lisp
0.5
0.68
0.76
0.78
0.71 0.77 0.78
0.79
0.77 0.77 0.81 0.72 0.78
0.78
0.79
0.81
0.77
0.8 0.76
0.76
0.76
0.78
Octave
0.49
0.74
0.74
0.8
0.71 0.76
0.77
0.77 0.74 0.72 0.76
0.74
0.71
0.73
0.75
0.77
0.76
0.72
0.76
0.74
0.76
0.79
Bash
0.5
0.72
0.74
0.88
0.71
0.72 0.76
0.71 0.77 0.77 0.77
0.71
0.75
0.76
0.77
0.71
0.75
0.75
0.73
0.73
0.74
0.8
Perl
0.40
0.8
0.72
0.74
0.79 0.77 0.79
0.71
0.74
0.77
0.76
do
0.5 0.74 0.76 0.78
0.6 0.78 0.92
0.81 0.78
0.77 0.81
0.8
0.77
0.8
0.82
0.79
0.81
0.62
0.84
0.8 0.63
0.82
Fortran
0.52 0.71
0.7 0.76
0.76
0.77 0.76
0.72
0:9
0.78
0.8
0.78
0.77
0.8
0.81
0.81
0.77
0.79
0.75
0.76
0.76
0.8
Haskell
0.49 0.72 0.74 0:83
0.77 0.77 0.79
0.76
0.84
0.74
84
0.75
0.78
0.8
8.
0.8
0.81
0.8
0.8
0.8 0.79
0.82
F#
0.48 0.71 0.74 0.81 0.78
0.81
0.78
0.77
0,82
0.79
0.9
0.79
0.8 0.82
0.82
0.83
0.77
0.77
0.84
0.79
0.81
0.81
ml
0.49 0.74 0.75 0.75
0.6 0.81
0.63
0.8
0.82
0.89
0.87
0.64
0.68
Visual Basic
0.48 0.67 0.75 0.79
0.76
0.8
63
0.66
0.79
86
62
O
82
0.86
0.63
84
0.8
84
0.73
0.77
0.61 0.81
Clejure
0.49 0.68
0.75 0.79
0.75
0.79
0.81
O
86
84
0.74
0.78
0.79
0.82
0.84
0.83
0.82
0.79
0,81
0.78
0.75
0.8 0.78
Scheme
0.49 0.68
0.7 0.76
8.82
0.78
0.74
0.74
8.83
82
0.8
0.8
0.84
0.86
0.89
0.9
0.84
0.94
0 83
0.83
0.83
0.85
Rust
0.46
0.8
0.73 0.73 0.79
0.78
7.73
0.76
0.74
0.76
0.77
0.8
0.84
0686 0.63 0.79
0.81 0.83
O. 88 0.95
0.84 0.89
C++
0.73
0.76 0.75
0.77
0.8
0.81
0.79
0.79
Cython
0.5
0.85
0.76
0.61 0.77
0.75
0.78
0.79
0.81
0.6
088
0.8
0
83
0 84 0.62 0.83 0.82 0.79
0. 0.84 0.62 0.63
0.94
Ruby
0.49
0.77
0.71
0.78 0.8
0.77
0.76
0.75
0.78
0.79
0.77
0.63
0 84
0. 94 0.87 0.82 0. 0.
0. 0.82
0.8 0.82 0.82 0.84
0.83
D
0.5
0.75
0.74
0.78 0.78
0.76
0.74
0.74
0.81
0.78
0.8
0.8
0.52
0 0:83 0.84 0.86 0.84
0.8 0.79 0.82 0.76
0.83
Nim
0.53
0.64
0.76
0.81 0.65
0.71 0.74
0.71 0.74
0.74
0.72 0.68
0.69 0.75 0.72 0.66
0.66
0.7 0.61 0.6 0.59
0.71
Sed
0.78
0.75
0:69
0.64
0.75
0.71 0.72 0.71
0.68 0.66
0.68
0.72
Vim
0.49
0.58
0.79
0.61 0.72 0.79 0.61 0.81 0.65 0.78 0.77 0.75
0.78
0.81 0.75
0.8 0.74
0.62
0.68
0.78
0.65
0.65
COBOL
0.5
0.57
0.72
0.75
0.73 0.7
0.71
0.75
0.75 0.79 0.82 0.63
0.8
0.81
0. 0.84
0.87 0.7
0.77
0.75
0.7
0.77
0.74
Moonscript
0.5
0.53
0.72
0.71
0.89
0.74 0.72 0.69 0.72 0.81 0.76
0.83
0.88 0.78
0. 0.89 0.91
0.81 0.74 0.66
0.7
0.73 0.68
bc
Zero Shot
Python
Awak
Pert
Go
Pascal
Fortian
Lisp
Haskell
PHP
mi
Java
Nim
Javascript
Scala
Kotlin
Rust
Ruby
(a) Clone Delection: 21 source languages x 41 larget languages. Metric: F1 Score.
Figure 2: Transfer scores heat map. The figure shows scores for every combination of source and target language. Each column corresponds to a source language, with "Zero Shot" showing zero shot performance, i.e., without fine-tuning on any source language. Each row corresponds to a target language. The languages whose language-name label uses red font were seen during pre-training. Framed black boxes highlight the performance of a source language (column) on itself as the target language (row). The dendrograms show results of hierarchical clustering based on similarity of the performance vectors. The row and column order is also determined by the same clustering.
Base datasets. The five tasks studied are derived from three multilingual code datasets: CodeNet, XCodoEval, and MBXP CodeNet (Puri et al., 2021) consists of about 14M code samples in 55 different programming languages, derived from submissions to online coding judge websites. The dataset comes with benchmarks for code classification and code similarity. XCodeFval (Khan et al., 2023) contains 25M samples
5

---

Published in Transactions on Machine Learning Research (06/2025)
10
10x10
aax
7xx00
6xx10
5x110"
0.67
0.79
0.61
0.86
0.65
0.87
0.86
0.83
0.78
0.96
0.65
0.84
0.84
0.85
0.88
0.88
0.83
0.6
0.81
0.79
0.89 bf
0.17
0.74
0.99
0.81
0.62
0.84
0.83
0.85
0.73
0.76
0.8
0.81
0.8
0.81
0.62
0.76
0.71
0.74
0.7
0.71
0.74
D
0.28
0.77
0.88
0.82
0.62
0
0.84
0.87
0.85
0.8
0.75
0.79
0.62
0.82
0.8
0.84
0.81
0.7
0.77
0.72
0.68
0.75
Dart
0.28
0.82
0.84
0.99
0.85
0.87
0.85
0.82
0.83
0.79
0.83
0.82
0.82
0.83
0.89
0.86
0.75
0.81
0.79
0.68
0.89
PHP
0.34
0.81
0.91
0.86
0.89
0.92
0.99
0.97
0.79
0.77
0.81
0.84
0.85
0.81
0.88
0.82
0.7
0.77
0.72
0.7
0.78
C
0.24
0.81
0.86
0.86
0.89
0.9
0.93
0.99
0.77
0.8
0.8
0.82
0.85
0.8
0.83
0.81
0.7
0.77
0.73
0.73
0.78
C++
0.2
0.79
0.9
0.87
0.99
0.89
0.9
0.91
0.79
0.78
0.81
0:85
0.83
0.81
0.87
0.82
0.74
0.77
0.77
0.76
0.79
C##
0.25
0.83
0:94
0.89
0.93
0.99
0.93
0.94
0.8
0.77
0.83
0:85
0.85
0.85
0.87
0.83
0.73
0.79
0.78
0.74
0.82
0.6
0.83
0.86
0.85
0.88
0.88
0.89
0.88
0.82
0.89
0:86
0.87
0.88
0.88
0.86
0.74
0.8
0.78
0.8
0.85
Sed
0.34
0.64
0.9
0.89
0.91
0.92
0.9
0.91
0.84
0.8
0.86
0.88
0.86
0.68
0.99
0.86
0.77
0.84
0.79
0.74
0.8
Javascript
0.25
0.62
0.86
0.87
0.87
0.86
0.86
0.86
0.82
0.78
0.84
0.86
0.84
0.85
0.95
0.84
0,76
0.81
0.78
0.7
0.78
Typescript
0.46
0.68
0.7
0.72
0.74
0.75
0.74
0.7
0.94
0.76
0.86
0:67
0.75
0.72
0.77
0.71
0.56
0.61
0.55
0.65
0.67
0.55 0.61
0.59
0.62
0.65
0.68
0.69
0.64
0.82
0.75
0.72
0.7
0.8
0.67
0.77
0.58
0.65
0.42
0.53
0.53
0.25 0.78
0.72
0.76
0.76
0.75
0.73
0.78
0.7
0.75
0.75
0.78
0.74
0.75
0.74
0.8
0.62
0.78
0.97
0.67
0.67
0.99
0.74
0.79
0.78
0.81
0.8
0.81
0.73
0.78
0.79
0.81
0.78
0.78
0.61
0.69
0.74
0.73
0.69
0.77
0.75
0.7
0.78
0.79
0.75
0.76
0.77
0.78
0.8
0.79
0.79
0.63
0.99
0.71
0.64
0.66
0.67
Fortran
0.74
0.66
0.69
0.67
0.7
0.72
0.74
0
0.73
0.72
0.79
0.62
0.71
0.67
0.66
0.68
0.69
0.73
0.76
0.75
0.73
0.72
0.73
0.76
0.67
0.66
0.64
Haskell
0.65
0.7
0.67
0.69
0.65 0.66
0.73
0.69
0.72
Kotlin
0.17
0.75
0.71
Scala
0.39
0.67
0.65
Bash
0.08
0.75
0.67
0.75
COBOL
0.21
0.77
0.77
0.7
Elixir
0.09
0.71
0.73
0.68
Visual Basic
0.29
0.63
0.71
0.72
0.72
0.75
0.77
0.75
0.61
0.68
0.64
Julia
0.23
0.65
0.7
0.72
0.79
0.64
0.68
0.68
Swift
0.06
0.65
0.67
0.72
0.74
0.72
0.76
0.73
0.69
0.72
0.72
0.67
0.7
0.68 F#
0.55
0.68
0.72
0.71
0.75
0.73
0.74
0.76
0.76
0.74
0.73
0.73
0.78
0.78
0.67
0.67
0.63
0.57
0.67 Octave
0.23
0.76
0.67
0.76
0.72
0.72
0.74
0.74
0.77
0.74
0
09
0.8
0.75
0.77
0.84
95
0.68
0.69
0.61 0.66 Cython
0.2 0.8
0.7
0.78
0.74
0.73
0.75
0.75
0.78
0.74
0.81
8.83
0.76
0.79
0.87
0.99
0.71
0.87
0.72
0.63
0.68
Python
0.2
0.76
0.69
0.77
0.78
0.75
0.77
0.78
0.77
0.75
82
0.8
0.8
0.8
0.87
0.87
0.73
0.77
0.7 0.66
0.71
Lua
0.41
0.76
0.67
0.7
0.72
0.75
0.75
0.77
0.85
0.78
0
82
0.74
0.8
0.79
0.87
0.85
0.74
0.73 0.66
0.65
0.64
Moonscript
0.37
0.68
0.71
0.76
0.75
0.77
0.74
0.73
0.7
0.74
0.67
0.72
0.73
0.75
0.76
0.76
0.63
0.64
0.63 0.63
0.96
Perl
0.07 0.7 0.73
0.74
0.73
0.75
0.74
0.75
0.68
0.77
0.8
0.74
0.75
0.73
0.77
0.73
0.65
0.66 0.65
0.98
0.7
Lisp
0.14
0.68
0.71
0.73
0.71
0.74
0.73
0.74
0.68
0.76
0.8
0.72
0.73
0.72
0.77
0.72
0.62
0.63
0.6
0.83
0.68
Scheme
0.08
0.64
0.65
0.7
0.68
0.7
0.68
0.89
0.65
0.77
0.73
0.69
0.68
0.68
0.73
0.65
0.58
0.59
0.58
0.75
0.63
Clojure
0.48
0.68
0.88
0.84
0.68
0.71
0.74
0.72
0.63
0.75
0.85
0.88
0.72
0.7
0.72
0.78
0.57
0.58
0.58
0.61
0.68
Vim
Zero Shat
Rust
PHP
C#
Jeva
Haskall
G0
Kollin
Scala
Javasolfpt
Python
Fortran
Nim
Pascal
Lisp
(b) Code Repair: 20 source languages x 38 targel languages. Metric: BLEU score.
Figure 2: (continued; see caption from 2a)
in 11 programming languages from another different online coding judge website. This dataset comes with execution-baas based evaluation framework and several different benchmarks: from classification (solution do- main classification and error detection) to generation (program synthesis, automatic program repair, and code translation). MBXP (Athiwaratkun el al., 2023) is a multilingual extension of MBPP (Austin el al., 2021) to Java, JavaScript, TypeScript, Go, Ruby, Kotlin, PHP, C#, Scala, C++, Swift, and Perl. A com- panion code package provided perform exccution in all supported languages. The datasct is designed for investigating the effectiveness of few-shot prompting, zero-shot translation, as well as the robustness prompt perturbation. In our experiment, we utilize the few shot prompting setup.
6

---

Published in Transactions on Machine Learning Research (06/2025)
5.6 10
5 10
4 10
0.01
0.47
0.47
0.56
0.53
0.54
Javascript
Rust
Go
0.47
0.46
0.51
0.51
0.57
0.5
0.51
0.51
Kotlin
0.0
0.47
0.44
0.53
0.52
0.48
0.53
0.49
0.52
0.5
0.5
0.51
РHP
0.01
0.44
0.41
0.43
0.48
0.46
0.5
0.52
0:48
0.48
0.47
0.5
Ruby
0.0
0.5
0.5
0.45
0.47
0.45
0.48
0.48
0.48
0.5
0.49
0.49
C
0.05
0.43
0.45
0.48
0.48
0.47
0.49
0.45
0.49
0.5
0.46
0.48
C8
0.01
0.37
0.36
0.32
0.35
0.34
0.4
0.39
0.39
0.39
0.36
0.41
Python
0.0
0.44
0.48
0.35
0.37
0.38
0.43
0.4
0.42
0.44
0.43
0.44
0.01
0.44
0.4
Zero Shot
C+f
PHP
Javasoria
Rust
Kollin
Ruby
C#
Java
6
Python
(c) Solution Domain Classification: 11 source languages x 11 target languages. Metric: Fl score.
Figure 2: (continued; see caption from 2a)
Tasks, To study how learning transfers between programming languages, we explore two type of tasks: classification (error detection, solution domain classification, clone detection) and generation (code repair, code generation). For the classification tasks, we follow Lewis et al. (2020) by predicting class labels basod on the final decoder hidden state. We use F1 score for evaluating the classification lasks, which is a widely accepted practice since it is good at dealing with class imbalance. We use BLEU score for evaluating the iiccution-baset metric would provide value, it infeasible due to the large
code repair task; while number of languages in CodeNet (Puri al., 2021). (Other prominent benchmarks for the code repair task also adopt BLEU score (Lu et al., 2021).) Finally, we use execution-based Pass@1 score for evaluating the code generation task. The execution-based evaluation checks the functional correctness of the generated code by running it against the provided tests.
Error Detection: Given a code C language L, do binary classification of whether C compiles (or can be loaded by an interpreter) without error.
Solution Domain Classification: Given code C' in language L, do multi-label classification into set of tags corresponding to algorithmic techniques required to write the solution (c.g., sorting, graphs).
Clone Delection: Given two code samples C and C2 in language L, do binary classification of whether they are type-IV clones (semantically similar) (Roy Cordy, 2007). We derived the dataset from CodeNet as follows. Given all combinations of solutions (clones) as pairs of accepted solutions for the same problem and the others as negative examples. To balance the data across problems, we ensure ratio of 0.15 of positive samples across different languages.
all problems in language L, we identify positive samples

---

Published in Transactions on Machine Learning Research (06/2025)
8.8
8 10
0.49
0.8
0.66
0.58
0.65
0.67
0.67
Javascript
Java
0.40
0.54
0.68
0.77
0.67
0.64
0.67
Python
Kotlin
0.54
0.68
0.64
0.66
0.68
0.68
0.63
C#
PHP
0.45
0.64
0.66
0.61
0.67
0.64
0.67
Ruby
Rust
0.51
0.85
0.87
0.65
0.88
0.65
0.87
C++
Go
Zero Shot
Javascript
Python
Java
C++
Kolinn
(d) Error Detection: source languages target languages. Metric: F1 Score.
Figure 2: (contimued; see caption from 2a)
Code Repair: Given a buggy code C' in language I,, generate the corresponding repaired code. We derived the dataset from CodeNet, modifying samples by sequentially inscrting, removing, or replacing tokens of different types for fixed ratios for different token lypes.
Code Generation: Given a prompt P consisting a function signature and a docstring in language L, the task generate the function body completion. In the few shot prompting setup, few-shot prompts consisting of three correct. functions from the respective MBXP dataset provided. Few shot examples precede the function completion prompt for each evaluation.
Data sampling. Our objective is to facilitate an extensive study on transfer across the maximum number of language pairs, which can be helpful for those working with low-resource languages. Sources are selected based on the rolative performance of finc-tuned checkpoints compared to a random bascline, in order to determine if there is enough signal in the data to facilitate learning. Languages that demonstrate higher performance than the baseline are included as sources. Given variation the size of datasets for different languages, we follow the sampling procedure of de Vries et al. (2022) to select the number of training samples and identify potential source high-resource languages. We first finetune the model with datasets of different number of samples N=(10K, 30K, 50K, 70K, 100K). Depending on the number of training examples Nr. for language L, we randomly upsample languages with We filter the set of source languages for given task based on the relative performance compared to a baseline model. We select sample sizes of 50K for solution domain classification, 70K for error detection, and 100K for clone detection and code repair.
Models. Our finetuning experiments are based on CodeT5-base (220M parameters) using the Hugging Face transformers library (Wolf et al., 2020). Code-T5 is open-source model, pre-trained eight programming

---

Published in Transactions on Machine Learning Research (06/2025)
0.8
0.6
pass
0.2
0.14
0.25
0.05
0.42
0.06
0.29
0.42
0.22
Javascript
0.23
0.22
0.33
0.29
0.12
0.09
0.37
0.06
0.13
Kotlin
0.03
0.01
0.27
0.16
0.17
0.28
0.00
0.22
0.02
0.06
0.02
Ruby
0.14
0.07
0.01
0.07
0.18
0.03
0.01
0.16
0.03
0.09
0.03
Swift
0.75
0.73
0.65
0.54
0.71
0.70
0.63
0.67
0.64
0.53
0.58
C++
0.67
0.97
0.69
0.70
0.75
0.93
0.40
0.72
0.61
0.51
0.53
0.6
0.44
0.49
0.16
0.20
0.51
0.39
0.49
0.47
0.42
Go
0.42
Python
0.10
0.37
0.47
0.66
0.72
0.72
0.56
0.55
0.44
0.55
Java
0.66
0.00
0.57
0.20
0.47
0.61
0.64
0.54
0.63
0.65
0.59
Perl
Zero Shot
swift
Java
PHР
Javascript
Kotlin
Python
perl
Ruby
Code Generation: 10 source languages X 10 targel languages. Metric: Pass@1 Score.
Figure Few shot prompting transfer scores heat map. The figure shows scores for every combination of source and target language. Fach column corresponds to source language, with "Zero Shot" showing shot performance. Each row corresponds to target language. Similarly, dendrograms show results of hierarchical clustering and framed black boxes highlight the performance of a source language on itself the target language.
languages (Ruby, JavaScript, Go, Python, Java, PHP, C, and C#). We consider these languages as seen languages and Figure highlights the names these pre-training languages red font. Due to its encoder- decoder nature, the model performs well on both code generation and code understanding tasks (Wang al., 2021) like code repair, error detection, and clone detection, among others. This, along with its relatively small size, makes it good fit. for empirical study, which requires 58 finetuned models with numerous inference runs each. We keep the same hyperparameters for all the experiments: learning rate of 2e-5, batch size of 8, and the number of epochs set to 20. The few-shot prompting experiment use LLAMA 3.3 70B- 70B-Instruct, an instruction-tuned multilingual LLM (70B parameters) (Grattafiori et al., 2024).
Data Splits Train and test splits for Error Detection and Solution Domain Classification are provided from the original dataset as described in the dataset statistics from xCodeEval (Khan et al., 2023). For Code Repair, 50,000 training examples and 1,000 test examples are synthetically generated for each language. For Clone Detection, distinct set of problems used for train and test splits, including languages with
htttp::/hmugginggacc.co cw/meta- /Llamm-amm 3-3-770--11

---

Published in Transactions on Machine Learning Research (06/2025)
Language
Train
Test
Language
Train
Test
Language
Train
Test
php
914,924
1,975
ml
175,056
2,008
sed
21,584
1,349
rb
912,159
3,963
f
172,046
1,253
lua
20,688
1,253
hs
843,011
2,045
cpp
150,492
1,792
pyx
18,184
931
py
785,760
4,235
pas 136,961
1,370
de
17,322
1,706
731,575
1,552
rs
128,271
478
vb octave
7,003
1,132
js
713,854
1,600
awk 95,312
1,035
5,370
1,476
c
555,924
2,363
88,700
1,737
vim
4,176
745
scala
473,594
1,705
sh
64,899
1,070
cob
3,437
1,038
go
352,181
867
Is
47,414
799
clj
1,058
910
java
266,248
1,338
1
45,986
585
dart
506
480
d
247,954
1,355
swift
38,531
737
moon
242
453
cs
233,059
1,151
fs
31,001
962
ex
330
549
pl
185,685
1,762
scm 22,787
826
nim
175,839
765
cr 21,733
1,356
Table 1: Number of train and test samples for Clone Detection for each language.
minimum of 450 test examples. Table shows the number of examples for Clone Detection for each language. In the few-shot prompling experiment, we conduct cross-lingual evaluation across all language pairs, using nearly 1,000 examples for each. We additionally use the provided synthetic canonical solutions to measure dataset features across overlapping problem solutions.
4 Results and Discussion
This section addresses the research questions RQ1-RQ3 stated in the introduction.
4.1 Transfer analysis
We explore RQ1 via extensive experiments every combination of source and target programming language, for each task. The four heat maps of Figure 2 show the results. A source programming language is the language used for fine-tuning a model, with samples in the training data. A taryet programming language is the language used to evaluate model, with samples in the test data. Figure 2 shows source languages on the horizontal axis and target languages on the vertical axis. Monolingual scores (where the source and the target language are the same) are highlighted with bold boxes. The first column from the left indicates zero-shot performance of the base Code-T5 model different larget languages. For RQ1 we analyze model performance trends with respect to task, source language, and target language.
Task dependency. The heat maps in Figure 2 show that transferability of source languages varies de- pending on the task. Visually this can be seen by how the color gradient varies for each task. To quantify this observation, we calculate the mean scores across all the language combinations. The mean cross-lingual scores are 0.78 for clone detection, 0.75 for code repair, 0.65 for error detection, and 0.47 for solution domain classification. The monolingual scores (in bold boxes) are higher (more red) than the cross-lingual scores, averaging 0.91, 0.98, 0.78, and 0.51 for the same four tasks. The zero-shot columns, where the model is not finetuned any source language, have lower scores (more blue) for all heat maps, averaging 0.49, 0.28, 0.49, and 0.01. Variations in the mean monolingual, cross-lingual, and zero-shot scores across tasks confirm that model performance is task dependent. The difference in overall zero-shot and cross-lingual scores indicates that in the absence of fine-tuning data for any task in any programming language, it is better to finetune with some other language in that task than to use a zero-shot setting. This cross-lingual advantage holds for every task.
Looking at these scores in absolute terms reveals that the choice of tasks also affects transferability. For solution domain classification, the difference in mean cross-lingual and monolingual scores is only 0.04,
10

---

Published in Transactions on Machine Learning Research (06/2025)
Mean Score
Task
Mono
Cross
Overall
Zero-Shot
Clone Detection
0.91
0.78
0.79
0.49
Code Repair
0.98
0.75
0.76
0.28
Error Detection
0.78
0.65
0.66
0.49
Solution Domain Classification
0.51
0.47
0.47
0.01
Table 2: Mean Scores by Task. Monolingual implies finetuning where the train and test data language is the same. Cross-lingual implics finctuning where the train and test data languages differ. Overall scores include both monolingual and cross-lingual scenarios. Zero-shot means the performance of the base pre-trained model (Code 15- (-20)) on the test set without finetuning.
Clone Detection
Code Repair
Error Detection
Solution Domain
Target
cross
O-shot
cross
O-shot
cross
O-shot
cross
0-shot
Mean Rank
Java
0.86
0.44
0.84
0.25
0.66
0.60
0.42
0.009
4.25
Go
0.86
0.50
0.71
0.16
0.66
0.55
0.51
0.003
4.25
Rust
0.80
0.49
0.77
0.23
0.66
0.36
0.53
0.001
4.50
JavaScript
0.83
0.48
0.85
0.34
0.65
0.49
0.52
0.008
4.75
Kotlin
0.84
0.49
0.76
0.17
0.66
0.48
0.50
0.005
5.00
PHP
0.84
0.51
0.82
0.28
0.65
0.38
0.50
0.004
5.25
C#
0.84
0.48
0.82
0.2
0.64
0.54
0.47
0.046
5.50
0.83
0.45
0.82
0.34
0.66
0.57
0.48
0.004
5.75
C++ 0.79
0.46
0.81
0.24
0.66
0.51
0.41
0.004
7.00
Python 0.83
0.50
0.76
0.20
0.64
0.49
0.37
0.006
8.75
Table 3: Score distribution by target languages common for all tasks. Scores are the mean of the score of every source for given larget. language. Mean rank is used to rank the languages based their ranking for each task. All the languages shown are high-resource target languages for Clone Detection, Code Repair and Solution Domain Classification, which means they are also among the source languages for those tasks. C#. Go, PHP, Ruby, and Rust are low-resource languages for Error Delection. The table shows that all languages benefit from transfer learning. Java and Go seem to benefit the most and C++ and Python seem benefit the least.
whereas for code repair, it is 0.23. The difference in mean cross-lingual and zero-shot scores for the two tasks is 0.46 and 0.48. Although overall trends show learning transfers well in all tasks, cross-lingual performance in relation to monolingual performance is highly task dependent. Within each task, model performance depends on the choice of source and larget programming language, which we explore next.
Table shows the mean score by task.
Learning transfers well for all tasks. Cross-lingual learning works better than zero-shot.
Target: language dependency. In Figure 2, high-resource target programming languages are represented on both axes. Low-resource target programming languages are represented only on the vertical axis. Models are finetuned only high-resource programming languages and tested on both high- and low-resource programming languages.
To understand overall trends across all four tasks, we consider ten target programming languages that are common to all four tasks. Six (Kotlin, JavaScript, Java, Python, C, and C++) are high-resource in all four tasks. Four (Go, Rust, РHP, and C#) are low-resource in error detection. For each of these ten languages, for cach of the four tasks, we calculate the mean cross-lingual score using all scores in the row of Figure for each language, excluding the monolingual score in the bold box and the zero-shot score in the extreme left. Based on this mean we rank each of the 10 languages for each task, and then calculate the mean rank for
11

---

Published in Transactions on Machine Learning Research (06/2025)
Clone Detection
Code Repair
Target Language
0-shot
cross
O-shot
Mean Rank
Dart
0.87
0.51
0.79
0.28
2.0
TypeScript
0.83
0.49
0.83
0.25
3.5
Elixir
0.83
0.51
0.75
0.21
5.0
Lua
0.82
0.51
0.77
0.20
5.0
Swift
0.85
0.52
0.72
0.23
6.5
Visual Basic
0.80
0.49
0.75
0.09
7,5
Cython
0.79
0.51
0.75
0.23
7.5
Julia
0.83
0.49
0.72
0.29
8.0
Moonscript
0.74
0.50
0.75
0.41
9.0
Sed
0.69
0.53
0.85
0.60
9.5
Scheme
0.78
0.49
0.71
0.14
11.5
Shell
0.74
0.49
0.74
0.39
12.0
Clojure
0.80
0.48
0.67
0.08
12.5
COBOL
0.73
0.49
0.74
0.03
12.5
F#
0.78
0.49
0.71
0.06
13.0
Octave
0.76
0.50
0.71
0.55
13.0
bc
0.73
0.50
0.67
0.55
16.5
Vim
0.69
0.49
0.67
0.48
16.5
Table 4: Score distribution for low resource target languages common for Clone Detection and Code Repair. Scores are the mean of the score of every source language for the given larget language. Mean rank is used to rank the languages based on their ranking across each task. All languages benefit from transfer learning. Dart, TypeScript benefit the most and be, Vim benefit the least.
each language. The most transferrable target languages from best worst are Java, Go, Rust, JavaScript, Kotlin, PHP, C#. C, C++, and Python. Some target programming languages are good at monolingual performance, but much worse at cross-lingual performance. For example, in-language performance for C++ is 0.95, 0.99, and 0.46 for clone detection, code repair, and solution domain classification, whereas its mean cross-language performance is 0.79, 0.81, and 0.41 for the same lasks. This might. be explained by the fact that C++ has reputation as being one of the hardest programming languages to learn for human software developers. All numbers are Table 3.
We do a similar analysis for low-resource target languages in clone detection and code repair, since these two tasks have wider and more varied set of low-resource languages. Here, the most transferrable target languages from best to worst are Dart, TypeScript, Elixir, Lua, Swift, COBOL, F#, Octave, be, and Vim. See Table 4 for details.
For error detection, only a smaller set of languages can be considered low resource. Average scores vary less across different sources with highest being Rust with 0.66 and lowest being C## with 0.64. For solution domain classification there are no low-resource target languages.
While all larget languages benefit from cross-lingual training, some benefit much more than others. Among low-resource target languages, Dart and TypeScript benefit the most.
Most Transferable Source Language. The number of source languages vary for each of the four tasks. To understand the source language dependency and to identify the most transfcrrable source language, we start from languages all four lasks have in common (C, C+++, Java, JavaScript, Kotlin, and Python). For each task, for each language, we calulate the mean gross-lingual score. Once we have the cross-lingual mean for each language, we rank the source languages based on their mean score for cach task. Basod on these rankings, we calculate the mean rank, for overall cross-lingual performance of source languages, across all target languages, across all tasks.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Clone Detection
Code Repair
Solution Domain
Error Detection
Source
score
rank
rank
score
rank
score
rank
Mean Rank
Kotlin
0.8
0.78
0.5
1
0.67
2.0
JavaSeript
0.82
0.81
0.47
4
0.62
6
3.0
Java
0.8
0.78
0.49
3
0.66
2
3.5
Python
0.73
0.79
0.49
2
0.66
5
3.75
C
0.8
2
0.78
0.45
5
0.66
4
3.75
C++
0.79
0.78
0.44
6
0.66
3
Table 5: Source languages common for all lasks ranked by mean score across larget languages and tasks. Score the mean of the scores of all target languages for given source language. We see that Kotlin is relatively the best source language and C++ the worst. This result is surprising since Kotlin is not a seen language during pre-training.
Clone Detection
Code Repair
Error Detection
Source
score
rank
score
rank
score
rank
Mean Rank
JavaScript
0.81
1
0.82
1
0.63
6
2.67
Java
0.79
2
0.75
5
0.66
1
2.67
Kotlin
0.78
3
0.77
3
0.66
4
3.33
C
0.77
0.76
0.66
2
3.33
Python 0.71
0.79
2
0.65
5
4.33
C++
0.76
0.75
0.66
3
4.67
Table 6: Source languages common for all tasks ranked by mcan across low-resource target languages and three tasks. Solution Domain Classification is missing because all the target languages for this task are resoures Score is the mean of the scores of all target languages for given source language. We see that JavaScript, Java are relatively the best source languages and C++ the worst.
The most transferrable source languages from best to worst are Kotlin, JavaScript, Java, Python, C, and C++. It may be surprising that Kotlin is the best-performing source language, since it rarely discussed in the context of training LLMs and is not part of the pre-training corpus for CodeT5 (Wang et al., 2021).
Prior work transfer learning for natural languages (de Vries et al., 2022) yielded similar surprise, with Romanian as the best-performing source language. A possible explanation could be that Romanian and Kotlin both have strong roots in dominant languages of the past (Latin and Java), while sitting al a cross-roads that helped them absorb many influences. Detailed scores and rankings are in Table
The above analysis considers all target languages, including high-resource target languages. In more realistic setting, can ignore high-resource larget languages because the monolingual option available for every high-resource language in every task. If repeat the above analysis with only low-resource languages, are left with a smaller set of target languages and only three tasks (all target languages for solution domain classification are high-resource). For the remaining three tasks and six common source languages, the most transferrable source languages from best to worst are JavaScript, Java, Kotlin, C, Python, and C++. JavaScript and Java contribute the most data to CodeT5 pretraining: Kotlin and C++ are absent (Wang et al., 2021). For details see Table 6.
If we consider only clone detection and code repair, get much broader set of about 20 common source languages. Repeating the analysis for these two tasks for low-resource target languages only, we find once again that JavaScript is the most transferable and C++ is among the least. More details are in Table 7.
Kotlin is the best source language over all target languages and tasks. JavaScript is the best source language for low-resource target languages. C++ poor source language.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Clone Detection
Code Repair
Source
score
rank
rank
Mean Rank
JavaScript
0.81
0.82
1.5
Scala
0.81
0.77
3.5
Haskell
0.81
4
0.78
3.5
Kotlin
0.78
11
0.77
5
8
D
0.81
3
0.71
15
9
Java
0.79
9
0.75
9
9
PHP
0.79
7
0.74
13
10
Nim
0.80
5
0.71
16
10.5
Python
0.71
0.79
10.5
Rust
0.79
0.72
14
11
Perl
0.79
6
0.68
17
11.5
C
0.76
15
0.76
8
11.5
Go
0.76
16
0.77
7
11.5
C#
0.77
12
0.74
12
12
0.55
20
0.77
12
Fortran
0.78
0.67
14
Awk
0.76
17
0.75
11
14
C++
0.76
18
0.75
10
14
Lisp
0.77
13
0.67
19
16
Pascal
0.77
14
0.66
20
17
Table 7: Source languages common for Clone Detection and Code Repair ranked by mean score across low-resource target languages. Score is the mean of the scores of all target languages for a given source language. We see that JavaScript, Scala, and Haskell are relatively the best source languages and C++, Lisp, and Pascal the worst.
Language Pair Performance The best model performance for each task is almost always in the mono- lingual setting. However, some cross-lingual pairs do stand out and show very good model performance, comparable monolingual. Another reason consider individual language pairs is that from the perspec- live of developers working with low-resource languages, it would be helpful to know which programming languages act the best source language.
For clone detection in Figure 2(a), a model finetuned on Ruby does almost well on Crystal, language whose syntax is inspired by Ruby. Figure 2(a-b) also shows that surprisingly, Haskell does very well on COBOL, much better than FORTRAN does on COBOL. Lisp does very well on Scheme, which can expected Scheme dialoct of Lisp. Similary Python does very well on Cython. Figure 2(a-d) also show that C-category languages (C, C++, and C#)) do well each other. Details are in Table 8.
Some language pairs show particularly good results. We need to analyze their features to understand why, motivating Section 4.3.
Dendrograms. Dendrograms for each task cluster programming languages with similar transfer learning performances. In the code repair task, syntaxically and semantically related languages are next to each other. For example, is paired with C++, JavaScript with TypeScript, Python with Cython, Lisp with Schemn. On the other hand, relatedness of languages plays a smaller role in the dendrograms for other tasks. This might be because while code repair requires deep language understanding, other tasks depend more upon other features. For instance, Ahmed & Devanbu (2022) show that some tasks depend heavily identifiers.
Relatodness of languages plays a big role for code repair but less so for other tasks.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Clone Detection
Code Repair
Error Detection
Target
Source
score
Source
Source
Crystal
Ruby
0.93
Dart
Scala
0.93
0.88
Swift
JavaScript
0.91
Kotlin
0.79
be
Scala
0.91
JavaScript
0.87
TypeScript
JavaScript
0.90
JavaScript
0.95
Elixer
C
0.89
JavaScript
0.83
Clojure
I.
0.89
bf
0.77
Visual Basic
Scala
0.89
JavaScript
0.81
Julia
Ruby
0.89
JavaScript.
0.81
Cython
Python
0.88
Python
0.95
Lua
Ruby
0.88
JavaScript
0.87
Moonscript
Scala
0.87
JavaScripi
0.87
Scheme
Lisp
0.86
L.
0.83
COBOL
Haskell
0.85
Python
0.78
F#
Haskell
0.84
Haskell
0.80
Sed
perl
0.81
0.89
Octave
ml
0.81
Python bf
0.78
Bash
perl
0.80
0.83
de
perl
0.80
Awk
Vim
0.78
C#
C++
0.68
Rust
Python
0.67
Ruby
Kotlin, Java
0.67
PHP
Kotlin, C++, C
0.67
Go
Kotlin
0.67
Table 8: The best performing source language for each low resource target language, in each task. For high resource languages, the best performing source language is usually the same language. Ruby is high-resource for Clone Detection and not a target for Code Repair, hence its results are not show for these two tasks. Similarly C#, Rust, PHP, and Go are high-resource for Clone Detection and Code Repair.
4.2 Performance Prediction
Given a task and target language, how should we pick the source language for best performance? Knowing this without having to train models on different source languages can save significant resources. It also provides basis to study important features that characterize successful transfer from language to another. To predict the performance of the models the different tasks for different source languages, we train ranking model based on gradient-boosted decision trees (Ke et al., 2017).
We consider the features defined in Table 9, grouped into four categories. (1) Lènguistic features: general properties characterizing a language pair like whether they support the same paradigms, the same style of memory management (garbage collector or not), or the same kind of type-system. properties of the syntax of language pair measured by overlap in the counts of certain token types, determined using the Pygments library the corpus of code in the given languages. (3) Dataset-specific features: propertics of the problems associated with code samples for the language pair in the online coding judge website from which the dataset came. (4) Model-specific features: whether the model saw the source, the larget, or both languages during pretraining. The linguistic features of each language are given Table 10.
(2) Syntactic features:
To predict the top source languages for given task and target language L, the set of target languages T, we train ranker using the LightGBM (Ke et al., 2017) implementation of the LambdaRank algorithm (Burges, 2010). The model takes features in Table as inputs and scores source languages L., in terms of their
15

---

Published in Transactions on Machine Learning Research (06/2025)
Feature
Mcasure Description
Linguistic Features
Object oriented
Is the language object oriented?
Type strength
e
Is the language strongly or weakly typed?
Type checking
Is the language typed statically or dynamically? Is the language type safe?
Type safety
Garbage collection
Does the language use garbage collector? Is the language standardized by a committee?
Standardized
Expression of types
Are the types written explicitly?
Paradigm
Paradigms supported (e.g., functional, imperative)
Type compatibility
Is language nominally-lyped or structurally-typed? How are params. passed (e.g. by value, by name)?
Parameter passing
Syntactic Features
Name
Number of names which overlap Number of text data which overlap Number of keywords which overlap
Text
Keyword
Literal
Number of literals which overlap
Punctuation
Number of punctuation signs which overlap
Operator
Number of operators which overlap Number of comment lokens which overlap
Comment
Syntax
Number of AST nodes which overlap
Tokens
Number of tokens which overlap
DSpasit-Specifie Features
Difficulty
Average difficulty of dataset problems
Length
Average number of tokens
Time limit
Average time limit of datasel problems
Memory limit
x Average memory limit of dataset problems
Model-Specific Features
Pretrained
S Source language is included during pretraining t Target language is included during pretraining Both source and target languages pretraining
Pretrained
Pretrained
Table Features of language pairs. Column "Measure" indicates how the feature is computed: (e)quivalence, (o)verlap, (s)ource, (t)arget, (b)oth source and target, (rd) for relative difference. For dataset specific features, (z) can be t, rd. Details of the linguistic features are in Table 10.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Garbage Collection
D data-driven
M
D
E eeventdriven
T task task-driven
drivrrien
by value, by reference
by value,, by reference by value by value, by reference by value, by reference by value, by reference by value, by reference by valne by vale..
by reference by value by value by name by value by valne, by reference by valte by value by value by value by value by value by value by value, by reference by value by reference, by value by value, by reference by value by value by value by value, by reference by whee, by name by value by valne, by reference by value by value by reference, by value by value
Parameter Passing
array
C process oriented
compiled
concurrent logic
Checking dynamic dynamic
dynamic
dynamic
dynamic dynamic
dynamic
dynamic
dynamic
dynamic
dynamic
dynamic
dynamic
dynamic
dynamic
dynamic
dynamic
dynamic
Type
static
static
static
static
static
static
static
static
static
static
static
static
static
static
static
static
static
static
static
static
P
L
M: multistaged
structural
nominal, structural
nominal structura
structural
structural
nominal, structural
nominal, structural
aspect-oriente orrieneed
P: procedural
Type Compatibility
nominal
struclural
nominal
nominal
nominal
structural
nominal
nominal
nominal
structural
nominal
nominal
structural
nominal
structural
structural
structural
nominal
nominal
struct
structural
nominal
structural
nominal
structural
nominal
structural
aspect-
nominal,
nominal,
nominal,
macro
object -orientee
Exprcssion of Types
implicit
implict
explicil
implict
explicit
explicit
implicit
explicit
inferred
implicit
explicit
implicit
explicit
inforred
explicit
inferred
implicit
explicit
implicit
implicit
implicit
inferred
implicit
explicit
implicit
explicit
implicit
implicit
implicit
implicit
explicit
implicit
impliciit
implicit
inferred
explicit
implicit
implicit
structured
pipeline
Type Safety
S
X
X
X
X
X
O
imperative
functionn tonal
Type Strength strong wesks
weak
stromg
strong
strong
strong
strong
strong
strong
strong
strong
strong
strong
strong
strong
weak
strong
strong
strong
strong
strong
weak
strong
weak
strong
weak
strong
strong
strong
strong
strong
strong
weak
strong
strong
strong
weak
block structured
D
Standardized
scripting agent- Agent-orien orientee
X
X
X
X
X
X
X
modular B:
Object Oriented
X
Table 10: Linguistic features. Paradigms:
A
F esoteric R rellective C generic
mnetaprorammmin D: distributed M:
BD
BD
declarative P: process-driven
Paradigm
F
R
R
R
S
Visual Basic
Langrage
Script
Moonscript
Typescript
Awk
Bf
C
Clojure
COBOL
C++
C#
Dart
Elixir
Fortran
F#
Go
Haskell
Java
Julia
Kotlin
Lisp
Lisp
Lua
ocaml
Nim
Octave
Pascal
PHP
Python
Cython
Ruby
Rust
Scala
Scheme
Bash
Swift
Vim
17

---

Published in Transactions on Machine Learning Research (06/2025)
Ranker
Regressor
History
0.4
0.2
Solution Domain Clessification
Code Repair
Code Generatin deneration
Detection
Detection
Figure 4: NDCG@3 scores for different rankers and tasks corresponding LOO evaluation over the set of target languages. The first four tasks are evaluated on ranking predictions on Codel15 transfer experiments, while code generation is evaluated over ranking predictions LLAMA 3.3 70B-Instruc..
relevance to the target L,. Our ranker uses boosting ensemble of 100 decision trees with 16 leaves each. We consider the normalized discounted cumulativv gain score (NDCG@K) at K = 3 as our evaluation metric. We evaluate the model using leave-onc-out. (LOO) cross-validation on the set T. For each target language Le, we train a ranker to predict rankings of different sources for each language in leaving out the source ranking for Ta as test set. For each fold, we compute the NDCG@3 score on the test set.
We compare the performance of our ranker with regression-based ranker and history ranker. The ranker
regression-bastt based ranker uses LightGBM regressor with the same hyperparameter settings and selects the top K source languages with the highest predicted scores. The history ranker selects the top K source languages with the shortest path to the given target language in a programming languages history graph² (contracted to merge different versions of the same language). Figure 4 shows the mean and standard deviation of NDCG@3 using LOO on T. While the error bars are overlapping, both regression and rankers outperform the history ranker on all tasks, indicating the importance of considering a varioty of features for predicting top-performing sources rather than simplified heuristics. Our model outperforms baseline rankers all tasks with the exception of clone detection, for which it shows slightly lower performance than the regression model. This indicates that ranking is the best suited method for predicting top performing sources. Based on these results, we use our ranker for the feature analysis in RQ3.
Rankers outperform regressors for predicting lop-performing sources. Using historical relationships among programming languages alone is insufficient.
4.3
Feature analysis
While Section 4.1 explored how well transfer learning performs for different languages different tasks, we would like to dig deeper into why they perform that way. That is, how do the features of language pairs alfect transfer? Which characteristies of the source and target language best explain the variation of transfer across tasks, the dependence of transfer on source and target pairing, as well as the superior transfer supported by specific sources? These questions can be answered by measuring feature importance.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Keywords (o)
Memory limit (rd)
Names (o)
Literals
Tokens
0.B
Operators
Length (rd)
Paradigm
Tags (js)
0.6
Difficulty (rd)
Length (t)
Syntax (o)
Time limit (rd)
Parameter Passing (o)
Expression of Types (e)
Punctuation (o)
standardized (e)
0.2
Length
Type Safety
Type Checking
0.0
Error Detection
Solution Domain Classification
Clone Detection
Code Repair
Code Generation
Figure Normalized SHAP values aggregated by tasks for the features defined in Table 9. The features are sorted by mean rank. Grayed-out features were unavailable for the given task. The first four tasks are evaluated on ranking predictions CodeT5 transfer experiments, while code generation evaluated over ranking predictions LLAMA 3.3 70B-Instruct.
The state-of-tho-ars technique for measuring feature importance, SHAP values (Lundberg Lec, 2017). considers all possible subsets S F of features. Then, the importance of a given feature is the difference it makes model performance in combination with all possible sets of other features. In the context. of ranking different source languages for cross-lingua transfer, SHAP values measure how different features contribute to transfer. Figure 5 shows SHAP values computed based on the ranking models evaluated in Section 4.2. For presentation purposes, it rescales the values so the most important feature for each task has importance 1.
Task-dependent feature importance. While previous works transfer learning across programming languages emphasize the importance of specific fcatures for cross-lingual transfer, the lack of comparison on different tasks limits their conclusions. The supported cross-lask analysis demonstrates lask-dependent feature importance whereby different features contribute differently to cross-lingual transfer depending on the task. For example, the top features for the solution domain classification task are Difficully, Literals, Nomes, and Operators. Being able to predict the solution domain for code solution requires some knowledge attribute. The difficulty score is another attribute
of the underlying problem for which the tags of the problem. Problems with similar difficulty scores require similar algorithms. The literals, names, and operators other indicators of the algorithms used in code sample. In comparison, the most significant features for the clone detection task are Token, Names, and Keywords. Detecting a clone requires different skills than classifying domains. deeper understanding of the code semantics irrespective of the similar algorithms used is needed. The overlaps in names, keywords, and more generally tokens is key for understanding the semantics code (Ahmed & Devanbu, 2022).
Prior work on transfer learning in natural languages (Lin et al., 2019) showed feature importances to vary greatly across tasks. We confirm this for programming languages.
Range of important features. Different tasks seem to not only focus on different features, but also on different number of features. While the feature importance for solution domain classification and error detection is concentrated on small number of features, is more spread out for clone detection and code repair. For example, the code repair task requires transfer of knowledge from overlapping keywords, names,
19

---

Published in Transactions on Machine Learning Research (06/2025)
and more generally different tokens from source language for fixing code in target language. On the other hand, solution domain classification focuses on fewer features, related to the problems attributes.
Higher-level tasks seemn to draw upon more features to best understand the code.
Comparison across categories. Comparing the importance of different feature categories across tasks, we observe relatively higher significance of syntactic features as demonstrated by the top two features Keywords and Names. Compared to transfer learning across natural languages, model-specific and linguistics features seem to carry less significance for transfer in programming languages. While de Vries et al. (2022) find the patential of cross-lingual transfer dropping significantly for sources and targets that are not seen by the model, our finding brings hope to not only transferring learning to low-resource target languages, but even finctuning the model on unseen languages that could serve as good sources, such as Kotlin.
Overall, the most important features across the four tasks are keywords and names.
Within-category differences. While different categories seem to have a higher impact on the model predictions, the impact of single features within category varies across task. Our results are consistent with prior work that found Nomes and Keywords to be the most important features for cross-lingual transfer in certain tasks (Ahmed& & Devanbu, 2022). However, half of the tasks show keywords to be more important than names, while the other half has the opposite order of importance.
4.4 Few-shot prompting
We examine cross-lingual transfer learning for programming languages by pairing low- and high-resource languages across classification and code-repair tasks. Whereas our initial study employed CodeTT5-baa we now replicate a representative subset of experiments with the larger LLAMA 3.3 70B-Instruct model to test the generalizability of our conclusions to contemporary large-scale LLMs. For evaluation we adopt the few-shot prompting scheme described in Section 3; this technique has been shown boost out-of-domain code-generation quality for previously unseen languages and to markedly reduce the compilation and parsing failures that arise when the model lacks explicit exposure to target language (Athiwaratkun et al., 2023).
We investigate whether rankers constructed from the features introduced earlier can accurately predict the best source language(s) for few-shot prompting. We compare different rankers explored in the previous section and test whether our ranker model can generalize to larger models. We train a ranker model on the same features described in Table 9 on Pass@1 scores. Figure 4 shows mean NDCG@3 scores. The learned ranker still surpasses other rankers and heuristic baselines, underscoring that considering multiple features is essential for accurately identifying the top source languages. SHAP analysis further reveals that expression of types and overlap in literals dominate the other features in predicting top performing sources. Languages whose type systems are richly expressed in code provide stronger semantic signals that transfer directly for code generation of other similarly typed languages solutions. Meanwhile a high lexical overlap in literals such as numeric and string constants offers immediate token-level anchors that reduce aat-of-vocabulld risk during generation. Together, these two features jointly encode semantic structure and surface- similarity, making them the most dependable indicators of best source languages for code generation.
5
Limitations
This section describes the limitations of our study, including the rationale for key design choices and the steps we took address those limitations.
5.1 External validity
One limitation that, to keep the large mumber of experiments feasible and reduce their carbon footprint, we constrained our fine-tuning experiments to modcrately sized model. We address this by using C (Wang al., 2021), a commonly adopted model in Al-for-Code research. Even if a larger model were used, insights from one model would not necessarily generalize to another. Accordingly, we included model-specific
20

---

Published in Transactions on Machine Learning Research (06/2025)
features in our study to account for potential model dependence. These features are described in Table 9 and analyzed as part of Section 4.3. Despite model-specific features, we do acknowledge that experimenting with different models, not just of varying size but also architecture, would have made our study more robust. Because the source and target languages are usually different and come in many combinations, we do not necessarily run the risk of overfitting. However, using models of varying strength could have exposed a broader range of more interesting patterns which would have made our study richer and more general. To mitigate this limitation, we conduct separate experiments with larger model and few-shot prompting.
Another limitation is that, to cover multiple languages, we limited ourselves to only five tasks. We address this by selecting tasks that span a broad range of difficulty and demonstrate diversity, as indicated by their differing performance and feature importance. Nonetheless, our study includes more tasks than any previous cross-lingual transfer study on programming languages.
A further limitation is that our datascts include synthetic clements- instance, type-IV clones (Roy & Cordy, 2007) in the clone detection data or fault injection in the code repair data. However, using exclusively natural data imposes demanding constraint that few academic studies meet, and it is also uncommon in industrial practice. To address this, we explicitly measure the importance of dataset-specific features for each language pair.
Another limitation is that we focused on the setting of zero data in the target language. This represents the lowest barrier to entry, as gathering even a small number of examples can be tricky. Therefore, we defer exploring scenarios that include some sargetllanguage data to future work.
5.2 Internal validity
We believe that our study avoids one common internal validity limitation, which revolves around insufficient runs. Across all experiments, each source-target pair is evaluated only once. But thanks to the sheer number of language pairs for each task, we get thousands of different experiments, whose results are consistent with each other. Moreover, each of these experiments involves fine-tuning with tens of thousands of samples, giving us confidence that the choice of samples is not a limitation either. However, having results based single runs does limit our ability to asses variability in experiments. Since our study is the first of its kind at this scale, it lacks directly comparable baselines. We mitigate this limitation by noting that, wherever comparisons are possible, our results corroborate those of narrower studies for instance, prior work on transfer for identifier-centee tasks (Ahmed Devanbu, 2022).
6 Conclusion
We perform a systematic and extensive study of LLM transfer learning covering up to 41 programming languages, across 5 tasks including both classification and generation. Programming languages covered include several low-resource but often still widely-used languages. Cross-lingual transfer learning performs much better than zero-shot with an LLM that has been pre-trained on code. One interesting finding is that even language not seen during pre-training, like Kotlin, can be good fine-tuning source language across several target languages and tasks. On the other hand, certain languages that are often used extensively to pre-train LLMs, like C++ and Python, are worse source or target languages relative to others.
To understand relative differences in lingsllingual performance between different programming languages, we define several linguistic, syntactic, dataset, and model-specific features of language pairs and then analyze the feature importance for a model that predicts transfer performance. We show how different features are needed for ranking source languages, compared to ad-hoc heuristics such as overlap names. To explain how language that was unseen during pre-training, like Kotlin, can be good source language, we show that seen language features are less significant compared to dataset, linguistic, and syntactic source language features. Similar to previous work of Ahmed& & Devanbu (2022), we show that keywords and names are top features on average for CodeT5. On the other hand, unlike previous work, we cover more languages and more diverse tasks, and find that feature importance vary strongly across tasks. Replicating a representative subset of experiments with a larger model under few-shot prompting confirms that these feature hicrarchies and the performance prediction gains they enable generalize to state-of-the-art L.M.ss. Overall, believe
21

---

Kabir Ahuja, Shanu Kumar, Sandipan Dandapat, and Monojit Choudhury. Multi task learning for zero shot performance prediction of multilingual models. In Proceedings of the 60th Annual Meeling of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5454 5467, May 2022. doi: wttp:///w//222aclllaae Published in Transactions on Machine Learning Research (06/2025)
that this paper sheds light how learning transfers among programming langnages, and that this ultimately leads to better models to assist users of those languages, particularly low-resource ones.
7
Data Availability
The experiments are based the publicly available CodeT5-basc (220M parameters) model (Wang et al., 2021) and the open-sourced datasets CodeNet (Puri et al., 2021) and XCodeBval (Khan et al., 2023). Our code is available at https://githun commbaltaci-r/XPL
References
Toufique Ahmed and Premkumar Devanbu. Multilingual training for software engineering. In Proceedings of the 44th International Conference on Software Engineering, ICSE '22 pp. 1443-1455, 2022. ISBN 9781450392211. doi: 00.1145/3510003335110
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xisopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ra- manathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. Multi-lingual evaluation of code generation models. In The Eleventh International Conference Learning Representattons, 2023. URL https: hatpas://pppeeeveee
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. 1xXi=:210.007722 2021.
Chris J.C. Burges. From RankNet to LambdaRank to LambdaMART: An overview. Technical Report MSR-TR-2010--2, Microsoft Research, 2010.
Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Anders Freeman, Carolyn Jane Anderson, Molly Q Feldman, Michael Greenberg, Abhinav Jangda, and Arjun Guha. Knowledge transfer from high-resource to low-resource programming languages for code llms. In Conference on Object- Oriented Programming, Systems, Languages, and Applications (OOPSLA), 2024. doi: 10.1145/3689738
Fuxiang Chen, Fatemeh H. Fard, David Lo, and Timofey Bryksin. On the transferability of pre-trained language models for low-resource programming languages. In Proceedings of the 30th IEEE/ACM Inter- national Conference on Program Comprehension, ICPC '22, pp. 401-412, 2022. doi: 10. 10.1145/3524610.
3527917.
Coursera.
Programming in Swift: Benefits of this popular coding language, 2024. hups::/ www.coursera..og/arriieeee programmingg-i-swillll
Wietse de Vries, Martijn Wieling, and Malvina Nissim. Make the best of cross-lingual transfer: Evidence from POS tagging with over 100 languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1676-7685, May 2022. doi: 10.18653/v1/
2022..cl--ongg.52.
Paul N. Edwards and John Loslic King. Institutions, infrastructures, and innovation. Computer, 54(1): 103 109, January 2021.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh
22

---

Published in Transactions on Machine Learning Research (06/2025)
Goyal, Authony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Tourct, Chunyang Wu, Corinne Wong, Cristian Canton Ferver, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyath, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evti- mov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiccao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Josoph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia,, Kalyan Vasuden Alwala, Karthik Prasad, Kartikey Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Airassy, Pengchuan Zhang, Peng- wei Li, Petar Vasic, Peter Weng, Prajiwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Sil- veira Cabral, Robert Stojnic, Roberta Raileamn, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Romnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hos- scini, Sahana Chennabasappa, Sanjay Singh, Scan Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarck Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Uijwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ra- manathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajufeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yumus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annic Dong, Annic Franco, Anuj Goyal, Aparajita Saraf, Arka- bandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assal Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo WU, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Ed- ward Dowling, Fissa Jamil, Elaine Montgomery, Fleonora Presani, Fmiiy Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Flo- rez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Takshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elac Elena Veliche, Itai Gat, Jake Weiss- man, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean- Baptist Gaya, Jeff Marcus, Jeff Tang,
23

---

Published in Transactions on Machine Learning Research (06/2025)
Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Vecraraghavan, Kelly Michelena, Koqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Moham- mad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Nor- man Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyag- ina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodrigues, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyn Yang, Zhiwei Zhao, and Zhiyu Ma. The Llama 3 herd of models. Xiv, 2024.
Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. On the naturalness of software. Comsmunications of the ACM, 59(5):122 131, 2016. doi: 10.1145/2902362.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tin-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems, 30, 2017.
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. xCodeEval: A large scale multilingual multitask benchmark for code understanding, genera- tion, translation and retrieval, 2023.
TTv:2303.030044:
Anne Lauscher, Vinit Ravishankar, Ivan Vulié, and Goran Glavaš. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Longuage Processing (EMNLP), pp. 4483 4499, November 2020. doi:
10.18653/ 1/1////vv//// /tt//2200..mmpp.. 363.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising requence-to-sequene pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Länguistics, pp. 7871-7880, July 2020. doi: 10.18653 v1 /00::1...65333///0000aaaalll /20200.aclmmmin
Raymond Li, Loubna Ben Allal, Yanglian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskki, Terry Yue Zhuo, Thomas Wang, Olivier Dehacne, Mishig Davaadorj, Joel Lamy-Poirier, João Montciro, Olch Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Ho Yee, Logesh Kumar Umapathi, Jaan Zhu, Ben- jamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel,
24

---

Published in Transactions on Machine Learning Research (06/2025)
Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Dan- ish Contractor, Siva Reddy, Daniel Fried, Damitry Bahdanau, Yacine Jernile, Carlos Muñoz Ferrandis, Scan Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder: may the source be with you! Xiv, 2023.
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios A Anastasopoulos, Patrick Littell, and Graham Neubig. Choosing transfer languages for cross-lingual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3125-3135, 2019. doi: 1118633/v1/199/
Shuai Lu, Daya Guo, Shuo Ren, Junjic Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement,
Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. Xrxi 2021.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, 2017.
MicroFocus. Cobol market shown to be three times larger than previously estimated in new indepen- dent survey, 2022. URL https://www.iirrre cobol hrw.pp/wo.....t...eeeeeeeeeeeeer
Weiguo Pian, Hanyu Peng, Xunzhu Tang, Tiezhu Sun, Haoye Tian, Andrew Habib, Jacques Klein, and Tegawendé F. Bissyandé. MetalPTrans: A meta learning approach for multilingual code representation learning. In Proceedings of the Thirth-Seventh AAAT Conference on Artificial Intelligence, AAAT23, 2023. doi: 10.1609
Ruchir Puri, David Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian T Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika. Thost, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Urich Finkler, Susan Malaika, and Froderick Reiss. Codenet: A large- scale AI for code dataset for learning a diversity of coding tasks. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, Virtual, 2021. Curran
Chanchal Kumar Roy and James R. Cordy, A survey on software clone detection research. Technical Report 2007-541, Queen's University, 2007.
Ao. Together ai, 2025. URL https: ai. Large Language Model Platform.
Yue Wang, Weishi Wang, Shafiq Joly, and Steven C.H. Hoi. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696-8708, November 2021. doi: 10.18653/v1/ 102lle main.685.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Touf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-arl natural language processing, 2020.
Wei Yuan, Quanjun Zhang, Tieke He, Chunrong Fang, Nguyen Quoe Viet Hung, Xiaodong Hao, and Hongzhi Yin. CIRCLE: Continual repair across programming languages. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2022, pp. 678-690, 2022. doi: 10.1145/ 3533767-3534211
25

---

Published in Transactions on Machine Learning Research (06/2025)
Wen Zhou, Seohyun Kim, Vijayaraghavan Murali, and Gareth Ari Aye. Improving code autocompletion with transfer learning. In International Conference on Soflware Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 161 162, May 2022, doi: 10.1145/3510457..0 15511457.3313000
26


Reviews:
[{"review_id": "4CdAVijbRf", "reviewer": "Reviewer_fe5e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength:\n\n1. **Extensive Coverage of Low-Resource Languages:**  \n   The study significantly broadens the scope of prior work by evaluating cross-lingual transfer across a large number of low-resource programming languages—many of which are often overlooked in LLM research. This enhances the practical relevance of the findings, especially for underrepresented developer communities.\n\n2. **Open Access to Code and Models:**  \n   The authors provide the source code and commit to releasing the fine-tuned models and data generation framework, facilitating reproducibility and further research. This commitment to openness is commendable and aligns with best practices in empirical machine learning.\n\n3. **Comprehensive Analysis Beyond Performance Metrics:**  \n   In addition to basic accuracy comparisons, the paper offers deeper insights through performance prediction modeling and feature importance analysis using SHAP. This extra layer of interpretability helps illuminate *why* certain transfers work better than others and provides actionable guidance for future LLM applications in software engineering.\n\n\n# Weakness:\n1. **Limited LLM Diversity:**  \n   The study evaluates cross-lingual transfer using only a single language model—CodeT5-base. While the authors acknowledge this constraint in Section 5 as a tradeoff to reduce computational cost and maintain feasibility, the findings may not fully generalize to other LLM architectures or larger-scale models.\n\n2. **Single Trial per Source–Target Combination:**  \n   Each source-target language pair for a given task is evaluated using only one trial. This lack of repeated runs limits the ability to assess variance or statistical confidence in the reported results, which is important given the stochastic nature of model training.\n\n3. **Lack of Hyperparameter Description:**  \n   The paper does not disclose the specific hyperparameters used during fine-tuning (e.g., learning rate, batch size, number of epochs). This omission hampers reproducibility and makes it difficult to assess whether observed performance differences could be influenced by tuning decisions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Experimentation that examines the cross-lingual transfer learning in programming languages over four software engineering tasks:\n\n- The study ran over 1,800 experiments across 41 programming languages to measure how effectively learning transfers between languages on real-world code tasks.\n\n- A prediction model was developed to identify the best source language for a given task and target language using a wide range of language pair features.\n\n- The analysis revealed that syntactic overlaps—such as shared names and keywords—play a bigger role in transfer success than language family or pretraining exposure.\n\n- Cross-lingual fine-tuning significantly outperforms zero-shot learning, and Kotlin and JavaScript emerged as top-performing source languages.\n\nIn addition, The paper develops a ranking model using gradient-boosted decision trees (LightGBM LambdaRank) to predict the best source programming language for transferring knowledge to a target language on a specific task. It uses features like linguistic similarity, syntactic overlap, dataset characteristics, and whether the languages were seen during pretraining to guide efficient and effective source language selection.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Acknowledge Single-Trial Limitation:**  \n   Each source–target pair is evaluated only once, limiting the ability to assess variability. This should be explicitly addressed in Section 5.\n\n2. **Add Fine-Tuning Hyperparameters:**  \n   The paper omits key hyperparameter settings (e.g., learning rate, batch size), which hinders reproducibility. Including these would improve clarity.\n\n3. **Discuss Model Diversity More Fully:**  \n   While Section 5 notes that results may not generalize across models, evaluating even a few additional LLMs would improve the study’s robustness. This trade-off should be discussed more thoroughly.\n\n4. **Clarify Scope of Contributions:**  \n   In section 1, the paper should clearly state that its findings are based on a single model and a specific experimental setup. For example, using “LLMs” (plural) in the introduction is misleading and should be corrected to reflect the preliminary nature of the results. Phrases like “shed light” in the conclusion appropriately reflect the exploratory nature of the work, the introduction should be more careful in framing the results to avoid overstating their generality.\n\nThe requested changes are primarily focused on improving the clarity and accuracy of the writing, which affects the assessment of the match between claims and evidence.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TuRsmWTvei", "reviewer": "Reviewer_2XXU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The experiments are extensive, covering many pairings between languages. This provides a comprehensive picture on each language's potential in the transfer learning task.\n2. Section 4.2 on Performance Prediction is interesting as it provides a way to determine what a best source language is given a target language and a task. The results show that the collected features have some predictive power.\n3. The paper is well-written overall and easy to follow.\n\nWeaknesses:\n1. The model used in the experiment is quite small (CodeT5-base has only 220M parameters) so it is unclear whether the findings in the paper holds for larger and more recent models.\n2. Code repair is the only generative task studied. Text-to-code is arguably more prevalent in evaluating LLM code generation. Furthermore, BLEU score is used to evaluate code repair, which measures shallow string matching not functional correctness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors study cross-lingual transfer learning in programming languages for language models. They conducted extensive experiments pairing low and high-resource programming languages to study transfer learning effects on two tasks: classification and generation. They also created language and dataset features to enable prediction on transfer outcomes. The findings are likely useful for improve LLMs capability on low-resource programming languages by leveraging high-source ones for transfer learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Could the authors add some additional experiment results with a larger and more recent base model? Of course we don't expect as comprehensive as the main paper's study with CodeT5-base, but a representative subsets that can corroborate the conclusions in the paper and showcase the performance prediction generalizes to new models could strengthen the paper.\n2. In addition to the BLEU score, can the authors conduct some experiments measuring functional correctness?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fEE8ldF4GW", "reviewer": "Reviewer_44MQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The experiments are extensive and extremely thorough, capturing something like 41 programming languages, and looking at performance in diverse tasks.\n* The experiments are conducted on contemporary larger models, making them highly relevant.\n* As a result, any conclusions drawn from the findings are valid and more broadly generalizable.\n* Good use of figures/visualization/color to depict the pipeline and results.\n\nWeaknesses:\n* Related work is more like a laundry list of facts and less of a synthesis/analysis of information to motivate where this work fits in and how it closes a gap.\n* Because the primary research contribution is the findings of the analysis, it would be important to describe these more clearly in detail. For instance, these sections are currently short and lack specificity. (Specifically, they do not explicitly link back to the overarching questions or suggest practical implications.) For the contributions of this paper to be effective, strong and clear take-home messages are needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides large-scale results for fine-tuning a model on one programming language and testing (on tasks like clone detection and code repair) on another.  These cover a diverse medley of types of programming languages and unique pairings.  The authors use these results to provide analysis on when the knowledge transfer works well and when it does not with respect to features of the programming language pair.  The results themselves are a very strong resource contribution, and the analysis provides an initial research contribution in this area.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition to addressing the weaknesses above:\n* phrase \"naturalness of software\" unclear\n* It seems like the paper does not apply any methods to improve cross-lingual transfer, rather, it applies the straightforward pipeline of fine-tuning on the source and testing on the target without modification to this process. Because of the incredibly thorough and holistic experiment results, this is not a problem at all, but it should be clarified early in the paper that it’s an evaluation paper rather than a comparison of methods.\n* Similarly, the title of the paper is misleading because it made me think the paper would evaluate methods of promoting cross-lingual transfer.\n* The conclusion is extremely short and would benefit from a clear overview of the main findings in the analysis of the results. \n* Including some sort of figure of the analysis results would be a big bonus to strengthen the paper further. (I can imagine a really good figure summarizing the results from your impressive large scale experiments going \"viral\" on academic twitter or being used in keynote talks)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NTsFZGXMXs", "reviewer": "Reviewer_hhQA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Comprehensive Cross-Lingual Study. The paper evaluates cross-lingual transfer across up to 41 programming languages.\nIt investigates four tasks, covering both classification (error detection, solution domain classification, clone detection) and generation (code repair).\n- Novel Contribution to Transfer Learning in Programming Languages. This study provides a systematic exploration of transferability at scale. It presents an empirical heatmap analysis, highlighting optimal source-target language pairs.\n\n- Performance Prediction Model for Transfer Learning\n\n- Feature Importance Analysis with SHAP Values. The study quantifies the impact of various features on transfer learning using SHAP analysis. The results confirm that different tasks depend on different features, providing a fine-grained understanding of transfer learning mechanisms.\n\n- Empirical Validation on Large Datasets. The study fine-tunes 58 models, conducting 1,808 transfer learning experiments, ensuring statistical robustness. The use of CodeNet and XCodeEval, two large multilingual code datasets, enhances the credibility and reproducibility of results.\n\nWeaknesses\n- Focus on Zero-Data Transfer, Excluding Few-Shot Learning. The study assumes zero data availability for target languages.\nFew-shot or semi-supervised learning scenarios could be explored to evaluate transfer with limited in-domain data.\n\n- Lack of Evaluation on More Complex Software Engineering Tasks. The study does not address code generation, completion, summarization, or translation, which are key applications of LLMs in software engineering.\n\n- Synthetic Data in Some Experiments\n\n- Limited Scope of Model Architectures. The study is restricted to CodeT5 (220M parameters), which may not generalize to larger-scale models (e.g., GPT-4, PaLM, or StarCoder).\nResults may differ for transformer architectures trained on different corpora.\n\nWhile the paper acknowledges the computational burden, scaling to larger models and broader experiments would show more significance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an extensive empirical study on cross-lingual transfer learning for programming languages, addressing the challenge of improving LLM performance for low-resource programming languages. The study evaluates transferability across up to 41 languages and four key software engineering tasks and introduces a predictive model to select the best source languages for effective transfer. This work is a significant step toward democratizing AI-assisted software engineering for low-resource programming languages by leveraging transfer learning insights.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Broaden the Scope Beyond Zero-Shot Transfer\n\nRecommendation: Include experiments with few-shot learning (e.g., instruct-tuning or prompt with a small amount of target language data) to provide a more realistic assessment of cross-lingual transfer.\n\n- Expand the Set of Software Engineering Tasks\n\nRecommendation: Evaluate at least one additional generation task beyond code repair to test transferability in more complex settings.\n\n- Provide a Detailed Computational Cost Analysis\n\nRecommendation: Offer an estimate of training costs (e.g., GPU hours, energy consumption) and discuss strategies for efficient transfer learning.", "broader_impact_concerns": "Intellectual Property and Licensing Risks\n\nThe datasets used (e.g., CodeNet, XCodeEval) are based on public code, but some low-resource languages have restrictive licensing (e.g., R). Unintended use of proprietary or GPL-licensed code for LLM training may lead to legal challenges.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an impressive body of work with extensive experiments that address important questions. The reviewers highlighted the following strengths and weaknesses:

**Strengths:**

- Comprehensive Experiments: The paper covers an impressive set of experiments across 41 languages 
- Relevance: The study tackles a key issue in LLM research, offering insights that could benefit real-world applications 
- Predictive Model and Feature Analysis:  The paper offers deeper insights through performance prediction modeling and feature importance analysis using SHAP

**Weaknesses:**

- Single Model Limitation:The use of only a single small model (CodeT5-base) is seen a limitation for generalization understanding of the method.
- Lack of multi trials evaluation: Each source-target combination for a given task is evaluated using only one trial this raises concerns about the statistical confidence in the results.
Decision: Accept (poster)