Venue: NAACL 2025

# PLEX: Adaptive Parameter-Efficient Fine-Tuning for Code LLMs

using Lottery-Tickets

Jaeseong Lee\({}^{1*}\) Hojae Han\({}^{1*}\) Jongyoon Kim\({}^{2}\) Seung-won Hwang\({}^{12}\)+

Naun Kang\({}^{3}\) Kyungjun An\({}^{3}\) Sungho Jang\({}^{3}\)

{\({}^{1}\)CSE, \({}^{2}\)IPAI} Seoul National University

Samsung SDS

{tbvj5914,stovecat,john.jongyoon.kim,seung wonh}@snu.ac.kr

{naun.kang,kyungjun.an,shl19.jiang}@samsung.com

###### Abstract

Fine-tuning large language models (LLMs) for code generation is challenging due to computational costs and the underrepresentation of some programming languages (PLs) in pre-training. We propose PLEX, a lotterly-ticket based parameter-efficient fine-tuning (PEFT) method that adapts LLMs to either well-supported and underrepresented PLs. During lottery ticket selection, PLEX employs a dual strategy: for well-represented PLs, it leverages the LLM's full parametric knowledge by selecting from full layers, while for underrepresented PLs, it narrows the selection scope to dense layers, prioritizing the most influential parameters. Additionally, PLEX-E, a low-rank extension of PLEX, further reduces computational costs by limiting the scope of fine-tuning. On MultiPL-E benchmarks, PLEX achieves state-of-the-art performance among PEFT methods, while PLEX-E maintains competitive results with reduced computational overhead. Both variants demonstrate effective adaptation across diverse programming languages, particularly for those underrepresented in pre-training.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

## 1 Introduction

Code generation is a critical task in software development, and large language models (LLMs) have shown great promise in this domain Chen et al. (2021); Allal et al. (2023).

In industrial settings, serving code LLMs often requires optimized generation for a target PL. Moreover, the target language can be proprietary, such as those used in chip design, which are typically absent from pretraining corpora.

Fine-tuning LLMs for specific PLs faces a computational bottleneck: While scaling laws suggest that larger models yield better performance, fully adapting the model for each target PL is prohibitively resource-intensive. This underscores the importance of parameter-efficient fine-tuning PEFT Hu et al. (2022); Ansell et al. (2022), which addresses this issue by adapting only a subset of parameters, making fine-tuning more feasible and efficient.

An additional challenge arises when the target PL is underrepresented. We specifically use the term 'underrepresented', distinct from 'low-resourced', as low-resourced is defined with respect to an amount of public training resources per language. Meanwhile, underrepresentation is defined with respect to a specific PL-model pair. For instance, one language that is abundantly observed in pretraining one model, can be underrepresented in another.

Needs for adapting to an underrepresented PL are common in industrial setting, when supporting a rare PL or a private or proprietary language unavailable during pre-training. However, in our preliminary experiments, we observe that no existing PEFT method is one-size-fits-all for both well-supported and underrepresented PLs.

To address these issues, we propose PLEX, a novel PEFT method designed to efficiently adapt LLMs to both well-supported major PLs and underrepresented or private PLs. Our method employs an adaptive parameter selection using lotteryticket Ansell et al. (2022); Frankle and Carbin (2019); Chen et al. (2020), adjusting the parame

Figure 1: PLEX adaptively finentunes code LLMs by selectively updating parameters. For well-supported PLs, it uses a full parameter space, while for underrepresented or private PLs, it focuses on dense layers.



ter groups based on whether the target PL is supported by the LLM. For well-supported PLs, PLEX leverages the full parameter space for ticket selection, maximizing the use of the model's pretrained knowledge. For underrepresented or private PLs, PLEX narrows the focus to dense layers, ensuring that the most influential parameters are prioritized during fine-tuning Meng et al. (2022).

To further improve computational efficiency, we introduce PLEX-E, an extension of PLEX that replaces full fine-tuning with low-rank LoRA tuning Hu et al. (2022) for parameter selection. This reduces the computational burden while minimizing performance drops, making it feasible to apply to larger models.

Our experimental results on the MultiPL-E HumanEval and MBPP benchmarks Cassano et al. (2023) demonstrate the effectiveness of PLEX. The method performs well not only on well-supported languages like Java, PHP, C++, and Swift for StarCoder-7B Li et al. (2023) and Java for SantaCoder-1.1B Allal et al. (2023), but also on underrepresented languages like PHP and C++ for SantaCoder-1.1B. These results validate the adaptability of PLEX across a diverse range of programming languages, including those underrepresented during pre-training. Additionally, in the StarCoder-7B experiments, PLEX-E, the computationally efficient version of PLEX, generally outperforms existing PEFT baselines while remaining competitive with PLEX. The code and dataset are publicly available.1

Footnote 1: [https://github.com/thknightfly/PILEX](https://github.com/thknightfly/PILEX)

## 2 Related Work

### Multilingual LLMs

LLMs trained on large programming-related corpora, such as GitHub or The Stack, inherently support code generation across diverse PLs. However, their performance often declines when focusing on a specific PL, due to the curse of multilinguality Conneau et al. (2020).

This degradation is more pronounced for an underrepresented PL. An immediate solution is rebalancing the pretraining corpus by adding substantial data for the target PL. However, this incurs pretraining costs.

A widely deployed solution is finetuning a pretrained multilingual model specifically on the target PL Chen et al. (2021); Nijkamp et al. (2023); Guo et al. (2024).

Our distinctionWe question a widely adopted approach of training an LLM with a large number of PLs, or fine-tuning the whole LLM. Our distinction is employing parameter-efficient fine-tuning (PEFT) to adapt to a target PL. This is different from works utilizing PEFT to code LLMs Zhuo et al. (2024) for a different purpose of adapting to other tasks, not PLs.

### PEFT: Parameter-Efficient Fine-Tuning

To adapt pretrained LMs to specific PLs for code generation, PEFT methods, which aim to add a handful number of parameters for fine-tuning, were a popular solution. A common PEFT employed for code LMs was LoRA Zhuo et al. (2024), which fine-tuned a low-rank subspace of each weight matrix. However, LoRA often missed important information outside of low-rank space Chen et al. (2023).

A promising alternative was LT-SFT Ansell et al. (2022), which aimed to find lottery tickets Frankle and Carbin (2019); Chen et al. (2020), which is a subnetwork whose performance is similar or better when fine-tuned. However, it required full fine-tuning, which was not practical for ever-enlarging LMs. Moreover, we noticed sometimes it underperforms LoRA significantly.

Our DistinctionWe find that both LoRA and LT-SFT are suboptimal for PL adaptation, and propose PLEX, a best practice for parameter-efficient adaptation to both under- and well-represented PL as a target PL. We observe when LT-SFT underperforms despite a higher cost, to aim at reducing such cases. Moreover, we devise PLEX-E, a computation-efficient version of PLEX, that is more suitable for large LMs.

## 3 Proposed Method

### Both LoRA and LT-SFT are Suboptimal In PL Adaptation

Our first finding is that both LoRA and LT-SFT are suboptimal when adapting pretrained LMs to diverse PLs.

While LoRA is a popular PEFT method Zhuo et al. (2024), it may miss important information outside of low-rank space Yu et al. (2017). A promising alternative would be the lottery-ticket-based PEFT method, LT-SFT Ansell et al. (2022).

Although we observe it usually outperforms LoRA, we observe sometimes it significantly underperforms LoRA (Table 1). To investigate why, 

we analyze the updated parameter distribution over layers, and find that the updated parameters in dense layers are too small (Figure 3 blue in sub-subsection 4.3.2). This can be a problem, since most knowledge is believed to reside in dense layers Meng et al. (2022).

### Proposed: PLEX

To address this, we propose PLEX, which moves beyond the limitations of the low-rank assumption Hu et al. (2022), known to overlook critical information Yu et al. (2017). Instead, we focus on finding lottery tickets within the network and ensure the selected parameters reside in dense layers, where most knowledge is concentrated Meng et al. (2022), overcoming the shortcomings of existing methods. Last but not least, for large language models, we make our version to be computation-efficient, avoiding the expensive full fine-tuning of LT-SFT.

One promising alternative of LoRA, LT-SFT Ansell et al. (2022), the lottery-ticket-based PEFT method, first finds lottery-ticket- a subnetwork whose fine-tuned performance is comparable to fine-tuning the full model. Formally, given a neural function with pretrained weight \(\theta\in\mathbb{R}^{N}\), finding a ticket corresponds to finding a mask \(m\in\{0,1\}^{N}\), by choosing the parameter with the largest movement Sanh et al. (2020) after fully fine-tuning the model Ansell et al. (2022). Then it restricts the training updates to be \(\Delta\theta\odot m\), converting the given parameters as follows:

\[f_{LT}(\theta)=\theta+\Delta\theta\odot m \tag{1}\]

where \(\odot\) is element-wise multiplication. \(\epsilon=\frac{\|m\|_{0}}{N}\) is naturally referred to as the density, tuned as a hyper-parameter, but expected to be \(\ll 1\) for sparsity, where \(\|\cdot\|_{0}\) counts the number of non-zero values.

However, LT-SFT underperforms in underrepresented PLs, likely due to the low proportion of parameters in dense layers (Figure 3 blue in sub-subsection 4.3.2). Therefore, PLEX focuses the updates to dense layers only, for efficient adaptation to PLs. Formally, we update the given parameter as follows:

\[f_{PLEX}(\theta)=\theta+\Delta\theta\odot m\odot(1-\mathbbm{1}(l\in\mathrm{ UR})m_{D}) \tag{2}\]

where \(\mathbbm{1}(l\in\mathrm{UR})\) is an indicator function conditioning whether given PL \(l\) is underrepresented2 or not, and \(m_{D}\in\{0,1\}^{N}\) is 1 where the index does not correspond to any dense layer. Here, density is defined as \(d=\frac{\|m\odot(1-\mathbbm{1}(l\in\mathrm{UR})m_{\overline{D}})\|_{0}}{N}\).

Footnote 2: \(l\in\mathrm{UR}\) can be empirically decided based on zero-shot performance (Section 4).

### PLEX-E: Computation-Efficient Variant for Large LMs

The proposed PLEX could overcome the downside of LT-SFT, but it would not be practical to apply to large LMs, since it requires fully fine-tuning the dense layers.

Inspired by LoRA, given a dense layer weight \(W\in\mathbb{R}^{a\times b}\), instead of directly optimizing the weight, we reduce the computational cost by applying low-rank updates as follows:

\[\Delta W=W_{u}W_{d} \tag{3}\]

where \(W_{u}\in\mathbb{R}^{a\times r},W_{d}\in\mathbb{R}^{r\times b}\) are the optimization target. The computational cost is controlled by reducing \(r\).

## 4 Experiments

In this section, our goal is answering to the following research questions:

* RQ1: How do existing PEFT methods (LoRA, LT-SFT) exhibit complementary strengths and weaknesses across different PLs?
* RQ2: Can we design a PEFT method that combines the advantages of both LoRA and LT-SFT while mitigating their limitations?
* RQ3: How can we maintain the benefits of our approach while achieving computational efficiency for LLMs?

### Experimental Setup

Model SelectionWe evaluate the effectiveness of PLEX on code generation with pretrained LMs. We strategically select SantaCoder-1.1B Allal et al. (2023)4 for our main experiments due to its focused pretraining on only three PLs Python, Java, and JavaScript). This focused pretraining provides a controlled setting for simulating underrepresented scenarios with PLs absent from pretraining data (see Figure 2). To investigate the scalability and 

computational efficiency of PLEX, particularly PLEX and PLEX-E, we extend our evaluation to StarCoder-7B [13].5

Footnote 5: [https://huggingface.co/bigcode/starcoderbase-7b](https://huggingface.co/bigcode/starcoderbase-7b)

Evaluation MetricsConsistent with existing works [20, 14], we use Pass@\(k:=\mathbb{E}_{\text{Problems}}[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}]\)[68] as the main metric to evaluate the code generation abilities of pretrained LMs. Note that, for an unbiased evaluation, Pass@\(k\) calculates the average probability of selecting at least one of \(c\) correct code snippets from every combination of \(k\) samples chosen from \(n\) given samples.

Datasets and Languages for EvaluationWe evaluate PLEX on MultiPL-E [9], which expands the Python-only benchmarks HumanEval (\(\mathcal{H}\)) and MBPP (\(\mathcal{M}\)) to support diverse PLs.

For adaptation to specific PL, we utilize MegaCodeTraining corpus,6 filtered to contain the specific PL of our target. As PEFT performance is generally bounded by full finetuning performance, we first verify that full finetuning shows clear improvements over the pretrained model for each candidate PL. This ensures our evaluation meaningfully assesses PEFT effectiveness rather than dataset limitations. Based on these criteria, we use Java, C++, PHP, and Swift.

Footnote 6: huggingface.co/datasets/rombdawg/MegaCodeTraining

BaselinesWe compare PLEX with the following approaches: 1) _LoRA_[14]: a popu

Figure 2: SantaCoder 1.1B was pre-trained on the Java, Python, and JavaScript subset of the Stack-v1.1.3 Accordingly, Java is a well-supported programming language, whereas PHP and C++ are underrepresented.

 

lar parameter-efficient fine-tuning method, which assumes a low-rank update of parameters. We focus on the Q,K,V attention matrices.7 2) _LT-SFT_(Ansell et al., 2022): an alternative parameterized fine-tuning method, which is supported by the lottery-ticket hypothesis. 3) _SoRA_(Ding et al., 2023): an efficient variant of LoRA which reduces the rank adaptively. 4) _P-Tuning_(Liu et al., 2021): prepending trainable prefix vectors to inputs.

Footnote 6: The code for this paper is publicly available at [https://github.com/facebook/safetransed/safet](https://github.com/facebook/safetransed/safet) 



#### 4.3.2 Visualization of Our Distinction

In this section, we visually analyze why PLEX is superior to LT-SFT or LoRA. Specifically, we investigate the case when LT-SFT underperforms, such as in C++ where it falls short of LoRA (see Table 1).

First, LT-SFT updates too few parameters in the dense layers, where most of the knowledge resides [56]. To delve deeper, we examine the heatmap of affected parameters across layers--embedding, attention (attn), layernorm, and MLP-- on the 22nd layer for comparison.

Figure 4 illustrates that LT-SFT (Figure 3(a)) updates fewer parameters in the attention and MLP layers (where dense layers are concentrated) and instead updates other layers, like the embedding layer. In contrast, PLEX (Figure 3(c)) prioritizes updates in dense layers, effectively targeting the knowledge stored in the language model [57].

Second, LoRA-variants densely affect the parameters, but they require low-rank assumption to do it in a parameter-efficient way, which is known to overlook critical information [89]. In contrast, PLEX is free of low-rank assumption, by achieving parameter-efficiency with the lottery-ticket hypothesis.

## 5 Conclusion

We studied an adaptive PEFT method using lottery tickets. We propose PLEX, which effectively adapts PEFT to any PL, whether well- or under-represented. We also introduce PLEX-E, a computation-efficient version of PLEX, which reduces the full fine-tuning cost during ticket selection, making our method applicable to large LMs.

Figure 4: Heatmap of affected parameters of layers in LT-SFT, LoRA-variant (SoRA), and PLEX. We investigate a SantaCoder-1.1B case study on C++, where LoRA outperforms LT-SFT. The analysis covers the embedding layer (_emb_), attention layer (_attn_), layermorm layer (_layernorm_), and multilayer perceptron (_mlp_).





## Acknowledgments

This work was partly supported by Samsung SDS. This work was also partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.RS-2021-II211343, Artificial Intelligence Graduate School Program (Seoul National University)]

## References

* L. Ben Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, et al. (2023)Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.03988. Cited by: SS5.
* A. Ansell, E. Ponti, A. Korhonen, and I. Vulic (2022)Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 1778-1796. External Links: Link Cited by: SS1, SS2.
* F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M. Yee, Y. Zi, C. Jane Anderson, M. Q. Feldman, et al. (2022)Multipl-e: a scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227. Cited by: SS1, SS2.
* F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M. Yee, Y. Zi, C. Jane Anderson, M. Q. Feldman, et al. (2023)Multiple-e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering. Cited by: SS1, SS2.
* M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillett, F. Petroski Such, D. Cummings, M. Plappert, F. Chantzisz, E. Barnes, A. Herbert-Voss, W. Hebgen, G. Aleschhol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba (2021)Evaluating large language models trained on code. CoRRabs/2107.03374. Cited by: SS1, SS2.
* T. Chen, J. Frankle, S. Chang, S. Liu, Y. Zhang, Z. Wang, and M. Carbin (2020)The lottery ticket hypothesis for pre-trained BERT networks. In Advances in Neural Information Processing Systems, Vol. 33, pp. 15834-15846. Curran Associates. External Links: Link Cited by: SS1, SS2.
* X. Chen, T. Chen, W. Chen, A. Hassan Awadallah, Z. Wang, and Yu Cheng (2023)DSEE: dually sparsity-embedded efficient tuning of pre-trained language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 8208-8222. External Links: Link Cited by: SS1, SS2.
* A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzman, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov (2020)Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 8440-8451. External Links: Link Cited by: SS1, SS2.
* N. Ding, X. Lv, Q. Wang, Y. Chen, B. Zhou, Z. Liu, and M. Sun (2023)Sparse low-rank adaptation of pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, Singapore, pp. 4133-4145. External Links: Link Cited by: SS1, SS2.
* J. Franklin and M. Carbin (2019)The lottery ticket hypothesis: finding sparse, trainable neural networks. In International Conference on Learning Representations, Cited by: SS1, SS2.
* The Rise of Code Intelligence. External Links: Link Cited by: SS1, SS2.
* J. Hu, yelong shen, Phillip Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022)LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: SS1, SS2.
* R. Li, L. Ben allal, Y. Zi, N. Muennighoff, D. Koecetkov, C. Mou, M. Marone, C. Akiki, J. LI, J. Chim, Q. Liu, E. Sheltonozhskii, T. Yue Zhuo, T. Wang, O. Dehaene, J. Lamy-Poirier, J. Monteiro, N. Gontier, M. Yee, L. Kumar Umapathi, J. Zhu, B. Lipkin, M. Oblokulov, Z. Wang, R. Murthy, J. Stillerman, S. Sanak Pal Patel, D. Abulkhanov, M. Zocca, M. Dey, Z. Zhang, U. Bhattacharyya, W. Yu, S. Luccioni, P. Villegas, F. Zhdanov, T. Lee, N. Timor, J. Ding, C. S. Schlesinger, H. Schoelkopf, J. Ebert, T. Dao, M. Mishra,

Alex Gu, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Munoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro Von Werra, and Harm de Vries. 2023. StarCoder: May the source be with you! _Transactions on Machine Learning Research_.
* Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, PoSen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Mallory, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmect Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition-level code generation with alpha-code. _Science_, 378(6624):1092-1097.
* Liu et al. (2021) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT Understands, Too.
* Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In _Advances in Neural Information Processing Systems_, volume 35, pages 17359-17372. Curran Associates, Inc.
* Nijkamp et al. (2023) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In _International Conference on Learning Representations_.
* Sanh et al. (2020) Victor Sanh, Thomas Wolf, and Alexander Rush. 2020. Movement pruning: Adaptive sparsity by fine-tuning. In _Advances in Neural Information Processing Systems_, volume 33, pages 20378-20389. Curran Associates, Inc.
* Yu et al. (2017) Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. 2017. On compressing deep models by low rank and sparse decomposition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 7370-7379.
* Zhuo et al. (2024) Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. 2024. Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models 



## Appendix A PLEX-E on SantaCoder-1.1B

Table 3 shows that PLEX-E shows comparable Pass@1 scores to LoRA and SoRA when applied to SantaCoder-1.1B. Note that PLEX-E still significantly improves in adapting to underrepresented PLs (PHP and C++) over the existing lottery-ticket based PEFT approach LT-SFT, though PLEX is computationally more efficient.





\begin{table}
\begin{tabular}{l r|c c c c c c c} \hline \hline \multirow{2}{*}{Method} & \multicolumn{3}{c|}{\(\Delta\) param.} & \multicolumn{3}{c}{Java} & \multicolumn{3}{c}{PHP} & \multicolumn{3}{c}{C++} \\  & size (\(\downarrow\)) & \(\mathcal{H}\) & \(\mathcal{M}\) & \(\mathcal{H}\) & \(\mathcal{M}\) & \(\mathcal{H}\) & \(\mathcal{M}\) \\ \hline Full FT & 4.2GB & 17.9 & 26.4 & **11.3** & 17.5 & 10.4 & **22.0** & 17.6 \\ LoRA & 205MB & 16.7 & 28.6 & 11.1 & 16.0 & 9.4 & 20.3 & 17.0 \\ SoRA & 162-199MB & 17.8 & 24.0 & 10.0 & **19.1** & 11.4 & 21.1 & 17.2 \\ LT-SFT & 194MB & **18.2** & **29.7** & 3.4 & 7.0 & 8.2 & 18.3 & 14.1 \\ PLEX-E & 194MB & 17.1 & 29.2 & 9.5 & 15.6 & 8.9 & 21.4 & 16.9 \\ PLEX & 194MB & **18.2** & **29.7** & 10.1 & 17.1 & **12.3** & 21.3 & **18.1** \\ \hline \hline \end{tabular}
\end{table}
Table 3: **SantaCoder-1.1B Pass@1 scores of various PEFT methods across diverse PLs on the HumanEval (\(\mathcal{H}\)) and MBPP (\(\mathcal{M}\)) benchmarks in MultiPL-E. \(\Delta\) param. signifies the size of trainable parameters. PLEX-E is a computationally efficient version of PLEX. Java is included in the pretraining corpus, while PHP and C++ (gray highlighted) are not. Best scores are in bold; second-best are underlined.**