CrossPL: Systematic Evaluation of Large Language Models for Cross Programming Language Interoperating Code Generation
Authors: zhanhang xiong
Dongxia Wang
Yuekang Li
Xinyuan An
Wenhai Wang
Venue: ICLR 2026 Poster (2026)
Keywords: cross programming language interactions
LLM based workflow
benchmark
code generation
Primary area: datasets and benchmarks

Abstract: Large language models (LLMs) have shown strong performance in single-language code generation, but how well they produce cross-programming-language (CPL) interoperating code, which is widely used in cross-platform and complex software systems, remains underexplored. Therefore, a benchmark for evaluating CPL interaction code generation is essential. However, Constructing such a benchmark is challenging owing to sparse interoperating code in real-world multi-programming-language projects, diverse Inter-process Communication (IPC) mechanisms, vast Foreign Function Interface (FFI) language pairs, and the difficulty of evaluation. To address this gap, we introduce CrossPL, the first benchmark for systematically assessing LLM performance of CPL code generation across two primary interoperation modes and 2534 tasks, specifically 1,982 IPC tasks spanning six languages and 522 Python–C FFI tasks. Its construction involved a review of CPL documentation, 156 finite state machines, and analysis of 19,169 multi-language GitHub repositories. Two LLM-based workflows are designed for automating the benchmark construction and evaluation, and assess 20 state-of-the-art LLMs. Results reveal clear limitations: the best model achieves only 19.5\% Pass@1 and 26.46\% Pass@5 on the FFI subset, in sharp contrast to the strong performance of these models on single-language benchmarks. These findings underscore the urgent need for improving LLMs regarding CPL interoperating code generation. The benchmark and code are available at https://github.com/newxzh/crosspl.

Published as a conference paper at ICLR 2026
CROSSPL: SYSTEMATIC EVALUATION OF LARGE LANGUAGE MODELS FOR CROSS PROGRAMMING LANGUAGE INTEROPERATING CODE GENERATION
Zhanhang Xiong Dongxia Wang 1.2. Yuekang Li
Xinyuan An
Wenhai Wang*
1 *Zhejiang University Huzhou Institute of Industrial Control Technology
The University of New South Wales
ABSTRACT
Large language models (LLMs) have shown strong performance in single- language code generation, but how well they produce russ-programming language (CPL) interoperating code, which is widely used in cross-platform and complex software systems, remains underexplored. Therefore, a benchmark for evaluating CPL interaction code generation is essential. However, Constructing such a benchmark is challenging owing to sparse interoperating code in real- world paall--pogrammingg language projects, diverse Inter-process Communica- tion (IPC) mechanisms, vast Foreign Function Interface(FFI) language pairs, and the difficulty of evaluation. To address this gap. we introduce CrossPL, the first benchmark for systematically assessing LLM performance of CPL code genera- tion across two primary interoperation modes and 2534 tasks, specifically 1,982 IPC tasks spanning six languages and 522 Python-C C FFI tasks. Its construction involved review of CPL documentation, 156 finite state machines, and analy- sis of 19,169 169 multi-language GitHub repositories. Two L.LM-based workflows are designed for automating the benchmark construction and evaluation, and as- sess 20 state-of-the-arr LLMs. Results reveal clear limitations: the best model achieves only 19.5% Pass@1 I and 26.46% @ on the FFI subset, in sharp contrast to the strong performance of these models on single-language bench- marks. These findings underscore the urgent need for improving LLMs regarding CPL interoperating code generation. The benchmark and code are available at https: tttp::///ggtuuuuuoee/ www.hhwww.oooooooeeeeeeeeeeeeeee com wttpc///wwwwww///e/eeeeeee http:///.....tttoo.....////////r
INTRODUCTION
Large Language Models (LLMs) have recently attracted substantial attention, showing impressive capabilities in natural language processing and code generation tasks (Jin et al. 2024 Lozhkov et al., 2024]. In particular, using LLMs for programming has emerged as a promising direction, with many software engineers already adopting them as auxiliary tools for software development (Aguiar et al. 2024
To evaluate the programming ability of LLMs, researchers have proposed a variety of benchmarks focusing on single programming language code generation (Chen et al. 2021 Austin et al., [02T] Cao et al.) 2024 Du al. 2024 Xue et al. 2024). On these tasks, LLMs demonstrate strong performance, particularly in generating syntactically valid and functionally correct code(Lozhkov
et al. 2024 Jin et al. 2024 Islam et 2024).
However, in modern software system development, the use of multiple programming languages (MPL) has become increasingly common, both in industry and open-source projects. Studies show that over 80% of software systems involve two or more programming languages (Tomassetti & Torchiano 2014). The rationale lies in exploiting the unique strengths of different languages to enhance performance, modularity, and scalability ( et al. 20244 Delorey et al.] [2007] Ray et al., 2014 Li et al. 2021: Grichi al., 2020). Consequently, it is crucial to extend evaluation beyond single e- language tasks and towards gooosppoorrnmiin--aagggge (CPL) settings.
"Corresponding author; dxwang@ edu.ccn

---

Published as a conference paper at ICLR 2026
Python Server side
Socket()
Bind()
Liseet)
Acceptt)
Recerve
Sumil
Catching
Close()
C++ Clist side
Connecti
Scndt)
Recelvet)
Closel)
+
Sockeet) and
db
Prepare C shared library
Load shared library with stypes
Define function protséype (argtsps and restype
Call C function
(b) Example Pythoa-C FVI (via Crype)
Figure I: Examples of CPL, interoperating (IPC and FFI)
Despite this demand, existing benchmarks for evaluating MPL-related code generation scarce. More importantly, the few available works (Zheng et al.) 2023 Cassamo et al. 2023 Xu et al.] all] (2024) primarily focus code translation across languages, rather than assessing L.Mss ability generate interoperating code across heterogeneous languages. This leaves a critical research gap: there is still benchmark specifically designed to evaluate LLMss capacity generate code involving CPL. collaboration.
In real-world MPL systems, such collaboration is typically enabled by two mechanisms: Inter- process Communication (IPC) and Foreign Function Interface (FFI) (Yang et al., 2024a Li et al.) 20226 ap. IPC facilitates communication between independent processes using techniques such as Sockets, gRPC, message queues, and Websocket, requiring careful handling of data serialization, synchronization, and protocol compliance to avoid deadlocks or message loss. FFI, in contrast, en- ables direct invocation of functions across languages within the same process by managing function signatures, type conversions, and memory layouts, where even minor mistakes may lead to unde- fined behavior (Yang et al. 2024b). Fig. provides representative examples of IPC and FFI in MPL systems
Constructing benchmark that incorporates both IPC and FFI poses several challenges: In real- world MPL software systems, CPL interaction code snippets are typically sparse and implicitly embedded within large codebases, making it difficult to accurately detect the boundaries of CPL interoperation. Manual collection is time-consuming and incurs substantial overhead. @ IPC en- compasses a wide range of mechanisms, including but not limited to pipe, message queue, gRPC, TCP, UDP, HTTP and Websocket, each exhibiting distinct characteristics, usage patterns, and pro- gramming paradigms (Yang et al. (2024a). This diversity adds complexity interface categorization and identification. the space of possible language combinations
@ Moreover, constructing an FFI benchmark is particularly challenging, because
vas Yang et al. 2024b; Li et al. 2022a 2023
Bae et al. 2019 Hwang et al.) [221]] Bai et al. (2088) making exhaustive coverage infeasible and necessitating careful selection of representative cases. MPL projects are typically large-scale, involving complex dependency environments, making direct code execution often infeasible.
To address these challenges in benchmark construction, we propose three core strategies: We de- signed two LLM-based workflows to support both the construction and evaluation of the benchmark. We conduct comprehensive interface analysis by reviewing official documentation of mainstream IPC technologies and designing 156 scenario-speciff finite state machines (FSMs) for real-world IPC pattern detection. @ As both Python and C are among the top ten most widely adopted pro- gramming languages, C offers exceptional performance and memory management, while Python is lauded for its simplicity and readability (Yang et al. 2024h). This combination provides access to a vast ecosystem of libraries and modules essential for tasks in machine learning and artificial intelligence (c.g., Numpy(Hamm (Harris et al. 2020), SciPy (Virtannn et al., 2020), Pytorch(Paszke et al.) 2019), Tensorflow (Abaii et al.) 2016), etc.). Consequently, we constructs a of FFI-related tasks specifically within the context of Python-C interoperation.
+ For IPC tasks, we employ the cor-
'Fig. I lillustrates representative examples for the mechanisms of CPL interoperability. The upper part depicts the workflow of IPC hetween Python and C++ using sockets, while the lower part shows the key steps for Python invoking low-level C functions via ctypes. HHigh-level languages (e.g., Python, Java, Ruby, Go) interact with C/C++ via FFI to leverage the high- nerformance capabilities of low-level modules.
2

---

Published as a conference paper at ICLR 2026
responding FSMs as static analysis tools to verify protocol compliance. For FFI tasks, focusing on Python-C as the target setting, we carefully construct controlled compilation and execution en- vironments together with assertion-based test cases to validate both the syntactic correctness and functional soundness of the L.M--genertee code.
Following these strategies, present CrossPL, the first benchmark for systematically evaluating LLM performance CPL interoperating code generation, comprising 2,534 tasks. It encompasses Iwo subsets: CrossPL-IPC with 1,982 tasks spanning six languages, and CrossPL-FFI with 522 Python-C tasks. Furthermore, we raise three key research questions (RQs) as follows:
RQ1: How effectively can LLMs generate complete IPC code covering key state transitions, and what are the eπror distributions in cases of failure?
RQ2: How effectively can LLMs generate FFI-based CPL interoperating code, and what are the error distributions in cases of fallure?
RQ3: How do model characteristics impact the CPL interoperating code generation performance of LLMs?
We systematically investigate these RQs and have the following key findings: LLMs tend to un- derperform on most tasks involving different programming languages, and their capabilities vary significantly across different subsets of programming languages; @ Similarly, LLMs show limited effectiveness in generating code for various IPC techniques, with substantial differences in perfor- mance depending the specific technique; ① The performance of LLMs on the CrossPL-FFI subset notably poor, with the best-performing model achieving only 19.50% at Pass@ and 26.46% at Pass @5. However, many LLMS have already surpassed 90% pass 1 single language code gen- B ! Recent ad advances in model thinking techniques do not significantly improve the accuracy of IPC code generation by LLMs and, in some cases, even perform worse than the base models. These findings reveal the significant inadequacy of existing LLMs in generating CPL interoperating code compared with their performance on single-lang laggu-la code generation.
eration benchmarks
Our main contributions can be summarized as:
CrossPL benchmark: To the best of our knowledge, we propose CrossPL as the first benchmark designed to evaluate LLMSs ability to generate CPL. interoperating code involving IPC and FFI. CrossPI. encompasses two subsets: CrossPI-IIPC with 1,982 tasks spanning six languages, and CrossPL-FFI with 522 Python-C tasks.
Methodology for Automated Benchmark Construction: We propose unified methodology for automated benchmark construction, combining FSM-based IPC interface characterization and LLM-based workflows. First, we designed 156 FSMs based on official CPL interface specifi- cations formally characterize IPC-based interaction interfaces. These FSMs enable automatic detection and extraction of IPC code snippets from real-world GitHub repositories. They also serve as a tool to evaluate LLMs' capability in generating CPL code under specific IPC scenarios. Second, we developed two LLM-based workflows: one leveraging FSMs to identify, extract, and validate IPC snippets, followed by instruction generation and performance evaluation; the other focusing constructing FFI tasks and systematically assessing LLMs' effectiveness FFI code generation.
Large-scale empirical study:
We evaluate 20 representative LLMs to answer the key question: whether LLMs can accurately generate cross- gross-language interoperating code. The findings highlight the need for more dedicated effort in this critical yet underexplored area.
RELATED WORK
Single Language Code Generation Benchmarks. Early benchmarks, such as BLEU (Rabinovich eet al., (2017) and CodeBLEU (Ren et al. 2020) focus on surface or structural similarity, but fail capture functional correctness. HvaanEval (Chen et al. 202I) introduces Pass for func- tional evaluation, which has become standard metric. However, these benchmarks primarily target
g1 gishub. 1o/leadezbcard h:ml) is open, con- tinuously updated ranking that evaluates and compares LLMSs single programming language code generation nerformance expanded and rigorous henchmarks.
EvalPlus leaderboard bottps://evalle
3

---

Published as a conference paper at ICLR 2026
Mining and medellng IPC MPI repositaries
CreePL--IPC
GO
JS
php
Reposituries
Preection leve
Eatradivr
Draluaton
SEPE repertery sdfection and aequiiition
FSM
FXM
11
JPC internctionn pallien aloination for FSM
Filie!
Clan-Retrcctr
MatricPuook
FSM
Enshed lnts grontt
CoceGeek
Generator
Condidore sample
Qwen
Add urmee indormation the grazpt
Pins
Evaltaaion
MatricPuunjk
Provide roquired dependlonde
Starod Oboect
Roneate Linse
Brackneak-OFF1
Foecate
Figure 2: Framework for CrossPL. construction and evaluation.
single-lange languae, function-level tasks, mostly in Python (e-g. MBPP(Austin et al. 2021) and Hua- manEval et al. 2021)) )). As evaluation expands, ClassEval (Du et al. (024) CodeAgentBench (hang et al. 2024a) assess class-level and repository-level performance. SWE-bench (?) evalu- ates real-world software engineering problem solving by requiring models to generate patches that correctly fix issues in large, open-source repositories. However, ClassEval, CodeAgentBench and SWE-bench are all still within Python. Domain -specff benchmarks in data science (Huang et al. 2024 Zhang et al. 2024b; Nejjar et al. 2025), embedded systems (Englhardt et al.) 2024), robotics (Hu al.) 2024), and other areas (Zhu al. 2025) are likewise restricted to single language in their respective fields.
Multilingual Code Generation Benchmarks. While early efforts focus single-language set- tings, recent work shifts toward multilingual code generation. Xu et al. (Zheng et al., (003) intro- duce HumanEval-X, extending evaluation five programming languages. Cassano et al. (Cassano et al.] (023) propose MultPL-EE, translating HumanEval and MBPFAustin et al. (2021) tasks into 18 languages analyze multilingual performance. CRUXEVAL-X (Xu al.] 2024) further assesses LLMss reasoning and generation across languages. HumanEval-XX (Peng et al. (024] expands this effort, linking 23 natural and 12 programming languages to create a large-scale multilingual bench- mark. Multi-SWE-bence (Zan et al. 2025) extends SWE-bench multilingual settings, but still evaluates bug fixing within single-languan projectsand few CPL interaction problems.
Although these benchmarks span multiple programming languages, they evaluate each language in isolation rather than testing CPL interoperability. As a result, it remains unclear whether LLMs can reliably generate interoperating CPL code. To the best of our knowledge, existing benchmark directly assesses the model capability of CPL interoperability.
3
METHODOLOGY
Fig 2 presents an overview of the core pipeline of this work, spanning from code analysis and col- lection code generation and evaluation. The framework consists of three main components: (1)
Mining and modeling IPC interoperating patterns from MPL repositories; (2) LLM-bassed workflow for CrossPL-IPC construction and evaluation; (3) LLM-based pipeline for CrossPL-FFI conssuution
and evaluation.
3.1 MINING AND MODELING IPC INTEROPERATING PATTERNS FROM MPL REPOSTTORIES
Repository Selection and Localization for CPL Analysis. We begin by curating 19169 MPL projects from GitHub, each with 1000-30000 stars and employing 2-5 programming languages. For

---

Published as a conference paper at ICLR 2026
each project repository, record the key metadata R (e.g., ID, name, star count, language set, description, timestamps) and store it in a structured CSV format. Using the provided clone URLs, we locally replicate all the repositories via Git to enable downstream analysis and extraction of IPC interoperating code.
Modeling CPL interoperating Patterns. The main advantages of FSMs in clear structure and logic make them well-suited for managing complex processes with distinct stages and transitions. They thus are particularly suitable for analyzing IPC workflows, where interactions typically involve well-defined states (e.g., initialization, data transmission, termination) and deterministic transitions between them. Fig. Dillustrates an example of modeling CPL interoperating using FSMs, a detailed explanation of this example is given in subsection A.
PolyFax (Li et al. 2022a) first proposes leveraging FSMs to identify and classify interoperating pat- terns in MPL projects. However, it defines only 8 IPC-related FSMs, whose state patterns are overly coarse, often contain few states, and rely on fuzzy matching. This can lead to misclassification, especially since the FSMs do not account for code comments, making them prone to incorrectly matching commented-out code. In this study, after analyzing extensive documentation and real- world MPL projects, we design suite of 156 FSMs. The detailed distinctions between the FSMs we designed and those developed in PolyFax are provided in the |subsection A.2
The entire workflow of IPC patterns modeling, CrossPL-IPC construction and evaluation is formal- ized in Algorithm I
3.2 LLM-BASED WORKFLOW OF CROSSPL-IPC CONSTRUCTION AND EVALUATION
This component aims to identify IPC interoperating instances across the repositories using the 156 FSMs, generate their corresponding instructions and also to append the validation FSM of the cor- responding task, for CrossPL-IPC construction. For each matched instance, we record the file path p, interaction type interaction technique name 6, programming language L, FSM identifier and key procedural steps K.
To reduce the manual load and the limitations of single LLM in handling complex tasks with long contextual dependencies, we develop a LLM-based workflow powered by the DeepSeek-V3 V3 (Liu al. 2024| to construct CrossPL-IPC. Input code instances identified by FSMS as containing IPC Techniques are first validated by the Judger to confirm IPC implementation and classify them as function-level or class-level. Subsequently, instances are processed by the corresponding Function Extractor or Class Extractor, which extract minimal, logically complete IPC code snippets guided by technique-specific descriptions and one-shot examples. Extracted snippets undergo FSM-based validation, with up to five extraction attempts at increasing temperature if validation fails. Validated snippets are then passed to the
Instructor to generate natural language task descriptions.
Though currently a common practic to ensure the effectiveness of such generated descriptions, we randomly sample and manually inspect a subset of them, confirming that they do clearly and accurately describe the intended tasks. Furthermore, the benchmark's 's task information, comprising metadata, code, and the generated instructions, are stored in structured JSON format.
Finally, We provide the instruction corresponding to each task in the benchmark to an LLM under evaluation, prompting it to generate the code for the respective functionality, Correctness is assessed by matching the generated code against the predefined FSMs to ensure protocol compliance. The use of FSMs is particularly advantageous here, as they not only offer formal and modular way to represent the state transition logic of IPC technologies but also effectively capture subtle protocol violations or misuse patterns in the generated code.
The entire workflow is formalized in Algorithm 2 Figs 9 present the prompts of all LLMs in the workflow; detailed information is provided in subsection A.3
'Selected for its state-of-the-art performance among open-source L.MSs availahle prior April 2025 and also its cost-effectiveeess.
'The feasibility of L.M--based code instruction generation has heen shown in Zhu et al. 2025 and Xie
2024
5

---

Published as a conference paper at ICLR 2026
3.3 LLM-BASED WORKFLOW OF CROSSPL-FFI CONSTRUCTION AND EVALUATION
Algorithm illustrates the construction of the CrossPL-FFI for Python-C external function calls. The underlying C code is sourced from the GNU Scientific Library (GSL) 2009), a widely used and self-contained library of mathematical and statistical functions. GSL is chosen because real-world Python-C FFI scenarios involve complex dependencies, making large-scale execution environments difficult to reproduce, whereas GSL provides stable, compilable C code suitable for benchmark construction.
The workflow begins by compiling the GSL library into shared object (.so) files using Autotools and Make, establishing the runtime environment. C source files are then cleaned and applied using an initial FFI prompt and an error-revision prompt. Execution of the candidate solution is performed the environment where the precompiled .so files are available for FFI calls; successful executions saved benchmark entries, while failures are iteratively refined via the LLM (powered by (Liis et al., 2024)). This approach ensures a scalable, reproducible, and controlled benchmark for assessing LLMs' ability to generate correct Python-C FFI code.
Additionally, key information from the canonical solution, including class names, function names, and parameter names, is incorporated into the "Instruction" field of the benchmark. To ensure the effectiveness of such generated descriptions, we randomly sample and manually inspect a subset of them, confirming that they clearly and accurately describe the intended tasks. Finally, these bench- mark entries are provided as tasks to the I.MS under evaluation. 'he outputs from the L.LMs are combined with automatically generated assertion test cases verify correctness, enabling system- atic execution and testing. This approach ensures a scalable, reproducible, and controlled benchmark for assessing LLMs' ability to generate correct Python-C FFI code. Figs TO present the prompts of all LLMs in the workflow; detailed information is provided in subsection A.4
3.4 STATISTICS OF CROSSPL. BENCHMARK
CrossPL comprises 2,534 tasks for systematically evaluating LLMs' CPL code generation perfor- mance. CrossPL encompasses two subsets: CrossPL-IPC with 1,982 tasks spanning six languages and seven IPC technologies, and CrossPL-FFI with 522 Python-C tasks. Figs. [4 16 present the staties of the CrossPL from different view. Additional details regarding Figs. [4] are provided in Appendix B Table and shows the detailed statistics on the number of samples retained after
each key filtering step CrossPL-IPC and CrossPL-FFI construction pipeline respectively.
EXPERIMENT
We design experiments answer three research questions:
How effectively can LLMs generate complete IPC code covering key state transitions, and what are the error distributions in cases of failure?
RQ1:
RQ2: How effectively error distributions in cases of failure?
LLMS generate FFI-based CPL interoperating code, and what the
RQ3: How do model characteristies impact the CPL interoperating code generation performance of LLMs?
4.1 EXPERIMENTAL SETTINGS
In our evaluation, we adopt the unbiased Pass@ metric (Chen et al. 2021 to precisely measure the functional correctness of code snippets generated by LLMs. Consistent with previous studies (Zhu et al.] 2025), we report both Pass@1 I and P results under zero-shot setting and use macro-averaging to caleulate the overall scores. For Pass@ I, we employ greedy decoding by setting the temperature to 0. For Pass @5, we set the temperature at 0.2 and top-p sampling at 0.95. For proprietary models and large-scale open-source models, we perform inference using their official
where the total number of elements, is the number of correct elements and & is the number of top-ranked predictions. set 10 in pass@5.
"pass @k Eprollens

---

Published as a conference paper at ICLR 2026
APIs. For open-source models such as LLaMA3--8 --AMMMA-------- 8b-Instruct (Grattafiori al. 2024), Gemma-7b Team et al. 2024b), CodeLLaMA-7b-Insst (Roziere et al. 2023), and CodeGemma- (Team
al.
2024a). deploy them locally using Ollama the NVIDIA RTX 3090 GPU.
4.2 IPC CODE GENERATION PERFORMANCE (RQ1)
Overall findings. Fig. 3a and Fig. 3b show that LLMS varying effectiveness in generating IPC code across different programming languages and IPC techniques. Generally, larger and domain- specialized models perform better. LLMs exhibit the best performance generating IPC code written in C++, while their performance is relatively weaker when generating IPC code based Go. This could be due to Go's reliance on struct-based abstractions, whereas LLMs are predominantly trained class-based languages, leading structural mismatches in Go's IPC code generation. Regarding IPC techniques. higher-level protocols such as gRPC perform best, while lower-level protocols like Pipe and HTTP exhibit weaker results due their minimal abstraction and system dependence. In terms of error distribution, Taking GPT-4o and Llama3-8h-instruet as examples, GPT-4o demonstrates relatively balanced state coverage, with errors primarily occurring in IPC protocol link configuration and data transmission setup. In contrast, Llama3-8b-instee is worse, encounters errors during library file configuration in many cases, preventing effective state machine matching and resulting in incomplete IPC code generation.
All experimental results related RQ1 are summarized in Tables 3 with detailed analyses pro- vided in the (subsection C.I Futhermore, a detailed case study of CrossPL-IPC is also provided in the subsection D.1
Answer to RQ1
LLMs generate IPC code with varying effectiveness, performing better with C++ and gRPC and worse with Go and low-level protocols like Pipe. Failures for models like GPT-4o often stem from protocol and data transmission setup, while -amma3--bbinnttuct frequently fails earlier at library configuration.
4.3
FFI CODE GENERATION PERFORMANCE (RQ2)
Overall findings. From Fig. and the Table 7 we find that LLMs perform unsatisfactorily in gen- erating Python-C FFI code. Even the strongest model, GPT-4o, achieves only 19.54% in Pass @ I, while Llama3-Sb-instrus performs worst with Pass of merely 0.77%. Although increasing the number of sampled outputs leads marginal improvements, the results remain far from satisfactory: GPT-4o reaches only 26.46% Pass and Llama3 -aiima----nnnntuee achieves 3.95%. To better under- stand these limitations, we conducted an error analysis of GPT-4o's outputs and categorized the failures into six types: (1) symbol resolution errors; (2) GSL runtime errors; (3) Python-level calling errors; (4) Nameeiror//nddefine symbols; (5) assertion/test failures; and (6) memory/crash errors. These error categories highlight the fundamental challenges that constrain the success rate of LLMs in generating reliable Python-C FFI code and underscore the need for more robust reasoning and domain-specific knowledge integration in future model development. We infer the main reasons are that LLMS have insufficient reasoning capability when generating FFI code, lack domain-specinf specific knowledge about CPL interoperating, and have limited understanding of low-level code. This sults in frequent symbol resolution failures, runtime errors, calling mistakes, undefined symbols, test assertion failures, and memory crash errors.
The detailed analysis of the data in Table 7 can found in subsection C.2 Detailed case studies and error distribution analysis for CrossPL-FFI are presented in subsection D.2 and subsection D.3)
Answer to RQ2
LLMs struggle with FFI-based CPL code generation (GPT-4o 19.54% Pass @ 8b-instruct < 1%), with failures dominated by symbol resolution, runtime, calling, and memory errors.
4.4
IMPACT OF MODEL CHARACTERISTICS (RQ3)
Our experiments with Qwen3 series models yield two main observations. First, think mode markedly improves performance on CrossPL-FFI tasks (Fig. 4c). indicating that explicit intermediate reason- ing helps manage the complexities of external library linking, type specification, and memory oper-

---

Published as a conference paper at ICLR 2026
Ceera ARE
ON
Dores 14h
Oweste:
Germaalt
Cemmert
Owerd Ecsdennoiiattttrd
(a) Radar Plot Pass@1 on CrossPL-IPC (pro- gramming languages view)
(b) Radar Plot of Pass@1 on CrossPL-IPC (IPC) technologies view)
25.00%
10.00%
20.00%
15.00%
10.00%
GPP40
orS4orain
demtl Svep
(LMM phter
Desplestrr
Germdt 201
Gennd R0b vip
Goved Adb
Geeek di
Garnd
demmmo 23
Gudeliete
Dovend croder Add Aostbrt
Creett coato Ziatstter
Colrtienntor 70
Zith
Casiet
1u spangs
(c) @ Pass@5 results various models on CrossPL-FFL.
Figure 3: Performance of LLMs CrossPL
ations inherent FFI. Second, such improvements are evident for CrossPL-IPC tasks (Fig. 4a 4b); in some cases, performance even degrades. This may be attributed to IPC's reliance on well- structured communication patterns (e.g., sockets, queues), which are sufficiently captured by the training corpus and do not benefit from additional reasoning steps. Overall, Qwen3 performs bet- ter on CrossPL-IPC, while think mode proves particularly beneficial reasoning-intenn intensive FFI scenarios. The detailed analysis of the data in Tables T0 can be found in subsection C.3
Answer to RQ3
Model characteristics, such think mode, improve performance reasoning-intensiny FFI tasks but have limited or even negative impact on IPC tasks, which rely well-structured communication patterns.
DISCUSSION
Effeets Model Scale in the Two Benchmark Pipelines: In the CrossPL-IPC benchmark con- struction pipeline, smaller models could in principle substitute for the "Judger" and "Instructor", since these components require shallow reasoning and few output tokens. However, smaller models exhibit significantly higher false- poseiioostive rates, which propagate through the pipeline and dramat- ically inflate unnecessary extraction attempts. For efficiency and precision, a stronger "Judger" remains preferable. The CrosPPL-FFI benchmark pipeline imposes much greater cognitive de-
8

---

Published as a conference paper at ICLR 2026
(a) Boxplot of Pass @ I performance of Qwen3 se-
(b) Boxplot of Pass @ performance of Qwen3 se- rics models across IPC techniques.
ries models across programming languages.
Pass@1 With think
Passg9 Without think
Qwen3-4b
Qwen33-8
Owen3-14b
Qwen3-
Owen3-326
Qwen3 (ainn3-23bbaae a22b
0.00%
2.00%
4.00%
6.00%
8.00%
10.00%
12.00%
14.00%
16.00%
18.00%
20.00%
(c) Performance of Qwen3 series models on CrossPL-FFI (Pass@ 1)
Figure Qwen3 series models on both CrossPL-IPC and CrossPL-FFL.
mands: analyzing low-level C code, generating tasks, producing reference solutions, and construct- ing assertion-based test cases. These steps benefit substantially from larger models with stronger instruction following and symbolic reasoning capabilities.
We conducted ablation studies to quantify the contribution of model scale. "Extractor" -onlyy replacement: Replacing the "Extractor" with Qwen3-4B reduces the number of valid samples from 1.982 to 1,723 (-259). @"Instructor"--nn
replacement: replacing only the Instructor with Qwen3-
4B reduces samples from 1,982 to 1,890 (-72). "Extractor" replacement: Using Qwen3-4B as the "Extractor" causes drastic drop in valid FFI samples: 522 to 65, confirming that high-quality FFI construction requires a strong. stable model. These findings clarify the differential roles of model scale across IPC and FFI pipelines. Further analysis be found in C.4
Risks of Data Leakage in CPL Code Generation Benchmarks: Because CrossPL-IPC samples extracted from public GitHub repositories and For the CrossPL-FFI subset, the primary risk of data leakage arises from the fact that Python-GSL. interactions are already implemented in existing open-source libraries (e.g.. pygs().
we analyzed potential data leakage through two experiments. 1 Temporal leakage in CrossPL-IPPC:
CrossPL-IPC samples were grouped by creation year: before 2022, 2022, 2023, and 2024-2025. The group of 2024-2025 contains 1.377 samples (69.48%). We evaluated Pass @ using greedy and Figure 20 there is no evidence that models systematically per- form better newer samples, indicating minimal temporal leakage. @Training-dar data contamination in CrossPL-FFI: The FFI pipeline designed avoid exposure to existing code: the model receives an isolated C function and must independently generate the task description, ctypes wrapper, and reference solution. We further computed lexical overlap between our prompts and the pygs/ code- base used for verification. The overlap is only 0.61% % for function names and 0.99% for class names, indicating negligible contamination risk. Together, the two analyses show that data leakage does not meaningfully affect model performance. Further analysis can be found in
decoding. As shown in Table and
Effects of Sampling Temperature and top-p on CPL Code Generation: We examined the effect of temperature and top-p across eight locally deployable models the CrossPL-IPC and CrossPL- FFI subsets (Tables [6 19). There are three key observations: Both sampling parameters affect model behavior, yet nonlinear. Performance varies irregularly across settings, and the optimal con-

---

Published as a conference paper at ICLR 2026
figuration differs by model. @ IPC tasks are largely insensitive to sampling variation. Their inter- action patterns follow fixed communication protocols and deterministic state transitions, yielding narrow solution space where stochasticity provides little benefit. Consequently, temperature and top-p adjustments lead to only marginal performance shifts. @ FFI tasks remain fundamentally constrained by low-level correctness. FFI requires strict adherence to function signatures, memory layouts and type coercions; minor deviations result in undefined behavior. Higher sampling diver- sity increases the likelihood of violating these constraints, while more conservative sampling does not substantively improve reliability. This explains the consistently large performance gap-for example, Qwen3-4B averages 55. 24% on IPC but only 3.08% on FFI across temperatures, and Qwen2.5-coder- shows 68.05% on IPC but only 3.89% across top-p.
Overall, sampling strategies alone cannot overcome the intrinsic difficulty of FFI generation. Sub- stantial improvements will likely require stronger low-level reasoning ability, expanded CPL. training data, and dedicated finetuning on FFI-oriented interactions. Further analysis can be found in C.6
Effects of Think Mode on CPL code generation Quality: Section 4.4 shows that think mode of- fers limited benefit for IPC code generation and can even degrade performance, while substantially improving FFI tasks. This contrast stems from task characteristics: IPC tasks follow rigid procedural patterns with a narrow solution space, where additional reasoning may introduce unnecessary devi- ations. FFI tasks, by comparison, demand multi-step reasoning over function signatures, memory layout, and type semantics, making think mode inherently advantageous. Appendix D.4 provides a representative failure example of think mode in IPC tasks.
LIMITATIONS AND FUTURE WORK
Limitations. While this is the first benchmark tailored for assessing LLMs in CPL code generation, it has limitations: (1) The distribution programming languages and IPC techniques in open-source projects is uneven, leading to limited samples in some sub-tasks; (2) To align with experimental settings in related work, we did not explore the impact of hyperparameter configurations on perfor- mance; (3) Since the Qwen3 series models encompass different parameter scales, selected this series to evaluate the impact of think mode, while not extending the test to other models.
Future Work. Despite progress in LLM-based CPL code generation, our findings reveal several limitations and suggest future directions. First, LLMs exhibit suboptimal performance on FFI tasks due to insufficient reasoning and limited domain-specift, knowledge about low-level code and li- brary linking, leading to symbol resolution failures, runtime errors, and memory crashes. Enhanc- ing models with explicit reasoning mechanisms, domain knowledge integration, and library-aware representations could mitigate these issues. Second, IPC code generation remains uneven across lan- guages and protocols: models perform well on C++ and high-level protocols like gRPC but struggle with Go and low-level mechanisms such as Pipe. Future work may explore specialized pretraining or fine-tuning strategies that strengthen L.MSs' ability in underrepresented languages and system- level IPC patterns. Third, while model scale generally improves performance, gains are not uniform across tasks, indicating that architectural innovations or task-specific reasoning strategies may be required. Finally, systematic evaluation frameworks and benchmark expansion, including more di- verse CPL projects, protocols, and library interactions, are essential to better understand LLMs' strengths and limitations to guide their development toward robust CPL interoperability.
CONCLUSION
CPL interoperation code is prevalent in MPL projects and is essential for their construction. How- ever, current LLMSs often fail to generate such code accurately, with performance being highly sen- sitive to both programming languages and interaction techniques. In particular, LLMs perform especially poorly on FFI tasks, where challenges such as external library linking, symbol resolu- tion, and memory management frequently lead to compilation model's 's thinking mode provides limited benefit and may even negatively impact CPL code genera- tion. Enhancing the ability of LLMs generate CPL interaction code, and especially to handle the complexities of FFI, will be critical for scaling up their application to the generation of large-scale MPL software systems in future work.
runtime failures. Moreover, the
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by the Huzhou Institute of Industrial Control Technology, China (Grant No. K-ZY-2024-009).
ETHICS STATEMENT
CrossPL. is collected entirely from public repositories whose licenses permit software use, and our contributions fully comply with these license terms. During the collection or evaluation process, do not collect any information about GitHub users, and CrossPL task instances only use GitHub data available via the public API and website. Our work does not involve any human subjects. CrossPL does not implicitly explicitly use any discriminatory biased heuristics in the selection of repositories.
REPRODUCIBILITY STATEMENT
We provide complete code and configuration files to fully reproduce all experiments. In |section 3 describe in detail the data collection process and evaluation methodology. In |section 4| present the full experimental settings. The code repository also includes scripts to regenerate all reported figures and results in the paper.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, el. Tensorflow: asystem asystemforlarge orlarge- scale machine learning. In /2th USENIX symposium on operating systems design and implemen- tation (OSDI 16), pp. 265-283, 2016.
Lucas Aguiar, Matheus Paixao, Rafael Carmo, Edson Soares, Antonio Leal, Matheus Freitas, and Eliakim Gama. Multi- Multi-language software development in the Ilm era: Insights from practitioners' conversations with chatgpt. In Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 489-495, 2024.
Zhipu AI. Glm-4 model api documentation. -moee//gmm 2024. Accessed: May 19, 2025.
ht tetps///im /ttp://pen......
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.00 07732, 2021.
Sora Bae, Sungho Lee, and Sukyoung Ryu. Towards understanding and reasoning about android interoperations. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 223-233. IEEE, 2019.
Junyang Bai, Weiping Wang. Yan Qin, Shigeng Zhang, Jianxin Wang, and Yi Pan. Bridgetaint: bi-directional dynamic taint tracking method for javascript bridges in android hybrid applications. IEEE Transactions Information Forensics and Security, (433):6776692 2018.
Jialun Cao, Zhiyong Chen, Jiarong Wu, Shing Chi Cheung, and Chang Xu. Javabench: A bench- mark object-oriented code generation for evaluating large language models. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pp. 870-882, 2024.
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl- e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions Software Engineering, 49(7):3675-366) 2023.

---

Published as a conference paper at ICLR 2026
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374. 2021.
Daniel P Delorey, Charles D Knutson, and Christophe Giraud-Carrier Programming language trends in open source development: An evaluation using data from all production phase sourceforge projects. In Second International Workshop on Public Data about Software Development (WoP- DaSD 07), pp. 1-5, 2007.
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engi- neering, pp. 1-13, 2024.
Zachary Englhardt, Richard Li, Dilini Nissanka, Zhihan Zhang, Girish Narayanswamy, Joseph Breda, Xin Liu, Shwetak Patel, and Vikram Iyer. Exploring and characterizing large language models for embedded system development and debugging. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pp. 1 1-9, 2024.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng. Hanlin Zhao, al. Chatglm: family of large language models from glm- 130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
Brian Gough. GNU scientific library reference manual. Network Theory Ltd., 2009.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama 3 herd of models. arXiv preprint arXi:2440.22178. 2024.
Manel Grichi, Ellis E Eghan, and Bram Adams. On the impact of multi-languaae development in machine learning frameworks. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp.
IEEE, 2020.
Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. Array program- ming with numpy. nature, 585(7825)3357-35 2020.
Zichao Hu, Francesca Lucchetti, Claire Schlesinger, Yash Saxena, Anders Freeman, Sadanand Modak, Arjun Guha, and Joydeep Biswas. Deploying and evaluating ilms to program service mobile robots. IEEE Robotics and Automation Letters, (3)) 2024.
Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, al. Da-code: Agent data science code generation benchmark for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13487-13521, 2024.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang. Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409 12186, 2024.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Sungjac Hwang, Sungho Lee, Jihoon Kim, and Sukyoung Ryu. Justgen: Effective test generation for unspecified jni behaviors jyms. In 202/ IEEE/ACM 43rd International Conference Software Engineering (ICSE), pp. 1708-1718. IEEE, 2021.
Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint 1rXiv:2405.11403 2024.
Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. From llms to llm- based agents for software engineering: A survey of current, challenges and future. arXiv preprint arXiv:2408. 02479, 2024.
12

---

Published as a conference paper at ICLR 2026
Wen Li, Na Meng, Li Li, and Haipeng Cai. Understanding language selection in multi-language software projects on github. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings VCCSE-Companions pp. 256-257. IEEE, 2021.
Wen Li, Li Li, and Haipeng Cai. Polyfax: A toolkit for characterizing multi- lanti-language software. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1662-1666, 2022a.
Wen Li, Jiang Ming, Xiapu Luo, and Haipeng Cai. Polycruise: A cross-language dynamic informa- tion flow analysis. In 31st USENIX Security Symposium (USENIX Security 22), pp. 2513-2530, 2022b.
Wen Li, Jinyang Ruan, Guangbei Yi, Long Cheng, Xiapu Luo, and Haipeng Cai. Polyfuzz: Holistic greybox fuzzing of multi- luntuila systems. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 1379-1396, 2023.
Wen Li, Austin Marino, Haoran Yang. Na Meng, Li Li, and Haipeng Cai. How multilingual systems constructed: Characterizing language use and selection in open-source multilingual soft- ware. ACM Transactions Software Engineering and Methodology, 33(3):1 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:24/2.19337 2024.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint arXiv:2402.19177 2024.
Mohamed Nejjar, Luca Zacharias, Fabian Stichle, and Ingo Weber. Llms for science: Usage for code generation and data analysis. Journal of Software: Evolution and Process, 37(1):e2723, 2025.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Qiwei Peng, Yekun Chai, and Xuhong Li. Humaneval-xl: A multilingual code generation bench- mark for cross-lingual natural language generalization. Proceedings of the 2024 Joint Inter- national Conference Computational Linguistics, Language Resources and Evaluation (LREC- COLING 2024), pp. 8383-8394, 2024.
Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract syntax networks for code generation and semantic parsing. arXiv preprint arXiv:1704.07533, 2017.
Baishakhi Ray, Daryl Posnett, Vladimir Filkov, and Premkumar Devanbu. A large scale study of programming languages and code quality github. Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pp. 155-165, 2014.
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: method for automatic evaluation of code synthesis. arXiv preprini 2009 10297, 2020.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code laaa:: Open foundation models for code. arXiv preprini
2tt.::2208...2111 2023.
CodeGemma Team, Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A Choquette- Choo. Jingyue Shen, Joe Kelley, et al. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406. /409, 2024a.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology, 2024. arXiv preprint arXiv: 08835 2024b.
13

---

Published as a conference paper at ICLR 2026
Federico Tomassetti and Marco Torchiano. An empirical assessment of polyglot-ism in github. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, pp. 1-4, 2014.
Pauli Virtanen, Ralf Gommers, Travis Oliphant, Matt Haberland, Tyler Reddy, David Coumapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: funda- mental algorithms for scientific computing python. Nature methods, 17(3):261-272, 2020.
Yiqing Xie, Alex Xie, Divyanshu Sheth, Pengfei Liu, Daniel Fried, and Carolyn Rose. Codebenchgen: Creating scalable execution-based code generation benchmarks. arXiv preprint arXiv:2404.00566 2024.
Ruiyang Xu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, and Le Sun. Cruxeval-x: A benchmark for multilingual code reasoning, understanding and execution. arXiv preprint arXiv:2408. 13001, 2024.
Pengyu Xue, Linhao Wu, Chengyi Wang, Xiang Li, Zhen Yang, Ruikai Jin, Yuxiang Zhang, Jia Li, Yifei Pei, Zhaoyan Shen, et al. Escalating llm-based code translation benchmarking into the class-level era. arXiv preprint arXiv:2411.06144 2024.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Haoran Yang, Yu Nong, Shaowei Wang, and Haipeng Cai. Multi-language software development: Issues, challenges, and solutions. IEEE Transactions on Software Engineering, 50(3):512-553 2024a.
Haoran Yang, Yu Nong, Tao Zhang, Xiapu Luo, and Haipeng Cai. Learning to detect and localize multilingual bugs. Proceedings of the ACM on Software Engineering, 1(FSE))2190-2211 2024b.
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving. arXiv preprint wrXiv:2504.02605 2025.
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13643-13658, 2024a.
Yuge Zhang, Qiyang Jiang. XingyuHan XingyuHan, Nan Chen, Yuqing Yang, and Kan Ren. Bench- marking data science agents. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5677-5700, 2024b.
Qinkai Zheng. Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue. Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilin- Proceedings of the 29th ACM SIGKDD Conference on 2023.
gual benchmarking on humaneval-x. Knowledge Discovery and Data Mining, pp.
Qiming Zhu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, and Shing-Chi Cheung. Domaineval: An auto-constructed benchmark for multi-domain code generation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 26148-26156, 2025.
A ALGORITHM
A.1 INTRODUCTION OF FSMS
As described in subsection 3.1 Fig. illustrates an example of modeling CPL interoperating using FSMs, specifically modeling typical client-server interactions based the java.net. The upper part of the diagram models the server-side workflow, beginning with the import of networking libraries, followed by creating ServerSocket, accepting connections, establishing input or output streams,
14

---

Published as a conference paper at ICLR 2026
and eventually closing the Socket. The lower part part represents the client-side process, including socket creation, connection initiation, stream handling, and termination. These workflows converge at the data exchange stage, representing the communication interface. This example demonstrates how FSMs can model API usage patterns real-world code, enabling structured and interpretable modeling of communication logic.
java.net server-side
New Server
getiaputStrean
accept
BerOutpurStream
$2
S3
import java.net.
close
SO
SI
S6
S7
New Socket
getinputStreant
connect
ge1OuiputStrcam
$4
SS
jav client-side
Figure 5: An example of FSM-modeled CPL interoperating.
A.2 FSMS DIFFERENCES BETWEEN THIS PAPER AND POLYFAX
Compared to that of PolyFax, our models offer far greater state and transition granularity, enabling accurate capture of each critical step in diverse CPL interoperating scenarios. Furthermore, FSMs include three key enhancements:
(1) Adding semantic descriptions for each key step of CPL interoperating code, providing contex- tual semantics for the LLMs (like for instruction generation) in benchmark construction work- flow.
(2) Increasing the number of states explicitly cover finer-grained IPC-specific steps, and design- ing clearer transition logic to reduce ambiguity in pattern detection.
(3) Skipping code comments avoid false positives from comment text matching critical step pat- terns.
A.3 LLM-BASED WORKELOW OF CROSSPL-IPC CONSTRUCTION AND EVALUATION
To ensure the reproducibility and transparency of our benchmark pipeline, we provide the exact prompt templates employed across all LLMSs in the CrossPL-IPC construction and evaluation work- flow. As described in subsection 3.2 Figs. 6
provide the detailed prompt information.
A.4 LLM-BASED WORKFLOW OF CROSSPL-FFI CONSTRUCTION AND EVALUATION
To ensure the reproducibility and transparency of our benchmark pipeline, we provide the exact prompt templates employed across all LLMs in the CrossPL-FFI construction and evaluation work- provide the detailed prompt information.
flow. As described subsection 3.3 Figs. I0
B BENCHMARK STATIC
B.1 DETAILED INFORMATION OF BENCHMARK
In constructing CrossPL-IPC benchmark. conducted thorough search and review of official documentation related to IPC technologics in CPL projects using keywords such as gRPC, Pipe, message queue, TCP, UDP, WebSocket, and HTTP across different programming languages.
Fig. 14a summarizes the distribution in CrossPL-IPC from different perspectives. Overall covers six programming languages and seven IPC technologies, comprising total of 1982 tasks. Among the programming languages, Java accounts for the highest proportion of IPC-related tasks with 615 instances (31. (31.03%) whereas C++ the fewest, with 51 tasks (2.57%). Among IPC technologies,
15

---

Published as a conference paper at ICLR 2026
Algorithm 1: IPC code analysis, extraction, generation and evaluation
Input
Wmin Wmax project star count bounds;
Lmnins Limma number of languages per project bounds;
P: all public GitHub repositories within bounds;
FSM: A list of FSMs.
Output:
True_List: LLM's correct task information list;
False LLM's wrong task information list
Step MPL Repositories Crawling
R & Crawler(P Wmin Wmax Liminn Lmmax)
save R CSV
Record metadata of satisfied MPL projects
Step Clone Local
foreach E R do
local dêr -- ramnmme + +_+ Clone(r;,CloneUri,, local_dir)
Clone to local with specific directory name
Step 3: FSM-based Interoperability Analysis
10 F
11
foreach E R do
12
foreach FSM, FSM do
13
if (r.) then
(pi, T, 02 T4, os K.) (SM,(r)
& Algorithm of FSM can find in Li et al. 2022a
Faappend((pi Tú, Oi, Ki)) break
16
18
else
19
continue
Step 4: Benchmark Construction (Algorithm) 2
Step 5: Benchmark Evaluation
22
foreach B, E do
code < (LM(I)
24
V + FSM(()
if (code) then
26
27
else
Folse List.oppend(
16

---

Published as a conference paper at ICLR 2026
Algorithm 2: CrossPL-IPPC Construction
Input
F {(p,, T,, 0, Li, o, k metadata of CPL interoperating instances detected by FSMs;
N: number of IPC instances identified by the FSMs;
P:: file path of instance ;;
T5 interaction type:
0: technique used;
Ly programming language;
0e FSM ID;
K2 key steps of IPC.
Output: IPC benchmark set B saved in JSON format.
Tid +-
D. Initialize task identifier
foreach (pi, Tr, 0, Li, oi, K;) E do
C 5 1..-reddpp read
D. Read the code file of the instance Judger
cl, + A1(1,, (C,, Ts, 0g Li, Ki)
if cl, == null then
continue
else
R, select reference Ci, cl:)
Select the required one-shot example
for k 5o do
11
f f 0.1
D Set LLM temperature
C + A2 Te, 0, Li, Ki Ri c4,
13
D Code extracter
if As(ck, σi) then
FSM-based validation
I4 An(ck, Ts. Ki)
Instructor
B₁ (TidsPe, Tú, O, Li, oe Ki, cl,, La ck)
Save(Bt, save sat, Tia) Tia f- Tia +
break
17

---

Published as a conference paper at ICLR 2026
Prompt template for "Judger"
You are senior software engineer with deep expertise multi -language interoperability and inter-process communication (IPC) technologies.
Your task is to analyze the provided "Source code", which written (PI.).
Please determine whether the LLM accurately extract either function -level code, class- code, which contains a complete implementation the { IPC Name} BechnqqeeJudgmeeee rules are as follows: 1.According to the "Source code" and the key steps of the inter -process communication technology, if the key steps of using this inter-process communication technology can be completed in one function, extract the function-leve code;
2. According to the "Souree code" and the key steps of the inter -process communication technology, if there are functions classes, and the key steps be completed solely within the script -level code, please extract the code encapsulate code into function -leee code;
3. According to the "Souree code" and the key steps of the inter -process communication technology, if the inter-
process communication technique requires multiple functions work together and involves defined classes, directly extract the glass-lever code:
4. According to the "Souree code" and the key steps of the inter -process communication technology, if the inter- process communication technique requires multiple functions work together, but the source code does define any classes, please extract these functions and encapsulate these functions into class -level code: 5. According to the "Source code" and the key steps of the inter mprccess communication technology, if you are absolutely sure that you cannot extract suitable class -level code or function-level code from the source code to accurately implement the key steps the inter-proces -process communication technology, nothing will be extracted and "null" will be returned.
The key steps required for a valid IIPC Name) implementation are: {Steps}
Source code: (Original code]
Your judgment is: (Your answer only contain one the following: "Function -leve!" "Class-level" "null". Please don't include any other characters, and do not give me the reason for your judgment.)
Figure 6: Prompt template for "Judger"
18

---

Published as a conference paper at ICLR 2026
Function-level code snippet extraction prompt template for "Function EEttractor
Task description:
Based on the original code and information related the original code, extract the expected function -level code multi-proge programriggg language interaction from the original code.
Requirements:
1. Only extract function-leva code involving multi-pree programming language interaction in the original code, including function name, parameters, return value and core logic.
2. According to the original code, if there
functions classes, and the key steps can be completed solely within the script-level code, please extract the code and encapsulate the code into function -level code, preserve the original semantics of the code as much as possible; do not introduce any content other than encapsulation
Extract all external library files and global variables that the function, and that these codes completely derived from the original code.
4. Only extract lask-related code, and the extracted code must strictly contain the key steps of the inter -process communication technology.
The output result should follow the format "Output" the example, Do not include any characters other than code, including "Output", "Python", or "Here's the extracted...".
6. Please strictly follow the instructions above.
Instruct:
You advanced programmer who focuses multi -lnguagge interaction and sierr-proeess communication technology.
Please complete the task according to the following example.
Example:
Input:
Original code: (Reference code]
Information related to original code:
Programming language: Python
Inter-proces, communication technology name: TCP Server side by using socket in python The key steps this inter-pon process communication technology:
Step 1: Import the socket module
Step 2: Create TCP socket object using IPv4
Step Connect
socket to the target server's port
Step 4: Send data to or receive data from the server
Step 5: Close the connection with the server
Output:
[Reference cinertinceefuniiinnnnnoooo function_level levelleeeeoooe level_code}
Request:
Now please follow the task description and the example to extract one expected multi -language interaction function-level code from the original code based the input.
Input:
Original code: (Original_code]
Information related to the original code:
Programming language: (PL)
Inter- prec-process communication technology name: {IPC, Name)
The key steps of this inter-process communication technology: (Steps)
Figure Function-level code snippet extraction prompt template for Crannction-Extratttnn
19

---

Published as a conference paper at ICLR 2026
Class-level code snippet extraction prompt template for "Class -Extractor"
Task description:
Based the original code and information related the original code, extract the expected class -level code for multi-pne poolrairoggnninnggggg language interaction from the original code.
Requirements:
1. Only extract one flassl code involving multi mulli-ppoogamming language interaction in the original code, including Class name, all relevant functions, parameters, return values and core logic.
According the "Souree code" and the key steps of the inter mnter-process communication technology, if the inter- process communication technique requires multiple functions work together, but the source code does not define any classes, please extract these functions and encapsulate these functions into class -level code, preserve the original semantics of the code much possible; do introduce any content other than encapsulation.
3. Extract all external library files and global variables that use the class, and ensure that these codes are completely derived from the original code.
4. Only extract task nek-related code, and extracted code must strictly contain key steps of inter -process communication technology.
5. The output result should follow the format of "Output" in the example. Do not include any characters other than code, including "Output", "Python", m or "Here's extracted.....
6. Please strictly follow the instructions above.
Instruct:
You are an advanced programmer who facuses on multi -lanuuage interaction and inter-process communication technology.
Please complete the task according to following example.
Example:
Input:
Original code: [Reference raw code)
Information related to the original code:
Programming language: Python
Inter-process communication technology name: HTTP Server side by using FastAPl in python TTe key steps of this inter-process communication technology:
Step Import FastAPI and necessary components from the FastAPI module
Step 2: Instantiate the FastAPI application object, which will handle HTTP requests
Step 3: Define route handlers using decorators (e.g., get, post) for each HTTP method
Step 4: Optionally, define request and response models using Pydantic BaseModel validation
Step 5: Extract query parameters, path variables, and request body using FastAPl's parameter extraction
mechanisms
Step Return the response from the handler, typically as JSON using FastAPl's built in response
classes
Output:
[Reference class level code)
Request:
Now please follow the task description and the example to extract one expected multi -language interaction class-level code from the original code based on the input.
Input:
Original code: (Original code]
Information related to the original code:
Programming language: (PL.)
Inter-process communication technology name: {IPC) Name)
The key steps of this inter-procee inter-process communication technology: (Steps)
Figure 8: Class-level code snippet extraction prompt template for "Class- Extractor"
20

---

Published as a conference paper at ICLR 2026
Instruction generation prompt template "Instructor"
You senior programmer specializing in multilingual interaction and inter -process communication (IPC) technologies.
Please carefully analyze the "Reference code", which is written in (PL) and utilizes the [PC_Name) technique. The key steps of this technique include {Steps).
Reference code: [Canonical }
If the reference code is at the function level, Please write clear instruction detailing the function, including its purpose, input parameters (if any), and output (if any), guide LLM in generating reference code. Use the following format structure your instruction(Finally. please strictly follow the format requirements and do not insert any characters are not relevant to requirements.):
Task Description....
Function Description:...
Input...
Output:...
If the reference code is at the class level, please write a clear instruction detailing the class, including its purpose, attributes, and methods (with input parameters and return values). Ensure the instruction guides LLM generating reference code.
Use the following format structure your instruction (Finally, please strictly follow the format requirements and do not insert any characters are not relevant to requirements.):
Task Description
Class Description:....
Attributes:
Altributel: [Data Type] [Description]
Attribute2: [Data Type] [Description]
Methods:
Method1: [Name]] [[Input Parameters]] [Return Type] [Description]
Method2: [[mpu]([nnut Parumeters]) >> [Return Type] [Description]
Figure 9: Instruction generation prompt template for "Instructor"
21

---

Published as a conference paper at ICLR 2026
Algorithm 3: CrossPL-FFI Construction
Input
P (PI. P2. ): file paths of C source files;
M: number of C files collected from the source library (derived from GSL); Prompt.filey CrossPL-FFI generation prompt template:
Prompt.file:: error-revision prompt template.
Output:
FFI benchmark set saved JSON format.
Preprocessing:
Compile GSL source code into Linux shared object (.so) files using Autotools and Make .etc compile tools.
Prepare runtime environment before iteration b Initialize task identifier
Tia f-
foreach p. E P do
C < read without comments(p:) Instr Sol 2, Err <- Ø
Read C file and remove comments
for ls t 0 to do
if k = 0 then
Prompl + forma((rompttfile.. C₁)
11
else
Prompt + format(Prompt file2, C, Instr, Sol, Err)
13
Resp & A1 (Prompt)
b extraction
Result Parse( Resp)
Parse the response generated by the L.M
15
Result = Ø then
break
Instr Resutt] Sol + CaaunilCannoncall. solution"] if execute(Sol, so files) success then
18
29
b ".so files serve as indispensable dependencies required for the execution of candidate code."
21
B: fT "FFT", Instr, Sol)
22
(ave(B,, save sate Tia)
Taa Tia
break
25
else
Err execution message
22

---

Published as a conference paper at ICLR 2026
Prompt template for generating CrossPL.FFI
You given low-level function from open-source project (GNU Scientific Library). This function has been compiled into dynamic library (.so on Linux) that can be loaded in Python using the ctypes module. The dynamic library paths are as follows:
1. //sr//ocall/ss//iiiii httal////ibbbbbgslcllasss.
2. /usr//ocal/gs// httpg///s/////bll.ss.
Extra Knowledge:
libgsl.so: The main GNU Scientific Library, provides high -level numerical routines.
libgslcblas.sor GSL's built-in C BLAS library, implements low-level linear algebra functions (e.g., matrix/vector operations).
Relationship: libgsI.so depends on libgslcblas.se so; you must load libgslcblas.so first (or globally) to resolve BLAS symbols, otherwise get "undefined symbol" errors.
The recommended loading order follows: ctypes. CDLLC haass//lcal/gs//ll/liiiii gsl ctypes. CaaiscDDLL"""uses ("/uss//l/ooal gsl//ibblibbss..s.
so", cereecccyyes..T.... GLOOBA)
The source code of a function from the dynamic library shown below: {source_com ]
You can refer the following link definitions functions related to GNU Scientific Library (GSL): https://www.gnu. www.w/wwwww....oo.oo.mmmm.oo...e www.:oorr/sofwrree...o httpwarr/gs//o//hmmll/
task is design both an Instruction and a Canonical solution that evaluate an LLM's ability to write
correct Python code that calls this C function via ctypes.
Instruction should include:
A concise and clear natural language task description specifying the intended functionality(e.g. compute mathematical result, manipulate data structures, process arrays).
2. An explicit requirement that the solution must use ctypes to load and call the corresponding C function from the compiled dynamic library.
Well-defined imput and output specifications to ensure correctness can verified automatically.
Canonical solution should include:
complete correct Python solution that fulfills the instruction.
Several ussertion-bast based test cases to validate correctness, robustness, and error handling.
All assertion tests must be encupsulated within a function named assert test'.
The 'assert_ test" function must executed within the name main block.
Respond ONLY with valid JSON object. use triple quotes (""")) your response.
Wrap code a single string with escaped newlines ('n).
The returned must be dictionary only with the following fields: "Instruction";
"Canonical solution":
Figure 10: Prompt template for constructing CrossPL-FFI
23

---

Published as a conference paper at ICLR 2026
Prompt template with error information generating CrossPI-FFI
previously, an attempt was made to generate a Python Canonical_solution solution that calls this C function via ctypes. The dynamic library paths follows:
1. /usr//ocal/// htt///il/igslllllas...o
2. /usr/local/ catlocal//s//liiii httpg//i//ibbgll....
The previously generated Instruction and Canonical solution are as follows:
Instruction:
(Instruction)
Canonical solution: (Canonical_solutien
However, when executing the Canonical solution, the following error occurred: )
Your task is to revise and generate a corrected Python Canonical solution that fixes the above error while still satisfying the original Instruction. Ensure that new solution correctly handles memory allocation, type conversion, buffer sizes, any error codes or return values.
Canonical solution requirements:
1. A complete and correct Python solution that fulfills the Instruction.
2. Several assertion-based test cases validate correctness, robustness, and error handling,
All assertion tests must be encapsulated within function named assert test"..
4. The asssrt test" function must be executed within the 'if" name main block.
Respond ONLY
valid JSON object.
Do not use triple quotes ("""") in your response.
Wrap code in single string with escaped newlines ('In).
The returned content must be dictionary only with the following field: "Canonical "
Figure 11: Prompt template with error information for constructing CrossPL-FFI
Add class information to the Instruction
uccomplish the above task, required classes are: class names); ), and required functions along with their parameter lists are: {funetin_names).. names}.
The recommended dynamic library loading order follows:
ctypes. ctypes.CDLL CDLLLCDus///oca httpi////gg//iibbibblclcllss so", mode=ctypes.RTI. 1odd=ctyys.RTLL__LLBBA)L GLOBAL) gsl = etypett Cwtppes.CDLLLL"usslccclelc /tbpa//l
You must return ONLY a valid JSON object.
Do not include explanations, comments, or extra text.
not use triple quotes (""").
The JSON object must contain exactly one field:
"Candidate Standidate_solution solution". <string>>
Inside "Candidate solution", ".adidate_solutioon.. provide the complete code single string. Escape line breaks In.
Figure 12: Add class information the Instruction
24

---

Published as a conference paper at ICLR 2026
Add function information Instruction
To accomplish the above task, the required functions along with their parameter lists ure: functin names).
The recommended dynamic library loading order is as follows:
ctypes. Copes.CDLL(") CDLLC (ODLLL"/s/lloaalgg///aaa http://////b/bslllllsssssssa//e/ mode=ctypes.RTI RTLD GLOBAL)
gsl etypes. COw.CCCDDLLLLloeeoo /ttp///o/o//////////a gsl lbblissssssss
You must return ONLY valid JSON object.
not include explanations, comments, or extra text. Do not use triple quotes ("""))
The JSON object must contain exactly one field: "Candidate "Candidate_solutioon" solution": <string>
Inside "Candidate solution", provide the complete code single string.
Escape all line breaks as In.
Figure 13: Add function information to the Instruction
HTTP accounts for the highest proportion of tasks with 779 (39.30%), while UDP for the lowest with 92 tasks (4.64%).
Fig. 14b] further details the distribution of IPC technologies used by different programming lan- guages within the CrossPL-IPC. Due to the distinct characteristies of each language, the proportional use of IPC technologies varies considerably; for example, Java tasks are predominantly associated with TCP, while JavaScript tasks mostly related to HTTP. Notably, some IPC techniques cannot be fully implemented within a single function and require implementation at the class level. Con- sequently, during IPC-related code extraction, distinguished between class-level and function- level code. Overall, class-level code accounts for 59.99% of the CrossPI--IPC, while function-level code constitutes 40.01 %. This distribution reflects language-specific design patterns: for example, class-level implementations dominate in Java and PHP due to their object-oriented paradigms, while languages like Go and JavaScript tend to favor function-level or lightweight constructs. Such vari- ations emphasize the necessity of handling both granularities in CrossPL-IPC to faithfully capture real-wor weal-world IPC usage across different programming ecosystems.
Fig. 15 presents the distribution of code line counts in the canonical solutions across CrossPL. Fig. 15a presents the distribution of Canonical solution Code Lines of CrossPL-IPPC Fig. T5b presents the distribution of Canonical solution Code Lines of CrossPL-FFII, Specifically, the code line the CrossPL-IPC exhibits a median of 51 lines, an average of 66.96 lines, and maximum of 483 lines. In contrast, the CrossPL-FFI shows more compact distribution, with median of 44 lines, an average of 47.1 lines, a maximum of 112 lines, and a minimum lines.
Fig. 16 illustrates the distribution of instruction lengths, measured in characters, across the canon- ical solutions of CrossPL. Fig. 16a shows the distribution of instruction length (characters) of CrossPL-IPC. Fig. presents the distribution of instruction length (characters) of CrossPL-FFL. For the CrossPL-IIPC, the instructions have a median length of 1204 characters, an average of 1284.64 characters, and minimum of 473 characters. In comparison, the length of instruction in CrossPL-FFI shows a slightly larger scale, with a median of 1277.5 characters, an average of 1329.74 characters, a maximum of 2421 characters, and a minimum of 973 characters.
B.2 DETAILED STATICS OF INSTANCES FILTERED AT EACH KEY STEP
To enhance transparency and ensure methodological reproducibility, we provide detailed statistics on the number of samples retained after each key filtering step in the pipelines used to construct CrossPL-IPC and CrossPL-FFL. Table I shows the detailed statistics on the number of samples re- tained after each key filtering step in CrossPL-IPC construction pipeline. Table 2 shows the detailed statistics on the number of samples retained after each key filtering step in CrossPL- construction pipeline.
2
25

---

Published as a conference paper at ICLR 2026
Java Python
JavaScript
WebSocket
Message Queue Function-level Class-level
PHP
TCP UDP
Pipe gRPC
Go
C++
13.67%
6.51%
7.62%
7.72%
9.99%
2.57%
9.99%
19.78%
59.99%
59.99%
4.64%
40.01%
40.01%
31.03%
39.30%
21.85%
25.33%
(a) Pic chart of CrossPL-IPPC dataset from different view.
615
HTTP
gRPC
600
54
TCP
Message Queue
117
UDP
Class-level
502
Websocket
Function-level
500
19
52
Pipe
II
36
392
400
38
232
:14
Count
11
91
561
102
271
30
259
29
18
30
200
81
177
151
270
31
242
53
202
96
133
121
94
112
98
51
F18
220
33
0
Java
Python
JavaScript Programming Languages
Go
PHP
(b) Distribution of different IPC technologies across different programming languages.
Figure 14: Distribution of CrossPL-IPC dataset from different view.
26

---

Published as a conference paper at ICLR 2026
Distribution of Canonical solution Code Lines
250
200
Number of Tasks
150
100
50
100
200
300
400
500
Code lines of CrossPL-IPC
Distribution of Canonical solution Code Lines of CrossPI.IIPC.
Distribution of Canonical solution Code Lines
35
25
Number of Tasks
20
15
20
40
60
80
100
Code lines of CrossPL-FFI
(b) Distribution of Canonical solution Code Lines of CrossPL-FFL.
Figure 15: Distribution of Canonical solution Code Lines of CrossPL-IPC and CrossPL-FFL.
27

---

Published as a conference paper at ICLR 2026
Distribution of Instruction Lengths
250
200
Number of Instructions
100
50
1000
2000
3000
4000
Instruction length (characters)
(a) Distribution of instruction length (characters) of CrossPL-IPC.
Distribution of Instruction Lengths
100
80
Number of Instructions
60
40
20
0
1000
1200
1400
1600
1800
2000
2200
2400
Instruction length (characters)
(b) Distribution of instruction length (characters) of CrossPL-FFL.
Figure 16: Distribution of instruction length (characters) of CrossPL-IPC and CrossPL-FFL.
28

---

Effectiveness of code models. Comparing models deliver comparable or even superior performance despite fewer parameters, emphasizing the effectiveness of LLMs' targeted domain adaptation. Analysis of Table 3 indicates that, given comparable parameter sizes, code-specialized models tend to achieve higher performance. Taking outperforms Qwen3-32B by substantial Published as a conference paper at ICLR 2026
Table Detailed number statistics of instances filtered each key step in CossPL-IPC construction pipeline
Longuige
Tocal MPL Projects
File Counts (FSM Filtered)
File Counss (Token Limit)
Instanee Counts (Judger Filtered)
Instance Counts (PSM Filtered)
70
58
51
Go
1534
2239
998
497
392
Java
8307
1038
615
JavaSeript
7560
3721
807
368
271
921
252
151
Python
5219
1600
913
522
502
Table 2: Detailed number statistics of instances filtered at each key step in CossPL-FFF construction pipeline
Language
Total C Functions
File Counts (Execution Filtered)
Python-C
1061
522
C EXPERIMENT RESULTS
Calculation method for 95% confidence interval: For all non-greedy decoding settings reported
in this paper, we provide 95% confidence intervals for both Pass@1 and Pass@5, computed using bootstrap procedure. The bootstrap resampling was performed with 5,000 iterations. For each resample, we recomputed the macro-average pass@5 and pass I; The 2.5th and 97.5th percentiles of the resulting distribution were taken the 95% confidence interval. This procedure provides distribution-free and statistically robust estimate of the uncertainty for pass@5.
C.1 RESULTS FOR ANSWER RQ1
C.1.1 HOW EFFECTIVELY CAN LLMs GENERATE IPC CODE ACROSS DIFFERENT PROGRAMMING LANGUAGES IN CPL PROJECTS?
Overall performance, Table and Table presen the Pass @ I and Pass@5 scores, respectively, for different LLMs across MPL on IPC tasks. Overall, Pass@5 success rates consistently ex- ceed their @ass@1 counterparts across all the task subcategories, reflecting improved performance through multiple sampling attempts. Among the evaluated models, GLM4-plus achieves the highest average Pass@] I score 75.88% followed closely by GPT-4o 73.81 %. In contrast, Qwen3-8b exhibits notably poor performance on Java IPC tasks, with a Pass @ 1 of only 10.73% and @ of 13.56%.
Further analysis of Table reveals that, among models with known parameter sizes, larger models generally exhibit better performance. However, this trend is not strictly linear. For instance, the 671b model (DeepSeek-V3) V33) and the 235b model (Qwen3-2235 -330--32-3b-22b) perform comparably, with an average difference of only 2.67%. In contrast, both significantly outperform the 30b model (Qwen3- 30b-a3b), with an average improvement of 9, 9.87%. Comparing Table 3 with Table 4 Pass typ- ically yields a substantial improvement over Pass@ However, some models with fewer than 10 billion parameters perform poorly on Java IPC tasks, achieving @ rates of only around 50% or even lower.
models with code-specific models, code
the Go subset an example, margin, with a 24.24% higher Pass @ score.
Values highlighted in bold red indicate the best results, while those underlined in blue represent the second- best results. Similarly for the later tables.
29

---

Published as a conference paper at ICLR 2026
Impact of Code Structural Granularity Model Performance. An analysis of Tables B] and reveals little correlation between model performance and code structural complexity. Although 91.22% of candidate samples in the Java subset the function level), the model does not exhibit the poorest performance on this subset. In contrast, the Go subset contains higher proportion of function-level candidates (66.07%). yet the model performs relatively poorly on it.
the class level (perceived more complex than
Language-specific performance disparities. From the analysis, it is evident that the models per- form best on C++ CPL tasks, while their performance on Go-related CPL tasks is relatively weaker. potential explanation for this disparity lies in the structural characteristics of the Go language. Specifically, Go does not natively support the concept of classes, whereas much of the code knowl- edge learned by current L.Mss is derived from class-based programming paradigms. class-level code samples for Go, we extracted code that emulates class-like behavior using structs. This structural mismatch may impose an additional cognitive burden on LLMs trained predomi- nantly on class-based languages, thereby reducing their effectiveness when generating code in Go for such tasks. Conversely, the better performance on C++ tasks can be attributed to the CrossPL focus widely used CPL techniques such TCP, UDP, HTTP, and WebSocket. These protocols are well-documented and frequently appear in public code repositories, increasing the likelihood of LLM pretraining exposure. In addition, C++ is closely associated with low-level system program- ming, which aligns well with the structural characteristics typical CPL. implementations. Fig. T7a illustrates the distribution of Pass @5 scores across different programming languages. The figure clearly highlights performance discrepancies among tasks in various languages. Overall, the models achieve the best performance on C++, while their performance Go is relatively weaker.
constructing
Over3-st O
GwarD 320
Oven23llo
Str
Over2:30
Over
Cwerlide-
Guisc st
Cadiijeenis
craiideenna 70
Spiut ratrect
Cvenu instrutt
Ib-intract
Gemmanth
cadee inatroct
(a) Radar Plot of Pass@S on CrossPL-IPC (pro- gramming languages view)
(b) Radar Plot of Pass@S on CrossPL-IPC (IPC) technologies view)
Figure 17: Radar Plot of Pass@5 on CrossPL-IPC
1.2 HOw DOES THEIR PERFORMANCE VARY ACROSS DIFFERENT IPC TECHNIQUES?
Overall performance. Observe from Tables and 6 that GLM4-plus achieves the best overall performance across the subsets of various IPC techniques. attains the highest Pass@ scores in the UDP, WebSocket, and Message Queue subsets, and ranks second-best in the remaining four subsets. In contrast, flama--Sbiinstuu exhibits the lowest average Pass score at only 40.67%, while Qwen3-8b 8b records the lowest average 5 score.
Notable outliers and analysis. Interestingly, Gemma-78 7b exhibits lower Pass 5 than Pass@ on the HTTP and Message Queue subsets, indicating that sampling introduces noise rather than diversity. Across other IPC subsets, limited P improvements further reveal Gemma-7b* 7b's difficulty in generating diverse yet correct alternatives, highlighting its reliance deterministic decoding.
30

---

Published as a conference paper at ICLR 2026
Table Performance of various models across programming languages (pass @ greedy decode)
Type
Python
JasaSeripi
2004
2004
75. 12.1
67.13%
71.94%
77.885
82.12%
73.81%
GPT-4-mm
Hurs
2034
2004
69.43%
62.15%
49.33%
73800
71.52%
$0.39%
64.63%
pm
G1I (0144y
A1
2004
72 7252
60.72%
23,47%
79.349
82.12%
96.08%
75.88%
DeepSeek
2004
2024
6718
78.21%
64.75%
06.345
82.66%
79.4%%
38.2416
73.51%
Qwmb-235) 77351-11
1035
67 919
65.11%
76 389
74.3%
72.55%
Quen3 32h
Ying 6t
1065
2025
12B
70.8916
62,759%
49.74%
74.17%6
75.50%
30.3916
05 .915
Quen3 14h
Yink &
2015
2025
14B
67.6416
57.57%
59.44%
77.12%
76.82%
70.391
54.54%
Qwen1-8h
Ying
2025
10.73%
55, 7696
46.47%
68.33
68.21%
64.711
43.29%
Qwen3
57.97%
52.81%
57.62%
64.71%
57.37%
GLM4 9t GLM
2024
98
66 (416
54,58%
56.12%
66.42%
70.86%
58.821
61.10%
Lammo
Gemma Th Team
30246
2024
52.52%
43.01%
58.67%
64.90%
43.14%
50,40%
CodeGzeXd/22ui (Zheng al
2023
2024
09.2716
62.55%
55.61%
69.00%
70.20%
30.3916
65.191
Qwen25 5-od----or----mmee
72.50%
Quen2. coder 14b instrve 8
2024
14B
70,41%
66.73%
69.90%
74.17%
78.81%
90.20%
71,04%
Code Model
Qwen2.5 ooder-/biinstncl 10
2004
68. 7819
63.55%
66.07%
75.65%
77.48%
84.311
68.92%
CodeLluna- instrud
53.33%
44.41%
49.24%
49.8216
54.30%
41.18%
50,15%
CodeGemma 7h Ieam 2t
X22
2024
78
$2.20%
53.78%
52.30%
65.31%
62.91%
58.92%
55.40%
Table Performance of various models across programming languages (pass@) 55, temperature-0.2.
top-p=0.95)
Model Type
Model
Publiched Time
Size
Java
Python
JavaSongt
PHP
C++
Mean
9516C1
GPE--o
2024
$2.19%
76.30%
75.795
$5.97%
87.695
59.76%
81.21%
[79524,
82.8%
GPT-4o-miii
54.01%
Gamini-1 Sypo
325
00.695
80.39
57.79%
725
(74.87%,
72.5298
GLMMi-pus
2024
$1.38%
72 142
75.51%
$2.27%
85 1445
171
$1.30%
671B
$4.99%
76 169
75.06%
82.67%
80.43%
03.64'6
90.425
176.77%
Qweni-tth
2025
79.79%
71 14.24%
77.77%
$1.598
80 1314
50.999
77.23%
(76.018 stri
General Model
308
0wex3-14b
66.15%
64,44%
75.94%
80/048
85.24%
71,69%
(69,99% 63.63%
Quen
2025
13.96%
63. 53.5%%
$4.88%
740014
77 0770
73.83%
40001%
9015 51 (.9)
Qwen3
61.
GLML-96
67.30%
71.66%
75.97%
56.17%
70.939
[65.99, 72.8665
Lamme--Sh-instrat
2024
208
61.54%
68 985
$4.81%
77.88%
67.54%
53.87%
63.21%
[63 943. 67. 67.1
71.23%
CodeGreX4
75.29%
79.211
172.025,
Owen25 codes 32b instract
2024
32B
77.96%
71 11.95%
77.78%
$1.56%
84.27%
94.07%
75.169
176.39%, 79.90%
Qwnn2 Seankes Hlbiiotras
2034
148
78.04%
74.43%
75.97%
81.41%
84 73%
06.08'8
75.149
[76 3099, 79 84.)
[15 8994,
Codet 7b instruct
2023
7B
80.38%
74.99%
80.89%
74.00%
80.955
86.56%
78.60%
177.00%, $0.228)
ColeGemmn-Th
2024
7B
72,44%
70.8593
66.22%
73.909
75.05%
74.31%
72.01%
(70) 109, 73.81%)
31

---

Published as a conference paper at ICLR 2026
IPC-specific performance disparities: Tables 6 reveal significant variation in LLM performance across IPC subsets for CPL interoperating code. The gRPC subset achieves the highest results, with 10 surpassing 80% Pass@ I and 14 on Pass@55 the top model reaches 90% and 93.17%, respectively. This success likely stems from gRPC"s standardized syntax, schema-driven design, and extensive representation in training data. In contrast, the low-level, platform-dependert Pipe subset underperforms, with 9 LLMS below 60% on Pass@ and 10 below 70% on Pass@ 5; minimum scores are 33.91% % and 44.23%. Similarly, the HTTP subset shows weaker results- 10 LLMs under 60% Pass and 14 under 70% Pass 5, likely due to its variable implementations and lack of strict interfaces, hindering LLM generalization. Fig. T7b illustrates the distribution of Pass@5 scores across different IPC technologies. The figure clearly highlights performance discrepancies among tasks in various IPC technologies.
Table 5: Performance of various models across IPC techniques (pass @1 1, greedy decode)
Model Type
Model
Pubiished Time
Size
HTPP1
TCP
UPP1
Webuocket
Pipe
2RPC1
Messago Quiee
Mean
71.63%
71.15%
52.611
GPT mini
2024
59.09%
65.13%
09.57%
72.73%
66.675
59 19.69%
67.68%
64.63%
Gcnini-1.5qqm
2014
61.49%
74.60%
73.91%
70.29%
68.65%
56.32%
72.229
69. 9599
Mi-plis
911
14
75.88%
DospSook-V3
2034
671B
66.345
78.79%
84.78%
82.839
60.785
$7.80%
76, 265
73.51%
Qwerd-235b-e
2028
25515
67.01%
77.078
51.00
74.75%6
65. 991
56.82%
22.22%
72.55%
Oweed
323
66.20%
69.45%
77.03%5
$4.52%
54.115
51.59%
73.91%
Genaal Midel
Qwen3-346-ucl
2025
308
$2.769
64.43%
60.875
62.639
56, 866
56.59%
$6,06%
57.57%
Qwend-14b
2028
14R
50.449
72.98%
70.65%
72.79%
$1. 51,6193
76.74%
67. 1714
545
Qwen3
5B
49.55%
29.79%
35.5716
52,50%
35 95.
52.79%
35,3516
Qwee3-4b
2025
4B
54.43%
62. 12%
63.045
$4,149
45.37%
68.22%
59.05%
57.57%
(il M4-Jh
2004
98
56.74%
60.289
60.87%
10.61%
600 1963
61.40%
18%1
61. 10%
Lammu3 Mb-instrucc
2014
8B
40.31%
44.80%
56.96%
44,45%
33.918
38.76/9
54.34%
Gemma-7h
3036
7H
47.50%
56 12:6
51.76%
40.4916
47 0998
52.71%
50,0056
50 40%
CodeGlee44
2014
60.08%
69.05%
66.30%
68.69%
62.75%
72.87%
69.70%
65.19%
Qwen2.5 coder 32b instratt
323
67.27%
72.52%
78,2616
98.7015
73.74%6
72.50%
Seed2---edde--BB 5-50ddg------iisttoe 4bb-isstmcct
2004
148
63.476
76.21%
67.39%
82.329
62. 09.09%
55.27%
69.70%
71.04%
Code Model
Quen?. wenn..5soeee---iinneeee
2024
78
64.11%
71. 959
70.65%
75.25%
62.0099
61.40%
71.725A
68. 9796
Cocel.lama instrect
46.98%
57.74%
55.4316
10%
00.47%
1619
CodoGenam-77
2004
7B
50.83%
63.05%
61.96%
62.63%
45.75%
54.26%
54.55%
53.40%
Table 6: Performance of various models across IPC techniques (pass@5, temperature=0.2, top- p=0.95)
Mode Type
Model
Patfidedd Tire
Size
HTTP:
TCP 1
UDP
Websocket
Pijet 1
1PPC1
Mossage Queue
Meas
Giff-4n
1004
78.58%
85.55
10.90%
$1.21%
179 (91911
GPT-4o-aiii
2004
62.95%
72.10 13.165
75.03%
73538
69:22
66.39%
72.536
69.33 10.35%
[67 1339899..44
Gemini- 5 pm
2004
51.3/%
83.428
$1.7%
75.30%
02105
71.85%
76 16122%
114875 (2.22)
GT plas
7004
11.475
90185
71.355
03.17%
86.555
70.05%
177.55 31 545)
DecpSeck-V3 V3
2004
6718
70065
$2.999
85916
65.58%
51.099
$0.34%
75.85%
(77.109.90.999)
Qwen 2120--20220 222h
2025
355
73.80%
53.45%
18.2%
$130%
7030%
02.42
Ower 22hh
2005
320
72.125
65.15
75.75
176009.50
General Model
30b-23
2005
161.09
Owen7-14h
Quest
2005
53.916
36.009
39.55
55.91%
4L.36
40.05%
14.999.51
Quen3 db
2005
48
59.935
165
0000
61.245
55,145
M0285
65.70%
61.55%
102.59 673)
GLM$4-56
2004
96
53.53%
72.129
60.00%
73.03%
72.455
10,099
76.52%
70.55%
[68999 969)
Lammra1-000 1-hrmmt-h-
3004
81%
50.7955
75. 195
59.81%
62.38%
63.145
61.21%
[65 545 [1N]
Geraeza.7 T6
2004
7B
43.316
57.366
60.24%
$1.23%
$2.386
(6.2%%
46:08%
68.05%
[66 1661299.09099 [29]
CocatinnA
2004
3
55.10%
76.58%
71.73%
75.355
B1325
76,055
70.00%
172 [
Qven2 Sooder 31h Sistttecct
2004
32B
71.07%
70.55
70.56%
21.025
73.36
15.99%
11.72%
78 73.16%
170.59,779
Code Model
Qwes2 Sonden-- i4b-moirncc
2004
148
71.21%
82.7%%
72.87%
87.1%%
6931%
91.54%
71.69%
78.38
Quen?. Th-terat
7054
71L
77.315
12%
74.66
$1.35%
67.75
41.716
77 77.88%
175 74 175)
Colellane Th
2003
50.416
74,548
(77.189.0) 22%
C.1.3 WHAT ARE THE ERROR DISTRIBUTIONS IN CASES OF FAILURE?
To further investigate the error distribution when LLMs generate IPC interaction code, we selected the berf-perg model, GPT-4o, and the worst-pe model, (lama--Sb-instnun for anal- ysis. Tables 18 and 19]preent the error distributions of GPT-4o and 1amma3--b-instruce in gener- ating IPC code across various programming languages. From these figures, it is apparent that LLMs exhibit varying degrees of success in correctly covering the states within the corresponding FSM state transition chains when generating erroneous IPC code. Analyzing Table [8 we observe that GPT-4o's 's state coverage is relatively balanced, with errors predominantly occurring in the config- uration of IPC protocol links and the setup for data transmission and reception. In contrast, Table 19 reveals that aamma--bb-instruct state coverage is heavily concentrated on the first two states,
32

---

Published as a conference paper at ICLR 2026
indicating that it encounters early as the configuration of necessary library files, preventing it from matching the corresponding state machine effectively.
Stupped FSWs Cheir Fiees State Cheir Length
Steaced stale
SuiaCruh
Date jagg
00
0000
0000000--
4000
2--00-0000 090
00
0000
10
50
20
70
10
(a) Error Distribution FSM state transitions ob- served in the Java IPC interaction code generated by GPT-4o.
(b) Error Distribution FSM state transitions served the Python IPC interaction code generated by GPT-4o.
Stopped Stat the PSM Stane Chair 1:30s state Chain Length
State the 15Ms State Hobess state Cha Longth
Caat (att-OWr Lmith
1.2
1.44
125
325
140
242
144
50
22
50
95
200
(c) Error Distribution of FSM state transitions ob- served in the PHP IPC interaction code generated by GPT-4o.
(d) Error Distribution FSM state transitions ob- served the JavaScript IPC interaction code gener- ated by GPT-4o.
SSuopped Suae the PSM Sueve Cheir Fiees State Cawin Length
Stoppec State 12lof VS Char Fibes tmity OWn (wngth
100
105
110
115
120
147.0
3475
146.0
342.5
140.5
14955
155.0
3505
153.0
(e) Error Distribution FSM state transitions ob- served in the Go IPC interaction code generated by GPT-4o.
(I) Error Distribution of FSM state transitions served the C++ IPC interaction code generated by GPT-4o.
Figure 18: Error Distribution FSM state transitions observed the IPC interaccti code generated by GPT-40.
C.2 RESULTS FOR ANSWER RQ2
Table presents the Pass 1 and Pass@5 performance of various LLMs on the CrossPL-FFI benchmark. Among general-purpose models, GPT-4o achieves the highest performance (Pass@ I: 19.54%, Pass@5. 26.46%), followed by Gemini-1.5-ppm and DeepSeek-V3 V3,, indicating that large,
33

---

Published as a conference paper at ICLR 2026
Stopper Sasa ohe rSts Sheae Cheir iat chein weggh
Sroppdd Boftte Sive camnensh
tangli
GB
80 00000 0 D0
D6
B6
0:00
900
000000
530
000
800
00000
0000 00 000 000 000
00
00
000
FSMID
(a) Error Distribution of FSM state transitions ob- served in the Java IPC interaction code generated by JJama3-Sb-instruct.
(b) Error Distribution of FSM state transitions ob- served the Python IPC interaction code generated by Llama3-8b-instme
Stopped Save of ite r5M's Siede Sher 1Shnts State Chain Length
Stoppett Suti 13 ih nSMs Sinic Chan Cra Langth
750
(c) Error Distribution of FSM state transitions ob- served in PHP IPC interaction code generated by Llama3-Sb-instrun
(d) Error Distribution of FSM state transitions ob- served the JavaScript IPC interaction code gener- ated by Llama3-8b ---nst---
Stoppes Stava the ISN Stere Cheir 13Pta State Chan Langth
Stoppes State 13lof the 1SMa State Cham Che Jangth
0
150
(c) Error Distribution of FSM state transitions ob- served in the Go IPC interaction code generated by 1lama3-Sb-instruct
(f) Error Distribution of FSM state transitions ab- served in the C++ IPC interaction code generated by -a-ma-----nsttruet.
Figure 19: Error Distribution of FSM state transitions observed in the IPC interaction code generated by Llama3-88 1033-88binnntett
34

---

Published as a conference paper at ICLR 2026
instruction-tuned models excel generating Python-C FFI code. Smaller Qwen3 variants (e.g., 30B, 14B) exhibit moderate performance, while very small models (c.g.. Qwen3-4b,, GLM4-9b, achieve minimal success. Among code-specialized models, Qwen2.5-coder- 32b-instruct outperforms other models, although their Pass rates remain below 17%, highlighting the overall difficulty of FFI code generation. These results demonstrate that model size, instruction tuning, and specialization significantly influence performance, yet all models show limited absolute success, suggesting substantial room for improvement in LLM-based FFI generation.
Table 7: Pass@) and @ results of various models on CrossPL-FFI
Model Type
Model
Published Time
Size
Pass@)
Pass@5
95%CI of Pass@5
GPT-4o
2024
19.54%
26.46%
[22.90%, 30.10%
GPT-4o-mini
2024
9.77%
13.74%
[11.0%; 16.59%]
G.5pii-1..-pru
2024
21.18%
[17.83%, 24.63%)
GLM4-plus
2024
8.62%
13.40%
[10.72%, 16.23%]
DeepSeek-V3
2024
671B
16.67%
21.50%
[18.23%, 10%
Qwen3-255b-s22b
2025
235B
13.03%
21.28%
[18.10%, 24.52%)
Qwen3-326
2025
32B
13.22%
19.92%
[16.63%, 23.23%)
General Model
Qwen3-30b-a3b
2025
30B
9.77%
14.32%
[11.50%, 17.26%]
Qwen3-14b
2025
14B
11.99%
16.69%
[13.64%, 19.83%]
Qwen3-8b
2025
8B
9.77%
12.06%
[9.48%, 14.86%]
Qwen3-4b
2025
4B
5.94%
9.68%
[7.28%, 12.23%]
GLM4-9b
2024
9B
4.60%
7.92%
[5.79%, 10.20%]
(ommma3-Bb-nnstrus
2024
8B
0.77%
3.95%
[2.59%. 5,48%]
Gemma-72 7b
2024
7B
1.34%
4.32%
[2.83%, 5.89%)
CodeGoeX4
2024
4.60%
6.99%
[4.96%. 9.16%)
(reen2.5-cooerr32bbbiee
2024
32B
7.28%
14.38%
[11.50%, 17.29%)
Code Model
Qwen2.5-coder- 14b-insrttt
2024
14B
4.80%
7.64%
[3.04%, 12.89%]
Owen2.5-coder- 7b--str----
2024
4.80%
8.36%
16.1, 10.73%]
7b-iitttttt
2023
7B
1.72%
5.69%
[4.09% 7.52%)
CodeGemma-7b
2024
7B
4.41%
8.20%
[6. 10%, 10.49%]
C.3 RESULTS FOR ANSWER RQ3
As described in subsection 4.3 this section provides detailed analysis of the experimental data for RQ3. Tables report the performance of LLMs on CrossPL.
Model scale effect: Larger models consistently outperform smaller ones across programming lan- guages and network protocols. For instance, Qwen3-2355 achieves the highest Pass scores on Java (77.40%), C++ (88.24%), and gRPC (86.82%), whereas smaller models (4B 8B) show sub- stantially lower performance.
Enabling think mode yields limited or inconsistent improvement IPC tasks. In some cases, it slightly decreases performance (e.g., Qwen3-235B Java: 75. 75.12% vs 77.40%), sug- gesting that structured IPC patterns do not strongly benefit from additional intermediate reasoning.
Think mode impact:
Language/protocol dependency: The models perform better on mainstream languages (Java, Python, C++) and commonly used protocols (TCP, gRPC), while less common languages or proto- cols (Go, Pipe) exhibit lower scores.
Overall difficulty: Pass scores for FFI tasks are substantially lower than for IPC, with the highest being 17-18%, reflecting the inherent challenges of library linking, type specification, and memory management in FFI.
Think mode efficacy: Think mode significantly enhances performance FFI tasks. For example, Qwen3-32B improves from 13.22% to 17.81 7.81%, %, indicating that explicit intermediate reasoning helps manage the complexity of cross-language function calls and low-level operations.
Model size effect: Larger models benefit more from think mode in FFI tasks, demonstrating that scale amplifies the model's ability utilize reasoning steps for complex scenarios.
35

---

Published as a conference paper at ICLR 2026
The experimental results indicate that Qwen3 models achieve higher performance on IPC than FFI, with large models consistently outperforming smaller ones. Think mode confers sub- stantial advantage in FFI tasks, which require reasoning over library linking and memory manage- ment, but offers minimal benefit for IPC tasks, where communication patterns are more structured and captured adequately in the training data. Overall, model scale and task complexity modulate the effectiveness of intermediate reasoning, highlighting the need for task-specific strategies when generating cross-language interoperable code.
Conclusion:
Table 8: pass @1 results across Qwen3 series models on IPC-related tasks
Model
Published Time
Size
Java
Python
Go
JavaScript
PHP
C++
Mean
Qwen3-235h 2onn3-25hha22bbb dhinkinnn
2025
75.12%
67.73%
61.48%
76.01%
24.83%
82.35%
72.92%
Qwen3-32b- 32b-i-ikb-
2025
61.14%
60.36%
60.97%
72.32%
72.19%
82.35%
68.22%
Qwen3-306e a3b aib-dhiinn
2025
50.33%
55.18%
52.04%
70.85%
73.51%
78.43%
63.39%
Qwen3- 1rren3-14b-ttinn 14b-thinking
2025
57.07%
62.75%
58.67%
69.74%
68.87%
72.55%
64.94%
Owen3-8b-tiiniin
2025
24.72%
55.58%
46,94%
69.74%
70.20%
66.67%
55.64%
Qwen Qwen3-4h-thinkee 4h-thinnking
2025
33.82%
57.77%
50.51%
61.62%
$2.32%
70.5995
54.44%
Qwen3 235h a22b
2025
77.40 %
67.93%
65.31
76.38%
24.83%
88.24%
75.02%
Qwen3-32b
2025
70.89%
62.75%
49.74%
74.17%
75.50%
80.39%
68.91%
Owen3-30b-aSb
2025
57.24%
51.99%
49.75%
70.48%
71.52%
66.67%
61.28%
Owen3-14b
2025
14B
67.64%
57.57%
59.44%
77.12%
76.82% %
70.59%
68,20%
Qwen3-8b
2025
10.73%
55.78%
48.47%
68.63%
68.21%
64.71%
52.76%
Qwen3-4b
2025
54.96%
57.97%
52.81%
66.79%
57.62%
64.71%
59.14%
Table 9: Performance of Qwen3 series models across network protocols (pass @ 1)
Model
Published Time
Size
HTTP
TCP
UDP
Websneket
1
Mossage Quene?
Mcan
Qwent-23fb- 221bb-22--2thhk
2025
235B
64.96%
76.67%
70.35%
73.74%
64.015%
$2.17%
22.27%
23.31
Qwen3-376-Biin
2025
32B
58.28%
70.44%
70.65%
71.72%
45.10%
80.62%
63 6466%
65.78%
Qwent-30h-gab-e tih-ghhh-kk
2025
30B
56.10%
2855
65.22 <4.22%%
64.14%
48.37%
66.67%
54.04%
59.265
Owead3 14b 14b-think
think
2025
14B
55.58%
66.51%
58.70%
72.73%
54.25%
79.07%
61.62%
64.07%
Qwen3-88 8b-think
2025
8B
50.19%
11.50%
45.65%
60.60%
35.29%
54.26%
41.4%%
47.32%
-hwen--4-tthink
2025
$1.865%
19%
43.48%
50.51%
32.68%
61.24%
46.46%
47.92%
Qwea3-2356-222 2356-2226
2025
235B
67.01%
77.60%
83.70%
74.75%
65.36%
86.82%
22.22
75.35%
Qwen3-37b
2025
328
20%
48%
77.03%
84.52%
54.11%
81.58%
73.91%
72.40%
Qwen?-3the 30b--2b
2025
30B
52.766%
64.43%
60.87%
62.63%
56.86%
56.59%
56.0666
58
Qwen2-1 14b
2025
14B
59.44%
72.98%
70.65%
72.73%
51.63%
76.74%%
67.17%
67.33%
Qwen3-8b 8b
2025
8B
49.55%
79%
35.87%
52.53%
35.95%%
62.79%
35.35%
43.12%
Qwen3-4t
2025
54.43%
126
63.04%
54.04%
48.37%
68.22%
59.099
58.47%
C.4 EFFECTS OF MODEL SCALE IN THE TWO BENCHMARK PIPELINES
Intuitively, in the IPC-related pipeline, smaller or less powerful LLMs could likely replace the Judger and "Instructor" components, as the associated reasoning and enstruction-generatin tasks relatively simple. However, function-level and class-level code extraction still require stronger custruction-followiii and structural reasoning capabilities, which are better handled by large models such as DeepSeck-V3 used in our original framework.
In the IPC-related pipeline, the Judger module only needs to determine whether the input file con- tains IPC-related logic and produce a short categorical output (e.g., "Yes," "Function-level." or "Class-level"). This step requires very few output tokens and minimal reasoning depth. Although smaller models could, in principle, achieve similar decision accuracy, they are more prone to false positives. Such false positives would propagate into the downstream extraction stage and dramat- ically increase the number of unnecessary extraction attempts. For this reason, we recommend keeping the Judger as stronger model to maintain pipeline efficiency and avoid computational overhead caused by spurious matches. In contrast, the FFI-related pipeline demands that the LLM simultaneously analyze low-level C code, design corresponding tasks, generate canonical solutions, and produce assertion-baat based test cases. These multi-step reasoning and synthesis tasks rely more heavily advanced models.
As a potential extension, we conducted an ablation study to quantify the impact on the number and quality of generated samples across both IPC-related and FFI-related pipelines.
36

---

Published as a conference paper at ICLR 2026
Table 10: Performance of Qwen3 series models on CrossPL-FFI (pass
Model
Published Time
Size
Pass @ ]
Qwen3-235b-225--hink
2025
235B
17.24%
Qwen3-32b-think
2025
32B
17.81%
Qwen3-30b-a3b-think
2025
30B
11.69%
1wre33-44b-thnkk
2025
14B
12.46%
Qwen3-8b-think
2025
8B
13.22%
Qwen3-4b-think
2025
4B
8.05%
Qwen3-235b-a22b
2025
235B
13.03%
Qwen3-32b
2025
32B
13.22%
Qwen3-30b-a3b
2025
30B
9.77%
Qwen3-14h
2025
14B
11.49%
Qwen3-8b
2025
8B
9.77%
Qwen3-4b
2025
4B
5.94%
For the ablation studies, we designed three experimental settings to isolate the impact of model size specific pipeline components: 1 "Extractor"-onfe replacement: We replace the "Extractor" with a smaller model (Qwen3-4B) while keeping all other components, including the "Instructor" and Judger, powered by DeepSeek- V3. "Instructor"" replacement: We replace only the "In- structor" with Qwen3-4B, while maintaining DeepSeek-V3 for extraction, judging, and verification- related reasoning. @ In the FFI sample construction pipeline, we replaced the "Generator" with smaller model (Qwen3-4B).
These ablations will quantify how much smaller models affect both the quantity and quality of generated benchmark samples across IPC and FFI pipelines.
From the experimental results, we observe that replacing the "Extractor" with Qwen3-4B substan- tially reduces the number of successfully constructed samples. The original pipeline produced 1,982 samples, whereas the replacement yielded only 1,723 samples-a a decrease of 259. contrast, re- placing the "Instructor" with Qwen3-4B has much smaller impact. The original pipeline also produced 1,982 samples, and the modified version generated 1,890 samples, reducing the total by only 72.
However, substituting the "Generator" with Qwen3-4B severely degrades performance in the FFI pipeline: the number of valid samples dropped from 522 just 65. This result highlights that constructing high-quality FFI tasks requires stronger and more consistent large-scale model. We sincerely appreciate the reviewer's insightful question, which helped better analyze and clarify the role of model size in different components of the pipeline.
C.5 RISKS OF DATA LEAKAGE IN CPL CODE GENERATION BENCHMARKS
To address potential concerns regarding temporal contamination, we clarify how time-based consid- erations were incorporated in the construction of both subsets of CrossPL.
37

---

Published as a conference paper at ICLR 2026
Table 11: Instance counts before and after "Extractor"/IInstrutoo"" replacement
Langunge
Original
"Extractor" replacement (Replaced by Qwen3-4B)
"Instructor" replacement (Replaced by Qwen3-4B)
C++
51
Java
615
606
620
JavaScript
271
157
262
PHP
151
159
162
Python
502
468
Table 12: Instance counts before and after "Generator" replacement
Language Pair
Original
"Generator" replacement (Replaced by Qwen3-4B)
Python-C
522
65
(1) Temporal analysis in the construction of CrossPL-IPPC. The IPC subset is built from real-world open-souree repositories on GitHub, and therefore requires explicit temporal validation. We cate- gorized all IPC samples into four time buckets: before 2022, 2022, 2023, and 2024-2025. Notably, the majority of IPC instances (1377 samples, 69.48% of the IPC subset) originate from 2024-2002220 2025. Table 13 show the counts of CrossPL-IPC Samples with different Release Year.
We then evaluated model performance across these temporal slices. The results (table [4 and figure 20 show evidence that models perform systematically better the newer subsets; in fact, model scores on older samples do not exceed those on more recent ones. This suggests that temporal leakage does not materially inflate performance in the IPC tasks.
(2) Contamination mitigation in the construction of CrossPL-FFL. For the FFI subset, we deliberately designed the pipeline to avoid exposure to pre-existing code. All FFI samples are constructed in self-contained and self-guided manner: the LLM is shown a standalone function and asked to generate task description, @ design Python wrapper using ctypes, and produce canonical reference solution.
To further quantify the risk of training-date contamination, measured the overlap between prompts and the Pygsl codebase used during verification. As shown the Table 15, the overlap is extremely mww-0.61% for function names and 0.99% for class names iideee-ndicating minimal risk that LLMs benefited from prior exposure to these specific instances. We thank the reviewer for raising this question, it prompted to conduct more systematic examination of temporal leakage and contamination across both IPC and FFI pipeline.
Table 13: Number of CrossPL-IPC Samples with different Release Year
samples before 2022
Number samples
of samples
Number samples in 2024-2022
222
143
240
1377
C.6 EFFECTS OF SAMPLING TEMPERATURE AND TOP-P ON CPL CODE GENERATION
To assess how decoding hyperparameters (temperature and top-p) influence cross-language code generation, we evaluated eight locally deployable models (Qwen3-4B, Qwen3-8B, GLM4-9B, -lamaa-8B-Instruet, Gemma-7B, on two CrossPL subsets: CrossPL-IPC (Java) and CrossPL-FFL. Two controlled experiments were conducted:
CodeGemma-7B)
temperature sweep with top-p fixed at 0.95 and temperature E (0.2, 0.4, 0.6, 0.8, 1.0, 1.2]:
A top-p sweep with temperature fixed at 0.2 and top-p E (0.1, 0.3, 0.5, 0.7, 0.9, 1.0).
38

---

Published as a conference paper at ICLR 2026
Table 14: Pass@1 Performance Across Release Years
Model
Before 2022
2022
2023
2024-2025
GPT-4o
70.27%
81.12%
75.42%
73.38%
GPT-4o-mini
64.41%
80.42%
64.17%
63.11%
Gemini-1.5-pro
65.77%
80.42%
69.58%
69.64%
GLM4-plus
71.62%
83.22%
75.42%
75.89%
DeepSeek-V3
72.97%
81.82%
72.08%
72.98%
Qwen3-235b-a226
69.55%
81.43%
76.69%
70.79%
Qwen3-32b
63.64%
77.86%
64.83%
64.34%
Qwen3-30b-a3b
59.09%
70.00%
58.05%
55.58%
Qwen3-14b 14b
64.09%
77.14%
66.53%
64.19%
Qwen3-8b
50.45%
32.14%
38.98%
43.22%
Qwen3-4b
55.00%
68.57%
62.29%
55.43%
GLM4-9b
59.91%
78.32%
61.25%
59.48%
Lamma3-8b-instruct
43.05%
39.44%
49.58%
38.85%
Gemma-7b
51.12%
59.86%
58.75%
47.86%
CodeGeeX4
62.61%
81.12%
70.42%
63.04%
Qween2.--coder-32-innntte
65.77%
83.92%
72.03%
72.48%
1wee225-codee--44biinntt
67.12%
78.32%
71.67%
70.81%
Qwenn25-coder-7-innstuu
66.22%
79.02%
69.58%
68.19%
CodeLLlama-7-iinsrruct
52.47%
66.90%
56.67%
46.91%
CodeGemma-7b
55.16%
69.72%
59.17%
53.30%
Mean
60.63%
73.08%
64.98%
60.88%
Figure 20: Classification of CrossPL-IPC Samples by Release Year
39

---

Published as a conference paper at ICLR 2026
Table 15: Overlap statistics between pygsl and CrossPL-FFI
Metric
Count and overlap Rate
Functions in pygsl
4,510
Functions in FFI-bench
14,822
Functions in both pygsl and FFI-bench
90
Function overlap rate
0.61%
Classes in pygsl
868
Classes in FFI-bench
304
Classes in both pygsl and FFI-bench
3
Class overlap rate
0.99%
Performance was measured using @ass@1 (computed with n and kt 1), and the results are
reported in Tables 16 19
Across both experiments. several consistent conclusions emerge. First, neither temperature nor top-p exhibits a linear or stable influence model performance. Adjusting sampling parameters changes model behavior, but the effects are irregular and strongly model-dependent.
Second, all models perform substantially better on IPC than on FFI. IPC tasks follow well-defined communication protocols, resulting in a constrained solution space. In contrast, FFI tasks require low-level reasoning over function signatures, ABI conventions, type conversions, and memory lay- out; even small errors may lead to undefined behavior. This makes FFI generation considerably more difficult. For example, the average Pass @ I of Qwen3-4B decreases from 55.24% on IPC to only 3.08% on FFI across different temperatures.
Third, parameter tuning has limited impact on FFI performance. Increasing temperature or top-p does not enhance creativity in a beneficial way, and lowering them does not improve output stability. Both strategies fail overcome the inherent difficulty of FFI generation. Mcaningful improvements will likely require enhanced cross-language alignment and komain-specifis fine-tuning with FFI- relevant knowledge.
Finally, optimal sampling settings vary across models. These differences reflect variations in training corpora, alignment procedures, and the extent to which each model has been exposed to cross- language code during training.
Table 16: Performance of different models on CrossPI.-IPC (Java) under varying temperature (fixed top-p=0.95). @ass@1 computed with and The values parentheses denote the 95% confidence intervals for @ 1
=
0.6
Qwen3-4h
54.50%
58.41%
53.92%
55.25%
53.82%
([11.28%, 57.79%)
((55 19%6 61.63))
([50.47%; 57.24%))
([51.90%) 60%)))
(550.45%, 57, 14%%)
Qwen3-8b
10.34%
12.55%
([8. 16%, 12.03%]
([8.42% 12.36%.
(9.0055 12.91%)
(19.59%, 13.56%)
(110.57% 14.54%)
GLM4-9b
51.58%
52.39%
51.80%
$2.07%
52.72%
(148.33/8, 54.80%)
(149.23%, 55.58%
(148.68% 54.96%
(148.11% 55.25%
(149.599, 55.93% $5.939%
Llama3-8B-Instruct
33.37%
32.94%
32.29%
33.37%
33.40%
(330.67%, 36.00%)
(330.34%, 35.615:
([29.69%, 34.93%)
(330.73%, 36.03%)
(330.73%, 36, 10%)))
Gemma-7B
52.20%
50.31%
49.95%
51.06%
50.44%
(148.85%, 55
9146,993, 53.66%%
(146.73%. 53.30%
(147.71% 54.47%
(147.06%% 53.79%
Qwen2.5 codder-
66.89%
66.11%
66. 66.80%
66.67%
67.77%
(663.74%, 70.02%)
(162.93%, 69.27%)
(f63.61%, 69.92%)
((63.45%) 69.79%)
(164.52% 70.96%)
CodeLllama--BBInsstue
$5.90%
56.23%
57.40%
55.97%
56.16%
(152.8 $8.89%
([53. 17%, 19.19))
([54.37%) 60.39%)
(52.998%, 59.06%))
(533.14% 19.12)))
CodeGemma-77
52.20%
52.91%
53.07%
53.63%
52.52%
(148.84%, 55.64%
([49.50%i 56.29%)
(149.66%, 56.52%
(150.245, 57.14%
(149.19% $5.87%.)
40

---

Published as a conference paper at ICLR 2026
Table 17: Performance of different models on CrossPL-IPPC (Java) under varying top-p (fixed tem- perature=0.2). Pass@] computed with = and k = 1. The values in parentheses denote the 95% confidence intervals for pass@I.
Model
Top-p 0.3
Top-p 0.5
Top-p 0.7
Top-p 0.9
Top-p 1.0
Qwen3-4b
54.31%
54.76%
54.83%
55.06%
54.41%
([50.44%,588 (4448888855)))
([50.89% 58.55%%
([51.02% (8.60)))
((51 355% (8.86))
(150.70%,58.028))
Qwen3-8b
9.17%
9.11%
9.63%
9.72%
9,92%
14.9998,1144499
(66.93% 11.28%)
(7.41%%.11.8880
([7.61% 18.11.906))
(17.77%.12.10%)
GLM4-9b
65.59%
65.37%
65.24%
64.72%
64.07%
([61.53% (161.53%6668.98%)) 66..88%)
(661.00%
(600.62%.67.4887
Lluma33 8B Instruct
34.63%
31.80%
32.94%
34.05%
33.17%
([3.007%_372279)
(29.3399,34.3155
(30.3118,355884)
(6113399,367222
(30.5449.35.7444
Gemma-7B
51.61%
51.09%
51.51%
51.54%
52.13%
([48.33%%554.86) (6.)))
([47.71 9.54.41%)
([48.03%,54.
(0100099554444411 54.8%%
[[48.23%.5549958))
(148.88%- 889.,55555551111 (.5511%) (%1)
Qwen2.5-coder-7h
67.87%
67.67%
68.13%
68.03%
68.39%
([64, (064,10%,7 0098,71.544 71.54%)
116.99%%.71.32555
([64.55% (0..5%%771.88)
(66462%%,71338))
([65.01% 11..111,77.119
CodeLlama-7B-Insst
57.17%
56.23%
55.35%
56,33%
56.29%
[154. 1144.18%%6001000 60.10)
(153.20 20% 39.22%)
(152.36% (I52.36%.58837%))) 58.3%)
(153.30%..99. (2..999..
(13.20%%.593324))
CodeGemma-7B
54.86%
53.92%
53.69%
54.37%
54.80%
(151.5498558.249)
(150.67%.57.30K%))
(150.7%%.57.019))
(151 136
(151.519,58.2448)
Table 18: Performance of different models on CrossPL-FFI under varying temperature (fixed top- p=0.95). Pass I computed with 5 and k 1. The values in parentheses denote the 95% confidence intervals for pass@1. 1.
Model
T=0.4
T=0.6
T=0.8
T=1.0
T=1.2
Qwen3-4b
2.72%
3.72%
2.80%
3.22%
2.84%
(11888%, 3.72%])
([2.57%. 4.98%.J
([1.92%
3.75%J
([2.22%, 4.37%)
([1 (1.88%, 3.95%)
Owen3-8h
6.21%
6.67%
6.59%
17%, 8811%)
(14.48%, 8.05%.)
(14,90,, 8.62%)
([4.22%, 9.57%)
(14.87%, 8.47%:
GLM4-9h
1.00%
1.30%
0.73%
1.03%
0.80%
(10.50%, 1.57%)
(I0.65%,
2.07%.J
(I0.34%,
1.23%)
([0.54%. 1.61%)
(10.38%,
1.30%)
Llama3-8B-Instuue
1.11%
1.15%
1.11%
0.96%
1.07%
((0.61% 1.69%)
(10.65%, 1. 1.69%)
([0.61%.
1.72%)
([0.54%, 1446%)
(00.61% 1.57%)]
Gemma-7B
1.38%
1.88%
1.57%
1.69%
1.84%
((0.73%. 2.15%)
([1.0%% 2. 2.80%)
((0.88%, 2.41%)
([0.96%, 2.49%)
([1003% 2.72%)]
Qwen2.5-coder-7b
4,25%
4.06%
4.18%
3.79%
3.75%
([3.10% 5,52%)
([2.91%, 5.33%)
([3.03% 5.44%)
([2.68%, 5.02%) ))
([2. (2.72%, 4.87%)
CodeLlama-7B-Instree
1.72%
1.49%
1.95%
1.76%
1.69%
([1.03% 2.49%))
(10.92%. 2.15%)
(l1.26% 2.72%)
([1.0%% 2.57%)
( (1.00%% 2.45%)
CodeGemma-7B
3.98%
4.25%
3.33%
3.49%
3.79%
([2.80%, 5.29% |)
(12.99%. 5.67%J)
([2.26% 4.52%)
([2.34%, 4.71%)
([2.615, 5.13%)
41

---

Published as a conference paper at ICLR 2026
Table 19: Performance of different models on CrossPL-FFI under varying top-p (fixed tempera- ture=0.2). Pass @ computed with = 5 and k = I. The values in parentheses denote the 95% confidence intervals for pass@1.
Model
Top-p=0.3
Top-p=0.3
=op-p=0.7
T.p-p=0.9
Top-p=
Qwen3-4b
6.63%
6,25%
6.44%
6.25%
6,90%
(44.71% 8.77%))
([4.37%, 8.31%)
(4.56%, 8.47%)
([4.41% 8.20%)
(4.99% 8.97%)
Qwen3-8b
9.66%
10.00%
9.60%
9,50%
9.77%
(07.20%, 12.34%)
([7.82%, 12.99%)
(17.28%. 12.26%)
((7.20%, 12.03%)
((7.4%% 12.30%)
GL.M4.9b
1.23%
0.80%
0.96%
1.07%
1.42%
(10.69%. 1.84%)
([0.38%, 1.30%)
(0.46% 1.53%))
([0.57%. 1. 1.65%)
((0.73%, 2.26%)
Llama3-8B-Insee
1.46%
1.15%
0.77%
0.57%
0,65%
(10.96%, 1.99%)
([0.65%, 1.76%)
(00.38% 1,23%)
([0.27%, 0.96%)
((0.27%, 1.11%)
Gemma 7B
1.34%
2.15%
1.92%
1.49%
1.80%
(00.69% 2.07%
(11.23% 3.18%)
(11.11% 2.84%)
(10.80% 2.30%
( (1.033% 2.68%)
Qwen2.5-oxde--7b
3.79%
4.79%
3.41%
3.68%
3,79%
(2.66%% 5.06%)
([3.49%, 6.21%))
([2.45%, 4,48%)
(2.64%% 4.87%)
((.766%, 4.94%)
Codelllama- -B-mtmtt-
1.26%
1,92%
1.38%
1.57%
1.84%
(0.73%. 1.88))
((119%, 2.76%))
(0.80% 2.03%)
((0.96%. 2.26%)
(L.11% 2.64%)
CodeGemma- 7B
3.75%
3,83%
3.64%
3.98%
2.95%
(1.661% %. 5.06%)
([268%, 5. 10%%)
(22449% 4,94%)
([280% 5 5,29%
(11.5%, 4.06%)
D CASES STUDY
D.1
CASES STUDY OF CROSSPL-IPC
Fig. 21 illustrates a sample alongside its corresponding "Instruction", "Canonical result", and an incorrect answer generated by DeepSeek-V3. The task involves implementing gRPC server using the grpeio library. Based on the official documentation, we have summarized the general procedure for such problems, represented the semantic descriptions of the various states in the FSM: (1) Import the necessary libraries required for server creation; (2) Create a gRPC server with a thread pool executor to support concurrency; (3) Bind the server to port using the add_insecure-pon method; (4) Start the server to begin listening for incoming gRPC requests.
From the "Canonical solution" shown on the middle part of Fig. 21 the implementation fully com- plies with this standard and passes validation by the predefined FSM. In contrast, the incorrect an- swer on the lower part lacks the critical step of binding the server to a port using add insseuue port.. Consequently, the server does not listen any port, rendering it incapable of receiving requests and thus functionally ineffective.
Notably, although the instruction does not explicitly mention the binding operation, it states: "Ini- tialize and start a gRPC server capable of handling concurrent gRPC requests." This implicitly requires binding the gRPC server to port. In practice, software engineers seldom specify every detail, as enumerating all functional steps often takes longer than implementing the functionality itself.
D.2 ERROR TYPE DISTRIBUTION ANALYSIS ON CROSSPL-FFI
As described in subsection 4.2) this subsection presents an overall error type distribution analysis on CrossPL-FFL. Fig. 22 presents the distribution of the eиror types observed GPT-4o-generated Python-C FFI interaction code. The histogram shows the frequency of different сттоr categories, including symbol resolution errors, GSL runtime errors, Python-level calling errors, undefined sym- bols (NameErrorr), assertion/test failures, memory/crash errors. GPT-4o, the best-performing model, exhibits errors primarily in symbol resolution errors (111), Python-level calling errors (84), and NameeEror/undefine symbols (94), while other errors such as GSL runtime errors (57), asser- tion/test failures (51), and memory/crash errors (54) occur less frequently. Overall, the error dis- tribution is relatively balanced, without any single category dominating. Llama3-8b -aimma3---niituuct.. the worst- porst-performing model, shows a more extreme distribution: Python-level calling errors (212) and symbol resolution enrors (166) account for the vast majority, with Namnennor/n symbols
42

---

Published as a conference paper at ICLR 2026
"Instruction":
"Task Description:
Create function up and tear down gRPC server with configurable interceptors and thread pool workers.
Function Description: The function set up initializes and starts a gRPC server with optional interceptors, while " __terr down" gracefully stops the server. The server is configured to handle gRPC requests concurrently using thread pool. Input:
interceptors (Optional[Listt grpe. wtw.. Serverinterceptor]]] A list of gRPC server interceptors for request/response processing. Defaults to None.
server " (grpe.Server): The gRPC server instance to be stopped (for " tear down").
Output:
set up returns the initialized and started "grpe.Server" instance. tear down" has return value (stops the server).".
import grpe
from concurrent import futures
from typing import List, Optional
def set inttuuiinterreptors:: Optiioaallislggpeseeeettttttrepe None): server grpe.server(
Tha.:se:TTTccadooooEEEcouuuon workkers=))..)) intereepoors-ineeccepooos
add gRPPCTeesterviceServier serven(TestService), server)
server.add insecure_ (3 (foffr"(:)"
server.star()
return server
def tear down(server: gepc.Server): server.stop(None)
A!
import grpc
from concurrent import futures
from typing import List, Optional
def settttttue inttuupinterrrptors:: OpiinnalLatiigoo.seveennttttg]]] None) >> Sepe.Server:
server = grpc.server(
futures. hiuursssTTheeddPPollEeeetttoaaa workers-10),
interepss if interceptors else []
server.start(
return server
def tear down(server: Sepe.Server) None: server.stop(I)
Figure 21: Case Analysis: the 495-th task the CrossPL-IPC. The upper part is the task instruction; the middle part is the Canonical solution; the lower part is the wrong result generated by Deepseek- V3
43

---

Published as a conference paper at ICLR 2026
(128) also significant. In contrast, GSL runtime errors (2), assertion/test failures (5), and mem- ory/crash errors (5) are rare. This is likely occurs because Llama3-8b-inn 8b-iisstruc frequently fails dur- ing the basic function call and symbol resolution steps. preventing the code from successfully com- piling and thus rarely triggering later assertion or runtime errors. Therefore, GPT-4o demonstrates greater robustness in FFI code generation, with errors relatively dispersed across categories, whereas Llama3-8b-instruee is particularly prone to errors in function calls and symbol resolution, highlight- ing its limitations in cross trogs-lan function interactions. Moreover, all LLMs exhibit suboptimal performance on the CrossPL-FFF benchmark, indicating substantial room for improvement.
D.3 CASES STUDY OF CROSSPL.-FFI
As described in subsection 4.2 this subsection presents detailed case analysis of the Python-C FFI code generated by GPT--4o. We conducted detailed case study on GPT-4o's outputs for Tasks 205 and 257 the CrossPL-FFI benchmark. Fig. 23 presents the prompt and reference solution for Task 205, while Fig. 24 illustrates the corresponding assertion test cases and the erroneous outputs. Similarly, Fig. 25 shows the prompt and reference solution for Task 257, and Fig. 26 displays its assertion test cases along with the incorrect outputs. GPT-4o exhibited symbol resolution failure on Task 205 and a GSL runtime error on Task 257.
D.3.1 TASK 205
For Task 205 in the CrossPL-FFI benchmark, GPT-4o generated Python function intended solve a triangular matrix system using the GSL library (libgsl (assssso) so) via ytep The provided im- plementation attempted to call the cblas_ct ram function directly from libgsl. so. However, execution failed with the following undefined symbol error:
Execution Error
AttributeBrror:
/
undefined
symbol: cblas etram
This failure indicates that cblas_ctrsm csr is not exported by libgsl. itself but resides in the separate 1 Libgs cblas as.so so library. Despite loading libgs ccllas.......................... as. with RTLD_GG RTLD.GLOBAL, GLOBAL, GPT-4o's solution did not correctly link the function, reflecting limitation in the model's under- standing of dynamic linking and cross lobss-i symbol resolution in Python FFI contexts. This case illustrates that GPT-4o can produce syntactically plausible code while failing account for library- level dependencies critical for correct execution.
D.3.2 TASK 257
For Task 257 the CrossPL-FFII benchmark, GPT-4o generated a Python function intended to per- form vector addition using the GSL library (libgsl. so) via ctypes. The implementation wraps Python lists into a custom GSLVect class and calls gsl.vector_ade to compute element- wise sum. During execution, the function triggered the following runtime error:
Execution Error
gsl: crt.source.c:2... ERROR: vectors must have same length Default GSL error handler invoked.
This failure indicates that GPT-4o's solution did not adequately validate the input vectors' lengths before calling the GSL function, which enforces strict dimension consistency. Although the function checks the Python lists' lengths at the Python level, mismatch in the underlying GSL vectors an improper mapping to ypep arrays can still lead to runtime errors. This case highlights GPT-4o's limited handling of library-level preconditions and runtime constraints in Python-C FFI contexts, showing that syntactically correct code may still fail during execution when domain-specific invari- ants not fully respected.
44

---

Published as a conference paper at ICLR 2026
Error Type Distribution
100
94
80
Count
51
40
20
Symbol resolution errors
GSL runtime error
Python-level calling errors
NameError/lundefined symbols
Assertion/test failures
Memory/crash errors
Error Type
(a) Distribution of error types observed in the Python-C FFI interaction code generated by GPT-4o.
Error Type Distribution
212
200
175
166
150
128
Count
100
75
Symbol resolution errors
GSL runtime errors
Pethon-level calling errors
Assertion/test NameErroriundefined symbols
failures
Memory/crash errors
Error Type
(b) Distribution of error types observed in the Python-C FFI interaction code generated by Llama3-88 Baanaa--b-inssee -b-instruut.
Figure 22: Distribution of error types observed in the Python-C FRI interaction code generated by GPT-4o and Jlama3-8b-instruce
45

---

Published as a conference paper at ICLR 2026
SUMMARY OF CASE STUDIES ON IPC AND FFI
The case studies presented in Sections D.I [ reveal several key limitations of current LLMs in generating cross-language interoperating code. In the IPC scenario, models often omit critical steps such as binding a gRPC server to port, resulting in implementations that are syntactically valid but functionally incomplete. This reflects a tendency of LLMs overlook implicit requirements that are not explicitly stated in the instruction but are essential for correct execution. In the FFI scenario, the observed issues include symbol resolution failures (e.g., attempting access functions not exported by the linked library) and runtime errors due to insufficient validation of input data consistency (e.g., mismatched vector lengths in GSL operations). These cases highlight the gap be- tween generating plausible-looking code and ensuring robust, semantically correct interoperability across programming languages.
Taken together, the findings underscore that current LLMs lack deep understanding of language- specific APIs, library-level dependencies, and domain-specific preconditions that are indispensable in languaee function calls.
D.4 CASES STUDY OF THINK MODE
Figure 27 presents a representative failure case that highlights how think mode degrades model re- liability IPC-oriented code generation. Although both answers were produced by the same base model, the think-mode version consistently deviates from the protocol-specific constraints required by the task, whereas the non-think version adheres closely to the expected Boost.Asio UDP com- munication pattern. This contrast provides clear window into the mechanism by which extended reasoning disrupts IPC code correctness.
In this case study, the think-mode model produced incorrect UDP client code by replacing the re- quired so/eecv/ee from datagram operations with commect/send/rrceivee,, effectively imposing a TCP-like workflow onto UDP task. This violates both the Instruction and our FSM, which explic- itly require message-oriented UDP semantics.
The underlying mechanism is that think mode encourages the model to generate longer, self- consistent reasoning chains before emitting code. While this can benefit FFI step reasoning about memory layout and type mapping is essential it becomes harmful in IPC tasks. IPC implementations demand strict, ppotocol-specific API usage, but think mode tends to over-generalize and substitute higher-level "universal socket templates." introducing steps that ap- pear logically coherent but are semantically incorrect.
multi-
As result, excessive reasoning amplifies small early misconceptions (e.g.. "UDP can use connect() like TCP"), causing the model to drift away from protocol constraints and break the precise com- munication logic required in IPC programming.
E
USE OF LLMS
The LLM's role was strictly limited to improving the quality of the prose. contributions included:
Assisting with grammar, spelling, and punctuation.
Rephrasing sentences and paragraphs for better clarity, conciseness, and flow.
46

---

Published as a conference paper at ICLR 2026
Write Python function that ctypes load and call the chlas function from the GNU Scientific Library's dynamic library. The function should perform triangular matrix solve operation complex matrices. The input should include the order, side, uplo, transpose, as inlegers, complex alpha value, complex matrix A, and a complex matrix B. The oulput should modificd matrix after the operation. Ensure solution uses ctypes to load librarics and correctly, To accomplish the above task, the required functions along with their parameter lists are: [(triangular matrix solve', ['order', 'side', 'uplo', 'transA,, 'diag', 'M, 'N,, 'alpha', 'A,, 'Ida', 'B, 'dbb'))]
diag parameters as intogers, dimensions M
function
The recommended dynamic library loading order is as follows: ctypes. ctypes.CDLLC CDLL("/us hatps/aallgl///i///iigggae higgs/iibbbbbsssssosoo modo-ctypes.RTI RTLD GLOBAL) gsl - wttpes. CDLLs ('Lus//r/lll //blii//ibgss.soo)
You must return ONLY valid JSON object.
Do not include explanations, comments, extra text.
Do not use triple quotes ("r").
The JSON object contain exactly field: "Candidate solution": <string?>
Inside "Candidate solution", provide the complete code single string. Escupe all line breaks
&A
import ctypes
A
import numpy as np
Load the libraries
CDLI CDLLL'Dsr/lolaa
mode- (
gsl - clypes. wrrssCDLL('Me (folL((/r//luallggg htsl/ss///bgsl......
Define the cblas cirsm function from libgslcblas.so
eblas ctrsm
etrsm
eblas ctrsm.argtypes [ctypes.c int,
ctypes.c_ime int, ctypes.c_int
etypes.c int,
etypes.c int,
ctypes.c
etypes.c int, ctypes. SypsE.PoINTERctcppcs..o float), PoppePPOINTERcctppc..c float). ctypes.c int, Eyyps.POINTER(cyypoc float), ctypes.c_ int]
Function to perform triangular matrix solve
def' triangular matrix solve(order, side, uplo, transA, diag, M, N, alpha, A, Ida, B, Idb):
alpha o= (ctypes.c float 2aalpha.rcal, alpha.imag)
A_ (clypes.c [foualllt (2 Idu (n)((tparraa(A
htpp:mmp.omlleexx))vevveei.. www.wwie.ww..333332 flout32).latttee flatten())
B_ (ctypes.c float (2 Idb N)X**pp.arrayd diype ppppmp.pllexx64)vvveeedeed
cblas crrsm, side, uplo, transA. diag, M, N, alpha_c.ppaaaaaaa A. Ida, B. Idb) return ap.array(B
wttp:/oooooooooooooooeeeoeeeeeee N))
Figure 23: Case Analysis: the 205-th task in the CrossPL.-FFI. The upper part is the task instruction and the lower part the Canonical solution.
47

---

Published as a conference paper at ICLR 2026
def assert assert_test(::
# Test case
order 101 # ChlasRowMajjor
side 141
CblasLeft
uplo = 121 # CblasUpper
transA = 111 # CblasNoTTamm
diag 131 # CblasNonUnit
M, 2,
alpha complex(1.0, 0.0)
[[complex[] 0.0). complex(0.0, 0.0)].
[complex(0.0, 0.0), complex(1.0) 0.0)]]
B [[complex[1.0 .0, 0.0). complex(2.0, 0.0)].
[complex(3.0, 0.0), complex(4.0, 0.0)]]
Ida, Idb = 2,
result = triangular Bectngular_matrix__._ maatrixssolveorrdee side, uplo, transA, diag, M, N, alpha, A, Ida, B, Idb)
expected mp.arayy(B)
assert ap.allclosffresult expected), T"Test case failed: [result) != (expected)"
# Test case order= 101
side 141
uplo 121
transA
diag 131
M, 3,
alpha = complex(2.0, 0.0)
A [[complex(1.0, 0.0), complex(0.0, 0.0), complex(0.0, 0.0)).
(complex00.0, 0.0), complex(1.0, 0.0), complex(0.0, 0.0)].
[complex(0.0, 0, 0.0), complex(0.0, 0.0), complex(1 (1.0, 0.0)]]
B [[complex(1,, (1.0, 0.0), complex(2.0, 0.0), complex(3.0, 0.0)],
(complex(4.0, 0.0), complex(5.0, 0.0), complex(6.0, 0.0)].
[complex(7.0, 0.0), complex(8.0, 0.0), complex(9.0, 0.0)]]
Ida, Idb 3,
result triangular matrix solve(order, side, uplo, transA, diag, alpha, A, Ida, B, Idb) expected np.array(B) alpha
np.allelose(result, expected), f"Test case failed: [result] != (expected)"
name main__"
assert test()
import ctypes import numpy np
ctypes.CDLL /it.il//iiiillllllg//////iiigggi
mode-ctypes.RTLD. (
gsl ctypes. wwwptssccc.L.CCC CDLL(Vusr/ocal//g///
def triangular matrix solve(order, side, uplo, transA, diag, M, N, alpha, A, B):
eblas ctrsm = gsl.cblas ctrsm
oblas cirsmaarglypes [clypes.c mi, clypes.c int, ctypes.c int, clypes.c int, ctypes.c int,
ctypes.c int, ctypes.c int, PoppesPPINNERtctyps..c.o float),
Prrres.POINTRRctppes... float), ctypes.c int,
ctypes. (cyneettontttccyyes.c..c. float), ctypes.c_ing
cblas tatnssctss.rrsttpeeee. None
A. Ayypes=A.ctypes.uue
float))
"_ctypes B.ctypes.dattt... awscttpes..OINTTEEEEEEEE....EEE POINTER(Ctypes.c_ooll f_ooat)) alpha ctypes = (ctypes.c float (allha..rel, alpha.imag) Ida 1
Idb B.shape[1]
eblas ctrsm(order, side, uplo, transA, diag. M, N, alpha clypes, A ctypes, Ida, B ctypes, Idb) return
Figure 24: Case Analysis: the 205-th task in the CrossPL-FFI. The upper part is the test and the lower part is the incorrect answer generated by the GPT-4o.
48

---

Published as a conference paper at ICLR 2026
Write Python function that clypes load and call specific function from GNU Scientific Library (GSL) compiled dynamic library. The function should perform mathematical operation vector of double precision floating "point numbers. The input to the function should be a list of doubles, and the output should be list of doubles representing the result of the operation. Ensure that libgsicblas.so so is loaded globally before libgsl.so to resolve dopendencies. To accomplish the above tas k. the required classes are: ['GGLLecctor J, and the required functions ulong with their parameter lists are: [('v (('vector add', I'a (bD)).
The recommended dynamic library loading order as follows: ctypes.CDLL (ww.LLLocoooallggeee hibp://alllgg//////ghlcllaaa gsl ctypes. CDLL (oustrollllll// gsl//bbb// httggg/////bbbsl..sse. lib"sl.so"")
mode--ttyes.RRII RILD GLOBAL)
You must return ONLY Y a valid ISON object. Do not include explanations, comments, or extra text. Do not use triple quotes (""").
object must contain exactly one field: "Candidate solution": <istring>
Inside "Candidate solution", provide the complete code as single string. Escape all line breaks as 'n.
import ctypes
A
import numpy as np
moderctypes.RII RTLD GLOBAL)
gsl etypes,CCl. CDLL
class GSLVector(ctypes.Sneee fields -
("size",ctypos.e ctypes.c_size sisssiize__ I).
("stride", ctypes.c size Q),
("data", ctypes.
ctdaet",,ttyps.PPP ettppes.PINTTII WOIINERR(typessccoooo. doublee),
("block", clypes.c_ void_p), PP).
("owner", cotypesscccmm
gsl.gsl vector alloo..rgtyppe [ctypes.c size 1]
gsl.gal vector_ alloocresttpp ctypes. (OSte
gol.gsl vector free.argtypes (etypes. POINTER(GSI Vector)]
gsl.gsl vector free.restype None
gsl. gsl vector set str_all.rggtypss= PotypeesPONNTRRGGSSLNeee ctypes.c crypes.c_doublee double] gsl.gsl vector
all.restype None
gsl. gsl vector add.argtypes (etypes. POINTER Vector) P Vector) gsl.gsl vector redyrestype etypes.e_int
def vector_ add(a, b): n len(a) assert len(b) n, "Vectors must same length" a- gsl.gal. vector alloc(n)
b gsl.gsl. vector alloc(n)
result gsl.gsll vector alloc(n)
for in range(n):
vee a.contents dataji] - alil
vee b.contents. - b[i]
error code gsl.gsl_ vector add(vec_d,.... voc_b)
I'error !-
raise RuntimeErron" "GSL vector addition fuiled with code; (1" (eroot code ))
result- - [vec a.comtents.aataji for in range(e))
return result
finally:
gilggsl vector free(vec__) a)
gsllggl vector free(vec b) gsl.gsl vector freefvec result)
Figure 25: Case Analysis: the 257-th task in the CrossPL-FFL. The upper part is the task instruction and the lower part the Canonical solution.
49

---

Published as a conference paper at ICLR 2026
def ussert inseert]est@)::
Test case I: Simple addition
a [1.0, 2.0, 3.0]
b [4.0, 5.0, 6.0]
expected - [5.0, 9.0)
assert vector_deee add(a, b) - expected, "Test case I failed"
Test case 2: Zero vector
[0.0, 0.0, 0.0]
b [1.0, 2.0, 3.0]
expected - [1.0, 3.0)
ussert vector add(a, b) - expected, "Test case 2 failed"
Test case 3: Negative numbers
[-1.0, -2.0.-3.0]
b- [1.0, 2.0, 3.0]
expected = [0.0, 0.0, 0.0]
ussert vector add(a. b) expected, "Test case 3 failed"
print("All test cases passed!")
if name main ":
TEST
assert test()
import ctypes
import numpy as np
class GSLVector:
def' init (self, data):
self.data (daaaa floa=1n))
self.size len(self.data)
self.c aaray swwf. dat..daaaa.ttpes.aata akCtypes.POINTER(cyyp.... (dtpcltcttppsccuuuull))) def vector_ add(a,dda b):
if len()) !- len(b):
raise Valucto must be of the same length ")
http:ii//iiigslbbb.ss.... so"", mooee-ctype..RI RTLD GLOBAL)
gsl =
gsl.gsl_ vcctor add.argtypes
Cypee.PPoNTERRecypss..o c
ctypes. (dcull
SyypesPPoNTEERcctyes..cc coublle) ctypes.c_size size_iie______
gsl.gs| vector add.restype None vec GSL
vec b GSI Vectorc)))())
result ap..zerosvvee a.size, dtype-np.foat64
result array result.ctypes.dat
PONNTEEc.P.. double))
gsl.gsl vector add(vec a.c. array, vec vec_ b.c b.c_arra,, result e array, ctypes.c size t(vec_ a.size )) return result.folis(()
Figure 26:
Case Analysis: the 257-th task in the CrossPL-FFL. The upper part is the test cases and the lower part is the incorrect answer generated by the GPT-4o.
50

---

Published as a conference paper at ICLR 2026
Function Descriptin: function estab ishe USP connection request recelves respenso.
Descritton
using
minimal wals
reseived data standard ougut. It usea Boost-Asio for anynchrenous UCP openationa.
Input:
std:stiing
the USP connect
None (yold function)
recetved standard
Note: The Landion uses foord serv ias nanme "chyrima" (port 15) for detraratiation purpossss In real auppl candoon, Bhis woulld sypkaell be contligumabi. The function handles iPul and UDP protocol operations.
Mindude <boost/asio.hge hpp> #include <icstream>
open think mode
dincludo abeostlasic hopp dinckude costreame
close think mode
vold udp clllnt strings host) namesspace asio boost:asio using udp asic:tb-udp::
vold sdpp cleeoccanst st string host) content;
conteety
type salp-s4f) "saytime");
asiocioo context selo:lo_contextt io, context: udp:ressolver resulvent to comtextl
auto
resaierrresevvehost, "daylime"),
boostrasto: nocket sncsetjo, contextyy
(endpaints amply()
socket open (bocet netto toruup wigh). requeet - "Requast"
udp::socket sockettio contexti; sockke, commeci)) enddpoints begin()):
sodket send kolbosttaaioo beoostarioobbutteeeqeueet "endpoints. begin0):
senkkttsendjassioobboe buffer"" O0% char reev. buff1024;
char raply 22);
brost:aado: bpcudpeenddee endpoit sender, enddointt
size, lon = socket http:attrroollleaaaoooobuuabue
buf):
srze len socket heeee, trambboost asto: www..tt buteerpplyyl sender_eendeeeeeeeeeeeeeeeeeeeee enndomtt wod-couttffainne wrhaarpppy, lent:
std:couu wwleerwwyybeeeee buu, len)
sodccoot stdcandi:
Case Analysis: the 37-th task in the CrossPL-IPCC (C++). The left part is the incorrect answer generated by the Qwen3-235b-a223 with think mode. The right part is the correct answer generated by the (wen3-235b-a22) without think mode.
Figure 27:
51


Reviews:
[{"review_id": "avGCN6751o", "reviewer": "Reviewer_DQS3", "summary": "This work presents a cross-programming-language benchmark designed to systematically evaluate large language models (LLMs) on cross-language code generation tasks. The benchmark includes 2,534,982 interprocess communication tasks across six programming languages, as well as 522 Python–C foreign function interface (FFI) tasks. The study is motivated by the growing prevalence of real-world systems that integrate two or more programming languages. Through extensive experimentation, the authors provide insightful analyses of LLM performance, highlighting differences in capabilities across languages and programming techniques.", "questions": "* Benchmark design: Multi-programming-language tasks can often be represented within repository-level benchmarks. Why do you believe your benchmark design offers a superior approach compared to, for instance, extending SWE-bench to include GitHub repositories that natively incorporate multiple programming languages?\n* FSM creation: Based on the main paper, I understand that the finite state machines (FSMs) were not generated by LLMs. Could you please confirm this?\n* Manual review details: What was the size of the manually sampled subsets used for human review (referenced in Lines 254 and 287)? Could you describe the review process in more detail and specify how many reviewers participated?\n* Filtering and model accuracy: Could you provide detailed percentages of instances that were filtered or pruned at each LLM-judging step? Additionally, do you believe that smaller or less powerful LLMs could achieve comparable accuracy on these tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The benchmark is large in scale, with the IPC task subset covering multiple real-world repositories and the FFI subset derived from a high-quality open-source repository.\n* The benchmark construction pipeline is fully automated, leveraging established software engineering methods to ensure correctness.\n* Although the benchmark is synthetically generated, the authors validate correctness through stochastic verification, including compilation and execution tests and a comprehensive suite of test cases.\n* The evaluation spans a diverse set of LLMs, encompassing both proprietary and large open-source models, providing a balanced comparative analysis.", "weaknesses": "* The manuscript employs an excessive number of acronyms within individual paragraphs across multiple sections, which negatively affects readability. Including a dedicated background or terminology section early in the paper would help introduce key terms and abbreviations, improving overall clarity.\n* The literature review omits several major code benchmarks, including SWE-bench, Multi-SWE-bench, and MBPP, which are essential for contextualizing the contribution.\n* Although a manual review was conducted, the methodology and outcomes are not reported, limiting transparency and reproducibility.\n* The pipeline’s performance appears to rely heavily on large, high-capacity LLMs, which may limit its reproducibility and scalability when using smaller open-source models.\n* Despite identifying outliers, most LLMs achieve similarly high scores on both IPC and FFI tasks, suggesting the need for further task annotation and filtering to better capture differences in task complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SazfepsXqu", "reviewer": "Reviewer_LuTN", "summary": "This paper introduces CrossPL, the first benchmark for systematically assessing LLM performance in Cross Programming Language (CPL) interoperating code generation.", "questions": "- Was there any manual review process for the 156 FSMs? Could insufficient coverage of the FSMs lead to an underestimation of model performance?\n\n- For IPC tasks, the FSM validation focuses on state transitions, but does it overlook performance aspects (such as latency or concurrency) in the evaluation?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The writing is fluent and logically clear.\n﻿\n- The dataset and evaluation code are open-sourced, facilitating replication of the work and future use by the community.\n﻿\n- The first benchmark for CPL interoperating code generation\n﻿\n- It highlights the widespread inadequacy of large models in this task, providing a new perspective for improving LLM capabilities.\n﻿\n- The benchmark is large-scale and provides a methodology for automated benchmark construction.\n﻿\n- The use of FSM-based validation simplifies the evaluation process.", "weaknesses": "- CrossPL-FFI is limited to Python–C tasks, and even focuses on the GSL library, raising concerns that the conclusions might be overly assertive or not generalizable.\n\n- There remains a gap between FSM-based validation and actual execution, which may affect the accuracy of performance assessment.\n\nTips\n\n- The abbreviations IPC and FFI are not fully defined upon their first appearance in the abstract, which could hinder readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IythNsWfJB", "reviewer": "Reviewer_aE3T", "summary": "CrossPL introduces a benchmark for cross-programming-language (CPL) interoperating code generation, covering two modes: IPC (1,982 tasks across 6 languages and 7 techniques) and Python↔C FFI (522 tasks). The benchmark is largely auto-constructed with 156 FSMs plus two LLM-driven workflows (for mining/templating IPC snippets and for building/validating FFI tasks from GSL). Evaluation of 20 LLMs shows stark gaps versus single-language coding: best Pass@1 is ~80%+ on some IPC slices but only 19.5% Pass@1 on FFI (GPT-4o), highlighting interoperability as an unsolved frontier. The paper also provides error taxonomies and an analysis of “think” mode (helpful for FFI, mixed for IPC). Overall, this is a timely benchmark with solid engineering, though several choices (Python-C-only FFI, FSM-only IPC validation, LLM-generated instructions) raise concerns about coverage, validity, and potential bias/contamination.", "questions": "PC validation fidelity: What fraction of IPC tasks are executed end-to-end vs only FSM-matched, and what are the FSM false-accept/false-reject rates?\n\nContamination/time split: Did you create a time-based holdout (repos/docs post-training) to estimate training-data leakage and its impact on scores?\n\nFFI generalization: Why restrict to Python↔C (GSL)? Any pilots for JNI/Rust/Node add-ons/CFFI—and a plan to broaden FFI coverage in v2?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear problem framing: Shifts evaluation from translation/single-language coding to true cross-language interaction.\n\nScale & breadth: 2,534 tasks; IPC spans 6 languages × 7 techniques; rare in this area.\n\nMethodological novelty: Use of 156 FSMs to mine/validate IPC protocols is systematic and reusable.", "weaknesses": "FFI narrowness: FFI limited to Python↔C (GSL); may not generalize to JNI, CFFI, Rust FFI, JS↔C++ addons, Swift bridging, etc.\n\nIPC validation fidelity: FSM conformance ≠ functional correctness; many IPC tasks are not executed end-to-end (risk of false positives).\n\nDataset skew: Language/technique distribution is uneven (e.g., heavy Java/HTTP), which can bias aggregate metrics.\n\nContamination risk: Mining popular repos and widely documented patterns without a contamination analysis could inflate results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BJmtwBLjvl", "reviewer": "Reviewer_re9k", "summary": "This paper presents CrossPL, a benchmark suite for systematically evaluating large language models (LLMs) for generating cross-programming language interoperating code (CPL interoperating code). The benchmark covers two major interoperability mechanisms: IPC and FFI, encompassing 2,534 tasks (1,982 IPC tasks and 522 Python–C FFI tasks). The authors modeled interfaces based on 156 FSMs, mined 19,169 multi-language GitHub repositories, and designed two LLM-driven build and evaluation workflows (one each for IPC and FFI), ultimately evaluating 20 representative LLMs. Key findings: On the FFI subset, the best model achieved only Pass@1=19.54% and Pass@5=26.46%, in stark contrast to the strong performance of single-language code generation, revealing the significant shortcomings of current LLMs in generating CPL interoperating code.", "questions": "- IPC currently defines \"correct\" based on FSM matching. Can you provide end-to-end run pass rates and abnormal scenario results?\n- Whether statistical significance tests (e.g., paired tests/confidence intervals) were performed for the improvement or decrease in Fig. 3/4\n- Why does think mode have a \"limited or even negative\" effect on IPC but is beneficial for FFI? I think you need to provide a mechanistic analysis of the failure example.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The problem definition is novel and important: this is the first systematic focus on \"cross-language interoperable code generation\" rather than translation/monolinguistic operation.\n- The experiments are comprehensive, making good use of FSM formalization, large-scale repository mining, LLM pipelines, and FFI executable environments for experiments.\n- The experimental results can reveal the difficulties of FFI and the heterogeneity of IPC in the cross-model/language/protocol panoramic comparison.", "weaknesses": "- The experiments and methods limit the scope of generalization, making it difficult to prove that the current approach is effective for multiple libraries, platforms, and languages simultaneously.\n- IPC correctness is determined only by FSM matching and runtime robustness is not measured, which lacks important evaluation metrics.\n- There is no systematic exploration of the impact of hyperparameters and decoding strategies on the overall method. Only the impact of fixed temperature and top-p on the experimental method is given.\n- The paper doesn't propose a novel technical path or implementation; its technical paths rely on elements of established, mature implementations. Furthermore, the FFI is limited to Python-C + GSL, and the IPC covers only six languages/seven technology categories, limiting the originality of this \"first system benchmark\" in broad CPL interoperability.\n- The paper's most important conclusion is a negative result (Pass@1 ≈ 19.5% in the FFI subset), but this finding is primarily confirmatory rather than surprising: given the narrow Python-C/GSL setting and the IPC caliber of FSM-compliant simulation, the conclusions are difficult to extrapolate to the broader CPL ecosystem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "10PJnJfIIj", "reviewer": "Reviewer_cZK6", "summary": "The goal of this paper is to build a benchmark to evaluate the ability of LLMs\nto generate code that spans multiple programming languages. It evaluates both\nlanguage interoperability using both IPC and FFI. For IPC, it considers several\ntechniques and language pairs. For FFI, it only considers FFI between Python\nand the GNU Scientific Library (GSL) which is written in C.\n\nThe benchmark has 2,000+ tasks, based on existing GitHub repositories. It\nachieves this scale with LLM-authored prompts.", "questions": "See the weaknesses listed -- there are two primary concerns that I have with the paper.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- An important task to benchmark. To the best of my knowledge, this is not\n  a well-studied task.", "weaknesses": "I really think the paper is on to something by working on this problem. However,\nI am not convinced by the benchmark construction methodology. There are two\nmain weaknesses:\n\n- Very high likelihood of benchmark contamination: the paper starts with\n  existing repositories, which makes contamination likely. I am aware that\n  there are other, popular LLM benchmarks with the same flaw, but that doesn't\n  mean that they are good benchmarks, even if they are very popular.\n\n  To be concrete, consider that the all the Python-C FFI tasks involve the\n  GNU Scientific Library (GSL). Unsurprisingly, there is a well-established\n  Python FFI to the GSL on GitHub: https://github.com/pygsl/pygsl\n\n  So, to what extent is the benchmark just measuring the willingness of LLM\n  developers to train on GPL code?\n\n- Model generated prompts can be unnatural: the problem with an LLM generated\n  prompt is that it can leak very peculiar implementation details unless it is\n  carefully crafted. For example, in my opinion, the prompt on Page 34 (Appendix) \n  literally specifies all implementation details. In fact, the prompt is\n  longer than the canonical solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)