CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilation
Authors: Anirudh Khatry
Robert Zhang
Jia Pan
Ziteng Wang
Qiaochu Chen
Greg Durrett
Isil Dillig
Venue: COLM 2025 (2025)
Keywords: Code translation
Code generation
Software engineering
Large Language Models
TL;DR: We introduce CRUST-Bench, a dataset of 100 C repositories, each paired with manually-written interfaces in safe Rust as well as test cases that can be used to validate transpilation correctness.

Abstract: C-to-Rust transpilation is essential for modernizing legacy C code while enhancing safety and interoperability with modern Rust ecosystems. However, no dataset currently exists for evaluating whether a system can transpile C into _safe_ Rust that passes a set of test cases. 
We introduce CRUST-Bench, a dataset of 100 C repositories, each paired with manually-written interfaces in safe Rust as well as test cases that can be used to validate correctness of the transpilation. By considering entire repositories rather than isolated functions, CRUST-Bench captures the challenges of translating complex projects with dependencies across multiple files. The provided Rust interfaces provide explicit specifications that ensure adherence to idiomatic, memory-safe Rust patterns, while the accompanying test cases enforce functional correctness. We evaluate state-of-the-art large language models (LLMs) on this task and find that safe and idiomatic Rust generation is still a challenging problem for various state-of-the-art methods and techniques. We also provide insights into the errors LLMs usually make in transpiling code from C to safe Rust. The best performing model, OpenAI o3, is able to solve only 19 tasks in a single-shot setting. Improvements on CRUST-Bench would lead to improved transpilation systems that can reason about complex scenarios and help in migrating legacy codebases from C into languages like Rust that ensure memory safety. Code and Data available [here](https://github.com/anirudhkhatry/CRUST-bench).

Published as a conference paper at COLM 2025
CRUST-Bench: A Comprehensive Benchmark for C-to-safe- Rust Transpilation
Anirudh Khatry
Robert Zhang
Jia Pan Isil Dillig
Ziteng Wang**
Qiaochu Chen 9
Greg Durrett
The University of Texas at Austin akhatry@cs wtw.as.eee.oooooo
New York University
Abstract
C-to-Rust transpilation essential for modernizing legacy C code while enhancing safety and interoperability with modern Rust ecosystems. How- ever, no dataset currently exists for evaluating whether a system can tran- spile C into safe Rust that passes a set of test cases. We introduce CRUST- Bench, a dataset of 100 C repositories, each paired with manually-written interfaces in safe Rust as well test cases that can be used to validate correctness of the transpilation. By considering entire repositories rather than isolated functions, CRUST-Bench captures the challenges of translating complex projects with dependencies across multiple files. The provided Rust interfaces provide explicit specifications that ensure adherence to id- iomatic, memory-safe Rust patterns, while the accompanying test cases enforce functional correctness. We evaluate soate-of-the-are large language models (LLMs) on this task and find that safe and idiomatic Rust genera- tion still challenging problem for various state-of-the-a methods and techniques. We also provide insights into the errors LLMs usually make in transpiling code from C to safe Rust. The best performing model, OpenAl 03, is able to solve only 19 tasks in single-shot setting. Improvements on CRUST-Bench would lead to improved transpilation systems that can reason about complex scenarios and help in migrating legacy codebases from C into languages like Rust that ensure memory safety. 1
Introduction
Code translation is essential for modernizing legacy systems, enabling cross-platform devel- opment, and improving software security (U.S. Government Accountability Office, 2022; Alexandrova et al., 2015; Egan, 2022; Giudice, 2024). As memory-safe alternative to C, Rust has gained widespread adoption due to its strong compile-time guarantees that eliminate entire classes of memory bugs without relying on garbage collection. Major companies such as Google, Microsoft, and Amazon have integrated Rust into their infrastructure, and open-source efforts like the Linux kernel and WebRender have embraced it reduce memory safety vulnerabilities.
However, translating C Rust is not just about achieving functional equivalence: it also involves transitioning from non-memory-safe C semantics to memory-safe, idiomatic Rust. While Rust supports unsafe code for low-level operations, the core value of migration lies in producing code that compiles and executes within Rust's s safe subset, allowing users to benefit from Rust's guarantees. Given that much of today's critical infrastructure is still written in C, there is pressing need for reliable automated techniques that support not only C-to-Rust translation, but more importantly, C-to-safe-Rau -uust migration.
Despite rapid progress in large language models (LLMs) for code generation, fully auto- mated C-to-safe-Rus transpilation remains an open challenge. Existing ML benchmarks
Authors contributing to dataset annotation.
I Code and Data available at https:/ https://ggttthubuubu.. http:///ghhhhucccommmmmchhhttttt

---

Published as a conference paper at COLM 2025
CRUST-Bench Instance
tepl Node
[[best]
ve Fintvvluee(
In find value(Bself,
In test FindO {
rode t self,
hyyhh u52, kty: Vecauls, key_len: usize)
saidt tey,
let ni Optionehalues rode, Fini_value(ke)))
$ize_t key_ lar)
curr care right:
Optiondralues eesented
}
[[best]
Itst.vluuss
void list.entries( E self, listy_list_t*
In ontrtes(keelf,
tree.ods(...);
Hist: Beut Entryl untepl (
rewweeentryyOt
tree. entryckey); ...
Source
Rust Interface
Rust lests
Challenges
Type definitions
Transpile
imp1 Kode
pub fn List. antriee) pub fn find_vo lue.vol key hash: u32, key: Ver-ai8,, usize)
Run tests
Generic pointers not alowed Rust
Borrowing safety
Referwnces muat be counted for maintarsing sefety
Options/nluue
Compile
right) sel child right
Wrapping pointer types
return right, okOZ umeraol)
Optiondval urs supports both and Velow types
Synchronization Guarantees
Lock must obtaied to access rece when doaling with Arce: types
Rust Implementetion
The task specification provided by CRUST-Bench, including the C source code (left), a safe Rust interface (middle), and Rust test cases (right). The code implements the find_value function, which traverses B-tree map to locate the value for given key. This implementation relies heavily raw pointers (e.g,, key). In contrast, the Rust interface uses safe, structured types such as Vec<u8>, requiring the transpiler to generate memory-safe, idiomatic Rust. Bottom right: The expected Rust implementation, representing the actual target of the transpilation task. Bottom left: Additional challenges of the transpilation task are highlighted, illustrating the complexity of translating low-level pointer operations to safe abstractions.
Figure 1: Example of a CRUST-Bench task: btree-map, Top:
largely focus on competitive programming problems (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021; Jain et al., 2025; Quan et al., 2025), which emphasize isolated, self- contained tasks rather than realistic, multi-file systems programming, Other efforts like SWE-bench (Jimenez et al., 2024) target
scenarios with localized edits, rather structural refactoring. Prior benchmarks for C-to-Rust Rust
than sthole-program translation transpilation have similarly been limited, typically focusing on individual functions without testing for correctness at the integration level, and with little or assessment of whether the resulting Rust code is safe or idiomatic. We elaborate on these limitations in Section 2.
To address these limitations, we introduce CRUST-Bench, a benchmark for evaluating automated C-to-Rust transpilation in realistic settings. CRUST-Bench is comprised of 100 C repositories, each paired with manually crafted Rust interfaces and test cases to assess the correctness and quality of transpilation. The Rust interfaces serve as formal specifications, defining function signatures, type annotations, and ownership constraints to guide the translation process. These interfaces enforce idiomatic and memory-safe Rust patterns, ensuring that transpiled code adheres to Rust's safety guarantees. The accompanying test cases provide an additional layer of validation, verifying that the generated Rust code correctly implements the specified behavior. Figure 1 provides a concrete example of CRUST-Bench task, illustrating the gap between low-level pointer-based C code and the corresponding safe Rust implementation.
CRUST-Bench is built through hybrid annotation process that combines automated tooling with human expertise. Annotators manually author safe and idiomatic Rust interfaces, along with corresponding test cases, for each C project. These interfaces are then validated via type checking using the Rust compiler to ensure they are well-formed and statically sound. This process ensures that CRUST-Bench provides rigorous framework for assessment.
Using CRUST-Bench, we evaluate 12 frontier LLMs and other models on the task of C-to- Rust transpilation. The strongest frontier reasoning models such OpenAl o3, Claude
2

---

Published as a conference paper at COLM 2025
Benchmark
# Projects
Multi-file?
Avg LoC
Rust Interface?
Rust Tests?
CROWN
20
31.7K
TransCoder-Rune
520
108
FLOURINE
112
68
C2SaferRust
9.3K
SYZYGY
2.5K
LAERTES
17
24K
CRUST-Bench (Ours)
100
958
Table 1: Comparison of C-to-Rust transpilation benchmarks.
Opus 4, OpenAl ol and Claude 3.7 Sonnet perform the best, successfully transpiling 13- 22% of tasks in the one-shot setting and 32-48% of tasks when using repair loop. The strongest open-source system is Virtuoso-Medium--220 (Arcee.ai, 2025), which outperforms the previous state-of-the of-thh--rttttttt distilled Arcee Nova-32B model, although underperforms the closed-source models substantially. We also employ agentic systems such SWE-agent (Yang et al., 2024c) and find that they do not outperform a ggenerate-the-reppee loop. Our error analysis highlights key challenges and directions for future work in LLM-powered transpilation.
Our primary contributions are: (1) We introduce CRUST-Bench, a new benchmark that enables systematic evaluation of C-to-Rust transpilation by incorporating repository-scale projects, annotated Rust interfaces, and sorrecthess-enforcing test cases. (2) We provide an empirical analysis of the performance and limitations of frontier LLMs on this task, identifying key challenges for future research in automated code migration.
2 Motivation and Related Benchmarks
number of benchmarks have been proposed to evaluate source code translation and transformation tasks, including transpilation (Sun et al., 2024), automated debugging (Li et al., 2024b; Jimenez et al., 2024), and safe code generation (Nitin et al., 2025; Eniser et al., 2024). While these benchmarks provide valuable insights, those that focus specifically on C-to-Rust transpilation tend to exhibit important gaps-in particular, limited task scope, lack of robust correctness validation, and insufficient evaluation of memory safety. Many are restricted to small, synthetic examples or overlook whether the generated Rust code adheres to safe and idiomatic practices. Table summarizes the defining features of the most relevant existing benchmarks.
CROWN (Zhang al., 2023) includes 20 C programs that are syntactically transpiled with guidance from Rust's ownership model. While the dataset features multi-file programs, it lacks structured interfaces or annotations to facilitate the generation of safe and idiomatic Rust code. TransCoder (Sun et al., 2024) is a widely used benchmark for training and evaluating neural machine translation models for code. For the C-to-Rust translation task, recent work (Yang et al., 2024a) typically uses subset of 520 C functions. However, these are primarily small, self-contained units and do not represent full projects with complex dependencies rich interfaces. FLOURINE (Eniser et al., 2024) evaluates large language models (LLMs) across 112 tasks derived from two larger real-world C projects. Nonetheless, the individual tasks remain narrow in scope and lack the structural complexity of complete software systems. C2SaferRust (Nitin et al., 2025) introduces curated dataset of seven large- scale C programs (averaging 9.3K lines code), each translated to Rust using combination of LLMss and syntax-driven transpilation tools. Despite the scale of the individual programs, the overall dataset size is too limited to serve as à comprehensive benchmark for general- purpose evaluation. SYZYGY (Shetty et al., 2024) offers single, large benchmark project of approximately 2,500 lines of code, fully transpiled to Rust, emphasizing correctness and idiomatic usage. LAERTES (Emre et al., 2021) provides small collection of C programs for evaluation but lacks corresponding Rust implementations and interface specifications, limiting its utility for assessing safe or idiomatic transpilation.
3

---

Published as a conference paper at COLM 2025
Compared to prior benchmarks, CRUST-Bench includes substantial number of real-world projects while keeping them within reach for LLM-guided transpilation techniques. With 100 repositories and an average size of 958 lines of code, it captures the challenges of migrating real-world software without requiring models to handle excessively large codebases. A key feature of CRUST-Bench is that it provides safe Rust interfaces and corresponding test files. This is important because existing transpilation tools, such as c2rust (Immutant Inc.), often produce naive translations that rely heavily on unsafe constructs, sidestepping Rust's safety guarantees rather than properly adapting C code. LLM-based approaches also risk Seerraressa generating inconsistent interfaces across files, making it difficult to produce coherent, working Rust codebase. By defining explicit Rust interfaces and tests, CRUST-Bench forces models to generate code that integrates cleanly with a well-defined target.
3 Benchmark Structure
Each instance in our benchmark is built on a C repository, which we formalize as collection of source files S (S, where S represents an individual C source file. The goal of transpilation is to produce a corresponding Rust repository R = R, with parallel file structure, ensuring that each R; is direct translation of the corresponding S,.
Validation of the transpiled Rust code is based on three criteria. First, it must conform to a well-defined interface I ((,),,..,,,),, (fi...., fm),), which specifies set of n abstract datatypes (e.g. struct, enums) and m functions. These interfaces enforce type constraints and function signatures to steer C-to-Rust transpilers to produce code that follows Rust's safety and ownership model. Second, the transpiled Rust code must compile without errors, meaning it must successfully type check and pass all borrow checker requirements enforced by the Rust compiler. Third, we provide a set of tests T (ft...,, fx),, where each test is a function 4; (R) + {pass, fail) that checks whether the transpiled code satisfies functional correctness requirements. Transpilation is considered successful if (1) R conforms to I, meaning all required functions are implemented with the appropriate types and ownership semantics, (2) R builds successfully and (3) all tests pass, i.e., 1; (R) = pass for all
PL infra
Benchmark sourcing. The projects in our benchmark are sourced from open-source repos- itories on Hut covering a diverse range of software categories. We consider repositories created between 2005 and 2025 that success- fully compile using GCC 11.4.0 and Clang 14.0.0. As shown in Figure 2, CRUST-Bend' includes projects from several domains, such as program- ming language infrastructure (e.g,, compilers), algorithmic libraries (e.g., solvers), system utili- ties (e.g., shell implementations), and others.
Algorithmic
Data structures
15
System utilities
Networking
23
11
Crypto security
Other
Figure 2: Application types.
Code Properties
Avg
Max
Preprocessing, To construct CRUST-Bench, we apply multi-stage preprocessing and selection pipeline. First, we filter for projects that are entirely written in and meet minimum complexity thresh- old: they must contain at least one dynamic memory allocation keyword, include build scripts, and have associated test files. We also perform automated de- duplication eliminate near-identical repositories. Following this initial filtering, we manually review the remaining projects to assess overall code quality and ensure test completeness for example, verifying that critical source or header files are not missing. We also exclude architecture-specffic repositories, selecting only those that can run on both x86 and x64 systems to maintain portability. Finally, we omit GUI-based projects, as they often rely external dependencies, platform-specific event loops, or graphical toolkits that are not easily testable in headless, compiler- and test-driven evaluation setting.
Test cases
76.4
Test files Test coverage
3.0
19
67%
100%
Lines of code Pointer dereferences
958
25,436
264
12,664
Functions
34.6
418
Table 2: Properties of C code
4

---

Published as a conference paper at COLM 2025
Table 2 provides statistics about the C projects selected for CRUST-Bench. These projects exhibit a wide range of codebase sizes, with an average of approximately 958 lines of code. 60% of the projects in CRUST-bench contain single source file with corresponding header files and test cases, and the remaining 40% projects contain more than one source file with associated header files and test cases. The average test coverage* is 67%, showing moderate level of testing across the projects.
Annotation process. A team of four annotators who are authors of this paper manually defined the Rust interface I for each benchmark, ensuring that function signatures and data structures are both safe and idiomatic in Rust. The first annotation step is to convert C custom types, such as structs and enums, into their Rust equivalents. Annotators ensure that all types are native to Rust and adhere to its safety guarantees, avoiding unnecessary unsafe constructs and enforcing Rust's s ownership model. After defining the types, they specify function signatures, including return types and ownership annotations. Instead of providing implementations, they replace function bodies with unimplemented! () to allow the Rust compiler to validate the interface independently.
Using these interfaces, annotators then construct Rust test files by adapting existing C tests. The test code invokes the corresponding Rust functions, ensuring that it correctly uses the defined interface. While the test files may include unsafe code for validation purposes, the interfaces remain safe and idiomatic.
Finally, annotators compile the annotated Rust code to verify that all function signatures and test invocations are well-formed. On average, each annotator spent around an hour and a half per benchmark, and the first author manually reviewed all benchmarks for correctness. To check if the interfaces were implemented correctly, we did a pilot study where we implemented 20 benchmarks adhering to the interfaces and ran tests to see they passed. We sampled long and short single-file projects, as well as projects containing multiple files with interddependsncies We also ensured that we included one sample from each domain from Figure The annotators were able to successfully implement the projects with the desired functionality and pass all provided test cases.
Evaluating the dataset. Our final benchmark consists of 100 C projects annotated with corresponding Rust in- terfaces, capturing wide spectrum of interface complexity and idiomatic Rust features, as summarized in Ta- ble 3. The dataset includes a total of 3,085 interface functions across 299 in- terface files, with an average of 30.9 functions per project. Many of these functions involve rich type-level struc- ture: 44% use custom types as argu-
Metric
Total
Avg
Max
Interface Structunt
Interface files
299
3.0
Interface functions
3,085
30.9
415
Function arguments
5716
57.2
Ownership and Type Features
Percent
Functions with reference args Custom types arguments
56%
44%
% Custom types in return types
50%
Functions with mutable references
30%
Table Statistics of the Rust interface.
ments, and 50% return custom types, reflecting reliance on user-defined data structures. These types often require associated methods and trait implementations to enable idiomatic usage within Rust. Ownership is also prominently featured-56% of functions use reference arguments, and 30% involve mutable references- demrnddngg precise handling of Rust's borrowing and mutability semantics.
Experimental Setup: Models and Systems Evaluated
Prompted LLMs. evaluate both closed-source and open-weight large language models (LLMs) on the CRUST-Bench benchmark. For closed-source models, we include o3, ol, ol- mini, and GPT-4o from ApenAT; Claude Opus (laude-37--oonnee and Claude-3.5-Sonnet from Anthropic; and Gemini-1.5-Pag from Google. On the open-weight side, we evaluate models with strong code generation capabilities: Virtuoso (Arcee.ai, 2025), distilled variant
We compute code coverage metrics over 25 repositories and discuss challenges of computing code coverage in Appendix A.
5

---

Published as a conference paper at COLM 2025
Ce--Codee--2--nssruut(((( et al., 2024b); Deepseek-Coder Serpssee--Coer-----nnsrvwee -3m---tttnnmtt et al., 2024); and LLaMA-3-8D (Llama-3
of DeepSeek-V3; QwQ-32B PPreview(Yang et al., 2024b);
Team, 2024). Open-weight models are executed using the vLLM inference framework (Kwon et al., 2023) on a server equipped with four NVIDIA A40 GPUs.
For models that support temperature control (e.g., GPT-40, Claude, Gemini), we use greedy decoding (T = 0) to ensure deterministic outputs and consistent pass@1 evaluation. Empiri- cally, we found that higher temperatures significantly degraded performance, likely due to the long-co long-context requirements and stringent correctness constraints of the tasks. On average, each model generates 5,165 tokens per task.
We experimented with variations in prompt formatting to identify the most effective strategy. The best results were obtained using a general task description followed by concise, point- based instructions. Notably, explicit directives such as "do not use libc" and "the code must compile" -rroved more effective than vague statements like "generate safe Rust." " Each model is provided with the full C source file, the corresponding header file (if available), and the Rust interface specification and tests. We provide the prompts in Appendix C.
LLM Scaling via Self-Repair. We investigate two strategies for leveraging additional compute to improve model performance through iterative self-repair. Both approaches begin with an initial -odel-generated candidate and refine it over multiple rounds based on feedback. The first strategy, referred to as Compiler repair, incorporates only compiler error messages into the prompt during each repair round. This allows the model to iteratively address syntactic and type-level issues flagged by the Rust compiler. The second strategy, called Test repair, extends this by also including information about failing test cases in the prompt, providing richer feedback about the correctness of the generated code.
For both strategies, we perform three rounds of self-repair per task. This bounded budget balances the benefits of iterative refinement with practical runtime constraints. In a pilot study on 10 benchmarks, we evaluated different configurations and found that starting with a single initial generation and applying greedy decoding across all repair rounds consistently outperformed sampling-baset strategies, including the approach proposed by Olausson et al. (2024). In particular, we observed that deterministic repair with feedback led to more stable improvements, especially under the long-context, high-precision demands of the C-to-Rust transpilation task. Based on this finding, we adopt the single-candidan greedy repair strategy for our full evaluation. We describe further in Appendix B.
Pipelined SWE-agent. Recent work on agent-based systems, such as SWE-agent (Yang et al., 2024c), has demonstrated strong capabilities in iteratively debugging and refining code by leveraging compiler feedback within isolated development environments. Notably, the combination of SWE-agent and Claude-3.7-Sonnet achieves state-of-the-an performance on the Full SWE-bench dataset as of May 2025.3 Although these systems are primarily designed to address GitHub issues apply localized patches in large codebases, their ability to interpret and act on compiler and testing errors makes them a promising candidate for supporting C-to-Rust transpilation.
To evaluate this potential, we adapt SWE-agent into a two-stage workflow we refer to as pipelined SWE-ngent. In this setup, an LLM (e.g., Claude or GPT-4o) first generates an initial Rust implementation from a C source file and its associated Rust interface. SWE-agent then attempts to repair any resulting compiler errors by iteratively editing the generated code. To support this workflow, we configure SWE-agent with a Docker environment that includes the Rust toolchain (cargo, rustc) and Python, and we supply a custom problem specification and demonstration tailored to transpilation, following guidance from the SWE-agent documentation.' All other parameters are left at their default values. This design allows us to evaluate SWE-agent as a targeted post-processing and repair mechanism applied on top of L.L --M-geeertted code.
We also investigated the feasibility of using SWE-agent (Yang et al., 2024c) and OpenHands' AcdedAct agent (Wang et al., 2025) for full end-to-end transpilation- iee, supplying C
http::::///wwwswwbenchhhcc

---

Published as a conference paper at COLM 2025
Model
Pass81
Pass Compiler
Pass81 + Test repair
repair (x=3)
(r=3)
Build
Test
Build
Test
Build
Test
OpenAA c3
35
19
68
31
63
48
Claude Opus
45
40
OpenAl ol
32
15
69
28
54
37
Claude
26
13
23
49
32
Claude 3.5
26
38
24
ol-mini
19
9
47
16
27
21
GPT-4o
18
Gemini 1.5 Pro
11
35
11
30
14
Virtuoso (Distilled Deepseek V3)
2
6
6
Codderr-Co
0
2
0
QwQ-32B QwQ-328-Prewieww
Qwen-225-Coder-228
Adapted SWE-agent (Claude-3.7)
41
32
Table 4: Pass rates on CRUST-Bench for different models in single-shot and repair settings.
repository input and expecting complete Rust output. However, experiments revealed that SWE-agent exhibited brittle and unreliable behavior in this setting. Furthermore, developer of OpenHands confirmed via personal communication that such usage falls outside the intended design and unlikely to yield satisfactory results. Consequently, we focus evaluation on pipelined workflow, which demonstrated significantly greater robustness and effectiveness in our experiments.
Excluded Baselines. We do not include syntax-directed C-to-Rust transpilers (Immutant Inc.; Zhang et al., 2023; Emre et al., 2021) in our evaluation, as these systems are incompatible with our framework. In particular, they frequently generate unsafe Rust code that relies on foreign function interfaces such as 1ibc, thereby violating the safe and idiomatic interface constraints enforced in CRUST-Bench. Similarly, C2SAFERRUST (Nitin et al., 2025), which augments C2RUST with LLM--asee post-processing, defaults to generating unsafe code when the target interface cannot be satisfied. Adapting these systems to support our interface-firee
evaluation would require significant reengineering and is beyond the scope of this work.
5 Results and Analysis
Single-Shot Transpilation Results. The first column of Table 4 reports pass@1 test success rates, capturing model performance in single-shot setting where each model is prompted once per task, without access to compiler or test feedback. Thus, these results evaluate model's ability to generate correct, safe Rust code its first attempt. Overall, performance is low across the board: the best-performing model, Claude Opus 4 from Anthropic, passes test cases for only 22% of CRUST-Bened tasks. We omit results for LLaMA-3-8B, which fails to produce any correct solutions.
Self-Repair Improves Success Rate. As shown in Table 4, applying iterative self-repair leads to substantial gains in both build and test success rates across all models. With three rounds of Compiler repair,
observe various improvements in build success (as high as 37% gain for o1), and in test pass rates (as high as 13%) compared to the single-shot pass@1 baseline. Appendix D qualitatively discusses the effects of repair. Extending this approach with Test repair, which incorporates failing Rust test cases into the repair loop, yields additional improvements in test success between 0% and 17% over the corresponding Compiler repair setting.
However, this gain in test correctness comes at the cost of build stability: we observe 5% to 20% drop in build success when moving from Compiler to Test repair. This degradation can attributed to the fact that incorporating test case information encourages the model to make more aggressive semantic changes, which may inadvertently introduce new compilation errors errors-partine patttsclaallllllyyy borrowing violations or type mismatches that are difficult to resolve without precise static analysis.
7

---

Published as a conference paper at COLM 2025
Model
Config
% projects with error
Mismatch
Borrow
Missing
Unimpl
Trait
Args
Unsafe
OpenAl
base
13
repair
9
2
2
27
4
2
Claude Opus 4
base
28
29
13
14
repair
11
OpenAl ol
base
30
42
13
14
17
repair
Claude 3.7
base
15
10
repair
Claude
24
14
repair
22
16
55
5
ol-mini
base
46
34
40
28
34
repair
21
13
10
29
1
GPT-4o
base
48
33
25
20
22
repair
12
6
Gemini 1.5 Pro
base
40
24
23
33
17
repair
18
8
Virtuoso (Distilled Deepseek
60
33
repair
38
23
17
50
15
Coepseel-Coder-229
hase
36
13
56
22
repair
28
8
59
36
10
QwQ-32B-Preview
base
9
2
94
5
repair
13
92
3.
Qwen-2.5-Coder
base
7
0
32
76
1
0
repair
5
Adapted SWE-Agent
15
8
Table 5: Error breakdown for different models and configurations
Agent-Based Debugging. The last row of Table 4 also presents the results of the pipelined SWE-agent described in the previous section. Compared to the pass@1 baseline (13% test pass rate for Claude 3.7 Sonnet), the pipelined SWE-agent workflow improves performance to 32%, demonstrating substantial benefit compared the single-shot performance. How- ever, this performance matches, rather than exceeds, that of the Claude 3.7 Sonnet + Test repair strategy, which also reaches 32% after Test repair.
We analyze the behavior of pipelined SWE-agent in Appendix E. We see that pipelined SWE-agent is able to successfully navigate files, edit files, invoke cargo build and cargo test to build and test the project, and more. However, does not successfully leverage large numbers of steps to fix errors: many of its steps are simply navigating and reading project files, and most fixed errors are fixed in relatively small number of steps. This result suggests that, while SWE-agent has broader capabilities dynamically choosing when to build and test, and autonomously deciding when to submit solution- it does not leverage these components successfully in current configuration to outperform the simpler Test repair approach.
such as reasoning across files,
Error Analysis. To better understand the failure modes, analyze the distribution of compiler errors Table 5. These errors were clustered together based on a manual inspection of their description using the rustc --explain command. We note that given project typically exhibits many of these error types at the same time. Based on our manual categorization, we determined that common errors include the following:
These errors arise when the generated Rust code has a type mismatch when calling a function from the tests, another file in the project, or incompatible return types.
Mismatch:
Borrowing: These errors arise due to violations of Rust's ownership, borrowing, mutabil- ity, and lifetime rules. As an example, such an error might arise if value is borrowed mutably while it is already borrowed immutably, or if lifetime annotation incorrect or unspecified.
Missing: These errors arise due to access to non-existent (or out of scope) variables.
8

---

Published as a conference paper at COLM 2025
Unimpl: These errors occur due to unimplemented functions incomplete code seg- ments in the generated output. They often arise when the model exceeds its token budget, leading to truncation before all necessary logic is emitted. In some cases, the model may also omit implementations by stating that certain functions can be implemented "similarly", leaving placeholders comments instead of actual code.
Args: These errors are observed when function invocation does not correctly supply the expected number of arguments. (e.g., passes single argument instead of the required three).
Trait-related errors: These occur due to a failure to implement the necessary traits for custom data types. In Rust, user-defined types often need to implement certain traits to function correctly in the language's 's type system, and these errors arise if the expected trait is not implemented.
Unsafe: This type of problem occurs when the generated code uses the "unsafe" keyword (so-called) unstable Rust feature. We note that these types of problems (e.g,, use of "unsafe" keyword) do not necessarily correspond to build failures.
As shown in Table 5, the occurrence of unsafe code is rare. This can be attributed to our prompt design, which explicitly instructs models to avoid using unsafe Rust features.
In contrast, type-related errors (i.e., those in the mismatch and borrow categories) are quite common. This is expected, given the Rust compiler's 's strong static guarantees and strict enforcement of type and ownership rules. These errors suggest that models often struggle to reason precisely about lifetimes, mutability, and type compatibility. The prevalence of such errors highlights the potential of incorporating static analysis signals into the fine- tuning process, which could substantially improve LLMs' ability to generate compilable, type-correct Rust code.
Finally, unimplemented errors are also widespread. These typically stem from models exceeding their output token limits, leading to truncated or incomplete code. This issue especially pronounced for models with shorter maximum output token limits. In some cases, the model may omit implementations entirely, inserting placeholder comments or referring to similar functions without emitting code, or violate the interface definitions further contributing to these errors.
6 Related Work
Code Generation Benchmarks. Prior work has introduced a range of benchmarks for evaluating code generation. Many focus generating Python code from natural language prompts (Chen et al., 2021; Austin et al., 2021; Jain et al., 2025; Hendrycks et al., 2021), typically framed as short, competition-stylle problems with test cases. HumanEval (Chen et al., 2021) has been extended to support additional programming languages (Cassano et al., 2023; Athiwaratkun et al., 2023; Orlanski et al., 2023). Other efforts have explored more realistic settings: incorporating external APIs (Yu et al., 2024), class-level code genera- tion (Du et al., 2023), repository-levv level tasks with inter-file dependencies (Li et al., 2024b), and rigorous testing of HumanEval (Liu et al., 2023a). In contrast, CRUST-Bench evaluates code generation over multi-file C projects with complex dependencies, using code as input rather than natural language, and targets the generation of safe, idiomatic Rust.
Repository-Level Benchmarks. Several benchmarks have explored code generation at the repository level. SWE-bench (Jimenez et al., 2024) evaluates model's ability to resolve GitHub issues using real-world repositories. RepoBench (Liu et al., 2023b) focuses on code completion across entire codebases, while Li et al. (2024a) examine generation across different stages of the software development lifecycle. In contrast, CRUST-Bench shifts the focus from editing or completing code to performing cross-language translation.
C-to-Rust Transpilation. Recent work on C-to-Rust transpilation spans both syntactic and learning-based approaches. Tools like C2RUST (Immutant Inc.) focus syntactic translation, often relying on external libraries like libe and payouu-prsserving annotations such #[repr(C)) to maintain compatibility. However, these approaches do not guarantee

---

Published as a conference paper at COLM 2025
memory safety and frequently produce Rust code with unsafe blocks. Other approaches incorporate Rust's ownership model to improve safety (Zhang et al., 2023), though they often still rely on unsafe code. More recent techniques combine syntactic translation with LLM-based post-processing to improve code quality (Nitin et al., 2025; Yang et al., 2024a). While C2RAFERRUST leverages LLMs to clean up C2RUST output, it falls back to unsafe code when constraints cannot be satisfied. In contrast, VERT (Yang et al., 2024a) uses both LLMs and property-based testing to verify semantic equivalence between the source and generated code. Eniser et al. (2024) also adopt LLMs for direct C-to-Rust generation, validating correctness through equivalence checking.
7 Conclusion
In this work, we presented CRUST-Bench, a benchmark of 100 C projects with target Rust interfaces and Rust test cases. This benchmark allows us to verify LLM-powered transpilation systems according to three criteria: (1) Do they successfully follow the given interface during transpilation? (2) Does the Rust code compile? (3) Does the code pass the provided Rust test cases? Our results show that even the best approach with state-of-the-art LLMs, ApenAl o3 with iterative repair from both compiler errors and test failures, only succeeds on 48% tasks in the benchmark, leaving significant room for future systems to improve.
Acknowledgments
This research was conducted within group supported by the National Science Founda- tion under awards CCF-1762299, CCF-1918889, CNS-1908304, CCF-1901376, CNS-2120666, CCF-2210831,, CCF-2319471, and and by the Defense Advanced Research Projects Agency (DARPA) under Agreement No. HR00112590133, We also thank the All Hands Al team for discussion the OpenHands CodeAct agentic framework applied to the C-to-Rust transpilation task.
References
A. Alexandrova, L. Rapanotti, and I. Horrocks. The legacy problem in government agencies: An exploratory study. Proceedings of the 16th Annual International Conference on Digital Government Research, pp. 150-159. ACM, May 2015.
InternationalCofffeeneee
Arcee.ai. Virtuoso-Lite & Virtuoso-Medium-v22 Distilling DeepSeek-V3 into 10B & 32B small language models (SLMs), 2025. URL https::/huggingfaaceccm www.aaac.cc....eeeaeeeeeeeeeeeee ai/
Virtuoso-Medium-
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. Multi-lingual evaluation of code generation models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevieee wwt.:eeteooooeeeeeeeeee dd-7mmm-eeee7mmmmmmm
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://///// wwtp://wwwwwrg///////.....
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feld- man, Arjun Guha, Michael Greenberg, and Abhinav Jangda. MultiPL-E: scalable and polyglot approach to benchmarking neural code generation. IEEE Trans. Softw. Eng., 4977))3655-3611, July 2023. ISSN 0098-5589, doi: 10.110//SS..00332224466 URL https://doi. org/10. 11t////....1191111 1SE0/SEE..... 2023. 3267446.
10

---

Published as a conference paper at COLM 2025
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastr,, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mo- hammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Πyya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https: /awxiv hwrggabb/2100 0b77231...3
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. ClassEval: A manually-crafted benchmark for evaluating LLMs on class-level code generation, 2023. URL https://arke org/abs/ 2308.0188
E. Egan. Federal IT
modernization needs a strategy and more
money, 2022.
URL https://itif.. http:///ggpplllaatios//////////0 sttpriaatttoneeee---tttte-----e strategy- strratee-aaddmorr-mnnee))
Mehmet Emre, Ryan Schroeder, Kyle Dewey, and Ben Hardekopf. Translating C to safer Rust. Proc. ACM Program. L 5(OOPLLA) October 2021. doi: 10.1145/3485498. URL https://doi, org.0/..rrr//1.0/ 1145////.111//345499
Hasan Ferit Eniser, Hanliang Zhang, Cristina David, Meng Wang, Maria Christakis, Brandon Paulsen, Joey Dodds, and Daniel Kroening. Towards Translating Real-World Code with LLMs: A study of translating to Rust, 2024. URL http:p/://axxv.oorg///a/
11514.
D. L. Giudice. The future of GenAl will rocket fuel modernisation of core legacy systems, July 2024, URL http://wwwwwaatooootrccoo///ooo/
GNU Project. Gcov Test Coverage Program. http:///ggcc...ooooommmmneeossss Gcov-Intro.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence, 2024. URL
abs//401.. .44664
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks' Track (Round 2), 2021. URL https::/ net/fforum2ddss09322H33.
Immutant Inc. C2Rust. URL httpps//gitthhi wttp://unaann///uuus...
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference Learning Representations, 2025. URL Forum?id=chhJJyCC31.
net/
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world Github issues? In The Twelfth International Conference on Learning Representations, 2024. URL
https://openreview wttpeeewwwwweeeoeuneeeeeeee www..et/oorum/dddTaaayyooooo
11

---

Published as a conference paper at COLM 2025
Woosuk Kwon, Zhuohan Li, Styuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposuum on Operating Systems Principles, 2023.
Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, and Kai Chen. Prompting large language models to tackle the full software development lifecycle: A case study, 2024a. URL https: htaps:://arivv org/abs/ 2403. 08604..
Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, and Mengfei Yang. DevEval: A Hamually-annotttte
code generation benchmark aligned with real-world code reposito- ries. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 3603-3614, Bangkok, Thailand, August 2024b, Association for Computational Linguistics. doi: 10.18653/ 2wtp22022...innssa-aaaa www.lloggyoog/222// org/2024. findings-acl. actgssaac..114. 214/.
URL
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation, 2023a. URL https://arxiv.oor//////////
01210.
Tianyang Liu, Canwen Xu, and Julian McAuley, RepoBench: Benchmarking repository-level code auto-completion systems, 2023b. URL
03091.
Llama-3 Team. The Llama 3 herd of models. arXiv preprint ar arx::22407.21788 2024. URL https://arxiv.org/abssss
21783.
Vikram Nitin, Rahul Krishna, Luiz Lemos do Valle, and Baishakhi Ray. C2SaferRust: Transforming C projects into safer Rust with neurosymbolic techniques, 2025. URL
https://arxww.org/bbbs//
14257.
Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Is self-repair silver bullet for code generation? In The Twelffh International Conference on Learning Representations, 2024. URL https: /ttp::///peeeeeeeeeeee net/forum?id= y0GJXRungR.
Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Mal- maud, Jacob Austin, Rishabh Singh, and Michele Catasta. Measuring the impact of programming language distribution. In Proceedings of the 40th International Conference on Machine Learning, ICML/23. JMLR.org, 2023.
Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. CodeElo: Benchmarking competition- level code generation of LLMs with human-comparable elo ratings, 2025. URL https: //arxiv. org/ ab////s//2/00 2201 01257.
Manish Shetty, Naman Jain, Adwait Godbole, Sanjit A. Seshia, and Koushik Sen. Syzygy: Dual code-test C to (safe) Rust translation using LLMs and dynamic analysis, 2024. URL 14234.
http:///axxiv..og/bbbbbs/
Qiushi Sun, Nuo Chen, Jianing Wang, Xiang Li, and Ming Gao. Transcoder: Towards unified transferable code representation learning inspired by human skills, 2024. URL http:///aaxxxv.o///bbbss//
07285.
U.S. Government Accountability Office. Information Technology: Agencies need to develop modernization plans for critical legacy systems, 2022.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan
12

---

Published as a conference paper at COLM 2025
Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for AI software developers as generalist agents. In The Thirteenth International Conference Learning Representations, 2025. URL https: /ttps://oppnneeveww. (erum?id=OJdaaylle
Aidan Z. H. Yang, Yoshiki Takashima, Brandon Paulsen, Josiah Dodds, and Daniel Kroening. VERT: Verified equivalent Rust transpilation with large language models as few-shot 18852.
learners, 2024a. URL httt::::/aaxxi..og//bb//////
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arviv preprint #rXio:240710067, 2024b.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In The Thirty-eighth Annual Conference on Neural Information Process- ing Systems, 2024c. URL
http:/rrreeiee.eeeooooommmmtrmee met//forumm1d===xooottt3333.
Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. CoderEval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE '24 New York, NY, USA, 2024. Association URL
for Computing Machinery. ISBN 9798400702174. doi: https://doi.org// 1145/1199503..... 33633999999333363666.
Hanliang Zhang, Cristina David, Yijun Yu, and Meng Wang. Ownership guided to Rust translation. In Constantin Enea and Akash Lal (eds.), Computer Aided Verification, pp. 459-482, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-37709-9
13

---

Published as a conference paper at COLM 2025
Appendix
A Code Coverage Analysis
We provide additional details on the code coverage analysis conducted over the reposito- ries in CRUST-Bench. Coverage was measured using gcov (GNU Project) on a subset of 25 benchmarks, selected from projects that successfully built using either make or cmake. For cmake-based builds, some projects linked external files that caused gcov to fail; we did not attempt to resolve these issues across the full benchmark.
Our manual inspection of benchmarks with low coverage revealed common patterns: (1) driver files that invoked core functionality but were themselves untested, and (2) files included for illustrative purposes -ucch usage examples demonstration scripts- contain calls to library functions but are not executed as part of the test suite, and therefore appear as uncovered in coverage analysis.
In the context of transpilation, we found that models rarely introduced localized errors. For example, when a benchmark included several related functions relying on the same abstract data type, the model typically succeeded or failed on all of them collectively. This suggests that the available test suites, while not exhaustive, are sufficiently representative for evaluating the correctness of transpiled Rust code.
Achieving significantly higher coverage would require writing targeted tests for each function in every benchmark an effort that entails deep understanding of program intent and edge cases, and is therefore outside the scope of this work.
B Greedy self-repair vs sample-and-repair
We describe in more detail our comparison between greedy self-repair vs. sampling multiple outputs from model and repairing each (balancing the total number of repairs). These also compared in Olausson et al. (2024), but in a different setting. Both techniques receive feedback in terms of compiler messages.
We conducted pilot study with 10 tasks in CRUST-Bench, consisting of both single and multiple file projects with a model calling budget of per task. The budget was selected to provide fair comparison between the two techniques. For the sumple-and-repal technique, we sample candidates at a temperature of 0.8, used in Olausson et al. (2024), and repair each once. For greedy self-repair, we greedily sampled a single candidate and repaired it up to 5 times with temperature of 0. We tested this with Claude-3.5 and GPT-4o. For both LLMs, we found that greedy self-repair outperformed cample-and-reant We attribute this to two factors: (1) The compiler may not surface all errors at once and multiple iterations of self-repair are required to address all errors, and (2) addressing errors might introduce new errors, which may require fixing in subsequent iterations.
For our full experiments, we found that 3 rounds of greedy self-repair were sufficient. Beyond three rounds we did not see substantial improvements.
14

---

Published as a conference paper at COLM 2025
C
Prompts
Prompt for Transpilation
You are an expert at converting To Rust.
You will be provided with C source files in the format:
{(filename.c filename.h>> h33
// Input C code
need you transpile the provided code files from C to Rust, with the following instructions that you MUST follow:
Each C file I provide MUST have the corresponding transpiled Rust file. You will also be given the Interface files with function signatures and return types. You MUST conform the specification given in the interface definitions.
Each transpiled Rust file MUST compile.
The transpiled Rust code MUST be observationally equivalent to the C code.
The transpiled Rust code MUST compile successfully. The transpiled Rust code MUST NOT contain Foreign Function Interface calls, such as the libc library.
The transpiled Rust code MUST NOT contain unsafe blocks.
All imports in the rust project (except main.rs) MUST be in the following format
rust
use crate: file_name: :
Imports main.rs rs should done the following format: **"uut
use project_name name:_nmmmmmmmm :odulee;
You MUST ensure that you include the required files that are referenced in each rust file.
Please think step-by-step and return your final solution for each transpiled file in the following format:
(ffilename.rs))
r*rust
Generated Rust Code
15

---

Published as a conference paper at COLM 2025
Prompt for Compiler Repair
You expert at repairing Rust Code. You will be provided with Rust code in the format: {(filename.rs]}
rrrsttt // Input Rust code
After the Rust code, you will be given the errors obtained from the compiler corresponding to the Rust code.
I need you to repair the provided Rust files,
with the following instructions that you MUST follow:
You MUST produce the entire file when repairing it and
not just the intended change.
You MUST NOT change the function signatures.
You MUST address each error by reasoning about it.
Each error MUST be solved using safe Rust code.
The transpiled Rust code MUST compile successfully.
The transpiled Rust code MUST NOT contain Foreign Function Interface calls, such as the libc library.
All imports in the Rust project must be in the following format -*rrust
use crate: file_name: :module;
You MUST ensure that you include the required files that are referenced in each Rust file.
You MUST ensure not to change the function signatures and return types of the functions when you performing repairs.
Please think step-by-step and return your final solution for each transpiled file in the following format:
{ffilename.rs)} rrust
// Generated Rust Code
16

---

Published as a conference paper at COLM 2025
Prompt for Test Repair
You are an expert at Rust.
You will be provided with Rust code in the format: {(filename.rs] rust
// Input Rust code
After the Rust code, you will be given the test files that were executed on the Rust source code in the format {[filename.rs]] "rust // Input Rust code
After that, you will be provided with the test failures obtained from the compiler corresponding to the Rust code.
need you repair the provided Rust files, with the following instructions that you MUST follow:
You MUST produce the entire file when repairing it and not just
the intended change.
You MUST not change the test code all. You must only make fixes the
Rust source files.
You MUST NOT change the function signatures.
You MUST address each failure by reasoning about it.
Each test failure MUST addressed using safe Rust code.
The corrected Rust code MUST compile successfully.
The corrected Rust code MUST NOT contain Foreign Function Interface calls,
such the libe library.
All imports in the Rust project must be in the following format **ruus
use crate: ffiee_ sariieeeneeeemeeeee :mddule;;
You MUST ensure that include the required files that are referenced in each Rust file.
You MUST ensure not to change the function signatures and return types of the functions when you are performing repairs.
Please think step-by-step and return your final solution for each transpiled file in the following format:
((filename.rs))
rruust
// Generated Rust Code
17

---

Published as a conference paper at COLM 2025
System instruction for SWE Agent
SETTING:
You are an autonomous programmer, and you re working directly in the command 1ine with special interface. Your task pertains to solving errors in repositrry with Rust code.
You must solve the problem by adding Rust code the repository. You can use the special interface to navigate and edit files. You can also use any bash commands to help you solve the problem.
You are provided with the Rust files and the associated Rust test files. Your implementations should go in the src directory. You can use the "cargo build" command for building the project and caargo test" for running the tests.
The special interface consists of a file editor that shows you {[WINDOW)} lines of file at time. In addition to typical bash commands, you can also Use the following commands help you navigate and edit files.
COMMANDS:
[[comand_ddoss]]
RESPONSE FORMAT:
Your shell prompt is formatted as follows:
(Open file: <path>) <cwd>
You need format your output using two fields: discussion and command. Your output should always include _one_ discussion and _one_ command field EXACTLY as in the following example:
DISCUSSION
First, I'll start by using 1s see what files are in the current directory. Then maybe we can look at some relevant files to see what they look like.
1s -a
You should only include a *SINGLE* command the command section and then wait for response from the shell before continuing with more discussion and commands. Everything you include in the DISCUSSION section will be saved for future reference.
If you'd d like to issue two commands at once, PLEASE DO NOT DO THAT! Please, instead first submit just the first command, and then after receiving response, you' 11 be able to issue the second command.
You'l re free to use any other bash commands you want (e.g., find, grep, cat, 1s, cd) in addition to the special commands listed above.
However, the environment does NOT support interactive session commands (e.g., python, vim), please do not invoke them.
18

---

Published as a conference paper at COLM 2025
Problem statement provided to SWE Agent
You are provided with a Rust project that must pass all the tests. In case the code contains errors, you must repair the provided Rust files such that all test cases pass.
You first run "cargo build" to determine what errors exist in the project. In case you don't t find any errors, you try running tests defined in the project using cargo test". case you find any errors when executing ccargo build" and "cargo test', you must address them by changing the Rust code in the src/ folder the project. Incase you don't t find any errors, you submit the project.
You must ensure that you follow the following instructions:
You must not change the function signatures and return types of the functions when you are performing repair on a file.
You must address each error by carefully reasoning about it.
Each error must be solved using safe Rust code.
The transpiled Rust code must not contain Foreign Function Interface calls,
such as the 1ibc library.
All imports in the Rust project must be in the following format "*"rust
use crate: file_name: :malluleee::oduee;
You must ensure that you include the required files and constants that are referenced in each Rust file.
You must ensure that you do not change the test code in the src/bin folder.
Please think step-by-step and resolve the issue.
D Analysis of the Self-Repair Pipeline
We observe that self-repair is effective at mitigating a range of errors encountered during the initial transpilation pass. For Claude 3.7 Sonnet in particular, we note the following trends:
Unimplemented functions and imports: The model generally fails to recover from unresolved function and import enrors during self-repair. This limitation arises from the fact that the LLM is more adept editing existing code than synthesizing entirely new logic during repair rounds. In 2% of benchmarks, the self-repair process increased the number of unimplemented functions, further exacerbating these errors.
Borrow-checker errors: Borrowing-related errors reduced by 75% after applying compiler-guided repair, suggesting that these errors are relatively tractable when explicit compiler feedback is available.
Trait-related errors: Errors related to missing trait implementations are reduced by 90%, indicating that trait errors are particularly amenable to iterative correction.
Type mismatches: Type errors cut by 50% in the first round of repair but plateau in subsequent rounds, suggesting diminishing returns on further iterations for this category.
Mutability issues: We observe that attempts to type and borrow errors sometimes introduce new mutability-relan issues, which may may not be corrected in later repair passes.
While self-repair improves many error types, we also identify cases where it fails to yield performance gains or introduces new issues. In particular, observe that attempts to address specific errors often cascade into other categories of failure.
For example, in the multi-file project impcheck, the model initially omits the hash. rs file containing the core HashTable implementation. In the first repair round, it adds hash-related functions but omits required Clone trait implementation. The second round corrects this omission, but introduces new error: the compiler reports that the capacity method is missing. In the third round, the LLM adds this method, but the implementation mishandles
19

---

Published as a conference paper at COLM 2025
ownership, leading to borrow-checker violations. This case illustrates how self-repair can make steady progress, yet ultimately fail due to deeper semantic issues related to Rust's ownership model.
project. The initial translation includes borrowing errors tied to custom Graph type defined in the library. Despite multiple repair attempts, the model is unable to satisfy the ownership constraints required for the correct use of this type. Additionally, the model fails to incorporate custom macro used for hashing graph nodes-hiighlighting its difficulty in reasoning about domain-specific constructs that are not easily inferred from compiler error messages alone.
similar pattern appears in the
E Results using SWE-Agent
Characterization of SWE-Agent Behavior. The SWE-Agent framework consists of an orchestrating LLM that executes sequence of structured actions to iteratively generate patches for target repository. A key question in our analysis is: what types of actions does SWE-Agent take in attempting to fix transpilation errors?
According to Table in the SWe-Agent paper (Yang et al., 2024c), the system can invoke standard Bash commands in addition to eleven custom actions: seven related to file viewing and search, three dedicated to editing, and one final action to submit the current patch for evaluation. In case, evaluation involves compiling the project and running the associated test suite.
From qualitative analysis, we observe consistent pattern in SWE-Agent's behavior:
It begins by exploring the repository, viewing both source and test files using a combina- tion of Bash navigation and custom file-view actions.
Once sufficient context is tythered-typically after observing all relevant files it issues the first cargo build command. On average, this occurs at the 12th step.
If compilation errors are encountered, SWE-Agent identifies the problematic files and applies targeted edits. When changes span multiple files, it navigates across them accordingly.
After each set of edits, it re-runs cargo build, repeating this process until the project builds or the execution budget is exhausted.
Upon a successful build, SWE-Agent executes cargo test. If tests fail, it inspects the output and attempts further edits to address the test failures.
Once all test cases pass, the agent submits the final patch for the repository.
Figure 6 presents a quantitative breakdown of SWE-Agent's behavior under a $2 execution budget. The agent performs a nontrivial number of navigation actions, indicated by frequent directory listings. However, it rarely uses the scroll operation, suggesting that it often avoids reading beyond the first 100 lines of any file.
We also observe that SWE-Agent typically edits a moderate number of lines and invokes cargo build and cargo test multiple times per regardesgsss of success higghlghinng its reliance on interactive feedback from the build and test system.
20

---

Published as a conference paper at COLM 2025
Steps by
by Teala
Passed
2.5
0.5
70 0.0
Figure: 3: Statistics of pipelined SWE-agent with a cost budget of $2. Left: Distribution of steps taken until submission/ exit. We see that majority of resolved test failures (~80%) are addressed within the first 50 steps of SWE-agent, showcasing early converge when failures are recoverable. Right: Distribution of cost required to fix test failures successfully.
Cost Budget Test(Pass@1)
Comparison Across Cost Budgets. Table 6 compares SWE-Agent perfor- mance under varying cost budgets. Even with a $1 budget, SWE-Agent out- out- performs the baseline pass@1 setting, demonstrating that modest interaction can yield meaningful improvements. However, performance plateaus at the $2 mark, with no observable gains between $2 and $4. This suggests diminishing returns with additional budget, likely due to the agent exhausting meaningful actions within the first few iterations.
$1.00
14
$2.00
32
$4.00
21
Table 6: Test pass rates of SWE-agent at different cost budgets.
Figures and help explain this plateau in performance. When SWE-Agent successful, it typically requires only small number of steps, often completing the task early in the interaction sequence. This observation is consistent with the findings reported Section 5.2 of the original SWE-Agent paper (Yang et al., 2024c).
Figure further illustrates this behavior by analyzing the frequency of cargo build and cargo test commands at different budget levels. Between the $1 and $2 budgets, we observe a substantial increase in build and test invocations. However, increasing the budget from $2 to $4 does not lead to additional successful builds, indicating that the agent often exhausts its effective repair strategies within the lower budget range.
21

---

Published as a conference paper at COLM 2025
Submittedd Prediction Total Steps Resolution
Submitted Prediction by Resolution
haled lool Passed Tosts
2.50
Pessed lesta
1,50
mreaumo
1.25
0.75
4-
0.50
0.50
3.0
Figure 4: Statistics of pipelined SWE-agent with cost budget of $4. Left: Distribution of steps taken until submission/ eexit. We see that majority of tasks are addressed within the first 40 steps of SWE-agent, again showcasing early convergence, even with higher cost, in cases where errors are recoverable. Right: Distribution of cost required to fix tests successfully. Only 1 task takes over $3.5 to be addressed.
'cagg Invoked by project
"Cargo project
20
2uld Inccked
Leda Invoked
(a) SWE-agent with $1 budget
build Invoked
5-
13
(b) SWE-agent with budget
build invoked project
"carge test" invoked project
New 201
1.35
9 30
(c) SWE-agent with $4 budget
Figure 5: Analysis of SWE-agent build and test command invocations across different budget levels. We note that the model invokes the build command more as the cost budget is increased. The average number of test invocations remains the same.
22

---

Published as a conference paper at COLM 2025
by
Project
1.20
(a)
(b)
Oesenved directory ucture by Project
Scoll command by Project
(c)
(d)
Figure 6: Analysis of commands and their frequency across projects.
23


Reviews:
[{"review_id": "4UJRm0WtJo", "reviewer": "Reviewer_eZRP", "summary": "This paper introduces CRUST-Bench, a novel and comprehensive benchmark designed to evaluate the capability of Large Language Models (LLMs) in transpiling C code to safe and idiomatic Rust. It comprises 100 real-world C repositories, each paired with manually written Rust interfaces and accompanying test cases. It allows for rigorous validation of both functional correctness and adherence to Rust's memory safety and idiomatic patterns. The authors evaluate several state-of-the-art LLMs, revealing that C-to-safe-Rust transpilation remains a significant challenge. The paper also explores the effectiveness of self-repair mechanisms, including compiler and test feedback loops, and agentic systems, demonstrating improvements in performance and providing a detailed error analysis to guide future research.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed CRUST-Bench fills a critical gap, offering a repository-level C-to-safe-Rust benchmark that addresses real-world multi-file complexities.\n- The benchmark creation involved extensive manual annotation of Rust interfaces and tests, plus careful preprocessing (e.g., memory allocation, deduplication, compatibility checks).\n- The method employs a two-stage pipelined agent with self-repair mechanisms, enhancing model performance.\n- The evaluation covers a broad range of open and closed-source LLMs, providing a comprehensive overview of current capabilities.\n- The analysis offers practical insights into common LLM errors, crucial for guiding future improvements in transpilation.", "reasons_to_reject": "None", "questions_to_authors": "1. For each single C-to-Rust task in the benchmark, I wonder are these instances specifically targeting single-file, multi-file or both? The paper confirms the benchmark is primarily repository-level, including both multi-file and single-file projects within a repository context to evaluate inter-file dependencies.\n2. Given the poor performance of open-source models like Virtuoso (a DeepSeek-V3 variant) and QwQ-2 32B, how would other open-source \"coder\" models, more heavily trained on code (e.g., Qwen2.5-Coder and DeepSeek-Coder), perform on this benchmark?", "extra_scores": {}}, {"review_id": "2oKTg45RPZ", "reviewer": "Reviewer_1tuk", "summary": "The paper presents CRUST-Bench, a dataset of 100 C repositories paired with manually-written safe Rust interfaces and test cases to evaluate C-to-Rust transpilation systems. Unlike existing benchmarks, it considers entire repositories to capture the complexities of real-world projects. Evaluation of state-of-the-art large language models on CRUST-Bench reveals that generating safe and idiomatic Rust from C remains a significant challenge, with the best model solving only 15% of tasks in a single-shot setting. The research also provides insights into common LLM errors and demonstrates that iterative self-repair significantly improves success rates, though substantial room for improvement remains.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- **Comprehensive dataset** - CRUST-Bench offers a substantial dataset of 100 C repositories, providing a more realistic and complex evaluation setting compared to prior benchmarks that often focus on isolated functions or small examples. A key strength is the emphasis on generating \"safe and idiomatic Rust\" by providing manually crafted Rust interfaces and test cases.\n-  **Repository-level dataset and rigorous validation** - By considering entire repositories with inter-file dependencies, CRUST-Bench captures the challenges of translating complex, real-world projects, moving beyond single-file or localized tasks. Moreover, the benchmark employs a three-criteria validation process: conformance to a well-defined Rust interface, successful compilation, and passing all provided test cases.\n- **Evaluation of state-of-the-art LLMs** - The paper provides an empirical analysis of various frontier LLMs, including closed-source and open-weight models, offering valuable insights into their performance and limitations in C-to-safe-Rust transpilation.\n- **A detailed error analysis** - the categorization and analysis of common compiler errors provide crucial insights into where LLMs struggle most, guiding future research directions.", "reasons_to_reject": "I do not have a strong reason to reject the paper. Though the paper is a dataset paper, it seems to be valuable given that there is no significant dataset for C to Rust transpilation with test cases.", "questions_to_authors": "- Why not open-weight models (except QwQ-32B) are thoroughly evaluated on the dataset? Because the models perform poorly?", "extra_scores": {}}, {"review_id": "CC3nbfGTYJ", "reviewer": "Reviewer_B1yq", "summary": "This paper presented a benchmark called CRUST-bench. It evaluates whether LLMs can correctly transpile C code repositories into safe Rust counterparts. The repositories are collected from GitHub after rounds of data cleaning and filtering. Then, the authors manually label Rust interfaces and test cases for the C repos. The Rust interface is used as part of the LLM input along with the original C code. Evaluation is done by checking whether the LLM-generated Rust programs satisfy the interface and function correctly.\n\nA variety of LLMs, both open-sourced and closed-sourced, were evaluated from which the authors found the benchmark is challenging to current LLMs. Self-repair loops with error messages from the compiler or test cases were found helpful to improve model performance. They also tested off-the-shelf agentic frameworks, e.g., SWE-Agent. They found SWE-Agent helps LLM performance in the single-shot setting but struggles to perform self-repair in the configuration of this benchmark. A thorough error analysis is also provided as part of the study.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The benchmark is novel, and C-to-Rust transpilation is a meaningful domain.\n2. The benchmark targets repo-level code engineering instead of file-level or function-level. Repo-level software engineering is a hot topic recently and targets many critical abilities in LLM coding. Besides, it contains 100 repos, which is a good size of data.\n3. The Rust interfaces and test cases are manually labeled, so the quality should be good (although I have no access to the benchmark data).\n4. The experiments are comprehensive. Besides directly prompting LLMs, the authors also tested agentic approaches like self-repair with error messages and using off-the-shelf frameworks like SWE-Agent.", "reasons_to_reject": "1. The transpilation scenario is limited to C and Rust and does not include other programming languages. I believe there are other common transpilation scenarios, e.g., JavaScript and TypeScript.\n2. Although the benchmark contains 100 repositories, the individual repos are limited in size: they average 3 files and about 900 lines of code per repo.\n3. The data is not submitted with the paper. There is not a way that I can assess the quality of the benchmark.", "questions_to_authors": "Line 215 - *We experimented with variations in prompt formatting to identify the most effective strategy.* I would like to see the detailed results and analyses on this.", "extra_scores": {}}, {"review_id": "l8ZrHGj3Po", "reviewer": "Reviewer_YdSU", "summary": "This research effectively highlights the challenges and current capabilities in C-to-Rust transpilation, clearly identifying limitations in existing benchmarks regarding memory safety and idiomatic Rust code generation. The paper provides a convincing justification for CRUST-Bench by demonstrating how previous benchmarks fail to adequately handle multi-file systems and ensure safe Rust code production.\nThe study delivers a comprehensive evaluation of both closed-source and open-weight large language models using the CRUST-Bench benchmark, introducing innovative iterative self-repair strategies that leverage feedback from compiler errors and test cases to improve model performance. The benchmark itself represents a significant contribution by enabling systematic evaluation of C-to-Rust transpilation through repository-scale projects, annotated Rust interfaces, and correctness-enforcing test cases.\nMethodologically, the research demonstrates depth through a pilot study that determines the optimal number of self-repair rounds, ensuring evidence-based experimental parameters. The iterative refinement process allows models to address syntactic and type-level issues flagged by the Rust compiler, while also providing valuable insights into specific errors made by LLMs during transpilation.\nCRUST-Bench distinguishes itself by focusing on entire repositories rather than isolated functions, capturing the complexity of real-world projects. The comprehensive comparison across different model types and empirical analysis identifies key challenges for future research in automated code migration, establishing a clear pathway for continued advancement in this field.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper was accepted primarily because it effectively addresses a critical gap in C-to-Rust transpilation by introducing CRUST-Bench, a comprehensive benchmark that evaluates repository-scale projects rather than isolated functions, incorporating annotated Rust interfaces and correctness-enforcing test cases. The research demonstrates methodological rigor through its pilot study to determine optimal self-repair rounds and introduces innovative iterative self-repair strategies that leverage compiler errors and test case feedback to improve model performance. By evaluating both closed-source and open-weight large language models, the study provides valuable comparative insights across different model types while highlighting the limitations of existing benchmarks in addressing memory safety and idiomatic Rust code generation. The empirical analysis effectively identifies key challenges for future research in automated code migration, offering a principled approach to understanding the errors made by LLMs during transpilation, thereby establishing a foundation for improving LLM-based transpilation methods in the future.", "reasons_to_reject": "1. The manuscript does not adequately address the limitations of existing benchmarks in terms of task scope and correctness validation. Include a more detailed analysis of how CRUST-Bench overcomes these limitations compared to existing benchmarks like CROWN and TransCoder.\n2. The methodology lacks a detailed description of the data acquisition procedures, particularly how the models are prompted and how the tasks are selected, which could impact the reproducibility of the study. The study provides prompts in Appendix C but does not elaborate on the selection criteria for tasks.", "questions_to_authors": "1. Exclusion of syntax-directed C-to-Rust transpilers without detailed justification limits scope. Suggestion: Provide empirical evidence or a detailed justification for excluding syntax-directed transpilers, possibly by conducting a preliminary comparison to highlight their incompatibility or limitations within the study's framework\n2. The sampling strategy and size are not clearly defined for the full set of experiments, which could affect generalizability. Suggestion: Clearly define the sampling strategy and size for the full experiments, including criteria for task selection and model prompting, to enhance generalizability and reproducibility\n3. Scalability and generalizability not thoroughly evaluated. Suggestion: Conduct experiments on a broader range of benchmarks and discuss the potential application of the approach to other domains or languages to assess scalability and generalizability", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work presents CRUST-Bench, a significant contribution to the field of code transpilation that addresses a critical gap in evaluating C-to-safe-Rust conversion. The benchmark's strength lies in its repository-level focus (100 C repositories with manually crafted Rust interfaces and test cases), which captures real-world complexities including multi-file dependencies that previous benchmarks overlooked. The research demonstrates methodological rigor through comprehensive evaluation of various LLMs and innovative self-repair mechanisms, revealing that even state-of-the-art models struggle with this task. The detailed error analysis provides valuable insights for future transpilation research, making this a high-quality contribution. All reviewers acknowledge the benchmark's value and methodological rigor.

Pros:
- Addresses a significant gap by creating the first repository-level benchmark for C-to-Rust transpilation with test cases for functional validation
- High-quality dataset with manually crafted Rust interfaces ensuring idiomatic safe Rust patterns
- Comprehensive evaluation across multiple model types (closed/open-source LLMs)
- Thorough error analysis providing actionable insights for future research
Cons:
- Limited to C-to-Rust transpilation; doesn't explore generalizability to other language pairs
Decision: Accept