Venue: NAACL 2025

# CSR-Bench: Benchmarking LLM Agents in Deployment of

Computer Science Research Repositories

 Yijia Xiao\({}^{\dagger}\)* Runhui Wang\({}^{\ddagger}\)** Luyang Kong\({}^{\ddagger}\)* Davor Golac\({}^{\ddagger}\) Wei Wang\({}^{\dagger}\)\({}^{\dagger}\)University of California, Los Angeles \({}^{\ddagger}\) Amazon Web Services

###### Abstract

The increasing complexity of computer science research projects demands more effective tools for deploying code repositories. Large Language Models (LLMs), such as Anthropic Claude and Meta Llama, have demonstrated significant advancements across various fields of computer science research, including the automation of diverse software engineering tasks. To evaluate the effectiveness of LLMs in handling complex code development tasks of research projects, particularly for NLP/CV/AI/ML/DM topics, we introduce CSR-Bench, a benchmark for Computer Science Research projects. This benchmark assesses LLMs from various aspects including accuracy, efficiency, and deployment script quality, aiming to explore their potential in conducting computer science research autonomously. We also introduce a novel framework, CSR-Agents, that utilizes multiple LLM agents to automate the deployment of GitHub code repositories of computer science research projects. Specifically, by checking instructions from markdown files and interpreting repository structures, the model generates and iteratively improves bash commands that set up the experimental environments and deploy the code to conduct research tasks. Preliminary results from CSR-Bench indicate that LLM agents can significantly enhance the workflow of repository deployment, thereby boosting developer productivity and improving the management of developmental workflows.

1
Footnote 2025: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 12705–12723

[Scale=title] **Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilin Xilinilin Xilin Xilin Xilin Xilinilin Xilin Xilinilin Xilin Xilinilin Xilin Xilinilin Xilin Xilinilin Xilin Xilinilin Xilin Xilinilin Xilinilin Xilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilinilin Xilinilin Xilinilinilin Xilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilinilin Xilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilinilin Xilinilinilinilinilinilin Xilinilinilinilinilinilin Xilinilinilinilinilinilin Xilinilinilinilinilinilin Xilinilinilinilinilinilinilin Xilinilinilinilinilinilinilin Xilinilinilinilinilinilinilin Xilinilinilinilinilinilinilin Xilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilinilin Xilinilinilinilinilinilinilin Xilinilinilinilinilinilinilin Xilinilinilinilinilinilin Xilinilinilinilinilinilin Xilinilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilinilin Xilin Xilinilin Xilin Xilinilin

expertise and iterative improvement with provided tools. From more than ten major NLP/CV/AI/M-L/DM conferences, we collected the 100 highly rated repositories1, which are carefully selected from an initial candidate set that has more than 1500 top-star repositories. Our selection criteria include topic diversity and self-containment2 so that CSR-Bench can provide a comprehensive evaluation of LLM agents on code deployment tasks including instruction generation, command execution, and self-improvement with tools.

Footnote 1: With appropriately permissive licenses.

Footnote 2: The information within the repository is mostly sufficient for a successful deployment.

To the best of our knowledge, CSR-Bench is the first benchmark for the deployment of computer science research projects, providing a reference for evaluating LLM agents. We note that the success of code deployment depends not only on code generation, but also on many non-coding factors including tasks like experimental environment setup, data/model preparation, correcting bash commands, searching for solutions, and etc. Our proposed multi-agent framework, CSR-Agents, aims to achieve automation of code deployment, which can accelerate the progress for computer science research projects.

Our contributions are as follows:

* We introduce CSR-Bench to assess LLM's ability to understand instruction manuals and complex project structures, generate executable commands for code deployment, and solve errors during deployment.
* We propose the CSR-Agents framework, which leverages multi-agent cooperation with specialized capabilities including instruction comprehension, command execution, error log analysis, and error correction with searching and retrieval tools.
* We design a standardized testing system for reproducibility in CSR-Bench, which can make CSR-Agents a CI/CD3 standard system of computer science code repository deployment, ensuring ease of use, reusability, and improving communication and collaboration efficiency in computer science research projects. Footnote 3: Continuous Integration and Continuous Delivery.
* We evaluated a wide range of foundation models for CSR-Agents on CSR-Bench. Results indicate that LLM agents can potentially accelerate the process of repository deployment, thereby boosting researcher productivity. However, it is still challenging to achieve full automation.

## 2 Related Work

Coding LLMs.Large Language Models (LLMs) have become the go-to solution for a wide array of coding tasks due to their exceptional performance in both code generation and comprehension Chen et al. (2021). These models have been successfully applied to various software engineering activities, including program synthesis Patton et al. (2024); Chen et al. (2021); Li et al. (2022); Iyer et al. (2018), code translation Pan et al. (2024); Roziere et al. (2020, 2021), program repair Xia et al. (2023); Xia and Zhang (2023); Monperrus (2018); Bouzenia et al. (2024), and test generation Deng et al. (2023); Xia et al. (2024); Deng et al. (2024); Lemieux et al. (2023); Kang et al. (2023). Beyond general-purpose LLMs, specialized models have been developed by further training on extensive datasets of open-source code snippets. Notable examples of these code-specific LLMs include CodexChen et al. (2021), CodeLlamaRoziere et al. (2023), StarCoderLi et al. (2023); Lozhkov et al. (2024), and DeepSeek-Coder Guo et al. (2024). Additionally, instruction-following code models have emerged, refined through instrumenting techniques. These include models such as CodeLlamaInst Roziere et al. (2023), DeepSeeK-Coder-InstGuo et al. (2024), WizardCoder Luo et al. (2023), Magicoder Wei et al. (2023), and OpenCodeInterpreter Zheng et al. (2024).

Benchmarking LLM-based coding tasks.To assess the capabilities of LLMs in coding, a variety of benchmarks have been proposed. Among the most widely utilized are HumanEvalChen et al. (2021) and MBPP Austin et al. (2021), which are handcrafted benchmarks for code generation that include test cases to validate the correctness of LLM outputs. Other benchmarks have been developed to offer more rigorous tests Liu et al. (2023), cover additional programming languages Zheng et al. (2023); Cassano et al. (2023), and address different programming domains Jain et al. (2024); Hendrycks et al. (2021); Li et al. (2022b); Lai et al. (2023); Yin et al. (2022).

More recently, research has shifted towards evaluating LLMs on real-world software engineering challenges by operating on entire code reposito 

ries rather than isolated coding problems Jimenez et al. (2024); Zhang et al. (2023); Liu et al. (2023). A notable benchmark in this area is SWEbench Jimenez et al. (2024), which includes tasks requiring repository modifications to resolve actual GitHub issues. The authors of SWE-bench have also released a more focused subset, SWE-bench Lite swe (2024), which contains 300 problems centered on bug fixing that only involves single-file modifications in the ground truth patches. MLBench Liu et al. (2023) is a benchmark for evaluating large language models and agents for Machine Learning tasks on reporitory-level code. It involves 18 repositories and focuses on code generation and interactions with Jupyter Notebooks.

**Repository-level coding.** The rise of agent-based frameworks Xi et al. (2023) has spurred the development of agent-based approaches to software engineering tasks. Devin dev (2024) (and its open-source counterpart OpenDevin ope (2024b)) is among the first comprehensive LLM agent-based frameworks. Devin employs agents to first perform task planning based on user requirements, then allows them to use tools like file editors, terminals, and web search engines to iteratively execute the tasks. SWE-agent Yang et al. (2024) introduces a custom agent-computer interface (ACI), enabling the LLM agent to interact with the repository environment through actions like reading and editing files or running bash commands. Another agent-based approach, AutoCodeRover Zhang et al. (2024), equips the LLM agent with specific APIs (e.g., searching for methods within certain classes) to effectively identify the necessary modifications for issue resolution. Beside these examples, a variety of other agent-based approaches have been developed in both open-source Gauthier (2024) and commercial products Bouzenia et al. (2024); Chen et al. (2024); Ma et al. (2024); lin (2024); fac (2024); ibm (2024); ope (2024a); mar (2024); ama (2024).

Unlike existing benchmarks and agent-based frameworks, which focus on the code generation/completion tasks, our proposed CSR-Bench and CSR-Agents focus on the code deployment task, which is under-studied in the field.

## 3 CSR-Bench

In this section, we provide the problem statement for code deployment in CSR-Bench, introduce the code repository collection process of computer science research projects, and show their statistics.

### Problem Statement

The CSR-Bench consists of a collection of computer science research repositories from GitHub and these repositories are used for evaluating the capabilities of LLMs in code deployment tasks. For each repository, the deployment tasks typically include: (1) setting up the environment; (2) preparing necessary data and model files; (3) conducting model training; (4) demonstration of inference; (5) performance evaluation. To complete these tasks, we prompt LLMs to generate executable bash commands by using the README file as the primary source of information and other repository contents (source code, bash scripts, directory structure, and etc.) as supplementary information.

**Metric.** During the evaluation in CSR-Bench, the large language model will be prompted to generate executable commands for the corresponding sections for each repository. we use the completion rate as a key metric, defined as the ratio between number of successfully executed commands and the total number of commands executed.

### Repository Collection

In CSR-Bench, we aim to collect a diverse and comprehensive collection of code repositories of computer science-related research projects. GitHub is a good data source for this purpose and it provides tags for identifying most relevant repositories. Some example tags are "nlp", "naacl", and "emnlp2024". Since CSR-Bench focuses on computer science-related repositories, we filter the repositories by tags of various conference names and categories to ensure they include diverse topics.

For repository selection, we use GitHub tags to obtain an initial set of over 1500 repositories that are relevant to computer science research topics and categorizing them into five areas: Natural Language Processing, Computer Vision, Large Language Models, Machine Learning, and Interdisciplinary topics. Notably, we collect repositories related to large language models because nowadays LLM-related research projects are increasingly popular due to its foundational impact in various areas of computer science.

We obtain 100 high-quality code repositories for CSR-Bench in the following steps. First, we categorize this initial set and sort them by the number of GitHub stars. Next, we manually check the content of each repository starting from the top of the sorted list. In this step, we only keep reposi 

for the reason reason to favor a positive value of \(0.15\), which is not surprising in our case.

### Statistics of CSR-Bench

This section provides an in-depth analysis of the traits of repositories in CSR-Bench. We examine the diversity and breadth of topics covered, as well as detailed statistics about the documentation and structure of these repositories.

In CSR-Bench, the README files and directory structures provide critical insights into the usability and organization of repositories. We use the following figures to analyze the lengths of README file and number of files, and offer a quantitative view of content complexity and organizational depth. The length of README file is an important metric because the most LLMs have limits on the input token length. The number of files indicate the complexity of the code repository.

Figure 3 shows the distribution of token counts in README files, highlighting the extent of documenting and repository usability. Since the mean token counts is just over 1000 and the maximum counts is around 3000, most LLMs can take the full README files as input.

Figure 4 depicts the distribution of file counts in repositories, reflecting their complexity and scale based on the number of files. Because the number of files in most repositories are in the lower hundreds, it is feasible to leverage directory structure for code deployment with LLMs.

 

Figure 5 shows the distribution of star counts in selected repositories. The average count is over 590, which means that these repositories receives significant attention and indicates they are generally well maintained.

Figure 6 shows the distribution of issues counts in selected repositories. On average, each repository contains over 48 issues, indicating these repositories have a good amount of engagements with the open-source community and sufficient support from the authors. Therefore, the information in the issues are valuable for the deployment tasks.

## 4 CSR-Agents

In this section, we propose CSR-Agents, a multi-agent framework that leverages LLMs for different tasks and achieves effective cooperation among agents for code deployment tasks. We introduce our standardized environment for code deployment, functionalities of different agents, and their cooperation workflow.

### Code Deployment with LLM

For each repository, we use the README file under the root folder as the major source of information and prompt LLMs to generate executable bash commands for different steps of deployment including environment setup, data preparation, training, inference and evaluation.

To achieve reproducbility and safety, we use the Docker container to isolate the code deployment environment. We build a standard Docker image that is equipped with essential tools like bash, Conda, GCC, Make, Python, etc. for evaluations across all repositories and various LLMs. In the container, we use a counter to count the total number of scripts that needed to be executed and the number of scripts that were executed successfully. In this way, we safeguard the entire computing system, especially from bash command executions that involve system-level permissions and could potentially break the whole system.

We note that the use of Docker is not part of the original repositories. The introduction of Docker into our evaluation system can ensure that each time we conduct the evaluation, we start from the same environment for fair comparisons across different LLMs, and the evaluation of one repository does not affect any other repositories. Another benefit of using Docker is that we can speed up the evaluation by running different evaluations in separate docker instances at the same time.

The LLM generated bash commands may not work well for successful deployment due to various reasons. First, in some README files, some basic directives (such as conda and pip) were missing, so the LLMs could not generate these commands at the first attempt. Secondly, the installation of packages may not be completed in a single attempt and usually needs several iterations of trial-and-error. Thirdly, some steps of the deplopyment require checking additional information like GitHub issues of the corresponding repositories and from the internet. To handle these issues, we design a group of LLM agents that cooperate effectively.

### CSR-Agent: LLM Agent Design

In CSR-Agents, we adopt an iterative trial-and-error process for successful code deployment. Specifically, LLM takes README, directory structure, and error logs from failed command execution as input and generates bash commands to complete the deployment tasks. The system comprises five agents: Command Drafter, Script Executor, Log Analyzer, Issue Retriver, and Web Searcher. These agents collectively facilitate the deployment of a code repository. The complete workflow is shown in Figure 7.

**Command Drafter**: This agent reads the README files and directory structure and generates a draft script containing bash commands for deployment. Then, it divides the entire script into five sections, each corresponding to a step in the code repository deployment. This sectional division also serves as an evaluation standard later on.

**Script Executor**: This agent receives the draft commands from the Command Drafter and execute the commands in our standardized Docker environment. After execution, it collects logs from bash, including standard output and errors. Note that during bash script execution, no explicit return code

Figure 6: Size of Issue Database



is provided by bash. We experimented with setting predefined special prompt to bash and parse the return code from the returned output and error message. However, we find that a good quantity of commands do not have return code, making the feedback from bash not informative to other agents. To address this challenge, we leverage the LLM in the executor, instructing it to provide feedback based on the standard output and error messages. We then parse this feedback to generate a return code: if the return code is zero, then the command is executed successfully; otherwise, the command, output, and error messages are logged and sent to the Log Analyzer.

**Log Analyzer**: This agent reads the logs and the associated bash command, and checks for updates, missing prerequisites, or script paths that need updating. It also identifies any other missing components and returns a curated command for a successful execution. We note that this agent only reply on the internal knowledge of the LLM and may not be able to correctly solve the errors.

**Issue Retriever**: This agent takes in the command, standard output and error message and search them against the issue database we collected from the repository. It leverages RAG and its pipeline requires a search algorithm to query the input against the database. In our design, the queries are the combination of the commands executed, standard output and error messages. The database contains GitHub issues and their communication records for the repository. We experimented with BM25 and Contriever as the retrieval algorithm and decided to use BM25 for (1) BM25's higher search speed and (2) the fact that error logs and issues generally share keywords, so sentimental search do not possess much advantage over lexical search.

**Web Searcher**: This agent utilizes the Perplexity API to obtain solutions for failed execution. If the pipeline reaches this stage, it indicates that the Log Analyzer and Issue Retriever failed to solve a failed command. The standard output, standard error, and the failed command are fed to Perplexity to search the web for solutions. The Web Searcher then analyzes the solutions from Perplexity and generates new bash commands to resolve the issue.

### LLM Coorporation Framwork

The workflow operates as follows: Deployment commands are drafted from repository documentation, executed in a bash environment, and adjusted based on log analysis if errors arise. Additional information is retrieved from an issue database and web search if needed. The process concludes with

Figure 7: Workflow of CSR-Bench



a summary that outlines successes and identifies steps needing further attention.

Drafting the Initial CommandsThe instructions from each repository's README are fed into the Command Drafter LLVM Agent to draft the necessary commands for deploying the repository. These commands are organized into five stages: prerequisite installation, data and model checkpoints downloading, training, inference, and evaluation. It is important to note that not every repository contains all these sections, and some sections may be empty.

Execution of CommandsAfter the draft stage, the script drafted will be sent to the Script Executor. To provide LLM Agents with the Python interface, we implement a BashExecutor that encapsulates the bash binary executable file in it. If the commands execute successfully, the process is deemed successful and the deployment pipeline will return True. Though looks simple, the bash simulation was not straightforward during our experiments, since we need to consider the environment variables, stdout, stderr, etc. Initially, we used a subprocess to handle it, but the success rate was extremely low. Upon analyzing the logs, we discovered that no commands related to environment variable changes were reflected in subsequent instructions. For instance, if the Command changes the environment to a Python virtual environment or another conda environment, this change only applies to that specific command. Afterward, the Python interpreter and package manager revert to their default settings. Besides, not all command executions have a valid return code, therefore, we utilize an LLM to parse the standard output and standard error (if any) to obtain the return state of the execution.

Analyze the Execution LogIf the execution is not successful and error occurs, the standard output, standard error, command, and return code are logged and analyzed. The Log Analyzer examines the error messages and attempts to refine the execution commands or adjust prerequisites to ensure the environment is prepared. We have a max_attempt argument that limits the number of retries for the log analyzer. If the issue persists after certain attempts, the workflow utilizes the web search tool to request external information.

Retrieve Augmented Generation from Issue DatabaseAfter analyzing the execution log and making initial adjustments, the next step is to retrieve more insights using a Retrieve Augmented Generation (RAG) approach. The Issue Retriever agent uses the logged command, output, and error messages to query the issue database for similar past problems or discussions. Leveraging the BM25 retrieval algorithm, it matches keywords from the logs to relevant entries. If a match is found, the agent extracts solutions or troubleshooting steps, feeding them back into the workflow to refine commands. If no relevant match is found, the process escalates to the Web Searcher for external information.

Search the Internet for External InformationUsing tools such as Perplexity, the WebSearcher agent integrates the command error logs and standard output with external information. This process refines the command and retries execution in the bash with a limited number of attempts. The agent records the information and transfers it to the deployment summarizer if the issue remains unresolved.

## 5 Evaluations

In this section, we evaluate CSR-Agents in CSR-Bench with a wide range of popular foundation LLM families, including Claude 4, GPT-4 [1], Llama-3 [7], and Mistral 5. For each LLM family, we experimented with different model sizes for thorough comparisons.

Footnote 4: [https://www.anthropic.com/news/claude-3-family](https://www.anthropic.com/news/claude-3-family)

Footnote 5: [https://mistral.ai/technology/#models](https://mistral.ai/technology/#models)

We show the completion rate in Table 1, Table 2, Table 3, and Table 4. In these tables, we use **S** to stand for the Setup stage, **D** to stand for the ownload stage, **T** to stand for the Training stage, **E** to stand for the Evaluation stage, and **I** for the inference stage.

### Initial Drafter

As shown in Table 1, all models perform well on _Setup_ and _Download_ tasks (success rates around 0.23 to 0.28) but struggle with _Training_, _Evaluation_, and _Inference_, where success rates are close to zero. This indicates that the drafter agent handles basic installation effectively but has difficulty with complex tasks requiring updates to file paths and environment variables.

We note that this is also similar to the deployment process of a real researcher, where their first execution is more likely to fail and they need to analyze the errors and leverage tools like GitHub issues or search engines to solve the problem.

 



### Log Analyzer

Table 2 shows noticeable improvements in all tasks compared to the drafter stage. Success rates for _Setup_ and _Download_ increase to around 0.34 to 0.40, while complex tasks see gains up to 0.18.

Analyzers leverage dynamic feedback from the execution of commands to refine scripts and try to correct errors from executed commands.

### Issue Retriever

In the Issue Retrieval stage (Table 3), success rates continue to improve, especially for complex tasks like **Training**, **Evaluation**, and **Inference**, reaching up to 0.25.

The results show that access to a knowledge base with informative discussions on issues of the code repository allows LLMs to retrieve solutions to execution errors from earlier stages, enhancing performance in complex operations.

### Web Searcher

The Searcher Success Metrics in Table 4 exhibits the highest performance. Success rates for _Setup_ and _Download_ reach up to 0.46, and complex tasks improve to between 0.15 and 0.29.

Web search enables models to find up-to-date solutions, resolving issues that previous agents could not, leading to substantial improvements.

### Aggregated Results

We show the aggregated results of a single LLM on different tasks with different level of engagements of multi-agents in Figure 8, Figure 9, Figure 10, and Figure 11. In short, with more agents contributing to solving the tasks, the success rate increases across all tasks for all LLMs, which demonstrates the effectiveness of our proposed CSR-Agents.



Inference and Evaluation: Performance was less consistent, with some models demonstrating moderate success, but generally struggling with the complexity of these tasks.

Training: Training tasks are particularly challenging, with lower success rates across the board, indicating that current LLMs require further refinement to handle training processes effectively.

#### 5.6.2 Overall Performance

The success metrics across different tasks and models indicate a wide variability in performance. Generally, models showed higher success rates in Setup and Download tasks, with performance tapering off in more complex tasks such as Inference, Evaluation, and Training. This pattern highlights the challenges LLMs face in handling the full deployment process autonomously.

The results demonstrate that while LLMs have made significant strides in automating repository deployment, their ability to manage complex tasks remains limited. Improvements are needed, particularly in the areas of Inference and Training, to achieve fully autonomous and reliable deployment of science repositories.

However, there is still a large gap between LLMs and real scientists even if the advanced tools are provided to the LLMs. To explain, it is not trivial to handle the nuances in the experiement environment setup for the science repositories. For example, the hardware and software compatibility issues are very common in code deployment and often causes confusions even for domain experts.

## 6 Conclusion

The work introduces CSR-Bench, a benchmark designed to evaluate the capabilities of LLM agents in automating the deployment of GitHub repositories for scientific research. Our study highlights that while LLMs show potential in handling tasks like environment setup and data preparation, they face challenges in complex tasks such as training and inference, where success rates are notably lower.

Our multi-agent framework, CSR-Agents, exemplifies how LLMs can collaborate to tackle deployment challenges, offering a promising approach to improving automation in software engineering. However, the results indicate that further advancements are needed to fully realize autonomous and reliable deployment processes.

Overall, CSR-Bench serves as a crucial tool for assessing and improving LLM-driven deployment workflows in scientific research, paving the way for more efficient and automated computer science projects exploration 



### Limitations

Although our benchmark framework supports several tools to facilitate large language model agents in the code deployment task, it does not actually improve the original reasoning capabilities of the large language models that are used in the agents. To improve LLMs' reasoning capabilities for this specific task, the community may resort to techniques like RLHF, which is orthogonal to this work. Our benchmark only focuses on code repositories that related to computer science research topics, and does not involve other types of repositories. Although this framework can be reused for other types of the repositories, we do not explore that direction in this work, and leave it to future works.

## References

* A. Agarwal (2023)Agent-101: a software engineering agent for code assistance developed by ibm research. Note: [https://github.com/swe-bench/experiments/blob/main/evaluation/lite/20240612_IBM_Research_Agent101/README.md](https://github.com/swe-bench/experiments/blob/main/evaluation/lite/20240612_IBM_Research_Agent101/README.md) Cited by: SS1.
* D. Chen, S. Lin, M. Zeng, D. Zan, J. Wang, A. Cheshkov, J. Sun, H. Yu, G. Dong, A. Aliev, et al. (2024)Coder: issue resolving with multi-agent and task graphs.. arXiv preprint arXiv:2406.01304. Cited by: SS1.
* E. Devin (2025)A software engineer. GitHub: [http://www.cognition.ai/introducing-devin](http://www.cognition.ai/introducing-devin). External Links: Link Cited by: SS1.
* F. de Vries, P. B. van der Laan, R. C. Lehouchelle, K. Lerer, T. W. Lee, and Y. Li (2027)Fashion-specific learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 987-990. Cited by: SS1.
* I. Bouzenia, P. Devanbu, and M. Pradel (2024)Repairagent: an autonomous, llm-based agent for program repair. arXiv preprint arXiv:2403.17134. Cited by: SS1.
* F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M. Yee, Y. Zi, C. Jane Anderson, M. Q. Feldman, et al. (2023)Multipl-e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering. Cited by: SS1.
* D. Chen, S. Lin, M. Zeng, D. Zan, J. Wang, A. Chesskov, J. Sun, H. Yu, G. Dong, A. Aliev, et al. (2024)Code: issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406.01304.



Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. _arXiv preprint arXiv:2107.03374_.
* Deng et al. (2023) Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In _32nd International Symposium on Software Testing and Analysis (ISSTA)_.
* Deng et al. (2024) Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. 2024. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. In _46th International Conference on Software Engineering (ICSE)_.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Gauthier (2024) Paul Gauthier. 2024. Aider is ai pair programming in your terminal. [https://aider.chat/](https://aider.chat/).
* Guo et al. (2024) Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YY Li, et al. 2024. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. _arXiv preprint arXiv:2401.14196_.
* Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranki, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge competence with apps. _NeurlPS_.
* Hong et al. (2023) Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023. MetagPT: Meta programming for multi-agent collaborative framework. _arXiv preprint arXiv:2308.00352_.
* Iyer et al. (2018) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1643-1652.
* Jain et al. (2024) Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Kotushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. _arXiv preprint_.
* Jimenez et al. (2024) Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kevin Pei, Ofr Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In _The Twelfth International Conference on Learning Representations_.
* Kang et al. (2023) Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large language models are few-shot testers: Exploring llm-based general bug reproduction. In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_, pages 2312-2323. IEEE.
* Lai et al. (2023) Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In _International Conference on Machine Learning_, pages 18319-18345. PMLR.
* Lemieux et al. (2023) Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In _45th International Conference on Software Engineering (ICSE)_.
* Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Koecetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you!
* Li et al. (2022a) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022a. Competition-level code generation with alphacode. _Science_, 378(6624):1092-1097.
* Li et al. (2022b) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprienden de Masson d'Autume, Igor Babuschkin, Xinyun Chen, PoSen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022b. Competition-level code generation with alphacode. _Science_.
* Liu et al. (2023a) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023a. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Liu et al. (2023b) Tianyang Liu, Canwen Xu, and Julian McAuley. 2023b. Repobench: Benchmarking repository-level code auto-completion systems. _arXiv preprint arXiv:2306.03091_.
* Liu et al. (2023c) Yuliang Liu, Xiangru Tang, Zefan Cai, Junje Lu, Yichi Zhang, Yanjun Shao, Zexuan Deng, Helan Hu, Zengxian Yang, Kaikai An, et al. 2023c. MDbench: Large language models leverage open-source libraries for machine learning tasks. _arXiv e-prints_, pages arXiv-2311.
* Lozhkov et al. (2023) Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noumanane Tazi, Ao Tang, Dhytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, 

Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen- Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltenozhskii, Nii Osse Qae Dade, Wenhao Yu, Lucas Kraub, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abat, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muthasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patty, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Munoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2024. Starcoder 2 and the stack v2: The next generation. _Preprint_, arXiv:2402.19173.
* Luo et al. (2023) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Dexin Jiang. 2023. Wiizardcoder: Empowering code large language models with evolirschort. _arXiv preprint arXiv:2306.08568_.
* Ma et al. (2024) Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. 2024. How to understand whole software repository? _arXiv preprint arXiv:2406.01422_.
* Monperrus (2018) Martin Monperrus. 2018. _The living review on automated program repair_. Ph.D. thesis, HAL Archives Ouvertes.
* Pan et al. (2024) Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pougem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbavand. 2024. Lost in translation: A study of bugs introduced by large language models while translating code. In _Proceedings of the IEEE/ACM 46th International Conference on Software Engineering_, pages 1-13.
* Patton et al. (2024) Noah Patton, Kia Rahmani, Meghana Missula, Joydeep Biswas, and Isal Dillig. 2024. Programming-by-demonstration for long-horizon robot tasks. _Proceedings of the ACM on Programming Languages_, 8(POPL):512-545.
* Roziere et al. (2023) Bapitste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jeremy Rapin, et al. 2023. Code llama: Open foundation models for code. _arXiv preprint arXiv:2308.12950_.
* Roziere et al. (2020) Bapitste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. _Advances in neural information processing systems_, 33:20601-20611.
* Roziere et al. (2021) Bapitste Roziere, Jie M Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lampe. 2021. Leveraging automated unit tests for unsupervised code translation. _arXiv preprint arXiv:2110.06773_.
* Wei et al. (2023) Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Source code is all you need. _arXiv preprint arXiv:2312.02120_.
* Xi et al. (2023) Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. _arXiv preprint arXiv:2309.07864_.
* Xia et al. (2024a) Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024a. Agentless: Demystifying llm-based software engineering agents. _arXiv preprint arXiv:2407.01489_.
* Xia et al. (2024b) Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. 2024b. Universal fuzzing via large language models. In _46th International Conference on Software Engineering (ICSE)_.
* Xia et al. (2023) Chunquiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023. Automated program repair in the era of large pre-trained language models. In _Proceedings of the ACM/IEEE 45th International Conference on Software Engineering_, ICSE '23.
* Xia and Zhang (2023) Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the conversation going: Fixing 162 out of 337 bugs for $0.42 each using chatgtpct. _arXiv preprint arXiv:2304.00385_.
* Yang et al. (2024) John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofri Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. _arXiv preprint arXiv:2405.15793_.
* Yin et al. (2022) Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yaming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henrykh Michalewski, Alex Polozov, and Charles Sutton. 2022. Natural language to code generation in interactive data science notebooks.
* Zhang et al. (2023) Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. _arXiv preprint arXiv:2303.12570_.
* Zhang et al. (2024) Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. Autocordover: Autonomous program improvement. _Preprint_, arXiv:2404.05427.
* Zheng et al. (2023) Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. 2023. Codgeex: A pre-trained model for code generation with multilingual evaluations on humaneval-\(x\). _arXiv preprint arXiv:2303.17568_.
* Zheng et al. (2024) Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. Opencodeinterpreter: Integrating code generation with execution and refinement. _arXiv preprint arXiv:2402.14658_ 

Appendix

### Categories and Subcategories

**Computer Vision and Graphics**

* 3D Vision and Reconstruction
* Structured3D: Structured 3D modeling dataset for computer vision and graphics (ECCV 2020)
* RayDF: Light ray-based 3D rendering and learning (NeurIPS 2023)
* SSDNeRF: Diffusion-based NeRF for 3D reconstruction (ICCV 2023)
* NeuralPull: Learning distance functions for point clouds (ICML 2021)
* DAD-3DHeads: Dataset for 3D head alignment and reconstruction (CVPR 2022)
* LeGO-LOAM: Lidar-based odometry and mapping for robotics
* **Image Segmentation, Processing, and Enhancement*
* Unsupervised-Semantic-Segmentation: Unsupervised methods for semantic segmentation (ICCV 2021)
* Seg-Uncertainty: Scene adaptation for self-driving cars (IJCAI 2020, IJCV 2021)
* RGBD_Semantic_Segmentation_PyTorch: RGBD scene recognition and segmentation (ECCV 2020)
* BCNet: Occlusion-aware segmentation (CVPR 2021)
* Deep_Metric: Metric learning for image retrieval
* Multiview2Noveliew: Novel view synthesis from multiple views (ECCV 2018)
* NeRCo: Low-light image enhancement using neural representations (ICCV 2023)
* Transfiner: High-quality instance segmentation with mask transformers (CVPR 2022)
* **Pose Estimation and Object Detection*
* HybrIK: Hybrid kinematics for 3D human pose (CVPR 2021)
* SmoothNet: Human pose refinement in videos (ECCV 2022)
* FSA-Net: Head pose estimation from images (CVPR 2019)
* VisualDet3D: Monocular 3D detection for autonomous driving
* DEVIANT: Monocular 3D object detection for autonomous driving (ECCV 2022)
* OWOD: Open World Object Detection with incremental learning (CVPR 2021)

**Machine Learning Models and Techniques**

* [noitemsep]
* **Generative and Diffusion Models*
* DCM2: Diffusion models for MRI denoising (ICLR 2023)
* DDNM: Diffusion models for zero-shot image restoration (ICLR 2023)
* diffusion-point-cloud: Generating 3D point clouds with diffusion models (CVPR 2021)
* GeoDiff: Diffusion models for molecular structures (ICLR 2022)
* probabilistic_unet: Conditional segmentation with probabilistic UNet (NeurIPS 2018)
* LTSF-Linear: Linear models for time series forecasting (AAAI 2023)
* unconditional-time-series-diffusion: Diffusion models for time series prediction (NeurIPS 2023)
* dyffusion: Spatiotemporal forecasting using diffusion models (NeurIPS 2023)
* **Graph Neural Networks and Transformers*
* GraphSAINT: Scalable graph neural networks with sampling (ICLR 2020, IPDPS 2019)
* Graph-Transformer: Applying transformers to graph data (WWW 2022)
* graphtransformer: Generalized transformers for graphs (DLG-AAAI 2021)
* GPT-GNN: Pre-training techniques for graph neural networks (KDD 2020)
* GraphMAE: Masked autoencoders for graph learning (KDD 2022)
* **Self-Supervised and Contrastive Learning*
* skip-connections-matter: Impact of skip connections on adversarial example transferability (ICLR 2020
 

* SparK: BERT-style pretraining on convolutional networks (ICLR 2023)
* imbalanced-semi-self: Semi-supervised learning for class-imbalanced datasets (NeurIPS 2020)
* CURL: Image enhancement via neural curve layers (ICPR 2020)
* **Model Interpretability and Optimization*
* rrl: Rule-based learning for interpretable classification (NeurIPS 2021)
* gradient-descent-the-ultimate-optimizer: Advanced optimization techniques (NeurIPS 2022)

Large Language Models (LLMs)

* LLM Development and Optimization
* llama-recipes: Fine-tuning scripts for LLLaMA models
* llama3: Official GitHub repository for LLLaMA 3
* alpaca-lora: Instruct-tuning LLaMA on consumer hardware
* MetaGPT: Multi-agent AI development framework using LLMs
* Stanford_Alpaca: Training and dataset generation for Alpaca models
* **Inference and Efficiency in LLMs*
* mistral-inference: Efficient inference for Mistral models
* PowerInfer: Fast LLM serving on consumer-grade GPUs
* direct-preference-optimization: Implementation of direct preference optimization (DPO)
* GPTCache: Efficient caching for LLMs integrated with LangChain and llama_index
* **LLM Applications and Evaluation*
* alpaca_eval: Automatic evaluation for instruction-following LLMs
* DART: Enhancing few-shot learning with differentiable prompts (ICLR 2022)
* LRV-Instruction: Reducing hallucination in multi-modal models (ICLR 2024)
* alpaca_farm: Simulated framework for reinforcement learning from human feedback (RLHF)
* FastChat: Open-source platform for training and evaluating large language models

Natural Language Processing (NLP)
* Text Classification and Information Extraction
* HVPNeT: Multimodal entity and relation extraction (NAACL 2022)
* PURE: Simple approach to relation extraction (NAACL 2021)
* SimCSE: Contrastive sentence embeddings (EMNLP 2021)
* Text-GCN: Text classification using graph convolutional networks (AAAI 2019)
* **Multimodal and Knowledge-Based Systems*
* OntoProtein: Protein function prediction using knowledge graphs (ICLR 2022)
* tree-of-thought-llm: Structured problem-solving with LLMs (NeurlPS 2023)
* storm: Automated report generation using large language models
* SWE-agent: AI-based code bug fixing using LLMs
* **Transfer Learning and Domain Adaptation*
* SubpopBench: Benchmark for subpopulation shifts in domain generalization (ICML 2023)
* naacl_transfer_learning_tutorial: Transfer learning techniques in NLP (NAACL 2019)
* private-transformers: Training transformers with differential privacy
* imbalanced-regression: Handling data imbalance in regression tasks (ICML 2021)

Domain-Specific Applications
* **Healthcare and Bioinformatics*
* imbalanced-regression: Addressing imbalances in healthcare data (ICML 2021)
* hyena-dna: Long-range genomic modeling with Hyena
* OntoProtein: Protein function prediction with pretraining (ICLR 2022)
* GeoDiff: Molecular structure generation using diffusion models (ICLR 2022
 

* tape: Benchmark tasks for protein sequence modeling
* **Robotics and Autonomous Systems*
* LeGO-LOAM: Lidar-based odometry and mapping for robotics
* SuperGlobal: Image retrieval using global features (ICCV 2023)
* Deep_Metric: Embedding learning for metric-based image retrieval
* FEARTracker: Robust visual tracking (ECCV 2022)
* **Computational Creativity and Visual Arts*
* CCPL: Artistic style transfer for images and videos (ECCV 2022)
* GANgealing: GAN-based visual alignment (CVPR 2022)
* NeRCo: Low-light image enhancement using neural representations (ICCV 2023)

### Word Cloud of Topics

Figure 12 presents a word cloud that visualizes the frequency of topics across the repositories, highlighting the primary focus areas within GSRBench100. It provides a quick overview of the thematic concentration of the dataset.

## Appendix B Repository list and Commit ID

We specify the commit ID we used in our benchmark dataset for reproducibility. Please see table 6 and table 7 for details.

 

* [105] M. C. B. Green, J. A. Harvey, and R. E. Hinton. The classification of the Crab Nebula in the Crab Nebula. In _Proceedings of the SPIE Conference on Gamma-Ray Astrophysics_, pages 237-248. Springer, Berlin, Heidelberg, Germany, September 2006.

[MISSING_PAGE_POST]

 

* [105] M. C. B. K., and J. A. L. van der Bijnen. The role of the \({}^{32}\)He-\({}^{46}\)Ni-\({}^{78}\)Li-\({}^{90}\ 



## 10.4 Summary

In this paper, we have presented a new method for computing the performance of the proposed algorithm in terms of accuracy and efficiency of the proposed algorithm. We also present an implementation of the proposed algorithm in terms of accuracy and efficiency of the proposed algorithm. The results are shown in Table 8.

\begin{table} \end{table}
Table 8: Commands and errors for setting up 3DDFA_V2 repository using GPT-4o.





**LLaMA 3.1:**

Extracted commands for Environment Setup / Requirement / Installation:

pip install -r requirements.txt

brew install libomp

**Claude 3:**

Extracted commands for Environment Setup / Requirement / Installation:

git clone [https://github.com/cleardusk/3DDFA_V2.git](https://github.com/cleardusk/3DDFA_V2.git)

cd 3DDFA_V2 sh ./build.sh

Error message from bash shell:

"stderr": "Traceback (most recent call last):\n File \"/workspace/3DDFA_V2/FaceBoxes/ utils/build.py\", line 15, in <module>\n from Cython.Distutils import build_ext\n ModuleNotFoundError: No module named 'Cython'\nTraceback (most recent call last):\n File \"/workspace/3DDFA_V2/Sim3DR/setup.py\", line 7, in <module>\n from Cython. Build import cythonize\nModuleNotFoundError: No module named 'Cython'",

Analyzer suggested solution:

"command": "pip install Cython && sh ./build.sh",

**Mistral:**

Extracted commands for Environment Setup / Requirement / Installation:

git clone [https://github.com/cleardusk/3DDFA_V2.git](https://github.com/cleardusk/3DDFA_V2.git) && cd 3DDFA_V2 && ch ./build.sh && brew install libomp

Error message from bash shell:

"stderr": "bash: brew: command not found"

Analyzer suggested solution:

"command": "sudo apt-get install homebrew && git clone [https://github.com/cleardusk/](https://github.com/cleardusk/)

3DDFA_V2.git && cd 3DDFA_V2 && sh ./build.sh && brew install libomp",

\begin{table} \end{table}
Table 9: Commands and errors for setting up 3DDFA_V2 repository using LLaMA 3.1, Claude 3 and Mistral.