On Improving Repository-Level Code QA for Large Language Models (2024)

Abstract: Large Language Models (LLMs) such as ChatGPT, GitHub Copilot, Llama, or Mistral assist programmers as copilots and knowledge sources to make the coding process faster and more efficient. This paper aims to improve the copilot performance by implementing different self-alignment processes and retrievalaugmented generation (RAG) pipelines, as well as their combination. To test the effectiveness of all approaches, we create a dataset and apply a model-based evaluation, using LLM as a judge. It is designed to check the model’s abilities to understand the source code semantics, the dependency between files, and the overall meta-information about the repository. We also compare our approach with other existing solutions, e.g. ChatGPT-3.5, and evaluate on the existing benchmarks. Code and dataset are available online1.

heading: 1 Introduction
text: Coding assistants (Zhu et al., 2024; Nam et al., 2024; Luo et al., 2024), are invaluable to any programming team for developing software applications, games, or machine learning models involves writing code using programming languages. Commercial AI-assisted programming Chatbot like GitHub Copilot2, Codeium3 or Starcoder (Li et al., 2023) help to understand the code better, to generate some code, and to fix errors faster.
However, it is important to note that coding assistants may generate incorrect information, also known as “hallucinations”, when requests go beyond the model training data or require additional knowledge (Nguyen and Nadi, 2022). Another drawback of such assistants is the data protection problem: users need to be extremely careful while sharing private code and data with commercial coding assistants. Sensitive or proprietary code could
1https://github.com/pesc101/ma_llm.git 2https://github.com/features/copilot/ 3https://www.codium.ai
be exposed to unintended parties. This could potentially lead to data breaches and intellectual property concerns (Niu et al., 2023). Moreover, most coding assistants are of general use and cannot be applied to solve context-specific issues or answer natural questions based on repository-level semantics.
To mitigate these limitations, we develop two methods to improve the LLMs response quality on repository-level programming in a more specific, cost-effective and privacy-focused manner. One promising solution is Retrieval-Augmented Generation (RAG) (Lewis et al., 2020), incorporating the repository-level data into the generative process, to deliver accurate and relevant responses. The second approach is inspired by Zheng et al. (2024) and aims to increase the performance of the models by fine-tuning them with synthetic self-generated data using the self-alignment procedure. Finally, we combine a RAG pipeline with a fine-tuned model trained on a self-augmented dataset, which can be considered as both cost-effective and privacyfriendly approach that improves the performance of coding assistants on a specific repository.
When working on the repository-level programming tasks, selecting the appropriate source is also crucial, as it should represent common repository structures and be big enough to generate training data. Therefore, we consider the Python Spyder IDE repository4 at version 5.5 due to its abundance of short functions and extensive documentation.
We use the open-source model Mistral 7B (Jiang et al., 2023) as a base and fine-tuned model, connected to RAG pipelines. Mistral 7B is a pretrained LLM that outperforms Llama 2 7B, 13B (Touvron et al., 2023) and CodeLlama 7B (Touvron et al., 2023) on most benchmarks.
Regarding the evaluation techniques, we apply the LLM-as-a-judge (Zheng et al., 2023a; Peng et al., 2023; Bubeck et al., 2023; Wang et al., 2023;
4https://github.com/spyder-ide/spyder/tree/ master
303
Fu et al., 2023; Mao et al., 2023) method that leverages a superior model to judge other models responses. We utilize it to test whether adding information through fine-tuning or RAG pipeline improves the response quality. SpyderCodeQA, our new evaluation dataset, is used as the test data for as LLM-as-a-judge evaluation. Additionally, we apply the HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) benchmarks to measure the catastrophic forgetting of code generation abilities after fine-tuning.
The contributions of the paper are as follows:
• We introduce a new benchmark for the repository-level programming called SpyderCodeQA, which includes 325 question-andanswer pairs (Q&A pairs) from three question categories: semantics understanding, dependency understanding, and knowledge of repository meta-information.
• We compare three different methods for repository-level programming: LLM fine-tuning with self-augmented data (selfalignment), Retrieval Augmented Generation, and their combination.
• We perform an ablation study regarding the training dataset size and the type of the judging model and perform a preliminary quantitative analysis of the results.

heading: 2 Related Work
text: This section provides an overview of the existing studies related to the paper: repository-level programming, code-based Question Answering, and LLM evaluation.

heading: 2.1 Repository-level Programming
text: Recent studies have explored the application of instruction fine-tuning with PEFT techniques for coding tasks. Wang et al. (2023) demonstrated the effectiveness of PEFT for coding tasks on various models, highlighting the effectiveness of QLoRA for fine-tuning. In a related study, Yuan et al. (2023) investigated the performance of instruction finetuned models on a range of coding tasks.
Researchers have also explored generating prompts for few-shot learning using RAG pipelines (Nashid et al., 2023) as well as the combination of fine-tuning and RAG pipelines using several open-source models to inject additional information (Ovadia et al., 2023).

heading: 2.2 Code-based Question Answering
text: Code-based question answering is a subfield of question answering that focuses on responding to code-related queries. Unlike generative approaches, retrieval-based code Q&A aims to find the most relevant code snippets from a large code corpus to satisfy user requests. To evaluate the performance of the models, Husain et al. (2019) introduced CodeSearchNet, a collection of datasets and benchmarks created by mining large-scale commentcode pairs from public GitHub repositories. Liu and Wan (2021) presented CodeQA, a free-form code question-answering dataset to assess the code comprehension capabilities of language models. CoSQA (Huang et al., 2021) mines real-world user queries from Bing search logs that were labeled if the provided answer is the solution to the question.
Although these Q&A datasets are useful for measuring the interaction of models and humans, they are unsuitable for repository-level programming tasks: CodeSearchNet and CodeQA have direct question-answer interaction. While CoSQA (Huang et al., 2021) consists of real human queries, they are only related to general coding tasks and have no label for a repository, which makes it difficult to use the Q&A pairs as training data to measure the performance of a specific repository.

heading: 2.3 Evaluation of LLMs
text: Evaluating the capabilities of LLMs has been challenging due to their vast and diverse abilities and the lack of standardized benchmarks to measure human preferences in this rapidly evolving field.
LLM-as-a-Judge LLM-as-a-judge is an evaluation method for LLMs in which a superior model is used to judge the results of other models. Zheng et al. (2023a) proposed three variations of Modelbased-evaluation referred to as LLM-as-a-judge. The first, pairwise comparison (Peng et al., 2023; Bubeck et al., 2023), involves directly assessing two answers to determine superiority or a tie. The second, single answer grading, assigns a score directly to a response (Wang et al., 2023; Mao et al., 2023). The third, reference-guided grading, incorporates a reference solution, beneficial for math problems (Bubeck et al., 2023).

heading: 3 Dataset Construction
text: In order to measure the performance the performance of the models on repository-level programming, we create a new evaluation dataset named
SpyderCodeQA comprising of 325 samples established on the Spyder IDE5. It is based on three dimensions: source code semantics comprehension (Subsection 3.1), dependency comprehension (Subsection 3.2), and meta-information comprehension (Subsection 3.3). Figure 1 presents an overview of the three dimensions of the evaluation dataset. The first one aims at understanding the containing text and code elements about the repository source code and being able to answer semantic questions about it. The second dimension evaluates the ability to understand the relationships between files within the repository and between files and imported libraries. The third dimension assesses the ability to understand general information about the repository using README files (build commands, requirements or legal information of the repository, unrelated to the source code).
The following subsections provide an overview of the creation process for each dimension in detail. Typical samples for each dimension are shown in Appendix A in Figure 8.

heading: 3.1 Source Code Semantic Comprehension
text: For creating the source code semantics comprehension dimension, ten experts computer science are asked to manually create the Q&A pairs using the Spyder IDE repository source code. For this purpose, we develop a custom web application using Python Django6 to write question pairs given the code snippet (see Appendix A for more details).
The first goal is to create Q&A pairs for one of the 5673 snippets (2000 characters max) from the 2083 Python files randomly selected from the open-source Python repository Spyder IDE. We
5https://github.com/spyder-ide/spyder/tree/ 0f8398a9a27d401b9984f6e049ef1199656900f1
6https://www.djangoproject.com
demonstrate those code snippets in the web application and ask the experts to create a question and the answer. Meta-information such as the module name, file name, and the start and end line of the code snippet is also given. The example of the interface is shown in Appendix A in Figure 9a.
The second task is to rate the created Q&A pairs from other participants to ensure the quality of the pairs on a 1-10 scale and optionally leave comments. The instructions for the rating task and the process for rating the Q&A pairs are shown in Appendix A. In the interface, the text areas are replaced with two rating forms (Figure 9b).
The last step of the dataset collection is the quality control of the collected Q&A pairs. In total, 189 questions were created and rated by the experts. The pairs scored with less than 3 points are automatically removed from the dataset. Pairs with a rating between 3 and 5 are manually curated. As a result, the final size encompasses 140 Q&A pairs.

heading: 3.2 Dependencies Comprehension
text: Q&A pairs for dependencies comprehension aim at measuring the ability to understand the dependencies between code files. Therefore, we present the AST algorithm (Appendix A) to identify dependencies between files, modules, and libraries.
It recognizes four types of imports: complete library imports, imports from libraries, complete file imports, and imports from files. We also identify the type of the imported artifacts (class, function, or assignment): whether is it a library-based or a file-based import. The algorithm also provides information on each Python file in the repository (file name, import category, and artifact name).
The raw dependencies are further processed with the OpenAI API using the “gpt-3.5-turbo-1106”7 model (temperature is set to 1.5, the maximum token limit is 256, and the top p-value is 1, the frequency and presence penalties are set to 0). In Appendix D, Figure 12 presents the full system prompt for generating the Q&A pairs along with the example to improve generation abilities.
As a result, 1319 Q&A pairs were generated using the OpenAI API from 686 unique file names. To ensure the quality of the dataset, a final set of 135 Q&A pairs was randomly chosen and manually verified for correctness by an expert annotator. This was done by cross-checking the repository’s source
7https://platform.openai.com/docs/models/ gpt-3-5-turbo
code to ensure that the questions and answers were both correct and made sense. The random selection process was implemented to minimize the amount of manual effort required for verification.

heading: 3.3 Meta-Information Comprehension
text: To understand the model ability to understand general information about a repository, such as its purpose, features, documentation, license, and contribution opportunities, we create Q&A pairs for the meta-information dimension. We first extract all files with the suffixes .md, .txt, and .yml, resulting in 29 files that included meta-information. We focus on the information about the repository installation, the available and supported versions of the packages, and the rules for contributing. We ask our expert annotator to create triplets containing questions, answers, and meta information (file name and the module) resulting in 50 questions.

heading: 4 Methodology
text: This section describes the methods we implement in the paper. First, we describe the data preprocessing step (Subsection 4.1), which is common for all approaches. Then we explain the self-alignment approach in Subsection 4.2 and our implementation of RAG in Subsection 4.3. Subsection 4.4 explains how both approaches can be combined.

heading: 4.1 Data Preprocessing
text: To fit the desired structure for fine-tuning models using self-alignment or creating a vector database for RAG, a pre-processing pipeline is created.
First, we fetch the Spyder repository and load each file type using individual loader classes. With a chunk size of 1500 characters and an overlap of 200, the file was divided into chunks of a maximum of 1500 characters, each overlapping by 200 characters. From the code chunks, all available metadata
is extracted: file name, module, flag whether the chunk contains a class or function, start and end line numbers, and all file imports. In the final step, the extracted metadata are added to the chunks and saved as .jsonl file and uploaded into Huggingface8.

heading: 4.2 Fine-tuning with Self-Alignment
text: This subsection overviews the fine-tuning process with self-alignment mainly inspired by Zheng et al. (2024). It comprises of the following steps: data generation (self-augmentation), data curation (SelfCuration). Afterwards, we perform the model finetuning on the generated dataset.
Self-Augmentation First, we provide the repository code chunks as input into the base model (Mistral 7B) to generate the dataset D0 that explains each line of code in the chunk and add one randomly selected question from a predefined question corpus (See Appendix B). Then, we generate the Q&A pairs (D1) from this source code explanations D0. We instruct the module to include file and module names to ensure the model always knows the file the question aims for. The prompt also specifies that code should be added to the answer. Both system prompts for generating D0 and D1 are shown in Appendix D in Figures 12 and 13.
In addition to the code chunk with explanations from D0, we also provide an example question selected from a question corpus inspired by Liu and Wan (2021). We manually limit possible question examples to be used, as the question should belong to one of three dimension types: source code semantics, dependencies and meta-information, likewise the dimension in the manually created dataset in Section 3. The list of selected questions can be found in Figure 10.
It is important to note that the pipeline to generate Q&A examples can be executed multiple times in a row, resulting in datasets that differ from each other. We execute the self-augmentation step twice for 7943 chunks to create two datasets D0, resulting in 15,886 data samples that are further processed to the curation step of the Q&A dataset.
Self-Curation To generate high-quality training data, we curate the data samples to collect the final dataset denoted in Figure 2 (a) as D2. We ask the base model (Mistral 7B) to evaluate the Q&A pairs on a scale from 1 to 5. The system prompt is displayed in Fig. 15. The model evaluates whether the
8https://github.com/pesc101/ma_llm/blob/main/ README.md
response is a good example of how an AI Assistant should respond to user instructions. A score of 1 indicates that the answer is incomplete, not precisely what the user asked for, or off-topic. A score of 5 represents a clear and well-structured answer from an AI assistant that thoroughly answers the user’s question. All examples with a score lower than 4 are removed from the dataset. As a result, our training dataset comprises 14,434 Q&A pairs.
Fine-Tuning The base model (Mistral 7B) is trained for 5 epochs using supervised fine-tuning (SFT) (Ouyang et al., 2022), 4-bit Quantization Low-Rank Adapters (QLoRA) (Dettmers et al., 2023) on the generated Self-Aligned dataset and Flash Attention 2 (Dao, 2023). After the training, the LoRA layers were merged into the base model Mistral 7B to reduce the response time when using the model for inference. The training details can be found in Appendix C.

heading: 4.3 RAG Approach
text: The implemented RAG pipeline is illustrated in Figure 2 (b). We use the preprocessed chunks to generate 768-dimensional vector representations of chunks using the Instructor embedding model (Su et al., 2023). This pre-trained model with 110 million parameters generates embeddings that can be used for retrieval, classification, or semantic search tasks. The data is stored in the in-memory version of Chroma9, an optimized database for storing vector representations. The database is initialized by assigning an ID to each chunk and indexing the metadata. This ensures a quick response time and enables data retrieval based on metadata queries. For the retrieval step, we also use the Instructor model to transform the query into a standardized 768-dimensional vector. During the generation step, we use the system prompt displayed in Figure 16 as input to the LLM (the base Mistral 7B model) to generate the answer. It utilizes the question and the retrieved code chunks as input and generates the answer to the question as output. Thus, our RAG approach aligns with the concept of "inference" (Huang and Huang, 2024).
It is also important to note, that we apply both Instructor and Mistral 7B models without additional fine-tuning.
9https://www.trychroma.com

heading: 4.4 Combined Approach
text: As the combined approach, we replace the base Mistral 7B model with the fine-tuned model from the self-alignment step in Subsection 4.2. We expect the fine-tuned model might produce better results when enhanced with the correct chunks from the RAG pipeline. Additionally, retrieved chunks should also prevent the LLM from hallucinating.

heading: 5 Evaluation
text: This section describes two evaluation strategies applied in the paper: using LLMs (primarily GPT 3.5/4) as judges (Zheng et al., 2023a) and standard benchmark evaluation using metrics. LLM-as-ajudge methods are preferred over BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004), as they can only evaluate the semantic similarity between human and model-generated responses, which might not be related to the correctness of the responses.

heading: 5.1 LLM-as-a-judge
text: The performance of models on the Q&A evaluation dataset created in Section 3 is evaluated pairwise using strong LLMs (primarily GPT 3.5/4) as judges (Zheng et al., 2023a) (using the same hyperparameters for the judge model as in generation: temperature of 0.7, top-P of 0.9, and max token of 2500). We test the base model against its modified version (finetuned Self-alignment, RAG, or the two methods combined).
Figure 3 shows the model-based pairwise comparison pipeline. For each Q&A pair in the evaluation dataset, the two models M1 and M2 answer the question of the Q&A pair. Then the LLM (GPT3.5) model is instructed in the system prompt to act as a judge to evaluate the quality of responses A1 and A2. The prompt template is shown in Figure 18. It consists of a question (“User Question”) and
the generated answers (“Model Solution”). To ensure clarity, each piece of information is enclosed with an identifier in square brackets, indicating the type of information. The evaluation could also result in “No value” when the judge does not return the output in the correct format.
We utilize the Average Win Rate (AWR) metric for evaluation. AWR is the proportion of Q&A pairs the judge has decided that one model is better than the other or it is not a tie. The average is calculated over k runs executed with the same parameters to take into account possible deviations.

heading: 5.2 Existing Benchmarks
text: In addition to evaluating whether a coding assistant has become better at answering questions about a repository, we also test whether the code generation abilities have changed after fine-tuning. Therefore, two benchmarks are used to evaluate the “catastrophic forgetting”: HumanEval introduced by OpenAI (Chen et al., 2021) and Mostly Basic Programming Problems (MBPP) (Austin et al., 2021). Both benchmarks use the pass@k unbiased estimator which is computed as follows (n is the total number of samples, c is the number of correct samples and E is the expected value):
pass@k := E Problems
 1− ( n− c k )
( n k )
  (1)

heading: 6 Results and Analysis
text: In this section, we present the results using LLMas-a-judge and the existing benchmarks (Subsections 6.1 and 6.2). In Subsection 6.4, we discuss the additional experiments with the training size and applying GPT-4 as the judging model. The
qualitative analysis of the results can be seen in Subsection 6.3 and in more detail in Appendix F.

heading: 6.1 LLM-as-a-Judge on SpyderCodeQA
text: The average win rate results for k = 3 runs are shown in Figure 4 for all approaches. We describe them separately in the following paragraphs.
Fine-tuning with Self-Alignment The results in Figure 4 (a) suggest that in approx. 57% of the Q&A pairs, the answer of the fine-tuned model is preferred, while in approximately 36% of the pairs, the answer of the base model is preferred. The LLM-as-a-judge evaluation method consists of k = 3 runs, where in each run the order of the answers given to the judge is randomized to reduce position bias. The error bars indicate the standard deviation of the runs. The low variance for each output indicates that LLM-as-a-judge is consistent over several evaluation runs.
Additionally, the fine-tuned model performs best on the human-labeled dimension code semantics. With 62%, it won almost two-thirds of the Q&A pairs. For the dependency dimension, the finetuned model is also better than the base model but has only a 54% win rate. The model performed the worst in the meta-information dimension, indicating that the fine-tuning process reduced its performance in this dimension.
RAG Approach In Figure 4 (b), we can see that for 57% of the Q&A pairs, the judge prefers Mistral 7B with the RAG pipeline, which aligns with the previous approach. Also, the win rate for the base model and the percentage of Q&A pairs that aren’t
correctly judged is similar to the Self-alignment pipeline and are close to 37% and 5% respectively.
The results of the different dataset dimensions differ from those of the Self-alignment pipeline. Although both approaches perform the same with a 1% difference in the code semantics dimension, there is a difference of 2 standard deviations in the results for the dependencies. The meta-information dimension shows the biggest difference, with the base model using the RAG pipeline outperforming the base model. This suggests that the RAG pipeline supports the model in answering questions related to the meta-information but is less useful for answering questions regarding dependencies.
Combined Approach The results for the comparison with the combined approach are shown in Figure 4 (c). The average win rate is approximately 64%, which is higher than that of the two pipelines, respectively. This suggests that there is a positive interaction effect between them. When examining each dimension separately, the best results are achieved for the code semantics dimension. With an average of 70% win rate, the model is in 7 out of 10 questions better than the base model. That indicates that this combination is a further improvement regarding code semantic questions. The results for the dependencies dimension demonstrate an average win rate of 61% and also indicate the efficiency of the interaction of both pipelines. For the meta-information dimension, the model shows a 51% average win rate, which means no improvement over the base model.
GPT-3.5 Turbo In the last experiment, we compare our best-performing model with the gpt-3.5-turbo-1106 as a code assistant instead of the base model. We acknowledge that the gpt-3.5-turbo-1106 approach does not get code snippets as input, however, our main idea was to check whether the fine-tuned model indeed learns the context from the given repository. Otherwise, the results of the Self-alignment fine-tuned model and GPT-3.5 would be comparable. It is worth noting that GPT-3.5 was utilized as the judge as well; therefore, it rates its responses in this experiment.
The results are presented in Figure 4 (d). The combination of the fine-tuned model with an RAG pipeline outperforms GPT-3.5, with an average win rate of 72%. Only 20% of the Q&A pairs were won by GPT-3.5. However, it is worth noting that the rate of not finding a rating by the judge is slightly higher than with Mistral 7B.
The code semantics and dependencies results are even better at the dimensions, with 78.3% and 74.07%, respectively. That indicates that the finetuned model with the RAG pipeline is a better coding assistant on repository level than GPT-3.5.

heading: 6.2 Benchmark Results
text: Figure 5 presents the percentage of solved tasks by the base model Mistral 7B and the fine-tuned model with Self-Aligned data on the HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) benchmarks. For each benchmark, the pass@1 and pass@10 are calculated. However, the results for both benchmarks are not very promising. The base model outperforms the fine-tuned model on HumanEval on pass@1 with 6.8% and on pass@10 with 8%. Similar results were found on the MBPP benchmark with a difference of 11.5% on pass@1 and 12.6% on pass@10. This decrease in scores indicates that the general coding ability of the finetuned model has been reduced. The possible reason for the poorer performance could be the modified prompt template, as the model is fine-tuned for answering Q&A pairs and not for pure coding tasks.

heading: 6.3 Results by Question Type
text: We also take a closer look at the concrete examples and provide more qualitative insights about how the RAG pipeline affects the output of the LLM model and improves performance. The examples are shown in Appendix F. Each example consists of the original question and answer, the answer of the two models, and the judgment at the end.
Regarding the Source code semantics comprehension, we can see from Figures 21-23 that each approach demonstrates its benefits when combined. The fine-tuned model answer is nicely formatted, and the RAG pipeline answer is contextually correct. The combination of both approaches fulfilled both requirements, providing a well-formatted answer with a good explanation of the class and the correct code snippet. For the Dependencies types of question in Figures 24-26, we can see that the base and the fine-tuned models without RAG cannot provide information about imports used, therefore, they might not be able to perform well for these tasks. Meta-information types of questions show a similar trend in Figures 28 and 29 where approaches using RAG in the pipeline demonstrate a more accurate response.
Quantitative results in Tables 1-3 (Appendix E) demonstrate quite an opposite tendency: for the Dependencies Meta-information types of questions GPT-3.5/4 prefer the pipelines with RAG in fewer cases than the RAG and Combined approaches. Code Semantics questions are better solved when provided the context from RAG and the Combined approach. Nevertheless, all developed pipelines outperform the base model for all three dimensions.

heading: 6.4 Ablation Study
text: This section presents supplementary experiments that provide a deeper insight into the number of dataset samples and the choice of the judge model.
Training Dataset Size To create different sizes of the training dataset, the self-augmentation was executed once (a), twice (b), and quadruple (c). The related loss curves and learning rates are shown in Appendix C. From the results in Figure 6, we can see that in all three experiments, each fine-tuned model learned about the repository, as reflected in the higher average win rates compared to the base model. However, the best-performing model was achieved using the self-alignment pipeline twice to create the training dataset. The Average Win Rate is considerably higher than the models trained with one or quadruple datasets, with an improvement of approximately three standard deviations.
We assume that the reason for the optimal number (2) for the self-alignment step might be explained by the number of unique Q&A pairs. The quadruple design adds only a few new pairs while having many duplicates, which may cause the model to overfit.
Judgement with GPT-4 Turbo The results of comparing the GPT-3.5 and (more expensive) GPT4 models as judges are presented in Figure 7. The corresponding results for each dimension can be found in Appendix E in Table 3. Both judges rate the quality of the response of the fine-tuned model with the RAG pipeline higher. However, GPT-4 prefers more the fine-tuned model and chooses a tie in almost 10% as judgment, which is more often than GPT-3.5. Furthermore, only 0.3% of the answers belong to the “No value” type, indicating that GPT-4 can judge the performance of models more consistently and accurately.

heading: 7 Conclusion
text: In this paper we introduce a new manually created dataset — SpyderCodeQA – which includes 325 question-and-answer pairs (Q&A pairs) from the Spyder IDE repository. We split it into three question dimensions: semantics understanding, dependency understanding, and knowledge of repository meta-information. We also present a series of experiments using Self-alignment, RAG, and their combination to evaluate LLMs’ performance on repository-level code Q&A using the generated dataset. We show that the quality of the system can be significantly improved when applying both approaches together: the LLM-as-a-judge win rate is approximately 64%, which is 7% higher than both approaches separately. Regarding the models’ performance on different dataset dimensions, we can see that they perform exceptionally well for code semantics, which is the human-labeled dimension.
In future work, we plan to improve the Selfalignment pipeline to create a more diverse dataset that includes Q&A pairs mainly focused on code generation to prevent the “catastrophic forgetting” of the model. Another possible direction is to perform the human evaluation to better align the model with user needs. It would provide additional insights since humans are the target audience for Q&A on repository-level programming, and they often have more knowledge about the repository, allowing them to better judge the model’s responses.

heading: Limitations
text: This section outlines the limitations regarding the approaches and the created dataset.
Small dataset size Other datasets in this research area include CS1QA (Lee et al., 2022), a dataset for code-based question-answering in the programming education domain or CodeQA (Liu and Wan, 2021) for the code comprehension task have much bigger samples than the dataset that is introduced in this thesis. CS1QA with over 9k pairs and CodeQA with approx. 200k are much bigger. While both datasets aim for slightly different goals, it is important to mention that the generalizability and value of the evaluation results should be treated with caution.
Different Knowledge Level of Creators For the source code semantic dimension, the Q&A pairs were created by humans. While the number of participants was with ten people quite low, also the knowledge level of the participants about the Python programming language and the working experience was high. That could lead to a bias in the difficulty of the questions asked. Assuming you want to test whether a model can answer simple questions for beginner programmers, the questions from the semantic dimension may not necessarily be helpful and accurate.
Unknown repository The individuals who took part in the study considered themselves experts in Python, however, none of them had previously contributed to the Spyder IDE repository. Essentially, this means that none of the participants were experts in this specific code base. Although this may not pose as a disadvantage, it does suggest that the questions and answers provided may not be as in-depth as those provided by a Spyder IDE contributor.
Low heterogeneity of the Q&A pairs in dependency dimension The Q&A pairs in the source code semantic dimension have a great variety, but the ones generated automatically in the dependency dimension are often very similar. The reason behind this is to assess the model’s ability to answer these questions accurately. However, a wider range of questions would be preferable to test the model’s performance as a coding assistant. Therefore, a further improvement of the dataset would be adjusting the model’s system prompt that generates the Q&A
pairs or developing a new way to measure the dependencies of the different repository components.
Only 1-hop Dependencies The relationship between the two source code files is adequately described using the dependencies dimension. However, the dataset dimension lacks a mapping that goes beyond the linking of two files. Therefore, it would be beneficial to devise a way to create 2-hop or even n-hop structures that the models can comprehend.

heading: Meta-Information dimension is self-generated
text: The quality of the source code semantic dimension dataset was ensured through a rating process conducted by participants. The dependency pairs were also manually verified to be correct. However, the meta-information dimension lacks quality testing. The Q&A pairs were created exclusively by the author of the thesis, which could introduce bias in the formulation of the questions and answers and the selection of information to create the pairs. This dimension may not be as objective as others, as different people may create completely different pairs.
Self-generated training dataset The Q&A pairs generated by the self-alignment process may not be semantically and syntactically correct. Although the model has been trained to match questions with the corresponding answers, it is not guaranteed that the generated code is functionally correctly reproduced and that the generated question is similar to a user request. The model itself curates the Q&A pairs, but the curation can only verify if the question matches the answer and seems to be correct. Therefore, the curation/verification process could be further improved in this pipeline step.
Data is limited to one Python repository The evaluation is limited to one Python repository that has its unique structure. This is important to consider as the model may behave differently when applied to other repositories, which could result in biased results. In addition, the evaluation results only cover a limited set of questions that could arise concerning repositories. Given the vast range of programming languages, frameworks, and projects, these results may not be applicable in all scenarios.
Choice of Model & Embeddings There exist dozens of large pre-trained generative models and embeddings that could be applied to the task. However, we report the results for the Self-Alignment
technique only with Mistral-7B and the basic RAG approach with the Instructor embeddings. An alternative base LLM or embeddings could further improve the results.
Our goal was to compare RAG-based and finetuning approaches on the repository-level Question Answering task and not to make an exhaustive search of all models, embeddings, and pipelines. We leave these experiments as future work.
Using LLM-as-a-judge instead of human evaluation Regarding evaluating the model’s performance, the LLM-as-a-judge approach also has its limitations. Despite the elimination of the position bias and the attempts to use GPT-4 as a judge, the evaluation is not flawless. The superior model judges the answers, but sometimes, the criteria are chosen by the model itself and do not match those of humans. Also, the correctness of the produced code is often not sufficiently verifiable for the model, as it does not have access to the necessary source code.
Chunking of the code file Despite its advantages, the RAG pipeline has some limitations that must be considered. One major limitation is that the context provided to the LLM is always just a portion of the file, which means that knowledge about multiple files is not processed, and the connection between the files and the code cannot be considered. To address this, the context would need to be pre-processed better. One possible solution is to have a hierarchical structure that provides context at different levels and contains summarized knowledge. For example, a description of what a module is responsible for or how the general structure of the module could be added to each chunk of each file in the module. That additional information should further help the model gain a deeper understanding of the repository.
Number of chunks retrieved For all experiments, the number of chunks was set to N = 1, but it could also be interesting to test whether the number of chunks could further improve the model’s performance. Also, the size of the chunk and the overlapping characters are possible variables for optimizing the results.
Catastrophic Forgetting As mentioned in Sec. 5, the model’s performance decreased on both MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021) benchmarks following the fine-tuning
process. This shows that fine-tuning can change models’ abilities to perform certain tasks. Therefore, the conducted experiments do not clarify how the model enhances its capacity to handle the context, particularly source code, and grasp it deeper after the fine-tuning process.

heading: A SpyderCodeQA
text: Dataset Examples For each dataset dimension, three Q&A pairs from SpyderCodeQA are presented.
Django Web App Interface An online study was conducted to create these pairs, and a custom web application was developed using Python Django as a backend service and HTML, CSS, vanilla JavaScript, and Bootstrap 5 for the user interface. The web app was hosted on a private home server during the data collection. Fig. 9a shows the UI for creating Q&A, and Fig. 9b for rating the Q&A from other participants.
Creation of Code Semantics Q&A Participants were given a random code snippet from the open-source Python Spyder IDE code repository during the online study. These snippets were generated using the LangChain package’s document loader and text splitter1. The 2083 Python files in the repository were divided into 5673 text chunks to create these code snippets. The source code was chunked using Python syntax and specific cutting points like \nclass, \ndef, and \n\tdef. Each chunk was not larger than 2000 characters. If the splitter within the chunk size found none of these cutting points, the splitter uses
1https://python.langchain.com/docs/modules/data_connection/document_transformers/
(a) Web App frontend for creating Q&A pairs. Two input fields are on the left for entering questions and answers, and the Code snippet is on the right. Users submit a Q&A with the green button and mark it as Bad Code, e.g., the code snippet is not understandable, with the yellow button. (b) Web App frontend for rating Q&A pairs. Two slider inputs are on the left for entering a rating from 0 (bad) to 10 (perfect), and the Code snippet is on the right. Users submit a rating using the green button. Understanding problems with rating the Q&A pair resulted in submitting the red button.
Figure 9: Web App frontends for creating and rating Q&A pairs.
secondary cutting points such as \n\n, \n and " ". In addition to the source code, meta-information about the code snippets were stored. That included the file’s name and module and the start and end lines of the source code. The procedure for identifying the start and end line involved fetching the file path of the code snippet and comparing its content with the original file’s content. It then located the starting line of the snippet by matching its first line with the lines in the file and determined the end line based on the snippet’s length. The function also accounted for edge cases where the snippet may not be found within the file or consists of only one line. After creating chunks of source code and meta-information, the data was stored in an SQLite database using Django object-relational mapping in Python.
The interface for the creation task is shown in Figure 9a. The left side of the interface contained two text areas, one for entering the question and the other for entering the answer. On the right side, the code snippets from the repository were displayed, along with meta-information such as the module name, file name, and the start and end line of the code snippet.
Participants were given login credentials via messenger or email with a link to the web application. Before executing the study, each user was asked to provide personal information. The required information included their highest computer science degree (Bachelor’s, Master’s, PhD, etc.), the number of semesters studied in total (rated on a scale of 1-10+), their self-rated coding skills (general and Python, rated on a scale of 1-5), and their field of study. This information was only collected to filter out bad Q&A pairs when participants had low coding or working experience.
Users could pause the study by logging out and resuming where they left off later, as the app automatically saved their progress. The execution duration of the study lasted an average (median) of 1 hour and 22 minutes, with the fastest participant finishing in 38 minutes and the slowest in 8 hours and 18 minutes. This large number is because the participants could interrupt the study to continue it later.
Creation of Dependencies Q&A The keywords import or from are used in Python to import an artifact. The algorithm identified four types of imports: complete library imports, imports from libraries, complete file imports, and imports from files. It is possible to identify the type of imported artifact for the categories imported from the library and file. The algorithm provides information on each Python file in the repository, including the file name, import category, and artifact name. The analysis involves a DirectoryAnalyzer to evaluate directories and a FileAnalyzer class to analyze individual files.
The DirectoryAnalyzer class is designed to systematically analyze a given directory’s contents. Upon invocation of the analysis procedure with a specified directory as input, the algorithm initializes an empty list to store the results. Utilizing the walk() function from the os package in Python, the algorithm traverses through the directory hierarchy from the top-down, iteratively examining each file encountered.
For files with a ".py" extension, the algorithm constructs the full file path and instantiates a FileAnalyzer object to analyze the file further. The dependencies of the file are then retrieved through the analysis method of the FileAnalyzer object, and these dependencies are appended to the list of results. Finally, the algorithm returns the accumulated list of file dependencies, providing insights into the interdependencies within the directory’s Python files.
The FileAnalyzer class extracts the dependencies from the Python files. Upon invocation of the analysis procedure with a file object as input, the algorithm first reads the content of the file and initializes an empty list to store samples. Subsequently, it iterates through the Python code’s AST representation, identifying import statements. Depending on whether the import is of the form import module or from module import ..., the process_node procedure is called to extract the relevant dependency information. This information includes the imported library name, the category of import (either "file_import" or "library_import"), and the file path of the imported module.
The process_node procedure, implemented within the same class, is responsible for processing individual AST nodes corresponding to import statements. It discerns the library name and import category, retrieves the file path of the imported module, and appends this information to the list of dependencies. Furthermore, the get_artefact_type procedure, also part of the FileAnalyzer class, determines the type of artefact defined in the Python file (e.g., function, class, variable) by traversing the AST and inspecting its structure.
Additionally, the is_file_import() function aids in determining whether an import statement refers to a file within the project directory or an external library. This function evaluates the module name and checks if it corresponds to a file within the project directory structure. If the module name starts with a dot (indicating a relative import), it constructs the full file path and checks its existence. Otherwise, it searches for matching files within the project directory using a specified search pattern.
The analysis of the Spyder IDE repository revealed that it has 7907 dependencies. The data shows a significant difference between the types of imports used. The project heavily relies on libraries, with 3305 instances sourcing the whole library and only 27 instances sourcing the whole files directly. This suggests that the project prefers to use external resources instead of local file dependencies. Furthermore, the dataset indicates that 686 files were used in the project, indicating that the project operates at a moderate scale. When examining only the imports from files, the imports are mainly classes, with 1265 occurrences, followed by functions, with 1048 instances, and assigns, with 569 instances. Additionally, the algorithm failed to predict the correct artifact type in 140 instances where the artefact type was unknown. This distribution highlights the predominant use of classes and functions.
Data Aggregation The raw dependencies were processed further using the OpenAI API using the "gpt-3.5-turbo-1106" model. The temperature was set to 1.5 to ensure creativity in the creation process, the maximum token limit was 256, and the top p-value was set to 1. The frequency and presence penalties were set to 0. These parameters were carefully selected to create diverse, contextually relevant questions and concise, coherent responses within specified token limits. To ensure that good Q&A pairs are built, a system prompt must lead to the desired result. Fig. 12 presents the system prompt for generating the Q&A pairs. Before generating the pairs, the assistant was instructed to create questions that could be answered with a "no". This ensured that guessing the most common libraries would not be a viable solution. Example questions were provided to help guide the model, such as asking which libraries were used in a particular file or where a function belongs to a particular library.
1319 Q&A pairs were generated using the OpenAI API from 686 unique file names. Despite several attempts to modify the prompt to yield only one question and answer, the API often returned several questions and answers for a single request. To ensure the quality of the dataset, a final set of 135 Q&A pairs was randomly chosen and manually verified for correctness. This was done by cross-checking the repository’s source code to ensure that the questions and answers were correct and made sense. The random selection process was implemented to minimize the manual effort required for verification.

heading: B Question Corpus for Source Code Semantic

heading: C Training Conditions
text: The model has trained 5 epochs with batch size 32 on an NVIDIA RTX A6000 with 49GB VRAM. The computing cluster consisted of 128 CPUs and 1TB of RAM. The model was trained using BF16 precision, which reduces the model’s memory consumption and improves performance and gradient checkpointing to reduce memory accumulation. Cross-entropy loss was used, while the Adam optimizer was used with β1 = 0.9, β2 = 0.999, following the implementation by Zheng (Zheng et al., 2023b). The learning rate was set using a cosine decay scheduler, starting with an initial learning rate of 1e − 4 and a warm-up ratio of 0.03. During each training run, the loss consistently decreased, with a significant drop at the end of each epoch. The learning rate also behaved as expected, with the warm-up ratio leading to an initial increase in the learning rate, followed by a gradual decrease over the training duration.
For quantization: LoRA R and Alpha 64, following the approach of equalizing the number of R and Alpha to reduce noise, as suggested in this blog post10. LoRA dropout was set to 0.1 and the weights were calculated in 4-bit using normalized float-4 (NF4) for the calculation, as recommended by Dettmers et al. (2023).
Flash Attention 2 (Dao, 2023) was used to speed up model training by a factor of 3 (Dao, 2023). For the dataset with 14434 samples, the five-epoch training took four and a half hours. After the training, the LoRA layers were merged into the base model Mistral 7B to reduce the response time when using the model for inference.
10https://medium.com/@fartypantsham/what-rank-r-and-alpha-to-use-in-lora-in-llm-1b4f025fd133

heading: D Prompt Templates

heading: E Evaluation Results

heading: F Q&A Pairs from the LLM-as-a-Judge evaluation
text: We take a closer look at the concrete examples and provide more qualitative insights about how the RAG pipeline affects the output of the LLM model and improves performance. The examples are shown in Appendix F. Each example consists of the original question and answer, the answer of the two models, and the judgment at the end.
Code Semantics For the Code Semantics dimension example, the self-alignment and RAG pipeline evaluations are shown in Figures 19 and 20, respectively. The answers and judgments for both combined are presented in Figure 21.
The question seeks an explanation of the class functionality. As anticipated, the base model (Mistral 7B) states its inability to provide a precise answer due to lack of access to the code, attempting to infer the benefit from the name but remaining vague. Conversely, the fine-tuned model confidently explains the class’s usage and returns a code snippet it considers correct. GPT-3.5 favors the fine-tuned model in its judgment despite the model hallucination — the generated code is incorrect. The judge assumes the presented code snippet is correct and is satisfied with the answer, as it directly addresses the user’s question and includes the code.
The RAG pipeline evaluation in Figure 20 provides the base model with the correct code snippet which results in a decent explanation. Therefore, GPT-3.5’s judgment again favors the modified variant (RAG) and not the base model, recognizing that the answer correctly explains the code functionality.
When considering Figure 21, we can see that each approach demonstrates its benefits when combined. The fine-tuned model answer is nicely formatted, the RAG pipeline answer is contextually correct. The combination of both approaches fulfilled both requirements, providing a well-formatted answer with a good explanation of the class and the correct code snippet. This example provides valuable insight into the effect each approach has on the base model.
Dependencies This question presented in Figures 24, 25, and 26 is a typical example from the dependencies dimension, asking for all imported libraries in a file. The correct answer lists all imports used in the file. In evaluating the Fine-Tuned model (Figure 24), the base model correctly acknowledges its limitations, stating that without access to the file source code, it cannot provide information about the imports used. In contrast, the Fine-Tuned model provides a confident but entirely incorrect answer, denying the presence of external library imports and listing non-existent internal imports from other repository modules. Unfortunately, the judgment fails to recognize the Fine-Tuned model answer as incorrect, instead describing it as accurate.
In contrast, the answer provided by the base model with the RAG pipeline (Figure 25) is more accurate. The model manages to list all imports, including those from external libraries and internal repository modules, when given the correct code snippet as context. Although the response format is not aligned, the content is correct. The combined evaluation of both approaches (Fig. 26) generates an answer aligned to the provided format, listing only the external libraries without internal imports.
Meta-Information The evaluation results for the Q&A pair from the meta-information dimension, are presented in Figure 27 and Figure 28. This question is a typical example from this dimension, as it inquires about the minimum size of the Spyder logo, a piece of information easily verifiable for correctness.
When evaluating the fine-tuned model, as shown in Figure 27, we can see that the base model lacks access to the source code. The fine-tuned model attempts to guess the answer are also incorrect. The judge recognizes the fine-tuned model answer as incorrect, and instead of penalizing both models, it favors the base model response for its transparency in acknowledging the lack of information.
In contrast, the answers from the base model and fine-tuned model using the RAG pipeline, presented in Figure 28, demonstrate a more accurate response. By feeding the correct context into the prompt, both model answers align perfectly with the correct answer. The judgment also reflects a positive assessment of the RAG pipeline variant. This example also illustrates the fine-tuned LLM tendency to hallucinate without the context provided by the RAG pipeline, making it unable to answer the question about the repository.

heading: In comparison, Assistant B directly addresses the user's question by providing a detailed and specific explanation of the "InstallerMissingDependencies" class, its purpose, and its behavior. Assistant A, while cautious, does not provide as much specific information about the class.

heading: Question:

heading: It is used to represent errors related to missing dependencies. The class defines a constructor that takes a message as an argument and writes the error message to the console. The error message is formatted with a newline charactr instead of a '<br>' tag for better display. Here's the code snippet:

heading: If you don't have that information, I'm afraid I cannot provide a definitive answer.

heading: Fine-Tuned Model + RAG (B):

heading: Question:

heading: Fine-Tuned Model + RAG (B):

heading: Question:

heading: Fine-Tuned Model + RAG (B):

heading: Question:

heading: Fine-Tuned Model (B):

heading: Question:

heading: Question:

heading: Fine-Tuned Model + RAG (B):

heading: Question:

heading: Fine-Tuned Model (B):

heading: Question:

heading: Fine-Tuned Model + RAG (B):

heading: Question:

heading: To install Spyder kernels using Anaconda (conda), you can follow these steps: Open your terminal or command prompt and create a new conda environment with the desired dependencies:

heading: Fine-Tuned Model + RAG (B):

heading: Question:

heading: Future Work
text: We leave these experiments as future work. Using LLM-as-a-judge instead of human evaluation Regarding evaluating the model’s performance, the LLM-as-a-judge approach also has its limitations. Despite the elimination of the position bias and the attempts to use GPT-4 as a judge, the evaluation is not flawless. The superior model judges the answers, but sometimes, the criteria are chosen by the model itself and do not match those of humans. Also, the correctness of the produced code is often not sufficiently verifiable for the model, as it does not have access to the necessary source code. Chunking of the code file Despite its advantages, the RAG pipeline has some limitations that must be considered. One major limitation is that the context provided to the LLM is always just a portion of the file, which means that knowledge about multiple files is not processed, and the connection between the files and the code cannot be considered. To address this, the context would need to be pre-processed better. One possible solution is to have a hierarchical structure that provides context at different levels and contains summarized knowledge. For example, a description of what a module is responsible for or how the general structure of the module could be added to each chunk of each file in the module. That additional information should further help the model gain a deeper understanding of the repository. Number of chunks retrieved For all experiments, the number of chunks was set to N = 1, but it could also be interesting to test whether the number of chunks could further improve the model’s performance. Also, the size of the chunk and the overlapping characters are possible variables for optimizing the results. Catastrophic Forgetting As mentioned in Sec. 5, the model’s performance decreased on both MBPP (Austin et al. , 2021) and HumanEval (Chen et al. , 2021) benchmarks following the fine-tuning
process. This shows that fine-tuning can change models’ abilities to perform certain tasks. Therefore, the conducted experiments do not clarify how the model enhances its capacity to handle the context, particularly source code, and grasp it deeper after the fine-tuning process