The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation
Authors: Dung Manh Nguyen
Le Hai Nam
Anh T. V. Dau
Anh Minh Nguyen
Khanh Nghiem
Jin L.C. Guo
Nghi D. Q. Bui
Venue: EMNLP 2023 Findings (2023)
Keywords: deep-learning
ai4code
dataset
benchmark
code-understanding
code-generation
TL;DR: A new dataset for advancing code understanding and code intelligence

Abstract: We present The Vault, an open-source dataset of high quality code-text pairs in multiple programming languages for training large language models to understand and generate code. We propose methods for thoroughly extracting samples that use both rules and deep learning to ensure that they contain high-quality pairs of code and text, resulting in a dataset of 43 million high-quality code-text pairs. We thoroughly evaluated this dataset and discovered that when used to train common code language models (such as CodeT5, CodeBERT, and CodeGen), it outperforms the same models train on other datasets such as CodeSearchNet. These evaluations included common coding tasks such as code generation, code summarization, and code search. The Vault can be used by researchers and practitioners to train a wide range of big language models that understand code. Alternatively, researchers can use our data cleaning methods and scripts to improve their own datasets. We anticipate that using The Vault to train large language models will improve their ability to understand and generate code, propelling AI research and software development forward. We are releasing our source code and a framework to make it easier for others to replicate our results.

The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation
Dung Nguyen Manh", Nam Le Hail Anh T. V. Dau
Anh Minh Nguyen', Khanh Nghiem', .Jin Guo 4,5, Nghi D. Q. Bui
FPT Software Al Center {dungnm31, namlh35, anhdtv?, minhna4, @hanhnv22)@fpt... com
Fulbright University, Viet Nam
www.buuiiiuuubbiitt.c..c.. edu.
Hanoi University of Science and Technology, Viet Nam
School of Computer Science, McGill University, Canada Mila Quebec AI Institute
Abstract
these expansive datasets for training. However, it is important to note that these datasets, including The Stack [Kocetkov et al., 2022] and The Pile [Gao et al., 2020a], often comprise unprocessed data. Alternatively, there are established datasets, such as CONCODE [Tyer et al., 201 FunCom [LeClair et al., 2019], Deepcom [Hu et al., 2020] for code summarization tasks; APPS [Hendrycks et al., 2021J for text-to-code generation: and Code- SearchNet [Husain et al., 2019] for code search. These datasets contain carefully curated code-text pairs. Although considerably smaller in compari- son raw code datasets (e.g.. 2.3M functions in CodeSearchNet [Husain al., 2019] versus 197M files in The Stack [Kocetkov al., 2022]), they provide high-quality code-text pairings that signifi- cantly enhance the effectiveness of model training. Consequently, we identify two main types of datasets used to train CodeLLMs: large yet un- processed, and smaller yet well-structured (e.g., arranged into code-text pairs). The scaling law [Kaplan et al., 2020, Gordon al., 2021, Sorscher et al., 2022] indicates that the volume of train- ing data crucial for model performance. How- ever, other studies underscore the importance of dataset quality over quantity in training superior LLMs [Zhou et al., 2023, Sorscher et al.. 2022, Dau et al., 2022, Brown et al., 2020, Khan et al., 2020]. Given these observations, we propose that an ideal dataset for training CodeLLMs should combine both elements: it should be expansive in volume and meticulously processed ensure quality. In this paper, we present The Vault dataset, de- tailing its creation process, the toolkit developed
We present The Vault, dataset of high-quality code-text pairs in multiple programming lan- guages for training large language models to understand and generate code. We present methods for thoroughly extracting samples that both rule-based and deep learning- based methods to ensure that they contain high- quality pairs of code and text, resulting dataset of 43 million high-quality code-text pairs. Our extensive evaluations on common coding tasks including code generation, code search and code summarization show that when fine-tuning Code Large Language Models on The Vault, such models outperform the same models trained on other datasets such as Code- SearchNet. We also provide detailed analyses of our datasets assess the effects of various programming languages and docstrings on the performance such models.
1 Introduction
The advent of deep learning and advancements in large language models (LLMs) have spurred rev- olution in the field of code representation learning. These developments, supported by the growing ac- cessibility of vast open-source code repositories, have heralded the emergence code large lan- guage models (CodeLLMs) for code generation and understanding tasks. The sheer volume of these repositories and the rich, unprocessed raw data they contain, serve as unparalleled resources for training LLMs. Consequently, current state-of- the-art models for coding tasks effectively utilize
Equal contribution

---

for constructing and quality-controlligg code-text pairs from raw souree code, as well an analysis of The Vault's metrics. We also share empirical results obtained from utilizing The Vault to fine- tune well-known foundational models. Our specific contributions include the following:
et al., 2020, Bui et al., 2019]. A significant por- tion of recent research employs language models. originally developed for natural language process- ing, for handling code [Feng et al., 2020a, Wang et al., 2023, Guo et al., Ahmad et al., 2021b, Bui et al., 2021b, Elnaggar et al., 2021, Peng et al., 2021, Kanade et al., 2020, Chakraborty et al., 2022. Ahmed and Devanbu, 2022, Niu et al., 2022]. Such approaches largely regard code analogous to text and adapt pretraining strategies that mirror those used for natural languages. CodeBERT [Feng al., 2020a], for instance, modifies a Roberta model [Liu et al., 2019] to pretrain a code model on multiple programming languages. CodeT5 [Wang et al.. 2021] and CodeTT+ [Wang et al., 2023] employs unique identifier information from source code to pretrain the T5 model [Raffel et al., 2019] for code in a multi-modal fashion.
A dataset with approximately 43M pairs of high- quality code-text pairs (over 10 times larger than CoDesc), 243M unimodal samples, and 69M pairs of line comments with context from 10 pop- ular programming languages (Java, JavaScript, Python, Ruby, Rust, Golang, C#, C++, C, PHP), more diverse than NodeSearchNet, which has six programming languages.
A novel approach to use a pre-trainet language model for detecting and removing noisy samples to complement traditional rule-based methods.
Datasets for Code Representation Learning: Code is commonly represented in training datasets for foundational LLMs, including the ROOTS cor- pus [Laurençon et al., 2023] for training BLOOM [Scao et al., 2022] and The Pile [Gao et al., 2020a] for training LLaMA [Touvron et al., 2023]. The code data represented in these datasets are unla- beled raw source code from GitHub. There also a family of code-only datasets for training or fine- tuning coding-specifus L.Ms, including The Stack [oocekov et al., 20221, 3TB corpus of permis- sively licensed source code, preceded by CodePar- rot with 50GB of deduplicated source code [Tun- stall et al., 2022]. These massive datasets are ally used train CodeLLMs. However, labeled data are required for training and evaluating LLMs for coding tasks involving source code and natural language descriptions. CodeXGLUE a bench- mark dataset Lu et al. [2021] for 10 coding tasks that include 14 subsets, four of which are code-text pairs. Most of the code-text pairs in CodeXGLUE come from CodeSearchNNt.
A thorough process for transforming code into code-text pairs and filtering noisy sam- ples. We have released the toolkit used in this pro- cess to the open community via a public GitHub repository', including tools for parsing code and docstrings different programming languages.
source
We perform extensive evaluation where we fine- tuned different CodeLLMs with The Vault com- pared to other datasets, such CodeSearch- Net on various code understanding tasks, includ- ing code generation, code summarization and code search. The results show that models finc- tuned on The Vault outperform those fine-tuned on CodeSearchNet (code summarization, code search) and outperform the original model by significant margin (code generation on pass @k over HumanEval and MBPP datasets).
2
Related works
Code Large Language Models for Understand- ing and Generation Code large language models facilitate various code understanding and code gen- eration tasks, including but not limited to code gen- eration [Feng et al., 2020a, Wang et al., 2023, El- naggar et al., 2021, To et al., Luo et al., 2023, Shen al., 2023], code completion [Feng et al., 2020a, Wang et al., 2023, Peng et al., 2021]. program re- pair [Xia et al., 2022], program classification [Bui et al., 2021a,c,b] and code translation [Roziere
CodeSearchNet (CSN) has also been employed for pretraining LLMs, enabling supervised learning techniques to achieve state-of-the-art performance for models such as CodeT5+ [Wang et al., 2023] and UniXcoder [Guo et al., 2022]. A few code- text pair datasets set out to surpass CSN in size. CoDesc combines existing parallel datasets (CSN, DeepCom [Hu al., 2020], CONCODE [yyer al., 201 2018a], and FunCom [LeClair et al., 2019]), and then refines the results from the superset, which yielded 4.2M Java data samples. PyMT5 [Clement
'https://g' TheVault

---

al., 2020] is dataset with 7.7M Python code- text. However, both of these datasets each contains code for single programming language. Notable datasets created from Stack Overflow include the necessary code-text data for generating post titles [Gao et al., 2020b, Liu et al., 2022].
paired with any docstring or comments, thereby forming a unimodal dataset.
3.
The third and final dataset, Dolock includes pairs of arbitrary code blocks (node 4) and inline com- ments (node 5). To construct this set, we capture all inline comments. Each comment is paired with the preceding code block, tagged as the "previous context" (node 4a), and the following code block, "next context" (node 4b).
3 The Vault dataset
3.1 Overview
In The Vault. we leverage a subset of The Stack [Kocetkov et al., 2022], recognized as the most expansive publicly available, multilingual, permissive-licensed source code dataset weighing in 3TB. From this large-scale dataset, The Vault transforms raw source code into a collection of high quality pairs of code and text. Our transfor- mation pipeline is designed to efficiently extract data from source code, create text-code pairings, and remove noise, yielding three distinct output datasets, as detailed in Figure 2. We draw from subset of The Stack, which comprises code in 10 prevalent programming languages, such C, C#, C++, Java, JavaScript, GoLang, PHP, Python, Ruby, and Rust (out of the total 300 languages fea- tured in The Stack). Each language-specific raw source code feeds into a custom-built tree-sitter" parser.
A large number of block comments adhere widely accepted doestring formats (Appendix A.5), encompassing neatly organized details about the name (identifier) of the associated function or class, their parameters, arguments, and return types. We channel these block comments through docstring parsers, which we have developed and made pub- licly available, to extract this information meta- data for each sample in our dataset. We contend that this metadata could prove beneficial for down- stream tasks, prompt settings, and other applica- tions (Figure 8). Collectively, these three datasets (Dolock- Dunimodal- and paired) constitute The Vault. Note that through the evaluation process, only D paired is used since its contains data that is suitable for training and comparison with other datasets.
This parser is designed to extract functions, classes, methods, block code snippets, and their corresponding block or inline comments. The fig- ure 1 illustrated basic structure of a code file that contains multiple levels of code snippets. By ap- plying a breadth-first search on the Abstract Syntax Tree (AST) of the root node, the parser is able to traverse down different node and leaf levels (class, function, and inline), result three separate datasets:
3.2 Data Cleaning Pipeline
From preliminary survey of the output dataset con- taining pairs of classes and functions with their corresponding block comments D Dpaireds we ob- serve salient patterns that would impair the training quality for code related tasks. We implemented a set of rule-based filters (Section 3.2.1) to remove irrelevant information or reformat textual data to be more descriptive of the corresponding code block. To address cases where the code-text pairs have inadequate erroneous semantic correlation, we trained neural-based model based on CodeBERT (Section 3.2.2) to serve as a filter. Such a filter gen- erates a score, which is used to assess the aligument of a pair of code and text. Low-scoring samples are assumed to be unaligned and will be removed.
1. The first output dataset, referred to as Dpaired contains pairs of classes (node 1) and functions (node 3) with corresponding block comments that serve docstrings (node 2). After the ini- tial construction, this dataset proceeds through pipeline that employs both rule-based filters and neural-basee filters to remove noisy samples that fail to meet the criteria detailed in Section 3.2.
3.2.1 Remove Noisy Sample by Rules
Our data pipeline employs 13 rule-based filters to eliminate noisy patterns in the source dataset. These filters, detailed in Table 1, are categorized into three main groups: enhancing readability, pro- moting consistency, and preserving the intended usage of the code.
2. The second output datasct, denoted as Dunimodal> consists of standalone functions and classes, not
"nttss://staccoverfiioo com/ "nttps://tree-sieee thub. /o/ tree-sitter/

---

// Java progren for implerentation of QuickSort
Class Node
QuickSort
This
elerent pivot,
places the pivot elesent its correct position in sorted array, and places all snallen (smaller than pivet) to left pivot and greator elerents right pivot "i.t..........
Block comment Mode
Node 2. function
this
Function Node
partstion(int arr(), low, high)
int pivot - arr[high]; int i (low-1;; // index of snaller elesent
j=low;
J+e)
I/ current clesent snallar equal piont
if (arr[3] <a pivet)
are and arr(j]
int teep a arr[i];
arr[1] arr[3];
Node and are provious and next context respectively
arr[3] teeps
[77 swap arr[aes] and are [high] (or pivot)
Line comment Node
int temp arr[
arr[i=1] arr[high];
are[high] temp:
Figure I: The tree-sitter node structure. Classes (1) and functions (3) are extracted along with their corresponding docstring, which may be the form a block comment (2) or a line comment (5). The line comments (5) are extracted along with their preceding (4a) and succeeding (4b) code nodes for the inline dataset.
In terms of readability, we strip delimiters, math formulas, HTML tags, and metadata tags from the text. This ensures cleaner and more coherent code-text pairing. For consistency, remove ele- ments that may cause irregularities in the dataset. This includes stripping hyperlinks and embedded code, and removing empty comments, overly short long comments, non-English comments, auto- comments.
Categories
Percentage (%)
Readability
Strip Delimiters
13.430
Strip Formulas
0.021
Strip HTML Tags
3,180
Strip Metadota Tags
5,260
Consistency
Strip Hyperlink
0,510
Strip Rimbedded
12.680
Remove Empty Comments
71.470
generated blocks, and Lastly, to preserve the original purpose of the code, we remove comments that are questions or serve as examples or notes. This rigorous filtering process guarantees high-quality dataset, improving the effectiveness of unde-focused language models.
Remove Comments Too Short Long
4.100
Remove Nou-Eng Non-English Comments
3,230
Remove Auto-gen Blocks
0.050
Romove Wock-in-Progrees Comments
0.002
Intended usage
Remove Comments as Questions
0.020
Remove Comments as Examples Notes
0.460
3.2.2 Remove Low-Quality Samples with Neural-based Classifier
Table 1: The percentage of constructed code-text pairs from The Stack caught by each rule-based filter.
Beyond the use of aule-based filtering methods, crucial question arises: how do we ensure align- ment between code and text? Random comments unrelated to the functionality of the code snippet can contaminate the dataset, necessitating the re- moval of such misaligned samples to guarantee quality. To address this issue, we constructed clas- sifier utilizing CoT [Feng et al., 2020b], de-
signed to score the semantic relationship between function or class and its corresponding docstring. In our scoring model, we input code snippets and docstrings separated by a token < /s >. Approx- imately 12% of the already file-filered code-tex. text pairs dataset was randomly selected for training.

---

Functions & classes w/
Code blocks
Functions
Classes
Classffer
D pusined
Functions caassss
D smmimodal
docstring metadata
Functions & classes comment
Raw files
Tree-sitter Parsers
D Book
Code w/ comment
Figure 2: Pipeline to create datasets of code blocks with comments Ditocks unimodal code D Duiimodat, and code-text pairs Dpsired from raw source code.
Number of functions
#Tokens
Language
#Repositories
#Unique code token
#Unique docstring token
#Unique identifier
w/docstring
All
Python
7,825,291
39,221,539
628,069
22,050,020
1,633,062
3,423,694
PHP
4,696,756
30,323,578
439,514
11,203,390 3333
715,546
1,133,437
JavaScript
1,683,568
33,015,657
355,761
4,895,923
501,750
753,399
Java
6,667,422
69,744,181
321,129
16.536,979
1,749,151
2,525,492
C#
3,350.316
35,736,746
150,657
5,485,063
409,220
1,233,383
C++
1,709,448
28,684.400
116,897
5,630,067
678,063
1,155,241
C
1,685,96
13,762,988
88,556
5,764,837
150,146
1,197.164
5,153,436
23,832,763
241,238
6,818,885
2,472,000
1,918,773
Rust
864,987
8,230,575
68,615
2,130,327
221,877
315,331
Ruby
461,585
4,342,191
61,804
1,436,713
146,237
213,005
Total 34,098,775
286,894,618
2,364,144
73,077,761
7,351,960
12,869,338
Table
The size of extracted function data each programming language.
As labeled data was unavailable, we generated neg- ative samples by randomly pairing functions and docstrings within the programming language. We then passed the representation of the > token linear layer, which produced a semantic correlation score between 0.0 and 1.0. Code-text pairs were then filtered using a binary classification gate with a threshold of 0.5.
der the Curve (AUC) metric proved suitable due to primary goal of excluding misalignments while preserving matched examples. An average AUC of 0.89 indicates that our approach effectively reduced dataset noise without discarding numerous informative samples. Detailed configurations and eva avaluation results are available in Appendix A.2.
In addition, amples in the rule-based noise-remove version of CodeScarchNet in CodeXGlue. Table 3 presents some inconsistent examples found by our model for Python, Java, JavaScript, and PHP in CSN. can be observed that detected pairs show strong incon- sistency between doestring and code. For instance, the docstring of the example in Python does not give much insight into what the code does its pur- pose. The code defines a method named 'has 'has_url' urt' which checks if the attributes have non-empty value; however, the docstring mentions templates which does not provide enough context to fully understand how this code relates to templates or
model to find noisy
To validate our model, we employed GPT 3.5 for analogous predictions. A million predictions were generated from unseen instances, from which we selected 300 per language: 200 high-confidence instances (100 consistent and 100 inconsistent code-text predictions) and 100 low-confidence in- stances. GPT 3.5-turbo was instructed to assign consistency score (1-10) for each instance's is code- docstring pair, serving a benchmark for our model's predictions. For high-confidence instances, our model agreed with the GPT 3.5-turbo scores over 80% the time. Although our model faced challenges with ambiguous samples, the Area Un-

---

Languages
Inconsistent pairs
// Handy For templates.
has (se
if self, self.isbn_uu iiss_nsss oz self. self.isbn_us self.sfficial_url self.noces_uril
Python
return True
else: return False
// only change appenders
public MapContent Type TytMap inontein containerType).
Java
Javerslype keylype = get.averslyype (Integer class); : JaversIype valueType get.aavessyppe return new MapContensType IkeyType, valueType);
// we do not need Buffer pollyfill For now (ontiio(sstr))
JavaScript
var ret new ( (str. len str. lengtty wh ret [Ten] str.cha chareo (1en); return OintBArray. From(ret);
disini mo atur akan apa mo kamana private runction parss_rruteee
{
Suriminplode"1 Sthis-buri->segmeen
PHP
it (isset (Stais prouter| S $urill
return Sthis->_set, reequsettterrttttteeeeeeee (explode /",
/", Schis->router (Sarruttttttt[i]]0 1)17
Table 3: Examples of Inconsistent pairs in CodeSearchNet found by model in Python, Java, Javascript and PHP. "/f" represents for docstring section. More examples are demonstrated Table 15 in Appendix section.
Dataset
4PL
w/ docstring
AFuncilon
w/o docstring
Language
Smail
Trinnin sct
Medium
Full
Valid set
Test set
PyMTS [Clement et 2020)
7,700000
Python
370.657
1,952,110
7.772.647
30,992
21.652
ColDes [Hasan 2021]
1,556
Java
351,213
1.622,666
6,629 193
22,677
15.552
CodeSearchiNel [Hussin al., 2019]
2.326.976
4,125.470
JavaSeript
82,931
404,729
1.640.416
22,044
21,
ColeXGLUE CSN [Lu et al. 2021]
1.005.474
PHP
236,638
1,155,476
4.656.371
21.375
Deepeom (Ha all, 2020]
424,025
C
105,978
381,207
1.639.119
27,525
19.122
CONCODE [lyer
20180]
2,184,310
C+
141,090
783,166
3,305,891
24.787
Funcom [LeClair et al., 2019]
2,149.121
87,420
410,907
1,671,268
20,011
CodeTS (Wang al., 2011] 2021]
3,158,313
1.199.321 151,151,985
Go
267,535
1.319,547
5.109.020
19,102
25,314
THEVAULE
10
34,0987778
Ruby
23,921
112,574
424,339
17,338
Rust
35,367
224,015
825.130
16,716
23,141
Table 4: Comparison of THE VAULT function other code-text datasets.
Total
1,702,500
8,356,097
33,673.994
222,507
202.614
Table 5: The proportion of training, validation, and test set of THEVAULT.
its broader purpose. Besides, our model able to identify oon-Esglish samples, which are presented in the example PHP, that are not captured by the rule-based methods.
datasets and commonly employed metrics, such MRR, smoothed BLEU [Lin and Och, 2004], and pass @k [Chen et al., 2021].
4 Empirical Evaluation
4.1 Dataset Statistics
In this section, we aim to assess the quality of The Vault in comparison with other datasets, such CSN. To substantiate this quality, we fine-tune prominent CodeLLMs tasks that necessitate the involvement of both code and text, including code summarization, code search, and code generation. We then compare these models, which have been fine-tuned on The Vault, with those fine-tuned on CSN. The comparison made using the same test
Table 2 provides the statistics of the samples for each programming language after undergoing our data-cleaning pipeline. In total, we have approxi- mately 34M samples. The table also includes other information, like the number of tokens for code and docstrings, and the quantity of repositories.
Table 4 offers a comparison between The Vault and other parallel datasets frequently used for pre- training and fine-tume tee-tuning downstream tasks. These

---

Python
Java
JavaScript
Go
PHP
Ruby
Total/Avg
Model
Dataset
CODESEARCHNET TESTSET (BLEU-4)
raw/TheStack
16.18
9.06
6.23
19.05
7.07
5.78
11.84/10.56
CodeT5
CodeSearchNet
19.35
20.38
16.15
19.83
26.26
15.38
21.24/19.59
The Vault/small
18.94
17.72
13.96
19.92
20.43
15.22
18.83/17.70
raw/TheStack
0.86
3.06
0.59
10.91
2.29
0.47
3.23/3
PLBART
CodeScarchNet
17.99
17.38
14.84
17.98
22.54
14.08
18.78/17.47
The Vault/small
14.93
15.66
11.95
17.03
18.00
11.49
15.95/14.84
THE VAULT
TESTSET (BLEU-4)
raw/TheStack
16. 18
9.06
6.23
19.05
7.07
5.78
11.84/10.56
CodeTS
NodeSearchNet
10.86
8.00
8.42
17.87
17.85
10.26
16.11/12.21
The Vault/mmll
12.26
11.13
9.68
31.64
38.86
11.23
25.12/19.13
raw/TheStack
1.69
4.02
0.43
24.60
4.83
0.49
7.19/6.01
PLBART
CodeSearchNet
10.24
7.26
7.64
16.90
13.83
9.60
14.39/10.91
The Vaultsmalll
10.23
9.28
8.95
22.78
34.32
9.74
20.29/15.88
Table 6: Smoothed BLEU-4 results for code summarization. The "Total" column demonstrates combined data in all languages to calculate BLEU, while "Avg" is the average BLEU score on the language level.
datasets include Funcom [LeClair and McMillan, 2019], Deepcom [Hu et al., 2020], CONCODE [yyer et al., 2018b], CSN [Husain et al., 2019], CoDesc [Hasan et al., 20211, and non-public data used for pretraining [Clement et al., 2020, Ciurume- lea et al., 2020, Wang et al., 2021].
experiment data be found in Table Note that The Vault/small has comparable size with CSN, making it fair to assess and compare the quality these two datasets.
Besides, in order to validate the efficiency of our processing pipeline, we conduct a compari- son between the performance of models trained on The Stack (raw data) and The Vault (pro- cessed data). Specifically, function-level subsets, each approximately the size of The Vault/s/mlll 1= code-text instances). These subsets were created by randomly sampling the raw function-level dataset extracted from The Stack, without applying any filtering (referred to as raw/TheStack). We use three different seeds to sample raw/TheStack and report the average result. All experiments conducted using 4 NVIDIA A100 GPUs.
We split the training set into two smaller subsets: the small set and the medium set that contain 5% and 20% of the full training set, respectively. To re- duce data leakage during training, we employed the MinHash LSH technique [Zhu et al., 2023] to filter training instance clusters that are close to samples in the validation and test sets of CSN, HumanEval, and MBPP. Additionally, during dataset partition- ing, we prevented content from the same repository from appearing in multiple sets, thereby avoiding any potential internal data leakage. A more detailed analysis of The Vault the class and code block levels can be found in Appendix A.4.
established three
4.2 Experiment Setup
Code search: We select CodeBERT [Feng et al., 2020a]. RoBERTa [Liu et al., 2019] and UniX- Coder [Guo et al., 2022] as the encoder for embed- ding source code and natural language query. We train each model for 10 epochs with sequence max length of 512, and a learning rate of 2
Data splitting: During the experiment phase, The Vault (Dpoired) was split into three distinct datasets: training, validating, and testing sets. To avoid data leakage, we reinforced a policy where code sam- ples from the same repository must all be in the same set. In the splitting algorithm, we also in- cluded as goal the preservation of the token length distribution from The Vault's dataset in each subset, For richer comparisons, the training was fur- ther branched off two smaller sets, the small and medium training sets, sampling 5% and 20% the full training set, respectively. Details about
Code summarization: CodeT5 [Wang et al., 2021J and PLBART [Ahmad al., 2021a] em- ployed for the summarization task. We use the base versions and set the max input tokens to 512 and the max output tokens to 400. We train for epochs with batch size of 512 and a learning rate of

---

Python
Java
JavaScript
Go
PHP
Ruby
Avg
Model
Fine-tune data
CODESEARCHNET TESTSET (MRR)
raw/TheStack
0.3713
0.3492
0.3148
0.5519
0.2731
0.2748
0.3559
CodeBERT
CodeScarchNet
0,3793
0.4636
0.4437
0.6201
0.4741
0.5219
0,4838
Vault/mmmmlll
0.4074
0.4857
0.4466
0.6578
0.6578
0.5251
0,5301
NodeSearchNet
0.3479
0.448
0.4254
0.5684
0.4623
0.5147
0,4611
RoBERTa
ThaVaul/small
0.4849
0.5581
0.4962
0.7446
0,5166
0.59
0,5651
CodeSearchNee
0.3935
0.4549
0.4459
0.5861
0.489
0.5446
0.4857
UniXCoder
The Vault/small
0.4427
0.4909
0.4506
0.6416
0.4515
0.5702
0.5079
THEVAULT TESTSET
(MRR)
raw/TheStack
0.318
0.3245
0.1837
0.4194
0.1718
0.0878
0.2509
CodeBERT
CodeSearchNet
0.2881
0.3213
0.2409
0.4123
0. 1854
0.2579
0,2843
The Vault/mamll
0.3501
0.4214
0.3216
0.4864
0.2351
0.2904
0.3165
CodeScarchNet
0.2644
0.3329
0.2371
0.2375
0.1577
0.2574
0.2478
RoBERTa
The
0.4533
0.5519
0.4386
0.5021
0,2876
0.3717
0.4342
CodeSearchNet
0.2959
0.344
0.2508
0.185
0. 0.1646
0.2669
0,2512
UniXCoder
Vaul//m/mmlll
0.3852
0.4279
0.3491
0.4628
0.238
0.3201
0.3639
Table 7: Comparison between the models fine-tuned on the CODESEARCHNET and on different THE VAULT training subsets on code search task.
We use CodeGen 350M and 2B Multi INijkamp al., 2023] to evaluate code generation. We the same configuration in the code summarization task.
Code generation:
Appendix, Table 14. The results show that our pipeline has witnessed strong effectiveness com- pared to unprocessed data, raw/TheStack. Particu- larly, during training on the raw/TheStack dataset for the code summarization task, we found that the PLBART and CodeT5 generate outputs with substantial noise. These outputs are characterized by prevalence of special tokens like "//" and This finding strongly underscores the efficacy of our filtering process in enhancing the quality of the dataset. However, the result using CSN shows superior performance CSN's testset than using The Vault. The reason for this is our mention of the poot-procsssing step to reduce the difference between the CSN and The Vault filtering methods, where the syntactic distribution can still exhibit nonidentical characteristics, which can affect the BLEU score. However, this gap could be reduced by using the full version of The Vault shown in Table 14. Although the total performance gain when evaluated on the CSN test is marginal (21.73 versus 21 2224, is worth noting that, despite the intermediary processing, CSN a considerably smaller dataset with more consistent doestring pat- terns. In contrast, our dataset is substantially larger and exhibits greater diversity, thereby encouraging broader generalization. When evaluated against The Vault's test set, the model fine-tuned on CSN lags behind by over 10%.
4.3 Evaluation Results
4.3.1
Code Summarization
For this task, utilize the Vault and CSN to fine-tune CodeT5 and PLBART summarize the source code. The Vault and CSN exhibit significant differences in docstring format. The Vault retains the complete docstring format, offering compre- hensive descriptions of core logic, parameters, ar- guments, and return types. This feature enables versatile applications in code documentation and various downstream tasks. Additionally, the first sentence of each complete docstring as metadata, termed as short dooetrdocstingg. To facili- tate fair comparison between The Vault and CSN, we apply post-processing to our full docstrings and short_docstring training sets, thereby reducing format distribution disparity.
Table shows the results when comparing CodeT5 and PLBART trained CSN and The Vault for the code summarization task, re- port the best score when using full docstrings and short docstrings. We present further exper- imental outcomes using the Rouge-L [Lin, 2004] and ScoT [Zhang et al., 2020] metrics in

---

Model
Fine-tune delaset
passee
pass@:10 10
pass 100
The Vault to match the size of the Python subset in CSN with 250K samples in the first round of fine- tuning. Additionally, raw/PyTheStack is a subset of Python data from The Stack mirroring the size of Python data present in The Vault dataset, which helps us to demonstrate the advancements achieved 41.72
HUMANFVAI
6.67
10.51
16.84
PyCCadeSearhhNa
8.76
14.72
CodeGaa 350M
(250K) Py/Th
3.74
10.57
16.26
Vault
8.14
18.12
30,07
CodeGen 2B
Py/The Viult
14,00
25.74
in our data process pipeline.
ColeGen 350M
7.46
24.18
46.37
Vault
10.13
33,96
53.20
The results are shown in Table 8. We can see that fine-tuning the CodeGen Multi 350M on The Vault causes the model improve significantly in terms of pass@ pass @ 10, and pass 100 on the HvmanEval and MBPP benchmarks. Addition- ally, CodeGen 2B is used to assess The Vault on larger scale models. Similar to experiments on small models, Table 8 shows that The Vault can improve the performance of pretrained large-scale models. These results validate The Vault's ability to improve the performance of pre-existing pre- trained models. In the future, we will expand our evaluation to even larger scale models and assess The Vault's s impact on them.
CodeGia 2R
18.06
25.80
65.34
Py/TheVault
27.82
50.06
65.06
Table 8: Result on code generation benchmarks using CodeGen Multi 350M and 2B models.
4.3.2 Code Search
We utilize CodeBERT, RoBERTa and UniXCoder to fine-tune both The Vault and CSN for the pur- pose of the code search task. We also furnish a base- line Mean Reciprocal Rank (MRR) score. MRR is widely used metric for evaluating code search tasks, and in our case, it is trained on 10 different programming languages and assessed using the test set from CSN and The Vault. The results of this task, when fine-tuning the model on The Vault and CSN, are illustrated in Table 7. Remarkably, we attain superior results in most languages when fine- tund using the smallest dataset, The Vault/m/mall in contrast to solely fine-tuning on the CSN cor- pus. Surprisingly, RoBERTa, model pretrained natural language text, outperforms the two code- pretrained models when evaluated on code search. This could imply the importance of natural lan- guage text representation over code representation in this task. Furthermore, models trained on The Vault consistently outperform all baseline models trained on raw/TheStack, underscoring both the ef- ficiency of our processing pipeline and the dataset's s ability to generalize across different architectures.
Conclusion
In this paper, we presented The Vault, a large dataset of high-quality code-text pairs from ten programming languages, with over 43 million sam- ples. The Vault was carefully curated to ensure that each pair meets quality standards, with detailed and informative descriptions and consistent cod- ing styles. Our analysis uncovered a number of intriguing patterns and trends that shed light on the characteristics of programming languages and coding practices. We believe that The Vault will be valuable resource for rescarchers and practi- tioners in this rapidly evolving field, providing a solid foundation for developing novel approaches and advancing state-of-the- code large language models.
4.4 Code Generation
We experiment with two versions of CodeGen Multi [Nijkamp et al., 2023], which are 350M and 2B models on the HumanEval and MBPP bench- marks for code generation. The scope of our ex- periment was limited because the benchmarks only support Python. We use these checkpoints and continue fine-tuning them on The Vault because CodeGen Multi models are trained on the dataset with multiple languages.
To create PyC and Py/The Vault, we use the Python subsets of CSN and The respectively. We sampled the training Python set of

---

Limitations
and Y. Zhou, editors, Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. NAACL-HLT 2021, Online, June 6-11, 2021. pages 2655-2668. Association for Com- putational Linguistics, 202
In our approach, we employed 13 heuristic and context-specific rule-based filters, curated from manual data observations. While these filters effec- tively mitigated noisy patterns, their deterministic nature precluded comprehensive generalizability. To address this, we supplemented these rules with a neural-based approach as described in Section 3.2.2. However, the absence of labeled training data necessitated pseudo-random sample genera- tion, which could compromise model soundness and potentially eliminate quality code-text pairs. Although cross-validation with GPT 3.5-turbo oc- casionally revealed scoring inconsistencies, we be- lieve that human labeling and model fine-tuning could further refine the dataset.
T. Ahmed and P. Devanbu. Multilingual training for software engineering. In Proceedings of the 44th International Conference on Software Engineering. pages 1443-1455, 2022.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan,
P. Dhariwal, A. Ncelakantan, Shyam, G. Sastry,
A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger,
T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, Wu, C. Winter. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Compared to The Stack and The Pile, our dataset is smaller, mainly due to our rigorous quality con- trol procedures. Moreover, creating AST parsers for each programming language is non-trivial task, limiting our dataset to 10 popular program- ming languages compared to The Stack's 300. Nonetheless, our framework's 's codebase is publicly available, encouraging future contributions to ex- tend our parsers and rules to additional languages,
N. D. Bui, Y. Yu, and L. Jiang. Sar: learning cross- language api mappings with little knowledge. Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Sym- posium on the Foundations of Software Engineering, pages 196-806,77 2019.
The current study primarily utilized small mod- els with less than 2 billion parameters illustrate the value of The Vault. These models effectively demonstrated the dataset's potential, but further research with larger models would shed light on its robustness and scalability across more complex tasks. In future work, we plan to conduct experi- ments using large-scale language models to further assess the impact of our dataset.
N. D. Bui, Y. Yu, and L. Jiang. Infercode: Self- supervised learning of code representations by pre- dicting suburees. In 2021 IEEE/ACM 43rd Interna- tional Conference on Software Engineering (ICSE). pages 1186-1197 IEEE, 2021a.
N. D. Bui, Y. Yu, and L. Jiang. Self-supervised con- trastive learning for code retrieval and summarization via semantic-preserving transformations. In Proceed- ings of the 44th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, pages 511-521, 2021b.
References
W. Ahmad, Chakraborty, Ray, and
N. D. Bui, Y. Yu, and L. Jiang. Treecaps: Tree-based capsule networks for source code processing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 30-38, 2021c.
K. Chang. Unified pre-training for program un- derstanding and generation. In K. Toutanova, A. Rumshisky. L. Zettlemoyer, D. Hakkani- Tür,
I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Con- ference of the North American Chupter of the Asxo- ciation for Computational Linguistics: Human Lan- guage Technologies. NAACL-HLT 2021, Online, June 6-, 2021. pages 2655-2668. Association for Com- putational Linguistics, 20
S. Chakraborty, T. Ahmed, Y. Ding, P. T. Devanbu, and B. Ray. Natgen: generative pre-training by "natu- ralizing" source code. Proceedings of the 30th ACM Joint European Software Engineering Confer- ence and Symposium on the Foundations of Software Engineering, pages 18 18-30, 2022.
W. U. Ahmad, K. Chang. Unified Pre-training for Program Un- derstanding and Generation. In K. Toutanova, A. Rumshisky. L. Zettlemoyer, D. Hakkani-Tür, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty,
Chakraborty,
Ray, and
M. Chen, J. Tworek, H. Jun. Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint wrXiv:2107.03374, 2021.

---

Ciurumelea, Proksch, and H. C. Gall. Suggesting comment completions for python using neural lan- guage models. In 2020 IEFE 27th International Con- ference on Software Analysis, Evolution and Reengi- neering (SANER). pages 456-467, 2020.
EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 5915- 5922. Association for Computational Linguistics, 2021.
D. Guo, Ren. S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou. N. Duan, A. Svyatkovskiy, S. Fu, M. Tufano, S. K. Deng, C. B. Clement, D. Drain, N. Sundaresan, J. Yin, D. Jiang, and M. Zhou. Grapheodebert: Pre-training code representations with data flow. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
C. B. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan. Pymt5: multi-mode translation of natural language and python code with transform- ers. In B. Webber, T. Cohn, Y. He, and Y. Liu, ed- itors, Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 9052-9065. Association for Computational Linguis- tics, 2020.
D. Guo, S. Lu, N. Duan, Y. Wang, M. Zhou, and J. Yin. Unixcoder: Unified cross-modal pre- pre-training for code representation. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7212-7225. Association for Computational Linguisties, 2022.
A. T. V. Dau, N. D. Q. Bui. T. Nguyen-Dou Duce, and H. Thanh-Tung, Towards using data-influence meth- ods detect noisy samples in source code corpora. In 37th IEEE/ACM International Conference on Au- tomated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022, pages 148:1-144:3. ACM, 2022.
M. Hasan, T. Muttaqueen, A. A. Ishtiaq, K. S. Mehrab. M. M. A. Haque, T. Hasan, W. U. Ahmad, A. Iqbal, and R. Shahriyar. Codesc: A large code-description parallel dataset. In C. Zong. F. Xia, W. Li, and R. Navigli, editors, Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, On- line Event, August 1-6, 2021, volume ACL/UCNLP 2021 of Findings of ACL, pages 210-218. Associa- tion for Computational Linguistics. 2021.
A. Elnaggar, W. Ding, L. Jones, T. Gibbs. T. Feher. C. Angerer, S. Severini, F. Matthes, and B. Rost. Codetrans:
Towards cracking the language of sil- icon's 's code through supf-supe deep learning and high performance computing. arXiv preprint DXXiv:104402244: 2021.
7. Feng, D. Guo, D. Tang, N. Duan, X. Feng M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou. Code- bert: A pre-trained model for programming and natu- ral languages. In T. Cohn, Y. He, and Y. Liu, editors, Findings of the Association for Computational Lin- guistics: EMNLP 2020, Online Event, 16-20 Novem- ber 2020, volume EMNLP 2020 of Findings of ACL, pages 1536-1547, Association for Computational Linguistics. 2020a.
D. Hendrycks, Basart, S. Kadavath, M. Mazeika. A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Mcasuring coding chal- lenge competence with APPS. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Infor- mation Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
X. Hu, Li, X. Xia, D. Lo, and Jin. Deep code com- ment generation with hybrid lexical and syntactical information. Empir. Softw. Eng., 25(3):2179 2217772717 2020.
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2020. pages
H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt. Codesearchnet challenge: Eval- uating the state of semantic code search. arXiv preprint arXiv: I 19990999333 2019.
Online, Nov. 2020b. Association for Computational Linguistics.
L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang. H. He, A. Thite, N. Nabeshima, al. The pile: An 800gb dataset of di- verse text for language modeling. arXiv preprint 1rXiv:2101.0002 2020a.
S. Tyer, I. Konstas, A. Cheung, and Zettlemoyer. Map- ping language to code in programmatic context. In E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsu- jii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 1643-1652. Association for Computational Linguistics, 2018a.
Z. Gao, X. Xia, J. Grundy, D. Lo, and Y. Li. Generating question titles for stack overflow from mined code snippets. ACM Trans. Softw. Eng. Methodol., 29(4): 16:1-2,:37, 2020b.
S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 1643-1652, Brussels, Belgium, Oct.-Nov. 2018b. As- sociation for Computational Linguistics.
M. A. Gordon, K. Duh, and J. Kaplan. Data and pa- rameter scaling laws for neural machine translation. In M. Moens, X. Huang, L. Specia, and S. W. Yih, editors, Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing,

---

A. Kanade, P. Maniatis. G. Balakrishnan, and K. Shi. Learning and evaluating contextual embedding of source code. In International Conference on Ma- chine Learning, pages 51 5110-5121. PMLR, 2020.
on Software Analysis, Evolution and Reengineer- ing, SANER 2022, Honolulu, HI, USA, March 18, 2022, pages 577-588. IEEE, 2022.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, I. 7ettlemoyer, and V. Stoyanov. Roberta: a robustly optimized bert pretraining ap- proach (2019). arXiv preprint orXiv:1907. 1699, 364, 1907.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:200..8361. 2020.
Y. Liu, M. Oit, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemooer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.0. 1692. 2019.
S. S. Khan, N. T. Niloy, M. A. Azmain, and A. Kabir. Impact of label noise and efficacy of noise filters in software defect prediction. In R. García-Castro, edi- tor, The 32nd International Conference on Software Engineering and Knowledge Engineering, SEKE 2020, KSIR Virtual Conference Center, USA, July 9-19, 2020, pages 347-352. KSI Research Inc., 2020.
S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang. D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tu- fano, M. Gong. M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. In J. Vanschoren and S. Yeung, edi- tors, Proceedings of the Neural Information Process- ing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf. et al. The stack: 3 tb of permissively licensed source code. arXiv preprint arxi::2211.5555
2022.
H. Laurençon, L. Saulnier, T. Wang, C. Akiki, A. V. del Moral, T. L. Scao, L. V. Werra, C. Mou, E. G. Ponferrada, H. Nguyen, J. Frohberg, M. Saško, Q. Lhoest, A. McMillan-Major, G. Dupont, S. Bi- derman, A. Rogers, L. B. allal, F. D. Toni, G. Pistilli, O. Nguyen. S. Nikpoor, M. Masoud, P. Colombo, J. de la Rosa, P. Villegas, T. Thrush, S. Longpre, S. Nagel, L. Weber, M. Muñoz, J. Zhu, D. V. Strien, Z. Alyafeai. K. Almubarak. M. C. Vu, I. Gonzalez- Dios, A. Soroa, K. Lo, M. Dey, P. O. Suarez, A. Gokaslan, S. Bose, D. Adelani, L. Phan, H. Tran, I. Yu, S. Pai, Chim, V. Lepereg, S. Ilic, M. Mitchell, S. A. Luccioni, and Y. Jernite. The bigscience roots corpus: A 1.6tb composite multilingual dataset, 2023.
Z. Luo, C. Xu, P. Zhao. Q. Sun, X. Geng, W. Hu, C. Tao, Ma. Q. Lin, and Jiang. Wizardeoder: Empow- cring code large language models with evol-instruct. arXiv preprint arXiv:2306.008 085688 2023.
J. Mahmud, F. Faisal, R. I. Arnoh, A. Anastasopoulos, and K. Moran. Code to comment translation: A comparative study on model effectiveness & errors, 2021.
C. Nguyen, L. Ngo, and T. Nguyen. Retrieving rele- vant context to align representations for cross-lingual event detection. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2157- 2170, 2023.
A. LeClair and C. McMillan. Recommendations for datasets for source code summarization. pages 3931 3937. Association for Computational Linguistics, 2019.
E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang. Y. Zhou, S. Savarese, and C. Xiong. Code- gen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representa- tions, 2023. URL htt.t:://///// httpp/sss///pppvvi/eeee wwwpneeeeeee....m//////// n forum? ferum?id=ian iddiaacJKKY2Y___.
LeClair, Jiang, and C. McMillan. A neural model for generating natural language summaries of pro- gram subroutines. In J. M. Atlee, T. Bultan, and J. Whittle, editors, Proceedings of the 41st Interna- tional Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pages 795-806. IEEE/ / ACM, 2019.
C. Niu, C. V. Ng. J. Ge, L. Huang. and B. Luo. Spt- code:
pre-training for learning source code representations. In Proceedings of the 44th International Conference on Software Engineer- ing. pages 2006 20062208.1111 2022.
C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches ouf, pages 74-81, 2004.
D. Peng. S. Zheng, Y. Li, G. Ke, D. He, and T.-Y. Liu. How could neural networks understand programs? In International Conference on Machine Learning, pages 8476-8486. PMLR, 2021.
C.-Y. Lin and F. J. Och. Orange: method for evaluat- ing automatic evaluation metrics for machine trans- lation. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguis- tics, pages 501-507, 2004.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang. M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. arXiv preprint arXiv: 10683, 2019.
K. Liu, G. Yang, X. Chen, and C. Yu. Sotitle: A transformer-basee post title generation approach for stack overflow. In IEEE International Conference

---

Roziere, M.-A. Lachaux. Chanussot, and G. Lam- ple. Unsupervised translation of programming lan- guages, Advances in Neural Information Processing Systems, 33:20601-20611. 2020.
Boyd-Graber and N. Okazaki, editors, Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 782-792. Association for Computational Linguistics, 2023.
T. L. Scao,
A. Fan, C. Akiki, E. Pavlick, S. Ilié, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al. Bloom: A 176b-parameter open- access multilingual language model. arXiv preprint arXiv:2211.05/00 2022.
T. 7hang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with BERT. 2020.
B. Shen, J. Zhang, T. Chen, D. Zan, B. Geng, A. Fu, M. Zeng, A. Yu, J. Ji, J. Zhao, et al. Pangu-coder2: Boosting large language models for code with rank- ing feedback. arXiv preprint arXiv:2307.14933 2023.
Z. Zhang. W. Yu, M. Yu, Z. Guo, and M. Jiang. A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods. A. Vlachos and Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023. Dubrovnik, Croatia, May 2-6, 2023, pages 443-956. Association for Computational Linguistics, 2023.
B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 19523- 19536, 2022.
C. Zhou, P. Liu, P. Xu, Iyer, J. Sun, Mao, X. Ma. A. Efrat, P. Yu, L. Yu, et al. Lima: Less is more for alignment. arXiv preprint M.Xi:2305.112200 2023.
To, N. Bui. J. Guo, and T. Nguyen. Better language models of code through self-improvement (2023). DOL: http:://ooi.
2304.
E. Zhu, V. Markovtsev, A. Astafiev. C. Ha, Eukasiewicz, A. Foster, Sinusoidal36, A. Oriekhov, J. Halliwell, JonR, K. Mann, K. Joshi,
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-.. Lachaux, Lacroix, B. Rozière, N. Goyal, E. Ham- bro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models. 2023.
M. J. Rosenthal, Q TianHuan, S. Ibraimoski. S. Thakur, S. Ortolani, Titusz, Letal, Z. Bentley, fpug, hguhlich, long2ice. oisincar, and R. Assa. ekzhu/datasketch: v1.6.4, Oct 2023.
L. Tunstall, L. Von Werra, and T. Wolf. Natural lan- guage processing with transformers. "'RReilly Me- dia, Inc.", 2022.
L. N. Van, N. L. Hai, H. Pham, and K. Than. Auxil- iary local variables for improving regulariiation/prior approach in continual learning. In Pacific-Asia Con- ference Knowledge Discovery and Data Mining, pages 16 28. Springer, 2022.
Y. Wang, W. Wang. S. R. Joty, and H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In M. Moens, X. Huang, Specia, and S. W. Yih, ed- itors, Proceedings of the 2021 Conference on Em- pirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 8696- 8708. Association for Computational Linguistics. 2021.
Y. Wang, H. Le. Gotmare, N. D. Q. Bui, Li, and S. C. H. Hoi. Codet5+: Open code large language models for code understanding and generation, 2023.
C. S. Xia, Y. Wei, and L. Zhang. Practical program repair in the era of large pre-trained language models. arXiv preprint arXiv:2210. 14179, 2022.
P. Yadav, Q. Sun, H. Ding, X. Li, D. Zhang, M. Tan, P. Bhatia, X. Ma, R. Nallapati, M. K. Ramanathan, M. Bansal, and B. Xiang. Exploring continual learn- ing for code generation models. In A. Rogers, J. L.

---

A
Appendix
template ask GPT 3.5-turbo score each pair of code-text on scale of 10 for semantic cor- respondence with a detailed explanation and ran this prompting template on systematically selected 300 data points from each language with 100 data points in each of the following groups:
A.1 Rule-based filters
While some datasets eliminate all special characters +0_++//,,--+-- --/,,- ') and keep only the first sentence or the paragraph preceding the first double endline symbol [Hasan et al., 2021, Mahmud et al., 2021J, our heuristic rules take a different approach. Instead of discarding such characters outright, we selectively remove the noisy elements while aiming to capture as many informative sections as possible.
(1@#$% (004%%&*0---00
Consistency group: Examples that the model gives high confidence prediction to class Con- sistent. We select the top 100 based on the output probability for class 1.
We analyze each docstring block individually and retain the sections that meet quality crite- ria. Table 9 provides comprehensive descriptions of our 13 rule-based filters, accompanied by illustra- tive examples. Additionally, table 10 presents the corresponding percentages of code-text pairs gen- erated through the application of these rule-based filters.
Inconsistency group: Examples that the model gives high confidence prediction to class In- consistent. We select the top 100 based on the output probability for class O.
Uncertainty group: Examples that the model gives uncertain predictions. We select the low- est top 50 examples for each class.
A.2 Neural-based refinement method
The systematic sampling scheme helped us se- lect 2994 samples in function level to be scored out of millions, reducing the cost of requesting GPT 3.5-turbo API while enabling meaningful analysis. The prompt input to GPT 3.5-turbo is follow:
To detect semantic inconsistency between code-text pairs, considered fine-tuning large founda- tional models such CodeGen [Nijkamp al., 2023], BLOOM [Scao et al., 2022] or leverage 3.5 APIs. However, these approaches would incur very high costs in terms of financial resources, time, and computational power. We de- cided to train dedicated model to deal with this specific task and use GPT 3.5-turbo to cross-check the predictions.
want you to act docstring evaluator for code. will give you a docstring along with source code, and you will give me score for the consistency between them. The score will of 10, 10 means the doestring can effectively summarize the code while means they are inconsisteat. The response answers must contain the score and the exp explanation that follows the format in the response format.
unbiased
scale
Training: We trained our model based Code- BERT, [Feng et al., 2020a]. The model assigns a score for semantic correspondence between code and text, before passing through binary classifi- cation into Consistent and Inconsistent categories. We randomly chose 5M samples (500K for each language in The Vault) and divided them into train- ing, validation, and testing sets at ratio of 3:1:1. The input to the model is the concatenation of the doestring and the code together with the < /s > Code:
Respense format: Score: X
Explanation: Y
Docstring: "fdscatring]"
token used to separate them (Figure 3). We use the representation the < s > token and feed into a linear layer to obtain the output logit.
[code)"
Empirical Evaluation Results: Table 11 presents the performance of our model with GPT 3.5 turbo's scores as reference, along with the scoring result for each group. In groups with high confidence,
Since labeled data was unavailable, we utilized self-supervised learning. We created negative sam- ples by randomly pairing a function with a doc- string from the same programming language (Fig- ure 3).
witness strong correlation be- tween our model and GPT 3.5-turbo, with a high score for Consistency (7.81) and a low score for Inconsistency (3.15). A similar pattern is observed in the Uncertainty group, where the average score is close to the middle of the scale at 5.74.
Cross-check: We used GPT 3.5-turbo to per- form similar classifications for semantic consis-
tency of code-text pairs. We used a prompting

---

Calegories
Syntax Feature
Action
Docstring
Lexical esseatially Lokenicer.
Comment Delimiter
Unnecessary delimiter
Update
Lexical essentially zokenizer.
Deletea Mux asaet
8see
Hyperlink
URL Link
Update
attps://coce mix www.co///t//eeeerrrooootttttttte
Set the trust lavel for key in GPG keychain. code-block::
salt (o' trg.truss-key key-idaf key-ida#3e449910 3PAD9F1s' Lsust-level- maaveiilaaggnsslly
Inline embedded code snippets, command lines, script excerpts
Embedded Code
Update
+ Set the trust level for Pode-block:: beah
BeB keychsin.
isup <usl> Is it down for everyone, just you?
Question
Question: Why? How?,
Update
ssup <url>
Recursive filter design using least-squazes
method.
{1)8,4(1} KULEMALKIN, finds recursive rilter coefficients 5 and A.
Math formula
\sgro. (exp0. (mathbr....
Update
Recursive filter design using a least-squares method.
Creases slive of 'array' with 'nt elements dropped
[zom the end. @static AmenbezOf_ @since 3.0.0
Metadata Tag
Metadata annotations
Update
Creates alice 'arzay' with 'nr elementa dropped from the end.
Constructs <codexGenera aenteeGeeeaallttoreeeeaeae rodeettt from plain JavaScript object.
HTML lags: <p.>... </p>.....
Tags
Update
Special tags.
constructs SeneraaltooreFFroduutoooo plain JaveSeript object.
packages
note: Uses to access package's data diz. +
Example and note
Code example, from developers
Update
Puil packages datz dir.
Unsuitable Length
Length 5, length
Msite eb jects
Retorna uma escrutura con os argumentos passades paza programa.
written English
Remove
English
-e--bagin-weer-dee- ul-end-ueer-doe->>> @generated
Auto-ger gen
Auto-generated
Remove
Deprecate
will rebuilt
dever-dev
Under development
Remove
any other test run wants to use it.
com-
No docstring/somment in function
Remove
mol1
Table 9: Rule-based filters examples.

---

Categories
Python
PHP
JavaScript
Java
Cf
C++
C
Rust
Ruby
Go
Total
Comment Delimiter
12.02
33.38
9.94
11.98
16.7
6.92
13.28
8,43
9.13
4,95
13,43
Hyperlink
0.95
0.44
0.66
0.25
0.71
0.15
0.11
0.59
1.11
0.65
0.51
Embedded Code
31.65
1.09
1.38
1,41
1.39
6.51
6.16
0.67
3,18
2,41
12.68
Question
0
0.02
0.02
0.01
0.02
0.13
0.02
Math formula
0.1
0
0.01
0.01
0.01
0.02
0.02
0.01
0
0.021
Metadata Tag
6.81
2.69
2.15
4.35
6.14
1.69
0.46
5.26
HTML Tags
0.79
0.68
0.8
2.7
17.15
0.31
0.45
1.13
1.56
0.13
3.18
Example and note
1.4
0.26
0.36
0,34
0.22
0.18
0.4
0.45
0.79
0.3
0.46
Unsuitable Length
5.11
8.79
3,90
2.20
2.75
4.58
3.86
2.26
5.19
4.37
4.10
Non-Emn Mon-English
1.69
5.72
3.26
4.16
2.62
4.1
1.94
0.42
1.53
1.77
3.23
Auto-gen
0.01
0
0
0.2
0
0
0
0.05
Under-dev
0.02
0
0
0
0
0.002
No comment
60.54
49.0
78.5
77.15
76.16
80.95
72.28
80.43
71.55
69.75
71.47
Table 10: The percentage of constructed code-text pairs from The Stack caught by each rule-based filter, by programming language.
Java
Positive example


Docstring,




Code,
Code1
Codem
Negative example
random
select


Docstring,




Codem
m of
Code
Figure 3: Input representation and Negative sample generation for code-docstring inconsistency detection.
GPT 3.5-turbo's scores to generate pseudo-labels and calculate accuracy and AUC for our model. We set relative threshold of 5 to determine the labels. It can be witnessed that our model performs well in high-confidence groups but struggles in the uncertainty group. However, the accuracy is influenced by the choice of rela- tive threshold, we consider Area Under the Curve (AUC) to measure the false positive and true pos- itive rates. The metric shows convincing result averaging 0.89, enabling to effectively reduce a high amount noise in our dataset while avoid- ing excluding too many informative examples. Fi- nally, after removing noisy data using the proposed neural-based method, we notice a decrease of 1.3% in the dataset.
In addition,
pose. The code defines method named 'has_url' which checks if the attributes have non-empty value; however, the docstring mentions templates which does not provide enough context to fully un- derstand how this code relates to templates or its broader purpose. A similar pattern also presents in the remaining examples. An example that pro- vides more clarity is the second example in Ruby. The docstring describes function with 'YAML filePath' parameter, but the function itself does not actually have this parameter. Besides, model is able to identify non- non-English samples (the sec- ond example in PHP) that are not captured by the rule-based method.
A.3 Analysis of Function-Level Data in The Vault
We use our model to find noisy examples in the rule-based noise-remove version of CodeSearch- Net in CodeXGlue. Table 15 illustrates some exam- ples found in 6 programming languages. It can be observed that detected pairs show strong inconsis- tency between docstring and code. For instance, the docstring of the first example in Python does not give much insight into what the code does or its pur-
Detailed description of function level data The Vault can be found Figure
A.3.1 Code and Docstring Analysis
Token Length Distribution: When training seq- to-seq LLMs, maximum input and output lengths are typically required. By understanding the distri- bution of sequence lengths in the corpus, we can

---

GPT t.r score (aceuracy)
Language
Accuracy (%)
AUC
Consistency
Inconsistency
Uncertainty
Python
8.19 1.15 (99%)
3.76 = 1.96 (69%)
6.20 + 2.12 (44%)
70.67
0.8559
PHP
7.73 = 1.32 (96%)
3,01 = 1.45 (90%)
4 (49%)
78.33
0.8863
JavaScript
7.73 + 1.25 (99%)
2.95 = 1.40 (89%)
5.58 + 2.29 (49%)
79.00
0.8984
Java
7.65 + 1.71 (94%)
2.73 1.32 (93%)
2.12 (53%)
80.00
0.9014
C#
7.70 +: 1.35 (97%)
3.31 = 1.56 (82%)
5.35 2.09 (46%)
75.00
0.8606
C++
7.51 = 1.64 (92%)
2.82 1.46 (89%)
5 (57%)
79.33
0.8787
C
7.79 + 1.10 (98%)
2.99 1,48 (88%)
5,81 + 2.08 (47%)
77.67
0.9108
Go
8.08 1.21 (99%)
3.68 1.67 (74%)
6.09 + 2.06 (50%)
74.83
0,8819
Rust
8.03 + 1.20 (99%)
3.72 - 1.77 (75%)
6.83 + 1.62 (50%)
74.67
0.9051
Ruby
7.72 1.03 (98%)
2.51 1.04 (96%)
5.01 + 2.23 (49%)
81.00
0,9203
All
7.81 + 1.33 (97%)
3.15 1.59 (84%)
5.74 + 2.19 (49%)
77.05
0.8874
Table 11: Evaluate CodeBERT using the consistency score provided by GPT 3.5 t.r-t We report the mean = the standard deviation for the score in each subset.
w/docstring-fiteeeed
w/docstring-noose
wo/docstring
Python
Go
C++
C++
5.3%
PHP
Ruby
10.6%
4.8%
Ruby
Rust
2.5%
11.5%
JavaScript
12.9%
Java
C#
JavaScript
PHP
Python
Java
6
Number of functions
1e7
Figure 4: Distribution and the number of functions by the presence of docstrings. Functions with docstrings are further divided into categories: functions removed by rule-based filters and functions in the final code-text dataset.
Cose_iolinss
Dosstrangtokers
choose appropriate input and output lengths for training. This can help improve the performance of training a language model and prevent the resulting LLMs from producing outcomes too short or too long for the intended use cases [Kaplan et al., 2020, Brown et al., 2020].
0-
C++
.318
Our tokenization process utilizes the tree-sitter framework to parse source code into nodes on an abstract syntax tree; each node is considered a to- ken. For docstring tokenization, we tokenize by word and punctuation. The code and docstring tokens length distribution for each programming language illustrated in Figure 5. The number of tokens present in a function (average of around 100 tokens) considerably more than the number of tokens found the doestrings (average of 15-30 tokens) that describe it. particular, among the
Jestorst
Bytee
4.01
Figure 5: Code and Docstring tokens length distribution. The plot shows the lower to upper quartile values of the number of tokens in the data. The orange solid line indicates the median and the green triangle presents the mean.

---

10 programming languages, C and C++ have the highest number of tokens in a function. This can be attributed to the fact that these languages are low-level languages, which typically require more code to perform a task when compared to higher- level languages. In the case of doestrings, their number of tokens is determined not only by the naturalness of the description in practice but also by cleaning rules outlined in Section 3.2.1. From Figure 5-Right and Table 10, it can be observed that the docstrings in Java and C are lengthy but are slightly cleaned by update-action rules, indicat- ing that the doestrings in these two languages are typically long and more detailed in practice. Mean- while, the number of tokens of doestrings in C# is the lowest. The cleaning rules may have played a role, significant proportion of the samples in C# has been updated based on Comment Delimite (16,7%) and HTML Tags (17,15%) rules.
of extracted attributes for each programming lan- guage, with most having between 1 to 5 elements. We make our docstring-style parser available to the community to facilitate easy customization and enhancement.
A.4 Analyzing for Class and Inline Comment Set
In Table 12, we provide a statistical analysis of the number of classes and inline comments in both the raw set and the filtered set. Since the class structure is not defined in C and Go, we do not have their information give in this table.
Initially, we excluded substantial number of class samples from the raw dataset that lacked doc- strings. The remaining class-docstring pairs un- derwent additional processing. Since the nature of classes and functions is similar, their function- alities can be meaningfully defined by pairs of a code snippet and docstring. However, one of the problems when constructing paired data for class- comment samples is the large code snippet length of the class structure. As a result, we set the maxi- mum number of code tokens that class can have to 5000. On average, the code-token length of the class set is approximately 500, which is around five times longer compared to the average token length in the function set, while the number of docstring- token lengths is similar between the two sets, as shown in Figure 6. Each pair of class-docstring is also examined via a rule-based filtering process, as described in Section 3, serving as a sample point in D Dpair dataset.
Table 2 depicts the overall number of distinct tokens for each programming language. As our dataset contains extensive unique tokens, we be- lieve that model training The Vault can effec- tively handle unseen tokens. Besides, we find that multiple function names are reused due to the rela- tively small number of unique identifiers compared to the total number of functions in the dataset. This finding implies that even for humans, naming func- tions might be a difficult task.
Docstring Styles: Alongside typical docstrings that provide brief deseriptions of the source code, many adhere to formatting and style conventions like Google, Jsdoe, and reST styles, among others. Our toolkit, designed to parse docstrings and ex- tract metadata into a dictionary, supports 11 preva- lent doestring styles. The styles we support and the information we aim to extract are depicted in figures 10 and 8 in Appendix A.5. This rich dataset could inspire research on advanced problems, such controlling docstring style during generation or crafting explanations for function parameters.
In the DMtek: analysis, we initiate the initial for- mation of the sub-dataset by identifying and extract- ing inline comments within code functions. The ex- tracted comments undergo a series of cleaning pro- cedures similar those applied to the docstrings (as discussed in Section 3.2.1). After eliminating noisy samples, we proceed establish various in- tervals for the number of comment tokens, aiming to determine the optimal upper and lower bounds that yield high-quality collected comments. Our observations reveal that inline comments exceed- ing 15 tokens typically incorporate code snippets, while comments containing fewer than 3 tokens lack substantial meaningful information. Conse- quently, this interval serves as filtering criterion to generate the final version of D Figure 7 shows the distribution of code-token length and docstring-toker length in Dotock set.
Figure 9 provides statistics on the number of docstrings following a standard style. The data suggests that styled docstrings constitute small fraction of the overall code-text dataset. One pos- sible explanation is that our style detection rules are stringent, excluding doestrings with even mi- nor syntax deviations, which might result in un- derestimating the number of doestrings adhering to a specific format. For styled doestrings, Figure 9- 9-bottom presents the distribution of the number

---

Number of raw classes
Number classes after filtering
Number of raw inline comments
Number of inline comments
Language
comment
wo/ comment
after filtering
Python
497,550
1,440,539
422,187
24,066,884
14,013,238
РHP
2,223,472
6,232,188
1,173,916
9.892,486
5,873,744
JavaScript
494,819
2,409,932
291,479
4,426,086
1,438,110
Java
8,438,772
11,997,783
4,872,485
24,982,298
17,062,277
C#
2,378,379
9,097,968
1,437,800
10,130,
6,274,389
C++
285,184
791,355
174,370
20.770.494
10,343,650
Rust
188,517
3,591,465
93,311
2,998,368
2,063,784
Ruby
721,338
2,903,507
353,859
1,236,
767,563
C
16,009,812
6,778,239
Go
7,574,542
4,390,342
Total
15,228,031 38,464,729
8,819,407
122,087,817
69,005,336
Table 12: The number of classes and inline comments associated with the class and inline set. The symbol '-' indicates that this information is unavailable due to the nonexistence of traditional classes and Go.
Code token
Docstring token
Ruby
Ruby
C++
C+-1
CW
Java
Java
JavaScript
JavaScript
PHP
PHP
Python
Python
200
400
600
800
1000
1200
1400
Figure 6: Code and Doestring tokens length distribution of the Class set after filtering.
Code_token
Comment token
Rust
Rust
Ruby
Ruby
Cpp
Cop
C sharp
sharp
Java
Java
JavaScript
JavaScript-
PHP
PHP
Python
Python
100
200
300
400
500
10
Figure 7: Code Docstring tokens length distribution of Dotock set after filtering.
A.5 Docstring Styling
definition in programming languages. It is usu- ally placed the first statement in the code block (which can be inside outside the code block it- self) and enclosed by comment delimiter (c.g..
A docstring is a string literal used as a form docu- mentation for a module, function, class, or method

---

Python
Java
JavaScripe
Go
PHP
Ruby
Rust
C++ C#
Model
Fine-tune data
CODESRARCHNET TESTSET (MRR)
CodeScarchNet
0,3793
0.4636
0.4437
0.6201
0.4741
0,5219
0,4838
CodeBERT
The Vault/small
0.4074
0.4857
0.4466
0.6578
0.6578
0.5251
0.5301
The Vault/medium
0.6945
0.7096
Vault
0.6952
0,7242
0.6562
0,8789
0.6646
0,7474
0.7278
CodeeearccNeet
0.3479
0.448
0.4254
0,5684
0.4623
0.5147
0.6952
RoBERTa
TheVlul//small
0.4849
0.5581
0.4962
0,7446
0.5166
0.59
0.5651
UniXCoder
CodeSearchNet
0.3935
0.4549
0.4459
0.5861
0.489
0.5446
0.4857
The Vault/malll
0.4909
0.4515
0.5702
0.5079
THEVAULT TESTSET (MRR)
CodeScarchNet
0.2881
0.3213
0.2409
044233
0.1854
0.2579
0.2843
CodeBERT
TheVuuh/small
0,3501
0.4214
0.3216
0,4864
0.2351
0.2904
0.326
0.2996
0.3015
0.3483
TheVaull/medium
0.5929
0.6215
0.549
0.6862
0.3642
0.514
0.5705
0.5362
0.5264
0.5268
0.5488
TheVuult
0.6448
0.6633
0.592
0.7111
0.3891
0.5607
0.6243
0.5947
0.5932
0.5616
0.5935
CodeScarchNet
0,2644
0.3329
0.2371
0.2375
0.1577
0.2574
0.2478
RoRERTa
The Vaulll//smllll
0,4533
0.5519
0.4386
0,5021
0.2876
0.3717
0.4195
0.3805
0.37
0.4099
0.4342
UniXCoder
CodeSearchNet
0,2959
0.344
0.185
0.46
0.2669
0.2512
TheVanh/small
0.3852
0.4279
0.3491
0,4628
0.238
0.3201
0.363
0.2934
0.2861
0.3473
0.3639
Table 13: Code search results of various architectures and training datasct.
triple quotes (**) or a star slash ()))). Depending on developer comment habit docstring style format, docstrings can form two types: one-line docstrings and multi-line (or block) docstrings. A docstring can provide a concise summary of the functionality while also providing detailed description of the code block, including its parameters, return val- ues, exceptions, and other relevant information (as illustrated in Figure 8)
its corresponding docstring information is avail- able. Furthermore, the presence of various data types allows for the exploration of scenarios such as continual learning [Van al., 2022, Nguyen et al., 2023, Yadav al., 2023] and multitask learn- ing [Zhang et al., 2023], which have been lacking investigation in the context of source code data.
A.6 Experimental results on code summarization
The primary purpose of a docstring is provide clear, concise, and easily accessible documenta- tion for code block. Docstring styles are conven- tions followed while writing docstrings to ensure consistency, readability, and case of understanding throughout a codebase. This has become a standard for clean code in the industry and has developers saving tons of time when it comes to understanding (auto-) suuto-)generating documentation (using Sphinx, Doxygen, etc).
We report Rouge-L. BERTScore, and BLEU-4 met- ries on test sets of CSN and The Vault in Table 14. The results obtained from the experiments clearly indicate that models trained on dataset con- sistently outperform CSN on all three evaluation metrics. This notable improvement across the met- rics serves as strong evidence for the syntactic and semantic richness embedded within our dataset for code summarization. This highlights the effective- ness of our dataset in enabling models to grasp contextual information and generate high-quality summaries that accurately represent the underlying code functionality.
There are several popular docstring styles, such Google Style. NumPy Style, reStructuredText (reST) Style for Python programmers, JavaDoc Style or Doxygen for Java users, each with its own formatting rules, structure and target programming language (docstring style examples and preferred language are listed in Figure 10). The statistic for docstring style corresponding to function level is presented in Figure 9. We believe that information inside docstring is extremely useful and can pro- vide numerous advantages for various applications in the fields of AI for source code, such as pro- viding more precise and relevant search results for code search and retrieval tasks, or the performance code analysis or refactoring can be significantly improved while the identifier of a parameter and
A.7 Experimental results on code search
In this section, we assess Vaultss versatility and adaptability by providing additional experi- mental results on several architectures (RoBERTa [Liu et al., 1907], UniXcoder [Guo et al., 2022]. PLBART [Ahmad et al., 2021a]) for code search. Tables 13 illustrates the results for code search. As result, models trained The Vault consistently outperform all baseline models, underscoring both the efficiency of our pipeline and the dataset's 's abil- ity to generalize across different architectures.

---

CodeSearchNet
The Vault
Language
Finetune dataset
Rouge-I.
BERTScore
BLEU-4
Rouge-L
BERTScore
BLEU-4
CodeSearchNet
34.000
88.827
19.55
26.798
87.055
10.86
The TheVaul//mediumm
34.676
88,905
19.74
30.335
87.633
13.06
Python
Vault-
36.499
89.211
21.15
31.786
87.929
14.14
The Vautt///mmcuum--...
33.848
88.734
18.88
30.947
87.716
13.36
VaulttL
35.024
88,921
19.83
32.251
87.954
14.33
CodeSearchNet
35.625
89.132
20.38
27.297
87.385
8.00
The Vaul/mmuudu
33.385
88.490
18.62
31,320
87.897
11.17
Java
The
35.495
88,907
20.43
33.137
88.268
12.00
The The//al//medi
32.561
88.161
18.29
30.773
87.596
11.50
The
35,221
88.782
20.37
32.882
88.000
12.47
CodeSearchNet
28.330
87.568
16.15
24.895
86.519
8.42
The Vault/mediumm-
26.528
87.017
14.88
27.891
86.846
10.58
JavaSeript
The
28.345
87.384
16.30
29.817
87.320
11.71
The Vahl//i//me
27.062
87.057
14.95
28.290
86.936
10.83
The Vault-L
27.869
87.276
15.63
30.572
87.391
12.38
CodeScarchNet
41.346
89.981
26.26
39, 39.960
89.281
17.85
The Thelamm//me
34.802
88.125
21.78
63.984
93.287
37.72
РHP
VaultuS
37.297
88.676
23.53
65.401
93.580
38.30
The Vaht/tul/medi
33.325
87.963
20.27
65.195
93.679
39.13
The Vault-L.
36.478
88.641
23.21
67.089
94.012
40.13
CodeSearchNet
40.076
90.487
19.83
38.189
89.994
17.87
The Vaultmm/uuus
42.011
90.816
21.38
54.030
92.372
34.47
Go
Vault-S
44.649
91.188
24.37
54.889
92.541
35.44
The Vaull//m/ddiuu-LL
41.480
90.731
21.22
56.721
92.994
39.27
The V
44.063
91.108
23.96
57.681
93.130
40.38
CodeSearchNer
28.196
87.371
15.38
24,500
86.417
10.26
The Vault//emiiu
29.680
87.559
16.09
26.904
86.964
12.26
Ruhy
The
31.133
87.830
17.15
28.535
87.280
13.79
The LLul/uum/
29.389
87.565
15.42
27.485
87.044
12.63
The V
30.634
87.759
16.53
29.141
87.223
14.24
CodeScarchNet
36.739
89,341
21.24
30.563
87.853
16.1
The Vhut/m//mmd
34.935
88.755
19.91
39.589
89.278
26.02
Total
The
37.120
89,163
21.73
41.079
89.591
27.41
The Ihehaul//medium-..
34.086
88.585
19.16
40.544
89.473
27.71
The
36.305
89.024
21.14
42.187
89.753
29.32
The Vheetmuu/me
28. 132
86.277
10,21
The
33.275
87.353
13.39
C
The VheVaul//medium-L
29.151
86.566
11.32
Vault-L
35.009
87.807
14.86
The Vaultm/nuns
39.480
89.616
23.88
The Vault-S
46.854
90.819
31.11
The Vautt//mmdduu--L
39, 39.720
89.652
24.30
The
46,594
90.788
31.05
The Vaultm/mmuiss
28.00 28.029
86.719
14.55
VaultsS
29. 29.942
87.116
16.18
C++
The Vhheaa///mdiuu--L
28.815
86.827
14.85
Vaultu..
30.754
87.163
16.65
The Vault/medium-S
30.416
87.758
13.30
The Vault-S
32,535
88.126
14.72
Rust
30,999
87.862
13.75
The Vault-L.
32.857
88.143
15.18
Table 14: Experimental results for code summarization. For models that are finetuned on The Vault, "-S" annotation
refers to finetuning process using short docstring field summarization, while "-L" represents the docstring field.

---

Languages
Inconsistent pairs
// Handy for templates.
def has_urls :
if self.isbn_u self.isbo isbn_is self.official_ur) self. notes_uri:
Python
zeturn True else:
return False
// compresses the waveform horisontally: ene of // """norma">>y """resyc"*', ""zesyn2 def phase_type (self, value):
aelf __paamaaaaaa_ paaarama.paas_ttiee phase_type value ovefwtooerw disable
InWithIage, CutmithTage, Both, BethMishTaga Tune Predicates (from Shape, in booll Shape ( dir :- gubjeSubject
in dir jeett
return Unique (NodesFrom(
Go
Quads: Quads:
(Dir: diz, Values: from),
},
Diz: quad.Pred
// select Suzf rо PhontomJS func tself #DefaulsRequeat) GetDowsloaderID() int 1 (self. prepare) return flwn" caderTD TD
supplied callback fusction. public beolean (Pipe IMtriellandeer fune, XPub pub)
assert (pipe !- mall);
Java
(func 1- mill);
return rmHllpper (pipe, new byte[0], 0, 0, func, pub);
only change appe-ders
publin MapContentTypp getXapContent Type (CantainerTypp
containerType)) {
JaversType keyType get.averssype (Integer class); JaversType valueType getJaversType (containerType. getItemType return new TypCon (keyIype, valueType)

---

Languages
Inconsistent pairs
not Buffer pollyfil1
function (str)
var ret new Accay (str. len atr. str.lengt3;
JavaScript
while while(len--) (len--) zet [1en] stz, chazsooaaa (1en); UinteArray from(ret);
// ReakMap works in IEll, node 0.12 function (In, name)
function proxiedFn
'use strict';
var fields get // jshint ignore: ine fn.apply(fieldg argumenta);
deeect.eefnnePoooeee (pppee/eooptttypppxxeeeeeee 'name", value: name, configurable: true
11;
proxiedPh;
NEW
public function consumerId()
1f (Lsae (fttttttt---eeesooooddde ctatar/ateaattttttrtrre__e quatomer__d") ]1 --- true) {
PHP
return Sthis->sesslon--ddtt wtw.rdattaaatttttrreeeee "uatomer__dd"
seturn null;
dinini ba stur akan spa private function parsse_ruuess[
Suri-implode'' /', 9thie-buri-Dsegments//
if (Lsset (isset (fthis--rrtteerSSuu1111 return sthis->_set_reques (explode sthis-brouter (Suril));
foreach (8thia-brouter as Skey + Sval]
5key str_replace sany', f.+', str_replace inum', " 1[0-9++, Skeyl);
(pzeg_natch ("" "5H", $uril) { (sLepos (8val, )5*) PALSD (Expe, " (') !-- FALSE)
$val preg_raplace (nr.eyyssee Sskey. rst, Sval, Suri);
reture Sthis-> set set__set____ueeeeeeeeeeeeeeessse request ( (expoool Svall);
Sthis-b_aet request Sithis-ouze >eeegments ())

---

Languages
Inconsistent pairs
Initialize new page, which simply rendered or // persisted to the filesystem. def method missing (name, +arga, sbleck)
Ruby
return meta (name. if meta.key? (name,to_s) supex end
// Accepts the path of the YAML file to be parsed into // commands will throw a CommandDxception should it have
// invaalid paraneters
// @param fi filePath [String] Path for YAMI, file
def actiion_options
Attempt resolution outputs menitor
return Baction_eptions unless @manitor @manitor_class.outynne wlwsssoottuus..een..
acticn_op acticn_options Baction pection_eptions www.ettttiioo...ee...eeeeeeeee.e cons.oooose
Amonitoz cttess_caass.ucce wttssss.cttuutt.eeee.e
loutput, _typel
action_options each do loption epticn_valuel act iot_ ions (ottioon_ep option valueen_vllueee
@menitez. sendttrreeedddeee (output).
end end acticn_options end
Table 15: Inconsistent pairs in CodeSearchNet found by our model. "//" represents for doestring section.
Parameter
docstring
Docstring Style: Epylext
def intercept (m,
Retura the x intercept of the line Klyentxobi....
The Xix intercept) of a line is the point which it x (M(y=))).
This function can be used in conjuction with Liz transform) to find an arbitrary function's as zecus.
Stype number
8param slope of
Stype be nomber
Paran's docstring and
8parom b: The intorcopt of the line. The Xly intercept) of line the point at which it croases the axis (Mix=D)).
Stype count: string
Sparom count: The outlier param
docalting type
Ertype: number
Raturn's docstring and
freturn: the x intercept the Line Miymtx-b).
Eauthors Rpydoc' Rocumenta
Esees fitos://eoydde
Others
pass
Figure 8: Structure of docstring and its metadata.

---

1e7
1.00
w/style
0.75
all
0.50
0.25
0.00
Python
PHP
JavaScript
Java
C++
Ruby
#bsyle
2853520
8271
39295
14432
2754629
32517
25233
64427
156286
all
5893858
5455989
2362158
7886299
4011467
1994858
1978551
1076588
564867
Figure 9: Number of docstrings follows specific style all extracted code-text pairs. Upper figure and Middle table illustrate statisties for docstrings with style. Lower figures present the histogram of extracted attributes in the range of 1-20 for docstrings in each language. Golang does have supported style.

---

Google Style
JavaDoc
Python
Test Function.
paran1 (int): Description of param1. Test function.
Args:
@paran paran1 Description of param1.
param2 (str): Description of param2.
@paran param2 Description of param2.
Returns:
@return Description the return value.
bool: Description the return value.
RustDoc
reST
Rust
"*"Test function. Python
function.
:param param1: Description param1.
** Arguments
:type paran1; int
param1 Description of param1.
:paran param2: Description of param2.
param2": Description of param2.
:type paran2: str
Returns
:return: Description the return value.
Description of the return value.
:rtype: bool
Rdoc
Jadoc
=begin
Ruby
/
JavaScript
Test method.
Test function.
@param paran1 [Integer] Description of param1. of param2.
@param (int) param1 Description of param1. @param (string) param2 Description of param2.
@param paran2 [String] Description
@return [Boolean] Description of the return value.
@return (bool) Description of the return value.
=end
PHPdoc
Doxygen
/++
РHР
/*0
C++
C#
Test function.
Test function.
@brief Constructor.
@param Sparam1 Description of param1.
@param param1 Description of @param param2 Description of param2 gsee Test()
Sparam string Sparamz Description of paran2.
greturn bool Description of the return value.
XML
Epytext
11/ <summary>
C#
Python
1/1 Test function.
Test function.
11/ </summary>
@type param1:
/// <paran nane="parant" "xrremiitDeesiiiiiioo param1.
Sparam param1: Description param1
</param>
@type param2: string
<paran name-"param2" DDeccippto param1.
@param param2: Description of paran2
</param>
@rtype: bool
/// <returns>
greturn: Description of the return value.
1// Description of the return value.
111 </returns>
NumPy Style
Python
function. Parameters
param1 int Description of param1. param2 str Description of paran2. Returns
bool Description of the return value.
Figure 10: Supported doostring styles.


Reviews:
[{"review_id": "gPOmFmUSO6", "reviewer": "Reviewer_H3Qp", "summary": "", "questions": "A. Given that both CodeSearchNet and The Vault derive significant code samples from GitHub, have you assessed any overlap between the test set of one and the training set of the other to prevent potential data leakage?\n\nB. Recognizing that CodeSearchNet and The Stack are distinct datasets, how might model performance in code search and code summarization tasks differ if we were to draw random text-code pairs from The Stack as a baseline, in comparison to your refined The Vault?\n\nC. The term \"Multi-TheVault-raw\" in Table 7 isn't elaborated upon within the paper. Is this intended to represent a dataset randomly sampled from The Stack?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we would like to express our gratitude for your constructive feedback.\n\n**Q1. Although the dataset includes 10 widely-used languages, they are just a fraction of the 300 languages found in The Stack. This could reduce its attractiveness for researchers focused on less prevalent languages.**\n\nWe acknowledge your point about our toolkit's limited number of supported languages. There are a few obstacles that prevent us from expanding our dataset beyond the top ten popular languages:\n1) Because each programming language has its own AST parser and grammar, and tree-sitter is a parser generator tool that provides a platform for users to write grammar for each language and. As a result, not every language in The Stack's list of 300 has an AST parser.\n2) Our primary effort is on parsing the code into AST for each programming language using a specific parser and extract code-text pairs from the ASTs. Beyond the scope of ten languages, writing the parser for the missing ones (among 300) is extremely time-consuming.\n\nWe did investigate the possibility of including low-resource languages in addition to the initial set of ten popular languages. During our testing, however, we encountered difficulties in maintaining control over the quality and consistency of parsing across a wide range of languages. As a result, our strategy is to bundle our toolkit into a library that can be easily extended so that the research community can contribute to the expansion to other languages.\n\n**Q2. The code generation evaluation was based on small models like CodeGen 350M. This could restrict our understanding of the dataset's efficacy with large-scale models.**\n\nWe provide the results of CodeGen 2B and the additional MBPP dataset in the code generation experiment. As a result, The Vault can improve the performance of pre-existing pre-trained models in both datasets.\n\n|Model | Model size |  Dataset | pass@1 | pass@10 | pass@100 |\n|:-------|:------|:------|------:|-------:|-------:|\n|CodeGen-multi | 350m | HumanEval |  6.67 | 10.61| 16.84 |\n|Multi-TheVault | 350m | HumanEval |  **8.14** | **18.12** | **30.0** |\n|CodeGen-multi | 2B | HumanEval |  **14.51** | 24.67 | 38.56 |\n|Multi-TheVault | 2B | HumanEval |  14.00 | **25.74** | **41.72** |\n|CodeGen-multi | 350m | MBPP| 7.46| 24.18 | 46.37 |\n|Multi-TheVault | 350m | MBPP |  **10.13** | **33.96** | **53.20** |\n|CodeGen-multi | 2B | MBPP |  18.06 | 45.80 | **65.34** |\n|Multi-TheVault | 2B | MBPP |  **27.82** | **50.06** | 65.06 |\n\n\n**Q3. Only one model is included in each down stream task, which may not provide a comprehensive assessment of the dataset's versatility and adaptability across different architectures and frameworks.**\n\nWe acknowledge your concern about the comprehensiveness of our assessment due to the inclusion of a single model in each downstream task. While some existing datasets in related works might explore a variety of models, they often lack a thorough evaluation of data quality compared to other datasets. In contrast, we spend the effort on data assessment of The Vault to other datasets, hence, we fixed model architecture.\n\nTo address your concern about TheVault’s versatility and adaptability, we provide experiment results on several architectures (RoBERTa [5], UniXcoder [6], PLBART [7]) for code search and code summarization (and a large-scale model in the code generation task mentioned in the Q2). As a result, We witness a similar trend across various architectures.\n\n*Code Search:*\n- Evaluation on TheVault (MRR)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| CodeBERT | CodeSearchNet | 0.2881 | 0.3213 | 0.2409 | 0.4123 | 0.1854 | 0.2579 | 0.2843 |\n|  | TheVault/small | 0.3501 | 0.4214 | 0.3216 | 0.4864 | 0.2351 | 0.2904 | **0.3508** |\n| RoBERTa | CodeSearchNet | 0.2644 | 0.3329 | 0.2371 | 0.2375 | 0.1577 | 0.2574 | 0.2478 |\n|  | TheVault/small | 0.4533 | 0.5519 | 0.4386 | 0.5021 | 0.2876 | 0.3717 | **0.4342** |\n| UniXCoder | CodeSearchNet | 0.2959 | 0.344 | 0.2508 | 0.185 | 0.1646 | 0.2669 | 0.2512 |\n|  | TheVault/small | 0.3852 | 0.4279 | 0.3491 | 0.4628 | 0.238 | 0.3201 | **0.3639** |\n\n- Evaluation on CodeSearchNet (MRR)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| CodeBERT | CodeSearchNet | 0.3793 | 0.4636 | 0.4437 | 0.6201 | 0.4741 | 0.5219 | 0.4838 |\n|  |  TheVault/small | 0.4074 | 0.4857 | 0.4466 | 0.6578 | 0.6578 | 0.5251 | **0.5301** |\n| RoBERTa | CodeSearchNet | 0.3479 | 0.448 | 0.4254 | 0.5684 | 0.4623 | 0.5147 | 0.4611 |\n|  | TheVault/small | 0.4849 | 0.5581 | 0.4962 | 0.7446 | 0.5166 | 0.59 | **0.5651** |\n| UniXCoder | CodeSearchNet | 0.3935 | 0.4549 | 0.4459 | 0.5861 | 0.489 | 0.5446 | 0.4857 |\n|  | TheVault/small | 0.4427 | 0.4909 | 0.4506 | 0.6416 | 0.4515 | 0.5702 | **0.5079** |\n\n*Code Summarization:*\n- Evaluate on TheVault (BLEU-4)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg | \n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | CodeSearchNet | 10.86 | 8.00 | 8.42 | 17.87 | 17.85 | 10.26 | 16.11/12.21 |\n|  | TheVault/small | 12.26 | 11.13 | 9.68 | 31.64 | 38.86 | 11.23 | **25.12/19.13** |\n| PLBART | CodeSearchNet | 10.24 | 7.26 | 7.64 | 16.90 | 13.83| 9.60 | 14.39/10.91 | \n|  | TheVault/small | 10.23 | 9.28 | 8.95 | 22.78 | 34.32| 9.74 | **20.29/15.88** |\n\n- Evaluate on CodeSearchNet (BLEU-4)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg |\n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | CodeSearchNet | 19.55 | 20.38 | 16.15 | 19.83 | 26.26 | 15.38 | **21.24/19.59** |\n|  | TheVault/small | 18.94 | 17.72 | 13.96 | 19.92 | 20.43 | 15.22 | 18.83/17.70 |\n| PLBART | CodeSearchNet | 17.99 | 17.38| 14.84 | 17.98 | 22.54 | 14.08 | **18.78/17.47** |\n|  | TheVault/small | 14.93 | 15.66 | 11.95 | 17.03 | 18.0 | 11.49 | 15.95/14.84 |\n\n\n**Q4. Even though The Vault emphasizes quality, its scale is considerably smaller than some raw datasets like The Stack and The Pile. This may limit its effectiveness in some applications where sheer volume is paramount.**\n\nAs you also mentioned, The Vault's goal is to emphasize quality over quantity. Recent work on fine-tuning LLMs also demonstrates that we only need a small amount of high-quality [1] [2] data to fine-tune a model that performs better than that. In our case, The Vault is not intended to replace larger scale datasets such as The Stack and The Pile, but rather to supplement the model that has been trained on such datasets. In our evaluation of code generation, we found that CodeGen [3] when fine-tuned on The Vault outperforms the original pretrained on The Pile and BigQuery (Table 7 in our paper).\n\n**Q5. Given that both CodeSearchNet and The Vault derive significant code samples from GitHub, have you assessed any overlap between the test set of one and the training set of the other to prevent potential data leakage?**\n\nYes, in our experiment setup, we employed the MinHash LSH technique [4] to filter training instance clusters that are close to samples in the validation and test sets of CodeSearchNet, HumanEval, and MBPP. This filtering was based on code and docstring similarity, in addition to employing exact match deduplication. Additionally, during dataset partitioning into training, testing, and validation sets, we prevented content from the same repository from appearing in multiple sets, thereby avoiding any potential internal data leakage.\n\n**Q6. Recognizing that CodeSearchNet and The Stack are distinct datasets, how might model performance in code search and code summarization tasks differ if we were to draw random text-code pairs from The Stack as a baseline, in comparison to your refined The Vault?**\n\nIn response to your concern, we have established 3 function-level subsets, each approximately the size of TheVault/small (~1.7M code-text instances). These subsets were created by randomly sampling the raw function-level dataset extracted from The Stack, without applying any filtering (referred to as raw-TheStack). We use 3 different seeds to sample raw-TheStack and report the average result. The tables below illustrate the impact of our cleaning pipeline, presenting a comparison between models fine-tuned on TheVault/small and those trained on raw-TheStack. These comparisons clearly reveal the filtering methods in The Vault. During training on the raw-TheStack dataset for the code summarization task, we find that the PLBART and CodeT5 generates outputs with substantial noise. These outputs are characterized by a prevalence of special tokens like //; and *. This finding strongly underscores the efficacy of our filtering process in enhancing the quality of the dataset.\n\n*Code Search*\n- Evaluation on TheVault (MRR)\n| Model | Python | Java | JavaScript | Go | PHP | Ruby | Rust | C++ | C | C# |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| raw-TheStack | 0.318 | 0.3245 | 0.1837 | 0.4194 | 0.1718 | 0.0878 | 0.1949 | 0.2024 | 0.2204 | 0.2309 |\n| TheVault/small | **0.3501** | **0.4214** | **0.3216** | **0.4864** | **0.2351** | **0.2904** | **0.326** | **0.2996** | **0.3015** | **0.3483** |\n\n- Evaluation on CodeSearchNet (MRR)\n| Model | Python | Java | JavaScript | Go | PHP | Ruby |\n| --- | --- | --- | --- | --- | --- | --- |\n| raw-TheStack | 0.3713 | 0.3492 | 0.3148 | 0.5519 | 0.2731 | 0.2748 |\n| TheVault/small | **0.4074** | **0.4857** | **0.4466** | **0.6578** | **0.6578** | **0.5251** |\n\n*Code Summarization:*\n- Evaluate on TheVault (BLEU-4)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg | \n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | raw-TheStack | 16.18 | 9.06 | 6.23| 19.05 | 7.07 | 5.78 | 11.84/10.56 |\n|  | TheVault/small |12.26 | 11.13 | 9.68 | 31.64 | 38.86 | 11.23 | **25.12/19.13** |\n| PLBART | raw-TheStack | 1.69  | 4.02 | 0.43 | 24.60  | 4.83  | 0.49  | 7.19/6.01 |\n|  | TheVault/small | 10.23 | 9.28 | 8.95 | 22.78 | 34.32| 9.74 | **20.29/15.88** |\n\n- Evaluate on CodeSearchNet (BLEU-4)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg |\n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | raw-TheStack | 16.18 | 9.06 | 6.23| 19.05 | 7.07 | 5.78 | 11.84/10.56 |\n|  | TheVault/small | 18.94 | 17.72 | 13.96 | 19.92 | 20.43 | 15.22 | **18.83/17.70**|\n| PLBART | raw-TheStack | 0.86 | 3.06 | 0.59 | 10.91 | 2.29 | 0.47 | 3.23/3.03 |\n|  | TheVault/small | 14.93 | 15.66 | 11.95 | 17.03 | 18.0 | 11.49 | **15.95/14.84** |\n\n**Q7. The term \"Multi-TheVault-raw\" in Table 7 isn't elaborated upon within the paper. Is this intended to represent a dataset randomly sampled from The Stack?**\n\nYes, \"Multi-TheVault-raw\" refers to a specific scenario where the CodeGen-350m-multi model is finetuned on a subset of Python data from The Stack. This subset mirrors the size of Python data present in The Vault dataset. Through this comparison, we demonstrate the advancements achieved in our refinement process and its potential to yield improved results when used for training.\n\n[1] LIMA: Less Is More for Alignment, https://arxiv.org/abs/2305.11206 \\\n[2] Instruction Mining: High-Quality Instruction Data Selection for Large Language Models, https://arxiv.org/abs/2307.06290 \\\n[3] Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., ... & Xiong, C. (2022, September). CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In The Eleventh International Conference on Learning Representations. \\\n[4] Eric Zhu, & Vadim Markovtsev. (2017). ekzhu/datasketch: First stable release (v1.0.0). Zenodo. https://doi.org/10.5281/zenodo.290602 \\\n[5] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. \\\n[6] Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022). Unixcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv:2203.03850. \\\n[7] Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K. W. (2021). Unified pre-training for program understanding and generation. arXiv preprint arXiv:2103.06333.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a substantially larger dataset of high-quality code-text pairs, surpassing existing datasets like CodeSearchNet. Evidence from three downstream tasks indicates that models trained on the Vault outperform those trained on CodeSearchNet.\n- The authors present an innovative data cleaning methodology that blends traditional rule-based techniques with deep learning classifiers, effectively filtering out many inconsistent code-text pairs.\n- By making the toolkit for dataset creation and cleaning publicly available, the authors not only show dedication to open research but also encourage reproducibility and further community-driven improvements.\n- The paper is well written and easy to follow. Detailed experiments are conducted to help evaluate and analyze the capabilities of the new dataset.", "reasons_to_reject": "- Although the dataset includes 10 widely-used languages, they are just a fraction of the 300 languages found in The Stack. This could reduce its attractiveness for researchers focused on less prevalent languages.\n- The code generation evaluation was based on small models like CodeGen 350M. This could restrict our understanding of the dataset's efficacy with large-scale models.\n- Only one model is inluced in each down stream task, which may not provide a comprehensive assessment of the dataset's versatility and adaptability across different architectures and frameworks.\n- Even though The Vault emphasizes quality, its scale is considerably smaller than some raw datasets like The Stack and The Pile. This may limit its effectiveness in some applications where sheer volume is paramount.", "questions_to_authors": "A. Given that both CodeSearchNet and The Vault derive significant code samples from GitHub, have you assessed any overlap between the test set of one and the training set of the other to prevent potential data leakage?\n\nB. Recognizing that CodeSearchNet and The Stack are distinct datasets, how might model performance in code search and code summarization tasks differ if we were to draw random text-code pairs from The Stack as a baseline, in comparison to your refined The Vault?\n\nC. The term \"Multi-TheVault-raw\" in Table 7 isn't elaborated upon within the paper. Is this intended to represent a dataset randomly sampled from The Stack?", "paper_topic_and_main_contributions": "The paper introduces \"The Vault,\" a dataset tailored for training large language models to understand and generate code. The dataset consists of 41 million pairs of high-quality code-text pairs extracted from 10 prevalent programming languages. \n\nIt surpasses existing datasets both in quality and scale. Empirical evaluation of \"The Vault\" by finetuning popular code language models, illustrating its superior performance compared to other datasets. And the entire toolkit used for dataset creation and quality assurance open-sourced to foster community engagement.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "- “CodeBert” in L450 and the title of Table 10 should be “CodeBERT”.\n- “Codegen” in L459, L468 and table 7 should be “CodeGen”."}, "extra_text": {}}, {"review_id": "5Mnp6Mytcx", "reviewer": "Reviewer_rHmi", "summary": "", "questions": "A. This paper mentions obtaining a consistency score through analogous prediction with GPT-3.5. It would be better if this paper could provide a more detailed explanation of how the authors perform the analogous prediction. Showing a prompting template or a prompt example would be very helpful.\n\nB. In Table 1, various rule-based filters are categorized into three categories: Readability, Consistency, and Intended usage. However, what practical significance does this tripartite categorization have? And the criteria for classification do not seem clear. For example, in the case of Strip Hyperlink, this paper classifies it as promoting consistency, but it seems that it could also be categorized as enhancing readability.\n\nC. Why is only one base model used for each task? In this paper, the CodeBERT model is used for the code search task, the CodeT5 model for the summarization task, and the Codegen multi-350m model for the code generation task. It would be better to use a few baselines for each task instead of only one. \n\nD. Does the deep-learning based filter model produce an output in the form of a two-dimensional vector?\n\nE. In line 890, this paper mentions that the lowest top 50 from each class are chosen when selecting the uncertainty group. But if there exists a data sample where the score for class 0 is very low but the score for class 1 is very high, wouldn't that sample be included in both the Consistency group and the Uncertainty group?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere gratitude your valuable insights and thoughtful evaluation of our work.\n\n**Q1. For the code summarization task and the code search task, the dataset size of Vault and CSN should be the same in order to demonstrate that Vault is better in terms of data quality than CSN.**\n\nWe understand your concern about the uneven and unfairness between experimenting dataset. In table 12, we highlight that TheVault/small (~1.7M instances) which is comparable in size to CodeSearchNet training set and in the Table 6 and 7, we reported that model trained on TheVault/small is better than model trained on CodeSearchNet in task code search and code generation. \n\nFor code summarization, we update the result in the table below. The model trained on TheVault outperform those trained on CodeSearchNet with a significant gap in TheVault testset, however, we expect the evaluation result on CodeSearchNet’s testset will be lower. Explain to this is our mentioned of the post-processing step to reduce the difference of the CSN and TheVault filtering methods (lines 418-421 in our paper), where the syntactic distribution can still exhibit nonidentical characteristics, which can affect the BLEU score. \n\n*Code Summarization*\n* Evaluate on TheVault\n| Model | Fine tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg | \n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | CSN | 10.86 | 8.00 | 8.42 | 17.87 | 17.85 | 10.26 | 16.11/12.21 |\n|  | TheVault/small | 12.26 | 11.13 | 9.68 | 31.64 | 38.86 | 11.23 | **25.12/19.13** |\n\n\n* Evaluate on CSN\n| Model | Fine tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg |\n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | CSN | 19.55 | 20.38 | 16.15 | 19.83 | 26.26 | 15.38 | **21.24/19.59** |\n|  | TheVault/small | 18.94 | 17.72 | 13.96 | 19.92 | 20.43 | 15.22 | 18.83/17.70 |\n\n**Q2. The compared dataset, CSN, is relatively old. It would be more convincing if the authors could include more recent datasets such as XLCoST for comparison.**\n\nCodeSearchNet is a well-known and widely recognized dataset in the field, known for its quality and usability, and it covers a wide range of languages. Current state-of-the-art models (e.g., CodeT5+ [1]) continue to benefit from and are evaluated on this dataset from CodeSearchNet. In contrast, the XLCost dataset is relatively new and has not been widely used as a benchmark for Code Large Language Models. Furthermore, no analysis in XLCost shows that the quality of XLCost is higher than CSN, so there are no compelling reasons to benchmark on XLCost. However, we recognize the importance of comparing the most recent datasets, and we will perform additional comparisons with XLCost in our future extension.\n\n**Q3. The authors run experiments on three tasks only: Code Search, Code Summarization, and Code Generation. To show the general performance improvement, it would require to include more tasks such as code translation or completion.**\n\nWe want to highlight that The Vault introduces large-scale code-text pairs data, which facilitates a deeper understanding of the relationships between programming languages and natural language. Consequently, we chose to focus our analysis on Code Search, Code Summarization, and Code Generation tasks, as they inherently encapsulate the integration of code and natural language. While tasks like code translation or code completion are undoubtedly important and interesting, they involve code-to-code interactions that fall beyond the primary scope and objectives of The Vault dataset. \n\n\n**Q4. In the experiments, only one baseline model is considered for each task. It would be better if there are more baselines to show the generality of Vault. Without it, one might suspect that only the well-performing results are reported.**\n\nWe acknowledge your concern about the comprehensiveness of our assessment due to the inclusion of a single model in each downstream task. While some existing datasets in related works might explore a variety of models, they often lack a thorough evaluation of data quality compared to other datasets. In contrast, we spend the effort on data assessment of The Vault to other datasets, hence, we fixed model architecture.\n\nTo address your concern, we provide experiment results on additional several models (RoBERTa [2], UniXcoder [3], PLBART [4]) for code search, code summarization and a large-scale model CodeGen-2B-multi for code generation task. We train on both *short_docstring* and *docstring* and report the best setting. As a result, We witness a similar trend across various architectures.\n\n*Code Search*\n* Evaluation on TheVault (MRR)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| CodeBERT | CodeSearchNet | 0.2881 | 0.3213 | 0.2409 | 0.4123 | 0.1854 | 0.2579 | 0.2843 |\n|  | TheVault/small | 0.3501 | 0.4214 | 0.3216 | 0.4864 | 0.2351 | 0.2904 | **0.3508** |\n| RoBERTa | CodeSearchNet | 0.2644 | 0.3329 | 0.2371 | 0.2375 | 0.1577 | 0.2574 | 0.2478 |\n|  | TheVault/small | 0.4533 | 0.5519 | 0.4386 | 0.5021 | 0.2876 | 0.3717 | **0.4342** |\n| UniXCoder | CodeSearchNet | 0.2959 | 0.344 | 0.2508 | 0.185 | 0.1646 | 0.2669 | 0.2512 |\n|  | TheVault/small | 0.3852 | 0.4279 | 0.3491 | 0.4628 | 0.238 | 0.3201 | **0.3639** |\n\n* Evaluation on CodeSearchNet (MRR)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| CodeBERT | CodeSearchNet | 0.3793 | 0.4636 | 0.4437 | 0.6201 | 0.4741 | 0.5219 | 0.4838 |\n|  |  TheVault/small | 0.4074 | 0.4857 | 0.4466 | 0.6578 | 0.6578 | 0.5251 | **0.5301** |\n| RoBERTa | CodeSearchNet | 0.3479 | 0.448 | 0.4254 | 0.5684 | 0.4623 | 0.5147 | 0.4611 |\n|  | TheVault/small | 0.4849 | 0.5581 | 0.4962 | 0.7446 | 0.5166 | 0.59 | **0.5651** |\n| UniXCoder | CodeSearchNet | 0.3935 | 0.4549 | 0.4459 | 0.5861 | 0.489 | 0.5446 | 0.4857 |\n|  | TheVault/small | 0.4427 | 0.4909 | 0.4506 | 0.6416 | 0.4515 | 0.5702 | **0.5079** |\n\n\n*Code Summarization*\n* Evaluate on TheVault (BLEU-4)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg | \n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | CodeSearchNet | 10.86 | 8.00 | 8.42 | 17.87 | 17.85 | 10.26 | 16.11/12.21 |\n|  | TheVault/small | 12.26 | 11.13 | 9.68 | 31.64 | 38.86 | 11.23 | **25.12/19.13** |\n| PLBART | CodeSearchNet | 10.24 | 7.26 | 7.64 | 16.90 | 13.83| 9.60 | 14.39/10.91 | \n|  | TheVault/small | 10.23 | 9.28 | 8.95 | 22.78 | 34.32| 9.74 | **20.29/15.88** |\n\n* Evaluate on CodeSearchNet (BLEU-4)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg |\n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | CodeSearchNet | 19.55 | 20.38 | 16.15 | 19.83 | 26.26 | 15.38 | **21.24/19.59** |\n|  | TheVault/small | 18.94 | 17.72 | 13.96 | 19.92 | 20.43 | 15.22 | 18.83/17.70 |\n| PLBART | CodeSearchNet | 17.99 | 17.38| 14.84 | 17.98 | 22.54 | 14.08 | **18.78/17.47** |\n|  | TheVault/small | 14.93 | 15.66 | 11.95 | 17.03 | 18.0 | 11.49 | 15.95/14.84 |\n\n\n*Code Generation*\n|Model | Model size |  Dataset | pass@1 | pass@10 | pass@100 |\n|:-------|:------|:------|------:|-------:|-------:|\n|CodeGen-multi | 350m | HumanEval |  6.67 | 10.61| 16.84 |\n|Multi-TheVault | 350m | HumanEval |  **8.14** | **18.12** | **30.0** |\n|CodeGen-multi | 2B | HumanEval |  **14.51** | 24.67 | 38.56 |\n|Multi-TheVault | 2B | HumanEval |  14.00 | **25.74** | **41.72** |\n|CodeGen-multi | 350m | MBPP| 7.46| 24.18 | 46.37 |\n|Multi-TheVault | 350m | MBPP |  **10.13** | **33.96** | **53.20** |\n|CodeGen-multi | 2B | MBPP |  18.06 | 45.80 | **65.34** |\n|Multi-TheVault | 2B | MBPP |  **27.82** | **50.06** | 65.06 |\n\n**Q5. This paper mentions obtaining a consistency score through analogous prediction with GPT-3.5. It would be better if this paper could provide a more detailed explanation of how the authors perform the analogous prediction. Showing a prompting template or a prompt example would be very helpful.**\n\nWe provide the template that is used to request GPT-3.5-turbo and some output examples as follows. We will also release these samples and GPT output in our public repository.\n\n**Prompt**\n```python\nI want you to act as an unbiased docstring evaluator for code. I will give you a docstring along with a source code, and you will give me a score for the consistency between them. The score will be on a scale of 1 to 10, 10 means the docstring can effectively summarize the code while 1 means they are inconsistent. The response answers must contain the score and the explanation that follows the format in the response format.\n\n- Response format:\nScore: X\nExplanation: Y\n\n- Docstring: \n\"{docstring}\"\n\n- Code:\n\"{code}\"\n```\n**Examples**\n```\n- Prediction: Consistence\n- Group: Consistence\n- Docstring: Reads in a GSLIB-style Geo-EAS data file\n- Code: \ndef read_gslib(flname, headeronly=False, delimiter=r'\\s*', tmin=None):\n    _test_file_open(flname)\n    if headeronly:\n        nrows = 1\n    else:\n        nrows = None\n    with open(flname, 'r') as datafl:\n        _ = datafl.readline().strip()  \n        nvar = int(datafl.readline().split()[0])\n        varnames = []\n        for _ in range(nvar):\n            varnames.append(datafl.readline().strip())\n        nrowtoskip = 2 + nvar\n    engine = Parameters['data.io.pandas_engine']\n    try:\n        tpdf = pd.read_csv(flname, skiprows=nrowtoskip, header=None,\n                           delimiter=delimiter, skipinitialspace=True,\n                           nrows=nrows, engine=engine, chunksize=100000)\n        data = pd.concat(tpdf, ignore_index=True)\n    except (ValueError, NotImplementedError):\n        data = pd.read_csv(flname, skiprows=nrowtoskip, header=None, delimiter=r'\\s*',\n                           skipinitialspace=True, nrows=nrows, engine='python')\n    data = _data_trim(data, tmin=tmin)\n    data.columns = varnames\n    return data\n- GPT output:\nScore: 8\nExplanation: The docstring states that the function reads in a GSLIB-style Geo-EAS data file, which is consistent with what the code does. However, the docstring could be more detailed and provide information on the expected inputs and outputs of the function.\n```\n\n```\n- Prediction: Inconsistence\n- Group: Uncertainty\n- Docstring: Here we make the assumption that all functions are not native\n- Code:\npublic Tuple<String, LibraryFunction> call(String library, String[] args) {\n                String funct_name = \"__\" + name.toUpperCase() + \"__\";\n                String[] used_args = usedArgs(args);\n                String funct_call = ((args[0] == null) ? \"\" : Services.strip(args[0]) + \" \") + \n                        funct_name + \"(\" + Services.join(used_args, \", \") + \")\" + \n                        ((args[args.length - 1] == null) ? \"\" : \" \" + args[args.length - 1]);\n                PreprocessData.addPlaceholderFunction(funct_name);\n                return new Tuple<String, LibraryFunction>(funct_call, this);\n        }\n\n\n- GPT output:\nScore: 6\nExplanation: The docstring mentions an assumption about the functions, but it is not clear how this assumption relates to the code. The code itself seems to be a method that takes a library name and an array of arguments, and returns a tuple containing a function call and a LibraryFunction object. There is no clear connection between the docstring and the code, so the score is 6.\n```\n\n**Q6. In Table 1, various rule-based filters are categorized into three categories: Readability, Consistency, and Intended usage. However, what practical significance does this tripartite categorization have? And the criteria for classification do not seem clear. For example, in the case of Strip Hyperlink, this paper classifies it as promoting consistency, but it seems that it could also be categorized as enhancing readability.**\n\nThe three categories correspond with our data cleaning objectives, which we derived through analyzing the dataset and closely monitoring the outcomes yielded by models trained on the dataset. Categorization gave us practical guidelines to brainstorm and implement rule-based filters. We made available on GitHub the source code for our data cleaning framework so that the community can tailor it to build custom filters that meet their own data cleaning objectives. \n\n**Q7. Why is only one base model used for each task? In this paper, the CodeBERT model is used for the code search task, the CodeT5 model for the summarization task, and the Codegen multi-350m model for the code generation task. It would be better to use a few baselines for each task instead of only one.**\n\nWe have updated the additional experiments. Please refer to Q4.\n\n**Q8. Does the deep-learning based filter model produce an output in the form of a two-dimensional vector?**\n\nYes, the output in the form of a two-dimensional vector as it is a binary classification problem. The model is trained to classify if a pair of code-text is aligned or not.\n\n**Q9. In line 890, this paper mentions that the lowest top 50 from each class are chosen when selecting the uncertainty group. But if there exists a data sample where the score for class 0 is very low but the score for class 1 is very high, wouldn't that sample be included in both the Consistency group and the Uncertainty group?**\n\nThe Uncertainty group aims to select the samples that introduce high entropy in the model prediction (the output vector is around 0.5 in each class). For each class, we select samples that have the lowest score (which is always higher than 0.5) based on the prediction probability for the class they belong to create the Uncertainty group. \n\nThe case where the score for one class is notably low while another class has a significantly high score can fall within the Uncertainty category if the model exhibits high confidence in its predictions even on the examples that have low scores. However, we avoid overlapping between each group by not repeating the selection of examples already selected in another set. The snippet code below shows our selection strategy.\n\n```python\nunmatch_set = sorted([x for x in data if x[“class”] == 0], key= lambda x: x[“prediction”][1])\nmatch_set = sorted([x for x in data if x[“class”] == 1], key= lambda x: -x[“prediction”][1])\n\nunmatch_subset = unmatch_set[:num_samples]\nmatch_set_subset = match_set[:num_samples]\nuncertain_set = unmatch_set[num_samples:][(-num_samples//2):] + match_set[num_samples:][(-num_samples//2):]\n```\n\n\n\n[1] Wang, Y., Le, H., Gotmare, A. D., Bui, N. D., Li, J., & Hoi, S. C. (2023). Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922. \\\n[2] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. \\\n[3] Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., & Yin, J. (2022). Unixcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv:2203.03850. \\\n[4] Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K. W. (2021). Unified pre-training for program understanding and generation. arXiv preprint arXiv:2103.06333.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The issues with the data quality and quantity of the existing code datasets are well analyzed, and practical solutions are proposed well. \n\n2. The performance improvement is noticeable. When performing the code summarization task and code search task, a model trained on the training set of Vault performs better on the testing set of Code Search Net (CSN), than a model trained on the training set of CSN.\n", "reasons_to_reject": "1. For the code summarization task and the code search task, the dataset size of Vault and CSN should be the same in order to demonstrate that Vault is better in terms of data quality than CSN.\n\n2. The compared dataset, CSN, is relatively old. It would be more convincing if the authors could include more recent datasets such as XLCoST for comparison. \n\n3. The authors run experiments on three tasks only: Code Search, Code Summarization, and Code Generation. To show the general performance improvement, it would require to include more tasks such as code translation or completion. \n\n4. In the experiments, only one baseline model is considered for each task. It would be better if there are more baselines to show the generality of Valut. Without it, one might suspect that only the well-performing results are reported.\n", "questions_to_authors": "A. This paper mentions obtaining a consistency score through analogous prediction with GPT-3.5. It would be better if this paper could provide a more detailed explanation of how the authors perform the analogous prediction. Showing a prompting template or a prompt example would be very helpful.\n\nB. In Table 1, various rule-based filters are categorized into three categories: Readability, Consistency, and Intended usage. However, what practical significance does this tripartite categorization have? And the criteria for classification do not seem clear. For example, in the case of Strip Hyperlink, this paper classifies it as promoting consistency, but it seems that it could also be categorized as enhancing readability.\n\nC. Why is only one base model used for each task? In this paper, the CodeBERT model is used for the code search task, the CodeT5 model for the summarization task, and the Codegen multi-350m model for the code generation task. It would be better to use a few baselines for each task instead of only one. \n\nD. Does the deep-learning based filter model produce an output in the form of a two-dimensional vector?\n\nE. In line 890, this paper mentions that the lowest top 50 from each class are chosen when selecting the uncertainty group. But if there exists a data sample where the score for class 0 is very low but the score for class 1 is very high, wouldn't that sample be included in both the Consistency group and the Uncertainty group?\n", "paper_topic_and_main_contributions": "1. This paper presents a new dataset of high quality code-text pairs in multiple programming languages. The previous code datasets have problems: 1) Despite having a large amount of data, the quality of the data is low, or 2) while the quality is good, the amount of data was small. This paper suggests an ideal dataset satisfying both data quality and quantity. \n\n2. This paper proposes a data cleaning method for constructing and quality-controlling from raw source code data. By applying this method, the output dataset, called Vault, shows a performance improvement by a large margin on the three downstream tasks: code summarization, code generation, and code search. \n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "1. In line 245, The ')' character has been mistakenly included. It seems it needs to be removed.\n\n2. In Figure 8, number 4, \"Oulier\" should be corrected to \"Outlier\".\n"}, "extra_text": {}}, {"review_id": "efxsFEE1G4", "reviewer": "Reviewer_EqfE", "summary": "", "questions": "See Reasons To Reject.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely value your thoughtful feedback and constructive questions.\n\n**Q1. The Vault is built on another already existing Stack dataset, reducing the contribution**\n\nThe Vault's contribution is in the pipeline of cleaning up and providing high-quality pairs of code and natural language. This pipeline works with any of the large-scale datasets available, including The Pile, The Stack, and CodeParrot. \n\nIt should be noted that larger-scale code datasets, such as The Stack, CodeParrot, and The Pile, are collected from public code on Github; the difference between them is mainly in size and some efforts to address privacy concerns, not in data quality. In our case, we could collect public GitHub and use the pipeline to extract high-quality pairs of code and docstring, but this is unnecessary and time-consuming given the availability and comprehensiveness of datasets like The Stack. \n\n**Q2. The proposed pipeline lacks enough novelty, not only for rules and the trained classifier**\n\nOur contributions lie in the rich and diverse information extraction and innovative filtering methods we've introduced. We've implemented a multi-level approach, encompassing functions, inlines, and classes, which effectively addresses the granularity of data extraction. This multilevel type of data provides a valuable foundation for subsequent tasks. A novel assessment approach with carefully grouping examples to prove the effectiveness of our filtering model.\n\nIn comparison to existing datasets, we provide a larger set of rules to filter data in reducing noise. In addition, a distinguishing aspect of our work is the application of deep learning techniques to the filtering process. Unlike the mentioned datasets that solely rely on rule-based filtering, we introduce a strategic integration of DL within our pipeline. \n\n**Q3. The paper needs more details about the D_unimodal and D_block datasets. For example, does the proposed pipeline process these two datasets? Besides, the authors do not evaluate the quality of these two datasets**\n\nWe acknowledge the reviewer’s observation about the missing experiment in the D_unimodal and D_block datasets. Our main reason for focusing primarily on the D_paired dataset lies in our objective to showcase the dataset's quality and its applicability to various code-related tasks.\n\nTo our knowledge, we notice that these two datasets present unique challenges in terms of proper task modeling. Unlike function stand-alone pairs (in D_paired), which are able to demonstrate the benefit of our dataset quality, D_unimodal overlaps with the scope of other raw code collections and often serves as a resource for pre-trained language models [1, 2]. On the other hand, D_block is a lack of clear task definitions or specific problem formulations (e.g., the author in [3] claims class code snippet level might not be as effective for code summarization tasks due to their extended length).\n\nWhile comprehensive insights and evaluations of these two datasets are not our main focus in this paper, we acknowledge the potential value of future work to delve into the unique characteristics and applications of these datasets.\n\n\n**Q4. Some flaws in experiments:**\n\n**Q4.1. The authors compare the Vault and CSN datasets in the code summarization task. However, the authors do not control the sizes of these datasets to be equal or similar. Specifically, the full and medium versions of Vault have ~32 and ~6 million samples, which are much larger than the ~2 million of the CSN (shown in Table 3). As a result, the evaluating results can not firmly prove that the dataset quality of Vault is better than CSN**\n\nIn our Code Search and Code Generation reports, we emphasize the evaluation of TheVault/small; a dataset of comparable size (~1.7M instances) to the CodeSearchNet training set, which presents better performance compared to CSN. It is accurate to acknowledge that we inadvertently omitted experiments with equal dataset sizes in the Code Summarization report. To rectify this, we have included the updated results of our fine-tuning on TheVault-small in the tables below.\n\n- Evaluate on TheVault\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg | \n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | CodeSearchNet | 10.86 | 8.00 | 8.42 | 17.87 | 17.85 | 10.26 | 16.11/12.21 |\n|  | TheVault/small | 12.26 | 11.13 | 9.68 | 31.64 | 38.86 | 11.23 | **25.12/19.13** |\n| PLBart | CodeSearchNet | 10.24 | 7.26 | 7.64 | 16.90 | 13.83| 9.60 | 14.39/10.91 | \n|  | TheVault/small | 10.23 | 9.28 | 8.95 | 22.78 | 34.32| 9.74 | **20.29/15.88** |\n\n- Evaluate on CodeSearchNet (BLEU-4)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg |\n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | CodeSearchNet | 19.55 | 20.38 | 16.15 | 19.83 | 26.26 | 15.38 | **21.24/19.59** |\n|  | TheVault/small | 18.94 | 17.72 | 13.96 | 19.92 | 20.43 | 15.22 | 18.83/17.70 |\n| PLBart | CodeSearchNet | 17.99 | 17.38| 14.84 | 17.98 | 22.54 | 14.08 | **18.78/17.47** |\n|  | TheVault/small | 14.93 | 15.66 | 11.95 | 17.03 | 18.0 | 11.49 | 15.95/14.84 |\n\nEven though we observe the result evaluation on CodeSearchNet testset is lower, the model trained on TheVault outperform with a more significant gap in TheVault testset. This phenomenon can be explained by the mention of the post-processing step to reduce the difference of the CSN and TheVault filtering methods (lines 418-421) in our paper, where the syntactic distribution can still exhibit nonidentical characteristics, which can affect the BLEU score. However, this gap can be overcome by using the full dataset of The Vault to achieve better results in both test sets.\n\n\n**Q4.2. The paper needs to include the ablation studies for the proposed process pipeline. The following experiments should be considered: the experiment of the un-processed raw Stack dataset vs. Vault; the experiment about the influence of rule-based and dl-based process methods, individually. Likewise, the authors should set the above compared datasets with equal sizes**\n\nIn the experiment of code generation, we had already compared the unprocessed The Stack (Multi-TheVault-raw) dataset and The Vault  (Table 7). We provide additional results for code summarization and code search tasks in the below tables. As a result, the performance of models trained on The Vault surpasses trained models on The Stack, presenting the effectiveness of our cleaning process.\n\nIn order to address your concern, we randomly select subsets from the raw code-text function pairs that were retrieved from The Stack, without performing any filtering (raw-TheStack), each set approximately the size of TheVault/small (~1.7M code-text instances). We use 3 different seeds to sample raw-TheStack and report the average result. The impact of our cleaning procedure on raw, extracted The Stack is presented in the tables below; these comparisons show that models trained on The Vault significantly outperform models that are trained on raw-TheStack.\n\n*Code Search*\n- Evaluation on TheVault (MRR)\n| Model | Python | Java | JavaScript | Go | PHP | Ruby | Rust | C++ | C | C# |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| raw-TheStack | 0.318 | 0.3245 | 0.1837 | 0.4194 | 0.1718 | 0.0878 | 0.1949 | 0.2024 | 0.2204 | 0.2309 |\n| theVault-small | **0.3501** | **0.4214** | **0.3216** | **0.4864** | **0.2351** | **0.2904** | **0.326** | **0.2996** | **0.3015** | **0.3483** |\n\n- Evaluation on CodeSearchNet (MRR)\n| Model | Python | Java | JavaScript | Go | PHP | Ruby |\n| --- | --- | --- | --- | --- | --- | --- |\n| raw-TheStack | 0.3713 | 0.3492 | 0.3148 | 0.5519 | 0.2731 | 0.2748 |\n| theVault-small | **0.4074** | **0.4857** | **0.4466** | **0.6578** | **0.6578** | **0.5251** |\n\n*Code Summarization*\n- Evaluate on The Vault (BLEU-4)\t\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg | \n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | raw-TheStack | 16.18 | 9.06 | 6.23| 19.05 | 7.07 | 5.78 | 11.84/10.56 |\n|  | TheVault/small |12.26 | 11.13 | 9.68 | 31.64 | 38.86 | 11.23 | **25.12/19.13** |\n| PLBart | raw-TheStack | 1.69  | 4.02 | 0.43 | 24.60  | 4.83  | 0.49  | 07.19/06.01 |\n|  | TheVault/small | 10.23 | 9.28 | 8.95 | 22.78 | 34.32| 9.74 | **20.29/15.88** |\n\n- Evaluate on CodeSearchNet (BLEU-4)\n| Model | Fine-tune data | Python | Java | JavaScript | Go | PHP | Ruby | Total/Avg |\n| :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| CodeT5 | raw-TheStack | 16.18 | 9.06 | 6.23| 19.05 | 7.07 | 5.78 | 11.84/10.56 |\n|  | TheVault/small | 18.94 | 17.72 | 13.96 | 19.92 | 20.43 | 15.22 | **18.83/17.70**|\n| PLBart | raw-TheStack | 0.86 | 3.06 | 0.59 | 10.91 | 2.29 | 0.47 | 3.23/3.03 |\n|  | TheVault/small | 14.93 | 15.66 | 11.95 | 17.03 | 18.0 | 11.49 | **15.95/14.84** |\n\nFurthermore, the practical instances showcased in Table 8 and Table 14 within our paper provide a rational basis for both our rule-based and deep learning-based methodologies, respectively.\n\n**Q4.3. Besides, the authors could also apply the proposed pipeline to another dataset, such as CSN, and then compare it with the raw dataset, which is helpful to show the generalization ability of these methods.**\n\nWe acknowledge your recommendation to apply the pipeline to CSN. However, there are several difficulties since CSN is already processed and filtered from raw GitHub files in different initial steps. Thus, we can not filter CSN from beginning. Table 8 and Table 14 within our paper apply to CSN can show a rational basis of our filtering approach.\n\nWe refer that the comparison experiment to apply the pipeline to CSN could be simulated by comparing the raw version of The Stack (raw-TheStack) and TheVault/small, since they have similar in size and are sourced from GitHub. The results are shown in Q4.2 indicate the effectiveness of our process.\n\n[1] Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., ... & Xiong, C. (2022, September). CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In The Eleventh International Conference on Learning Representations. \\\n[2] Wang, Y., Le, H., Gotmare, A. D., Bui, N. D., Li, J., & Hoi, S. C. (2023). Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922. \\\n[3] Parvez, M. R., Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K. W. (2021). Retrieval augmented code generation and summarization. arXiv preprint arXiv:2108.11601. \\", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper presents a new pre-process pipeline that could improve code-related datasets' quality.\n2) The authors evaluate the proposed Vault dataset across different tasks and models, which shows much improvement compared with the CodeSearchNet dataset.", "reasons_to_reject": "1) The Vault is built on another already existing Stack dataset, reducing the contribution.\n2) The proposed pipeline lacks enough novelty, not only for rules and the trained classifier.\n3) The paper needs more details about the D_unimodal and D_block datasets. For example, does the proposed pipeline process these two datasets? Besides, the authors do not evaluate the quality of these two datasets.\n4) Some flaws in experiments:\\\n4.1) The authors compare the Vault and CSN datasets in the code summarization task. However, the authors do not control the sizes of these datasets to be equal or similar. Specifically, the full and medium versions of Vault have ~32 and ~6 million samples, which are much larger than the ~2 million of the CSN (shown in Table 3). As a result, the evaluating results can not firmly prove that the dataset quality of Vault is better than CSN. \\\n4.2) The paper needs to include the ablation studies for the proposed process pipeline. The following experiments should be considered: the experiment of the un-processed raw Stack dataset vs. Vault; the experiment about the influence of rule-based and dl-based process methods, individually. Likewise, the authors should set the above compared datasets with equal sizes.\\\n4.3) Besides, the authors could also apply the proposed pipeline to another dataset, such as CSN, and then compare it with the raw dataset, which is helpful to show the generalization ability of these methods.", "questions_to_authors": "See Reasons To Reject.", "paper_topic_and_main_contributions": "This paper presents a new multilingual code-related dataset named The Vault for training LLMs to understand and generate code. The authors build The Vault on another huge dataset called The Stack and present a novel data cleaning pipeline to process the raw dataset, including the rule-based and dl-based methods. The authors compare the Vault dataset with the baseline CodeSearchNet dataset using various models (CodeBERT, CodeT5, and CodeGen) in different code-related tasks, including code search, summarization, and generation, which finally show improvement.\n\nIn conclusion, the main contributions of this paper include: \n1) a huge multilingual code dataset\n2) a novel data cleaning pipeline improving the quality of code-related datasets", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The references of BLEU, Rouge-L, and Bertscore metrics."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) Replace 'they' with 'researchers' in line 020\n2) remove ')'  of 'D_paired)' in line 245\n3) In lines 406-407, perhaps 'we utilize the Vault and CodeSearchNet to fine-tune CodeT5' is better\n4) Replace 'RougeL' with 'Rouge-L' in line 429\n5) Replace 'Model' with 'Dataset' in the head of Table 5\n6) Better to add total/avg results in Table 6, like Table 5\n7) The naming reason for 'Vault'"}}]

Author rebuttal:

Meta-review: The paper under consideration introduces "The Vault," a novel dataset designed for training large language models to comprehend and generate code. It consists of 41 million high-quality code-text pairs extracted from 10 prevalent programming languages, surpassing existing datasets in both quality and scale. The authors employ innovative data cleaning techniques, combining rule-based methods with deep learning classifiers, and open-source the toolkit used for dataset creation and quality assurance. The paper provides detailed experiments showcasing the dataset's advantages and is well-written and easy to follow.


**Reasons to Accept:**
1. The paper presents a significantly larger and higher-quality dataset compared to existing alternatives, with empirical evidence demonstrating its superiority in downstream tasks.
2. The authors' data cleaning methodology is innovative and effectively filters out inconsistent code-text pairs.
3. The open-sourcing of the toolkit for dataset creation aligns with the principles of open research and encourages community engagement.
4. The paper's clarity and comprehensive experiments contribute to the understanding of the new dataset's capabilities.


**Reasons to Reject:**
1. The dataset's limitation to 10 programming languages may restrict its appeal to researchers interested in less prevalent languages.
2. The evaluation of code generation relies on small models, potentially limiting insights into its performance with larger-scale models.
3. The paper's evaluation of the dataset across three tasks using only one model for each task may not provide a holistic assessment of its adaptability across different architectures.
4. The dataset, while high-quality, is smaller in scale compared to some raw datasets, potentially limiting its effectiveness in applications requiring larger volumes of data.

The paper appears to be sound, with reviewers generally agreeing on its methodological rigor and contributions. However, excitement levels are mixed, with some reviewers expressing reservations about the novelty and impact of the work.
Decision: Accept-Findings