The Stack: 3 TB of permissively licensed source code
Authors: Denis Kocetkov
Raymond Li
Loubna Ben allal
Jia LI
Chenghao Mou
Yacine Jernite
Margaret Mitchell
Carlos Muñoz Ferrandis
Sean Hughes
Thomas Wolf
Dzmitry Bahdanau
Leandro Von Werra
Harm de Vries
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) play an ever-increasing role in the field of Artificial Intelligence (AI)--not only for natural language processing but also for code understanding and generation. To stimulate open and responsible research on LLMs for code, we introduce The Stack, a 3.1 TB dataset consisting of permissively licensed source code in 30 programming languages. We describe how we collect the full dataset, construct a permissively licensed subset, present a data governance plan, discuss limitations, and show promising results on text2code benchmarks by training 350M-parameter decoders on different Python subsets. We find that (1) near-deduplicating the data significantly boosts performance across all experiments, and (2) it is possible to match previously reported HumanEval and MBPP performance using only permissively licensed data. We make the dataset available at https://hf.co/BigCode, provide a tool called "Am I in The Stack" for developers to search The Stack for copies of their code (https://hf.co/spaces/bigcode/in-the-stack), and provide a process for code to be removed from the dataset. 

The Stack: 3 TB of permissively licensed source code
Denis Kocetkov* ServiceNow Research
Raymond Li ServiceNow Research
Loubna Ben Allal* Hugging Face
Jia Li Independent Researcher
Chenghao Mou Independent Reseorcher
Carlos Muñoz Ferrandis Hugging Face
Yacine Jernite
Hugging Face
Margaret Mitchell
Hugging Face
Sean Hughes ServiceNow
Thomas Wolf
Hugging Face
Dzmitry Bahdanau ServiceNow Research
Leandro von Werrat
Hugging Face
Harm de Vriest ServiceNow Research
Abstract
Large Language Models (LLMs) play an ever-incrcasing role the field of Artificial In- telligence (AI) not only for natural language processing but also for code understanding and generation. To stimulate open and responsible research on LLMs for code, intro- duce The Stack, 3.1 TB dataset consisting of permissively licensed source code in 30 programming languages. We describe how collect the full dataset, construct per- missively licensed subset, present data governance plan, discuss limitations, and show promising results on text2code benchmarks by training 350M-parameter decoders on dif-
"Corresponding authors (denoted by t) can be contacted priratttbigcodee-rjjjett org

---

ferent Python subsets. We find that (1) near-deduplieating the data significantly boosts performance across all experiments, and (2) it is possible to match previously reported HumanEval and MBPP performance using only permissively licensed data. We make the dataset available at htttp:://hf..o/Biggeee
provide a tool called ' "Am I in The Stack" for developers to search The Stack for copies of their code, and provide a process for code to be removed from the dataset by fol- lowing the instructions at
lhttps://hit ct///pptee//igooodd/iirr-ssssssa
1 Introduction
Large Language Models (LLMs) have emerged as a powerful tool for Natural Language Processing (NLP) (Brown el al., 2020 Bommasani et al. 2021 Zhang el al. 2022 Chowdhery el al. 2022 BigScience
Workshop 2022). These large transformer models are pre-trained on large internet corpora and have shown impressive zero and fow-shot performance
numerous NLP tasks, often by prompting the model with natural language description of the task at hand (Brown el al. 2020). More recently, researchers have started exploring LLMs for coding applications (Chen et al. 2021 Fried et al. 2022 Nijkamp et al., 2022). Code LLMs are trained on large collections of source code and enable the synthesis of programs from both natural language descriptions and other code snippets. Such models can assist professional developers with pro- gramming tasks, for example, by auto-completing code snippets, generating docstrings for a given function signature and body, or suggesting unit tests for codebase.
One of the challenges faced by researchers working code LLMs is the lack of openness and transparency around the development of these systems. While a handful of papers code LLMs has been published, they do not always give full insight into the development process. Some rescarch groups have made their code LLMs available through a paid API service CChen et al. 2021 or a commercial produc Other groups have published the model weights (Nijkamp et al. 2022 Fried et al. 2022 but did not release the training data. It is, therefore, difficult for researchers to fully reproduce these models bocause they first need investigate the nuanced details of the data processing. For example, several groups have reported that (near) deduplication of the training data is an important preprocessing step for both natural language (Kandpal et al. 2022: Hernandez et al., 2022) and source code {Allamanis 2019). Instead of letting research groups independently iterate over such preprocessing details, we argue that the rescarch community would make progress faster if
high-quality pre-training da taseess supported by data cards (Gebru el al. 2021 were more broadly shared.
Bender & Friedman 2018
We believe openness around the pre-training data is also important given the ongoing legal discussions around the use of open-source code repositories for training (commercial) LLMs. Some have argued that machine learning models are derivative work of the training material, and that therefore the resulting models and inferred outputs must comply with the terms of any licenses on the training material. This argument has been made most forcefully by advocates of copyleft licenses (Kuhn) 2022). but has also been made for permissive licenses (Butterick) (022)). Others have taken the position that code LLM developers likely benefit from copyright law exceptions, such as fair use under U.S. copyright law, when using available code repositories for model training purposes (Rothchild & Rothchild (022). But even if regulations currently permit the use of public source code for training ML models, it worth discussing whether these practices align with the ethical values of society and the broader rescarch community. For example, one could argue that code creators should have the rights to meaningfully control whether their data is included in the training set (Jernite et al. 2022). One could also have ethical concerns around the inclusion of Personally Identifiable Information (PII) and malicious code in the training data, as code LLMs might output such sensitive data (Carlini et al., 2021 Kandpal et al.) 2022) or unsafe programs (Khlaaf et al.] 2022) during deployment. We believe open datasets benefit. from external scrutiny addressing such data issues, as researchers can investigate the dataset and report issues directly to the maintainers.
In this paper, we take step towards open and responsible research code LLMs and release The Stack, large dataset of permissively licensed source code. By releasing a dataset that can be shared, inspected, and
For example, Microsoft's Copilot http:e///gttaa..co///tttssses//p and Amazon's CodeWhisperer (https://aws.
amazon. mmnzoo..omr/codeehssperrr//
2

---

Language
The Stack!
CodeParrot
AlphaCode
CodeGen
PolyCoder
Assembly
2.36
0.78
Batchfile
1.00
0.7
C
222.88
183.83
48.9
55
C#
128.37
36.83
38.4
21
C++
192.84
87.73
290.5
69.9
52
CMake
1.96
0.54
CSS
145.33
22.67
Dockerfile
1.95
0.71
FORTRAN
3.10
1.62
GO
118.37
19.28
19.8
21.4
15
Haskell
6.95
1.85
HTML
746.33
118.12
Java
271.43
107.7
113.8
120.3
41
JavaScript
486.20
87.82
24.7
22
Julia
3.09
0.29
Lua
6.58
2.81
2.9
Makefile
5.09
2.92
Markdown
164.61
23.09
Perl
5.50
4.7
PHP
183.19
61.41
13
PowerShell
3.37
0.69
Python
190.73
52.03
54.3
55.9 (217.3)
16
Ruby
23.82
10.95
11.6
4.1
Rust
40.35
2.68
2.8
3.5
Scala
14.87
3.87
4.1
1.8
Shell
8.69
3.01
SQL
18.15
5.67
TeX
4.65
2.15
TypeScript
131.46
24.59
24.90
9.20
Visual Basic
2.73
1.91
Total
3135.95
872.95
715.1
314.1
253.6
The size of The Stack (in GB) compared to other source code datasets used for pro-training LLMs. indicates the dataset is publicly released. The Stack is more than three times the size of CodeParrot, the next-largest released code dataset.
Table 1:
used for pre-training, we hope to make the LLM development process more reproducible and transparent. This work describes how we collected the data from Github and present evidence that the dataset is a useful resource for developing competitive code LLMs. More specifically, we make the following contributions:
We present The Stack, large dataset with 3.1 TB of permissively licensed source code in 30 programming languages. We release this dataset along with near-deduplicated version at https:
//hf.co/Bigteeee co/iggode
We train 350M decoder-only transformers on several python subsets of the data and find that removing near-duplicates siguificantly boosts performance in all experiments. We show it is possible to reproduce text2code performance of Codex (Chen et al. 2021 and CodeGen (Nijkamp et al. 2022) by only using permissively licensed data. We outperform these models by a large margin if we train on the all-license version of the datasct.
We acknowledge that some developers do not wish their code be used for pre-training LLMs and, therefore, start experimenting with giving developers the possibility to have their data removed from
3

---

the dataset. We present the details of this opt-out process in data governance plan in Section 3.2 We also provide further instructions for removal requests at https///wwwwbbcccoeepjjeeeeee org/ docs/about/the-staa
2
Related Work
Code LLMs A growing body of research has trained large-scale transformer models on source code. Several groups have explored decoder-only models with a causal language modeling objective (Chen et al. 2021 Austin et al. 2021 Nijkamp et al. 2022 Christopoulou ct al. 2022 Izadi et al. 2022; Xu et al.
2022 and generally found that larger models are increasingly capable of synthesizing programs from natural language descriptions. A few studies have used such decoder-only models for code-infilling tasks via a causal masking mechanism (Fried et al.) 2022 Bavarian et al.) 2022). Researchers have also investigated encoder masked language models (Feng et al. 2020 Kanade et al. 2020 and encoder-decoder architectures with
various training objectives (Li et al. 2022
Ahmad et al. 2021 Wang et al. 2021 Roziere et al. 2021
Datascts for pro-training code LLMs GitHub has been froquently used resource for pretraining large- provides snapshot of permissively licensed repositories on GitHub and can be filtered through a SQL query. AlphaCode (Li et al.) 2022). BLOOM (Laurençon et al., 2022 CodeGen (Nijkamp et al., )022). and InCoder (Fried et al. 2022) all included this resource in their pre- training dataset. A snapshot of this GitHub data also publicly available on the HuggingFace hut as the GitHub-Cod Code dataset under the CodeParrot project. PolyCoder (Xu et al. (2022) collected a code dataset by scraping GitHub repositories and released a catalog of their downloaded repositories and files. We compare our work against these code datasets in Table [ and claborate more on these differences in Section 3.3
scale code LLMs. Google BigQuery
33
Another frequently used resource is the CodeSearchNet corpus (Husain et al. 2019). This corpus first collected large number of public and permissive repositories from GitHub, which were then further processed with trecsitten to extract pairs of functions (methods) and docstrings. The dataset contains such pairs for six programming languages: Go, Java, JavaScript, Python, PHP and Ruby. CodeBERT (Feng et al., 2020) and CodeT5 (Wang et al.] 2021) used this dataset for pre-training. CodeNet (Puri et al., [202]]
p is another
large code dataset collected from online programming contest websites. It consists of 14M code snippets and is therefore significantly smaller than The Stack (317M permissively licensed files).
Some studies have reported results by training on non-public datasots. Codex (Chen et al.
2021 | collected 179
GB of python files from 54M public GitHub repositories, but did not release the data nor disclose information regarding the licensing. CodeGen collected a private GitHub dataset of 217.3 GB of permissively licensed python files. They fine-tuned models on this dataset after pre-training on the Pile (Gao et al., 2020) and the GitHub snapshot on BigQuery. While CodeGen open-sourced the model weights, they did not release the private python dataset.
Evaluation of code LLMs Code LLMs are froquently evaluated on text-to-code benchmarks, in which models are lasked with completing code snippets from natural language descriptions (Austin al., 2021; Hendrycks et al. 2021 Chen et al., 2021 Lai et al. 2022). These benchmarks usually measure the execution accuracy of generated code against mumber of unit tests. Two popular text-to-code benchmarks are HumanEval (Chen et al., 2021) and MBPP (Austin et al. 2021). Recently, these benchmarks have been extended from Python to other programming languages (Athiwaratkun et. al. 2022 Cassano et al. 2022). Code LLMs have also been used to solve a variety of other tasks. CodeXGLUE (Lu et al.) 2021) is a set of 14 datascts for evaluating code generation models. The tasks include code-to-code tasks like clone detection, code repair, and code translation; text-to-code tasks like code search and code generation; and code-to-test tasks like generating documentation. CoSet (Wang & Christodorescu
2019 introduced dataset of 84,165.
programs for semantic evaluation of neural program embeddings.
T http:///ugghiifeecc.oo/ssssssssp http://gthaa..cc//nneessttteerre

---

Deduplication Recent studies have shown that deduplicating the training set can significantly improve the performance of LLMs. [eee et al. (021) show that training corpora for language models contain many near- duplicates, and that LLM performance improves when long repetitive substrings are removed. [Hernandez ct al. (002)) also found that repeating even a small portion of the training data can significantly hurt model performance, polentially due to a fraction of its capacity being consumed by data memorization. Data duplication is even more present in code datascts, since it is common practice to reuse and clone code repositories of others. Indeed, Lopes al. (2017) observed that large proportion of GitHub data consists of clones, resulting in high ratio of exact and near-duplicates. [Allamanis (2019) study the effect of code duplication on machine learning models and show that it can result in highly inflated performance metries. However, many existing code LLMs (Nijkamp et al. 2022 Xu et al. 2022 Li et al. 2022) only apply exact deduplication, which leaves a large number of near-duplicales in the training data.
3 Dataset
We describe how we collect the code dataset and creale permissively licensed and near-deduplicated subsets We present a data governance plan in Section 3.2 and further analyze the data in Section 3.3
in Section 3.1
3.1 Dataset Creation
Dataset Collection We first collected set of active GitHub repository names from GHArchive open-source project working archiving and releasing the public GitHub timeline. Note that these archives only contain GitHub events, such as forking a repository and commenting on an issue, and not the code repositories. We extracted unique repository names from the event archives published between Jamuary 1st, 2015 and March 31st, 2022. This resulted in a list of 220.92M unique repository names, Next, we ran a distributed compute cluster for a couple of months to clone this list of repositorics. We did not successfully download all of these repositories because some of them were deleted or made private. In the end, we successfully downloaded 137.36 337.36M repositories, resulting in clone success rate of over 62%. All repositories were downloaded between November 2021 and June 2022.
Note that we do not store binary files as we cannot use this data for pre-training. We provide the exact list of excluded binary file extensions in Appendix C We also do not store files larger than 1 MB except: if the extension is from an approved list of programming language extensions. See Appendix D for the full list. Furthermore, we avoid storing exact file duplicates by using git hashes. All repositories contain 51.76B files, but only 5.28B of them are unique (i.e., slightly more than 10%). The uncompressed size of all stored files is 92.36 TB.
License detection We describe how we extract license information for each repository. GHArrhiie pro- vides license information when the repository owner explicitly sets the code license through the web interface. We find that licenses from GHArchive are available for 26.4M repositories. For the remaining 110.9M repos- itories, we run the go-license-detectooff This detector scans the repository and returns list of predictions with the associated file, the SPDX license identifie and confidence score. We found that the detector did not detect license for more than 80% of the repositories. MIT and Apache 2.0 are the most frequently detected licenses for 9.6% and 2.7% of the total repositorics, respectively. We present the top-20 predicted SPDX identifiers in Table 2
Permissive license dataset We develop a dataset of source code with only permissive licenses, i.c., with minimal restrictions on how the software can be copied, modified, and redistributed. We first provide the list of licenses which classified as permissive in Appendix A. Note that we intentionally exclude copyleft licenses like GPL, as this community has strongly expressed the concern of machine learning models and inferred outputs violating the terms of their licenses Kuhn 2022).
htttss//gharchhive.or
bttps://github.. httpa//.cooooccc/eeeeeee htl.:oo--icenee-eeteee http:://apdx.org/Iicoeees
5

---

SPDX identifier
Number of repos (in M)
Percentage
not found
112.51
81.91
MIT
13.16
9.58
Apache-2.0
3.72
2.71
BSD-3-Clause
0.76
0.55
error
0.58
0.42
GPI-3.0-only
0.55
0.4
GPL-3.0-or-later
0.55
0.4
deprecated GPL-3.0+
0.55
0.4
deprecated GPL-3.0
0.55
0.4
GPL.3.0
0.52
0.38
Unlicense
0.38
0.28
CC0-1.0
0.29
0.21
GP--2.0-or-later
0.28
0.2
depreeatodGGPI-220 GPI-2.0
0.28
0.2
BSD-2-Clause
0.24
0.17
CC-BY-400
0.2
0.15
CC-BY-3.0
0.13
0.1
GPL-2.0
0.11
0.08
MPL-2.0
0.1
0.07
AGPL-3.0
0.09
0.06
Table 2: The top 20 detected licenses in the collected repositories. We took the highest confidence prediction per repository.
After rumning all our experiments, it was brought to our attention that licenses such as MPL, LGPL, and EPL were erroneously labeled as permissive when they are in fact weak copyleft licenses. We have removed these weak copyleft license files from The Stack and will release the updated version to the community. The weak
licensed data is only a small part of the overall dataset (below 0.5% for the Python subset), hence we decided to not rerun experiments as we expect findings to remain unchanged. For the new version of The Stack, we rely on the Blue Oak Counci to classify the set of permissive licenses see Appendix B for the full list and the updated programming language statistics.
One challenge in compiling the permissive license dataset. is that each file might be part of multiple repos- itories. We opt to include file in the permissive license dataset if at least one the repositories containing the file has permissive license. With this procedure, we keep permissively-lienneed files that were copied into non-permissively licensed repositories. However, it is possible that non-permissively licensed files are part of the dataset if a developer erroneously copied a non-pernissively licensed file into permissively licensed repository. We encourage users of the dataset report files that might have been misclassified permissively licensed.
We mark an individual repository permissive by using the list of license predictions as follows. We first take the highest confidence prediction for each separate file that the license detector flags as potential license. We then check whether all predictions are permissive licenses (see the list in Aj. If so, we classify the repository as permissive. Additionally, empty files, files larger than MB, and files that could not decoded are removed from the dataset. Finally, we point out that we give developers the opportunity have their data removed from this dataset by following the instructions at httph:ttp/// htttps://ww...e httpss//www.bbbbgdeerppjjjee bigcbboeeccee.ojecct.. erg/docas httg///c//abbt////e/oo theuut///hssttcck/
Near-deduplication
We apply near-deduplicatior our pre-processing pipeline on top of exact dedupli- cation. We largely follow the implementation of Allamanis (2019) and rely on a MinHash approach Broder
6 httpe://blueakcouuiilor/////
g See ottp:///po///c.ooo.occcccccoeean http::///..ttt......oooooaaaaaan

---

All-licenses
Permissive
Perm. + near-dedup
Language
Size (GB)
Files (M)
Size (GB)
Files (M)
Perc.
Size (GB)
Files (M)
Perc.
Assembly
36.04
1.34
2.36
0.32
23.8%
1.55
0.24
75.0%
Batchfile
31.05
2.82
1.00
0.42
14.9%
0.33
0.28
66.7%
C
1461.23
95.57
222.88
19.88
20.8%
73.21
10.95
55.1%
C#
644.28
105.96
128.37
20.54
19.4%
56.75
12.79
62.3%
C++
1106.54
62.72
192.84
13.54
21.6%
185.60
7.23
53.4%
CMake
11.25
3.59
1.96
0.56
15.8%
0.68
0.25
44.6%
CSS
1040.53
50.47
145.33
5.73
11.4%
35.60
3,54
61.8%
Dockerfile
3.89
3.74
1.95
1.27
32.6%
0.55
0.65
51.2%
FORTRAN
26.67
1.21
3.10
0.24
19.8%
1.77
0.17
70.8%
GO
271.92
23.34
118.37
12.08
51.8%
34.87
6.17
51.1%
Haskell
15.79
2.06
6.95
0.92
44.7%
3.29
0.64
69.6%
HTML
9491.23
267.81
746.33
32.31
12.1%
279.37
15.55
48.8%
Java
1311.99
279.16
271.43
43.01
15.4%
119.19
26.05
60.1%
JavaScript
5820.23
209.51
486.20
39.28
18.7%
220.94
25.22
64.2%
Julia
21.75
0.88
3.09
0.47
53.4%
1.78
0.34
72.3%
Lua
88.39
5.18
6.58
0.81
15.6%
3.60
0.58
71.6%
Makefile
39.36
6.34
5.09
1.09
17.2%
1.69
0.62
56.9%
Markdown
706.8
135.69
164.61
28.97
21.4%
73.09
20.91
72.2%
Perl
49.21
2.74
5.50
0.55
20.1%
2.16
0.31
56.4%
PHP
779.66
115.53
183.19
34.18
29.6%
90.21
22.65
66.3%
PowerShell
13.26
1.39
3.37
0.52
37.4%
1.66
0.33
63.5%
Python
737.89
106.91
190.73
23.58
22.1%
80.38
15.03
63.7%
Ruby
78.63
30.74
23.82
6.39
20.8%
22.39
4.12
64.5%
Rust
78.97
6.3
40.35
3.09
49.0%
13.65
1.73
56.0%
Scala
28.37
6.06
14.87
2.64
43.6%
6.04
1.55
58.7%
Shell
71.56
14.01
8.69
3.66
26.1%
3.98
2.49
68.0%
SQL
1438.73
10.2
18.15
1.27
12.5%
11.47
1.00
78.7%
TeX
69.4
4.01
4.65
0.45
11.2%
3.56
0.38
84.4%
TypeScript
4145.01
75.8
131.46
19.44
25.6%
120.19
12.90
66.4%
Visual Basic
28.57
1.97
2.73
0.2
10.2%
1.20
0.12
60.0%
Total
29648.2
1633.05
3135.95
317.41
19.4%
1450.75
194.79
61.4%
Table 3: An overview of the amount of data we collected for 30 popular programming languages. We show the size and number files for different splits of the data: the all-license, permissive license, and permissive license with deduplicaiia The "Perc." columns show the percentage of files kept after the permissive license selection and the removal of near-duplicates, respectively.
2000 Lee et al. ) to find near-duplicates. We first split the files into words/unigrant /niigrass based on non- alphanumeric characters and remove files with fewer than 10 tokens. Next, we compute the MinHash with 256 permutations of all documents, and use Locality Sensitive Hashing (Har-Poled et al. 2012) to find clus- ters of duplicates. We include another false positive check by ensuring that each file in the original cluster is similar to (Jaccard similarity exceeding 0.85) at least one other file. We find that in the permissive license dataset, 38.6% of the files are near-duplicates of other files and removed, they also represent 53.7% of the volume of the dataset. See Table for breakdown per programming language. We find that CMake, C, C++, and HTML subsets contain many near-duplicaees (45-55%) whereas SQL, Tex, and Assembly contain relatively few ncar-duplicates (15-2%%).
3.2 Data Governance
One of the goals of this project is to give developers agency over their source code and let them decide whether not it can be used to develop and evaluate LLMs, acknowledge that not all developers may

---

wish to have their data used for that purpose. Our first step to that end was to select source code with permissive licenses, i.e. those with minimal restrictions on how the software can be copied, modified and redistributed (see Section 3.1). In addition, we are giving developers the ability to have their code removed from the datasct upon request. The process for submitting and enacting removal requests will keep evolving throughout the project as we receive leedback and build up more data governance tools. The following FAQ Q presents the current state of this process, as well as the planned next steps.
How do I know my data is in The Stack? We have a developed a tool to help users understand whether their data is in The Stack. See https://hf.co/spcceettt
co/ http///cce//iggooeee/---eeeeesee
How can I request that my data be removed from The Stack? In order to request that data from your repositories be removed from The Stack, we ask that you first fill out the following form with your GitHub username and the email address associated with your git activity. After submitting the form, we will invite you to a private repository on the BigCode organization and ask you to open an issue with the topic "remove my Github repositories from The Stack". This will verify your Github username and we will mark all public repositories under your username for removal in the next dataset release cycle. The verification process is manual at the moment but we are looking into ways fully automate it.
What data can I request be removed from The Stack? Currently, you can request that we remove all public repositories under the provided username. In future work, we will expand the scope of data removal requests to address requests at finer gramularity (specific repositories, specific files) and to greater range of contribution types (for example, based on whether a file or repository contains push events associated with your username according to GHArchive).
Can I also prevent my data from being included in future versions of The Stack? The removal request form will be used to validate removal requests and remove appropriate data. Validated requests and associated code pointers will also be stored in order to ensure that the code does not appear in future versions of The Stack.
What happens to my data once I have requested its removal? For as long as we are maintaining The Stack dataset, we will provide regular updates to the dataset to remove data that has been flagged since the last version. The current plan is to update the dataset every 3 months, although the schedule may change based on the volume of requests received. If we are not in a position to continue maintaining the dataset, we plan to stop distributing it in its current format and update its terms of use to limit its range of applications further, including for training new LLMs. Finally, we require that people who download the dataset agree to use the most recent allowed version in order to incorporate the removal requests.
3.3 Dataset Analysis
To gain more insight into the collected dataset, we first analyze the amount of data per programming language, then compare against other code datasets, and finally present a more extensive analysis on the python subset.
Data per programming language We present the amount of data available for 30 programming lan- guages in Table 3 We see that the all-license dataset contains 1.6B files (29.6 TB data). Only selecting permissively licensed files reduces the dataset to 317M files (3.1 TB), ie.. only roughly 19.4% of the files are kept. For certain programming languages (e.g. CSS, Tex, Visual Basic), we find that less than 12% of the files have a permissive license. We might be able to increase this percentage by adding more licenses to the permissive license list (see Appendix A If we further apply near-dedupliaation to the permissive license dataset, we end up with 194M files (1.4 TB), a reduction of almost 40%. For example, more than half of the files are removed for CMake and HTML. Another important observation is that the the majority of the datasct are coming from a few programming languages, as can be seen in Figure D For the permissive license dataset, the four biggest languages-HHi HTML (746 GB), Javascript (486 GB), Java (271 GB), and C (222 GB) conssume more than 55% of the dataset volume.
8

---

1 TB
100 M
100 GB
10
Number files
Size of files
GB
100
HTML
JavaScript
C++
Pychon
РHР
Markdown
CSS
TypeScript
CW
GO
Rust
Ruby
SQL
Scala
Shell
Haskell
Lua
Peri
Makefile
TeX
PowerShell
FORTRAN
Julia
Visual Bosic
Assembly
CMake
Dockerfile
Datchfille
Programming languages
Figure Histogram of the amount of data per programming language for the permissive license dataset.
Note that we plot the dataset size on a log scale.
10000
1000
Number of files
100
10
English
Chinese
French
Portuguese
Spanish
Russian
German
Koreen
Japanese
Uzbek
Italiaan
Indonesian
Arabic
Catalan
Romanion
Persian
Dutchh
Malayalam
Hungarian
Turkish
Telugu
Esperanto
Greek
Bangla
Latvian
Polish
Galician
Gujarati
Czech
Sundanese
Serbian (Latin)
Georgian
Latin
Swedish
Urdu
Macedonian
Kannada
Inter Interlingua
Cebuano
Mongolian
Languages
Figure Histogram of natural languages in docstrings and comments from 10,000 Python files. Note that plot the number of files log scalc.
Comparison with other code datasets We compare The Stack with CodeParro 1 AlphaCode (022), PolyCoder[Xu al.) (202)) in Table Note that AlphaCode
et al. 2022), CodeGen (Nijkamp and CodeGen did not release their data. but provided statistics the amount of data per programming language. It is also worth mentioning that CodeParrot includes files with copyleft license (c.g. GPL) while our permissive license dalaset does not. PolyCoder does not filter for license information and also

---

likely contains files with copyleft license. While The Stack and CodeParrot provide source code for 30 programming languages, AlphaCode, PolyCoder, and CodeGen only provide data for 12, 12, and 6 of the languages, respectively. Furthermore, we observe that our dataset is more than 3x the size of CodePParrot, the next largest publicly available code dataset. We also see that our dataset is bigger in size than CodeParrot for each individual programming language.
Python subset analysis First, we investigate how many configuration and test files are in the Python subset of the permissive license dataset. These config files are abundant in GitHub repositories but do not capture the complex features of source code. To detect them, we look for mentions of keywords such as "configuration file' and 'test file" in the first lines. If these keywords are not present, see if the occurrence number of either consig or lest literals is higher than 5% of the number of lines in the file. This filter selects 15% of the files which represent 23% of the data by volume.
Next, we estimate the number valid Python files by using the py_compile module on 10,000 samples from our dataset. We find that only 0.7% of the files do not compile due to syntax errors. Specifically, we find that around 0.1% of the Python files had tokenization errors (c.g., due to inconsistent use of tabs and spaces).
Finally, analyze the docstrings and comments in the files. This data is essential for applications such documentation generation and natural language to code translation (Chen et al. 2021 Feng et al.) 2020). We analyze a subset of 10,000 files. We first use the ast and tokenize modules to extract docstrings and comments. We find that they make up 18% of the volume of the subset and that 20% of the files had little (less than 20 characters) to no natural text.
To identify the language of the extracted docstrings and comments, we use model from the faslerl library We find that 94% of the files are in English. Among the other languages, Chinese and French are popular with over 100 samples. We show the full distribution of natural languages in Figure 2 It is important to note that this language detection is imperfect since docstrings often include code examples with English keywords.
4 Experiments
In this work, we investigate the quality of the Python subset of The Stack. Subsequent work by the BigCode community has investigated the the quality of other subsets by training larger Code LLMs that support more
programming languages (Allal et al.) 2023 Li et al. 2023). Here, we train several LLMS on the Python subset
of The Stack and evaluate the models on Human Eval (Chen et al. 2021) and MBPP Austin et al. 2021
We compare against CodeGen (Nijkamp et al., 2022). InCoder (Fried et al., 2022). and Codex (Chen el al. 2021) models of similar size. We discovered late the research process that some of the pretraining sets were contaminated with examples from the evaluation benchmarks. As running experiments is expensive and takes long, we decided to re-run only a few experiments to investigate the impact of the data contamination. In addition, we did not rerun experiments after finding out labelling mistake in the permissive licenses (see Appendix A for full details). The results we present for permissively licensed Python data, therefore, do contain a small fraction (less than 0.5%) of weak copyleft licensed data. However, we think our main findings remain unchanged.
4.1 Experimental setup
Datasets We experiment with 4 versions of our python dataset, as well as the python subset of Code- Parrot. For our datasets we either use python files from all repositories (all-license) or from repositories with permissive licenses (permissfive-lienss))) To this data we either apply near-deduplication (near-dedup) or no further filtering (none). is worth resssng that the near deduplication process on top of the exact deduplication that we applied during the dataset collection. For CodeParrot we only experiment with the near-deduplicatted version 13
15 http:///ee.cccccccc/ooooooo/oe/n
10

---

Dataset
Filter
Human Eval
MBPP
Python-all-license
None
338
1
Near-dedup
291
1
Pythan-perrmisiie--ienne
None
336
0
Near-dedup
292
0
CodeParrot
Near-dodup
0
0
Table 4: Data contamination per training set. We report the number of files in which we find natural language prompt of test example.
For all datascts, follow the filtering methods of Codex Chen et al. 2021 and remove files with:
an average line length greater than 100 characters
a maximum line length above 1,000 characters
less than 25% of the characters being alphamumeric characters
keywords in the first few lines of the file indicating that the file was likely automatically generated
Data contamination After training the models, we found that some of the training sets contained ex- amples from the evaluation benchmarks. We detected this contamination issue by searching for exact string matches of the natural language prompts of HumanEval and MBPP. Table g shows how many contami- nated files were found for each of the subsets. All our subsets contain HumanEval examples but only the python-all-licens: subsets contain MBPP examples. To investigate the impact of the data contamination, we rerun experiments on the near-deduplicated versions of the python-all-lieesse and suuhon-permissivee datasets. To this end, we remove the contaminated files from these datasets. Note that we only eliminate files with exact copics of the prompts and thus do not detect paraphrases.
Evaluation We evaluate models on the HumanEval CChen et al. 2021 and MBBP (Austin et al. 2021 benchmarks. HumanEval a lext2python benchmark containing 164 programming problems. Each problem consists of a function signature, docstring, body, and some test cases that allow to verify the correctness of the generated program. Models are evaluated with the pass@k metric (Chen et al.. 2021): model samples are generated for each problem, and problem is considered solved if at least one of the samples passes all the unit tests. We report the average fraction of problems solved. In practice, sample 200 programs for each problem, and estimate pass@k with the unbiased estimated proposed by CCen et al. (2021) We use nucleus sampling with top-p = 0.95, temperature E {0.2, 0.8) and report the pass@k for the best temperature.
We also evaluate on MBPP (Austin et al. 2021 a tex:2python benchmark consisting of crowdsourced programming problems. Each problem consists of a short natural language description and unit tests. We evaluate the test set of 500 examples. While Austin et al. (2021) include all three unit tests in the prompt, Fried et al. (022)) only include a single unit test because they observed that smaller language models (i.c., with few billion paramcters) did not benefit from more unit tests. We follow Fried et al. (0022) ) and only add the first unit test to the natural language prompt. Besides pass@1, we also evaluate pass@10 and pass@100 100 by sampling 200 programs for each problem. Similarly to HumanEval, we use nucleus sampling with top-p = 0.95, temperature T E {0.2, 0.8) and report the scores for the best temperature.
Training details We experiment with decoder-only transformers trained via a causal language modeling objective. We opt for a 350M parameter model with 24 layers, a hidden dimension of 1024, 16 attention heads, and a sequence length of 2048. The model is trained for 300K iterations with a global batch size of 384 using Adam (Kingma Ba 2015) with 31 = 0.9, B2 = 0.95, 10-8 and a weight docay of 0.1. The learning rate set
10 + warmed up for 175 steps, then follows cosine decay. The model processes 235.9B tokens during training. The Pyie-Pair Encoding tokenizer trained 50-50 mixture of the
11

---

Dataset
Filtering
Pass@1
Pass@10
Pass@l 100
Codex (300M)
13.17
20.17
36.27
CodeGen (350M)
12.76
23.11
35.19
Python all-license
None
13.11
21.77
36.67
Near-dedup
16.60
27.82
44.00
+ Decontamination
17.34
27.64
45.52
Python permissivo-liceso
None
10.99
15.94
27.21
Near-dedup
13.94
22.36
37.00
Decontamination
12.89
22.26
36.01
CodeParrot
Near-dedup
11.23
18.16
30.37
Table 5: HumanE'val performance of 350M model on different training sets.
Model
Filtering
Pass@1
Pass@10
Pass@100
InCodor-1B CodeGen (350M)
9.36
23.37
45.80
14.09
30.07
51.80
Python all-license
None
17.41
33.09
53.59
Ncar-dedup
22.99
39.62
61.00
+ Decontamination
21.82
37.55
58.28
Python permissive-licenss
None
11.60
23.13
44.99
Ncar-dedup
15.94
31.70
54.69
CodeParrot
Near-dedup
6.31
21.50
45.44
Table 6: MBPP performance of a 350M model on different training scts.
Pile
Gao et al. 2020) and Python files from The Stack. We use a fork of Megatron-T.M (Shoeybi et al.
2019 for training.
4.2
Discussion of results
We report the HumanEval and MBPP results in Table and respectively.
Near-deduplication improves performance Applying near-deduplication the training data yields dramatic improvement, on both the all-license and permissive-license datasets. On the permissive-licent dataset, near-deduplication improves HumanEval performance from 27.21% to 37.00% pass@100 and MBPP performance from 44.99% to 54.69% pass@100 100. We see similar boost in results for the all-license dataset, where HumanEval performance improves from 36.67% to 44.00% pass@100 and MBPP performance from 53.59% to 61.00% pass@100.
Reproducing performance with permissive licenses We are able to reproduce text2python results of previous work with only permissively licensed source code. On HumanEval, we observe that, without dear-deduplication, the performance of the permissive license dataset (27.21% pass@100) is significantly be- low Codex (36.27% pass@ 100) and CodeGen(35.193 pass@100) However, we match Codex and CodeGen performance after applying near-dedhuliiattinn (37.00% pass@100). On MBPP, observe similar findings. Without near-deduplication, the performance of the python-permissiv dataset. (44.99% pass@100) is sig- nificantly below CodeGen (51.80% pass@100) However, the near-deduplicated version (54.69% pass 100) surpassos the CodeGen results.
Comparison to CodeParrot We see that training on CodeParrot another released code dataset achieves 30.37% pass@100 on HumanEval, which is significantly below the performance of our released
12

---

dataset (37.00% p0ss@100). On MBPP, CodeParrot also underperforms released dataset (45.44% 54.69% pass@100). CodeParrot consists of data extracted from BigQuery and is not enough to obtain a competitive model on HumanEval and MBPP.
Impact of data contamination Surprisingly, we find that removing contaminated files has very little impact on the text2python results. On HumanEval, there is very small drop for the permissive license dataset (37.00% vs 36.01% pass@100), while there a small gain for the all-license dataset (44.00% 45.52% pass@100). We also see small impact for the all-license dataset. MBPP (61.00% 58.28% pass@100). We speculate that the impact of data contamination minimal because (1) small models are unlikely to memorize training data and (2) there were few contaminated files.
5 Conclusion and Future Work
We introduce The Stack, large dataset of more than 3 TB of permissively licensed source code. This paper described the details of the dataset collection, presented a brief dataset analysis, and showed promising results on the HumanEval benchmark. Our experimental results show that near-deduplication is an important pre-processing step for achioving competitive results on text2code benchmarks. We release all permissively licensed files for 30 common programming languages, along with near-deduplicated version. In future work, we would like further improve the released dataset. We are open to releasing data of other programming languages, plan work on methods for removing PII and malicious code, and start experimenting with giving developers the possibility to have their data removed from the dataset. We hope The Stack will be useful resource for open and responsible research on Code LLMs.
Ackuowledgement We thank Christopher Akiki, Evgenii Zheltonozhskii, Sebastien Paquet, Torsten Scholak, and Luis Villa. for their feedback on an early draft this paper. We also thank Fanny Ran- court and Masoud Hashemi for help with the data cards. Lastly, we are grateful to ServiceNow and Hugging Face for the provided compute resources.
13

---

6 Limitations
Social Impact of the Dataset The Stack is an output of the BigCode Project 15 responsible by design and by default. The project conducted in the spirit of Open Science, focused on the responsible development of LLMs for code.
BigCode aims to be
With the release of The Stack, aim to increase access, reproducibility, and transparency of code LLMs in the research community. Work to de-risk and improve on the implementation of ethical best practices of code LLMs is conducted in various BigCode working groups. The Legal, Ethics, and Governance working group has explored topics such as licensing (including copyleft and the intended use of permissively licensed code), attribution of generated code original code, rights to restrict processing, the inclusion of Personally Identifiable Information (PIT), and risks of malicious code, among other topics. This work is ongoing as of October 25th, 2022.
We expect code LLMs to enable people from diverse backgrounds to write higher quality code and develop low-code applications. Mission-critical software could become casier to maintain as profcssional developers are guided by code-generating systems on how to wrile more robust and efficient code. While the social impact intended to be positive, the increased accessibility of code LLMs comes with certain risks such as over-reliance on the generated code and long-term effects on the software development job market.
We refer the reader to Section 7 of Chen al. 20021) for a broader impact analysis of Code LLMs, as well as Khlaaf et al. 2002 for an in-depth risk assessment and hazard analysis of this emerging technology.
Biases of the Datasct The code collected from GitHub does not contain demographic information proxy information about the demographics. However, it is not without risks, as the comments within the code may contain harmful or offensive language, which could be learned from the models.
Widely adopted programming languages like and Javascript are overrepresented compared to niche pro- gramming languages like Julia and Scala. We found that some programming languages such as SQL, Batch- file, TypeScript are less likely to be permissively licensed (4% vs the average 10%). This may result. in biased representation of those languages. Permissively licensed files also tend to be longer.
We also found that the English language is over represented in the docstrings and comments, as it makes up 96% of the data for Python files.
HTML not WCAG-compliant One of the current limitations of The Stack is that scraped HTML for websites may not be compliant with Web Content Accessibility Guidelines (WCAG). This could have an impact on HTML-generated code that may introduce web accessibility issues.
Personally Identifiable Information We noted that Personally Identifiable Information (PII) such names and email addresses are contained in the data. This PII is already exposed to the public on GitHub, however, we do plan to remove PII in future work.
Malicious code In the context of cyber-sccurity, there is risk that large language models trained on datasets containing ransomware, malware, or other malicious code, could be used to build harmful applica- tions. The ability to spawn new harmful applications by prompting the model using known indicators of compromise (1OCs) could reduce the experience needed for hackers learn these techniques [T and increase the risk of aaansomaar--ssaa-Srveeee general public domain by cilizen-hackers and activists. While more advantaged companies and communilies may have resources to manage these new threats, the negative social impact of ransomware on disadvan- taged communities and society in general, along with the potential legal consequences for companies that pay ransom to bad actors, is something that requires more consideration.
kits being developed and distributed, not the Dark Web, but in the
httpss:/www.bbooodeaaee 16
article hhtpp:://wwe htt.e//www.....ceoeee//////eeeee hewpeee-eepoeeeeeeppiaee hackers training. inneecrr-nn-tttininnn...n.
14

---

A handful of repositories were removed because they triggered an internal security tool during the down- loading phase. However, we did not fully scan the dataset for malware and, as such, warn future users of the potential for malicious code bases in The Stack. Please report your findings of malicious code to contact@bigcodeprrojec.. org so that it can be removed in a future release.
Data licensing While we did our best to filter for permissively licensed source code, it is possible that the license detector incorrectly classified number of repositories. Please reach out contact bigcode-project... org org in case you encounter files that do not have a permissive license. We also stress that The Stack is compilation of source ilesuincluding attribution and license notices-in the form of a dataset. Each source file in The Stack carries its own permissive license which should be respected by users of the dataset.
Model limitations We show promising text2code results for smaller models on the python dataset. More research is necessary to find out whether strong results can be obtained for larger models and other pro- gramming languages than Python.
References
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Chang. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Techoologies, pp. 2655 2668, Online, June 2021. Association for Computational Linguistics. TRLhhttp:://www htttpp::/www.aallee..orogg. wtw.://borrgg/antoooo////////// naacl-main 211
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferran- dis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoclkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, lan Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, and Leandro von Werra. Santacoder: don't reach for the stars!, 2023.
Miltiadis Allamanis. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2019, pp. 143 153, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450369954. doi: 10.1145/3359591.B8 33359735.
URL
https://bbi.orr/10..11 0ttr://1..11///33555999 3359735
Bon Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang. Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ra- manathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. Muulii-liguut evaluation of code generation models, 2022. URL https: //arxi/ wow. ats//bbb/220.. 14868
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108. 07732, 2021.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle, 2022. URL httpp://arrxv..org
abs/2207 14255
Emily M. Bender and Batya Friedman. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 2018. doi: 10.1162/tacl 00041. URL hutp:://accaattboooo.o//////1
15

---

BigScience Workshop. BLOOM (revision 4ab0472), 2022. URL https://aagghggat...///gggeeee bloom
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannetle Bohgg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dal- las Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Good- man, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas leard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshie Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. On the opportunities and risks of foundation models. CoRR, abs/2108.07258, 2021. URL https: / /arxivv org/abs//106 2ttp//1218.008888888
Andrei Z Broder. Identifying and filtering near dupl-ctp documents. In Annual sympostum on combinatorial pattern matching, pp. 1-. Springer, 2000.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arViv preprint ar 2rXx:2005.14165, 2020.
Matthew Butterick. This CoPilot is stupid and wants to kill me. httper//aathheeuuuttrrr.....oo.o this-coppllo--i-sstpp--- to-tr---st--ppdddd--a-a--------- html 2022.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In USENIX Security Symposium, 2021. URL
org/aba/ aba/
2012.07805
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. A scalable and extensible approach benchmarking nl2code for 18 programming languages, 2022. URL https://arxi:: org.orgggaa......o..o htsp//0bb/2222...a/// 088227
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Fe- lipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jic Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyaashchnko,, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thamumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Hells-eel, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR, abs/2204.00 /222222442211,, 2022. doi: 10.48550 Xiv. 220222222211111 URL httpp::////a http////..or///..////// 18850//axXa a2X44.
02311
16

---

Fenia Christopoulon, Gerasimos Lampouras, Milan Gritta, Guchun Zhang, Yinpeng Guo, Zhongqi Li, Qi Zhang, Meng Xiao, Bo Shen, Lin Li, Hao Yu, Li Yan, Pingyi Zhou, Xin Wang, Yuchi Ma, Ig- nacio lacobacci, Yasheng Wang, Guangtai Liang, Jiansheng Wei, Xin Jiang, Qianxiang Wang, and Qun Liu. Pangu-coder: Program synthesis with function-level language modcling, 2022. URL https:
//arxiv.org org wwtpiiv.rrg/bbb//20000 http//bb/222.111//
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural lan- guages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1536-1547, On- line, November 2020. Association for Computational Linguistics. doi: (1/6653/11/220.ffi 139. URL httph: ttttps://aclamthollogg. org/ 222022020.22 findings--mmll. 139
Danicl Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis, 2022. URL htttps://arxiv wttp:/i.i..rrrvoooooo.oo abs/ 2abbs2200
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling, 2020.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):
86 96-92,25 2021.
Sariel Har-Peled, Piotr Indyk, and Rajeev Motwani. Approximate nearest neighbor: Towards removing the curse of dimensionality. 2012.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. Xix preprint ar59w:2105.09938, 2021. doi: 1.48650/AARXVV.21000000 URL https: //arxiv org/ abs/ 219339990888888
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiu preprint ar Xiv:2200. 10487, 2022.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Mare Brockschmidt. CodeSearchhee challenge: Evaluating the state of semantic code search. arXiv preprint arXic: 09436, 2019.
Maliheh Izadi, Roberta Gismondi, and Georgios Gousios. Codefill: Multi-token code completion by jointly learning from structure and naming sequences. In Proceedings of the 44th International Conference on Software Engineering, ICSE '22, pp. 401 412, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450892211. doi: 10.1145/551000..500011 3510003 3510172
URL https: /ttp:::///oooor/11..... 1145/
Yacine Jernite, Huu Nguyen, Stella Biderman, Anna Rogers, Maraim Masoud, Valentin Danchev, Samson Tan, Alexandra Sasha Luccioni, Nishant Subramani, Isaac Johnson, Gerard Dupont, Jesse Dodge, Kyle Lo, Zecrak Talat, Dragomir Radev, Aaron Gokaslan, Somaich Nikpoor, Peter Henderson, Rishi Bommasani, and Margaret Mitchell. Data governance in the age of large-scale data-driven language technology. In 2022 ACM Conference on Fuirness, Accountabilily, and Transparency, FAceT '22, pp. 2206 2222, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393522. doi: 10.1145/3531144 3534637. URL http:::://ooooo/////// 1145/3531146 35346377
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In Proceedings of the 37th International Conference on Machine Learning, ICML/20. JMLR.org, 2020.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. arXiv preprint ar arXi::2202.0655 2022.
17

---

Heidy Khlaaf, Pamela Mishkin, Joshua Achiam, Gretchen Krueger, and Miles Brundage. A hazard analysis framework for code synthesis large language models. aiv preprinl arXiv:2207.14 14177, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Bradley M. Kuhn. II Software is My Copilot, Who Programmed My Software? https:://sfoonservant 2022.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211.JJ 111501, 2022.
Hugo Laurengon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Saško, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Scbastian Nagel, Leon Weber, Manuel Romero Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Vu Minh Chien, Itziar Gonzalez- Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifcoluwa Adelani, Long Phan, Hicu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violctte Lepercq, Suzana Ilie, Margaret Mitchell, Sasha Luccioni, and Yacine Jernite. The bigscience ROOTS corpus: A 1.6TB composite multilingual dataset. In Thirty-sinth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreviee: htwp:/pprrreiiiiieeeeorrrrrrraaa net/formmmtt UoEw6KigœUn
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499) 2021.
Raymond Li, Loubna Ben Allal, Yanglian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehacne, Mishig Davandorj, Joel Lamy-Poirier, João Monteiro, Olch Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Ben- jamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Davan-Gavitt, Dan- ish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Fre- itas, Koray Kavukeuoglu, and Oriol Vinyals. Competition-leve code generation with alphacode. arXiv preprint 2rx::22003.0 07814, 2022.
Cristina V Lopes, Petr Maj, Pedro Martins, Vaibhay Saini, Di Yang, Jakub Zitny, Hitesh Sajuani, and Jan Vitck. Déjåvu: a map of code duplicates on github. Proceedings of the ACM on Programming Languages, 1(0OPSLA):1 28, 2017.
Shuai Lu, Daya Guo, Shuo Ren, Junjic Huang, Alexcy Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. ar Xie preprint or arxix:102. 04664, 2021.
18

---

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. ar preprint, 2022.
Ruchir Puri, David Kung, Geert Janssen, Wei Zhang, Giacomo Domenicomi, Vladimir Zolotov, Julian T Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika. Thost, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Uhrich Finkler, Susan Malaika, and Froderick Reiss. Codenet: A large- scale ai for code dataset for learning a diversity of coding tasks. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL ott.reoo0ooofaaaaaaaac4444444443
bencharkss pdf
neurips www..ps...ppppparree.ae /c/pappe//22///i1le/
John A. Rothchild and Daniel Rothchild. Copyright Implications of the Use of Code Repos- itories to Train Machine Learning Model.
fsf htt.://....r//toooonnoo/oooo////
of- Sngriighh-imlicationss ofoe::ooo-----t-eeee-ee------eee repcsdddrrpstsiiiieeeeeeeee to- www.ttraann-nneeeeee machinem Lew.ahhhee----nnnng--- 2022.
Baptiste Roziere, Marie- Anne Lachaux, Mare Szafraniec, and Guillaume Lample. Dobf: A deobfuscation pre-training objective for programming languages. arXiv preprint arXim:2102.07402 2021.
Mohammad Shocybi, Mostofa. Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megairon-Im: Training multi-billion parameter language models using model parallelism. orXiv preprinl. arXiv:1909.0800 08053, 2019.
Ke Wang and Mihai Christodorescu. Coset: benchmark for evaluating neural program embeddings. arXiv preprint ar Xiv: 1 14445, 2019.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. Code15: identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Melhods in Natural Language Processing, pp. 8696 8708, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10. 18.13/611 /2021 2tt///22111mmppp/ -emnlp-main. 685. URL
org.ollooy..og////. org/2021. emnlp-maine 685
Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, MAPS 2022, pp. 1-10, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392730.. doi: 10.1155/5520312.3554866
URL https:://di..rrg// 1145/
0820312.3534855
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chon, Christopher Dowan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
A Permissive Licenses
For the experiments presented in the paper, we the following SPDX identifiers for our permissive license dataset:
MIT-0
MIT
MIT-feh
Apache-2.0
BSD-3-Clause
19

---

BSDD-2-Cluuse
CC0-1.0
EPT-1.0
MPT-2.0
Unlicense
ISC
Artistic-2.0
deprecated 1Pprc-ated__GPL-33.0 LGPL-3.0+
deprecated LGPL-2.1+
ECL-2.0
SHL-0.51
MPL-2.0-no-coppp -o-pppl--oooppleeffeeeeppooooo
After rumning all experiments, it was brought to our attention that licenses such as MPL. LGPL, and EPL. were erroneously labeled as permissive when they are in fact weak copyleft licenses. We have removed these weak copyleft license files and will release the updated version The Stack. The weak copyleft- licensed data only a small part of the overall dataset (below 0.5% for the Python subset), hence we expect the experimental findings of the paper to remain unchanged. In the next section, describe how updated The Stack.
B
The Stack v1.1
For the Stack v1.1, we rely the Blue Oak Counci to classify the licenses. The new classification process results in 193 permissive licenses (which we list, for completeness, at the end of this section). The Stack v1.1 also includes more programming languages. We used the following list of programming language extensions http::://istt.iitiiie
and obtained data for 370 programming languages. We show the updated statistics for the 30 popular programming languages in Table In general, we find that the amount of data increased for most programming languages. Specifically, we see a large increase in data for C# (128.37 GB 215.07 GB), Tex (4.65 GB vs 8.19 GB), and Markdown (164.61 GB 245.26 GB). On the other hand, GB) and Rust (40.35 GB vs 39.85 GB). We release this updated version to the research community.
minor decrease in data for Go (118.37 GB 112.86
MIT
Apacho-2.0
BSD-3-Clause
Unlicense
CC0-1.0
BSD-2-Clause
httpe::/bbleeacccmcil..o/////
20

---

All-licenses
Permissive
Perm. + near-dedup
Language
Size (GB)
Files (M)
Size (GB)
Files (M)
Size (GB)
Files (M)
Assembly
36.04
1.34
2.57
0.36
1.65
0.26
Batchfile
31.05
2.82
1.06
0.44
0.33
0.28
C
1461.23
95.57
255.29
21.38
75.93
11.21
C++
644.28
105.96
215.07
14.82
65.97
7.60
C#
1106.54
62.72
133.55
21.7
57.98
13.28
CMake
11.25
3.59
2.1
0.59
0.68
0.25
CSS
1040.53
50.47
150.2
5.89
34.86
3.59
Dockerfile
3.89
3.74
1.9
1.27
0.52
0.64
FORTRAN
26.67
1.21
3.81
0.29
2.08
0.19
GO
271.92
23.34
112.86
11.65
32.01
5.89
Haskell
15.79
2.06
5.85
0.8
2.75
0.58
HTML
9491.23
267.81
812.73
35.6
291.46
16.60
Java
1311.99
279.16
266.41
42.43
112.82
25.12
Javascript
5820.23
209.51
496.22
40.11
166.24
25.43
Julia
21.75
0.88
3.4
0.48
1.75
0.33
Lua
88.39
5,18
7.09
0.93
3.77
0.64
Makefile
39.36
6.34
6.88
1.48
2.14
0.80
Markdown
706.8
135.69
245.26
40.75
95.84
25.66
Perl
49.21
2.74
7.08
0.83
2.99
0.48
PHP
779.66
115.53
185.79
34.85
89.46
22.63
PowerShell
13.26
1.39
3.42
0.53
1.65
0.33
Python
737.89
106.91
200.93
24.21
80.13
15.15
Ruby
78.63
30.74
25.95
7.21
9.78
4.46
Rust
78.97
6.3
39.85
3.06
12.92
1.68
Scala
28.37
6.06
15.56
2.79
6.06
1.61
Shell
71.56
14.01
9.07
3.77
4.07
2.54
SQL
1438.73
10.2
19.94
1.39
12.68
1.07
TeX
69.4
4.01
8.19
0.71
5.86
0.59
Typescript
4145.01
75.8
131.01
19.59
36.61
12.82
Visual Basic
28.57
1.97
3.81
0.4
1.71
0.19
Total
29648.2
1633.05
3372.85
340.31
1212.7
201.90
Table 7: An overview of the amount of data we collected for The Stack v1.1. We show the size and number of files for different splits of the data: the all-license, permissive license, and permissive license with near- deduplication.
CC-BY-4.0
CC-BY-3.0
OBSD
RSA-MD
WTFPL
MIT-0
ISC
ADST.
BSI-1.0
21

---

Zlib
2rtistic-2.0
FTL
MS-PL.
FSFAP
BSD-Source-Code
Apacho-1.1
BSD-4-Clause
Ruby
Artistic-1.0
MulanPSI.-1.0
BSD-1-Clause
X11
CNRI-Python
Becrware
Condor-1.1
PostgreSQL
CECILL-B
Intel
Vim
Naumen
OML
BSD-3-Clause-Clen
AML
PHP-3.01
OpenSSL
PSF-2.0
Xnet
Linux-OpenIB
LSD---CCluue-LLBNL
UPL-1.0
AFL-3.0
22

---

BlucOak-1.0.0
Info-ZIP
BSD-4-Clause-UC
AAL
LPPL-1.3c
bxip2-1.0.6
W3C
W3C-20150513
AFL-1.1
DOC
ICU
CC-BY-2.0
curl
MTLL
OLDAP-2.2.1
ECL-2.0
Adobe-Glyph
Ceal-PyytoonGPL--Cmmpiiii
BSD---Clause-Pateet
IJG
PHP-3.0
ZPL-2.1
MIT-advertising
NCSA
Fair
BSD-3-CUe alau--llltueeetbbtttitn
OLDAP-2.3
NLPL
SED-3-Clause-Open-MM
ClArtistic
Python-2.0
NASA-1.3
TCL.
23

---

Artistic-L.0-Perl
blessing
ImageMagick
Net-SNMP
Artistic-1.0-cl8
OLDAP-2.5
MIT-feh
OLDAP-2.4
MITNFA
AFL-2.1
1ibpng-2.0
EFL-2.0
OLDAP-2.7
IBM-pibs
libtiff
OLDAP-2.8
Cube
Adobe-2006
RSD-2-Clauss-NeLBSDD
zlib-acknowledgement
OLDAP-2.6
OLDAP-14
Libpng
MIT-CMU
AFL-2.0
JasPer-2.0
LPL-1.02
Zend-2.0
TCP-wrappers
XFree86-11 1.1
FSFUT
24

---

OLDAP-13
SGI-B-2.0
NetCDF
CNRI-Jython
Zed
ZPT.2.0
AFL-1.2
Apache-1.0
CC-BY-1.0
OLDAP-2.1
OLDAP-1.2
OLDAP-20
NTP
LPL-1.0
AMPAS
Barr
mpich2
ANTLR-PD
Xerox
Spencer-94
AMDPLPA
HPND
ECL-1.0
MirOS
Qhuil
ZPL-1.1
TU-Berlin-2.0
Spencer-86
SMLNJ
xinetd
OLDAP-222.2
OGTSL
25

---

MIT-enna
Font-exception-2.. 2.0
FSFULLR
TU-Berli-1.0
xpp
NRL
W3C-19980720
EFL-1.0
eGenix
Mnicode-DFS-200
SWT
Spencer-99
Plexus
VSL-1.0
Leptonica
Unicodc-DES-2015
Mup
Giftware
OLDAP-2.2
APAFMI.
NBPL-1.0
OLDAP-1.1
Entessa
Multics
Newsletr
psutils
bzip2-1.0.5
Afmparse
diffmark
SSD-2-Clauso-Views
DSDP
MIT-Modern-Varte Variant
ANTIRR-DD-fallacc
26

---

Bahyph
BSD-3-Claus-Moodiicttinn
BSDD-4Clause-Shorttene
HTMLTIDY
MTT-open-group
MulanPSL-2.00
2.D1
Saxpath
Borceux
Crossword
CrystalStacker
Rdise
Wsuipa
C
Excluded file extensions
Part of this list was taken from http:///hhhhh..ooooieetettttttte Bagoocacasccaaaeeeeooooffeteeeea
'apk', 'app', "bin', 'bmp', 'bz2', 'class', 'osv', 'dat', 'db', 'deb', 'dll., 'dylib,, 'egg', 'eot', 'exe', 'gif", 'gitignore', "glif", 'gaadle', 'gz', 'ico', 'jar', 'jpeg', 'ipg', 'lib', 'lo", 'lock', 'log', 'mp3', 'mp4', 'nar', 'o,, 'ogg'. 'otf', 'p', "pdb', "pdf", 'png', 'pickle', 'pkl', 'ppt', 'pptx', 'pyc', 'pyd', 'pyo', 'rar', 'rkt', 'so', 'ss", 'svg', 'tar', 'tif, 'tiff', ttsv,, 'ttf,, 'war', 'wav', 'webm', 'woff", 'woff2', 'xz', 'zip', 'zst'
Included programming language extensions
This list of programming language extensions is taken from https://gist..iiim cow www.whww..coo/pppsccy....kk (3922166672220511774444477848
abap aasc .ash ammpl mmdd g4 appi appl .dyalog asp asax asex .ashx .asmx .aspx .axd ,dats .hats .sats as adb ada ads agda als apacheconf whost .els applescript scpt are ino asciidoc adoc aase .aj aasm .a51 .ine nasm aug ahk ahkl au33 aawk auk gaak .mawk .awk bat .end .befunge bison .bb .bb ,decls .mxx .bs boo .b .f brs .bro .c .cats Jh idc .w .cs ccake .cshmll .osx .opp .c++ .cp .cxx .h .h++ .hh .hpp .hxx line inl ipp tec tpp .c-bjduum chs .epp .make .maae..i .cob .ebl .cep .cobol .epy .oss .csv .capup .mss .ceylon chpl .ch ok .cirru .ll icl .del .click .clj boot .c12 .clic .cljs .cjs.hl .eljsen .ciix .iic .coffee coffee cake .cjsx .son iced .efm .ffml cefc lisp asd .el .1 Isp ny ppodsl .sexp .ep .cps .el .coq x .cpoobjdump .ct +-bj-uuu tobjdump cpp-objdump .cxx-objdump .croole .er ffeature ,eu .cuh .cy -pyx .pxd .pxi .d .di .d-objdump .com .dm zone arpa ,d darcspatch .dpatch ,dart .diff' .patch .dockerfile .djs .dylan .dyl intr .iid .E .ecl .eclmml eel sch brd .epj fsx fx flux .90 .f f.3 f08 .777 .955 for fpp factor fy fancypack fan fs for .eam.fs .tth ,4th ff for forth fr frt fs ftl fr g gco .godd gmss .g gap .gd gi .tst .ms .gd glsl .p .frag frg fs fish fshader .goo ggom .lssl gshader shader vvert .vx .vsh vshader gmm kid .ebuild ,eclass .p .pol glf -gp gnu gnupllt -llot -plt .o golo gs gst -gxx vark grace gradle of gml graphql .dot -g man .1 lin .m .Ix . 3. .in .3m .qt .3x .4 4.5 6.7 8 .me .ms .mo roff groovy grt gtpl -gyy ggpp .hcl .ff hhsl fx .fxh hhllli .htll .htm .ttmlhl ince xht xhtml mustache jinja ,ex .erb .erbddeface .phmml .http .hh -hhp .aam hefac hhadleebars hbs .bb hss hscc hi .xsl by .f pro ddl ipf ini .cgg prefs ppro properties iircoo weechatlog .idr .idd .ni .7x
,exs .elm .el .emacs .ww.e...eeeeee.......... desktop .em emberrcript .erl eeccrpp hrl xrl .yrl .s fsi
27

---

iiss io iik thy ijs flex jflex jsson geojson .lock topojson jsoon .jsonld .jq .jade -j -java jsp js jjs bones les .es6. frag -g8 jake jjsb jsccad jisfl jjsm jss .njs -pac .js .sjs .sublime-build .sulline-commands ssublime- completions keymaa settings sublime-thene
.sublime-projeee projecct sublime-- .sublime Doelimme_mmtrissssee metrics sublime_session xxjss .xsjslib jll ipynb .rl sch .brd _kicad peb .kit .kt .ktm .kts .Ife .Il .lol .ssl .slp .lvproj .lasso las lass088 .lasso9 .ldml .latte .eaa .llann Jess .1 .ex .ly Jily .b am .Id .lds modd Jiquid lagda .litcoffee .lhs .ls ._Is Is xxm x xx .lgt Jogtalk .lookml .ss .lua Segi .nse .pd lua .rbss .wlua .mmmps .m .m4 .m4 .ms .ner .mtml .muff an .nak .d .mk .nkfile mako .aoo .nd markdown .nkd .nknn .nkdoon .ron .mask mathematica .edf . .m .nt .nb .ubp ww wlt matlab .m .maxpat maxhellp maxproj .mxt .pat .mediawiki wwiki .m .moo .eetal miindd .druby .duby .iir .mirah .mo .nod .mms ,mmk monkey .moo .noon .myt .ncl .nl .nsi .nsh in axs axi axs.erb .axierb .llgo .ul Jisp Isp _ngincoonf .hhost .nim .uimrod .ninja .nit .nix .nu .numpy .numpyw .nunse .ml .eliom eliomi ml44 mml .mll mll .ojduump m .h mm j .sj .omgrofl opaa oppal .cl .ppencl .p .cls scadd org ox .ohh .xxo .xygenne .z2 .pwn .inc -php .aw .tp fegi .inc -php3 phhp4 .php5 .phps -phpt -pls .pck .pkb .pks -plb -plsql sql ssql -pov .ine pan .psc -paroot .pasm -pir -pas .dfm .dr .inc .pr .pp -pl aal .egi .sogi -perl -ph pix .mm pod psgi it 66p 6pm .qp .66 .p61 .pmm .pl .l66 .pm .mm6 . .pkl I .pig pike .mmod .pod .pgo pony .ps .eps .ssl .psd1 psmml .dde ppl .pro .prolog yyap .ppin proooo .asc .ubb .pp .dd .pb .pbi .purs .py bzl .egi fegi .gVp .mi pydee .yp pyt -pyw .pyy .tac wsgi xpy -yybb .qml .qbs .pro ppr ir ard asx rraml rdoc .rbbas .rbfrm .bmnu .rbres rbtbarr rbuistate .hhmml .mnd .rkt .rkid .rttl serbl .rl .raw .reb .r .22 .3 rebol red reds .ww .py .ss rsh robot rg .Ib buulder fegi gemnspec god iibre jbuilder .mspec ppuginspec podspec rabl .raee rbuild .bw .bx .ru rubyy .thor wwatchr .rs .rs.in .as sees .mmt2 .smt .spargl .rq sqf haf sql eqql .ddl inne prre .tab uuf viw ssl .d22 .ston svg sage .sagews sls sass scala .sbt isc .scaml .sm sld .sss sps .s .cc see ,tst self sh bash bbts .egi .command fegi .ssh .sh.in .tmux .tool .2sh ss--esssion shen sl .slm .snali .st .cs .tpl .s .ine .sma .nut .tan .ML fun sig smml .do .do .doh .ihlp .mata maathh sthlpp .styl se sedd .swift .sv .sh .h .tom .xx .tc adp .m ,tssh .csh .tex .aux .bbx .bib .cbx .cls .dtx ins .bx .Itx .nkii ankiv .mkvi .sty .toc .taa .t .txt frr .nb .ncl no textile .hhrift .t .tu .ttl .twig its itxx upc anim .asset .mat .meta preffab Junity .uno .uc .ur .urs .vel .vhdl .yhd vvhf .vhi .hho whs whht .vhw vala vapi .v .vo wii .vb .ba .cls frm .frx .vba .vbhtml .vbs vvolt .vue .owl .webidl .x10 xxe .xmil .ant axml .ccmml .elixml .projeect .esl .ssproj .ct ,dita .ditamap .ditaval .dll.config .dotsettings .fltters .fsproj fxmm glade -mml -rxmll .iml .ivy -jelly jjsproj kml launch .ndpolicy .mm .mod .mxml .nproj .nuspec .odd .osm plist pluginspec .props .psxmml .psc1 .pt rdf .ass .scmml .rddf .storyboard .stTheme .sublime-snipper .targets AnCommand .uml .tmm .imaangagg .tmrrfeerence .tmSniipet. .tmTheme .ts .ax .ui .urdf .ux .vbproj .veppro .sssetting .xml .wsdl wws wxi .wxl wxs x3dd xacro xaaml xii xff xliff xmi .iml.dist .xproj xsdd xxul zcml xxsp-config xsp.metadata xxpl .xproc xqquery .xq xxi xxqm .xqy xs xslt xsl xojo__ode -xojo menu -xojo report xojo script xxjoo toolbar .xjoo window .xtend .yml .reek .riiz sublime-syntax ssyntax yaml yyml-tmmaangugee -yang -y yyace -yy zzep zimpl .zmpl .2pl ddesttop ddesttop.n .cc ,ch .edn fish an ne .000 .sst rest .xtt rst.txt wiis -prg .ch .pww
molline-macco ,sublime-menu sublime-mousenae
sublime-workspen
28


Reviews:
[{"review_id": "Po0a0gaDaa", "reviewer": "Reviewer_tEET", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Stack has the potential to have a transformative impact on LLM for code by becoming the standard dataset for developing LLM for code models. In general, better datasets tend to lead to breakthroughs in AI. \n\n+ The paper is in general well-written, easy to read, and accessible to a broad community. \n\n+ The evaluation results on the Python language tasks show the promise of the proposed dataset.\n\n- From a technical perspective, I did not see any particular algorithmic challenges solved here. The presentation in Section 3 describes relatively low-level implementation details (maybe there could have been some challenges in de-duplication but the authors rely on existing methods). \n\n- Given that Stack supports code in 30 languages, I find it underwhelming that the authors chose to evaluate the quality of models trained with their dataset on only Python. Is there any good reason why other languages are not considered? Having these experiments will help the readers better judge the suitability of the proposed dataset for a variety of tasks. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a new dataset called \"The Stack\" to enable the learning of large language models (LLMs) for code. The dataset contains code written in 30 diverse programming languages. Overall, the dataset is larger than the existing ones. The authors implement near de-duplication in their collected dataset and describe methods to remove licensed code as much as possible. They also describe a mechanism through which developers can pull their data out of the stack. The authors evaluate the usefulness of their dataset on two challenging tasks: HumanEval and MBPP. They compare the performance of the trained models against models trained with state-of-the-art datasets. The results show that models trained with the stack can achieve better or comparable performance than existing methods.   ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Given that Python represents a small subset of all the collected code, I would recommend the authors show the suitability of the proposed dataset in other languages where they have more data as shown in Figure 1 (e.g., Javascript, HTML, or Java).\n\n* I did not get the definition of the pass@k metric completely. You mention \"k samples are generated for each problem, a problem is solved if at least one of the samples passes the test cases, and the total fraction of problem solved is measured\", what is the meaning of samples here? input output example from the generated program? also as per the provided definition for pass@1, even if one sample out of 100 passes the test cases, the program is correct, isn't it wrong?\n\n* In the paragraph on \"How can I request that my data be removed from The Stack\", what are the criteria you are going to apply for processing a user request? Only username, email address, and git activity need to be provided? How will you prevent developers from asking to remove permissive code from the database? More details here need to be provided. \n\n* In Figure 2, it will help to specifically name the languages as you do in Figure 1.", "broader_impact_concerns": "The authors have sufficiently addressed the potential negative impact of their proposed research in Section 6.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1hgXU2FeQ1", "reviewer": "Reviewer_xNBc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The large dataset can be a useful resource for the research community.\n- The dataset collection and curation process is designed to address various societal concerns that have been raised, such as copyright issues and allowing code authors to opt out of collection.\n- The authors provide empirical evidence about the benefits of near-deuplication through their experiments.\n\n# Weaknesses\n- There is no empirical justification for how the precise near-deduplication method was selected, for example to justify the pipeline or the choice of 0.85 for Jaccard similarity.\n- The fraction of data remaining after near-deduplication varies significantly depending on the language. It would be useful to have a study of why this might be, and if some of it is due to the language itself (e.g. a language requiring more common bolierplate than another) versus how people happen to use the language on GitHub.\n- The experiments are only done using the Python subset of the dataset, and not any other languages.\n- A significant fraction (about 1/4 before deduplication) of the dataset is HTML, for which we may expect that a significant fraction of the content is actually natural language, due to the nature of markup. As such, it may be that the HTML data is significantly different from the other programming language data.\n- I think there would be no particular legal issues with the authors distributing copyleft data as part of the dataset, as long as it's clearly marked, and the downstream users can decide what to do with it. However, it looks like that part of the dataset is relatively small (<5%) anyways.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors compile a large dataset (called The Stack) of permissively-licensed open-source code, with the intention that the dataset can be used to train a language model. Unlike prior datasets of code, The Stack only includes code identified as having a permissive license such as the MIT or Apache 2.0 licenses, excluding code with a copyleft license; empowers developers to request removal of their code from the dataset; and is much larger than the earlier CodeParrot dataset. The authors also train some models on the Python subset of the dataset to evaluate how near-deduplication and using only the permissively-licensed data (versus all data) impact the results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To secure recommendation:\n- Please include an experiment or other empirical analysis which covers a part of the dataset other than only Python.\n\nTo strengthen the work:\n- The end of page 7 states that 3 langauges only have less than 4% of the data marked with a permissive license. It would be nice if there was an augmented version of Table 3 to include this information for all languages.", "broader_impact_concerns": "Regarding the \"Malicious code\" paragraph of the Limitations section, I'm not sure if I agree that removing malicious code from the dataset necessarily would have positive social impact. For example, LMs which are aware of malicious code may be generally useful for analyzing and detecting malicious code, not just for generating new copies of it. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PRbrxpMf8d", "reviewer": "Reviewer_qZns", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes a dataset of licensed code which is clearly valuable to the community. However, I find the current version of the manuscript needs significant improvement before publication. See detailed comments below\n\n- The paper needs to give a systematic comparison between The Stack and prior works [1,2,3] (which by the way are not even cited in the work) to illustrate the strengths and weaknesses of your datasets. \\\n[1] CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation \\\n[2] CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks \\\n[3] COSET: A Benchmark for Evaluating Neural Program Embeddings \n\n- In order to show the utility of the dataset, authors tried to reproduce the results of Codex and CodeGen. This is on the right track, unfortunately, it falls short on several fronts. First, why are only two models considered? Second, Why only Codex and CodeGen but not others? Third, why only consider the task of text2code. As a trademark of LLMs that are powered by few-shot learning paradigm (not the fine-tuning approach towards the specific downstream tasks), The Slack needs to enable such LLMs in a variety of code text related tasks.\n\n- The authors need to make the link of the dataset public for evaluation. Since this is the probably the only contribution of the work, I don't believe the paper is acceptable without the open access to The Slack.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a dataset that consists of 3.1 TB of licensed source code in 30 programming languages.  The authors also manage to reproduce the results of Codex and CodeGen with only licensed code.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See strengths and weaknesses section", "broader_impact_concerns": "concerns on the ethical implications of the work have been sufficiently addressed", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As mentioned, the absence of results on languages
other than Python is a significant weakness. On balance, however, the
ML-for-code community can benefit from more large, open, permissively
licensed datasets, and this dataset appears to be a valuable
resource. Given this, I am recommending acceptance. Please incorporate
all reviewer feedback into the final version.
Decision: Accept (poster)