# DRAGON: Robust Classification for Very Large Collections of Software Repositories

Stefano Balla, Stefano Zacchiroli, Thomas Degueule, Jean-Rémy Falleri, Romain Robbes

► To cite this version:

Stefano Balla, Stefano Zacchiroli, Thomas Degueule, Jean-Rémy Falleri, Romain Robbes. DRAGON: Robust Classification for Very Large Collections of Software Repositories. 2026. hal-05498254

HAL Id: hal-05498254

https://hal.science/hal-05498254v1

Preprint submitted on 9 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# DRAGON: Robust Classification for Very Large Collections of Software Repositories

Stefano Balla¹

Stefano Zacchiroli²

Thomas Degueule³

Jean-Rémy Falleri³,⁴

Romain Robbes³

¹DISI – Università di Bologna, Italy

²LTCI, Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France

³Univ. Bordeaux, CNRS, Bordeaux INP, LaBRI, UMR 5800, France

⁴Institut Universitaire de France, France

stefano.balla2@unibo.it, stefano.zacchiroli@telecom-paris.fr,
thomas.degueule@labri.fr, falleri@labri.fr, romain.robbes@labri.fr

## Abstract

The ability to automatically classify source code repositories with “topics” that reflect their content and purpose is very useful, especially when navigating or searching through large software collections. However, existing approaches often rely heavily on README files and other metadata, which are frequently missing, limiting their applicability in real-world large-scale settings.

We present DRAGON, a repository classifier designed for very large and diverse software collections. It operates entirely on lightweight signals commonly stored in version control systems: file and directory names, and optionally the README when available. In repository classification at scale, DRAGON improves F1@5 from 54.8% to 60.8%, surpassing the state of the art.

DRAGON remains effective even when README files are absent, with performance degrading by only 6% w.r.t. when they are present. This robustness makes it practical for real-world settings where documentation is sparse or inconsistent. Furthermore, many of the remaining classification errors are near misses, where predicted labels are semantically close to the correct topics. This property increases the practical value of the predictions in real-world software collections, where suggesting a few related topics can still guide search and discovery.

As a byproduct of developing DRAGON, we also release the largest open dataset to date for repository classification, consisting of 825 thousand repositories with associated ground-truth topics, sourced from the Software Heritage archive, providing a foundation for future large-scale and language-agnostic research on software repository understanding.

**Keywords:** Software repository classification, mining software repositories, machine learning, GitHub, Software Heritage

## 1 Introduction

Exhaustive collections of software source code repositories have grown to hundreds of millions. GitHub alone reports more than 420 million repositories, both public and private, in early 2025.¹

¹https://github.com/about, accessed 2025-03-10 (like other URLs in the paper, unless otherwise stated)
Software Heritage, abbreviated as SWH (Di Cosmo and Zacchiroli, 2017), the largest public archive of software source code, has archived more than 350 million.<fnref n="2" /> The World of Code, known as WoC (Ma et al., 2019), is in a similar ballpark. Browsing, searching through, and discovering relevant projects in such vast collections remain very challenging to this date. This limitation impacts both professional software developers looking for open source components to reuse, and empirical software engineering researchers assembling large sets of repositories to analyze.

*Repository classification* (Ugurel et al., 2002; Kawaguchi et al., 2004) associates descriptive labels (or “topics”) with software repositories to support navigation and discovery. Most existing approaches use the topics that GitHub users assign to their own repositories<fnref n="3" /> as ground truth to train *automated* classifiers, which can then label repositories that lack such metadata (Di Rocco et al., 2020, 2023; Dang et al., 2024; Izadi et al., 2021, 2023). However, these user-provided topics cover only a small portion of the ecosystem: roughly 4% of repositories on GitHub and just 2% of those archived in Software Heritage include them. This scarcity underscores the need for automated classifiers that can assign meaningful topics to the vast majority of unlabeled repositories, enabling large-scale search and analysis.

However, State-of-the-art repository classification methods perform poorly in very large and heterogeneous collections such as SWH and WoC. Three main limitations stand out (discussed further in Section 2):

1. **Topic quality.** GitHub topics are user-generated and often noisy (Sas and Capiluppi, 2022). They frequently include programming languages, which are trivial to infer via file extensions (Fratantonio et al., 2024) and inflate reported accuracy. Focusing on high-value applications and domain topics (e.g., *game*, *database*, *computer configuration*) is both harder and more useful, but is rarely studied at large scale (LeClair et al., 2018; Ma et al., 2018; Yusof and Rana, 2010; Soll and Vosgerau, 2017; Linares-Vásquez et al., 2014; McMillan et al., 2011).

2. **Platform dependency.** Several classification approaches rely on metadata such as repository descriptions and wiki pages, which are not recorded in version control and thus absent from repository collections like SWH and WoC. This limits their applicability beyond GitHub.

3. **README dependence.** Many classification methods rely heavily on README content, yet README files are missing in 42% of repositories in SWH—over 140M projects. A classifier whose performances degrade sharply without README is ill-suited for large-scale archives.

To address these challenges, we introduce **DRAGON**, a new *robust and scalable multi-label repository classifier* that tags repositories with *high-value topics* from the curated GitRanking taxonomy (Sas et al., 2023).

DRAGON is trained on a novel open dataset of over 825 000 repositories extracted from Software Heritage and 239 topics from GitRanking (excluding programming languages). The dataset, presented in Section 4, is the largest ever used for repository classification, 5.5× larger than previous ones.

DRAGON integrates two sources of information: (1) file and directory names from the source tree and (2) the README file, if available. It employs a *sentence-pair BERT* model (Devlin et al., 2019), which keeps file tree and textual signals distinct. A key design objective of DRAGON is robustness, understood as the ability to preserve most of its predictive performance when README

<footnote n="2">²https://archive.softwareheritage.org</footnote>

<footnote n="3">³https://github.com/topics</footnote>
content is missing. This property is essential because real-world repositories vary widely in doc-
umentation quality, and many entirely lack a README. At the same time, achieving robustness
without retraining ensures that a single model can operate seamlessly across both README-rich
and README-scarce cases, avoiding the need to maintain separate models for different input
conditions. This combination makes DRAGON more practical and directly applicable to large,
real-world repository collections.

The dataset used for the training is highly imbalanced, with a few very frequent topics such as *computer configuration* or *game* and many rare ones. This long-tailed distribution makes learning difficult for infrequent classes. We mitigate this class imbalance using *focal loss* (Lin et al., 2017), emphasizing harder examples during training, and refine multi-label outputs by comparing global (Lipton et al., 2014) and per-class thresholding strategies (Narasimhan and Menon, 2021). The model architecture and training procedure are described in detail in Section 5; the experimental setup is reported in Section 6.

1.1 Research Questions

To evaluate DRAGON, we investigate the following research questions:

*   **RQ1 (Effectiveness on domain topics):** How accurately can repositories be classified on domain-only labels (excluding those related to languages) at scale?

*   **RQ2 (Robustness and design under README scarcity):** How robust is performance when README is absent at inference *without retraining*, and does sentence-pair encoding outperform single-sequence concatenation when README is missing?

*   **RQ3 (Evaluation protocol):** Which thresholding strategy best balances performance and class coverage?

*   **RQ4 (Error characteristics):** Which kinds of errors occur, and how much are they due to semantic near-misses?

1.2 Results

DRAGON achieves F1@5 score of 60.8%, surpassing the best previous approach in the literature (Dang et al., 2024) by 6%. When README files are missing, a common occurrence in our target use cases, DRAGON, *without retraining*, degrades by only 6%, versus -9.5% in the state-of-the-art with retraining (Izadi et al., 2021).

Beyond aggregate metrics, our error analysis shows that a large portion of apparent misclas-
sifications are semantically close to the correct topics—so-called near misses. This makes the
predictions useful in practice even when exact matches differ from the ground truth, since they still
reflect the project’s domain with meaningful proximity.

Sections 7 and 8 detail these findings, which we discuss in Section 9 before concluding in Section 10.

1.3 Data availability

A complete replication package (Balla et al., 2025) for this work is available from Zenodo.
## 2 Background and Motivation

We articulate the constraints of exhaustive archives (resource limits, forge heterogeneity, and missing documentation), motivating our design choices in Section 5 and the evaluation criteria in Section 6.

Large-scale project classification has several valuable use cases. It allows to refine navigation and searches for relevant projects in large-scale software repositories and ecosystems, which is useful for developers searching for projects as well as scientists studying the practice of software engineering. Applied to exhaustive software archives such as Software Heritage, it allows for more representative empirical studies, which is important for generalizing findings (Nagappan et al., 2013). Trujillo et al. (2022) found that open-source projects not on GitHub differ from the open-source projects found on GitHub: notably, projects not on GitHub tended to be maintained for longer, had more collaborators, and focused on different domains. Such exhaustiveness further allows for larger-scale studies of specific categories of software repositories.

### 2.1 Assumptions on data availability

While several project classification approaches exist, they take assumptions that are not suitable for such large-scale scenarios targeting an exhaustive software archive.

In truly large scale scenarios, accessing all the content is an expensive operation, due to the sheer size of it. For instance, a copy of Software Heritage, the largest public source code archive, currently requires about 2 PiB of storage. Just storing this amount of data was estimated to cost around 50,000 US$ in 2023.<fnref n="4" />

This provides a strong incentive to minimize the resources required to build an automatic repository classification system. Classification approaches that require access to source code cannot scale to such volumes of data.

Not all software forges have the same metadata; approaches that work on GitHub may rely on metadata specific to GitHub, such as project descriptions of wiki pages, that are not available in other forges. Topics, which may be used to train a project classifier, are not present in all the forges.

Taking into account the diversity of metadata and the size constraints, a common set of features naturally emerges: the file tree data of the project (in terms of its name and its file names) as a lightweight approximation of its content; as well as the README file, a high-signal source of high-level documentation about the project. Downloading and storing a single and small text file is a much more scalable solution in a large-scale scenario.

Even relying on this common denominator is not valid in many cases. In Software Heritage, 42% of projects lack a README file. This could be for several reasons. The main documentation file (the README) may be found in different locations in different forges, and writing heuristics to find it has diminishing returns. The project may use other forms of documentation that are hard to detect via heuristics, that, by necessity, can not look at the content of entire project. Even when a README is found, it can be difficult to use for multiple reasons: it can be very terse (e.g., if other sources of documentation are available); a significant portion of projects may be written in languages other than English, or that use encodings other than ASCII. Several project classification approaches filter out examples whose README do not contain enough english data, e.g. (Dang et al., 2024; Izadi et al., 2023), once again limiting the applicability of these approach. Thus, an approach that works on exhaustive archives should have good results even in the absence of a README. While previous works illustrate the feasibility of multi-label classification using textual

<footnote n="4">⁴https://ourworldindata.org/grapher/historical-cost-of-computer-memory-and-storage</footnote>
metadata, existing approaches suffer from severe performance degradation (Izadi et al., 2021) or
become entirely inapplicable (Izadi et al., 2023; Widyasari et al., 2023) when README files are
absent. The project file tree (directory and file names) is always present and retrievable, while
being also a small source of information.

## 2.2 Assumptions on label quality

In addition, many project classification approaches make assumptions about label quality.

A convenient source of labels for projects are GitHub Topics and GitHub Featured Topics. GitHub Topics <fnref n="5" /> are user-defined tags that repository maintainers assign to describe the content, language, or domain of a project (e.g., machine-learning, bioinformatics). Featured Topics <fnref n="6" /> are a curated subset of these topics, selected and maintained by GitHub staff. However, this data source features a large amount of variability in terms of quality of the data, distribution of the data among classes, and in terms of the domains of the labels. Most recent approaches rely on GitHub Featured Topics, which feature a level of data curation, as opposed to regular GitHub topics. However, Featured Topics are still low quality classes (e.g., terse topics such as #rna-seq, or #ci), as demonstrated by Sas et al. (2023) and Sas and Capiluppi (2022) who explored their inconsistencies and limitations.

In particular, a large portion of such labels represent programming languages (e.g., a web library written in python may have the topics #web and #python). Classifying programming languages in this way impacts the performance of the classifier in several ways: first, these labels might be often very easy classes to guess (especially if a classifier can access file names) and they are also common (see Table 2). This leads to an over-optimistic performance on datasets that feature these programming language topics. Second, the model dedicating some of its capacity to these labels may lead it to allocate less capacity for other labels (less common and more difficult), negatively affecting performance there. Finally, the task of predicting programming languages is not necessarily a machine learning task, as many tools exist for this very topic, from the elaborate such as Guesslang <fnref n="7" /> and Linguist <fnref n="8" />), to more crude heuristics based on file extensions.

More generally, performance on a few labels should not dominate the overall performance, and should not be at the detriment of the performance of the other labels. A project classifier that targets exhaustive software archives should have performance that is well distributed among all possible classes.

To sum up, the choice of labels in the classification has a major impact on the actual task that the classifier solves (i.e., whether it is a practical task or not) and on the performance of the classifier (and whether it reflects reality). Therefore, the selection of labels should be approached with methodological rigor and clear justification.

Sas et al. (2023) introduced an active-sampling approach to identify and create a taxonomy of the most informative topics for repository labeling. This process condensed 130k GitHub topics into a well-structured taxonomy of 301 topics, specifically focusing on application domains while excluding programming languages, a trivial category to predict, as previously demonstrated by Di Sipio et al. (2020). The resulting taxonomy, named GitRanking, provides a structured mapping from raw GitHub topics to curated taxonomy topics, allowing for the classification of repositories based on more meaningful topic representations. The authors also determine the depth of each

<footnote n="5">⁵https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/classifying-your-repository-with-topics</footnote>

<footnote n="6">⁶https://github.com/topics/featured</footnote>

<footnote n="7">⁷https://github.com/yoeo/guesslang</footnote>

<footnote n="8">⁸https://github.com/github-linguist/linguist</footnote>
topic within the taxonomy, specifying the hierarchical level at which a topic resides. This taxonomy should serve as a foundational reference for any repository classification problem. Following this taxonomy creation, Sas et al. (2023) applied a probabilistic methodology, inspired by their prior work (Di Sipio et al., 2020), to solve the multi-class classification problem. While Sas et al. made an important first step toward improving label quality, their methodology did not yet achieve state-of-the-art performance: they report an F1 score of 34%, which is below the results achieved by later approaches discussed in Section 3 and summarized in Table 5. It is worth noting, however, that Sas et al. (2023) are the only prior work that, like our study, removes programming language labels from the taxonomy. This makes their classification task substantially harder and renders direct comparison with models evaluated on language-heavy topic sets less meaningful.

Building on this foundation, our work addresses the remaining limitations by targeting robustness under sparse documentation and scalability to exhaustive archives—challenges that prior approaches, even with curated taxonomies, have yet to resolve. The following section situates our contribution within this broader landscape of repository classification methods.

# 3 Related work

Guided by **RQ1–RQ4**, we analyze prior work along three axes: (i) availability of input features across platforms and archives, (ii) quality and nature of the label taxonomy, and (iii) scalability to collections containing hundreds of millions of repositories. Within this space, two main paradigms have emerged: (A) *source-code-based methods* and (B) *textual or metadata-based methods*. These paradigms differ in the signals they exploit (code vs. documentation), the type of learning (unsupervised vs. supervised), and the scale of datasets they can handle.

## 3.1 Source-code-based methods

Many early studies attempted to infer domain or functionality from raw source code. For instance, Kawaguchi et al. introduced MUDABlue (Kawaguchi et al., 2004), applying Latent Semantic Analysis (LSA) to categorize SourceForge repositories of only a few thousand projects. Others employed latent Dirichlet allocation (LDA) on code bases (Linstead et al., 2007; Tian et al., 2009), or techniques like SVM and decision trees (Linares-Vásquez et al., 2014; Yusof and Rana, 2010). More modern approaches, such as LeClair et al. (LeClair et al., 2018), leverage Neural Networks on roughly 9k projects with curated domain labels. While code-level analysis can yield meaningful categories, it is often *expensive to mine* at scale, and so not a feasible solution for our use case. Most code-centric efforts remain limited to only a few hundred (Kawaguchi et al., 2004; Tian et al., 2009) or, at best, a few tens of thousands of repositories (Linares-Vásquez et al., 2014), making them unsuitable for massive archives like SWH or WoC.

## 3.2 Textual and metadata-based methods

Another extensive line of research leverages *textual documentation* to categorize software. In supervised multi-label settings, researchers often adopt a *TF-IDF* or embedding-based transformation of textual data and then map it to preexisting topics. For instance, Izadi et al. (2023) proposed a *semantically-enhanced* recommendation pipeline using Knowledge Graphs and Logistic Regression (LR), trained on approximately 150k GitHub repositories. Izadi et al. (2021) used DistilBERT or TF-IDF + LR on README, wiki, and file names for multi-label classification, showing that TF-IDF + LR achieves stronger results in practice. Both studies, however, rely on topic sets
<table><caption>Table 1: Comparison of repository classification studies based on dataset characteristics.</caption><thead><tr><th>Study</th><th>Size</th><th>Data</th><th>Label source</th><th>Prog. langs. as topics</th></tr></thead><tbody><tr><td>Kawaguchi et al. (2004); Linstead et al. (2007); Tian et al. (2009); Linares-Vásquez et al. (2014); Yusof and Rana (2010)</td><td>&lt; 3300</td><td>Source Code</td><td>SourceForge Domains</td><td>No</td></tr><tr><td>LeClair et al. (2018)</td><td>9000</td><td>Source Code, Project descriptions</td><td>Curated Domains</td><td>No</td></tr><tr><td>Izadi et al. (2023, 2021)</td><td>150 000</td><td>Readme, Wiki, Project Description, file names</td><td>Featured Topics</td><td>Yes</td></tr><tr><td>Widyasari et al. (2023)</td><td>22 000</td><td>Readme, Project Description</td><td>Featured Topics</td><td>Yes</td></tr><tr><td>Dang et al. (2024)</td><td>22 000</td><td>Readme</td><td>Featured Topics</td><td>Yes</td></tr><tr><td>Sas et al. (2023)</td><td>48 000</td><td>Readme, Project Description</td><td>GitRanking Topics</td><td>No</td></tr><tr><td>Our work</td><td>825 000</td><td>File/Repository names, Readme (optional)</td><td>GitRanking Topics</td><td>No</td></tr></tbody></table>

that include a substantial proportion of programming languages: in Izadi et al. (2023), six of the twenty-five most frequent topics are languages, and in Izadi et al. (2021), nine of the top twenty-five fall in that category. This inflates apparent performance, since language prediction is a comparatively trivial task. Although these approaches scale to roughly 150k repositories, they still depend on the presence of documentation (e.g., README files). In their ablation study, removing the README led to a sharp performance drop, with F1@5 declining from 47% to 37.5%. Widyasari et al. (2023) reformulated the same task, under the same assumptions of complete documentation and language-heavy topic sets, as an extreme multi-label classification problem evaluated on about 22k repositories, reporting improved metrics over previous techniques. Dang et al. (2024), building on the same dataset, further approached the task as a long-tail problem, introducing a distributed balance loss and quality filtering mechanism to improve performance metrics. Sas et al. (2023) proposed a smaller but higher-quality dataset of 48k repositories labeled using the curated GitRanking taxonomy, focusing on domain topics rather than programming languages.

## 3.3 Positioning and contributions

The successes of prior work are often tied to two favorable conditions: (i) the use of programming language topics, which are easy to predict and inflate reported performance, and (ii) evaluation on repositories with rich textual documentation, particularly README files. In contrast, large-scale archives such as SWH contain millions of repositories without README content, and practical discovery tasks require domain-oriented topics rather than programming languages. This gap motivates our focus on robust classification under sparse textual conditions and with higher-value labels.

Table 1 summarizes representative repository classification studies by dataset size, input signals,
and label sources. The comparison highlights four persistent issues across prior work:

1. *Dataset sizes* rarely exceed 150k repositories, and many studies cover only a few thousand.

2. *Overreliance on code or textual documentation* hampers performance when README or other textual fields are incomplete or absent, or make approaches reliant on code inapplicable at a large scale.

3. *Low-quality classes* often yield only superficial or uninformative labels.

4. *Prediction of programming languages* dominates many efforts, even though simpler standalone tools (e.g., Guesslang) can solve that problem more efficiently.

To address these shortcomings, we propose:

* *The creation and the use of a large-scale, real-world dataset* of over 825k repositories from Software Heritage, surpassing previous studies by a wide margin.

* *Excluding programming language detection* and focusing instead on a high-quality taxonomy of domain-specific classes (GitRanking).

* *Combining file tree and textual data* within a sentence-pair BERT architecture, achieving strong predictive power even for repositories that lack a README.

This work thus advances the state of the art in both *scale* and *robustness*, delivering a more resilient and domain-focused classification pipeline aligned with the realities of massive open-source ecosystems.

# 4 Dataset

We detail in this section how we assembled our dataset of software repositories and associated topics, the largest available to date for repository classification tasks. The main stages of the dataset extraction and cleaning pipeline are summarized in Figure 1.

The dataset design reflects our research questions. It prioritizes domain-oriented topics (RQ1), includes repositories even when README files are absent (RQ2), maintains a long-tail topic distribution to study thresholding trade-offs (RQ3), and preserves sufficient provenance to support a semantic error analysis (RQ4).

## 4.1 Initial dataset

We obtained from the Software Heritage archive (Di Cosmo and Zacchiroli, 2017) an export of all archived Git repositories hosted on GitHub and associated topics on that platform. The choice of starting from GitHub repositories is due to the large availability of topics on that platform, but the obtained classifier is independent on any specific platform and only need information stored in version control systems (VCSs) to operate. The initial dataset contained 216 775 237 repository addresses (called “origins” in SWH jargon) and the topics associated with each of them. We also obtained the most recent version of the SWH graph dataset (Pietri et al., 2019), to be used later for repository mining purposes.
<figure><img src="image_2.png" /><figcaption>Figure 1: Overview of the data processing and filtering pipeline. The Sankey diagram shows the progressive reduction of repositories from the initial 8047 951 GitHub projects archived by Software Heritage with at least 1 topic, to the final dataset of approximately 825 000 repositories. Each split corresponds to a major operation described in the following subsections: 4.2 Topic filtering, 4.3 Repository mining and 4.4 Data cleaning.</figcaption></figure>
## 4.2 Topic filtering

To prepare an annotated dataset for training, we only kept repositories with at least one associated topic. This reduced the dataset to 8 047 951 repositories. Approximately 96% of GitHub repositories (98% of all repositories) archived in SWH lack associated topics, strengthening the need to label them in order to peruse this huge collection.

As user-provided topics (those in the initial dataset) tend to be noisy and low-quality (Sas et al., 2023; Sas and Capiluppi, 2022), we remapped them to higher-quality ones using the GitRanking taxonomy (Sas et al., 2023) and associated mapping. This step also filters out implicitly programming language topics, as they are not part of the GitRanking taxonomy.

Table 2 compares the top 30 topics before and after GitRanking mapping, illustrating the prevalence of programming languages (before) and that of higher-value domain topics (after). Before the mapping, the most frequent topics were dominated by specific technologies and frameworks (e.g., *React*, *Node.js*, *Android*, *Docker*, *MongoDB*, *Express*, *Vue*, *Django*), reflecting transient or tool-oriented interests typical of user tagging behavior on GitHub. In contrast, after applying the GitRanking-official remapping, the topic distribution becomes more balanced and oriented toward broader, longer-lived software domains (e.g., *database*, *security*, *blockchain*, *data visualization*). This shift highlights how GitRanking improves label consistency and conceptual abstraction, better supporting the study of repository functionality rather than popularity of individual technologies. Nonetheless, technology-specific topics remain valuable indicators of emerging trends and could be investigated in future work to analyze temporal dynamics or framework adoption patterns. After the mapping we obtain 4 784 812 repositories with at least one (GitRanking) topic and a total of 239 distinct topics used across all remaining repositories. We discard all repositories with no remaining topics. In summary, this re-labeling directly supports RQ1 by focusing evaluation on good-quality domain topics instead of programming languages.

## 4.3 Repository mining

We collected three kinds of information from each remaining repository: *file tree data*, *textual data*, and *metadata*:

* *File tree data:* to capture the organization of each repository, we extracted its file and directory hierarchy from the HEAD commit (i.e., the most recent commit in each repository). We recorded the name (including extensions) of every directory and every file in the HEAD commit, preserving the hierarchical file-system structure. We also preserved symbolic links, without following them, to preserve file structure as shown to developers.

* *Textual data:* we looked for the presence of README files located at the root directory of the HEAD commit. We considered common variants of the README name (e.g., README.md, README.txt, README.rst, ignoring case differences) and excluded files larger than 100 MiB. Each identified README was retrieved from the SWH archive and stored in the dataset based on its intrinsic SHA1 identifier.

* *Repository metadata:* finally, we retained from the initial dataset the following repository metadata, as originally stored on GitHub and archived by SWH: URL (e.g., https://github.com/user/repository), whether it is a fork or not (only for deduplication purposes later on), the topics.

This repository mining step has been implemented using the compressed graph representation of the SWH archive (Boldi et al., 2020), via its Rust API. Code for this step is provided as part of the replication package for this paper (Balla et al., 2025).
<table><caption>Table 2: Top-30 topics before/after GitRanking mapping, with programming language topics highlighted in bold.</caption><thead><tr><th colspan="2">Before applying GitRanking</th><th colspan="2">After applying GitRanking</th></tr><tr><th>Topic</th><th>Frequency</th><th>Topic</th><th>Frequency</th></tr></thead><tbody><tr><td>javascript</td><td>93 384</td><td>computer conf.</td><td>93 494</td></tr><tr><td>r</td><td>83 627</td><td>game</td><td>49 613</td></tr><tr><td>python</td><td>81 674</td><td>website</td><td>42 235</td></tr><tr><td>c</td><td>75 459</td><td>algorithm</td><td>29 360</td></tr><tr><td>react</td><td>62 258</td><td>database</td><td>22 110</td></tr><tr><td>nodejs</td><td>48 671</td><td>internet bot</td><td>18 227</td></tr><tr><td>java</td><td>44 725</td><td>blockchain</td><td>17 511</td></tr><tr><td>css</td><td>44 298</td><td>web application</td><td>17 494</td></tr><tr><td>html</td><td>40 952</td><td>data structure</td><td>15 683</td></tr><tr><td>typescript</td><td>32 724</td><td>video game dev.</td><td>15 623</td></tr><tr><td>d</td><td>29 195</td><td>artificial neural net.</td><td>14 901</td></tr><tr><td>php</td><td>27 034</td><td>front end</td><td>13 741</td></tr><tr><td>hacktoberfest</td><td>26 040</td><td>automation</td><td>13 167</td></tr><tr><td>android</td><td>23 411</td><td>security</td><td>12 927</td></tr><tr><td>docker</td><td>23 064</td><td>server</td><td>12 523</td></tr><tr><td>machine-learning</td><td>21 632</td><td>visualization</td><td>12 518</td></tr><tr><td>api</td><td>18 572</td><td>microservices</td><td>12 163</td></tr><tr><td>v</td><td>17 778</td><td>software testing</td><td>11 857</td></tr><tr><td>mongodb</td><td>17 262</td><td>digital image proc.</td><td>11 655</td></tr><tr><td>express</td><td>14 754</td><td>deep neural network</td><td>11 507</td></tr><tr><td>csharp</td><td>13 307</td><td>cryptography</td><td>10 578</td></tr><tr><td>git</td><td>13 113</td><td>animation</td><td>10 282</td></tr><tr><td>bootstrap</td><td>12 913</td><td>cryptocurrency</td><td>9977</td></tr><tr><td>vue</td><td>12 658</td><td>websocket</td><td>9976</td></tr><tr><td>react</td><td>12 422</td><td>music</td><td>9865</td></tr><tr><td>django</td><td>12 373</td><td>data</td><td>9633</td></tr><tr><td>cpp</td><td>12 338</td><td>mathematics</td><td>9479</td></tr><tr><td>mysql</td><td>11 854</td><td>simulation</td><td>8742</td></tr><tr><td>deep-learning</td><td>11 805</td><td>robotics</td><td>8717</td></tr><tr><td>angular</td><td>11 797</td><td>back end</td><td>8675</td></tr></tbody></table>
<figure><img src="image_3.png" /><figcaption>Figure 2: Topic distribution vs. power law ($y \propto x^{-0.77}$).</figcaption></figure>

## 4.4 Data cleaning

Some repositories did not have a retrievable file tree in their HEAD commit. The most common reason for this is that they were empty repositories at the time of archival by SWH. Excluding them led to 1873 605 repositories with an associated file hierarchy. Most of the excluded repositories had `github-config` as their only topic, indicating a repository used to configure GitHub-specific features (e.g., user profiles, wiki pages).

To address class imbalance, we employed an inverse-frequency sampling approach, where weights were assigned inversely proportional to the frequency of topic combinations. This ensured that less common topics received higher weights in the sampling process. This approach targeted 900 000 repositories, determined iteratively using the elbow method to balance coverage. We adjusted the dataset size until the most frequent topic was not disproportionately dominant over the second most frequent, ensuring a more even representation. After applying balancing strategies, the final topic distribution, shown in Figure 2, still exhibits a long-tail pattern closely following a power law, which is desirable as it aligns with real-world data conditions.

To avoid contamination between training and test sets, we deduplicated repositories in the dataset using two approaches. First, we removed repositories declared as forks on GitHub, which resulted in the removal of 25 583 repositories. Second, we removed repositories having the same
<figure><img src="image_4.png" /><figcaption>Figure 3: Distribution of README lengths (in words) for our dataset, excluding outliers. Most repositories have minimal documentation.</figcaption></figure>

URLs, which could be present in the dataset due to SWH archiving the same repositories via differ-
ent protocols or through renames over time; this resulted in removing a further 48 856 repositories.
To break ties when deduplicating, we always kept the most recently archived repository. After
deduplication, we obtain a final dataset of 825 561 repositories.

Following Izadi et al. (2021), we apply a standardized cleaning pipeline also to the content of README files, to enhance consistency and reduce noise. We remove emails, URLs, usernames, embedded code snippets, punctuation, digits, and non-ASCII characters. The latter step is necessary as the bert-base tokenizer, which we use, has an English-centric vocabulary and replaces many non-ASCII symbols with [UNK] tokens, leading to data contamination. Additionally, we strip HTML/XML tags, markdown symbols, and overlong whitespaces.

4.5 Dataset statistics

The final dataset contains 825 561 repositories, with 277 907 (34%) of them lacking a README file.

READMEs in the dataset tend to be short, with a median length of 75 words (see distribution in Figure 3): 25% are under 20 words, and 75% contain fewer than 200, with very few exceeding 500.

The number of labels per repository is also highly skewed (Figure 4): 83.6% have one topic, 13.6% have two, 2.3% have three, 0.4% have four, and only 0.1% have five. This reflects the general sparsity of topic annotations.

5 Methodology

DRAGON is based on the BERT machine learning model, trained on the dataset of Section 4, and evaluated against the RQs above. We describe in this section the high-level building blocks of DRAGON, leaving the technical details of its training to Section 6.

5.1 Classification pipeline

DRAGON is trained on two primary inputs:

*   *File tree data:* repository name plus up to *k*=50 distinct file and directory names sampled uniformly from the repository's tree; names are cleaned (trimmed, empty dropped) and deduplicated per repository, with casing preserved. Sampling is deterministic per repository and sampled names are sorted for stable artifacts. This keeps inputs within BERT's 512-token budget and leaves room for the README when present.
<figure><img src="image_5.png" /><figcaption>Figure 4: Distribution of the number of labels per repository. Most repositories have only one or two associated labels, showing the sparsity of topic annotations.</figcaption></figure>

*   **Textual input:** The README content, if available.

We applied the BERT tokenizer's default behavior: if the combined length of the two segments exceeds the 512-token limit, tokens are truncated from the longer segment first.

We feed these inputs into a BERT model configured for *sentence-pair* classification (Devlin et al., 2019). This design leverages segment embeddings to distinguish the file tree (first segment) from the README content (second segment). The final BERT [CLS] token representation encodes information across both segments. A fully connected layer with a sigmoid activation produces probability scores for each of the 239 topic labels. This separation of file-tree and README inputs is designed to preserve robustness when documentation is absent, directly supporting our investigation of RQ2.

To address the high class-imbalance present in the dataset, we use *focal loss* (Lin et al., 2017), which down-weights easy examples and places greater emphasis on *hard-to-classify* instances. Formally, for a given label $y \in \{0, 1\}$ with predicted probability $p$, the focal loss is expressed as:

$$FL(p) = -\alpha(1 - p)^{\gamma} \text{BCE}(x, y) \quad (1)$$

where BCE($x, y$) is the binary cross-entropy, $\alpha \in [0, 1]$ is a weighting factor for the positive class, and $\gamma \ge 0$ controls the strength of down-weighting. High values of $\gamma$ focus the loss more heavily on difficult or misclassified examples. This strategy not only mitigates skewed label frequencies, but also helps stabilize gradients for rare labels, leading to better overall performance.

## 5.2 Model architecture and design rationale

Modern transformer encoders provide a wide design space. DistilBERT (Sanh et al., 2019), for instance, trades some accuracy for faster inference, while very long-context variants are unnecessary in our case, as both the file-tree sample and the README text comfortably fit within BERT's
512-token window. We therefore adopt the original BERT architecture (Devlin et al., 2019) as a practical balance between quality and efficiency.

The choice of a sentence-pair formulation, is motivated by robustness considerations. Multi-modal learning research shows that naive early fusion (simple concatenation) allows a dominant modality to suppress weaker ones, reducing performance when one input is missing (Baltrusaitis et al., 2019; Gao et al., 2025). By contrast, segment embeddings act as explicit boundaries: if the README segment is empty, the model still receives a clean representation of the file-tree input and does not conflate “missing” with “short.”

Empirical evidence supports this design. CodeBERT encodes source code and documentation as two segments separated by [SEP], preserving distinct modality information while still enabling cross-attention (Feng et al., 2020). Similar paired-segment formulations consistently outperform single-sequence concatenation in natural-language inference and question-answering tasks—the same benchmarks on which BERT was originally validated (Devlin et al., 2019).

Our own experiments (Section 8) confirm these findings: the sentence-pair encoding maintains high performance even when the README is absent, whereas naive concatenation leads to measurable degradation. This architectural choice directly contributes to the robustness goals defined in RQ2.

## 5.3 Thresholding strategies

To deal with the fact that the dataset rarely contains more than three labels per repository, we introduce a *quality filter* on top of the predicted probabilities, inspired by previous work (Dang et al., 2024). This filter prevents low-confidence classes from being assigned, improving precision and ensuring that only the most relevant topics are predicted.

To implement this filtering mechanism, we explore two thresholding strategies: a *single global threshold*, which uniformly removes low-confidence predictions, and a *per-class threshold*, which assigns a distinct confidence threshold to each label based on its frequency and distribution, in details:

*   **Single threshold:** as done in prior work (Dang et al., 2024), a *single threshold* selects the top-k predicted labels for each repository, discarding those with confidence below a predefined common value $\tau$.

*   **Per-class thresholds:** A single global threshold may not be optimal for all labels, as different topics exhibit varying levels of confidence distribution and frequency in the dataset. To take this into account, per-class thresholds are distinct values $\tau_c$ for each topic c. Predictions for every class are initially produced, yielding a predicted probability $P(c)$. Then all classes are ranked in descending order by $P(c)$, and the selection of the top k prediction is denoted as $O_{\text{origin}}^k$. For each class c in $O_{\text{origin}}^k$ we apply the corresponding $\tau_c$, retaining only classes whose predicted probability is greater or equal than the respective thresholds. The final set of topics O for a given repository is hence:

$$O = \{c \in O_{\text{origin}}^k \mid P(c) \geq \tau_c\} \qquad (2)$$

Each per-class threshold $\tau_c$ is tuned independently using a validation set, optimizing for a balance between precision and recall to maximize the overall F1-score and class coverage (Narasimhan and Menon, 2021; Ding et al., 2023). This strategy allows us to account for the varying confidence distributions across different labels, yielding finer control over multi-label predictions. The precise tuning methodology is detailed in Section 6.
# 6 Experimental setup

We now detail the experimental setup of training DRAGON, covering: train/validation/test partitioning, thresholding, comparison baselines, and evaluation metrics.

## 6.1 Dataset partition

We randomly partition the final dataset from Section 4 of 825 561 repositories into three training/-validation/testing datasets using the following ratios: 81%, 9%, and 10%.

## 6.2 Training configuration

Unless otherwise stated, we fine-tune **BERT-base-uncased** with a multi-label classification head using focal loss. We use the Hugging Face tokenizer for **bert-base-uncased**. Inputs are sentence pairs: Segment A concatenates the repository name with up to 50 sampled file and directory names, Segment B is the README text when available.

We use **max_length** equal to 512 with pairwise truncation set to *longest first*. When the pair exceeds the budget, tokens are removed from the longest segment, which preserves the shorter segment intact.

We train with AdamW and a linear learning rate schedule using the default learning rate of $5 \times 10^{-5}$. Mixed precision is enabled. We set per device train and evaluation batch size equal to 2 and use gradient accumulation with 4 steps, yielding an effective batch size of 8 per device. We train for 3 epochs. Other AdamW parameters follow the library defaults.

We use focal loss with $\alpha = 0.5$ and $\gamma = 2.0$ applied to the logs produced by the classification head. This down weights easy examples and emphasizes hard and rare labels.

We evaluate at each epoch end and save a checkpoint at epoch end, keeping the best checkpoint by micro F1@5. We reload the best model at training end.

During validation we compute precision, recall, and F1 at $k \in \{1, 3, 5\}$ with micro averaging, as well as threshold based scores used later for threshold tuning. The selection metric for early model choice is micro F1@5.

Imbalance is handled by the loss, we do not use data re sampling in the default setting. We also experiment with the resampling loss from LEGION in a separate variant and with a weighted BCE variant; both use the same data and evaluation pipeline as the focal loss model.

We fix the random seed for the tokenizer and data loader in our training scripts and report exact configurations in the replication package, including model name, all **TrainingArguments**, and per model hyperparameters.

## 6.3 Evaluation metrics

We evaluate repository classification using standard multi-label metrics, including precision@k, recall@k, and F1@k for $k \in [1, 5]$, and report *micro-averaged* scores to capture predictive accuracy. We include top-k metrics because most repositories have few labels (83.6% with one, 13.6% with two; see Section 4.5). For deployment alignment, we also consider threshold-only evaluation without fixing k when discussing thresholding strategies.

We also measure class coverage (Narasimhan and Menon, 2021; Ding et al., 2023), which assesses how well the model distributes its predictions across the ground-truth categories, preventing bias
toward frequent labels. We define it as:

$$ \text{ClassCoverage} = \frac{|\{c \mid \sum_{i=1}^{N} \hat{y}_{i,c} > 0\}|}{|\{c \mid \sum_{i=1}^{N} y_{i,c} > 0\}|} \qquad (3) $$

where $y_{i,c}$ and $\hat{y}_{i,c}$ represent the true and predicted labels for class c in instance i, and N is the total number of instances.

## 6.4 Threshold tuning

We tune the various thresholds for the approaches described in Section 5.3 as follows:

* **Global threshold:** to tune the global threshold we perform a grid search over $\tau \in [0.0, 1.0]$ on the validation set, selecting the threshold that maximizes F1@5, following the approach of Dang et al. (2024).

* **Per-class thresholds:** we optimize per-class thresholds using coordinate descent (Wright, 2015), an iterative method that updates one variable at a time while keeping others fixed. The goal is to balance the micro-averaged F1-score with class coverage, ensuring a broader distribution of predicted classes.

The process starts by initializing all thresholds to a default value. Then, for a fixed number of iterations, we cycle through each class c, evaluating candidate thresholds $\tau_c \in [0.0, 1.0]$. At each step, $\tau_c$ is temporarily updated while others remain unchanged, and model predictions are re-evaluated to compute micro F1-score and class coverage. The objective function is:

$$ L = \text{F1}_{\text{micro}} + \lambda_{\text{class}} \cdot \text{ClassCoverage} \qquad (4) $$

where $\lambda_{\text{class}}$ is a weighting factor promoting diverse class predictions. The optimal threshold for each class is the candidate maximizing this function.

Once all classes have been updated for a full cycle, the algorithm evaluates whether the improvement in the composite score exceeds a predefined tolerance. If the improvement is negligible, the optimization stops early; otherwise, another iteration is performed.

## 6.5 Baselines

We compare our approach against both internal and external baselines. Internally, we validate our methodology by re-implementing LEGION (Dang et al., 2024) from its original replication package<fnref n="9" /> and training it on our dataset. This guarantees an equitable comparison between LEGION and DRAGON: performance differences will be attributable to methodological advancements only, rather than dataset discrepancies.

As external baselines, we train our model on the benchmark introduced by LEGION to evaluate all state-of-the-art repositories classifiers, up to that point in the literature, namely:

* **LEGION** (Dang et al., 2024) itself, the best performing repository classifier in the literature (before DRAGON);

* **ZestXML** (Widyasari et al., 2023), a classifier optimized for long-tail label distributions;

<footnote n="9">⁹https://github.com/AI4Code-HUST/LEGION</footnote>
<figure><img src="image_6.png" /><figcaption>Figure 5: Distribution of README lengths (in words) in the LEGION dataset. Compared to our dataset (median 75), LEGION READMEs are substantially longer (median 463 words), indicating higher documentation quality.</figcaption></figure>

* **TF-IDF+LR** (Izadi et al., 2021) (or just **LR** in the following, for conciseness), that combines file tree and textual features using a logistic regression classifier.

The benchmark introduced by LEGION to compare these classifiers, which we use as external baselines, consists of 15 262 repositories crawled from GitHub, annotated with 665 unique labels. This dataset contains, for each repository: repository name, README file (in general a well-curated one), and associated topics. Unlike our dataset, LEGION does not include file tree information, and its topics also cover programming languages—both factors that *disadvantage* DRAGON relative to other baselines.

Figure 5 shows the distribution of README lengths in the LEGION dataset. Compared to our dataset (median of 75 words), LEGION's READMEs are considerably more detailed, with a median of 463 words, an interquartile range from 238 to 914, and no missing READMEs. This confirms that LEGION repositories tend to be well-documented, which can benefit models that rely heavily on textual features.

By evaluating on both standard benchmark and real-world repositories (with, among other variations, frequently missing README files), we assess the robustness of DRAGON across very diverse conditions.

# 7 Results

We breakdown the presentation of our experimental results by research question (see Section 1).

## 7.1 Evaluation on large-scale repository collections (RQ1)

We evaluate DRAGON on our *domain-only* dataset, which excludes programming language topics and contains repositories that often lack comprehensive READMEs, thereby providing a realistic and challenging setting for repository classification. Our goal is to assess whether the approach of DRAGON, designed to handle sparse project descriptions, can achieve state-of-the-art performance in identifying high-value topics.

Table 3 reports precision@k, recall@k, and F1@k for $k \in \{1, 5\}$ under three thresholding methods (cf. Section 5.3): *no threshold*, *single global threshold*, and *per-class thresholds*.

* **No threshold** simply ranks the top-k predictions without filtering, capturing the highest recall (53.6% at $k = 1$, 82.1% at $k = 5$), but at the cost of many spurious predictions that degrade F1 (down to 31.8% at $k = 5$).
<table><caption>Table 3: Comparison of DRAGON variants vs LEGION on our domain-only dataset (without programming languages). All values are percentages, and improvements are reported as relative percentages.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">k = 1</th><th colspan="3">k = 5</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>DRAGON, no threshold</td><td>64.2</td><td><strong>53.6</strong></td><td>58.4</td><td>19.7</td><td><strong>82.1</strong></td><td>31.8</td></tr><tr><td>DRAGON, per-class threshold</td><td>70.1</td><td>50.9</td><td>59.0</td><td>57.6</td><td>58.6</td><td>58.1</td></tr><tr><td>DRAGON, single threshold</td><td>68.7</td><td>52.1</td><td><strong>59.3</strong></td><td>62.0</td><td>59.7</td><td><strong>60.8</strong></td></tr><tr><td>LEGION Dang et al. (2024)</td><td><strong>77.1</strong></td><td>42.5</td><td>54.8</td><td><strong>72.0</strong></td><td>44.2</td><td>54.8</td></tr><tr><td>DRAGON improvement (%)</td><td>-10.9</td><td>+22.6</td><td>+8.2</td><td>-13.9</td><td>+35.1</td><td>+10.9</td></tr></tbody></table>

* **Single global threshold** imposes a single confidence cutoff, boosting precision and yielding the best overall F1 scores at both *k* = 1 (59.3%) and *k* = 5 (60.8%). Even a modest threshold effectively discards low-confidence labels while retaining enough moderate-confidence predictions to maintain a solid recall.

* **Per-class thresholds** slightly trail the single global threshold in F1 (58.1% vs. 60.8% at *k* = 5), but achieve broader *class coverage*. Across the 239 topics in our dataset, the single global threshold leaves 15.1% of topics without predictions, while no-threshold and per-class thresholding reduce this to 6.5% and 6.9%, respectively. The per-class thresholds approach balances filtering low-confidence predictions while *cutting topic loss by more than half*, ensuring better coverage of rare topics without excessive noise.

### 7.1.1 Comparison with LEGION

To further evaluate DRAGON we also conducted a head-to-head comparison against LEGION, the current state of the art in repository classification, under the same domain-only conditions of the previous experiments. As shown in Table 3, LEGION achieves the highest precision at both *k* = 1 (77.1%) and *k* = 5 (72.0%), but its recall is substantially lower (42.5% and 44.2%, respectively). Our single global threshold variant, by contrast, maintains a stronger balance, translating into the top F1 scores at *k* = 1 (59.3%) and *k* = 5 (60.8%). Performing better than LEGION under identical training conditions positions DRAGON as a state-of-the-art approach for large-scale repository classification in diverse environments where one cannot rely on high-signal information like README files.

*Answer to RQ1:* on domain-only labels at scale, DRAGON achieves F1@5 = 60.8%, exceeding LEGION's 54.8% by +6.0 points (Table 3).

## 7.2 Evaluation on curated repository datasets (external validity)

Although our primary objective lies in classifying large-scale collections of repositories with sparse documentation and domain-only labels, we also evaluate our pipeline on the more conventional curated dataset used in prior repository classification work (Dang et al., 2024; Izadi et al., 2021; Widyasari et al., 2023). The most recent version of this benchmark is from the LEGION paper (Dang et al., 2024).
<table><caption>Table 4: Comparison of original LEGION results vs. our own replication of them for the following metrics: precision (P), recall (R), and F1 score at *k* = 1, 3, 5. All values are percentages. Improvements are reported as relative percentages.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">k = 1</th><th colspan="3">k = 3</th><th colspan="3">k = 5</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>LEGION (orig.)</td><td>74.4</td><td>29.3</td><td>42.1</td><td>61.6</td><td>45.1</td><td>52.1</td><td>60.0</td><td>46.7</td><td>52.5</td></tr><tr><td>LEGION (repl.)</td><td>76.3</td><td>28.6</td><td>41.6</td><td>68.0</td><td>43.5</td><td>53.1</td><td>67.2</td><td>44.6</td><td>53.6</td></tr><tr><td>Improvement (%)</td><td>+2.6</td><td>-2.4</td><td>-1.2</td><td>+14</td><td>-3.5</td><td>+1.9</td><td>+12</td><td>-4.5</td><td>+2.1</td></tr></tbody></table>

<table><caption>Table 5: Comparison of state-of-the-art approaches for repository classification on the LEGION dataset. Reported values are F1 scores at *k* = 1, 3, 5. All values are percentages, and improvements are reported as relative percentages.</caption><thead><tr><th>Model</th><th>F1@1</th><th>F1@3</th><th>F1@5</th></tr></thead><tbody><tr><td>ZestXML Widyasari et al. (2023)</td><td>37.9</td><td>46.5</td><td>41.6</td></tr><tr><td>LR Izadi et al. (2021)</td><td>38.8</td><td>50.7</td><td>50.0</td></tr><tr><td>LEGION Dang et al. (2024)</td><td><strong>42.1</strong></td><td><strong>52.1</strong></td><td><strong>52.5</strong></td></tr><tr><td>DRAGON</td><td>40.8</td><td>51.5</td><td>52.1</td></tr><tr><td>Improvement (%)</td><td>-3.1</td><td>-1.2</td><td>-0.8</td></tr></tbody></table>

This dataset diverges from our intended use case in three key aspects: (1) each repository includes a *README*, (2) programming languages are included in the target labels, and (3) file tree data are missing. Even if these aspects put DRAGON at a disadvantage, testing it on an established curated benchmark helps establish its generality.

### 7.2.1 Validating our LEGION replication

As a prerequisite to comparing DRAGON with other methods, we first re-implemented LEGION, starting from its replication package and training it on the benchmark dataset. Table 4 shows the original scores from the LEGION paper compared to our replicated results. Although our implementation shows a slight drop in recall but a marginally higher overall F1 on some ranks, these numbers remain close to or exceed the originally reported performances, validating our reimplementation. These results also further validate our earlier comparison on our domain-only dataset: observed gains of DRAGON over LEGION cannot be attributed to a flawed reimplementation.

### 7.2.2 DRAGON vs. established baselines

Table 5 compares DRAGON with other representative baselines, including ZestXML (Widyasari et al., 2023), LR (Izadi et al., 2021), and LEGION (Dang et al., 2024) (original results from the paper, not our replication). Although LEGION achieves the highest F1 on this more README-rich dataset (e.g., F1@1 = 42.1%, F1@5 = 52.5%), DRAGON remains highly competitive, trailing by only 0.8% at F1@5 in relative terms. While DRAGON is not the top performer under these more polished conditions than the reality of large-scale collections, it still surpasses other strong baselines by a considerable margin.

In sum, DRAGON demonstrates near state-of-the-art performance on a benchmark where:
<table><caption>Table 6: Sentence pair BERT (SP BERT) versus traditional BERT under three thresholding strategies. Values are F1 scores at $k \in \{1, 3, 5\}$, as percentages. The last row shows the relative improvement of SP BERT over BERT.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">No Threshold</th><th colspan="3">Single Threshold</th><th colspan="3">Per-Class Threshold</th></tr><tr><th>F1@1</th><th>F1@3</th><th>F1@5</th><th>F1@1</th><th>F1@3</th><th>F1@5</th><th>F1@1</th><th>F1@3</th><th>F1@5</th></tr></thead><tbody><tr><td>BERT</td><td>57.2</td><td>42.3</td><td>31.4</td><td>58.1</td><td>59.7</td><td>59.7</td><td>57.7</td><td>57.4</td><td>56.9</td></tr><tr><td>SP BERT</td><td>58.4</td><td>42.9</td><td>31.8</td><td>59.3</td><td>60.8</td><td>60.8</td><td>59.0</td><td>58.6</td><td>58.1</td></tr><tr><td>Improvement (%)</td><td>+2.1</td><td>+1.4</td><td>+1.3</td><td>+2.1</td><td>+1.8</td><td>+1.8</td><td>+2.3</td><td>+2.1</td><td>+2.1</td></tr></tbody></table>

every repository has a README, file tree data are not available, and programming languages are part of the labels. The fact that DRAGON closely matches LEGION in such a setting, remarkably different from our target scenario, underscores its adaptability. Meanwhile, on a large-scale and “messy” corpus, DRAGON clearly surpasses LEGION, achieving higher recall and, consequently, higher F1 scores.

## 7.3 Enhancing multi-label classification for README-scarce repositories (RQ2)

In real-world repositories, documentation quality varies widely: many projects include detailed READMEs, while others provide none at all. A practical classifier must therefore remain effective under both conditions. We refer to this ability as *robustness*, meaning that the model maintains strong predictive performance when README content is missing, without requiring retraining or specialized variants. Such robustness ensures that a single model can adapt seamlessly to repositories of different completeness, which is essential for large-scale collections like Software Heritage.

To verify this, we employ a *sentence-pair BERT model* (SP-BERT), which keeps file tree and README inputs as separate segments, and compare it with a standard *BERT model* that concatenates them into a single sequence. This design addresses two real-world challenges: (1) 34% of repositories lack a README; (2) 75% of present READMEs are shorter than 200 words. Below, we compare the two approaches in three scenarios, establishing that **SP-BERT consistently outperforms its single-sequence counterpart** when README data are scarce.

### 7.3.1 Full test set results

Table 6 compares F1 scores at $k \in \{1, 3, 5\}$ for SP-BERT and standard BERT across the three thresholding strategies. The single global threshold yields the highest overall F1 in both models, but SP-BERT provides an additional gain at every $k$ (e.g., F1@5: 59.7% → 60.8%). This suggests that maintaining file tree and textual signals as distinct segments helps the classifier learn more robust features, even when READMEs are scarce.

### 7.3.2 Forced README removal

To measure the impact of keeping the two inputs separate, we evaluate a scenario where we forcibly remove READMEs from all repositories in the test set. Table 7 reports F1@1...5 for both models. Without any textual data at inference, SP-BERT still outperforms standard BERT by a clear margin (e.g., single global threshold at $k = 5$: 53.9% → 55.2%). This indicates that training the model with separate file tree and README segments encourages more robust learning from file tree data, maintaining strong performance when README is missing at test time.
<table><caption>Table 7: Forced README removal: F1 Scores at $k \in \{1, 3, 5\}$ for sentence-pair BERT (SP-BERT) and traditional BERT. All values are percentages and the last row shows the relative improvement of SP-BERT over BERT.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">No Threshold</th><th colspan="3">Single Threshold</th><th colspan="3">Per-Class Threshold</th></tr><tr><th>F1@1</th><th>F1@3</th><th>F1@5</th><th>F1@1</th><th>F1@3</th><th>F1@5</th><th>F1@1</th><th>F1@3</th><th>F1@5</th></tr></thead><tbody><tr><td>BERT</td><td>51.1</td><td>38.3</td><td>28.8</td><td>52.3</td><td>53.9</td><td>53.9</td><td>47.6</td><td>47.9</td><td>47.5</td></tr><tr><td>SP-BERT</td><td>52.4</td><td>38.9</td><td>29.2</td><td>53.6</td><td>55.2</td><td>55.2</td><td>53.1</td><td>52.6</td><td>52.1</td></tr><tr><td>Improvement (%)</td><td>+2.5</td><td>+1.6</td><td>+1.4</td><td>+2.5</td><td>+2.4</td><td>+2.4</td><td>+11.6</td><td>+9.8</td><td>+9.7</td></tr></tbody></table>

<table><caption>Table 8: Real-world README-less repositories: F1 Scores at $k \in \{1, 3, 5\}$ for sentence-pair BERT (SP-BERT) and traditional BERT. All values are percentages and the last row shows the relative improvement of SP-BERT over BERT.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">No Threshold</th><th colspan="3">Single Threshold</th><th colspan="3">Per-Class Threshold</th></tr><tr><th>F1@1</th><th>F1@3</th><th>F1@5</th><th>F1@1</th><th>F1@3</th><th>F1@5</th><th>F1@1</th><th>F1@3</th><th>F1@5</th></tr></thead><tbody><tr><td>BERT</td><td>47.6</td><td>37.8</td><td>28.9</td><td>48.4</td><td>50.7</td><td>50.7</td><td>47.6</td><td>47.9</td><td>47.5</td></tr><tr><td>SP-BERT</td><td>48.9</td><td>38.4</td><td>29.2</td><td>49.9</td><td>52.2</td><td>52.1</td><td>49.3</td><td>49.5</td><td>49.0</td></tr><tr><td>Improvement (%)</td><td>+2.7</td><td>+1.6</td><td>+1.0</td><td>+3.1</td><td>+3.0</td><td>+2.8</td><td>+3.6</td><td>+3.3</td><td>+3.2</td></tr></tbody></table>

When comparing DRAGON's performance using the entire dataset to its performance after removing the README at *test time*, the results show a decrease of only 5.7% (from F1@1 = 59.3% to 53.6%) and 5.6% (from F1@5 = 60.8% to 55.2%). In relative terms, this represents a drop of approximately 9.6% for F1@1 and 9.2% for F1@5. A similar experiment by Izadi et al. (2021) investigates the effect of removing README and wiki data, but at *training time*. Even with that tailored training regime, they reported an F1@5 drop from 47.0% to 37.4%—a relative loss of more than 20%. In contrast, our method operates *without* re-training and sees only a relative 9.2% decline.

### 7.3.3 Real-world README-less repositories

Finally, we measure performance on the *actual* subset of repositories that lack a README in our dataset (34%). This real-world scenario (Table 8) confirms the trend: SP-BERT retains higher F1 scores across all *k*, compared to standard BERT. Notably, under single global thresholding at *k* = 5, SP-BERT reaches F1 = 52.1%; standard BERT remains at 50.7%. Even on real-world README-less repositories, training with a dedicated second segment confers better robustness.

*Answer to RQ2:* DRAGON maintains robustness without retraining under README removal (F1@5 drop ≈ 9.2% relative). Across all tested conditions and thresholding policies, sentence-pair BERT (file-tree — README) achieves higher F1 than single-sequence BERT; the margin is ≥2 points at *k*=5 and is more pronounced when README is missing, underscoring the benefit of treating file-tree signals as a separate input segment for README-scarce settings.
## 7.4 Thresholding trade-offs and evaluation policy (RQ3)

We evaluate three thresholding strategies—*no threshold*, *single global threshold*, and *per-class thresholds*—across all experimental settings introduced in Sections 7.1 and 7.3. This analysis quantifies how thresholding affects model performance under different input and model conditions, including the absence of README files.

### 7.4.1 Domain-only dataset

Table 3 summarizes the main results for our large-scale *domain-only* dataset, which excludes programming languages and includes many repositories with sparse documentation.

Here, thresholding strongly impacts the balance between precision and recall. The *no-threshold* ranking achieves the highest recall (82.1% at *k* = 5) but yields the lowest F1 (31.8%), confirming that unrestricted ranking introduces substantial noise. Applying a *single global threshold* improves overall balance, reaching the best F1 scores at both *k* = 1 (59.3%) and *k* = 5 (60.8%), as low-confidence predictions are effectively filtered. *Per-class thresholds* slightly trail in F1 (58.1% at *k* = 5) but increase topic diversity: the share of unpredicted topics drops from 15.1% (single threshold) to 6.9%. Thus, threshold selection directly controls the precision-coverage trade-off.

### 7.4.2 SP-BERT vs. BERT

Table 6 compares the effect of thresholding when using *SP-BERT* versus a conventional *BERT*. For the baseline BERT model, thresholding shows the same pattern observed with SP-BERT. Without any threshold, performance remains low (*F*1@5 = 31.4%), as unfiltered rankings introduce many incorrect labels. Applying a *single global threshold* improves performance, reaching *F*1@5 = 59.7%, while *per-class thresholds* yield a slightly lower score (*F*1@5 = 56.9%).

### 7.4.3 Impact of README availability

We also examine how thresholding behaves when README content is missing, both in a controlled removal setting and on the real subset of README-less repositories. Across both conditions, absolute F1 scores decrease but the relative ordering of thresholding strategies remains unchanged. The *no-threshold* variant continues to yield the lowest F1, while introducing a *single global threshold* again provides the best overall performance. At *k* = 5, F1 under the single-threshold policy declines moderately—from 60.8% with full inputs to 55.2% after README removal, and to 52.1% on repositories that lack README files entirely—whereas *per-class thresholds* remain slightly lower.

*Answer to RQ3:* a single global threshold maximizes F1 on this dataset; per-class thresholds trade a small F1 loss for substantially higher class coverage.

# 8 Error and Semantic Similarity Analysis

We now study where errors concentrate and whether misclassifications are semantically meaningful. This section connects four pieces: distribution of errors across labels, their aggregate contribution, the role of frequency, and the semantic structure of confusions. Together these answer RQ4.

## 8.1 Distribution and nature of errors

Unless otherwise noted, we report results as recall@5 under the single threshold policy introduced in Section 5.3, because here we localize where errors arise by counting missed ground truth labels
per class. This makes it the most appropriate metric for identifying which classes are systematically underpredicted. Later in this section, we also translate recall values into absolute miss counts by multiplying with class support.

Because the test set, as the original dataset, is strongly imbalanced in the number of ground truth topics per repository, error counts should be interpreted with that distribution in mind. In particular, 69 050 repositories have a single topic (83.6%), 11 207 have two (13.6%), 1858 have three (2.3%), 340 have four (0.4%), and only 76 have five (0.1%).

### 8.1.1 Lowest recall classes

We first report the labels with the lowest absolute recall@5, restricting to classes with at least ten test instances. Table 9 lists the top-30. Most of these classes are infrequent, but some with meaningful support, such as computer vision and engineering, also show low recall. The distribution plots in Figure 6 reveal which alternative labels the model tends to predict in these cases. For example:

* computer vision (support 44, recall 0.091) is most often confused with digital image processing (52.3%), deep neural network (43.2%), and artificial neural network (40.9%).

* engineering (support 125, recall 0.376) is replaced by broad categories such as algorithm (14.4%), website (13.6%), and web application (10.4%).

* Other low recall labels such as user interface are confused with web application (31.6%), data structure (26.3%), and software design pattern (26.3%).

* natural language processing is substituted by deep neural network (17.1%), search (17.1%), and classification (14.6%).

* autonomous car tends to be replaced with robotics (32.4%), simulation (17.6%), or algorithm (14.7%).

* Even high profile research topics such as generative adversarial network are frequently confused with deep learning (50%) and artificial neural network (43.3%).

### 8.1.2 Main error contributors

The previous analysis focused on labels with the lowest recall, i.e., those the model struggles to predict correctly. However, such labels may be rare and therefore have little impact on the overall performance. We now shift focus to identify which labels contribute most to the total number of errors across the test set. To measure this, we estimate for each class c the number of ground-truth instances that were missed by the model, computed as:

$$ \text{Missed}(c) = \text{support}(c) \times (1 - \text{recall@5}(c)). $$

This quantity represents the absolute count of missed labels for each class and highlights labels whose residual errors dominate the aggregate error budget, even if their individual recall is moderate. Table 10 shows the top contributors. Broad categories such as web application, database, website, and server dominate the error budget. Machine learning appears in both the lowest recall table and the highest error contributors, reflecting its combination of difficulty and prevalence. Distribution plots in Figure 6 confirm that machine learning is often substituted with artificial neural network (26.6%), deep neural network (23.2%), or classification (16.3%).
<figure><img src="image_7.png" /><figcaption>Figure 6: Prediction distribution analysis for lowest performing labels. For each problematic label, the chart shows the percentage of correct predictions and the most frequent incorrect predictions.</figcaption></figure>
<table><caption>Table 9: Top-30 topics with the lowest recall@5 with support at least ten. The label shown in bold appears in both tables.</caption><thead><tr><th>Label</th><th>Support</th><th>Recall@5</th><th>Missed</th></tr></thead><tbody><tr><td>social network</td><td>13</td><td>0.000</td><td>13</td></tr><tr><td>artificial intelligence</td><td>20</td><td>0.000</td><td>20</td></tr><tr><td>computational biology</td><td>13</td><td>0.000</td><td>13</td></tr><tr><td>computer vision</td><td>44</td><td>0.091</td><td>40</td></tr><tr><td>graph database</td><td>11</td><td>0.091</td><td>10</td></tr><tr><td>functional programming</td><td>10</td><td>0.100</td><td>9</td></tr><tr><td>electronic trading platform</td><td>26</td><td>0.115</td><td>23</td></tr><tr><td>computer science</td><td>24</td><td>0.125</td><td>21</td></tr><tr><td>knowledge graph</td><td>15</td><td>0.133</td><td>13</td></tr><tr><td>user interface</td><td>38</td><td>0.184</td><td>31</td></tr><tr><td>telecommunications network</td><td>16</td><td>0.188</td><td>13</td></tr><tr><td>geographic information system</td><td>18</td><td>0.278</td><td>13</td></tr><tr><td>long short term memory</td><td>12</td><td>0.333</td><td>8</td></tr><tr><td>text mining</td><td>12</td><td>0.333</td><td>8</td></tr><tr><td>generative model</td><td>24</td><td>0.333</td><td>16</td></tr><tr><td>collaborative filtering</td><td>20</td><td>0.350</td><td>13</td></tr><tr><td>augmented reality</td><td>16</td><td>0.375</td><td>10</td></tr><tr><td>engineering</td><td>125</td><td>0.376</td><td>78</td></tr><tr><td>semantic web</td><td>47</td><td>0.383</td><td>29</td></tr><tr><td>virtual reality</td><td>52</td><td>0.385</td><td>32</td></tr><tr><td>reverse engineering</td><td>36</td><td>0.389</td><td>22</td></tr><tr><td>natural language processing</td><td>41</td><td>0.390</td><td>25</td></tr><tr><td>operating system</td><td>10</td><td>0.400</td><td>6</td></tr><tr><td>generative adversarial network</td><td>30</td><td>0.400</td><td>18</td></tr><tr><td>image editing</td><td>17</td><td>0.412</td><td>10</td></tr><tr><td>autonomous car</td><td>34</td><td>0.412</td><td>20</td></tr><tr><td>autonomous driving</td><td>14</td><td>0.429</td><td>8</td></tr><tr><td><strong>machine learning</strong></td><td><strong>319</strong></td><td><strong>0.433</strong></td><td><strong>181</strong></td></tr><tr><td>face detection</td><td>20</td><td>0.450</td><td>11</td></tr><tr><td>hyperparameter optimization</td><td>51</td><td>0.451</td><td>28</td></tr></tbody></table>

### 8.1.3 Concentration and frequency effects

Errors are not uniformly spread. The top-30 error contributing labels account for 7139 of 17695 misses, that is 40.3% of all errors, and the top-10 alone account for 17.5%. In parallel, label frequency is a first order driver of performance: across the 239 labels we observe a strong positive association between log support and recall@5 with Pearson correlation of 0.859. Thus, frequency improves recall on average yet also magnifies the absolute number of misses whenever any residual error remains. Rare classes are systematically brittle. Among labels with average support near 6, the mean recall@1 is 0.034 and the mean recall@5 is 0.133, with 33 labels at recall@1 equal to zero and 23 labels at recall@5 equal to zero.

## 8.2 Semantic structure of confusions

We next examine whether misclassifications are semantically close to the target labels. This connects the error distributions above with the semantic organization of the label space.
<figure><img src="image_8.png" /><figcaption>Figure 7: Each cell shows the proportion of true instances of a label predicted as another, after removing the diagonal. Rows are normalized *before* diagonal removal, so values represent actual error rates per class rather than relative shares among errors, highlighting how much of each class leaks into others.</figcaption></figure>
<table><caption>Table 10: Top-30 error contributing topics ranked by number of misses with support at least ten. The label shown in bold appears in both tables.</caption><thead><tr><th>Label</th><th>Support</th><th>Recall@5</th><th>Missed</th></tr></thead><tbody><tr><td>web application</td><td>1763</td><td>0.778</td><td>391</td></tr><tr><td>database</td><td>2222</td><td>0.830</td><td>377</td></tr><tr><td>website</td><td>4144</td><td>0.911</td><td>367</td></tr><tr><td>server</td><td>1231</td><td>0.755</td><td>301</td></tr><tr><td>data</td><td>982</td><td>0.697</td><td>298</td></tr><tr><td>automation</td><td>1318</td><td>0.775</td><td>297</td></tr><tr><td>algorithm</td><td>2916</td><td>0.906</td><td>273</td></tr><tr><td>search</td><td>758</td><td>0.646</td><td>268</td></tr><tr><td>computer programming</td><td>693</td><td>0.620</td><td>263</td></tr><tr><td>security</td><td>1267</td><td>0.800</td><td>254</td></tr><tr><td>visualization</td><td>1259</td><td>0.799</td><td>253</td></tr><tr><td>education</td><td>558</td><td>0.548</td><td>252</td></tr><tr><td>front end</td><td>1365</td><td>0.818</td><td>248</td></tr><tr><td>back end</td><td>873</td><td>0.729</td><td>237</td></tr><tr><td>software testing</td><td>1167</td><td>0.805</td><td>227</td></tr><tr><td>web application security</td><td>407</td><td>0.459</td><td>220</td></tr><tr><td>mathematics</td><td>1007</td><td>0.799</td><td>202</td></tr><tr><td>statistics</td><td>797</td><td>0.749</td><td>200</td></tr><tr><td>game</td><td>5085</td><td>0.961</td><td>199</td></tr><tr><td>animation</td><td>1013</td><td>0.808</td><td>195</td></tr><tr><td>client</td><td>571</td><td>0.664</td><td>192</td></tr><tr><td>design</td><td>594</td><td>0.687</td><td>186</td></tr><tr><td>mathematical optimization</td><td>613</td><td>0.701</td><td>183</td></tr><tr><td>analytics</td><td>529</td><td>0.658</td><td>181</td></tr><tr><td><strong>machine learning</strong></td><td><strong>319</strong></td><td><strong>0.433</strong></td><td><strong>181</strong></td></tr><tr><td>digital image processing</td><td>1141</td><td>0.842</td><td>180</td></tr><tr><td>image</td><td>886</td><td>0.798</td><td>179</td></tr><tr><td>real time computing</td><td>426</td><td>0.580</td><td>179</td></tr><tr><td>websocket</td><td>1041</td><td>0.829</td><td>178</td></tr><tr><td>classification</td><td>846</td><td>0.790</td><td>178</td></tr></tbody></table>

### 8.2.1 Observed confusions

Figure 7 is a confusion matrix restricted to the thirty most frequent classes. The figure shows that many wrong predictions are semantically close to the true label. For instance, repositories annotated as *web application* are often predicted as *website* or *server*, and *database* is often replaced by *data* or *analytics*. For low recall classes, distribution plots in Figure 6 highlight that the model confuses *natural language processing* with *deep neural network* and *classification*, *autonomous car* with *robotics*, and *computer vision* with *digital image processing*. These substitutions indicate that errors concentrate within topical neighborhoods rather than being unrelated.

### 8.2.2 Critical groups of labels

Taken together, the previous results delineate where effort should focus next. The findings consistently point to three groups:

1. Rare classes with low recall such as *artificial intelligence* or *computational biology*.
2. Common classes with moderate deficits that generate many absolute errors such as *web application*, *database*, and *machine learning*.

3. Overlapping classes prone to mutual confusion such as *web application* versus *website*, or *computer vision* versus *digital image processing*.

This typology motivates the next analysis, where we examine whether these confusions reflect
meaningful semantic proximity rather than arbitrary mistakes.

8.2.3 Semantic similarity of labels

To examine whether these confusions reflect semantic proximity, we embed all label names with a SentenceTransformer and compute cosine similarity between pairs. The similarity matrix reveals clusters of related terms. High scoring examples include *debugger* and *debugging* with 0.91, *deep learning* and *deep neural network* with 0.87, *data visualization* and *visualization* with 0.86, and *autonomous car* and *autonomous driving* with 0.85. These are candidates for merging or aliasing prior to training and evaluation.

8.2.4 Similarity and confusion

For the thirty most frequent labels we correlate pairwise similarity with empirical confusion counts. We find a positive and statistically significant association (Pearson *r* = 0.34, Spearman *ρ* = 0.39, *p* < 10<sup>-25</sup>). In other words, labels that are closer in embedding space are more likely to be substituted in predictions. Examples include *video game development* predicted as *game* in 487 cases with similarity 0.48, and *deep neural network* predicted as *artificial neural network* in 213 cases with similarity 0.72. Other high similarity confusions include *front end* versus *back end* with 0.76 and *data* versus *data structure* with 0.70.

8.2.5 Synonym aware evaluation

Having established that confusions correlate with similarity, we now examine whether the most error-prone classes occur in semantically dense regions of the label space. To quantify this, we measure the average embedding similarity among the top error-contributing classes and compare it to the global label space. Labels that dominate the error budget tend to occur in dense semantic regions. The mean similarity among the top error contributing classes is 0.3354, compared to 0.2025 across all label pairs. Between 14% and 16.4% of all apparent errors can be reinterpreted as synonym confusions if pairs with similarity above 0.5 are treated as equivalent, which indicates that many errors are near misses in meaning.

Building on this observation, we perform a relaxed evaluation that explicitly accounts for such semantic proximity by accepting synonyms above a similarity threshold as correct. This allows us to estimate how many of the observed “errors” are actually semantically valid alternatives rather than true misclassifications. To help interpret the effect of different thresholds more clearly, Figure 8 visualizes the changes in precision, recall, and F1 across top-k settings (k = 1, 3, 5). Gains persist across different similarity thresholds: from 0.5 to 0.7. At 0.5, precision@5 increases from 0.197 to 0.314 and recall@5 from 0.821 to 0.880 with 48245 corrections. At 0.6, precision@5 rises to 0.266 and recall@5 to 0.861 with 28438 corrections. Even at 0.7, precision@5 improves to 0.233 and recall@5 to 0.845 with 14883 corrections. Moreover, false positives are often close to the ground truth in embedding space. At rank one, the mean similarity of false positives to their true labels is 0.345 with 14.6% above 0.5, and the same pattern holds for ranks three and five with 16.2% and 14.6%.
<figure><img src="image_9.png" /><figcaption>Figure 8: Comparison of evaluation methods across top-k settings. Rows correspond to metrics and columns to top-k values. Bars show scores for semantic-aware thresholds alongside Baseline (No Threshold), Single-Threshold, and Dynamic-Threshold methods. Numeric labels report absolute scores; annotations above bars report percent improvement relative to the Baseline.</figcaption></figure>
*Answer to RQ4:* frequent labels achieve higher recall on average, yet because they appear often and lie in dense semantic neighborhoods, their residual mistakes account for most of the absolute error budget and are frequently near synonyms of the ground truth. Between 14% and 16.4% at rank five can be reclassified as acceptable under synonym aware evaluation, and similarity correlates with confusion, indicating that many residual errors are semantically close rather than arbitrary.

# 9 Discussion

## 9.1 On the need of a “messy”, real-world dataset for large-scale repository classification

The performances of automated classifiers depend heavily on the dataset(s) they are tested on. When initially faced with the need of classifying repositories from very large real-world collections, we quickly realized that previous approaches in the state-of-the-art had not been benchmarked on comparable datasets, resulting in unmet expectations.

To address this, we assembled a realistic dataset, corresponding to the target problem.

The first design choice behind the dataset is to exclude programming languages as target labels because, on the one hand, they are trivial to detect with existing heuristics and tools and, on the other, their presence would inflate performance metrics without yielding deeper domain insights about input repositories.

The second design choice is to avoid relying on all user-provided labels, like GitHub topics, because they can be inconsistent, redundant, or both. Instead, we adopted the curated taxonomy of GitRanking (Sas et al., 2023), which offers a structured set of meaningful domain-specific topics, mitigating the noise associated with crowdsourced annotations.

Third, most existing approaches rely heavily on README content, in contrast with the real-world of large repository collections where README files are often either absent (34% of the repositories in our dataset lack a README, up to 42% of the entire SWH archive) or very short. By incorporating these README-scarce repositories in our dataset, we encourage the development of classifiers that perform well in README-scarce contexts.

Finally, we also wanted a *both large-scale and real-world dataset*, in order to be able to scale to scenarios like SWH, where 96% of the archived 350 million repositories do not have any topic.

Testing DRAGON and baselines on different benchmarks shed light on the applicability of repository classifiers in different contexts.

LEGION (Dang et al., 2024) achieves slightly higher scores on a polished dataset where every repository has a README and topics include programming languages. But on a real-world “messy” benchmark, with scarce README information and only predicting high-value topics, DRAGON outperforms LEGION by 8.2% at F1@1 and 10.9% at F1@5 in relative terms, ultimately achieving F1@1 = 59.3% and F1@5 = 60.8%. These results underscore DRAGON’s robustness in the real-world conditions of large repository collections, where other methods have never been tested.

Significant work went in the creation of our dataset, which is an important byproduct of the development of DRAGON; **we strongly encourage future work on repository classification to benchmark against it.**

## 9.2 Sentence-pair BERT is key for handling README scarcity

Our evaluation in Section 7.3 shows that sentence-pair BERT improves classification performance when README content is missing. Across all test scenarios (full dataset, forced README removal, real-world README-less repositories) DRAGON outperforms standard BERT concatenation. The
effect is particularly evident when READMEs are absent, where sentence-pair modeling effectively leverages file-tree data, whereas naive concatenation in traditional BERT leads to a significant drop in predictive power.

A key advantage of DRAGON is its ability to maintain classification performance without requiring specialized re-training. In contrast, Izadi et al. (2021) observed that removing README and wiki content during training led to an F1@5 drop from 47.0% to 37.4%—a relative decline of 20%. DRAGON, even *without re-training*, experiences only a 9.2% F1@5 relative loss when README files are removed at test time, nearly halving the gap in a much more challenging scenario. This demonstrates how effectively DRAGON exploits file-tree data, making it more robust for real-world collections where READMEs are scarce.

These results suggest that the file-tree structure of a repository provides a latent signal sufficient for meaningful classification, even without textual descriptions. Thanks to its sentence-pair architecture, DRAGON captures these signals more effectively than prior approaches. This aligns with findings from related work in different contexts (Rokon et al., 2021; Zhang et al., 2019), where separating structural and textual information also improved software representation learning.

## 9.3 Thresholding strategies: precision vs. coverage trade-offs

The analysis of thresholding strategies in Section 7.1 reveals that single-threshold filtering consistently yields the best overall F1 scores. This finding aligns with prior work from Dang et al. (2024), showing that filtering out low-confidence predictions helps prevent spurious topic assignments.

However, our per-class thresholding approach provides a crucial advantage: broader class coverage. While it slightly underperforms in raw F1 score (58.1% vs. 60.8% at F1@5), it ensures that more distinct topics appear in model predictions. The impact is significant: the number of unpredicted topics *shrinks by more than half* (15.1% vs 6.9%). This suggests a precision vs. class coverage trade-off: stricter filtering improves accuracy at the cost of reducing diversity in the assigned labels. Our approach allows prospective users to reason on this tradeoff, which depends on the target application: if precision is paramount (e.g., automated tagging systems), a single-threshold approach is preferable; if maximizing class coverage is critical (e.g., exploratory search engines), per-class thresholds may be more appropriate. Finally, by pointing out the issue, we encourage future work to improve on this target.

## 9.4 What errors and similarity tell us about evaluation and utility

The error analysis shows two concentrations of mistakes. First, rare labels have low recall even with careful sampling and focal loss. Second, broad and frequent labels such as *web application* or *database* accumulate many absolute errors even when recall is acceptable.

The similarity analysis clarifies that many substitutions are semantically close to the ground truth. Confusions are more likely between labels that are close in embedding space, and the classes that dominate the error budget lie in dense semantic regions. When we accept close synonyms above a fixed similarity threshold, between 14% and 16.4% of apparent errors at rank five are reclassified as acceptable matches. This complementary view does not replace strict metrics but helps interpret them: strict metrics favor conservative selectivity, whereas synonym aware views favor semantic coverage. For future works, a hierarchical or merged taxonomy for very close concepts could reduce artificial confusion and improve interpretability while keeping strict metrics as the primary score.
## 9.5 Why Top-5 suggestions are useful with sparse ground truth

Many repositories in our data have very few annotated topics (83.6% have 1 topic, 13.6% have 2, 2.3% have 3, 0.4% have 4 and only 0.1% have 5). This sparsity means that the ground truth often captures only a narrow slice of a project. In practice, a model can propose additional labels that are reasonable for the same repository even if they are not listed in the annotations.

Our similarity analysis shows that a large share of apparent errors are semantically close to the target labels. Hence, a short list of Top five suggestions is valuable in real workflows such as tagging, curation, or search enrichment. Even when not all five match the annotated ground truth, several suggestions are near synonyms or closely related concepts and therefore useful to users.

## 9.6 Threats to validity

While DRAGON is an effective large-scale classification pipeline, some factors may impact the generalizability, fairness, and reproducibility of our results.

### 9.6.1 Internal validity

A primary concern lies in the quality of the ground-truth labels. Our dataset is derived from user-assigned GitHub topics mapped through the GitRanking taxonomy. These annotations may be incomplete, repositories can lack valid topics or be noisy, with incorrect or overly generic labels. Such inconsistencies could bias reported metrics by overstating or understating true performance. Although GitRanking reduces this risk through curation and normalization, it cannot eliminate the original label noise entirely.

Another potential threat is the reliance on a single train-validation-test split. As is customary for large-scale transformer-based models, we do not perform multiple resampling runs. This may cause moderate variance in reported scores depending on the initial data partitioning. Future work could explore repeated stratified sampling or cross-validation to assess the sensitivity of results.

Finally, while focal loss mitigates class imbalance, some residual bias toward frequent topics may persist, affecting per-class recall estimates.

To ensure methodological transparency, we used standard evaluation metrics (scikit-learn), the official Hugging Face BERT implementation, and verified our replication of LEGION against its original results. All code, data, and trained models are publicly available through our replication package (Balla et al., 2025).

### 9.6.2 External validity

While our dataset is designed to reflect large-scale software archives with missing or minimal README files, models trained on it might underperform on repositories that rely heavily on curated textual documentation.

To mitigate this, we also tested our methodology on the benchmark dataset used in prior work.

## 10 Conclusion

In this article, we introduced DRAGON, a robust and scalable multi label repository classifier that assigns high value, domain specific topics at scale. DRAGON excludes programming languages since they are easy to infer by other means, and instead targets application and domain labels that are more useful for discovery. The method operates with light inputs that are preserved in version
control, combining file and directory names with optional README text through a sentence pair BERT encoder, and addresses class imbalance with focal loss.

The approach is designed for scarce data scenarios that arise in exhaustive archives, where source code level mining is impractical at scale and READMEs are frequently absent. In Software Heritage, 42% of repositories lack a README, yet DRAGON remains effective without retraining, showing about a 9% relative drop in F1@5 when README is removed at inference. On domain only labels and at large scale, DRAGON achieves F1@5 equal to 60.8% and improves over the best prior approach by about 11% in relative terms. An analysis of errors shows that most mistakes are semantic near misses among closely related topics. Thresholding studies further show that a single global threshold maximizes micro F1, while per class thresholds can be used to increase topic diversity and class coverage, which is valuable for exploratory search.

A key contribution of this work is the largest dataset to date for repository topic classification, with more than 825 thousand repositories mapped to 239 GitRanking topics. We release the dataset together with code and models to support replication and reuse. Since these 825 thousand repositories represent the entire set of labeled projects among more than 200 million preserved in Software Heritage, over 99% of repositories remain unlabeled today, which highlights the need for automated classification approaches such as ours.

Few limitations remain. Some labels are rare or overlapping which complicates strict evaluation. The tokenizer and cleaning choices favor English, which can affect multilingual projects. Future work includes taxonomy consolidation to reduce overlap, multilingual modeling, strategies for the long tail, confidence calibration, and human in the loop workflows. Taken together, these directions can further strengthen DRAGON as a practical building block for navigating and mining very large collections of open source software.
## References

Balla S, Zacchiroli S, Degueule T, Falleri JR, Robbes R (2025) Replication package for dragon: Robust classification for very large collections of software repositories. DOI 10.5281/zenodo.15020642, URL https://doi.org/10.5281/zenodo.15020642

Baltrusaitis T, Ahuja C, Morency L (2019) Multimodal machine learning: A survey and taxonomy. IEEE Trans Pattern Anal Mach Intell 41(2):423-443, DOI 10.1109/TPAMI.2018.2798607, URL https://doi.org/10.1109/TPAMI.2018.2798607

Boldi P, Pietri A, Vigna S, Zacchiroli S (2020) Ultra-large-scale repository analysis via graph compression. In: Kontogiannis K, Khomh F, Chatzigeorgiou A, Fokaefs M, Zhou M (eds) 27th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2020, London, ON, Canada, February 18-21, 2020, IEEE, pp 184-194, DOI 10.1109/SANER48275.2020.9054827, URL https://doi.org/10.1109/SANER48275.2020.9054827

Dang YT, Le-Cong T, Nguyen PT, Bui AMT, Nguyen PT, Le B, Huynh QT (2024) Legion: Harnessing pre-trained language models for github topic recommendations with distribution-balance loss. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, Association for Computing Machinery, New York, NY, USA, EASE '24, p 181-190, DOI 10.1145/3661167.3661168, URL https://doi.org/10.1145/3661167.3661168

Devlin J, Chang M, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein J, Doran C, Solorio T (eds) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Association for Computational Linguistics, pp 4171-4186, DOI 10.18653/V1/N19-1423, URL https://doi.org/10.18653/v1/n19-1423

Di Cosmo R, Zacchiroli S (2017) Software heritage: Why and how to preserve software source code. In: iPRES 2017: 14th International Conference on Digital Preservation

Di Rocco J, Di Ruscio D, Di Sipio C, Nguyen P, Rubei R (2020) TopFilter: An Approach to Recommend Relevant GitHub Topics. In: Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), ACM, Bari Italy, pp 1-11, DOI 10.1145/3382494.3410690, URL https://dl.acm.org/doi/10.1145/3382494.3410690

Di Rocco J, Di Ruscio D, Di Sipio C, Nguyen PT, Rubei R (2023) HybridRec: A recommender system for tagging GitHub repositories. Applied Intelligence 53(8):9708-9730, DOI 10.1007/s10489-022-03864-y, URL https://doi.org/10.1007/s10489-022-03864-y

Di Sipio C, Rubei R, Di Ruscio D, Nguyen PT (2020) A Multinomial Naïve Bayesian (MNB) Network to Automatically Recommend Topics for GitHub Repositories. In: Proceedings of the 24th International Conference on Evaluation and Assessment in Software Engineering, Association for Computing Machinery, New York, NY, USA, EASE '20, pp 71-80, DOI 10.1145/3383219.3383227, URL https://dl.acm.org/doi/10.1145/3383219.3383227

Ding T, Angelopoulos A, Bates S, Jordan MI, Tibshirani RJ (2023) Class-conditional conformal prediction with many classes. In: Oh A, Naumann T, Globerson A, Saenko K, Hardt M, Levine S (eds) Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA,
USA, December 10 - 16, 2023, URL http://papers.nips.cc/paper_files/paper/2023/hash/cb931eddd563f8d473c355518ce8601c-Abstract-Conference.html

Feng Z, Guo D, Tang D, Duan N, Feng X, Gong M, Shou L, Qin B, Liu T, Jiang D, Zhou M (2020) Codebert: A pre-trained model for programming and natural languages. In: Cohn T, He Y, Liu Y (eds) Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, Association for Computational Linguistics, Findings of ACL, vol EMNLP 2020, pp 1536-1547, DOI 10.18653/V1/2020.FINDINGS-EMNLP.139, URL https://doi.org/10.18653/v1/2020.findings-emnlp.139

Fratantonio Y, Invernizzi L, Farah L, Thomas K, Zhang M, Albertini A, Galilee F, Metitieri G, Cretin J, Petit-Bianco A, et al. (2024) Magika: Ai-powered content-type detection. arXiv preprint arXiv:240913768

Gao X, Cao B, Zhu P, Wang N, Hu Q (2025) Asymmetric reinforcing against multi-modal representation bias. In: Walsh T, Shah J, Kolter Z (eds) AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025, Philadelphia, PA, USA, AAAI Press, pp 16754-16762, DOI 10.1609/AAAI.V39I16.33841, URL https://doi.org/10.1609/aaai.v39i16.33841

Izadi M, Heydarnoori A, Gousios G (2021) Topic recommendation for software repositories using multi-label classification algorithms. Empirical Software Engineering 26(5):93, DOI 10.1007/s10664-021-09976-2, URL https://doi.org/10.1007/s10664-021-09976-2

Izadi M, Nejati M, Heydarnoori A (2023) Semantically-enhanced topic recommendation systems for software projects. Empirical Software Engineering 28(2):50, DOI 10.1007/s10664-022-10272-w, URL https://doi.org/10.1007/s10664-022-10272-w

Kawaguchi S, Garg P, Matsushita M, Inoue K (2004) MUDABlue: an automatic categorization system for open source repositories. In: 11th Asia-Pacific Software Engineering Conference, IEEE, Busan, Korea, pp 184-193, DOI 10.1109/APSEC.2004.69, URL https://ieeexplore.ieee.org/abstract/document/1371919, iSSN: 1530-1362

LeClair A, Eberhart Z, McMillan C (2018) Adapting Neural Text Classification for Improved Software Categorization. In: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, Madrid, Spain, pp 461-472, DOI 10.1109/ICSME.2018.00056, URL https://ieeexplore.ieee.org/document/8530052, iSSN: 2576-3148

Lin TY, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp 2999-3007, DOI 10.1109/ICCV.2017.324

Linares-Vásquez M, McMillan C, Poshyvanyk D, Grechanik M (2014) On using machine learning to automatically classify software applications into domain categories. Empirical Software Engineering 19(3):582-618, DOI 10.1007/s10664-012-9230-z, URL https://doi.org/10.1007/s10664-012-9230-z

Linstead E, Rigor P, Bajracharya S, Lopes C, Baldi P (2007) Mining concepts from code with probabilistic topic models. In: Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, ACM, Atlanta Georgia USA, pp 461-464, DOI 10.1145/1321631.1321709, URL https://dl.acm.org/doi/10.1145/1321631.1321709
Lipton ZC, Elkan C, Narayanaswamy B (2014) Thresholding classifiers to maximize f1 score. arXiv preprint arXiv:14021892

Ma Y, Fakhoury S, Christensen M, Arnaoudova V, Zogaan W, Mirakhorli M (2018) Automatic classification of software artifacts in open-source applications. In: Proceedings of the 15th International Conference on Mining Software Repositories, ACM, Gothenburg Sweden, pp 414–425, DOI 10.1145/3196398.3196446, URL https://dl.acm.org/doi/10.1145/3196398.3196446

Ma Y, Bogart C, Amreen S, Zaretzki R, Mockus A (2019) World of code: an infrastructure for mining the universe of open source VCS data. In: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), IEEE, pp 143–154

McMillan C, Linares-Vásquez M, Poshyvanyk D, Grechanik M (2011) Categorizing software applications for maintenance. In: 2011 27th IEEE International Conference on Software Maintenance (ICSM), IEEE, Williamsburg, USA, pp 343–352, DOI 10.1109/ICSM.2011.6080801, URL https://ieeexplore.ieee.org/abstract/document/6080801, iSSN: 1063-6773

Nagappan M, Zimmermann T, Bird C (2013) Diversity in software engineering research. In: Meyer B, Baresi L, Mezini M (eds) Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE’13, Saint Petersburg, Russian Federation, August 18-26, 2013, ACM, pp 466–476, DOI 10.1145/2491411.2491415, URL https://doi.org/10.1145/2491411.2491415

Narasimhan H, Menon AK (2021) Training over-parameterized models with non-decomposable objectives. In: Ranzato M, Beygelzimer A, Dauphin YN, Liang P, Vaughan JW (eds) Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp 18165–18181, URL https://proceedings.neurips.cc/paper/2021/hash/9713faa264b94e2bf346a1bb52587fd8-Abstract.html

Pietri A, Spinellis D, Zacchiroli S (2019) The software heritage graph dataset: public software development under one roof. In: Storey MD, Adams B, Haiduc S (eds) Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019, 26-27 May 2019, Montreal, Canada, IEEE / ACM, pp 138–142, DOI 10.1109/MSR.2019.00030, URL https://doi.org/10.1109/MSR.2019.00030

Rokon MOF, Yan P, Islam R, Faloutsos M (2021) Repo2vec: A comprehensive embedding approach for determining repository similarity. In: 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp 355–365, DOI 10.1109/ICSME52107.2021.00038

Sanh V, Debut L, Chaumond J, Wolf T (2019) Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR abs/1910.01108, URL http://arxiv.org/abs/1910.01108, 1910.01108

Sas C, Capiluppi A (2022) Antipatterns in software classification taxonomies. J Syst Softw 190:111343, DOI 10.1016/J.JSS.2022.111343, URL https://doi.org/10.1016/j.jss.2022.111343

Sas C, Capiluppi A, Di Sipio C, Di Rocco J, Di Ruscio D (2023) GitRanking: A ranking of GitHub topics for software classification using active sampling. Software: Practice and Experience 53(10):1982–2006, DOI 10.1002/spe.3238, URL https://onlinelibrary.wiley.com/doi/10.1002/spe.3238
Soll M, Vosgerau M (2017) ClassifyHub: An Algorithm to Classify GitHub Repositories. In: Kern-Isberner G, Fürnkranz J, Thimm M (eds) KI 2017: Advances in Artificial Intelligence, Springer International Publishing, Cham, Lecture Notes in Computer Science, pp 373–379, DOI 10.1007/978-3-319-67190-1_34

Tian K, Revelle M, Poshyvanyk D (2009) Using Latent Dirichlet Allocation for automatic categorization of software. In: 2009 6th IEEE International Working Conference on Mining Software Repositories, IEEE, Vancouver, Canada, pp 163–166, DOI 10.1109/MSR.2009.5069496, URL https://ieeexplore.ieee.org/document/5069496/?arnumber=5069496, iSSN: 2160-1860

Trujillo MZ, Hébert-Dufresne L, Bagrow J (2022) The penumbra of open source: projects outside of centralized platforms are longer maintained, more academic and more collaborative. EPJ Data Science 11(1):31

Ugurel S, Krovetz R, Giles CL (2002) What’s the code?: automatic classification of source code archives. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 23-26, 2002, Edmonton, Alberta, Canada, ACM, pp 639–644, DOI 10.1145/775047.775141, URL https://doi.org/10.1145/775047.775141

Widyasari R, Zhao Z, Cong TL, Jin Kang H, Lo D (2023) Topic Recommendation for GitHub Repositories: How Far Can Extreme Multi-Label Learning Go? In: 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, Taipa, Macao, pp 167–178, DOI 10.1109/SANER56733.2023.00025, URL https://ieeexplore.ieee.org/abstract/document/10123501, iSSN: 2640-7574

Wright SJ (2015) Coordinate descent algorithms. Math Program 151(1):3–34, DOI 10.1007/S10107-015-0892-3, URL https://doi.org/10.1007/s10107-015-0892-3

Yusof Y, Rana OF (2010) Classification of Software Artifacts Based on Structural Information. In: Setchi R, Jordanov I, Howlett RJ, Jain LC (eds) Knowledge-Based and Intelligent Information and Engineering Systems, Springer, Berlin, Heidelberg, pp 546–555, DOI 10.1007/978-3-642-15384-6_58

Zhang Y, Xu FF, Li S, Meng Y, Wang X, Li Q, Han J (2019) Higitclass: Keyword-driven hierarchical classification of github repositories. In: Wang J, Shim K, Wu X (eds) 2019 IEEE International Conference on Data Mining, ICDM 2019, Beijing, China, November 8-11, 2019, IEEE, pp 876–885, DOI 10.1109/ICDM.2019.00098, URL https://doi.org/10.1109/ICDM.2019.00098