DOI: 10.5281/zenodo.15412179
title: An Empirical Study on the Accuracy of GitHub's Dependency Graph and the Nature of Its Inaccuracy: A Replication Package

description: Replication Package

This is the Replication Package for the paper:“An Empirical Study on the Accuracy of GitHub's Dependency Graph and the Nature of Its Inaccuracy”

Below is a detailed explanation of the directory structure, contents, and instructions for running the scripts to replicate the study.



Prerequisites



Python ≥ 3.8

R (for statistical analysis)

GitHub Personal Access Token (used in API-based scripts)

Required Python packages: pandas, openpyxl, requests, etc.


Before starting a new analysis, delete all previously generated files in the assets directories. This avoids issues like “file already exists” or name conflicts during script execution.



Directory Structure

▸ java/ & python/

Each of these contains data and scripts specific to Java and Python repositories, respectively.



java/
├── assets/           # Input/output CSV/XLSX files
└── script/           # Scripts to gather and analyze data

python/
├── assets/
└── script/


Subfolders:



assets/: Contains datasets and analysis results.

script/: Python scripts to extract, filter, analyze, and validate GitHub dependency data.


▸ rq1_analysis/

Contains scripts and output files used for analyzing Research Question 1 (RQ1).



rq1_analysis/
├── scripts/
│   ├── rq1_analysis.py
│   └── statistics.R
└── assets/
    ├── java_repo_metrics.xlsx
    └── python_repo_metrics.xlsx


▸ rq2_analysis/

Contains scripts, results, and samples related to Research Question 2 (RQ2).



rq2_analysis/
├── assets/
│   ├── *.csv
│   └── *.xlsx
├── sample_data/
│   ├── *.csv
│   └── *.xlsx
└── script/
    ├── 1.bw_sample_extraction.py
    └── ...




Script Execution Order

Each language has its own scripts but they follow the same logic:



Generate a GitHub token and add it to scripts (InsertYourToken)

Run scripts in numerical order (e.g., 1_, 2_, … 11_)

Inspect output files in /assets after each stage


Scripts handle dependency graph retrieval, forward/backward analysis, and manifest/lock file matching.



Data Gathering and Analysis

Java and Python Data Gathering

Folder: /{prg_lang}/

Scripts:



1_dependencies_repositories_retriever.py : Retrieves dependency's repositories from GitHub Dependency Graph.

2_backward_analysis.py : Performs backward analysis on the retrieved data.

3_dependencies_filtering.py : Filters dependencies based on specific criteria (e.g., Package Manager).

4_backward_results_analysis.py: Extract a summary of the backward analysis aggregate by error cases.

5_dependents_repositories_retriever.py: Retrieves dependent's repositories from GitHub Dependency Graph.

6_dependents_count.py: Counts the number of dependents for each repository.

7_forward_analysis.py: Performs forward analysis on the retrieved data.

8_forward_results_analysis.py: Extract a summary of the forward analysis aggregate by error cases.

9_manifest_lock_extraction.py: Extracts the manifest/lock file information from the dependencies retrieved in the 1_dependencies_repositories_retriever.py by GitHub's dependency graph.

10_manifest_lock_dependencies_extraction.py: Retrieves raw file of the manifest/lock file from GitHub and then parse these to extract the dependencies.

11_manifest_lock_to_dg_match.py: Matches the dependencies retrieved from manifest/lock files with the dependencies retrieved from GitHub's dependency graph.




RQ1 Analysis

Folder: /rq1_analysis/

Scripts:



rq1_analysis.py: Executes RQ1 analysis pipeline

statistics.R: Performs Chi-squared, Fisher’s Exact Test, Krippendorff’s alpha


Output:



Excel files (e.g., *_repo_metrics.xlsx, *_match_count.xlsx) with per-language statistics used in the paper’s tables and figures.




RQ2 Analysis

Folder: /rq2_analysis/

This folder contains data and scripts used to validate and analyze the findings for Research Question 2, especially around error types and their frequency.

Structure:



rq2_analysis/
├── assets/            # Intermediate and final results (CSV/XLSX)
│   ├── bw_occurence_not_found.csv
│   ├── fw_analysis_results.csv
│   ├── kripp_test_bw.xlsx
│   └── ...
│
├── sample_data/       # Data samples for manual verification and statistics
│   ├── bw_analysis.csv
│   ├── non_match_sample.csv
│   └── ...
│
└── script/            # Python scripts used for sampling, result aggregation, and enrichment
    ├── 1.bw_sample_extraction.py
    ├── 5.fw_sample_extraction.py
    ├── 8.fw_final_results.py
    └── 10.match_add_link.py


Scripts:



1.bw_sample_extraction.py: Random sampling of backward error cases

2.bw_sample_dep_link.py: Add dependency links for backward analysis

3.bw_pkg_not_found_analysis.py: Analyze packages not found in backward analysis

4bw_pkg_not_found_results.py: Generate results for packages not found

5.fw_sample_extraction.py: Random sampling of forward error cases

6.fw_analysis.py: Processes forward analysis output

7.fw_res_manipulation.py: Manipulates forward analysis results

8.fw_final_results.py: Finalizes forward analysis results

9.match_add_date.py: Adds temporal dimension to matching results

10.match_add_date.py: Adds date information to matching results

11.match_add_link.py: Adds links to matching results


Notes:



Data in sample_data/ includes manually labeled samples.

Output files in assets/ include the output of each manual/automated analysis referenced in the tables of the paper.




Statistical Analysis (R)



Performed via rq1_analysis/statistics.R

Tests used:



Chi-squared test (for distribution comparisons)

Fisher’s exact test (for 2x2 tables)

Krippendorff’s alpha (for inter-rater agreement)






Tips



Make sure paths are OS-consistent (Windows paths may need double-backslashes or raw strings).

If Excel files don't open or save properly, check for open handles in Python.

Use a virtual environment (venv) to manage dependencies cleanly.




For any questions, please refer to the comments inside each script or contact the authors of the study.