Package: magi-dataset
MAGI DatasetInstallpipinstallmagi_datasetIf you plan on using magi_dataset to periodically crawl data, set the following variables in your environment:exportGH_TOKEN="Your token"ReadCreating a personal access tokenfor more information on creating GitHub personal access token. If using the default data without crawling new data, you may safely ignore this token. You can either provide the GitHub token usinggh_tokenargument when initializing theGitHubDatasetobject, or setting it as an environment variableGH_TOKENin your shell. If neither provided, the GitHub API will be initialized with no token, and the rate limit will be not sufficient for subsequent operations.UsageInitiate Using DefaultsYou may initiate aGitHubDatasetobject directly using source we provided. Currently supported sources can be viewed atlist.json. For example:>>>frommagi_datasetimportGitHubDataset>>>github_dataset=GitHubDataset(...empty=False,...file_path='rust-latest'...)Which will downloadghv10_rust-metadata.json,ghv10_rust-0.jsonandghv10_rust-1.jsonunder./magi_downloads, and use them to create a dataset. Downloading from curated sources only cost time of downloading files, which is usually <500MB.Pull Data by ChunksPulling data from original sources is time-consuming. The recommended way to usemagi_datasetis to run the collection process in chunked mode. First create an empty dataset and initiate index from GitHub:>>>frommagi_datasetimportGitHubDataset>>>github_dataset=GitHubDataset(...empty=True...)>>>github_dataset.init_repos(fully_initialize=False)>>>github_dataset.dump('./outputs/gh_data.json')After this process, the fingerprint./outputs/gh_data-metadata.jsonwill be generated, which contains both metadata of this dataset and a fixed index of the repos to pull. Based on this metadata file, you can run multiple instances ofGitHubDatasetto pull data from online sources by chunks. For example:# create a new GitHubDataset object in another terminal>>>frommagi_datasetimportGitHubDataset>>>github_dataset=GitHubDataset(...empty=True，...# use tokens from different accounts to increase throughput...gh_token='ghp_token1'...)>>>github_dataset.load_fingerprint('./outputs/gh_data-metadata.json')>>>github_dataset.update_repos(...chunks=range(0,50）...)>>>github_dataset.dump(...'./outputs/gh_data.json',...chunks=range(0,50）...)Which dumpsgh_data-0.json,gh_data-1.json, ...,gh_data-49.jsonunder the./outputsdirectory. You can also copy the fingerprint metadata file to other machines to pull different chunks, in order to relieve some stress on IP address limits of the translation API. Make sure to use tokens from different GitHub accounts in diffenent terminals/on different machines.Alternatively, we provide an entry from shell to do this. You can run for each coding language:magi_dataset--langPython--file./outputs/gh_data_python-metadata.json--meta_onlyTrue--gh_token$GH_TOKENAnd after copying./outputs/gh_data_python-metadata.jsonto other machines, run on them separately:magi_dataset--langPython--file./outputs/gh_data_python.json--meta_onlyFalse--load_meta./outputs/gh_data_python-metadata.json--gh_token$GH_TOKENPull Data All TogetherIf the data is not much (for example, settingGitHubDataset.MIN_STARS_PER_REPO > 2000), you can also pull all data together once.To do so, initialize an empty instance and collect data:>>>frommagi_datasetimportGitHubDataset>>>github_dataset=GitHubDataset(...empty=True...)>>>github_dataset.init_repos(fully_initialize=True)Or, download the default data (not guranteed to be the newest):>>>frommagi_datasetimportGitHubDataset>>>github_dataset3=GitHubDataset(...empty=False...)The default data may be found atEnoch2090/github_semantic_searchon HuggingFace. We will update the data periodically.After the dataset is created, access the data with either number index:>>>github_dataset[5]GitHubRepo(name='ytdl-org/youtube-dl',stars=114798,description='Command-line program to download videos from YouTube.com and other video sites',_fully_initialized=True)Or the full name:>>>github_dataset['ytdl-org/youtube-dl']GitHubRepo(name='ytdl-org/youtube-dl',stars=114798,description='Command-line program to download videos from YouTube.com and other video sites',_fully_initialized=True)And you can access the corpus by accessing thereadmeandhn_commentsattributes ofGitHubRepoobjects.>>>github_dataset[5].readme[0:100]'[![Build Status](https://github.com/ytdl-org/youtube-dl/workflows/CI/badge.svg)](https'Future WorksThe current idle handler design is primordial, will switch to async pipelines to relieve CPU sleep time.Elasticsearch database builderPinecone database builder (wrapper only)Hash verification of persistence filesChangelogsv1.0.7Temporary fix. Addedredownloadparameter toGitHubDatasetto avoid redownload of the same file in multiple local runs. Fixed a bug where local files can not be load.v1.0.5Updated default list of files to ghv10. Users may also retrieve default files with keyname in the latest list. For example if the list states{"python-latest":["https://huggingface.co/datasets/Enoch2090/github_semantic_search/resolve/main/ghv10_python-metadata.json","https://huggingface.co/datasets/Enoch2090/github_semantic_search/resolve/main/ghv10_python-0.json","https://huggingface.co/datasets/Enoch2090/github_semantic_search/resolve/main/ghv10_python-1.json",]}User may retrieve these files by simply calling>>>github_dataset3=GitHubDataset(...empty=False,...file_path="python-latest"...)Which is similar to Huggingface model initiation.v1.0.4Added chunked update/dump/loads. Now when saving to files, only.jsonis allowed. Due to the unsafe nature of.pklfiles and other reasons,.pklfiles will not be supported in the future.If saving aGitHubDatasetwith $N$ chunks to file namedata.json, will createdata-metadata.json, anddata-0.json,data-1.json...data-$(N-1).json.