Package: github-to-sqlite
github-to-sqliteSave data from GitHub to a SQLite database.DemoHow to installAuthenticationFetching issues for a repositoryFetching pull requests for a repositoryFetching issue comments for a repositoryFetching commits for a repositoryFetching releases for a repositoryFetching tags for a repositoryFetching contributors to a repositoryFetching repos belonging to a user or organizationFetching specific repositoriesFetching repos that have been starred by a userFetching users that have starred specific reposFetching GitHub Actions workflowsScraping dependents for a repositoryFetching emojisMaking authenticated API callsDemohttps://github-to-sqlite.dogsheep.net/hosts aDatasettedemo of a database created byrunning this toolagainst all of the repositories in theDogsheep GitHub organization, plus thedatasetteandsqlite-utilsrepositories.How to install$ pip install github-to-sqliteAuthenticationCreate a GitHub personal access token:https://github.com/settings/tokensRun this command and paste in your new token:$ github-to-sqlite authThis will create a file calledauth.jsonin your current directory containing the required value. To save the file at a different path or filename, use the--auth=myauth.jsonoption.As an alternative to using anauth.jsonfile you can add your access token to an environment variable calledGITHUB_TOKEN.Fetching issues for a repositoryTheissuescommand retrieves all of the issues belonging to a specified repository.$ github-to-sqlite issues github.db simonw/datasetteIf anauth.jsonfile is present it will use the token from that file. It works without authentication for public repositories but you should be aware that GitHub have strict IP-based rate limits for unauthenticated requests.You can point to a different location ofauth.jsonusing-a:$ github-to-sqlite issues github.db simonw/datasette -a /path/to/auth.jsonYou can use the--issueoption one or more times to load specific issues:$ github-to-sqlite issues github.db simonw/datasette --issue=1Example:issues tableFetching pull requests for a repositoryWhile pull requests are a type of issue, you will get more information on pull requests by pulling them separately. For example, whether a pull request has been merged and when.Following the API of issues, thepull-requestscommand retrieves all of the pull requests belonging to a specified repository.$ github-to-sqlite pull-requests github.db simonw/datasetteYou can use the--pull-requestoption one or more times to load specific pull request:$ github-to-sqlite pull-requests github.db simonw/datasette --pull-request=81Note that themerged_bycolumn on thepull_requeststable will only be populated for pull requests that are loaded using the--pull-requestoption - the GitHub API does not return this field for pull requests that are loaded in bulk.You can load only pull requests in a certain state with the--stateoption:$ github-to-sqlite pull-requests --state=open github.db simonw/datasettePull requests across an entire organization (or more than one) can be loaded with--org:$ github-to-sqlite pull-requests --state=open --org=psf --org=python github.dbYou can use a search query to find pull requests.  Note that no more than 1000 will be loaded (this is a GitHub API limitation), and some data will be missing (base and head SHAs).  When using searches, other filters are ignored; put all criteria into the search itself:$ github-to-sqlite pull-requests --search='org:python defaultdict state:closed created:<2023-09-01' github.dbExample:pull_requests tableFetching issue comments for a repositoryTheissue-commentscommand retrieves all of the comments on all of the issues in a repository.It is recommended you runissuesfirst, so that each imported comment can have a foreign key pointing to its issue.$ github-to-sqlite issues github.db simonw/datasette
$ github-to-sqlite issue-comments github.db simonw/datasetteYou can use the--issueoption to only load comments for a specific issue within that repository, for example:$ github-to-sqlite issue-comments github.db simonw/datasette --issue=1Example:issue_comments tableFetching commits for a repositoryThecommitscommand retrieves details of all of the commits for one or more repositories. It currently fetches the SHA, commit message and author and committer details; it does not retrieve the full commit body.$ github-to-sqlite commits github.db simonw/datasette simonw/sqlite-utilsThe command accepts one or more repositories.By default it will stop as soon as it sees a commit that has previously been retrieved. You can force it to retrieve all commits (including those that have been previously inserted) using--all.Example:commits tableFetching releases for a repositoryThereleasescommand retrieves the releases for one or more repositories.$ github-to-sqlite releases github.db simonw/datasette simonw/sqlite-utilsThe command accepts one or more repositories.Example:releases tableFetching tags for a repositoryThetagscommand retrieves all of the tags for one or more repositories.$ github-to-sqlite tags github.db simonw/datasette simonw/sqlite-utilsExample:tags tableFetching contributors to a repositoryThecontributorscommand retrieves details of all of the contributors for one or more repositories.$ github-to-sqlite contributors github.db simonw/datasette simonw/sqlite-utilsThe command accepts one or more repositories. It populates acontributorstable, with foreign keys toreposandusersand acontributionstable listing the number of commits to that repository for each contributor.Example:contributors tableFetching repos belonging to a user or organizationThereposcommand fetches repos belonging to a user or organization.Without any other arguments, this command will fetch all repos that the currently authenticated user owns, collaborates on or can access via one of their organizations:$ github-to-sqlite repos github.dbTo fetch repos belonging to a specific user or organization, provide their username as an argument:$ github-to-sqlite repos github.db dogsheep # organization
$ github-to-sqlite repos github.db simonw # userYou can pass more than one username to fetch for multiple users or organizations at once:$ github-to-sqlite repos github.db simonw dogsheepAdd the--readmeoption to save the README for the repo in a column calledreadme. Add--readme-htmlto save the HTML rendered version of the README into a column calledreadme_html.Example:repos tableFetching specific repositoriesYou can use-rwith thereposcommand one or more times to fetch just specific repositories.$ github-to-sqlite repos github.db -r simonw/datasette -r dogsheep/github-to-sqliteFetching repos that have been starred by a userThestarredcommand fetches the repos that have been starred by a user.$ github-to-sqlite starred github.db simonwIf you are using anauth.jsonfile you can omit the username to retrieve the starred repos for the authenticated user.Example:stars tableFetching users that have starred specific reposThestargazerscommand fetches the users that have starred the specified repos.$ github-to-sqlite stargazers github.db simonw/datasette dogsheep/github-to-sqliteYou can specify one or more repository usingowner/reposyntax.Users fetched using this command will be inserted into theuserstable. Many-to-many records showing which repository they starred will be added to thestarstable.Fetching GitHub Actions workflowsTheworkflowscommand fetches the YAML workflow configurations from each repository's.github/workflowsdirectory and parses them to populateworkflows,jobsandstepstables.$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqliteYou can specify one or more repository usingowner/reposyntax.Example:workflows table,jobs table,steps tableScraping dependents for a repositoryThe GitHub dependency graph can show other GitHub projects that depend on a specific repo, for examplesimonw/datasette/network/dependents.This data is not yet available through the GitHub API. Thescrape-dependentscommand scrapes those pages and uses the GitHub API to load full versions of the dependent repositories.$ github-to-sqlite scrape-dependents github.db simonw/datasetteThe command accepts one or more repositories.Add-vfor verbose output.Example:dependents tableFetching emojisYou can fetch a list of every emoji supported by GitHub using theemojiscommand:$ github-to-sqlite emojis github.dbThis will create a table calledemojiswith a primary keynameand aurlcolumn.If you add the--fetchoption the command will also fetch the binary content of the images and place them in animagecolumn:$ github-to-sqlite emojis emojis.db -f
[########----------------------------]  397/1799   22%  00:03:43You can then use thedatasette-render-imagesplugin to browse them visually.Example:emojis tableMaking authenticated API callsThegithub-to-sqlite getcommand provides a convenient shortcut for making authenticated calls to the API. Once you have created yourauth.jsonfile (or set aGITHUB_TOKENenvironment variable) you can use it like this:$ github-to-sqlite get https://api.github.com/gistsThis will make an authenticated call to the URL you provide and pretty-print the resulting JSON to the console.You can omit thehttps://api.github.com/prefix, for example:$ github-to-sqlite get /gistsMany GitHub APIs arepaginated using the HTTP Link header. You can follow this pagination and output a list of all of the resulting items using--paginate:$ github-to-sqlite get /users/simonw/repos --paginateYou can outline newline-delimited JSON for each item using--nl. This can be useful for streaming items into another tool.$ github-to-sqlite get /users/simonw/repos --nl