Package: ghcloneall
ghcloneallIt’s a script to clone/update all repos for a user/organization from GitHub.Target audience: maintainers of large collections of projects (for example,
ZopeFoundation members).Usage examplesFirstpip install ghcloneall.Clone all mgedmin’s vim plugins:mkdir ~/src/vim-plugins
cd ~/src/vim-plugins
ghcloneall --init --user mgedmin --pattern '*.vim'
ghcloneallClone all mgedmin’s gists:mkdir ~/src/gists
cd ~/src/gists
ghcloneall --init --user mgedmin --gists
ghcloneallClone all ZopeFoundation repositories:mkdir ~/src/zf
cd ~/src/zf
ghcloneall --init --org ZopeFoundation
ghcloneallHere’s a screencast of the above (running a slightly older version so the
script name differs):DetailsWhat it does:clones repositories you don’t have locallypulls changes for repositories you already have locallywarns you about local changes and other unexpected situations:unknown files in the tree (in –verbose mode only)staged but not committed changesuncommitted (and unstaged changes)non-default branch checked outcommitted changes that haven’t been pushed to default branchremote URL pointing to an unexpected location (in –verbose mode only)You can ask it to not change any files on disk and just look for pending
changes by runningghcloneall--dry-run.  This will also make the
check faster!SynopsisOther command-line options:$ ghcloneall --help
usage: ghcloneall [-h] [--version] [-c CONCURRENCY] [-n] [-q] [-v]
                  [--start-from REPO] [--organization ORGANIZATION]
                  [--user USER] [--github-token GITHUB_TOKEN] [--gists]
                  [--repositories] [--pattern PATTERN] [--include-forks]
                  [--exclude-forks] [--include-archived] [--exclude-archived]
                  [--include-private] [--exclude-private] [--include-disabled]
                  [--exclude-disabled] [--init] [--http-cache DBNAME]
                  [--no-http-cache]

Clone/update all user/org repositories from GitHub.

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -c CONCURRENCY, --concurrency CONCURRENCY
                        set concurrency level (default: 4)
  -n, --dry-run         don't pull/clone, just print what would be done
  -q, --quiet           terser output
  -v, --verbose         perform additional checks
  --start-from REPO     skip all repositories that come before REPO
                        alphabetically
  --organization ORGANIZATION
                        specify the GitHub organization
  --user USER           specify the GitHub user
  --github-token GITHUB_TOKEN
                        specify the GitHub token
  --gists               clone user's gists
  --repositories        clone user's or organisation's repositories (default)
  --pattern PATTERN     specify repository name glob pattern to filter
  --include-forks       include repositories forked from other users/orgs
  --exclude-forks       exclude repositories forked from other users/orgs
                        (default)
  --include-archived    include archived repositories
  --exclude-archived    exclude archived repositories (default)
  --include-private     include private repositories (default when a github
                        token is provided)
  --exclude-private     exclude private repositories
  --include-disabled    include disabled repositories (default)
  --exclude-disabled    exclude disabled repositories
  --init                create a .ghcloneallrc from command-line arguments
  --http-cache DBNAME   cache HTTP requests on disk in an sqlite database for
                        5 minutes (default: .httpcache)
  --no-http-cache       disable HTTP disk cachingConfiguration fileThe script looks for.ghcloneallrcin the current working directory, which
should look like this:[ghcloneall]
# Provide either github_user or github_org, but not both
# github_org = ZopeFoundation
github_user = mgedmin
pattern = *.vim
# Provide github token for authentication
# github_token = <my-github-token>
# You can also uncomment and change these if you wish
# gists = False
# include_forks = False
# include_archived = False
# Listing private repositories requires a valid github_token
# include_private = True
# include_disabled = TrueYou can create one withghcloneall--init--{user,org}X[--patternY][--{include,exclude}-{forks,archived,private,disabled}][--gists|--repos].TipsFor best results configure SSH persistence to speed up git pulls – in your~/.ssh/config:Host github.com
ControlMaster auto
ControlPersist yes
ControlPath ~/.ssh/control-%r@%h-%pIt takes about 80 seconds to rungit pullon all 382 ZopeFoundation
repos on my laptop with this kind of setup.Changelog1.11.0 (2022-10-27)Add support for Python 3.10 and 3.11.Drop support for Python 3.6.Fixghcloneall--user...--github-token...--include-privatenot
including any private repositories (GH: #16).1.10.1 (2021-05-26)When determining if a repository is dirty, use the repository’s
configured default branch from GitHub instead of assuming that the
default is “master”.1.10.0 (2021-04-10)Allow authentication with GitHub token.Depend on requests-cache < 0.6 on Python 2.7.Add support for Python 3.9.Drop support for Python 3.5.1.9.2 (2019-10-15)Add support for Python 3.8.1.9.1 (2019-10-07)Reuse HTTP connections for GitHub API requests.1.9.0 (2019-09-06)Can now clone all user’s gists.Command line args: –gists, –repos.1.8.0 (2019-08-28)Skip forks and archived repositories by default.Command-line args: –include-forks, –exclude-forks.Command-line args: –include-archived, –exclude-archived.Command-line args: –include-private, –exclude-private.Command-line args: –include-disabled, –exclude-disabled.Use a custom User-Agent header when talking to GitHub.1.7.1 (2019-08-14)Drop support for Python 3.3 and 3.4.Add a test suite.Fix AttributeError: ‘str’ object has no attribute ‘format_map’ on Python 2.1.7.0 (2018-12-19)Command line args: -q, –quietFix display corruption on ^C1.6.1 (2018-10-19)Fix TypeError: get() got an unexpected keyword argument ‘fallback’ on
Python 2.1.6 (2016-12-29)Comprehensive rebranding:Rename the GitHub repository tohttps://github.com/mgedmin/ghcloneallRenamecloneall.pytoghcloneall.pyRename the config file to.ghcloneallrc, and rename the config
section to[ghcloneall].Don’t print tracebacks on ^C (this regressed in 1.5).1.5 (2016-12-29)Released to PyPI asghcloneallAdded Python 2.7 support1.4 (2016-12-28)Command line args: –user, –pattern, –initLoad (some) options from a.cloneallrcStop using--organization=ZopeFoundationby default, require an
explicit option (or config file)Rename clone_all_zf_repos.py to cloneall.py1.3 (2016-12-28)Command line args: -cShow progress while fetching the list of repositories from GitHubUpdate repositories concurrently by defaultHighlight items in progressHighlight failed items in redTweak progress bar style from[===]to[###..]Clear the progress bar on ^CHandle git errors nicelyBugfix: -vv could fail with NameError if unknown files were present in a
working treeBugfix: correctly show progress when using –start-fromBugfix: script would hang (for 10 minutes) if you didn’t already have an
SSH control master process runningBugfix: –dry-run didn’t show which repos were new1.2 (2016-11-09)Command line args: –dry-run, –verboseCache HTTP responses on disk for 10 minutes to avoid GitHub API rate limitsReport about forgotten uncommitted and staged changesWarn about local (unpushed) commits tooWarn about other branches being checked outDefault to SSH URLs again (faster when using SSH’s ControlPersist)1.1 (2015-11-07)Command line args: –version, –start-from, –organizationOutput formatting: shorter repository names, totals at the endUse ANSI colors to indicate changesDon’t print tracebacks on ^CDefault to HTTPS URLs1.0 (2015-11-07)Moved from a gist to a proper GitHub repository.