Package: all-repos
all-reposClone all your repositories and apply sweeping changes.Installationpipinstallall-reposCLIAll command line interfaces provided byall-reposprovide the following
options:-h/--help: show usage information-C CONFIG_FILENAME/--config-filename CONFIG_FILENAME: use a non-default
config file (the defaultall-repos.jsoncan be changed with the environment
variableALL_REPOS_CONFIG_FILENAME).--color {auto,always,never}: use color in output (defaultauto).all-repos-complete [options]Addgit clonetab completion for all-repos repositories.Requiresjqto function.Add to.bash_profile:eval"$(all-repos-complete-C~/.../all-repos.json--bash)"all-repos-clone [options]Clone all the repositories into theoutput_dir.  If run again, this command
will update existing repositories.Options:-j JOBS/--jobs JOBS: how many concurrent jobs will be used to complete
the operation.  Specify 0 or -1 to match the number of cpus.  (default8).Sample invocations:all-repos-clone: clone the repositories specified inall-repos.jsonall-repos-clone -C all-repos2.json: clone using a non-default config
filename.all-repos-find-files [options] PATTERNSimilar to a distributedgit ls-files | grep -P PATTERN.Arguments:PATTERN: thepython regexto match.Options:--repos-with-matches: only print repositories with matches.Sample invocations:all-repos-find-files setup.py: find allsetup.pyfiles.all-repos-find-files --repos setup.py: find all repositories containing
asetup.py.all-repos-grep [options] [GIT_GREP_OPTIONS]Similar to a distributedgit grep ....Options:--repos-with-matches: only print repositories with matches.GIT_GREP_OPTIONS: additional arguments will be passed on togit grep.
seegit grep --helpfor available options.Sample invocations:all-repos-grep pre-commit -- 'requirements*.txt': find all repositories
which havepre-commitlisted in a requirements file.all-repos-grep -L six -- setup.py: find setup.py files which do not
containsix.all-repos-list-repos [options]List all cloned repository names.all-repos-manual [options]Interactively apply a manual change across repos.note:all-repos-manualwill always run in--interactiveautofixing mode.note:all-repos-manualrequiresthe--reposautofixer option.Options:autofix options:all-repos-manualis
an autofixer and supports all of the autofixer options.--branch-name BRANCH_NAME: override the autofixer branch name (defaultall-repos-manual).--commit-msg COMMIT_MSG(required): set the autofixer commit message.all-repos-sed [options] EXPRESSION FILENAMESSimilar to a distributedgit ls-files -z -- FILENAMES | xargs -0 sed -i EXPRESSION.note: this assumes GNU sed. If you're on macOS, installgnu-sedwith Homebrew:brewinstallgnu-sed# Add to .bashrc / .zshrcexportPATH="$(brew--prefix)/opt/gnu-sed/libexec/gnubin:$PATH"Arguments:EXPRESSION: sed program. For example:s/hi/hello/g.FILENAMES: filenames glob (passed togit ls-files).Options:autofix options:all-repos-sedis
an autofixer and supports all of the autofixer options.-r/--regexp-extended: use extended regular expressions in the script.
Seeman sedfor further details.--branch-name BRANCH_NAMEoverride the autofixer branch name (defaultall-repos-sed).--commit-msg COMMIT_MSGoverride the autofixer commit message.  (defaultgit ls-files -z -- FILENAMES | xargs -0 sed -i ... EXPRESSION).Sample invocations:all-repos-sed 's/foo/bar/g' -- '*': replacefoowithbarin all files.ConfiguringA configuration file looks roughly like this:{"output_dir":"output","source":"all_repos.source.github","source_settings":{"api_key":"...","username":"asottile"},"push":"all_repos.push.github_pull_request","push_settings":{"api_key":"...","username":"asottile"}}output_dir: where repositories will be cloned to whenall-repos-cloneis
run.source: the module import path to asource, see below for builtin
source modules as well as directions for writing your own.source_settings: the source-type-specific settings, the source module's
documentation will explain the various possible values.push: the module import path to apush, see below for builtin push
modules as well as directions for writing your own.push_settings: the push-type-specific settings, the push module's
documentation will explain the various possible values.include(default""): python regex for selecting repositories.  Only
repository names which match this regex will be included.exclude(default"^$"): python regex for excluding repositories.
Repository names which match this regex will be excluded.all_branches(defaultfalse): whether to clone all of the branches or
just the default upstream branch.Source modulesall_repos.source.json_fileClones all repositories listed in a file.  The file must be formatted as
follows:{"example/repo1":"https://git.example.com/example/repo1","repo2":"https://git.example.com/repo2"}Requiredsource_settingsfilename: file containing repositories one-per-line.Directory locationoutput/
+--- repos.json
+--- repos_filtered.json
+--- {repo_key1}/
+--- {repo_key2}/
+--- {repo_key3}/all_repos.source.githubClones all repositories available to a user on github.Requiredsource_settingsapi_key: the api key which the user will log in as.Usethe settings tabto create a
personal access token.The minimum scope required to function ispublic_repo, though you'll
needrepoto access private repositories.api_key_env: alternatively API key can also be passed via an environment variableusername: the github username you will log in as.Optionalsource_settingscollaborator(defaultfalse): whether to include repositories which are
not owned but can be contributed to as a collaborator.forks(defaultfalse): whether to include repositories which are forks.private(defaultfalse): whether to include private repositories.archived(default:false): whether to include archived repositories.base_url(default:https://api.github.com) is the base URL to the Github
API to use (for Github Enterprise support set this tohttps://{your_domain}/api/v3).Directory locationoutput/
+--- repos.json
+--- repos_filtered.json
+--- {username1}/
    +--- {repo1}/
    +--- {repo2}/
+--- {username2}/
    +--- {repo3}/all_repos.source.github_forksClones all repositories forked from a repository on github.Requiredsource_settingsapi_key: the api key which the user will log in as.Usethe settings tabto create a
personal access token.The minimum scope required to function ispublic_repo.api_key_env: alternatively API key can also be passed via an environment variablerepo: the repo which has forksOptionalsource_settingscollaborator(defaulttrue): whether to include repositories which are
not owned but can be contributed to as a collaborator.forks(defaulttrue): whether to include repositories which are forks.private(defaultfalse): whether to include private repositories.archived(default:false): whether to include archived repositories.base_url(default:https://api.github.com) is the base URL to the Github
API to use (for Github Enterprise support set this tohttps://{your_domain}/api/v3).Directory locationSee the directory structure forall_repos.source.github.all_repos.source.github_orgClones all repositories from an organization on github.Requiredsource_settingsapi_key: the api key which the user will log in as.Usethe settings tabto create a
personal access token.The minimum scope required to function ispublic_repo, though you'll
needrepoto access private repositories.api_key_env: alternatively API key can also be passed via an environment variableorg: the organization to clone fromOptionalsource_settingscollaborator(defaulttrue): whether to include repositories which are
not owned but can be contributed to as a collaborator.forks(defaultfalse): whether to include repositories which are forks.private(defaultfalse): whether to include private repositories.archived(default:false): whether to include archived repositories.base_url(default:https://api.github.com) is the base URL to the Github
API to use (for Github Enterprise support set this tohttps://{your_domain}/api/v3).Directory locationSee the directory structure forall_repos.source.github.all_repos.source.gitoliteClones all repositories available to a user on agitolitehost.Requiredsource_settingsusername: the user to SSH to the server as (usuallygit)hostname: the hostname of your gitolite server (e.g.git.mycompany.com)The gitolite API is served over SSH.  It is assumed that whenall-repos-cloneis called, it's possible to make SSH connections with the username and hostname
configured here in order to query that API.Optionalsource_settingsmirror_path(defaultNone): an optional mirror to clone repositories from.
This is a Python format string, and can use the variablerepo_name.This can be anything git understands, such as another remote server (e.g.gitmirror.mycompany.com:{repo_name}) or a local path (e.g./gitolite/git/{repo_name}.git).Directory locationoutput/
+--- repos.json
+--- repos_filtered.json
+--- {repo_name1}.git/
+--- {repo_name2}.git/
+--- {repo_name3}.git/all_repos.source.bitbucketClones all repositories available to a user on Bitbucket Cloud.Requiredsource_settingsusername: the Bitbucket username you will log in as.app_password: the authentication method for the above user to login withCreate an application password within youraccount settings.We need the scope: Repositories -> Readall_repos.source.bitbucket_serverClones all repositories available to a user on Bitbucket Server.Requiredsource_settingsbase_url: the bitbucket server URL (egbitbucket.domain.com)username: the Bitbucket username you will log in as.app_password: the authentication method for the above user to login withCreate an application password within youraccount settings.We need the scope: Repositories -> ReadOptionalsource_settingsproject(defaultNone): an optional project to restrict the search for repositories.Directory locationoutput/
+--- repos.json
+--- repos_filtered.json
+--- {username1}/
    +--- {repo1}/
    +--- {repo2}/
+--- {username2}/
    +--- {repo3}/all_repos.source.gitlab_orgClones all repositories from an organization on gitlab.Requiredsource_settingsapi_key: the api key which the user will log in as.Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a
personal access token.We need the scope:read_api,read_repository.api_key_env: alternatively API key can also be passed via an environment variableorg: the organization to clone fromOptionalsource_settingsbase_url: (defaulthttps://gitlab.com/api/v4) the gitlab server URLarchived(default:false): whether to include archived repositories.Directory locationoutput/
+--- repos.json
+--- repos_filtered.json
+--- {org}/
    +--- {subpgroup1}/
        +--- {subpgroup2}/
            +--- {repo1}/
        +--- {repo2}/
    +--- {repo3}/
    +--- {repo4}/Writing your own sourceFirst create a module.  This module must have the following api:ASettingsclassThis class will receive keyword arguments for all values in thesource_settingsdictionary.An easy way to implement theSettingsclass is by using anamedtuple:Settings=collections.namedtuple('Settings',('required_thing','optional'))Settings.__new__.__defaults__=('optional default value',)In this example, therequired_thingsetting is arequiredsetting
whereasoptionalmay be omitted (and will get a default value of'optional default value').def list_repos(settings: Settings) -> Dict[str, str]:callableThis callable will be passed an instance of yourSettingsclass.  It must
return a mapping from{repo_name: repository_url}.  Therepo_nameis the
directory name inside theoutput_dir.Push modulesall_repos.push.merge_to_masterMerges the branch directly to the default branch and pushes.  The commands it
runs look roughly like this:gitcheckoutmain
gitpull
gitmerge--no-ff$BRANCHgitpushoriginHEADOptionalpush_settingsfast_forward(default:false): iftrue, perform a fast-forward
merge (--ff-only). Iffalse, create a merge commit (--no-ff).all_repos.push.github_pull_requestPushes the branch tooriginand then creates a github pull request for the
branch.Requiredpush_settingsapi_key: the api key which the user will log in as.Usethe settings tabto create a
personal access token.The minimum scope required to function ispublic_repo, though you'll
needrepoto access private repositories.api_key_env: alternatively API key can also be passed via an environment variableusername: the github username you will log in as.Optionalpush_settingsfork(default:false): (if applicable) a fork will be created and pushed
to instead of the upstream repository.  The pull request will then be made
to the upstream repository.base_url(default:https://api.github.com) is the base URL to the Github
API to use (for Github Enterprise support set this tohttps://{your_domain}/api/v3).all_repos.push.bitbucket_server_pull_requestPushes the branch tooriginand then creates a Bitbucket pull request for the branch.Requiredpush_settingsbase_url: the Bitbucket server URL (egbitbucket.domain.com)username: the Bitbucket username you will log in as.app_password: the authentication method for the above user to login withCreate an application password within youraccount settings.We need the scope: Repositories -> Readall_repos.push.gitlab_pull_requestPushes the branch tooriginand then creates a GitLab pull request for the branch.Requiredpush_settingsbase_url: the GitLab server URL (eghttps://{gitlab.domain.com}/api/v4)api_key: the api key which the user will log in as.Use the settings tab (eg https://{gitlab.domain.com}/-/profile/personal_access_tokens) to create a
personal access token.We need the scope:write_repository.api_key_env: alternatively API key can also be passed via an environment variableall_repos.push.readonlyDoes nothing.push_settingsThere are no configurable settings forreadonly.Writing your own push moduleFirst create a module.  This module must have the following api:ASettingsclassThis class will receive keyword arguments for all values in thepush_settingsdictionary.def push(settings: Settings, branch_name: str) -> None:This callable will be passed an instance of yourSettingsclass.  It should
deploy the branch.  The function will be called with the root of the
repository as thecwd.Writing an autofixerAn autofixer applies a change over all repositories.all-reposprovides several api functions to write your autofixers with:all_repos.autofix_lib.add_fixer_argsdefadd_fixer_args(parser):Adds the autofixer cli options.Options:--dry-run: show what would happen but do not push.-i/--interactive: interactively approve / deny fixes.-j JOBS/--jobs JOBS: how many concurrent jobs will be used to complete
the operation.  Specify 0 or -1 to match the number of cpus.  (default1).--limit LIMIT: maximum number of repos to process (default: unlimited).--author AUTHOR: override commit author.  This is passed directly togit commit.  An example:--author='Herp Derp <herp.derp@umich.edu>'.--repos [REPOS [REPOS ...]]: run against specific repositories instead.
This is especially useful withxargs autofixer ... --repos.  This can be
used to specify repositories which are not managed byall-repos.all_repos.autofix_lib.from_clideffrom_cli(args,*,find_repos,msg,branch_name):Parse cli arguments and produceautofix_libprimitives.  Returns(repos, config, commit, autofix_settings).  This is handled separately fromfixto allow for fixers to adjust arguments.find_repos: callback takingConfigas a positional argument.msg: commit message.branch_name: identifier used to construct the branch name.all_repos.autofix_lib.fixdeffix(repos,*,apply_fix,check_fix=_noop_check_fix,config:Config,commit:Commit,autofix_settings:AutofixSettings,):Apply the fix.apply_fix: callback which will be called once per repository.  Thecwdwhen the function is called will be the root of the repository.all_repos.autofix_lib.rundefrun(*cmd,**kwargs):Wrapper aroundsubprocess.runwhich prints the command it will run.  Unlikesubprocess.run, this defaultscheck=Trueunless explicitly disabled.Example autofixerThe trivial autofixer is as follows:importargparsefromall_reposimportautofix_libdeffind_repos(config):return[]defapply_fix():passdefmain(argv=None):parser=argparse.ArgumentParser()autofix_lib.add_fixer_args(parser)args=parser.parse_args(argv)repos,config,commit,autofix_settings=autofix_lib.from_cli(args,find_repos=find_repos,msg='msg',branch_name='branch-name',)autofix_lib.fix(repos,apply_fix=apply_fix,config=config,commit=commit,autofix_settings=autofix_settings,)if__name__=='__main__':raiseSystemExit(main())You can find some more involved examples inall_repos/autofix:all_repos.autofix.azure_pipelines_autoupdate: upgrade pinned azure
pipelines template repository references.all_repos.autofix.pre_commit_autoupdate: runspre-commit autoupdate.all_repos.autofix.pre_commit_autopep8_migrate: migratesautopep8-wrapperfrompre-commit/pre-commit-hookstomirrors-autopep8.all_repos.autofix.pre_commit_cache_dir: updates the cache directory
for travis-ci / appveyor for pre-commit 1.x.all_repos.autofix.pre_commit_flake8_migrate: migratesflake8frompre-commit/pre-commit-hookstopycqa/flake8.all_repos.autofix.pre_commit_migrate_config: runspre-commit migrate-config.all_repos.autofix.setup_py_upgrade: runssetup-py-upgradeand thensetup-cfg-fmtto migratesetup.pytosetup.cfg.