Package: mirrorshades
mirrorshadesmirrorshadesis a tool for mirroring data from remote sources.Copyright (c) 2020-2022 Paul Barker.In this repository,
code files are distributed under theApache 2.0 License,
documentation files are distributed under theCC BY 4.0 Licenseand trivial data files are distributed under theCC0 1.0 License.SummaryMirrorshades was written to help maintain mirrors and backups by regularly
pulling content from remote locations. Tools such as rsync, rclone,git clone --mirror, etc already exist and can easily be used to create local mirrors of
remote content. What mirrorshades provides is a single command to update a set
of mirrors of different types of content from various remote sources. For
example, the author uses mirrorshades to sync content from Dropbox, GitLab and
multiple mail servers to a local mirror on a nightly basis for disaster
recovery purposes.All the details of how to pull down the desired content are stored in the
configuration file, it is intended that command line arguments to mirrorshades
remain as minimal as possible and that the config file is the single source of
truth to control the operation of the utility. This makes it very easy to invoke
mirrorshades regularly and reliably using cron, systemd timers or any other
automation mechanism.Mirrorshades is intended to be trivial to extend to handle new types of remote
source. Users familiar with Python are encouraged to look at the source code
for the existing mirroring agents and to add new agents as required. Such
extensions or other modifications to mirrorshades are welcome to be submitted
following thecontribution guidelines, we will greatly
appreciate them!InstallationMirrorshades is published onPyPIso the following command
is usually sufficient to install the application:pip install mirrorshadesThe following mirroring agents have additional dependencies which must be
installed if you wish to use them in your configuration:git: Requires thegitcommand line tool.github: Requires the Python modulePyGithuband thegitcommand line tool.gitlab: Requires the Python modulepython-gitlaband thegitcommand line tool.rclone: Requires therclonecommand line tool.UsageAll significant options for mirrorshades are set via a YAML configuration file
so command line invocation is very straightforward:usage: mirrorshades [-h] [--version] [config_path]

Data mirroring tool

positional arguments:
  config_path  path to the configuration file (defaults to 'mirrorshades.yml' in the current directory)

optional arguments:
  -h, --help   show this help message and exit
  --version    show program's version number and exitConfiguration exampleThe following example shows all the options and mirroring agents supported by
mirrorshades. Not all of these entries are required in your configuration file,
the minimum usable configuration is just one entry undersources.# Global options for mirrorshadesoptions:# Select the destination where mirrored content will be written. If no# destination is given, the current directory will be used.dest:/srv/mirror# The core of a mirrorshades configuration file is the dictionary of# sources to mirror. The key of each source entry is used as the default# for the 'name' and 'agent' properties of that source if they are not set.sources:# 'git' agent: Mirror one or more git repositories.# Repositories are cloned into subdirectories of the path# '<options.dest>/<source name>'. If local repositories are already# present (for example on a subsequent invocation of mirrorshades) then# they will be updated instead of freshly cloned.git:# URL prefix applied to all entries under 'repositories'. This reduces# duplication when mirroring multiple repositories from the same# upstream server.url_prefix:https://git.example.com/# List of repositories to mirror.repositories:# For example, this entry (along with the other example configuration# shown here) will mirror 'https://git.example.com/myrepository.git'# to '/srv/mirror/git/myrepository.git'.-myrepository.git# 'github' agent: Mirror repositories from a Github instance by full name,# user or organization. Note that this agent only mirrors git repository data# and not issues, merge requests, etc. Each git repository is mirrored using# the 'git' agent.github:# Access token. This may be set to allow access to private repositories.access_token:...# List of users to mirror. Each user will be looked up using the Github API# and the git repositories which they own will be mirrored. The destination# path is formed in the same way as for the 'git' agent using the repository# clone url without the 'https://github.com/' prefix.users:-myusername# List of organizations to mirror. Each organization will be looked up using# the Github API and their git repositories will be mirrored. The# destination path is formed in the same way as for the 'git' agent using# the repository clone url without the 'https://github.com/' prefix.organizations:-myorganization# List of individual repositories to mirror, identified by their full name# (formed of the user or organization name, a '/' and then the repository# name). The destination path is formed in the same way as for the 'git'# agent using the repository clone url without the 'https://github.com/'# prefix.repositories:-torvalds/linux# 'gitlab' agent: Mirror groups and projects from a GitLab instance.# Note that this agent only mirrors git repository data and not issues,# merge requests, etc. Each git repository is mirrored using the 'git'# agent.gitlab:# GitLab server address. If not set, this defaults to# https://gitlab.com.server:https://gitlab.example.com# Private token used to connect to the GitLab API. Keep this secret!private_token:...# List of projects to mirror. Each project will be looked up using the# GitLab API and the git repository will be mirrored. The destination# path is formed in the same way as for the 'git' agent using the# relative repository path given by the GitLab server.projects:-othergroup/myproject# List of groups to mirror. Each group will be recursively enumerated# using the GitLab API and all git repositories which are found will be# mirrored. Destination paths will be determined in the same way as for# individually listed projects.groups:-mygroup# 'rclone' agent: Mirror remote or cloud data which can be accessed using# rclone. Note that rclone must be configured separately before invoking# mirrorshades for this agent to work.## In this example we use our own key 'my_cloud_data' instead of the agent# name 'rclone' so that the mirrored data will be placed under# '/srv/mirror/my_cloud_data'.my_cloud_data:# As the key we used does not match the agent we wish to use, we need# to explicitly specify the agent.agent:rclone# The rclone remote from which data will be mirrored. This should be# configured as a remote within rclone.remote:myremote# Paths within the given remote to mirror. To mirror all data from the# given rclone remote we can specify a single '.' path as shown here.paths:-'.'# 'command' agent: Invoke a custom command to mirror arbitrary data. In# this example we use the 'mbsync' command to create a local mirror of# one or more email accounts. This example assumes that appropriate# sources and destinations are setup in the mbsyncrc file, for other# commands you may similarly need to setup the relevant configuration# or pass appropriate command line arguments.mbsync:# As with the 'rclone' agent example above, the key does not match the# agent we wish to use so we need to explicitly specify the agent here.agent:command# The command to run along with any arguments.command:mbsync -a# Number of attempts to make. If the command fails, it is retried up to this# number of times. If the command succeeds it is not retried. This property# is optional, if it is unspecified the default is to only make one attempt# to run the command.attempts:5Contributionmirrorshades is developed on GitHub athttps://github.com/unnecessary-abstraction/mirrorshades.If you find any bugs or have a feature request feel free to open a ticket in theissue tracker.To submit patches to mirrorshades please fork the repository on GitHub and open
apull requestwhere your changes are ready to merge. Pull requests should generally be
targeted at thedevbranch.