Package: bandersnatch
This is a PyPI mirror client according toPEP 381+PEP 503+PEP 691http://www.python.org/dev/peps/pep-0381/.bandersnatch >=6.0 implements PEP691bandersnatch >=4.0 supportsLinux,MacOSX+WindowsDocumentationbandersnatch maintainersare looking for morehelp! Please refer to ourMAINTAINERdocumentation to see the roles and responsibilities. We would also
ask you read ourMission Statementto ensure it aligns with your thoughts for
this project.If interested contact @cooperleesInstallationThe following instructions will place the bandersnatch executable in a
virtualenv underbandersnatch/bin/bandersnatch.bandersnatchrequires>= Python 3.8.0DockerThis will pull latest build. Please use a specific tag if desired.Docker image includes/bandersnatch/src/runner.pyto periodically
run abandersnatch mirrorPlease/bandersnatch/src/runner.py --helpfor usageWith docker, we recommend bind mounting in a read onlybandersnatch.confDefaults to/conf/bandersnatch.confdockerpullpypa/bandersnatch
dockerrunpypa/bandersnatchbandersnatch--helppipThis installs the latest stable, released version.python3-mvenvbandersnatch
bandersnatch/bin/pipinstallbandersnatch
bandersnatch/bin/bandersnatch--helpQuickstartRunbandersnatch mirror- it will create an empty configuration file
for you in/etc/bandersnatch.conf.Review/etc/bandersnatch.confand adapt to your needs.Runbandersnatch mirroragain. It will populate your mirror with the
current status of all PyPI packages.
Current mirror package size can be seen here:https://pypi.org/stats/Ablocklistorallowlistcan be created to cut down your mirror size.
You might want toAnalyze PyPI downloadsto determine which packages to add to your list.Runbandersnatch mirrorregularly to update your mirror with any
intermediate changes.WebserverConfigure your webserver to serve theweb/sub-directory of the mirror.
For PEP691 support we need to respect the format the client requests.For annginxexample, please look at ourbanderxdocker container andnginx.confexample configuration.Note that it is a good idea to have your webserver publish the HTML index
files correctly with UTF-8 as the charset. The index pages will work without
it but if humans look at the pages the characters will end up looking funny.Make sure that the webserver uses UTF-8 to look up unicode path names. nginx
gets this right by default - not sure about others.For more information visit outofficial documentationfor instructions on how to use a NGINX example Docker Image.If you are looking to an docker-compose example head overhereCron jobsYou need to set up one cron job to run the mirror itself.Here's a sample that you could place in/etc/cron.d/bandersnatch:LC_ALL=en_US.utf8
    */2 * * * * root bandersnatch mirror |& logger -t bandersnatch[mirror]This assumes that you have aloggerutility installed that will convert the
output of the commands to syslog entries.SystemD Timersare also another alternative in today's modern world.Maintenancebandersnatch does not keep much local state in addition to the mirrored data.
In general you can just keep rerunningbandersnatch mirrorto make it fix
errors.If you want to force bandersnatch to check everything against the master PyPI:runbandersnatch mirror --force-checkto move status files if they exist in your mirror directory in order get a full sync.Be aware that full syncs likely take hours depending on PyPI's performance and your network latency and bandwidth.Other Commandsbandersnatch delete --help- Allows you to specify package(s) to be removed from your mirror (dangerous)bandersnatch verify --help- Crawls your repo and fixes any missed files + deletes any unowned files found (dangerous)Operational notesCase-sensitive filesystem neededYou need to run bandersnatch on a case-sensitive filesystem.OS X natively does this OK even though the filesystem is not strictly
case-sensitive and bandersnatch will work fine when running on OS X. However,
tarring a bandersnatch data directory and moving it to, e.g. Linux with a
case-sensitive filesystem will lead to inconsistencies. You can fix those by
deleting the status files and have bandersnatch run a full check on your data.Windows requires elevated promptBandersnatch makes use of symbolic links. On Windows, this permission is turned off by default for non-admin users. In order to run bandersnatch on Windows either call it from an elevated command prompt (i.e. right-click, run-as Administrator) or give yourself symlink permissions in the group policy editor.Many sub-directories neededThe PyPI has a quite extensive list of packages that we need to maintain in a
flat directory. Filesystems with small limits on the number of sub-directories
per directory can run into a problem like this:2013-07-09 16:11:33,331 ERROR: Error syncing package: zweb@802449OSError: [Errno 31] Too many links: '../pypi/web/simple/zweb'Specifically we recommend to avoid using ext3. Ext4 and newer does not have the
limitation of 32k sub-directories.Client CompatibilityA bandersnatch static mirror is compatible only to the "static",  cacheable
parts of PyPI that are needed to support package installation. It does not
support more dynamic APIs of PyPI that maybe be used by various clients for
other purposes.An example of an unsupported API isPyPI's XML-RPC interface, which is used when runningpip search.Bandersnatch MissionThe bandersnatch project strives to:Mirror all static objects of the Python Package Index (https://pypi.org/)bandersnatch's main goal is to support the main global index to local syncingonlyThis will allow organizations to have lower latency access to PyPI and
save bandwidth on their WAN connections and more importantly the PyPI CDNCustom features and requests may be accepted if they can be of apluginforme.g. refer to theblocklistandallowlistpluginsContactIf you have questions or comments, please submit a bug report tohttps://github.com/pypa/bandersnatch/issues/newDiscord: #bandersnatch now sit in thePyPA Discordserver. To join visithttps://discord.com/invite/pypaCode of ConductEveryone interacting in the bandersnatch project's codebases, issue trackers,
chat rooms, and mailing lists is expected to follow thePSF Code of Conduct.KudosThis client is based on the original pep381client byMartin v. Loewis.Richard Joneswas very patient answering questions at PyCon 2013 and made the
protocol more reliable by implementing some PyPI enhancements.Christian Theunefor creating and maintainingbandersnatchfor many years!