Package proxy cache

Related articles
pacman
If you want to install the same Arch packages over and over —e.g. for testing purposes— it could help if you would not have to get the packages every time from the internet. This article shows you how to share packages so that you can greatly decrease your download times.
Which solution is best depends on your individual use-case. The methods can be grouped into
#Package cache sharing
of the machines, or deploying a
#Proxy server
for extra caching on one machine and configuring the machines to use it accordingly.
Package cache sharing
For all solutions to share the package cache, keep in mind that, by default,
pacman -Sc
removes package tarballs from the cache that correspond to packages that are not installed on the machine the command was issued on. Because
pacman
cannot predict what packages are installed on all machines that share the cache, it will end up deleting files that should not be.
To clean up the cache so that only
outdated
tarballs are deleted:
/etc/pacman.conf
[options]
CleanMethod = KeepCurrent
Read-only cache
Pacman supports cache servers directly. Cache servers will be tried before any non-cache servers, will not be removed from the server pool because of HTTP 404 download errors, and will not be used for database files.
If you are looking for a quick solution, you can simply run a
basic temporary webserver
which other computers can use as their cache server.
Start serving this directory. For example, with
Python
http.server
module:
$ python -m http.server -d /var/cache/pacman/pkg/
Tip
By default, Python
http.server
listens on port
8000
of any interface. To use another port, or bind only to specific address, simply add a parameter and an argument:
$ python -m http.server -d /var/cache/pacman/pkg/ --bind 127.0.0.1 8080
Then
edit
/etc/pacman.d/mirrorlist
on each client machine to add this server:
/etc/pacman.d/mirrorlist
CacheServer = http://
server-ip
:
port
Warning
Do
not
append
/repos/$repo/os/$arch
to this custom server like for other entries, as this hierarchy does not exist and therefore queries will fail.
If looking for a more standalone solution,
darkhttpd
offers a very minimal webserver. Replace the previous
python
command with e.g.:
[http]$ darkhttpd /var/cache/pacman/pkg --no-server-id
You could also run darkhttpd as a
systemd
service for convenience: see
Systemd#Writing unit files
.
miniserve
, a small web server written in Rust, can also be used:
$ miniserve /var/cache/pacman/pkg
Then edit
/etc/pacman.d/mirrorlist
as above with the first url miniserve is available at.
If you are already running a web server for some other purpose, you might wish to reuse that as your local repository server instead. For example, if you already serve a site with
nginx
, you can add an
nginx
server block listening on port 8080:
/etc/nginx/nginx.conf
server {
    listen 8080;
    root /var/cache/pacman/pkg;
    server_name myarchrepo.localdomain;
    try_files $uri $uri/;
}
Remember to
restart
nginx.service
after making this change.
Tip
Whichever web server you use, make sure the firewall configuration (if any) allows the configured port to be reached by the desired traffic, and disallows any undesired traffic. See
Security#Network and firewalls
.
Overlay mount of read-only cache
It is possible to use one machine on a local network as a read-only package cache by
overlay mounting
its
/var/cache/pacman/pkg
directory. Such a configuration is advantageous if this server has installed on it a reasonably comprehensive selection of up-to-date packages which are also used by other boxes. This is useful for maintaining a number of machines at the end of a low bandwidth upstream connection.
As an example, to use this method:
# mkdir /tmp/remote_pkg /mnt/workdir_pkg /tmp/pacman_pkg
# sshfs
remote_username
@
remote_pkgcache_addr
:/var/cache/pacman/pkg /tmp/remote_pkg -C
# mount -t overlay overlay -o lowerdir=/tmp/remote_pkg,upperdir=/var/cache/pacman/pkg,workdir=/mnt/workdir_pkg /tmp/pacman_pkg
Note
The working directory must be an empty directory on the same mounted device as the upper directory. See
Overlay filesystem#Usage
.
Tip
If listing the
/tmp/pacman_pkg
overlay directory gives errors, e.g., "Stale file handle", try overlay mounting with options
-o redirect_dir=off -o index=off
.
After this, run
pacman
using the option
--cachedir /tmp/pacman_pkg
, e.g.:
# pacman -Syu --cachedir /tmp/pacman_pkg
Distributed read-only cache
There are Arch-specific tools for automatically discovering other computers on your network offering a package cache. Try
pacredir
,
pacserve
,
pkgdistcache
AUR
, or
paclan
AUR
. pkgdistcache uses Avahi instead of plain UDP which may work better in certain home networks that route instead of bridge between Wi-Fi and Ethernet.
Historically, there was
PkgD
and
multipkg
, but they are no longer maintained.
Read-write cache
In order to share packages between multiple computers, simply share
/var/cache/pacman/
using any network-based mount protocol. This section shows how to use
SSHFS
to share a package cache plus the related library-directories between multiple computers on the same local network. Keep in mind that a network shared cache can be slow depending on the file-system choice, among other factors.
First, install any network-supporting filesystem packages:
sshfs
,
curlftpfs
,
samba
or
nfs-utils
.
Tip
To use
sshfs
, consider reading
Using SSH Keys
.
By default,
smbfs
does not serve filenames that contain colons, which results in the client downloading the offending package afresh. To prevent this, use the
mapchars
mount option on the client.
Then, to share the actual packages, mount
/var/cache/pacman/pkg
from the server to
/var/cache/pacman/pkg
on every client machine.
Warning
Do not make
/var/cache/pacman/pkg
or any of its ancestors (e.g.,
/var
) a symlink. Pacman expects these to be directories. When
pacman
re-installs or upgrades itself, it will remove the symlinks and create empty directories instead. However during the transaction
pacman
relies on some files residing there, hence breaking the update process. Refer to
FS#50298
for further details.
Two-way with rsync or FTP
Another approach in a local environment is
rsync
. Choose a server for caching and enable the
rsync daemon
. On clients synchronize two-way with this share via the rsync protocol. Filenames that contain colons are no problem for the rsync protocol.
Draft example for a client, using
uname -m
within the share name ensures an architecture-dependent sync:
# rsync ... rsync://server/share_$(uname -m)/ /var/cache/pacman/pkg/ 
# pacman -Syu
# paccache --remove --keep 3
# rsync --delete ... /var/cache/pacman/pkg/ rsync://server/share_$(uname -m)/
Instead of relying on unencrypted
rsync daemon
a more secure security option is rsync over ssh,
Rsync#Automated backup with SSH
gives an overview.
In case rsync is not available in your local environment, a simple ftp service is suitable for the two-way sync as well.
lftp
provides a
--mirror
and a
--delete
option to sync a local with a remote storage.
Synchronize pacman package cache using synchronization programs
Use
Syncthing
or
Resilio Sync
to synchronize the
pacman
cache directories (i.e.
/var/cache/pacman/pkg
).
Proxy server
For proxy server solutions, keep in mind the machines should only use HTTP mirrors, because a proxy server cannot introspect
HTTPS
connections by default.
Dynamic reverse proxy cache using nginx
nginx
can be used to proxy package requests to official upstream mirrors and cache the results to the local disk. All subsequent requests for that package will be served directly from the local cache, minimizing the amount of internet traffic needed to update a large number of computers.
In this example, the cache server will run at
http://cache.domain.example:8080/
and store the packages in
/srv/http/pacman-cache/
.
Install
nginx
on the computer that is going to host the cache. Create the directory for the cache and adjust the permissions so nginx can write files to it:
# mkdir /srv/http/pacman-cache
# chown http:http /srv/http/pacman-cache
Use the
nginx pacman cache config
as a starting point for
/etc/nginx/nginx.conf
. Check that the
resolver
directive works for your needs. In the upstream server blocks, configure the
proxy_pass
directives with addresses of official mirrors, see examples in the configuration file about the expected format. Once you are satisfied with the configuration file
start and enable nginx
.
In order to use the cache each Arch Linux computer (including the one hosting the cache) must have the following line at the top of the
mirrorlist
file:
/etc/pacman.d/mirrorlist
Server = http://cache.domain.example:8080/$repo/os/$arch
...
Note
You will need to create a method to clear old packages, as the cache directory will continue to grow over time.
paccache
(which is provided by
pacman-contrib
) can be used to automate this using retention criteria of your choosing. For example,
find /srv/http/pacman-cache/ -type d -exec paccache -v -r -k 2 -c {} \;
will keep the last 2 versions of packages in your cache directory.
Squid
Squid
proxy can be setup to only cache arch packages and can be used with aif/pacman/wget/etc with minimal configuration on the client system.
Install Squid
Install
squid
.
Configure Squid
This is the minimum configuration to get squid cache arch packages.
Cache Rules
Before defining these rules, remove/comment (if you do not need them) all the default refresh_patterns
/etc/squid/squid.conf
refresh_pattern \.pkg\.tar\.   0       20%     4320      reload-into-ims
refresh_pattern .              0       0%      0
That should define that *.pkg.tar.* gets cached, and anything else should not.
Tip
https://www.squid-cache.org/Doc/config/refresh_pattern
Maximum Filesize
Objects larger than this size will NOT be saved on disk:
/etc/squid/squid.conf
maximum_object_size 256 MB
Tip
https://www.squid-cache.org/Doc/config/maximum_object_size
Cache Directory
Set the cache dir and its maximum size and subdirs:
/etc/squid/squid.conf
cache_dir aufs /var/cache/squid 10000 16 256
Tip
https://www.squid-cache.org/Doc/config/cache_dir
Shutdown Lifetime
Time to wait until all active client sockets are closed:
/etc/squid/squid.conf
shutdown_lifetime 1 seconds
Tip
https://www.squid-cache.org/Doc/config/shutdown_lifetime
Note
Every time you change the cache_dir path (and after fresh install), you need to (re)create this directory:
# squid -z
and it could be helpful to check the configuration file before running squid:
# squid -k parse
Start Squid
Just
start
squid.service
or if squid is already running
restart
it.
Note
It could be helpful to check the configuration file before running:
# squid -k check
Follow Squid access log
To see the access to squid:
# tail -f /var/log/squid/access.log
You should see this for packages that are directed to original host:
...TCP_MISS/200...DIRECT...
and for packages that are delivered from the cache:
...TCP_HIT/200...NONE...
Manual Arch Install
On the individual machines, add
environment variables
for your proxy. To do so for testing:
# export http_proxy='http://
your_squid_machine_ip
:3128/'
# export ftp_proxy='ftp://
your_squid_machine_ip
:3128/'
Now it should use your proxy. Watch the squid logs to verify this. Once it works, add the
http_proxy
and/or
ftp_proxy
variables in an appropriate place on the installed system, e.g. in
/etc/profile.d/proxy.sh
.
Intercepting local requests
If you want all HTTP requests on local machine
automagically
go through squid, we first need to add an intercepting port for squid:
/etc/squid/squid.conf
http_port 3127 intercept
and iptables rules to redirect all (except the ones from squid) port 80 requests to squid:
# iptables -t nat -A OUTPUT -p tcp --dport 80 -m owner --uid-owner proxy -j ACCEPT
# iptables -t nat -A OUTPUT -p tcp --dport 80 -j REDIRECT --to-ports 3127
Tip
https://wiki.squid-cache.org/ConfigExamples/Intercept/LinuxLocalhost
Note
if you get random slow download speeds in vagrant/packer/virtualbox, try using
virtio
network device type.
Pacoloco proxy cache server
Pacoloco
is an easy-to-use proxy cache server for
pacman
repositories. It also allows
automatic prefetching
of the cached packages.
It can be installed as
pacoloco
. Open the configuration file and add
pacman
mirrors:
/etc/pacoloco.yaml
port: 9129
repos:
  mycopy:
    urls:
      - http://mirror.lty.me/archlinux
      - http://mirrors.kernel.org/archlinux
Restart
pacoloco.service
and the proxy repository will be available at
http://
myserver
:9129/repo/mycopy
.
Flexo proxy cache server
Flexo
is yet another proxy cache server for
pacman
repositories. Flexo is available as
flexo-git
AUR
. Once installed,
start
the
flexo.service
unit.
Flexo runs on port
7878
by default. Enter
Server = http://
myserver
:7878/$repo/os/$arch
to the top of your
/etc/pacman.d/mirrorlist
so that
pacman
downloads packages via Flexo.