Package: docker-charon
docker-charonTransfer your Docker images to an air-gapped system efficiently.(An air-gapped system is a system that is not connected to the internet)From wikipedia:In Greek mythology, Charon or Kharon (/ˈkɛərɒn, -ən/; Ancient Greek: Χάρων) is a
psychopomp, the ferryman of Hades who carries souls of the newly deceased
who had received the rites of burial, across the river Acheron
(or in some later accounts, across the river Styx) that
divided the world of the living from the world of the dead.Installationpipinstalldocker-charonYou don't need Docker installed locally to run this tool.ExampleYou can run those examples directly from the command line. Here we use docker, but it's only for demonstration purposes.
In a real world scenario, you don't need Docker at all. Just two registries.Use as a command linepipinstalldocker-charon# we setup a local registry and will pretend it's air-gapped# we'll transfer docker images from dockerhub to our local registrydockerrun-d-p5000:5000--restart=always--nameregistryregistry:2


docker-charonmake-payload-f./payload.zippython:3.9.2-alpine,elasticsearch:7.14.1
docker-charonpush-payload-f./payload.zip--insecure--registry=localhost:5000# our images are now in the air-gapped registry# you can verify it withdockerpulllocalhost:5000/python:3.9.2-alpine# now let's upgrade images in our local registry without taking the layers that are already there# we take higher versions, python:3.9.2 -> python:3.9.3 for exampledocker-charonmake-payload-f./payload2.zip--already-transferred=python:3.9.2-alpine,elasticsearch:7.14.1python:3.9.3-alpine,elasticsearch:7.14.2# you'll see that some layers are skipped because they are already in the registry# the outputs will be something like this for those layers:# Skipping elasticsearch/sha256:7a0437... because it's already in the destination registry in the repository elasticsearch# the argument --already-transferred is the one that does the magicdocker-charonpush-payload-f./payload2.zip--insecure--registry=localhost:5000# you can verify it withdockerpulllocalhost:5000/python:3.9.3-alpineUse as a python librarypipinstalldocker-charon# we setup a local registry and will pretend it's air-gapped# we'll transfer docker images from dockerhub to our local registrydockerrun-d-p5000:5000--restart=always--nameregistryregistry:2fromdocker_charonimportmake_payload,push_payloadmake_payload("./payload.zip",["python:3.9.2-alpine","elasticsearch:7.14.1"])push_payload("./payload.zip",secure=False,registry="localhost:5000")# our images are now in the air-gapped registry# you can verify it with# docker pull localhost:5000/python:3.9.2-alpine# now let's upgrade images in our local registry without taking the layers that are already there# we take higher versions, python:3.9.2 -> python:3.9.3 for examplemake_payload("./payload2.zip",["python:3.9.3-alpine","elasticsearch:7.14.2"],docker_images_already_transferred=["python:3.9.2-alpine","elasticsearch:7.14.1"])# you'll see that some layers are skipped because they are already in the registry# the outputs will be something like this for those layers:# Skipping elasticsearch/sha256:7a0437... because it's already in the destination registry in the repository elasticsearch# the argument docker_images_already_transferred is the one that does the magicpush_payload("./payload2.zip",secure=False,registry="localhost:5000")# you can verify it with# docker pull localhost:5000/python:3.9.3-alpineUse as a docker imageIt has the same command line interface as the command line version, but we'll use stdin/stdout instead of-fto
avoid using docker volumes.# we setup a local registry and will pretend it's air-gapped# we'll transfer docker images from dockerhub to our local registrydockerrun-d-p5000:5000--restart=always--nameregistryregistry:2

dockerrungabrieldemarmiesse/docker-charonmake-payloadpython:3.9.2-alpine,elasticsearch:7.14.1>./payload.zip# here we use the -i argument of docker run to read from stdin (by default stdin is not available in docker)# we also use --net=host to be able to communicate with localhost:5000dockerrun-i--net=hostgabrieldemarmiesse/docker-charonpush-payload--insecure--registry=localhost:5000<./payload.zip# our images are now in the air-gapped registry# you can verify it withdockerpulllocalhost:5000/python:3.9.2-alpine# now let's upgrade images in our local registry without taking the layers that are already there# we take higher versions, python:3.9.2 -> python:3.9.3 for exampledockerrungabrieldemarmiesse/docker-charonmake-payload--already-transferred=python:3.9.2-alpine,elasticsearch:7.14.1python:3.9.3-alpine,elasticsearch:7.14.2>./payload2.zip# you'll see that some layers are skipped because they are already in the registry# the outputs will be something like this for those layers:# Skipping elasticsearch/sha256:7a0437... because it's already in the destination registry in the repository elasticsearch# the argument --already-transferred is the one that does the magicdockerrun-i--net=hostgabrieldemarmiesse/docker-charonpush-payload--insecure--registry=localhost:5000<./payload2.zip# you can verify it withdockerpulllocalhost:5000/python:3.9.3-alpineArgumentsCommand line and Docker imagedocker-charon make-payload$ docker-charon make-payload --help
Usage: docker-charon make-payload [OPTIONS] DOCKER_IMAGES_TO_TRANSFER

  Create a payload (.zip file) with docker images inside. This zip file can
  then be unpacked into a registry in another system.

  By providing images that were already transferred to the new registry, you
  can reduce the size and creation time of the payload. This is because
  docker-charon only takes the layers that were not already transferred.

  The payload is written to stdout by default. You can provide a file path
  to write the payload to by using the --file (or -f) option.

Arguments:
  DOCKER_IMAGES_TO_TRANSFER  docker images to transfer, a commas delimited
                             list of docker image names. Do not include the
                             registry name.  [required]


Options:
  -a, --already-transferred TEXT  docker images already present in the remote
                                  registry, a commas delimited list of docker
                                  image names. Do not include the registry
                                  name.

  -f, --file TEXT                 Where to write the payload zip file. If this
                                  is not provided, the payload will be written
                                  to stdout.

  -r, --registry TEXT             The registry to push the payload to. It
                                  defaults to dockerhub (registry-1.docker.io)

  -i, --insecure                  Use --insecure if the registry uses http
                                  instead of https

  -u, --username TEXT             The username to use to connect to the
                                  registry. If you want more security and
                                  don't want your username to appear in your
                                  shell history, you can also use the
                                  environment variable DOCKER_CHARON_USERNAME

  -p, --password TEXT             The password to use to connect to the
                                  registry. If you want more security and
                                  don't want your password to appear in your
                                  shell history, you can also use the
                                  environment variable DOCKER_CHARON_PASSWORDdocker-charon push-payload$ docker-charon push-payload --help
Usage: docker-charon push-payload [OPTIONS]

  Unpack the payload (.zip file) into a docker registry.

  The zip file must have been created by 'docker-charon make-payload ...'

  This command will output to stdout the list of images that were
  transferred. One image per line.

  By default, the payload is read from stdin. You can provide a file path to
  read the payload from by using the --file (or -f) option.

Options:
  -f, --file TEXT      The payload zip file. If this is not provided, the
                       payload will be read from stdin.

  -s, --strict         Fails if there is a mismatch between what was given
                       with --already-transferred and what is in the registry.
                       [default: False]

  -r, --registry TEXT  The registry to push the payload to. It defaults to
                       dockerhub (registry-1.docker.io)  [default:
                       registry-1.docker.io]

  -i, --insecure       Use --insecure if the registry uses http instead of
                       https

  -u, --username TEXT  The username to use to connect to the registry. If you
                       want more security and don't want your username to
                       appear in your shell history, you can also use the
                       environment variable DOCKER_CHARON_USERNAME

  -p, --password TEXT  The password to use to connect to the registry. If you
                       want more security and don't want your password to
                       appear in your shell history, you can also use the
                       environment variable DOCKER_CHARON_PASSWORDPython librarymake_payloadCreates a payload from a list of docker imagesAll the docker images must be in the same registry.
This is currently a limitation of the docker-charon package.If you are interested in multi-registries, please open an issue.Argumentszip_file: The path to the zip file to create. It can be apathlib.Pathor
astr. It's also possible to pass a file-like object. The payload with
all the docker images is a single zip file.docker_images_to_transfer: The list of docker images to transfer. Do not include
the registry name in the image name.docker_images_already_transferred: The list of docker images that have already
been transferred to the air-gapped registry. Do not include the registry
name in the image name. It's optional but if you use it, you can make the
payload a lot smaller.registry: The registry to pull the images from. The name of the registry
must not be included indocker_images_to_transferanddocker_images_already_transferred. Defaults to dockerhub (registry-1.docker.io).secure: Set toFalseif the registry doesn't support HTTPS (TLS). Default
isTrue.username: The username to use for authentication to the registry. Optional if
the registry doesn't require authentication.password: The password to use for authentication to the registry. Optional if
the registry doesn't require authentication.push_payloadPush the payload to the registry.It will iterate over the docker images and push the blobs and the manifests.Argumentszip_file: the zip file containing the payload. It can be apathlib.Path, astror a file-like object.strict:Falseby default. If True, it will raise an error if the
some blobs/images are missing.
That can happen if the user set an image indocker_images_already_transferredthat is not in the registry.registry: The registry to push the images to. Defaults to dockerhub (registry-1.docker.io).secure: whether to use TLS (HTTPS) or not to connect to the registry,
default isTrue.username: the username to use to connect to the registry. Optional
if the registry does not require authentication.password: the password to use to connect to the registry. Optional
if the registry does not require authentication.ReturnsThe list of docker images loaded in the registry.It also includes the list of docker images that were already present
in the registry and were not included in the payload to optimize the size.
In other words, it's the argumentdocker_images_to_transferthat you passed
to the functiondocker_charon.make_payload(...).Why such a package?The usual method: docker save and loadIt's a problem where a simple solution already exists.
You can use thedocker saveanddocker loadcommands to transfer your images to an air-gapped system with a tar.
This is actually what is recommended for simple use cases.Here is the recap of thedocker pull -> docker save -> docker load -> docker pushmethod:But let's say that you want to scale your deliveries, make regular updates, you'll soon
notice issues withdocker saveanddocker load:Speed when making the payload: The images must be pulled from the registry. Unpacked by the Docker engine, repacked again in a tar file. This is a LOT of disk access.Speed when loading the payload: The tar file is unpacked by the Docker engine. Then you can push the images to the registry. This involves a lot of unpacking and repacking of layers.Size of the payload.docker savetake all the layers and images you declare.
Even if some images and layers are already present in the air-gapped system.The docker-charon method:Docker-charon is a package that addresses these issues.It reads the registry directly to make the payload.
You don't even need docker on the machine making the payload and the machine loading the payload.It can also compute the diff in a smart way if you provide the images already present in the
air-gapped system registry. That means much smaller payloads because some layers are not
transferred a second time.Here is the recap of the docker-charon method:How does it work?docker-charon will query the registry, get the manifests of the docker images,
and then will only download the blobs that were not transferred yet to the
air-gapped registry.Everything is put in a single zip. Zip files are a good choice because it's
possible to randomly access files within it and then decompress them in
whatever order is needed.When in the air-gapped system, thepush_payloadfunction will
read the zip file index and push the blobs and the manifests to the registry on the fly.The Docker images are then ready to be pulled in your air-gapped cluster!