# CargoSync: Efficient Containerd Image Updates in Low-Bandwidth Edge Environments with Rsync-based Delta Compression

Ioannis Protogeros, Michail Rizakis, Antonios Porichis, Michalis Karamousadakis

► To cite this version:

Ioannis Protogeros, Michail Rizakis, Antonios Porichis, Michalis Karamousadakis. CargoSync: Efficient Containerd Image Updates in Low-Bandwidth Edge Environments with Rsync-based Delta Compression. 2024 IEEE International Conference on Edge Computing and Communications (EDGE), Jul 2024, Shenzhen, France. pp.22-28, 10.1109/EDGE62653.2024.00013.1. hal:04718136

HAL Id: hal-04718136

https://hal.science/hal-04718136v1

Submitted on 2 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
# CargoSync: Efficient Containerd Image Updates in Low-Bandwidth Edge Environments with Rsync-based Delta Compression

Ioannis Protogeros
TWI Hellas &

National Technical
University of Athens

Michail Rizakis
University of Essex

Antonios Porichis
University of Essex

Michalis Karamousadakis
Plaixus Ltd.

***Abstract***—Containers have emerged as the standard for software deployment, offering simplicity, low overhead, and portability. In the context of updating containers in Edge environments, the constrained bandwidth poses the critical challenge of minimizing data transfer and optimizing the update process. This paper introduces CargoSync, a Containerd-based tool designed for efficient image updates. It leverages the rsync tool for delta compression, reducing the size of updates towards saving network resources. The proposed method is evaluated against traditional pull-based updates showcasing notable improvements in data transfer efficiency as well as update times even for large container images when bandwidth is severely limited. Our tool competes favourably with Moby-based counterpart Balena-engine's rsync delta updates in our performance evaluations, making it the first Containerd-based tool to effectively adopt this approach. Additionally, CargoSync addresses the challenge of distributing updates to multiple machines concurrently, demonstrating scalability benefits. The results indicate the potential of CargoSync for optimizing container image updates in resource-constrained environments.

***Index Terms***—Containerization, Delta Encoding, Image Updates, Resource-constrained environments, Containerd, rsync

## I. INTRODUCTION

Containers are becoming the new standard of software deployment due to their simple usage, low overhead, and portability [1]. They allow software to run in an isolated, controlled environment in a way that is predictable, repeatable, and without the uncertainty of inconsistent code execution across diverse development and production environments. While containers and virtual machines (VMs) share similar objectives, containers have the advantage of being more lightweight. They package only the necessary files required to execute the specific software unit, eliminating the need for an entire operating system [2], [3].

Container orchestration becomes necessary to deal with the operational complexity of containerized microservice environments [4]. Among the various container orchestrators available, Kubernetes (K8s) has emerged as the widely adopted standard in the industry, with its advantages including extensive capabilities, portability, and being highly declarative [5]. As of v1.24, Dockershim is deprecated in Kubernetes and the default container runtime is Containerd [6], [7].

Containerd is an industry-standard container runtime that is recognized for its simplicity, robustness, and portability. It is responsible for managing the entire lifecycle of containers within its host system, encompassing tasks from image transfer and storage to container execution and supervision, as well as low-level storage and network attachments. The typical workflow towards deploying a container starts with a *pull* operation to download the container image and unpack its contents and then mount them using a union filesystem [8], [9]. Updating container images is a relatively efficient process due to the images being comprised of a stack of layers. There is no need to rebuild the whole image, rather than only the updated layers. Likewise, containers can be easily extended by adding layers on top of the image. This saves a considerable amount of data when the previous and the target image share common layers. However, when we take into account larger images (such as ML model containers), it is not uncommon to have layers sizing up to GBs [10] which change slightly over an update and as a result need to be pulled again in their entirety, even though they are largely identical [11]. This is especially problematic in Edge environments, where the network resources are often severely limited.

In this paper, we propose a method to reduce the size of updates using delta compression. The rsync [12], [13] tool is used, which specializes in syncing large filesystems efficiently using a rolling checksum algorithm [14]. More specifically, its ability to produce delta bundles is utilized to create updates that can be distributed to multiple client machines, saving substantial amounts of both time and data transferred over the network. We examine the scenario where the client(s) on the Edge are machines with limited resources requesting an image update from a registry served by a server machine on the Cloud with average computational resources. We examine updates of similar image versions for image sizes ranging from 100 MB to several Gigabytes, focusing on ML containers such as nvidia/tensorflow.

We observe at least 70% of transferred data being saved through delta encoding and zstd compression [15] for the examined minor updates, with the compression being almost 2% of the original size for a specific update. For a very
limited bandwidth (10Mbps), these updates require 25-70% less time compared to the baseline. For increased bandwidth, the updates require more time because of the trade-off of a more computationally intensive updating process for less data transferred through the network. For this exact reason, however, when many machines require the same update, the client-side updates happen in parallel, leading to the total process being completed considerably faster.

## II. RELATED WORKS

**Lazy Pulling** Lazy pulling [16], [17] is a strategy aimed at reducing pull times by fetching only the files necessary to start a container, deferring the retrieval of the remaining files until later. This approach aims to expedite the container startup process, allowing users to access the basic functionality of the container more quickly. Despite its efficiency in enhancing initial deployment speed, it's essential to note that by design lazy pulling will still need to resend whole essential files in the context of an image update. While it minimizes the files fetched until the container can be run, the complete set of files that need to be accessed still needs to be retrieved eventually, which can lead to an increase of redundant data transferred, especially for large ML libraries.

**Approaches to Delta Updates** Approaches have been made in the past to utilize delta encoding in the context of image updates that show a few promising results, but also have several limitations. Based on these, we motivated our work on the proposed solution that improves certain aspects and is analyzed in the next sections. **Starlight** is an accelerator for container provisioning that implements among other features both lazy-pulling and updates with delta differences. In their paper [18], Chen et al. examine the effect of only pulling changed files over updates, which substantially optimizes the update size. This is to be expected, especially due to an analysis of the Docker Hub registry [19] indicating that while only 10% of layers are referenced by more than one image, over 99.4% of files had duplicates. While this updating method offers substantial update size reduction for most images found on Docker Hub, we figure that when it comes to images that contain hundreds, if not thousands of Megabytes worth of files in shared libraries (which is usually the case in ML model containers), it can be beneficial to change the granularity of the update to a finer level, closer to the one of binary updates. This is backed by Natsumoto, et al. who in their work [20] develop a system (**d4c**) that among other features optimizes container provisioning by utilizing binary delta differences towards improving the compression rate of updates, and indeed confirm the improved data efficiency of updating files such as shared libraries with binary deltas. However, **d4c** utilizes the bsdiff tool, which may hinder performance for large image updates because of its high memory usage [21]. Another relevant work is **balena-engine**, which utilizes rsync-based differential updates but is based on Moby/Docker instead of Containerd [22], and since Moby has diverged from Kubernetes-focused development [23], this incompatibility hinders cloud-native ecosystems such as Kubernetes clusters to be benefited from it

<figure><img src="image_2.png" /><figcaption>Fig. 1: CargoSync Architecture - Interaction with external components.</figcaption></figure>

seamlessly. After the deprecation of Docker by Kubernetes [6], [7], we argue that a similar tool that directly communicates with Containerd should be created for container environments orchestrated by Kubernetes, or its lightweight, more suited for the Edge alternative K3s, which also uses Containerd as its runtime. In our work, we build a system that utilizes the rsync tool and Containerd's existing components to also implement rsync delta updates. Emphasis is put on creating an efficient and straightforward workflow, all while minimizing dependencies to ensure seamless compatibility with other systems.

## III. CARGOSYNC

During our analysis of the related works, we spot no example of a tool that leverages rsync's feature of creating batch differences efficiently for large filesystems towards updating large-size images in Containerd, similar to how balena-engine does this for Moby. To this aim, we implemented CargoSync, a tool that handles this task. It only uses the existing Containerd components, so this tool can be used immediately instead of a normal pull operation, the only dependency being rsync and the Go programming language. There is no need to install a custom Snapshotter since overlayfs is used. In the next sections the implementation of the CargoSync tool will be described, including the design considerations needed for integrating our task with Containerd's components.

### A. Overview

Figure 1 shows a high-level overview of CargoSync's architecture. It consists of two main Client and Server components that orchestrate the workflow of updating a Containerd image on the worker machine, using an existing older-version image. A high-level description of this event sequence follows.

The user makes a delta pull request through the CargoSync client ①, specifying the target image and optionally the source image. If the source image is not provided, the containerd client in the worker machine is queried for the
**Algorithm 1 Same Update Distribution**

1: Receive update request $A \to A'$
2: **if** mutex[$A \to A'$] does not exist **then**
3:     mutex[$A \to A'$] $\leftarrow 1$ // unlocked
4:     **end if**
5:     **while** mutex[$A \to A'$] $= 0$ **do**
6:         Wait for the delta to generate from the other process in the ($A \to A'$)-critical section
7:         **end while**
8:         mutex[$A \to A'$] $\leftarrow 0$ // lock the mutex
9:         ($A \to A'$)-critical section: Check if the ($A \to A'$) Delta Bundle already exists. If it does, continue. If it does not, generate it.
10:        mutex[$A \to A'$] $\leftarrow 1$ // unlock the mutex
11:    Transfer delta update to client

locally available images with the same name as the target image and the newest tag version is chosen as the source image. Once the server component receives the request containing the source and target image references, it pulls the images that are not present in the server (2). Then, these images are unpacked and mounted with `containererd`'s `Snapshotter overlayfs`. At this point, the server machine has an available view of the root filesystem of both images. It uses `rsync`, therefore, in batch mode, to produce a batch file containing the binary differences (deltas) of the two filesystems. Right after that the delta file is compressed with `zstd` and sent over the network (4) along with the target image's manifest as a response to the client's update request. Then the client mounts the source image (5), using `overlayfs` just like the server before. Now, the client machine can use `rsync` (6) to apply the delta batch file to the root filesystem of the source image. This results in the client having an available filesystem that represents the updated image. The client utilizes then `containererd`'s `Content and Diff` services to generate a layer of the updated image's entire content (7). A new image is then generated based on that layer, which as far as the filesystem content is concerned, is identical to the target image. The image size is also the same, if not slightly reduced because of the squashing that is done in the process. In the end, the image is unpacked, so that it can be used immediately.

## B. Delta Compression - rsync

`rsync`, or Remote Sync, is a powerful file synchronization tool that was chosen for its efficiency and reliability when dealing with large filesystems. It operates by transmitting only the differences between the source and target files, rather than sending entire files. This differential approach, often referred to as delta encoding, significantly reduces the amount of data that needs to be transferred, making `rsync` particularly effective for large filesystems. Moreover, it preserves symbolic links, file permissions, user and group ownerships and timestamps when synchronizing files across machines.

In the context of CargoSync, these features of `rsync` enable efficient, reliable, and secure synchronization of con-

tainer images across different machines. By transmitting only the differences between images, CargoSync can update large container images with minimal network overhead as opposed to pulling the updated image. Delta generation is faster too, due to `rsync`'s rolling checksum algorithm that allows the identification of changed data blocks without the comparison of files in their entirety [14].

`rsync` is most frequently used for synchronizing filesystems in remote directories. However, we mainly use `rsync`'s batch mode to create delta update bundles that can potentially be used across many worker machines. After creation, this delta bundle is compressed with the `zstd` tool. While `rsync` can natively use various compression methods, the only supported method for batch mode is (as of now) `zlib`, and `zstd` is substantially superior both in terms of compression & uncompression speed and compression ratio achieved [24].

## C. Creating the Updated Image

Once the client has a view of the updated image filesystem (after (6)), the new image has to be created based on that view. A naive approach that was initially examined was to use `nerdctl commit` on a container after applying the delta update at its root filesystem. However, this was not a viable approach because of the way committing container changes works. Since `overlayfs` provides a read-write "upper" layer [9] on top of the read-only layers that comprise the image filesystem, any changes made to the root filesystem are represented in that layer. Hence, committing these changes would put a layer on top of the existing image, and the new image would be considerably larger, which is highly undesirable.

What eventually worked was to create a new image based on the contents of the updated filesystem. Since `overlayfs` only provides a read-only view of the underlying layers, and thus they cannot be directly modified, we used the `containererd` diff service to create a layer containing the updated filesystem. To do that, we compare this filesystem with that of an empty layer. That creates a new layer which is stored in the local Content Store of `containererd`, effectively creating a squashed version of the initial target image.

The final step towards creating the new image is to reference the created layer in the manifest file. Towards that end, we used a modified version of the `godarch/darch` [25] tool's code for manipulating manifests. The worker, not having the target image available locally, retrieves the target image's manifest from the server. If the target image manifest is distributed as a manifest list, then the manifest corresponding to the worker machine's architecture and runtime is retrieved. After that, and after the aforementioned updated layer has been saved in the Content Store, we manipulate the manifest, only replacing the referenced layers with the digest reference of the squashed layer we created. We end up with a manifest that contains the same metadata as the original, but referencing the new layer that is available locally. Therefore the updated image can now be created and unpacked.
<table><caption>TABLE I: Container Image Updates and Sizes</caption><thead><tr><th>Update Ref.</th><th>Image</th><th>Update</th><th>Size</th></tr></thead><tbody><tr><td>zk1</td><td>docker.io/library/zookeeper</td><td>3.9.1 → latest</td><td>114.5MB</td></tr><tr><td>tf1</td><td>docker.io/tensorflow/tensorflow</td><td>latest → nightly</td><td>524.7MB</td></tr><tr><td>tf2</td><td>nvcr.io/nvidia/tensorflow</td><td>18.01-py3 → 18.02-py3</td><td>1.379GB</td></tr><tr><td>gcsdk1</td><td>docker.io/google/cloud-sdk</td><td>453.0.0 → 454.0.0</td><td>1.153GB</td></tr><tr><td>tf3</td><td>nvcr.io/nvidia/tensorflow</td><td>19.11-tf2-py3 → 19.12-tf2-py3</td><td>3.368GB</td></tr><tr><td>tf4</td><td>nvcr.io/nvidia/tensorflow</td><td>23.08-tf2-py3 → 23.09-tf2-py3</td><td>7.510GB</td></tr><tr><td>pt1</td><td>docker.io/pytorch/pytorch</td><td>{2.1.0→2.1.1}-cuda12.1-cudnn8-devel</td><td>8.862GB</td></tr></tbody></table>

<figure><img src="image_3.png" /><figcaption>Fig. 2: Times required for each update using different updating methods. For the rsync-based updates (Proposed, Balena), the ratios of the update times compared to baseline are also shown as a line graph. In both graphs, a lower value is better.</figcaption></figure>

## D. Update Distribution to Multiple Machines

A significant use case scenario for our system would be the distribution of the same image update to multiple edge workers in different machines. This could apply for example to robotic swarms [26]–[28], or workers of a distributed system that run the same image and an updated version is released. With that in mind, we design the server application to handle simultaneous requests for the same image update. It is obvious in this case that the delta bundle only needs to be created once, and then it can be distributed as-is to all the requesting machines. To do that, each time a request is received it is checked whether the requested delta bundle already exists. If it doesn’t exist, then the delta generation process commences. However, it must be ensured that if there is another request for the same update that has entered the process of generating the update, then the latter request will not try to generate the update too. Instead, it will wait for the delta diff generation of the first request and then send that bundle. Towards that end, we map each image update that is requested to a mutex [29]. Thus, we ensure that only one process associated with each specific image update enters the code at the same time. The other processes “wait” just before the part where it is checked whether the delta bundle exists locally, so when the first process finishes with the delta generation, the waiting processes continue and proceed with transferring the already generated bundle. Algorithm 1 gives a high-level overview of how the mutexes are used to ensure that only one process of each update can access the instructions to

generate the delta update.

## IV. EVALUATION

We examine a series of image updates, that are presented in table I. We focus on minor version updates of various image sizes, from around 100MB to over several Gigabytes in compressed size for certain ML library images. We are interested in the time needed to create and apply a container image update, the compression achieved with rsync, and the efficiency of distributing an update to multiple machines.

### A. Experimental Setup

We use Google Cloud VMs from the general-purpose e2 family to run our experiments. For the worker machines, we use e2-medium type machines with single-core (2 vCPUs) CPU at 2.25 GHz and 4GB memory [30]. For the server machine, we use a e2-highcpu-8 machine which has a 4-core CPU (8 vCPUs) at 2.25 GHz and 4 GB memory [31]. We try to simulate the realistic scenario where the workers have quite fewer resources than the server, without however using a server machine that is far more capable. We also use the Wondershaper [32] tool -which uses Linux’s tc [33]- to throttle the bandwidth, simulating a Cloud-Edge link.

As for rsync’s usage, the only way we deviate from the tool’s standard configuration is that we use a different **block size**. By default, rsync uses the following heuristic for determining the block size that will be used upon comparing
the file to be updated with its updated version: If the file is smaller than 490KB, then a block size of 700 bytes is used. Otherwise, a block size equal to the root of the file size is used [12]. We found that using a smaller and constant block size yielded significantly better results in terms of the compression achieved and the total time needed to update an image. We observed that for each update a different block size was "optimal" for the update (e.g. for the `gcsdk` update a block size of 128 bytes led to a compression ratio of 13.65 without increasing substantially the delta creation time, while the same block size proved inefficient for most other updates). We suppose that the optimal block size for an update is related to the way the differences between files are fragmented. In our experiments, we use a block size of 386 bytes for all updates, since we found that it performs well in general, even if for some larger image updates we observed a larger size would yield better results.

## B. Delta Update Time

First, we compare the time that is needed to update an image. We showcase the results in figure 2. We suppose that the server machine needs to pull both images that are to be compared. We compare our times to the time needed to pull the entire updated layers, which we consider the baseline method. We also measure the performance of the balena-engine and d4c tools. The former was chosen since it also uses an rsync based delta update mechanism - so we are confident that our Containerd-based tool can match the performance of Moby-based balena-engine's implementation of delta updates. The latter was chosen as a tool that worked with Containerd and implemented delta updates, though with the usage of bsdiff. We observe that the d4c tool takes more time for the image updates. We attribute this to two reasons: First, the docker-squash takes a considerable amount of time, and second, the usage of the bsdiff tool is not efficient for larger images due to its high memory usage [21]. We find that in environments with extremely low bandwidth (10Mbps), our updating method is about as efficient as balena-engine's. Balena-engine uses a different rsync block size at 512 bytes [34], [35], along with an internal configuration that differs from the default rsync one. At a bandwidth of 50Mbps, the baseline method is faster, since there is a trade-off of a more computationally intensive updating process for less data transferred over the network.

## C. Rsync update sizes

Next, we compare the update bundle sizes produced by different delta encoding methods. Specifically, we measure the sizes of the updates that are produced with rsync during the usage of both balena-engine and CargoSync. We also measure the sizes of whole-file delta updates, which are used in systems such as Starlight [18]. For these types of updates, we use rsync with the option of transferring whole files for updated files. We observe in Figure 3 that using binary/rsync updates generally improves the achieved compression ratio, especially for the updates that we are interested in, which are

for images that are comprised of many binary files, such as shared libraries. We observe a fluctuation of the compression achieved between balena-engine and our usage of rsync, which we can attribute to the differing configuration of rsync (and mainly the block-size, as mentioned before).

## D. Multiple-Machine Updates

Now we examine the scenario where multiple machines request the same update, as described in Section III-D. In Figure 4 we show two Gantt diagrams of the updating process via pulling the entire new image and using delta updates, for 4 machines, and one of the updates with the lowest compression ratio and largest sizes. The bandwidth of 50Mbps is equally divided between the machines. It becomes evident that the significant reduction of transferred data and the parallelization between the machines when it comes to the application of the update leads to a notable reduction in the total updating time. This indicates a solution that scales a lot better for many machines that need the same update.

# V. CONCLUSION

**Limitations** While our implementation of delta updates for Containerd significantly reduces the data that needs to be transferred over a Cloud-Edge link and the update time needed for limited enough bandwidth environments, areas of improvement exist. It becomes evident that unless the network speed is extremely limited, the processes of creating the update bundle and applying it become a bottleneck, especially for constrained worker machine resources. Furthermore, the constant rsync configuration may not always lead to the best possible compression. We notice from the results in Figure 3 that the rsync configurations in balena-engine and CargoSync lead to better compression results in different update scenarios.

**Discussion and Future Works** Based on the aforementioned limitations of our system, we want to examine the employment of certain techniques to further accelerate container update provisioning. Such an example would be implementing lazy pulling for delta differences so that the container can start earlier with only the needed updated files while still minimizing the transferred data. The delta-merging technique that is proposed in the work of Natsumoto et al. [20] also seems promising to reduce the time needed to generate a delta bundle if other deltas are already generated. Finally, it would be interesting to observe if we can use data about the updates (i.e. image sizes, number of files, file sizes, differences, file type composition) to train an ML model that predicts the configuration of the rsync algorithm (e.g. block size) which optimizes the updating time, compression with delta encoding, or a metric that combines the two.

## AVAILABILITY

The CargoSync tool is available as an open-source project at https://github.com/CargoSync/cargosync
<figure><img src="image_4.png" /><figcaption>Fig. 3: Comparison of update sizes in MBs (left) and in compression ratios with regards to the original Image size (right).</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4: Parallel comparison in the form of a Gantt chart of the updating processes with the baseline method (pulling the updated layers) and the proposed method. In the former method, we suppose that the update requests are made simultaneously, while in the latter the requests after the first one can be made anytime before the delta is created.</figcaption></figure>

## ACKNOWLEDGEMENTS

This work has received funding from the European Union's Horizon Europe programme under Grant Agreement N°101135959.

## REFERENCES

[1] Víctor Medel Gracia, O. Rana, J. A. Banares, and U. Arronategui. Modelling performance resource management in kubernetes. 2016 IEEE/ACM 9th International Conference on Utility and Cloud Computing (UCC), pages 257–262, 2016. https://www.researchgate.net/publication/311622401_Modelling_performance_resource_management_in_kubernetes.

[2] Amit M Potdar, Narayan D G, Shivaraj Kengond, and Mohammed Moin Mulla. Performance evaluation of docker container and virtual machine. Procedia Computer Science, 171:1419–1428, 2020. Third International Conference on Computing and Network Communications (CoCoNet’19) https://www.sciencedirect.com/science/article/pii/S1877050920311315.

[3] Docker. What is a container? Accessed: Dec 19, 2023 https://www.docker.com/resources/what-container/.

[4] Red Hat. What is container orchestration?, 2022. Accessed: Dec 29, 2023 https://www.redhat.com/en/topics/containers/what-is-container-orchestration.

[5] VMware. What is container orchestration?, 2022. Accessed: Dec 29, 2023 https://www.vmware.com/topics/glossary/content/container-orchestration.html.

[6] Sergey Kanzhelev, Jim Angel, Davanum Srinivas, Shannon Kularathna, Chris Short, and Dawn Chen. Kubernetes is moving on from docker-shim: Commitments and next steps. https://kubernetes.io/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/, 2022.

[7] Kubernetes. Dockershim deprecation faq. https://kubernetes.io/blog/2022/02/17/dockershim-faq/, 2022.

[8] Containerd Documentation. Content flow, 2023. Accessed: Dec 16, 2023 https://github.com/containerd/containerd/blob/main/docs/content-flow.md.

[9] kernel.org. Overlay filesystem - the linux kernel documentation. Accessed: Dec 16, 2023 https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html.

[10] Docker Hub. Pytorch - docker hub. Accessed: Dec 19, 2023 https://hub.docker.com/r/pytorch/pytorch/tags.

[11] Dimitris Skourtis, Lukas Rupprecht, Vasily Tarasov, and Nimrod Megiddo. Carving perfect layers out of docker images. In 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19), Renton, WA, July 2019. USENIX Association. https://www.usenix.org/conference/hotcloud19/presentation/skourtis.

[12] Linux Die. rsync man page. Accessed: Dec 15, 2023 https://linux.die.net/man/1/rsync.

[13] Samba. Rsync. Accessed: Dec 15, 2023 https://rsync.samba.org/.

[14] Andrew Tridgell, Paul Mackerras, et al. The rsync algorithm. 1996. https://openresearch-repository.anu.edu.au/handle/1885/40765.
[15] Yann Collet and Murray Kucherawy. Zstandard Compression and the application/zstd Media Type. RFC 8478, October 2018. https://www.rfc-editor.org/info/rfc8478.

[16] NTT Laboratories Kohei Tokunaga. Startup containers in lightning speed with lazy image distribution on containerd. https://medium.com/nttlabs/startup-containers-in-lightning-speed-with\-lazy-image-distribution-on-containerd\-243d94522361, 2020. Accessed: January 7, 2024.

[17] Tyler Harter, Brandon Salmon, Rose Liu, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. Slacker: Fast distribution with lazy docker containers. In *14th USENIX Conference on File and Storage Technologies (FAST 16)*, pages 181–195, Santa Clara, CA, February 2016. USENIX Association.

[18] Jun Lin Chen, Daniyal Liaqat, Moshe Gabel, and Eyal de Lara. Starlight: Fast container provisioning on the edge and over the WAN. In *19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22)*, pages 35–50, Renton, WA, April 2022. USENIX Association. https://www.usenix.org/conference/nsdi22/presentation/chen-jun-lin.

[19] Dimitris Skourtis, Lukas Rupprecht, Vasily Tarasov, and Nimrod Megiddo. Carving perfect layers out of docker images. In *11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19)*, Renton, WA, July 2019. USENIX Association. https://www.usenix.org/conference/hotcloud19/presentation/skourtis.

[20] Naoki Matsumoto, Daisuke Kotani, and Yasuo Okabe. Efficient container image updating in low-bandwidth networks with delta encoding. In *2023 IEEE International Conference on Cloud Engineering (IC2E)*, pages 1–10, 2023. https://ieeexplore.ieee.org/document/10305845.

[21] Daemonology. bsdiff. Accessed: Dec 19, 2023 https://www daemonology.net/bsdiff/.

[22] Argemiro Ferreira. Docker vs. containerd: Unraveling the differences and analyzing performance. Medium, 2023. Accessed: January 7, 2024 https://medium. com/@argferreira1/docker-vs-containerd-unraveling-the-\ differences-and-analyzing-performance-\9cdd369ead17.

[23] Loraine Johnson. The moby project post-kubernetes: 3 new releases in 2023, October 2023. Accessed: Dec 29, 2023 https://thenewstack.io/ the-moby-project-post-kubernetes-3-new-\releases-in-2023.

[24] Facebook. Zstandard - fast real-time compression algorithm. Accessed: Dec 21, 2023 https://github.com/facebook/zstd.

[25] Paul Knopf. godarch/darch: A tool for building and booting stateless and immutable images, bare metal. Accessed: Dec 21, 2023 https://github.com/godarch/darch.

[26] Simon Jones, Emma Milner, Mahesh Sooriyabandara, and Sabine Hauert. Dots: An open testbed for industrial swarm robotic solutions, 2022. https://arxiv.org/abs/2203.13809.

[27] Docker Hub. Ros - official image on docker hub. Accessed: Dec 27, 2023 https://hub.docker.com/_/ros.

[28] Docker Hub. Gazebo on docker hub. Accessed: Dec 27, 2023 https://hub.docker.com/_/gazebo.

[29] Ingo Molnar and Davidlohr Bueso. Generic mutex subsystem. Accessed: Dec 22, 2023 https://www.kernel.org/doc/html/latest/locking/ mutex-design.html.

[30] Google Cloud. General-purpose machine family for compute engine. Accessed: Dec 27, 2023 https://gcloud-compute.com/e2-medium.html.

[31] Google Cloud. Google compute engine machine type e2-highcpu-8. Accessed: Dec 27, 2023 https://gcloud-compute.com/e2-highcpu-8.html.

[32] magnific0. wondershaper: Command-line utility for limiting an adapter's bandwidth. Accessed: Dec 21, 2023 https://github.com/magnific0/ wondershaper.

[33] Linux man-pages project. tc(8) - linux manual page. Accessed: Dec 21, 2023 https://man7.org/linux/man-pages/man8/tc.8.html.

[34] Balena OS. image_delta.go in balena-engine source code, 2023. Accessed: Dec 22, 2023 https://github.com/balena-os/balena-engine/blob/ master/daemon/images/image_delta.go.

[35] balena os. librsync-go/signature_test.go. https://github.com/balena-os/ librsync-go/blob/7b435f8f590637e8ba24e72f7cfc2d62f17e3848/ signature_test.go#L22. Accessed on 4 Jan 2024.