Dev/Targets

Note: This page is about targets for DPoS/Warrior projects. Other targets, such as those in use for ArchiveBot or for offloading during IA outages are not included in this description. Their pages are yet to be written.
If you have any questions, you can ping rewby in .
= Terminology =
WARC: An item submitted by a worker. Usually a single .warc.gz or .warc.zst file containing just the data for their ‘item’. These files are typically fairly small. Usually <3MB.
MegaWARC: The IA doesn’t handle small files well, so we combine the WARCs into a MegaWARC of at least 10GB or more.
Pack: A set of files consisting out of a MegaWARC, a metadata json file, and a .tar file containing rejected/corrupted data.
Chunk: A directory full of WARCs that will be “packed” into a pack.
Target (pipeline): A single instance of the target pipeline that handles a single project’s uploads.
Target (server): A server hosting one or more target pipelines.
= Components of a target pipeline =
Each project’s pipeline consists of several docker containers and a “data” folder.
The data folder has a number of subfolders:
* incoming
* chunker-work
* packing-queue
* packer-work-in
* packer-work-out
* upload-queue
* uploader-work
The data folder is mounted into each container.
Please don’t use docker volumes; they’re prone to accidental deletion.
== Airsync ==
An instance of rsync that receives data from workers.
It writes them to the “incoming” folder.
It first writes them as temp files before renaming them to full fledged uploaded files.
It has a mechanism that keeps track of the amount of space “free” in the “incoming directory” (via df).
Two thresholds are specified: A Soft limit and a Hard limit.
When the soft limit is reached the “max connections” setting of rsync are set to -1 to prevent it from accepting new connections but letting it complete existing transfers.
This provides a form of backpressure when things are going too fast and we start running out of disk space. (See the space considerations below for why this matters.)
If we get too low on disk space, we stop being able to pack new packs and then everything locks up.
When the hard limit is reached, the rsync daemon is killed completely.
This ensures there is sufficient space available for the packers to complete what they’re working on.
== Chunker ==
A program that watches the contents of the “incoming” folder and moves completed uploads to “chunker-work/current/”.
When the size of “chunker-work/current/” is greater than the desired MegaWARC size, the “current” folder is moved/renamed to “packing-queue/timestamp_randomhex/”.
Two versions of this program exist: One in megawarc-factory that’s written in bash and a newer one written in rust (this one’s a lot faster).
== Packer ==
A program that watches the “packing-queue/” directory for new folders moved there by the chunker.
When one is found, it is atomically moved to the “packer-work-in” folder. (Not the contents of the directory, the whole directory is moved under “packer-work-in”.)
A corresponding empty folder is created in “packer-work-out”.
The packer then takes the files in the “packer-work-in/” folder and makes a MegaWARC in the folder made in “packer-work-out”.
When the MegaWARC is complete its folder is moved to “upload-queue”.
=== ZSTD dictionaries ===
Recently we've introduced .warc.zstd megawarcs.
These use custom dictionaries to improve compression ratios.
On the final MegaWARC files these dictionaries are embedded in a skippable frame at the beginning of the file.
But the WARC files from workers don't include a dictionary.
Instead, the filename encodes the dictionary that's needed.
This dictionary is downloaded from a special server during the packing process.
If this server goes down, all packers will halt.
=== Shutdown/restart ===
When a packer gets shut down or restarted, you need to move all the directories inside of "packer-work-in" back to "packing-queue" and remove everything under "packer-work-out".
== Uploader ==
A program that watches the “upload-queue” directory and when a new sub-directory is found, it moves that directory to “uploader-work” and uploads the data to the IA (before deleting it from disk).
=== Shutdown/restart ===
When an uploader gets shut down or restarted, you need to move all the directories inside of "uploader-work" back to "upload-queue".
= Space considerations =
If we take “n” to be the size of the MegaWARC, you need the following amount of space:
* rsync: 2-3 x n
* chunker: 1 x n
* m x packer: m x (2 x n) (one copy for “packer-work-in” and one for “packer-work-out”)
* o x uploader: o x n
= Performance considerations =
Packers are single threaded and quite CPU limited.
As such, you will likely need multiple instances of the packer for each target pipeline that you’re running.
There is no scientific way to determine how many you need.
Usually 4 is a good starting point.
Uploaders only do one pack at a time and quite often single tcp connections can be quite throughput limited.
As such, you will want to run multiple of these as well.
Experience teaches that you can get ~300mbps per instance.
Generally, starting with 4 is a good idea.
Note that if a pipeline lives in the EU, things get more complicated.
Transfers across the Atlantic Ocean are quite restricted in bandwidth.
Generally you can only expect 100mbps (if that) from a single uploader instance in the EU. 
(Even with BBR and other TCP tuning flags which we employ.)
As such, you will need more uploaders.
This has implications in the Operational considerations section.
Everything in the pipeline up to (and including) “packer-work-in” deals in Lots of Small Files.
As such, you will want to run this on a fast SSD.
Additionally, this is a very write-intensive workload and is known to destroy con-/prosumer SSDs in single-digit months.
Using enterprise or even optane SSDs is recommended.
Under no circumstances do you want to run this on a HDD.
Due to how we generally expect targets to be doing upwards of 1gbps these days, we run the main permanent targets on smaller (~0.5-1TB) SSDs and just ensure we have enough space (see previous section) to pack+upload at the same (or greater) throughput than we ingest.
In some cases, however, it is desirable to hold data on the target as a buffer before uploading it.
In such cases it may be useful to have the “packer-work-out”, “upload-queue”, and “uploader-work” directories on a pool of HDDs to hold the finished packs before they get uploaded.
This is quite a situational thing and for “normal operations” the previous setup is preferred.
IMPORTANT: When you do a setup like this, it is important to maintain a specific property; mv commands between the above directories need to be “atomic” (aka be handled by a rename syscall). There are a few requirements; primarily that all these directories are on the same filesystem.
= Operational considerations =
As most permanent targets are running multiple pipelines, there are a couple of things you must keep in mind.
Notably, the disk space is shared between all pipelines.
There is a failure mode where one project gets Really Busy and takes up all the space until all projects hit their soft or hard rsync limits.
On some new target nodes we have started using zfs to host the data folders.
This allows for quotas to be set on each project’s data directory.
The quotas ensure that when a project goes Too Fast, it only backpressures itself.
This prevents some projects (urls, telegram, youtube) from stalling the rest.
== File integrity ==
When rsync completes a transfer, the file is considered “done” by the tracker.
If we lose a file on the target it gets really fucky or even impossible to recover from this.
== Internet Archive Capacity ==
When we upload to the IA's S3 interface, we must consider a few things.
The S3 interface consists of an HAProxy instance and a number of "workers" behind it.
Each worker can only handle a certain amount of parallel connections.
We call these connection "slots".
As a result there is an upper limit to how many parallel uploads the IA can accept.
This is an important thing to consider.
We have agreements with the IA on how many slots we can use.
If we have a lot of uploads from the EU, the uploads take longer as we have less throughput.
As a result we end up using more slots in total.
As such, targets doing the upload from the US are preferred as they can get more bandwidth per connection.
And thus do more with less slots.
Additionally, there are also overall bandwidth limits on the S3's ingest.
We need to be careful with those too so we don't clog the whole S3.