saltstack/salt — Issue #285: salt as a distribution platform for large files
Author: pcarrier
Created: 2011-11-27 01:33:56+00:00
State: open

The idea is to stream large files between random minions with optimal network efficiency (given a well-designed salt topology) and without extra delays.

# Any minion has a module function to grab a file from a minion, with an ACL mechanism as already written for minion-to-minion execution.

# A syndic-like agent, let's call it "cacher" here, runs on each master and ensures that the file gets transmitted everywhere.

# The receiving minions requests request for the file to their master's cacher.

# For each request, the first common master ancestor has to be established. The file request is forwarded recursivelyfrom cacher to cacher down to the minion holding the file.

# The cachers forward the chunks as they arrive to the subscribing cachers (their parent if the file comes from a minion below them, and/or their minions if they subscribed).

# Each cacher caches every chunk, and locking ensures that a chunk is not requested multiple times by the same cacher.

# As each cacher gets new subscriptions, it serves every cached chunk and keeps receiving chunks.

# Once the file is entirely provided to a minion, the module call ends.