Package: mla-archive
Multi Layer Archive (MLA)MLA is an archive file format with the following features:Support for compression (based onrust-brotli)Support for authenticated encryption with asymmetric keys (AES256-GCM with an ECIES schema over Curve25519, based onRust-Cryptoaes-ctrandDalekCryptographyx25519-dalek)Effective, architecture agnostic and portable (written entirely in Rust)Small memory footprint during archive creationStreamable archive creation:An archive can be built even over a data-diodeA file can be added through chunks of data, without initially knowing the final sizeFile chunks can be interleaved (one can add the beginning of a file, start a second one, and then continue adding the first file's parts)Archive files are seekable, even if compressed or encrypted. A file can be accessed in the middle of the archive without reading from the beginningIf truncated, archives can be repaired. Files which were still in the archive, and the beginning of the ones for which the end is missing, will be recoveredArguably less prone to bugs, especially while parsing an untrusted archive (Rust safety)RepositoryThis repository contains:mla: the Rust library implementing MLA reader and writermlar: a Rust utility wrappingmlafor common actions (create, list, extract, ...)curve25519-parser: a Rust library for parsing DER/PEM public and private Ed25519 keys and X25519 keys (as made byopenssl)mla-fuzz-afl: a Rust utility to fuzzmlabindings: bindings for other languages.github: Continuous Integration needsQuick command-line usageHere are some commands to usemlarin order to work with archives in MLA format.# Generate an X25519 key pair {key, key.pub} (OpenSSL could also be used)mlarkeygenkey# Create an archive with some files, using the public keymlarcreate-pkey.pub-omy_archive.mla/etc/os-release/etc/issue# List the content of the archive, using the private keymlarlist-kkey-imy_archive.mla# Extract the content of the archive into a new directory# In this example, this creates two files:# extracted_content/etc/issue and extracted_content/etc/os-releasemlarextract-kkey-imy_archive.mla-oextracted_content# Display the content of a file in the archivemlarcat-kkey-imy_archive.mla/etc/os-release# Convert the archive to a long-term one, removing encryption and using the best# and slower compression levelmlarconvert-kkey-imy_archive.mla-olongterm.mla-lcompress-q11# Create an archive with multiple recipientmlarcreate-parchive.pub-pclient1.pub-omy_archive.mla...mlarcan be obtained:through Cargo:cargo install mlarusing thelatest releasefor supported operating systemsQuick API usageCreate an archive, with compression and encryption:usecurve25519_parser::parse_openssl_25519_pubkey;usemla::config::ArchiveWriterConfig;usemla::ArchiveWriter;constPUB_KEY:&[u8]=include_bytes!("samples/test_x25519_pub.pem");fnmain(){// Load the needed public keyletpublic_key=parse_openssl_25519_pubkey(PUB_KEY).unwrap();// Create an MLA Archive - Output only needs the Write traitletmutbuf=Vec::new();// Default is Compression + Encryption, to avoid mistakesletmutconfig=ArchiveWriterConfig::default();// The use of multiple public keys is supportedconfig.add_public_keys(&vec![public_key]);// Create the Writerletmutmla=ArchiveWriter::from_config(&mutbuf,config).unwrap();// Add a filemla.add_file("filename",4,&[0,1,2,3][..]).unwrap();// Complete the archivemla.finalize().unwrap();}Add files part per part, in a "concurrent" fashion:...// A file is tracked by an id, and follows this API's call order:// 1. id = start_file(filename);// 2. append_file_content(id, content length, content (impl Read))// 2-bis. repeat 2.// 3. end_file(id)// Start a file and add contentletid_file1=mla.start_file("fname1").unwrap();mla.append_file_content(id_file1,file1_part1.len()asu64,file1_part1.as_slice()).unwrap();// Start a second file and add contentletid_file2=mla.start_file("fname2").unwrap();mla.append_file_content(id_file2,file2_part1.len()asu64,file2_part1.as_slice()).unwrap();// Add a file as a wholemla.add_file("fname3",file3.len()asu64,file3.as_slice()).unwrap();// Add new content to the first filemla.append_file_content(id_file1,file1_part2.len()asu64,file1_part2.as_slice()).unwrap();// Mark still opened files as finishedmla.end_file(id_file1).unwrap();mla.end_file(id_file2).unwrap();Read files from an archiveusecurve25519_parser::parse_openssl_25519_privkey;usemla::config::ArchiveReaderConfig;usemla::ArchiveReader;usestd::io;constPRIV_KEY:&[u8]=include_bytes!("samples/test_x25519_archive_v1.pem");constDATA:&[u8]=include_bytes!("samples/archive_v1.mla");fnmain(){// Get the private keyletprivate_key=parse_openssl_25519_privkey(PRIV_KEY).unwrap();// Specify the key for the Readerletmutconfig=ArchiveReaderConfig::new();config.add_private_keys(&[private_key]);// Read from buf, which needs Read + Seekletbuf=io::Cursor::new(DATA);letmutmla_read=ArchiveReader::from_config(buf,config).unwrap();// Get a fileletmutfile=mla_read.get_file("simple".to_string()).unwrap()// An error can be raised (I/O, decryption, etc.).unwrap();// Option(file), as the file might not exist in the archive// Get back its filename, size, and dataprintln!("{} ({} bytes)",file.filename,file.size);letmutoutput=Vec::new();std::io::copy(&mutfile.data,&mutoutput).unwrap();// Get back the list of files in the archive:forfnameinmla_read.list_files().unwrap(){println!("{}",fname);}}:warning: Filenames areStrings, which may contain path separator (/,\,.., etc.). Please consider this while using the API, to avoid path traversal issues.Using MLA with others languagesBindings are available for:C/CPPDesignAs the name spoils it, an MLA archive is made of several, independent, layers. The following section introduces the design ideas behind MLA. Please refer toFORMAT.mdfor a more formal description.LayersEach layer acts as aUnix PIPE, taking bytes in input and outputting in the next
layer.
A layer is made of:aWriter, implementing theWritetrait. It is responsible for emitting bytes while creating a new archiveaReader, implementing bothReadandSeektraits. It is responsible for reading bytes while reading an archiveaFailSafeReader, implementing only theReadtrait. It is responsible for reading bytes while repairing an archiveLayers are made with therepairableproperty in mind. Reading them must never need information from the footer, but a footer can be used to optimize the reading. For example, accessing a file inside the archive can be optimized using the footer to seek to the file beginning, but it is still possible to get information by reading the whole archive until the file is found.Layers are optional, but their order is enforced. Users can choose to enable or disable them.
Current order is the following:File storage abstraction (not a layer)Raw layer (mandatory)Compression layerEncryption layerPosition layer (mandatory)Stored bytesOverview+----------------+-------------------------------------------------------------------------------------------------------------+
| Archive Header |                                                                                                             | => Final container (File / Buffer / etc.)
+------------------------------------------------------------------------------------------------------------------------------+
                 +-------------------------------------------------------------------------------------------------------------+
                 |                                                                                                             | => Raw layer
                 +-------------------------------------------------------------------------------------------------------------+
                 +-----------+---------+------+---------+------+---------------------------------------------------------------+
                 | E. header | Block 1 | TAG1 | Block 2 | TAG2 | Block 3 | TAG3 | ...                                          | => Encryption layer
                 +-----------+---------+------+---------+------+---------------------------------------------------------------+
                             |         |      |         |      |         |      |                                              |
                             +-------+--      --+-------       -----------      ----+---------+------+---------+ +-------------+
                             | Blk 1 |          | Blk 2                             | Block 3 | ...  | Block n | |    Footer   | => Compression Layer
                             +-------+--      --+-------       -----------      ----+---------+------+---------+ +-------------+
                            /         \                                                             /           \
                           /           \                                                           /             \
                          /             \                                                         /               \
                         +-----------------------------------------------------------------------------------------+
                         |                                                                                         |             => Position layer
                         +-----------------------------------------------------------------------------------------+
                         +-------------+-------------+-------------+-------------+-----------+-------+-------------+
                         | File1 start | File1 data1 | File2 start | File1 data2 | File1 end |  ...  | Files index |             => Files information and content
                         +-------------+-------------+-------------+-------------+-----------+-------+-------------+Layers descriptionRaw LayerImplemented inRawLayer*(i.e.RawLayerWriter,RawLayerReaderandRawLayerFailSafeReader).This is the simplest layer. It is required to provide an API between layers and
final output worlds. It is also used to keep the position of data's start.Position LayerImplemented inPositionLayer*.Similar to theRawLayer, this is a very simple, utility, layer. It keeps
track of how many bytes have been written to the sub-layers.For instance, it is required by the file storage layer to keep track of the
position in the flow of files, for indexing purpose.Encryption LayerImplemented inEncryptionLayer*.This layer encrypts data using the symmetric authenticated encryption with associated data (AEAD) algorithmAES-GCM 256, and encrypts the symmetric key using an ECIES schema based onCurve25519.The ECIES schema is extended to support multiple public keys: a public key is generated and then used to performnDiffie-Hellman exchanges with thenusers public keys. The generated public key is also recorded in the header (to let the user replay the DH exchange). Once derived according to ECIES, we getnkeys. These keys are then used to encrypt a common keyk, and the resultingnciphertexts are stored in the layer header.
This keykwill later be used for the symmetric encryption of the archive.In addition to the key, a nonce (8 bytes) is also generated per archive. A fixed associated data is used.The generation usesOsRngfrom craterand, that usesgetrandom()from crategetrandom.getrandomprovides implementations for many systems, listedhere.
On Linux it uses thegetrandom()syscall and falls back on/dev/urandom.
On Windows it uses theRtlGenRandomAPI (available since Windows XP/Windows Server 2003).In order to be "better safe than sorry", aChaChaRngis seeded from the
bytes generated byOsRngin order to build a CSPRNG(Cryptographically Secure PseudoRandom Number Generator). ThisChaChaRngprovides the actual bytes used in keys and nonces generations.The layer data is then made of several encrypted blocks, each with a constant size except for the last one. Each block is encrypted with an IV including the base nonce and a counter. This construction is close to theSTREAMone, except for thelast_blockbit. The choice has been made not to use it, because:At the time of writing, the archive writer does not know that the current block is the last one. Therefore, it cannot use a specific IV. To circumvent it, a dummy footer block has to be added at the end, leading to additional complexity for last block detectionIn STREAM, thelast_blockbit is used to prevent undetected truncation. In MLA, it is already the role of theEndOfArchiveDatatag at the file layer levelThus, to seek-and-read at a given position, the layer decrypts the block containing this position, and verifies the tag before returning the decrypted data.The authors decided to use elliptic curve over RSA, because:No ready-for-production Rust-based libraries have been found at the date of writingA security-audited Rust library already exists for Curve25519Curve25519 is widely used andrespects several criteriaCommon arguments, such as the ones ofTrail of bitsAES-GCM is used because it is one of the most commonly used AEAD algorithms and using one avoids a whole class of attacks. In addition, it lets us rely on hardware acceleration (like AES-NI) to keep reasonable performance.External cryptographic libraries have been reviewed:RustCrypto AES-GCM, reviewed byNCC GroupDalek cryptography library, reviewed byQuarkslabCompression LayerImplemented inCompressionLayer*.This layer is based on the Brotli compression algorithm (RFC 7932).
Each 4MB of cleartext data is stored in a separately compressed chunk.This algorithm, used with awindowof size 1, is able to read each chunk and
stop when 4MB of cleartext has been obtained. It is then reset, and starts
decompressing the next chunk.To speed up the decompression, and to make the layer seekable, a footer is used. It
saves the compressed size. Knowing the decompressed size, a seek at a cleartext
position can be performed by seeking to the beginning of the correct compressed
block, then decompressing the first bytes until the desired position is reached.The footer is also used to allow for a widerwindow, enabling faster
decompression. Finally, it also records the size of the last block, to compute the
frontier between compressed data and the footer.The 4MB size is a trade-off between a better compression (higher value) and faster seeking (smaller value). It has been chosen based on benchmarking of representative data. Better compression can also be achieved by setting the compression quality parameter to a higher value (leading to a slower process).File storageFiles are saved as series of archive-file blocks. A first special type of block
indicates the start of a file, along with its filename and a file ID. A second special type of
block indicates the end of the current file.Blocks contain file data, prepended with the current block size and the corresponding file ID. Even if the
format handles streaming files, the size of a file chunk must be known before
writing it. The file ID enables blocks from different files to be interleaved.The file-ending block marks the end of data for a given file, and includes its
full content SHA256. Thus, the integrity of files can be checked, even on repair
operations.The layer footer contains for each file its size, its ending block offset and an index of its block locations. Block location index enables direct access. The ending block offset enables fast hash retrieval and the file size eases the conversion to formats needing the size of the file before the data, such as Tar.If this footer is unavailable, the archive is read from the beginning to recover
file information.API GuidelinesThe archive format provides, for each file:a filename, which is an unicode Stringdata, which are a stream of bytesA few metadata are also computed, such as:the file sizethe SHA256 hash of the contentNo additional metadata (permissions, ownership, etc.) are present, and would probably not be added unless very strong arguments are given. The goal is to keep the file format simple enough, and to leave the complexity to the code using it. Things such as permissions, ownership, etc. are hard to guarantee over several OSes and filesystems; and lead to higher complexity, for example in tar. For the same reasons,/or\do not have any significance in filename; it is up to the user to choose how to handle them (are there namespaces? directories in Windows style? etc.).If one still wants to have associated metadata for its own use case, the recommended way is to embed an additional file in the archive containing the needed metadata.Additionally, the file format is expected to change slightly in the future, to keep an easier backward compatibility, or, at least, version conversion, and simple support.The API provided by the library is then very simple:Add a fileStart / Add file chunk / EndList files in the archive (unordered)Get a fileGet a file hashAs the need for a less general API might appear, helpers are available inmla::helpers, such as:StreamWriter: Provides aWriteinterface on aArchiveWriterfile (could be used when even file chunk sizes are not known, likely withio::copy)linear_extract: Extract an Archive linearly. Faster way to extract a whole archive, by reducing the amount of costlyseekoperationsIs a new format really required?As existing archive formats are numerous, probably not.But to the best of the authors' knowledge, none of them support the aforementioned
features (but, of course, are better suitable for others purposes).For instance (from the understanding of the author):tarformat needs to know the size of files before adding them, and is not
seekablezipformat could lose information about files if the footer is removed7zipformat requires to rebuild the entire archive while adding files to it
(not streamable). It is also quite complex, and so harder to audit / trust
when unpacking unknown archivejournaldformat is not streamable. Also, one writter / multiple reader is
not needed here, thus releasing some constraintsjournaldformat haveany archive +age:agecould be used jointly with an archive format to provide encryption, but would likely lack integration with the inner archive formatBackup formats are generally written to avoid things such as duplication,
hence their need to keep bigger structures in memory, or their not being
streamableTweaking these formats would likely have resulted in similar properties. The
choice has been made to keep a better control over what the format is capable
of, and to (try to) KISS.TestingThe repository contains:unit tests (formlaandcurve25519-parser), testing separately expected behaviorsintegration tests (formlar), testing common scenarios, such ascreate->list->to-tar, orcreate->truncate->repairbenchmarking scenarios (formla)AFLscenario (formla)Acommitted archive in format v1, to ensure backward readability over timePerformanceOne can evaluate the performance through embedded benchmark, based onCriterion.Several scenarios are already embedded, such as:File addition, with different size and layer configurationsFile addition, varying the compression qualityFile reading, with different size and layer configurationsRandom file read, with different size and layer configurationsLinear archive extraction, with different size and layer configurationsOn an "Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz":$cdmla/
$cargobench
...
multiple_layers_multiple_block_size/LayersENCRYPT|COMPRESS|DEFAULT/1048576time:[28.091ms28.259ms28.434ms]thrpt:[35.170MiB/s35.388MiB/s35.598MiB/s]...
chunk_size_decompress_mutilfiles_random/LayersENCRYPT|COMPRESS|DEFAULT/4194304time:[126.46ms129.54ms133.42ms]thrpt:[29.980MiB/s30.878MiB/s31.630MiB/s]...
linear_vs_normal_extract/LINEAR/LayersDEBUG|EMPTY/2097152time:[145.19us150.13us153.69us]thrpt:[12.708GiB/s13.010GiB/s13.453GiB/s]...Criterion.rs documentation explains how to get back HTML reports, compare results, etc.The AES-NI extension is enabled in the compilation toolchain for the supported architectures, leading to massive performance gain for the encryption layer, especially in reading operations. Because the crateaesnistatically enables it, it might lead to errors if the user's architecture does not support it. It could be disabled at the compilation time, or by commenting the associated section in.cargo/config.FuzzingA fuzzing scenario made withafl.rsis available inmla-fuzz-afl.
The scenario is capable of:Creating archives with interleaved files, and different layers enabledReading them to check their contentRepairing the archive without truncation, and verifying itAltering the archive raw data, and ensuring reading it does not panic (but only fail)Repairing the altered archive, and ensuring the recovery doesn't fail (only reports detected errors)To launch it:produce initial samples by uncommentingproduce_samples()inmla-fuzz-afl/src/main.rscdmla-fuzz-afl# ... uncomment `produces_samples()` ...mkdirinmkdirout
cargorunbuild and launch AFLcargoaflbuild
cargoaflrun-iin-oout../target/debug/mla-fuzz-aflIf you have found crashes, try to replay them with either:Peruvian rabbit mode of AFL:cargo afl run -i - -o out -C ../target/debug/mla-fuzz-aflDirect replay:../target/debug/mla-fuzz-afl < out/crashes/crash_idDebugging: uncomment the "Replay sample" part ofmla-fuzz-afl/src/main.rs, and adddbg!()when it's needed:warning: The stability is quite low, likely due to the process used for the scenario (deserialization from the data provided by AFL) and variability of inner algorithms, such as brotli. Crashes, if any, might not be reproducible or due to themla-fuzz-aflinner working, which is a bit complex (and therefore likely buggy). One can comment unrelevant parts inmla-fuzz-afl/src/main.rsto ensure a better experience.FAQIsMLAArchiveWriterSend?By default,MLAArchiveWriteris notSend. If the inner writable type is alsoSend, one can enable the featuresendformlainCargo.toml, such as:[dependencies]mla={version="...",default-features=false,features=["send"]}How to deterministically generate a key-pair?The option--seedofmlar keygencan be used to deterministically generate a key-pair. For instance, it can be used for reproductive testing or archiving a key in a safe.:warning: It is not recommended to use aseedunless one knows why she is doing it.
The security of the resulting private-key is dependent of the security of the seed. In particular:if an attacker known theseed, he knowns the private-keythe entropy of the resulting private-key is at most the one of theseedThe algorithm used for the generation is as follow:given aseed, encode it as an UTF8 sequence of bytesbytesprng_seed = SHA512(bytes)[0..32]secret = ChaCha-20rounds(prng_seed)secret, after being clamped as specified by the Curve-25519 reference, is used as the private keyHow to setup a "hierarchical key infrastructure"?mlarprovides a subcommandkeyderiveto deterministically derive sub-key from a given key along a derivation path (a bit likeBIP-32, except children public keys can't be derived from the parent one).For instance, if one wants to derive the following scheme:root_key
    ├──["App X"]── key_app_x
    │   └──["v1.2.3"]── key_app_x_v1.2.3
    └──["App Y"]── key_app_yOne can use the following commands:# Create the root key (--seed can be used if this key must be created deterministically, see above)mlarkeygenroot_key# Create App keysmlarkeyderiveroot_keykey_app_x--path"App X"mlarkeyderiveroot_keykey_app_y--path"App Y"# Create the v1.2.3 key of App Xmlarkeyderivekey_app_xkey_app_x_v1.2.3--path"v1.2.3"At this point, let's consider an outage happened and keys have been lost.One can recover all the keys from theroot_keyprivate key.
For instance, to recover thekey_app_v1.2.3:mlarkeyderiveroot_keyrecovered_key--path"App X"--path"v1.2.3"As such, if theApp Xowner only knowskey_app_x, he can recover all of its subkeys, includingkey_app_v1.2.3but excludingkey_app_y.:warning: This scheme does not provide any revocation mechanism. If a parent key is compromised, all of the key in its sub-tree must be considered compromised (ie. all past and futures key that can be obtained from it). The opposite is not true: a parent key remains safe if any of its children key is compromised.The algorithm used for the generation is as follow:Given aprivate key, extract it'ssecretas a 32-bytes value (the clamped private key of Curve 25519)For eachpathencoded as UTF8:Derive a seed from a HKDF-SHA512 function (RFC5869) with:HKDF-SHA512(salt="PATH DERIVATION" ASCII-encoded, ikm=secret extracted from the parent key, info=Derivation path)Use the first 32-bytes as a seed for a ChaCha-20 rounds PRNGThe first 32-bytes output of ChaCha, after being clamped as specified by the Curve-25519 reference, is used as the new private keyUse the last computed private key as the resulting key