LZMA

LZMA（Lempel–Ziv–Markov chain algorithm）是2001年以来得到发展的一个数据压缩算法，它用于7-Zip归档工具中的7z格式和 Unix-like 下的 xz 格式。它使用类似于LZ77的机制，在一般的情况下压缩率比bzip2为高，用于压缩的字典档案大小可达4GB。
C++语言写成的LZMA开放源码压缩库使用了区间编码支持的LZ77改进压缩算法以及特殊的用于二进制的预处理程序。LZMA 对数据流、重复序列大小以及重续序列位置单独进行了压缩。LZMA支持几种散列链变体、二叉树以及基数树作为它的字典查找算法基础。
特性.
BCJ / BCJ2二进制文件压缩.
BCJ / BCJ2压缩工具所附带的LZMA SDK包括：在X86、ARM、PowerPC、IA-64以及ARM Thumb处理器上在压缩之前跳转目标进行归一化处理。对于x86平台来说，这是一个近跳转、近调用以及近条件跳转需要从“向后跳1665字节”这样的机器语言归一化到“跳转到5554”这样的格式，但是短跳转及短条件跳转不需要进行这样的处理。
BCJ与BCJ2之间的区别在于前者只将近跳转及近调用目标地址转换到归一化的形式，而BCJ2只将x86平台下的近跳转、近调用及条件近跳转目标分别进行压缩。
实现和可移植性.
一些Windows作业系统专有的特性深深嵌入在原始程序中，使得最初很难生成一个与Unix等系统兼容的版本。然而，LZMA 由于其开放源码特性，仍然最终获得了各种平台的实现：
7-Zip/p7zip 参考实现.
在GNU通用公共许可证下发布的 7-zip 参考版本有以下几个特点：
这个特点使得这个这个算法的解压过程非常适合于嵌入式系统应用的场合。p7zip 为 7-zip 的 POSIX 系统移植。
xz 和 LZMA Unix Port.
LZMA Unix Port 是一个只移植了 7-zip 中 LZMA 压缩代码的版本，内含命令行参数类似于 gzip 的基于数据流的压缩工具。它不是一个归档工具，而只是一个普通的压缩工具，并且由于它在没有数据头中没有未压缩文件大小的UInt64变量，所以它与7-zip生成的LZMA数据流中不同。7-zip使用一种更加灵活的归档格式7z，因此不能由此工具解压。
后来类似的 xz 替代了 LZMA Unix Port，提供了更好的压缩功能，并最终以其优异的性能和压缩比成为了不少开源软件（例如 Linux 内核源码、Debian deb 和 Fedora rpm）的压缩方式之一，甚至是默认压缩方式。xz 命令行程序曾有过一个名为 pxz 的分支，提供多线程压缩功能，后来 xz 在 5.2 时本身就直接提供多线程了。
lzip.
Lzip 是另一个 Unix-like 系统下的 LZMA 压缩格式，其主要目的之一就是和 xz 竞争。与 xz 相比，它的最大亮点在于提供更简单的文件格式和因此得来的更方便的数据恢复。Lzip 的格式如此简单以至于其文档中就存在一个解压器实现，于是未来的数据考古学家即使在量子计算机使得 LZMA 无用多时之后只靠文档也能成功解压文件。
应用.
使用或者支持LZMA的软件有：
压缩格式表示.
LZMA的压缩输出流是一个比特流，采用自适应二进制行程编码器（adaptive binary range coder）。比特流划分为包（packet），每个包或者表示一个字节的受压缩数据，或者如同LZ77的压缩输出序列那样的长度与距离的对（pair）。每个包得每个部分作为独立的上下文（context），从而对每个比特的概率预测仅相关于前一个包的同类型比特值。
有7类包：
LONGREP[*] 表示LONGREP[0-3]四种包, *REP指称LONGREP 与SHORTREP, *MATCH指称MATCH或*REP.
LONGREP包删除了对距离的直接表示，而是使用包序列最近四个距离。
包的长度部分表示如下：
如同LZ77, 长度不一定要小于距离。
距离在逻辑上是32比特，距离0表示最近增加到词典的那个字节。
距离的编码以6比特"distance slot"开始，由此可知后面跟着多少比特来补全。这是可变长编码。
距离解码后为比特流，从最显著位到最不显著位。distance slots 0−3直接编码了0−3.
解压缩算法描述.
依据嵌入到Linux kernel的 XZ解码算法源文件。
32 − 1, and "code" to the 32-bit value starting at the second byte in the stream interpreted as big-endian; the first byte in the stream is completely ignored.
Normalization proceeds in this way:
Context-based range decoding of a bit using the "prob" probability variable proceeds in this way:
Fixed-probability range decoding of a bit proceeds in this way:
The Linux kernel implementation of fixed-probability decoding in rc_direct, for performance reasons, doesn't include a conditional branch, but instead subtracts "range" from "code" unconditionally, and uses the resulting sign bit to both decide the bit to return, and to generate a mask that is combined with "code" and added to "range".
Note that:
Range coding of integers.
The range decoder also provides the bit-tree, reverse bit-tree and fixed probability integer decoding facilities, which are used to decode integers, and generalize the single-bit decoding described above.
To decode unsigned integers less than "limit", an array of ("limit" − 1) 11-bit probability variables is provided, which are conceptually arranged as the internal nodes of a complete binary tree with "limit" leaves.
Non-reverse bit-tree decoding works by keeping a pointer to the tree of variables, which starts at the root. As long as the pointer doesn't point to a leaf, a bit is decoded using the variable indicated by the pointer, and the pointer is moved to either the left or right children depending on whether the bit is 0 or 1; when the pointer points to a leaf, the number associated with the leaf is returned.
Non-reverse bit-tree decoding thus happens from most significant to least significant bit, stopping when only one value in the valid range is possible (this conceptually allows to have range sizes that are not powers of two, even though LZMA doesn't make use of this).
Reverse bit-tree decoding instead decodes from least significant bit to most significant bits, and thus only supports ranges that are powers of two, and always decodes the same number of bits. It is equivalent to performing non-reverse bittree decoding with a power of two "limit", and reversing the last log2("limit") bits of the result.
Note that in the rc_bittree function in the Linux kernel, integers are actually returned in the ["limit", 2 * "limit") range (with "limit" added to the conceptual value), and the variable at index 0 in the array is unused, while the one at index 1 is the root, and the left and right children indices are computed as 2"i" and 2"i" + 1. The rc_bittree_reverse function instead adds integers in the [0, "limit") range to a caller-provided variable, where "limit" is implicitly represented by its logarithm, and has its own independent implementation for efficiency reasons.
Fixed probability integer decoding simply performs fixed probability bit decoding repeatedly, reading bits from the most to the least significant.
LZMA configuration.
The LZMA decoder is configured by an "lclppb" "properties" byte and a dictionary size.
The value of the "lclppb" byte is "lc" + "lp" * 9 + "pb" * 9 * 5, where:
In non-LZMA2 streams, "lc" must not be greater than 8, and "lp" and "pb" must not be greater than 4.
In LZMA2 streams, ("lc" + "lp") and "pb" must not be greater than 4.
In the 7-zip LZMA file format, configuration is performed by a header containing the "properties" byte followed by the 32-bit little-endian dictionary size in bytes. In LZMA2, the properties byte can optionally be changed at the start of LZMA2 LZMA packets, while the dictionary size is specified in the LZMA2 header as later described.
LZMA coding contexts.
The LZMA packet format has already been described, and this section specifies how LZMA statistically models the LZ-encoded streams, or in other words which probability variables are passed to the range decoder to decode each bit.
Those probability variables are implemented as multi-dimensional arrays; before introducing them, a few values that are used as indices in these multidimensional arrays are defined.
The "state" value is conceptually based on which of the patterns in the following table match the latest 2-4 packet types seen, and is implemented as a state machine state updated according to the transition table listed in the table every time a packet is output.
The initial state is 0, and thus packets before the beginning are assumed to be LIT packets.
The "pos_state" and "literal_pos_state" values consist of respectively the "pb" and "lp" (up to 4, from the LZMA header or LZMA2 properties packet) least significant bits of the dictionary position (the number of bytes coded since the last dictionary reset modulo the dictionary size). Note that the dictionary size is normally the multiple of a large power of 2, so these values are equivalently described as the least significant bits of the number of uncompressed bytes seen since the last dictionary reset.
The "prev_byte_lc_msbs" value is set to the "lc" (up to 4, from the LZMA header or LZMA2 properties packet) most significant bits of the previous uncompressed byte.
The "is_REP" value denotes whether a packet that includes a length is a LONGREP rather than a MATCH.
The "match_byte" value is the byte that would have been decoded if a SHORTREP packet had been used (in other words, the byte found at the dictionary at the last used distance); it is only used just after a *MATCH packet.
"literal_bit_mode" is an array of 8 values in the 0-2 range, one for each bit position in a byte, which are 1 or 2 if the previous packet was a *MATCH and it is either the most significant bit position or all the more significant bits in the literal to encode/decode are equal to the bits in the corresponding positions in "match_byte", while otherwise it is 0; the choice between the 1 or 2 values depends on the value of the bit at the same position in "match_byte".
The literal/Literal set of variables can be seen as a "pseudo-bit-tree" similar to a bit-tree but with 3 variables instead of 1 in every node, chosen depending on the "literal_bit_mode" value at the bit position of the next bit to decode after the bit-tree context denoted by the node.
The claim, found in some sources, that literals after a *MATCH are coded as the XOR of the byte value with "match_byte" is incorrect; they are instead coded simply as their byte value, but using the pseudo-bit-tree just described and the additional context listed in the table below.
The probability variable groups used in LZMA are those:
LZMA2 format.
The LZMA2 container supports multiple runs of compressed LZMA data and uncompressed data. Each LZMA compressed run can have a different LZMA configuration and dictionary. This improves the compression of partially or completely incompressible files and allows multithreaded compression and multithreaded decompression by breaking the file into runs that can be compressed or decompressed independently in parallel.
The LZMA2 header consists of a byte indicating the dictionary size:
LZMA2 data consists of packets starting with a control byte, with the following values:
Bits 5-6 for LZMA chunks can be:
LZMA state resets cause a reset of all LZMA state except the dictionary, and specifically:
Uncompressed chunks consist of:
LZMA chunks consist of: