Talk:Lempel–Ziv–Markov chain algorithm

message: Maybe information about XZ should also be added here. Maybe there should also be an XZ=>LZMA redirect. Mainly, because a lot of GNU/Linux distributions(Slackware, Gentoo, Zenwalk) and NetBSDs pkgsrc switch from tgz to txz these days. A lot of people will ask there self, what it is all about and since XZ is more or less the successor of LZMA I suggest adding a paragraph about it and do a redirect. Maybe also redirecting txz would be a good idea. I would have done it, but I have no account. --85.127.107.86 (talk) 13:46, 11 May 2009 (UTC)
thread_title:  XZ? 

message: I can't find anywhere in this article, or in the external links, any details on this algorithm (not the implementation or "7-zip"). Who invented it? When? What is the difference between it and the traditional LZ77, LZ78 and LZW algorithms, or is it a completely new algorithm? What's the acronym "LZMA"? Thanks.
Nyh 08:33, 15 Dec 2004 (UTC)
replies: text: *Igor Pavlov could have discovered and implemented it as he claims.  There has been no counter-argument or dispute on his ownership claims so far and the claims are plausable.  The theory behind the algorithm is quite simple.  Start with the older LZ77 compression.  Rather than using a complete dictionary of all tokens it uses Markov chains to generate a very small dictionary of available tokens visible in nearby data.  The tokens themselves are range encoded values which further improves the general dictionary size.  He also ignored LZ77's arbitrary limitation on data sizes when generating the archive, which allows the huge dictionary to tend toward a more optimal set of tokens.  All of these technologies are well-studied and understood, so there is no need for continued research on the format.  Based on his many years of comments in the SourceForge forums it started as his own little personal project to see what happens when the technologies were combined.  He noticed the results were excellent and then released to a few people for review.  It quickly grew in popularity even though there were no formal papers describing the algorithm.   Bwagstaff (talk) 23:22, 25 August 2009 (UTC)

text: Range encoding is incrementally more efficient than Huffman coding, but not nearly enough so to account for the dramatic performance increase over Deflate.  If couple LZMA with a Huffman coder, and LZ77 with a range/arithmetic coder, the gap between them would close somewhat, but LZMA would still generally beat the pants off LZ77.  The real "secret sauce" is the huge dictionary size, and the lookup algorithms that let the encoder search a dictionary of that size while staying usefully fast.  --Piet Delport 16:23, 23 December 2005 (UTC)
replies: text: Granted, Deflate's 32K limit doesn't help.  So the "secret sauce" is the combination of the bigger window size plus the encoding that allows the bigger window size to be represented economically.  See, I learned something!  Posting wild speculation until someone corrects you is the true spirit of Wikipedia. --24.203.227.176 18:23, 2 January 2006 (UTC)
replies: text: If by "encoding" you mean "data structure", then yes, that's it. :)  --Piet Delport 23:48, 2 January 2006 (UTC)
thread_title:  Algorithm Details 

message: On what benchmark are based the claim about a compression speed of 1MB/sec ? I've done some quick and dirty bench on a P4 2.8 Ghz and don't get something near the 1MB/sec with a 2Ghz processor as stated in the article. The test is done on fr page current (20050903_pages_current.xml.gz)
$ time lzma e pages_current.xml pages_current.xml.lzma
real    42m46.403s
user    41m47.801s
sys     0m2.220s
-rw-r--r--  1 ~~~ users 716524090 2005-09-23 23:39 pages_current.xml
-rw-r--r--  1 ~~~ users 141582811 2005-09-24 00:24 pages_current.xml.lzma
compression 280K/sec
$ time lzma d pages_current.xml.lzma pages_current.xml.1
real    0m42.048s
user    0m34.918s
sys     0m2.052s
decompression 17MB/sec
$ time bzip2 pages_current.xml
real    5m44.382s
user    5m31.425s
sys     0m1.560s
-rw-r--r--  1 ~~~ users 716524090 2005-09-24 00:27 pages_current.xml
-rw-r--r--  1 ~~~ users 164451016 2005-09-23 23:39 pages_current.xml.bz2
compression ~ 2 MB/sec
$ time bunzip2 pages_current.xml.bz2
real    1m50.196s
user    1m46.415s
sys     0m2.800s
decompression 6.5 MB/sec
$ time gzip pages_current.xml
real    1m13.921s
user    1m12.205s
sys     0m1.240s
-rw-r--r--  1 ~~~ users 716524090 2005-09-24 00:27 pages_current.xml
-rw-r--r--  1 ~~~ users 213544652 2005-09-23 23:39 pages_current.xml.gz
compression 9.8 MB/sec
$ time gunzip pages_current.xml.gz
real    0m14.064s
user    0m12.357s
sys     0m1.604s
decompression 51 MB/sec
Given the number above either the used data perform very differently from the benchmarked data or the compression speed is 1Mbits/sec rather than 1MBytes/sec. This test used the lzma sdk 4.27. Note also the memory usage seems pretty higher than gzip/bzip, using default option the compressor use 80MB of RAM. 80.15.144.16 23:58, 23 September 2005 (UTC)
replies: text: For compression, the rate should refer to input consumed, and for decompression, output produced.  In other words, it's always in reference to the uncompressed data.  (Otherwise, the rating would be largely determined by the compression ratio itself, which is not very useful.)  --Piet Delport 00:50, 17 April 2006 (UTC)
thread_title:  compression speed 

message: I did a quick comparision, default options for gzip,bzip2,lzma (from lzma utils] and rar:
  -rw-r--r--  1 root root 199M 2005-12-04 20:26 linux-2.6.11.tar
  -rw-r--r--  1 root root  36M 2005-12-04 20:28 linux-2.6.11.tar.bz2
  -rw-r--r--  1 root root  45M 2005-12-04 20:29 linux-2.6.11.tar.gz
  -rw-r--r--  1 root root  30M 2005-12-04 21:15 linux-2.6.11.tar.lzma
  -rw-r--r--  1 root root  34M 2005-12-04 21:24 linux-2.6.11.tar.rar
Quick decompression bench:
  # lzma d -so linux-2.6.11.tar.lzma |pv > /dev/null
  199MB 0:00:11 [17.2MB/s]
  # cat linux-2.6.11.tar.bz2| bunzip2 |pv > /dev/null
  199MB 0:00:20 [9.52MB/s]
  # cat linux-2.6.11.tar.gz| gunzip |pv > /dev/null
  199MB 0:00:03 [63.8MB/s]
:Previous section talk about compression speed, not decompression speed, any number on this ? phe 12:56, 4 December 2005 (UTC)
thread_title:  Comparision of compression ratio 

message: This page (and the page for 7z) claim variable dictionary size is up to 1GB. However, at 7zip page, claim is up to 4GB. Any reference for (it once being or practically being) 1GB? --Gojomo 19:53, 16 February 2007 (UTC)
replies: text: The current version of 7zip for Windows x64 (4.65) only supports a dictionary size of up to 1GB. From memory it's been like that always, or perhaps even lower (never paid much attention to such a high size, I admit, I don't have enough RAM to compressing something with such a large dictionary). Given that a dictionary of 4GB would need over 40GB of RAM for compression, I'm frankly not surprised that it's limited. I presume LZMA, and probably the 7z format is capable of supporting such a dictionary size, perhaps the current version is even capable of decompressing that large a dictionary if you have over 4GB of RAM and a file made with some other program. Nil Einne (talk) 18:25, 5 August 2009 (UTC)
thread_title:  Dictionary Size 

message: I added a tone cleanup tag to the section describing the algorithm, because it sounds a little unprofessional.  I don't think it should tell the reader to rely on forums for information, since forums aren't even an acceptable source of citations for Wikipedia.  Describing the lack of information, or the that most of the information is only available via the forums, might say the same thing without directly telling the reader to rely on them, but I think it'd be best to either cite the information from a refutable source or simply not include it until it does become available. Maybe even include speculation by a refutable source (so long as it's properly cited). --Sydius (talk) 00:17, 23 July 2008 (UTC)
replies: text: Forums are acceptable in some circumstances.  See WP:SOURCES for details.  I believe they meet the guidelines for acceptable sources.  There are no known scholarly articles or news sources that provide the information in secondary form.  The currently-best available references are those forum posts and other self-published comments by the subject's creator.  The author is relavent to the subject, in that he was the undisputed creator of the algorithm and therefore expert about it.  The sources are not self-serving or from third parties, as they are direct Q&A by the author to his technical audience.  There is no reason to dispute authenticity or accuracy, since the author's identity and expert status is clear.  Finally, the article is not based exclusively on the forum posts; the posts are simply the best known source of information.  Unless and until there are better sources available, his forum posts seem to be the best available source and allowed according to WP guidelines.   Bwagstaff (talk) 07:40, 10 January 2010 (UTC)
thread_title:  Tone cleanup 

message: The latest edit relating RPM and LZMA points to a comment on proposed, rather than actual implementation.  Unless there's a better source, the edit should be reverted. Tedickey (talk) 11:28, 16 December 2008 (UTC)
replies: text: Fixed that for you :) 192.118.27.253 (talk) 09:47, 31 December 2008 (UTC)
thread_title:  Use of LZMA in RPM 

message: The reason I requested an expert's attention is because the article first quotes a potentially non notable but I presume expert, then goes to say he's mistaken (without any source). If the person quoted is wrong, there's little reason to quote him. Instead we should just mention how LZMA works, with sources. Nil Einne (talk) 18:14, 5 August 2009 (UTC)
thread_title:  Expert attention 

message: The discussion is referring to "originally" in the context of Common Public License, but that license is dated more recently than the beginning of the project.  Replacing that by "earlier" would work with the existing sources; it is unsourced as is. TEDickey (talk) 10:45, 5 August 2012 (UTC)
replies: text: Actually, he's right. The earliest archived version of the LZMA SDK is dual-licensed under the CPL and the LGPL. And that came out 4 years before the CPL was published. Myconix (talk) 00:53, 12 February 2014 (UTC)

text: The comment states 4.62; your suggested source is 4.23 (and pointing the reader to a download site is less than helpful).  The statement should be rewritten, given the information you are providing TEDickey (talk) 01:03, 12 February 2014 (UTC)

text: He was referring to the first public domain release, not the first release overall. Regardless, such an edit is trivial at best, and the statement has been clarified. If you believe there is a better citation to place there with the same information, feel free to replace it. Myconix (talk) 02:16, 12 February 2014 (UTC)
thread_title:  "originally" used without source 

message: Why is the LZMA2 container format described in an article about the LZMA algorithm? I feel that too many unrelated formats and encoding schemes are described in this page (LZMA2 encoder, Upper encoding layers). They should be moved to the 7-zip or xz pages. Ant diaz 14:05, 9 March 2013 (UTC)
thread_title:  LZMA2 format? 

message: Hello fellow Wikipedians,
I have just modified one external link on Lempel–Ziv–Markov chain algorithm. Please take a moment to review my edit. If you have any questions, or need the bot to ignore the links, or the page altogether, please visit this simple FaQ for additional information. I made the following changes:
*Added archive https://web.archive.org/web/20100430180036/http://nullprogram.com/blog/2010/04/16 to http://nullprogram.com/blog/2010/04/16/
When you have finished reviewing my changes, you may follow the instructions on the template below to fix any issues with the URLs.
Cheers.—InternetArchiveBot (Report bug) 13:39, 20 December 2017 (UTC)
thread_title:  External links modified 

message: Wouldn't the compression speed also be in Mbit/s rather than MB/s, User:Bender235? Also, is there a source for these claims? They seem questionable at least as frequency tells very little about execution time when you consider the impact of cache size, cache speed, execution pipeline, and branch prediction. Not to mention that being written in C++, a newer compiler might be able to improve those numbers noticeably. BernardoSulzbach (talk) 21:57, 28 August 2019 (UTC)
replies: text: Sorry, this was an suggested typo fix by AWB. It might be wrong. Please correct it if so. --bender235 (talk) 23:45, 28 August 2019 (UTC)
replies: text: I just removed the numbers that could not be verified and explained the reasoning behind it. User:Bender235, please make sure to not change MB to Mb (and other multiples of bytes to multiples of bits and vice-versa) without being certain about the values. In this case, for instance, the previous values were much "closer to reality" than the ones you introduced. BernardoSulzbach (talk) 01:07, 29 August 2019 (UTC)
thread_title:  Decompression speed change