Talk:Burrows–Wheeler transform

message: The text says that EOF is ignored when sorting, but the example seems
to suggest that EOF is considered to come after all normal letters. AxelBoldt 14:14 Aug 26, 2002 (PDT)
The dot wasn't an EOF.  It was just a dot that was there to make the rotations stand out visually.  Hopefully it's clearer now.  --LC 08:06 Aug 27, 2002 (PDT)
Obviously sufficiently many repeats of the transform must eventually restore the original text, and obviously that provides another (usually inefficient?) way of inverting a transform (just store the inputs temporarily, and give back the last input before the text to be inverted turns up again). But one interesting thing is this: since the original is itself the result of a transform on some text, clearly the transform does not necessarily always produce something better suited for compression. It must be something that is statistically likely, based on properties that are common in many inputs. What are these, and what are the odds that a transform won't actually help and might even hurt? If we do get one of these cases, how can we tell (cheaply)? And so on. PML.
Assuming you're talking about the Bijective version (since your claim is false of the other versions), then the statistical property it takes advantage of is the tendency for letter sequences to be exactly repeated more often than merely similar letter sequences. Exact matches create runs of the same letter. Like, when you're alphabetizing rotations of a book and you get to the 'uick' section, it's a pretty good bet that the previous character (the output from the BWT for that letter) is 'q'; when you get to the 'the' section, there are going to be a lot of spaces as the output character, from all the instances of 'the', 'then', 'there', and 'their' and others, but some 'o' will be mixed in from 'other', etc.
If you already have a lot of repeats or are simply high-entropy, it's not going to help.
How can repeating the transform restore the original text?  The transform is a not a bijection between strings of length n.  It is a one-to-one function from strings of length n into the set of ordered pairs of a string of length n and an integer less than n.  Thus, different inputs can produce the same output string with a different index.  For example .BANANA. and ANA..BAN produce the same output string. -- Luke Somers
There is a simple way to avoid the use of an EOF symbol which is treated differently in the many verisons of BWT. Since any rotation of the original string gets you the same BWT string output. Which is the way the Burrows Wheeler Transform is usually first discussed in the literature. One problem with the Special symbol for EOF is that when dealing with a file on 256 character types one needs to add a special character in to do the BWT. Then one has to pick its value some make it the heaviest symbol some the lightest and in some binary versions they pick a weight half way in between. One method that I have used to get back to the traditional BWT is to rotate the string until it represents a single lyndon word. And then do the BWT as done in the start of this article and pass as an index the number of rotations needed to get the start of original string. One nice feature of this it that when the string is rotated to make a single lyndon word the string resulting from the classical BWT and the Bijective BWT are the SAME.  — Preceding unsigned comment added by 69.152.152.134 (talk) 15:23, 22 August 2012 (UTC)
thread_title:  EOF 

message: Nobody reading this page cares about Posix collation.  Why don't we pick a sample string with no such issues? --Doradus 17:45, 16 January 2006 (UTC)
replies: text: How do you get from "I don't care" to "nobody cares"?  People reading this page quite clearly do care about POSIX collation, or there wouldn't be a mention of it in the first place.  I certainly care, and I'm equally certainly reading this page.  The point that BWT is sensitive to collation issues is worth making, IMO. — Haeleth Talk 20:19, 22 March 2006 (UTC)
replies: text: BWT isn't sensitive to Posix; the sort will work in whatever way you implement the sort. Practically speaking, it's very unlikely anyone will be doing real BWT with Posix C string sorting functions; I found it to be quite a non-sequitur with little or no apparent relation to the subject at hand. The removed section follows: --Brion 03:31, 19 May 2006 (UTC)
thread_title:  Collation issues 

message: Could someone check the example, please? I was trying to verify something and my simple quick implementation of BWT outputs
 TEXYDST.E.IXIXIXXSSMPPS.B..E.S.EUSFXDIIOIIIT
not
 TEXYDST.E.XIIXIXXSMPPSS.B...S.EEUSFXDIOIIIIT
as the article writes.
(And note that I intend to at least replace the black dot with another character, or to remove it altogether. It is not a good style to unnecessarily convey meaning only with a color variation, mind e.g. blind people.)
--Mormegil 18:12, 18 June 2006 (UTC)
thread_title:  Mistake in example? 

message: The BWT is not just the transformed string. If using method above you need to include something
like the position of original data in example above where the SIX.MIXED row occurs. If we don't
do this then people we wrongly think the normal BWT is just a permutation. When in fact its two
outputs the permutation and an integer index.  —Preceding unsigned comment added by 75.54.98.144 (talk) 16:28, 2 March 2009 (UTC)
thread_title:  INCOMPLETE EXAMPLE 

message: Article says:
The Burrows-Wheeler transform (BWT, also called block-sorting compression)
Technically, BWT is not a compression. Should it be noted? It leads to wrong understanding of the subject —lim 16:54, 16 May 2007 (UTC)
thread_title:  compression? 

message: Putting aside the issue that this is a really horrible example of program code, there's a major inefficiency in it: In the function rotlexcmp() if 'li' is equal to 'ri', it will needlessly traverse the entire data and then just return 0. According to my own experiments this makes it almost 100 times slower than it could be (well, depending on the exact implementation of the sorting function, of course). Simply adding a "if(li==ri) return 0;" at the beginning solves this issue and makes the program a lot faster with large datasets.
Wikipedia is not about tweaking the last bit of Performance out of some Algorithm.
It only has to give the right oputput.  —Preceding unsigned comment added by 129.13.72.198 (talk) 08:35, 6 February 2009 (UTC)
thread_title:  Bad inefficiency in the example implementation 

message: We don’t need two “sample implementations” and should not have them here both. We are not a code repository. Maybe we should clean one of them to focus on the algorithm (e.g. remove those #includes and main() in the C version)? --Mormegil 17:54, 9 November 2007 (UTC)
replies: text: I agree with you, I am going to remove the C-implementation. I don't think it serves any purpose here. Sample implementations can be found around the web. The C-code also points to the Polish Wikipedia as the source. Wikipedia is not a reliable source. —ZeroOne (talk / @) 16:28, 14 December 2008 (UTC)
replies: text: I do not agree. Although the WP:NOT says Wikipedia is "not a code repository", this is only a suggestion, not a rule. I do not agree with this policy. I think Wikipedia should always have sample code whenever possible. It provides probably the best context to the article. I agree that we should primarily focus on the algorithm, but practical/actual code is always a help. It is yet another reason to visit Wikipedia. I am, and have always been, an inclusionist. PS: What about this entire category: :Category:Articles with example C code? Are you going to remove that as well? --Thorwald (talk) 00:02, 15 December 2008 (UTC)
replies: text: Well, I'm glad we got a discussion. I also think that sample code is good. In this case I removed the C code and left the Python code, because I found the Python code to be clearer. There's nothing wrong with having C code but I think a sample in one language is enough. If I have to choose between C and Python I remove the C example; if I had to choose between, say, C and assembly language, I'd probably remove the assembly example. Having an example in more than one language feels like artificially expanding the article. If we allow two example languages, why not three? If we allow three different example languages, why not four? Why not five, six or seven examples? —ZeroOne (talk / @) 13:40, 15 December 2008 (UTC)
replies: text: Okay. I agree. The Python code is sufficient (and clearer and cleaner). --Thorwald (talk) 01:20, 16 December 2008 (UTC)
thread_title:  Implementation 

message: Just for Fun.
BWT[str_] := Block[{list, n, data},
  list = StringSplit[str, ""];
  n = Length@list;
  data = Sort@Table[RotateRight[list, i], {i, 1, n}];
  StringJoin@dataAll, -1
]
inBWT[str_, endOfFile_: "@"] := Block[{list, n, K},
  list = StringSplit[str, ""];
  n = Length[list];
  K = Sort@Transpose@List@list;
  Do[K = Sort@Transpose@Prepend[Transpose@K, list] ;, {n - 1}];
  StringJoin@Select[K, Last@# == endOfFile &]1
  ]  —Preceding unsigned comment added by 189.83.138.252 (talk) 22:35, 24 October 2009 (UTC)
thread_title:  Mathematica Implementation 

message: Does anybody have independent confirmation of the Bijective BWT actually working? After having read the link to David Scott's method, I'm extremely sceptical that his technique works.
The main reason why I'm extremely sceptical is because he essentially appears to be using the 1st column of the BWT reversed. If that works, then there's no reason why you couldn't use a binary input (instead of ASCII characters) and end up with what would surely be the world's most compressible file, consisting merely of a count of 0's and 1's.
Tiggs (talk) 00:16, 30 August 2008 (UTC)
replies: text: Huh. I managed to completely overlook the Kufleitner paper, which seems to be a published source. Somebody ought to convert those external links to in-line citations ... –Henning Makholm (talk) 14:08, 6 December 2010 (UTC)
thread_title:  The Bijective BWT 

message: Would it be possible to explain why the bijective version is interesting?
Eg where is it used and why?
Saving the cost of one index does not seem to justify it.
The text seems to suggest it was created for more than its mathematical beauty.
128.16.7.220 (talk) 15:17, 7 June 2014 (UTC)
thread_title:  Purpose of bijective transformation 

message: I'm concerned that the only source for the bijective transformation is the paper by Gil and Scott that seems to be (in traditional terms) an "unpublished manuscript". That isn't quite up to Wikipedia standards. That isn't just useless policy-wanking in this case; the paper contains meaning-disturbing typos and mistakes that I don't think would have survived in a published paper. At least we shouldn't mislead readers into thinking this is as reliable a source as the ones we usually provide. Furthermore, Gil is an academic, but no reference to the paper appears on his web page, which could indicate that it was later retracted.
In particular: The paper presents a "string-sorting transform" with two variants, depending on how one compares strings of different length where one is a prefix of the other. It also presents an inverse transform, though one without any variants, and it is not quite stated which of the two main transforms it is an inverse of. According to my pencil-and-paper experiments, the inverse transform does seem to invert the "infinite-periodic" variant of the main transform. It does not work for the "usual lexicographical" variant, and indeed that variant is not a bijection! As a minimal example, the input strings "bab" and "abb" both transform to "bba".(*)
Unfortunately it is only the "usual lexicographical" transformation that the authors provide a linear-time algorithm for (though it is very sketchy, and I'm not convinced by the time analysis). For the infinite-periodic variant, the authors explicitly deny having a linear algorithm. They claim to have an O(nlogn) one, but do not describe it further.
While a bijective variant of the BWT is indeed an interesting and worthy topic, the one presented by the paper is not quite as ready for use as the paper's abstract suggests.
 (*)  Input:                 bab      abb
      Lyndon factorization:  b·ab     abb
      Cyclic rotations:      b,ab,ba  abb,bba,bab
      Sorted:                ab       abb
                             b        bab
                             ba       bba
      Output:                bba      bba
(The "infinte-periodic" variant sorts "b" after "ba" and therefore transforms "bab" to "bab").
–Henning Makholm (talk) 13:44, 6 December 2010 (UTC)
replies: text: Huh. I managed to completely overlook the Kufleitner paper, which seems to be a published source. Somebody ought to convert those external links to in-line citations ... –Henning Makholm (talk) 14:08, 6 December 2010 (UTC)
thread_title:  Source for bijective transformation 

message: Section "BWT in bioinformatics" and pages Compressed suffix array and FM-index should probably be merged into a single page. They are all about essentially the same data structure:
# Build the BWT of the text.
# Encode the BWT in a way that supports rank(c, i) (the number of occurrences of character c in BWT[1..i]) and select(c, i) (the position of the i th occurrence of character c in the BWT) efficiently.
# Use the BWT to simulate the suffix array.
The differences between various compressed suffix arrays and FM-indexes are mostly in the encodings they use. The following survey articles are probably the best resources about compressed self-indexes so far:
* G. Navarro, V. Mäkinen: Compressed Full-Text Indexes. ACM Computing Surveys, 2007.
* P. Ferragina, R. González, G. Navarro, R. Venturini: Compressed Text Indexes: From Theory to Practice. ACM Journal of Experimental Algorithms, 2009.
Jouni Sirén (talk) 23:48, 18 December 2010 (UTC)
thread_title:  BWT-based self-indexes 

message: The article is very nicely written and clear even for a beginner like me. But I didn't get one point:
Would it be possible to explain, why step 3 in inverse transformation is possible? Sort and concatenate with BW'ed string in step 2 is clarified in the text, but why one can sort again and concatenate the BW'ed string again? Though it works.
cheers,
Oliver Drechsel (talk) 08:39, 20 January 2011 (UTC)
thread_title:  Example - Inverse Transformation 

message: the added character @ should be lexicographically before any other character, so:
ANANA@^B
ANA@^BAN
A@^BANAN
BANANA@^
NANA@^BA
NA@^BANA
^BANANA@
@^BANANA
should be
@^BANANA
ANANA@^B
ANA@^BAN
A@^BANAN
BANANA@^
NANA@^BA
NA@^BANA
^BANANA@
and the resulting string would be ABNN^AA@  — Preceding unsigned comment added by 84.223.127.232 (talk) 19:33, 12 April 2012 (UTC)
thread_title:  WRONG transformation 

message: This article is beautiful. Thank you so much to all for the work in making this algorithm understandable. I mentally cried tears of joy because I was expecting the worst. I've never seen a Wikipedia article so carefully curated to its intended audience.
--Dominic (talk) 12:58, 23 December 2016 (UTC)
thread_title:  Beautiful 

message: The algorithm claims to be linear in time, but clearly it cannot be, since for a string of length N it requires the construction of an N×N matrix, or since it is symmetric (over negation), an upper or lower triangle N×N matrix, which require O(N^2) to manipulate.
That glaring flaw aside, it is actually quite a a nice construction algorithm, and elucidates some nice intuitions about the problem of BWT construction, but claiming it is efficient is simply wrong from an algorithmic perspective. Drtomconway (talk) 22:33, 22 December 2019 (UTC)
replies: text: Where does it claim to be linear? BernardoSulzbach (talk) 16:11, 23 December 2019 (UTC)
thread_title:  League Table Sort Algorithm 

message: It may be "by convention", but the Example claims that rotation is done 8 times. It. Is. Not. It is done 7 times for a total of 8 strings if you include the original un-rotated string. That is, N = 7 for a string of 8 characters. It should also, imho, be mentioned that sorting of 'special characters' (which in this case would include both the caret (^) and the EOS (end of string? end of line?) character (|)) depends on where they are arbitrarily positioned in the sort order (it isn't necessary that both occur before the 'a' or after the 'z' of "lexical order".98.21.213.235 (talk) 23:31, 10 February 2021 (UTC)
thread_title:  Example contains an error. 

message: In is common to use ^ and $ to start and end a string, respectively -- see Regular expression#POSIX basic and extended.  Should we use $ instead of  in this article?  —Quantling (talk | contribs) 03:26, 14 November 2023 (UTC)
replies: text: I made the change boldly.  If this instead needs discussion, let's do it. —Quantling (talk | contribs) 22:01, 14 November 2023 (UTC)
thread_title:  Dollar sign instead of pipe to terminate strings 

message: The original Burrows–Wheeler paper has neither a string-starting character (like ^) nor a string-ending character (like @ or | or $), and does not invoke them in any way.  Can I remove them from this Wikipedia article? —Quantling (talk | contribs) 00:33, 15 November 2023 (UTC)
replies: text: In the paper, “the index I of the original string S in the sorted list of rotations” is an additional output of the compression scheme that is needed as an input for decompression. The WP article instead uses the fact that $ being in the last column will identify M[I]. But I don't see why the start-of-text character ^ is used. stw (talk) 19:57, 23 January 2024 (UTC)
replies: text: I’m particularly confused as to why the start of string character is used in the bijective example NixCode (talk) 02:57, 31 January 2024 (UTC)
thread_title:  String-starting and string-ending characters