Branch for sorting improvments, including (I hope) unicode.


git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3029 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Remove the Simple8Encoder since it is not a real thing.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3031 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Save the actual transliterated characters that are written to the output.

Save as char the output characters in the character set they were written in.
TODO AnyCharsetEncoder.

This idea may not work.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3033 25d90789-57f7-4ee0-8453-03a3dfeeeb22
When no code-page, actually use an ascii charset.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3044 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Program for generating collation input files.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3045 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Rewrite srt input file reader in terms of unicode.

Syntax slightly modified to be more standard.  If a character is represented by
a number, then that number is in unicode and not the target character set.

All srt.txt files updated to match. They should also be more complete, eg the cp1252
one did not have ž,Ž etc previously.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3046 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Add all the other common single byte code pages.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3047 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Modify Sort to allow multiple byte values in the future.

There is no real change with this commit, there is still just one page
suitable for a single byte sort.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3048 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Remove the constructed default sort.

It has little purpose any more, nothing will work well if it is used anyway.
Now just use 1252 if we do not have a known code-page.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3050 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Use the actual transliterated characters to sort on.

We now save the characters that are placed into the file in their codepage
representation. (but not in the 6-bit format).

Still only 8-bit codepages supported, so all the top bits of the characters
will be 0.

This is a transition. New and old ways of doing things co-exist and the usage
of the old way will be gradually removed, while keeping everything working.

There may be debug statements that will cause an exception if turned on.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3054 25d90789-57f7-4ee0-8453-03a3dfeeeb22
All empty labels must map to offset 0 in the LBL section.


git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3056 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Sorting tests, a couple of minor fixes.


git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3061 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Sorting tests, a couple of minor fixes.
Made all sort descriptions consistent.


git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3066 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Final fixes to sort by keys and by collation.

Now both give the same results.

Sorting of ignored characters is now consistent.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/sort@3067 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Merge from the sort branch.

This solves multiple problems searching for addresses when the default
ascii character set is used, or when many characters cannot be 
represented in the target character set and have to be
transliterated.

Complete sort tables for code-pages: 1250, 1251, 1252, 1253, 1254,
1255, 1256, 1257 and 1258.

A few corrections to the sorting algorithm so that sorting by
sort-keys always gives the same result as sorting using the Collator.

If anyone uses there own sort descriptions, then there is one change:
everything is in terms of unicode. The file itself should be in utf-8
as before, but all characters with numeric references should be in
unicode, rather than the target character set.

There are numerous internal changes that are the beginnings of
unicode support, but these changes are not visible, so unicode
maps will still not be searchable.



git-svn-id: http://svn.mkgmap.org.uk/mkgmap/trunk@3076 25d90789-57f7-4ee0-8453-03a3dfeeeb22