Branch to implement unicode index and searching.


git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3271 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Add and update tests for the new features.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3272 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Add a multi command to the sort text file reader.

When this flag is not set, then the values are converted
to the single byte representation of the code page.

If the flag is set, then this is not done and the full value
is read and saved. This is not fully implemented with this commit.


git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3273 25d90789-57f7-4ee0-8453-03a3dfeeeb22
New code for CollationRules helper to produce unicode sort file.

Add a first attempt at a code-page 65001 text sort description.

Update icu4j to 53.1 (an optional dependency, not used at runtime for mkgmap).

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3274 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Convert to using char for the primary key.
Encapsulate all position fields of Page.
Junit version to 4.11

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3275 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Remove sort position parameters to functions.

All is now done by the getPos and writePos top level
functions, they correctly deal with the upper byte(s) of the character.


git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3276 25d90789-57f7-4ee0-8453-03a3dfeeeb22
When adding a character, ensure that we initialise a Page if required for it.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3277 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Convert all input arrays to char.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3278 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Keep multi byte character sets are char arrays.

So for sorts with the 'multi' flag set we take the char array directly
from the string to be sorted. For single byte code-pages we convert to
the single byte form and then create a char array with those values (so
that all the chars have the top byte equal to zero). May migrate everything
to use the unicode char values directly later, but this may create differences
with characters that are invalid for the code page.


git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3279 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Write out a full multi-byte SRT file.

New sections 7 & 8, and the record size of some sections is larger.


git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3281 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Set different flags for the multi-byte srt.

Now seems to work.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3283 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Skip characters that are in pages that are not present in srt.



git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3284 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Add comments and rename some variables.

git-svn-id: http://svn.mkgmap.org.uk/mkgmap/branches/unicode@3286 25d90789-57f7-4ee0-8453-03a3dfeeeb22
Merge the unicode branch.

Add a sort description for cp65001
Create and write the multibyte SRT file.

There should be no change when not using cp65001 and if any such
change is found it should be reported as a bug.



git-svn-id: http://svn.mkgmap.org.uk/mkgmap/trunk@3294 25d90789-57f7-4ee0-8453-03a3dfeeeb22