twitter/twitter-cldr-rb — pull request #108 by jrochkind (2013-10-08 15:28:50+00:00)
Title: Add maximum_level option to SortKeyBuilder

The Unicode Collation Algorithm technical report mentions that implementations may optionally support a 'maximum level' option for sort key generation.

I have need of this option. This pull request adds it.

It's not neccesarily quite ready for merging, I'm not sure the right way to expose this option to the Collator, and could use feedback. There's also a backwards incompatibilty in SortKeyBuilder#new and SortKeyBuilder#build, that I'm not sure if you'll mind. In general, I am quite happy to take directions on changes required to this pull request.

## The maximum level option:

UTS#10 section 4.3 says:

> An implementation may allow the maximum level to be set to a smaller level than the available levels in the collation element array. For example, if the maximum level is set to 2, then level 3 and higher weights are not appended to the sort key. Thus any differences at levels 3 and higher will be ignored, leveling any such differences in string comparison.

I have need of this feature, for a searching/matching application. UTS#10 hints that it may come in handy there:

> Strength. This refers to the number of levels that are to be considered in comparison, and is another important feature. Most of the time a three-level strength is needed for comparison of strings. In some cases, a larger number of levels will be needed, while in others—especially in searching—fewer levels will be desired.

Yep, that's pretty much what's going on with me. Supporting the optional `maximum level` option is pretty much what you need to implement that concept of variable 'strength'.

So this PR adds it to SortKeyBuilder. Pretty straightforward.

## Backwards compat?

Previously, `SortKeyBuilder.new(collation_elements, case_first)`. Needing to add another option, it seemed to make sense to change the second arg to an options hash. `SortKeyBuilder.new(collation_elements, :case_first => case_first, :maximum_level => max_level)`. And same with `SortKeyBuilder#build`.

This is technically not backwards compat, but I couldn't see any way to keep the API non-sucky and also be backwards compat; and I doubt using SortKeyBuilder directly is particularly popular.

But if you have another preference for how the API to choose 'maximum level' should work, just let me know and I'll make it so.

## Add to Collator api?

This PR doesn't yet expose the maximum level to the Collator api, but it probably needs to, to actually be useful. Appreciate feedback on how to do this.

I guess it should simply be an optional options second param to `Collator#get_sort_key`, eg:

collator.get_sort_key(string_or_code_points, :maximum_level => 2)

But then that got me thinking, perhaps it ought to be possible to pass in case_first here too, *overriding* what you may have gotten from the locale tailoring. UTS#10 implies that you might want to override locale tailorings here:

> Some dictionaries and authors collate uppercase before lowercase while others use the reverse, so that preference needs to be customizable. Sometimes the case ordering is mandated by the government, as in Denmark. Often it is simply a customization or user preference.

That implies it is allowable and useful to let the case_first setting be chosen on a case-by-case (doh) basis, overriding locale tailoring, right?

So maybe add that, while we're at it? The confusing thing is that `:case_first => nil` means something, so I guess passing that explicitly to `get_sort_key` might override a non-nil locale tailoring?

Or we could not allow case_first overriding yet, but just add the `maximum_level` option to `get_sort_key`?

Appreciate your thoughts/desires on this; but maybe I'll just go ahead and add what makes sense to me, if I don't hear from you soon.

## Thanks!

Thanks for this super awesome gem, and sharing it as open source! Thanks for any attention you can spare for this pull request! Appreciate it!