domnikl/highscore — pull request #17 by pdg (2013-01-31 22:22:39+00:00)
Title: Fix tokenization for words which contain non-ascii characters

Hi Dominik,

when processing words with non-ascii characters, highscore does not split up the content correctly for me.

~~~ ruby
content = "Herzliche Grüße"
Highscore::Content.new(content).keywords.to_a # => [Herzliche]
RUBY_DESCRIPTION # => "ruby 1.9.3p327 (2012-11-10 revision 37606) [x86_64-linux]"
~~~

According to the current docs `\w+` only matches ascii characters `[a-zA-Z0-9_]`: http://www.ruby-doc.org/core-1.9.3/Regexp.html#label-Metacharacters+and+Escapes

This commit replaces `/\w+/`, which is currently used as default `word_pattern`, with `/\p{Word}+/u` and includes a corresponding test, which checks that splitting of words with non-ascii characters works. This gives me positve results for...

* ruby 1.9.2-p290
* ruby 1.9.3-debian (from Debian package ruby1.9.3 1.9.3.194-5)
* ruby 1.9.3-p194
* ruby 1.9.3-p327
* jruby 1.6.7

Unfortunately ruby 1.8 seems not to support `\p{Word}` (tested with "ruby 1.8.7-debian" from Debian package ruby1.8 1.8.7.358-6).