heiglandreas/Org_Heigl_Hyphenator — pull request #14 by damienalexandre (2013-05-16 15:00:36+00:00)
Title: Infinite loop when using non breaking spaces

Something went strange when running a string with non "classic" whitespace in the Hyphenator.

$this->assertEquals('Ceci est à rem-pla-cer par une fâble'."\xE2\x80\xAF".':p',$h->hyphenate('Ceci est à remplacer par une fâble'."\xE2\x80\xAF".':p'));

The `\xE2\x80\xAF` char is a NO_BREAK_THIN_SPACE, commonly used in French. The more classic NO_BREAK_SPACE is also making the tokenizer going crazy.

The lib is actually creating an infinite quantity of token, I have no idea why but here is my proprosal for fixing this.

I'm just adding two new spaces to the WhitespaceTokenizer, removing `trim` test and tada, all the tests are green.