twitter/twitter-text — pull request #7 by edengol (2014-12-17 19:36:14+00:00)
Title: Internationalize hashtags

These changes are to properly and systematically internationalize hashtag support to cover all unsupported major scripts like Greek, Armenian, Georgian, Amharic, Devanagari (Hindi, Nepali, Marathi), Gujarati, Punjabi, Oriya, Tamil, Telugu, Malayalam, Kannada, Sinhala, Tibetan, Burmese, Laotian, Khmer, and all unicode supported scripts in general. Also, support for other scripts' numbers has been added, respecting the existing rules that define a hashtag. To do this, I generalized the manual ranges of characters currently in place with unicode categories, checking all existing supported characters to ensure no regressions. Only javascript does not support these unicode regexes (\p{L}, \p{M}, and \p{Nd}) and so manual ranges for the entire unicode standard were used. Some special characters were added which don't fall into the above regex categories which are essential to writing words in these scripts. I have tightened up the rules of a hashtag so that a hashtag must contain at least one unicode letter or mark and that numbers and special characters cannot appear on their own.

As this is my first time working in this code base and in some of these languages, your reviews are appreciated. Actual deployment of this is on hold until we can work out all the ramifications with search, but I wanted to get the ball rolling. Thanks for your help and comments.

Please review: @jakl @jugyo @psychs @niw