joto/taginfo — pull request #99 by kcwu (2014-12-07 06:34:34+00:00)
Title: using icu tokenizer in sqlite3 full text search

OpenStreetMap is an international project, so icu tokenizer is more appropriate than the default "simple" tokenizer.

icu tokenizer is very helpful for Chinese text search. For example, we can find "中山高速公路"(Jhongshan Freeway) in taginfo but not "高速公路" (freeway). This is because the "simple" tokenizer doesn't understand word boundaries of Chinese and doesn't know "高速公路" is a word as well. With icu tokenizer, we can search Chinese tags using substring. I guess it is helpful for other languages, too.

I have tested icu tokenizer with Taiwan extract and it works well. I don't have idea about the performance impact on large scale database, the main taginfo site. You may need to test it before apply.

p.s. you may need to recompile sqlite3 with "icu" support if it is not enabled yet.