recruit-tech/document-validator — pull request #103 by jmoriya (2014-02-08 13:53:58+00:00)
Title: Katakana SpellCheck Validator #55

Validate the correctness of Katakana word spelling.

Japanese Katakana words have orthographical variations.
For example, "index" is written in Katakana by"インデクス" or "インデックス".
Unfortunately, we can not detect whether is correct or not because there is no specific rule.
So our strategy is to validate whether a Katakana word is very similar to the other Katakana word in previous sentences or not.
The similarity is defined as Levenshtein distance(Edit distance).
We heuristically detecit a pair as similar if the distance is smaller than the 30% of the length of the target string.
Moreover, if the length of Katakana word is small, mostly the pairs are detected as similar.
To avoid the noisy detection, we set a threshold of the length of Katakana word. And if the length of a Katakana word is smaller than the threshold, we do not detect the similarity.