Project: talenddataquality
Issue TDQ-10690: Create new Pattern frequency indicators for Japanese characters
h3. high level description of the feature A new "East Asia Pattern frequency" indicator will be added to the system indicators in the pattern finder folder. Actually, two new indicators must be created: - the regular top n frequency indicator - and the low frequency indicator This indicator gives the same results as the Pattern Frequency indicator when the characters are latin characters. When values contain Japanese character, there is distinction between Hiragana, Katagana and Kanji characters. These characters are represented by H,K,C for uppercase characters, respectively and h,k,c for lowercase characters. This indicator is primarily a java indicator. As a system indicator, it will appear in the DQ repository view and in the indicator selection dialog. h3. technical information We currently use a pattern 'language' to describe the alpha chars with an 'a' and numerics with a '9'. We need to extend this language for: ||Character Type||Unicode Range||Replacement|| |Hiragana|\u3040-\u309F|use 'H' for regular chars, and 'h' for small chars| |Katakana|\u30A0-\u30FF|use 'K' for regular chars, and 'k' for small chars| |Katakana Phonetic Extensions|\u31F0-\u31FF|all of them are small chars, use 'k'| |Halfwidth Katakana|\uFF66-\uFF9F|use 'K' for regular chars, and 'k' for small chars| |Fullwidth Latin Numbers|\uFF10-\uFF19|use '9'| |Fullwidth Latin Uppercased Letters|\uFF21-\uFF3A|use 'A'| |Fullwidth Latin Lowercased Letters|\uFF41-\uFF5A|use 'a'| |Kanji|\u4E00-\u9FFF|use the letter 'C' for chinese, since 'k' is already taken| |Hangul|\uAC00-\uD7AF|use the letter 'G' for hangul, since 'H' is already taken| In case there are regular and small chars, one possible implementation is to use the range to determine the type firstly, and then use a static string which lists all small chars to see if the input char can be found in it, use the small h or k if found and big H or K if not found. This would be faster than using a replacement table, and make the code easier to maintain. However, for Fullwidth Latin Numbers and Letters, it looks better to use a simple replacement table as before. * We would keep ** 'a' for ascii latin chars ** 9 for ascii digits ** any other char/symbol will be kept as it is Useful links: https://gist.github.com/terrancesnyder/1345094 http://stackoverflow.com/questions/10809176/regular-expressions-regex-in-japanese http://www.localizingjapan.com/blog/2012/01/20/regular-expressions-for-japanese-text/ http://stackoverflow.com/questions/6787716/regular-expression-for-japanese-characters In Java, we can use the regex replace function to replace all characters. {code:java} public static void main(String[] args) { String value = "salut, comment ça va! 1 rue "; String regex = "[a-z]"; Matcher lowercase = Pattern.compile(regex).matcher(value); Matcher numeric = Pattern.compile("cat").matcher(value); System.out.println(numeric.replaceAll(lowercase.replaceAll("a"))); } {code} In SQL, we need to search in the document of each database whether they natively support the replace function or not. It seems that MysQL does not, but needs a UDF instead.
Story points: 20