Package: levenshtein-finder
Similar string search in Levenshtein-distanceInstallfrom sourcegit clone https://github.com/lovit/levenshtein_finder
cd levenshtein_finder
python setup.py installfrom PyPIpip install levenshtein_finderUsageDefine Tokenizerlevenshtein_finderprovides two basic tokenizers.In [1]fromlevenshtein_finderimportCharacterTokenizer,BigramTokenizerchar_tokenizer=CharacterTokenizer()bigram_tokenizer=BigramTokenizer()char_tokenizer.tokenize("abc de")# ['a', 'b', 'c', ' ', 'd', 'e']bigram_tokenizer.tokenize("abc de")# ['ab', 'bc', 'c ', ' d', 'de']char_tokenizer.train("path/to/text")# orchar_tokenizer.train(["list","of","words"])When you use word piece tokenizer in Huggingface'stokenizers, wrap the class instance withWordpieceTokenizersWrapperIn [2]from tokenizers import BertWordPieceTokenizer
from levenshtein_finder import WordpieceTokenizersWrapper

tokenizer = WordpieceTokenizersWrapper(BertWordPieceTokenizer())
tokenizer.train("path/to/text")
tokenizer.tokenize("abc")  # ["ab", "#c"]
tokenizer.encode("abc"  # [53, 2]
tokenizer.decode(tokenizer.tokenize("abc"))  # "abc"Set tokenizer which Levenshtein Finder uses. Default isCharacterTokenizerIn [3]fromlevenshtein_finderimportLevenshteinFinder()finder=LevenshteinFinder()finder=LevenshteinFinder(BigramTokenizer())finder=LevenshteinFinder(WordpieceTokenizersWrapper(BertWordPieceTokenizer()))IndexingIn [4]finder=LevenshteinFinder()finder.indexing("path/to/text")# when you use CharacterTokenizer or BigramTokenizer, you can index with list of wordsfinder.indexing(["list","of","words"])SearchingBasic search. The form ofsimilarsisdict{term: distance}.In [5]finder.search("query")If you want to check details in searching, setverbose=TrueIn [6]finder.search("분식회계",verbose=True)Out [6]query               : 분식회계
tokens              : ['분', '식', '회', '계']
num data            : 132864
num 1st candidates  : 10137
num final candidates: 7
num similars        : 4
elapsed time        : 0.00542903 sec
[{'idx': 84243, 'data': '분식회계', 'distance': 0},
 {'idx': 36211, 'data': '분식회', 'distance': 1},
 {'idx': 113942, 'data': '분식회계설', 'distance': 1},
 {'idx': 114122, 'data': '분석회계', 'distance': 1}]For decomposing Korean to Jamo (NFKD)To decompose Korean character to jamo components (감-> [ㄱ,ㅏ,ㅁ]), useNormalizers.In [7]normalizer=Normalizers.create_normalizer(unicodedata=True)tokenizer=CharacterTokenizer(normalizer=normalizer)finder=LevenshteinFinder(tokenizer=tokenizer)finder.indexing("text.txt")finder.search("분식회계",verbose=True)Out [7]query               : 분식회계
tokens              : ['ᄇ', 'ᅮ', 'ᆫ', 'ᄉ', 'ᅵ', 'ᆨ', 'ᄒ', 'ᅬ', 'ᄀ', 'ᅨ']
num data            : 132864
num 1st candidates  : 125925
num final candidates: 4
num similars        : 2
elapsed time        : 0.0863643 sec
[{'idx': 84243, 'data': '분식회계', 'distance': 0},
 {'idx': 114122, 'data': '분석회계', 'distance': 1}]DemoSearching similar wordsLevenshteinFinderneedsseconds.In [8]importjsonwithopen("data/nouns_from_financial_news.json")asf:words=list(json.load(f).keys())finder=LevenshteinFinder()finder.indexing(words)finder.search("분식회계",verbose=True)Out [8]elapsed time        : 0.00542903 secBrute-force distance computation needs1.495 secondsIn [9]importtimefromlevenshtein_finderimportlevenshteinquery='분식회계'begin_time=time.time()distance={word:levenshtein(word,query)forwordinwords}search_time=time.time()-begin_timeprint('search time ={}sec'.format('%.3f'%search_time))similars=sorted(filter(lambdax:x[1]<=1,distance.items()),key=lambdax:x[1])elapsed_time=time.time()-begin_timeprint('total elapsed time ={}sec'.format('%.3f'%elapsed_time))print(similars)Out [9]search time = 1.480 sec
total elapsed time = 1.495 sec
[('분식회계', 0), ('분식회', 1), ('분식회계설', 1), ('분석회계', 1)]Searching similar stringsFind similar sentence from NSMC corpus (20M).In [10]finder=LevenshteinFinder()finder.indexing("~/Korpora/nsmc/corpus.txt")finder.search("음악이 주가 된 최고의 음악영화",max_distance=1,verbose=True)Out [10]query               : 음악이 주가 된 최고의 음악영화
tokens              : ['음', '악', '이', ' ', '주', '가', ' ', '된', ' ', '최', '고', '의', ' ', '음', '악', '영', '화']
num data            : 200002
num 1st candidates  : 189663
num final candidates: 1
num similars        : 1
elapsed time        : 0.123322 sec
[{'idx': 6, 'data': '음악이 주가 된, 최고의 음악영화', 'distance': 1}]