Package: kshingle
kshingleUtility functions to split a string into character-level k-shingles,
shingle sets, sequences of k-shingles.The packagekshinglecan be deployed for the following use cases:Character-level Shingling for MinHash/LSH: The result is a set of unique shingles for each document.Transform text into Input Sequences for
NNs: The result is input sequence
with k features.Install packagepipinstall"kshingle>=0.10.0,<1"Usage for MinHashingPlease note that the packagekshingleonly addresses character-level
shingles, andnotcombining word tokens (n-grams, w-shingling).Generate Shingle SetsFor algorithms like MinHash
(e.g.datasketchpackage) a
document (i.e. a string) must be split into a set of unique shingles.importkshingleasksshingles=ks.shingleset_k("abc",k=3)# {'a', 'ab', 'abc', 'b', 'bc', 'c'}importkshingleasksshingles=ks.shingleset_range("abc",2,3)# {'ab', 'abc', 'bc', 'c'}importkshingleasksshingles=ks.shingleset_list("abc",[1,3])# {'a', 'abc', 'b', 'c'}Wildcard Shingle SetsTypos can lead to rare shingles, that don’t match with the correct
spelling. The longer the shingled text, the less important the effect of
typos. However, short text strings will produce less shingles, i.e. the
variance of the similarity due to typos is much higher for short text
strings than for large text documents. In order to smooth this effect,
we can generate variants of a specfic shingle by replacing characters
with a wildcard characters (e.g. special unicode characters such asU+FFFF).Example: Withk=5the document"aBc DeF"would result in 25
unique shingles without assumned typos. For each of these shingles, we
enumerate all variants of up to 2 typos. This leads to a total of 152
unique shingles with no typo, 1 typo, and 2 typos.importkshingleasksshingles=ks.shingleset_k("aBc DeF",k=5)# -> 25 shinglesshingles=shingles.union(ks.wildcard_shinglesets(shingles,n_max_wildcards=2))# -> 152 shinglesdatasketch usageimportdatasketchimportkshingleasks# Enable wildcard variants and check the resultswith_wildcard=Falses1=ks.shingleset_k("Die Zeitung wird zugestellt.",k=5)s2=ks.shingleset_k("Der Bericht wird zugestellt",k=5)ifwith_wildcard:s1=s1.union(ks.wildcard_shinglesets(s1,2))s2=s1.union(ks.wildcard_shinglesets(s2,2))m1=datasketch.MinHash(num_perm=128)forsins1:m1.update(s.encode('utf8'))m2=datasketch.MinHash(num_perm=128)forsins2:m2.update(s.encode('utf8'))m1.jaccard(m2)Utility functionsimportkshingleasksmetric=ks.jaccard_strings("Bericht","berichten",k=5)# 0.5128205128205128ReferencesA. Z. Broder, “On the resemblance and containment of documents,” in
Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat.
No.97TB100171), Salerno, Italy, 1998, pp. 21–29, doi:10.1109/SEQUEN.1997.666900Ch. 3 in: J. Leskovec, A. Rajaraman, and J. D. Ullman, Mining of
Massive Datasets, 2nd ed. Cambridge: Cambridge University Press,
2014. URL:http://infolab.stanford.edu/~ullman/mmds/book.pdf“MinHash,” Wikipedia. Apr. 17, 2021, Accessed: May 01, 2021.
Available:https://en.wikipedia.org/w/index.php?title=MinHash&oldid=1018264865.Usage for Input SequencesConvert a string to a sequences of shinglesUsing thekparameterimportkshingleasksshingles=ks.shingleseqs_k("aBc DeF",k=3)# [['a', 'B', 'c', ' ', 'D', 'e', 'F'],#  ['aB', 'Bc', 'c ', ' D', 'De', 'eF'],#  ['aBc', 'Bc ', 'c D', ' De', 'DeF']]Using a range forkimportkshingleasksshingles=ks.shingleseqs_range("aBc DeF",n_min=2,n_max=3)# [['aB', 'Bc', 'c ', ' D', 'De', 'eF'],#  ['aBc', 'Bc ', 'c D', ' De', 'DeF']]Using a specific list of k valuesimportkshingleasksshingles=ks.shingleseqs_list("aBc DeF",klist=[2,5])# [['aB', 'Bc', 'c ', ' D', 'De', 'eF'],#  ['aBc D', 'Bc De', 'c DeF']]PaddingThe functionsshingleseqs_k,shingleseqs_range, andshingleseqs_listcan pad the sequence with aplaceholderelement. Thepaddingmodes arecenter: Pad on both sides (Theevenpad='pre'| 'postparameter is only available forpadding='center', and applied on
sequences with evenn-shingles)pre: Pad at the beginning of the sequencepost: Pad at the end of sequenceimportkshingleasksshingles=ks.shingleseqs_list("1234567",k=5,padding='center',evenpad='pre',placeholder='x')[[f"{s:^5}"forsinseq]forseqinshingles][['  1  ', '  2  ', '  3  ', '  4  ', '  5  ', '  6  ', '  7  '],
 ['  x  ', ' 12  ', ' 23  ', ' 34  ', ' 45  ', ' 56  ', ' 67  '],
 ['  x  ', ' 123 ', ' 234 ', ' 345 ', ' 456 ', ' 567 ', '  x  '],
 ['  x  ', '  x  ', '1234 ', '2345 ', '3456 ', '4567 ', '  x  '],
 ['  x  ', '  x  ', '12345', '23456', '34567', '  x  ', '  x  ']]Identify Vocabulary of unique shinglesimportkshingleasksdata=['Cerato­saurus („Horn-Echse“) ist eine Gattung theropoder Dino­saurier aus dem Ober­jura von Nord­ame­rika und Europa.','Charak­teris­tisch für diesen zwei­beini­gen Fleisch­fresser waren drei markante Hörner auf dem Schädel sowie eine Reihe kleiner Osteo­derme (Haut­knochen­platten), die über Hals, Rücken und Schwanz ver­lief.','Er ist der namens­gebende Vertre­ter der Cerato­sauria, einer Gruppe basaler (ursprüng­licher) Thero­poden.']shingled=[ks.shingleseqs_k(s,k=6)forsindata]VOCAB=ks.identify_vocab(shingled,sortmode='log-x-length',n_min_count=2,n_max_vocab=20)print(VOCAB)Upsert a word to VOCABimportkshingleasksVOCAB=['a','b']# insert because "[UNK]" doesn't existVOCAB,idx=ks.upsert_word_to_vocab(VOCAB,"[UNK]")print(idx,VOCAB)# 2 ['a', 'b', '[UNK]']# don't insert because "[UNK]" already existsVOCAB,idx=ks.upsert_word_to_vocab(VOCAB,"[UNK]")print(idx,VOCAB)# 2 ['a', 'b', '[UNK]']Encode sequences of shinglesimportkshingleasksdata=['abc d abc de abc def','abc defg abc def gh abc def ghi']shingled=[ks.shingleseqs_k(s,k=5)forsindata]VOCAB=ks.identify_vocab(shingled,n_max_vocab=10)VOCAB,unkid=ks.upsert_word_to_vocab(VOCAB,"[UNK]")# Encode all sequencesencoded=ks.encode_with_vocab(shingled,VOCAB,unkid)Find kFor biggerkvalues, the generate longer shingles that occur less
frequent. And less frequent shingles might be excluded inks.identify_vocab. As a result at some upperkvalue the
generated sequences only contains[UNK]encoded elements. The
functionks.shrink_k_backwardsidentifieskvalues that generate
sequences that contain at least one encoded shingle across all examples.importkshingleasksdata=['abc d abc de abc def','abc defg abc def gh abc def ghi']# Step 1: Build a VOCABshingled=[ks.shingleseqs_k(s,k=9)forsindata]VOCAB=ks.identify_vocab(shingled,n_max_vocab=10)VOCAB,unkid=ks.upsert_word_to_vocab(VOCAB,"[UNK]")encoded=ks.encode_with_vocab(shingled,VOCAB,unkid)# Identify k's that are actually usedklist=ks.shrink_k_backwards(encoded,unkid)# Step 2: Shingle sequences againshingled=[ks.shingleseqs_list(s,klist=klist)forsindata]encoded=encode_with_vocab(shingled,VOCAB,unkid)# ...Collectively Exhaustive Wildcard Shingling (CEWS)CEWS is a selection algorithm for k-shingles with wildcards to build a
vocabulary list.Extract and count shinglesFirst, build a databasedbwith shingles as keys and the occurence
within a corpus as values.fromcollectionsimportCounterimportkshingleasksimportitertools# load the corporadocs=["...","..."]# loop over all documentsdb=Counter()fordocindocs:# extract all shingles of different k-length (no wildcards!)shingles=ks.shingleseqs_k(doc,k=5)# bump it up to 8# count all unique shingles, and add the resultdb+=Counter(itertools.chain(*shingles))db=dict(db)len(db)Extra: Augment text by adding typological errorsIn order to increase the generalizibility of a trained ML model, we can
use text augmentation to produce possible edge case of errornous text.
High quality corpora try to avoid such errors, and corpora based
laymen’s text might not include each possible edge case.importaugtxt.keyboard_layoutsaskblfromaugtxt.augmentersimportwordaugimportnumpyasnpfromcollectionsimportCounter# Augmentation settings: Probability of typological errorssettings=[{'p':0.50,'fn':'typo.drop_n_next_twice','args':{'loc':['m','e'],'keep_case':[True,False]}},{'p':0.50,'fn':'typo.swap_consecutive','args':{'loc':['m','e'],'keep_case':[True,False]}},{'p':0.25,'fn':'typo.pressed_twice','args':{'loc':'u','keep_case':[True,False]}},{'p':0.25,'fn':'typo.drop_char','args':{'loc':['m','e'],'keep_case':[True,False]}},{'p':0.25,'fn':'typo.pressed_shiftalt','args':{'loc':['b','m'],'keymap':kbl.macbook_us,'trans':kbl.keyboard_transprob}},]# Number of augmentation rounds (i.e. the total count will be 10-1000x larger)n_augm_rounds=10# maximum percentage of augmentionspct_augmented=0.1pct_augmented*=(1.0+np.prod([cfg['p']forcfginsettings]))# Count factor for original shinglen_factor_original=int((n_augm_rounds/pct_augmented)*(1-pct_augmented))# reproducibilitynp.random.seed(seed=42)# loop over shingle frequency database (`db`)db2=Counter()fororiginalindb.keys():augmented=[wordaug(original,settings)for_inrange(n_augm_rounds)]# count all unique augmented shingles, and add the resultdb2+=Counter(augmented)# count the original shingledb2[original]+=n_factor_originaldb2=dict(db2)len(db2)Select the shingles (CEWS), create pattern list, and encode data# use `db` or `db2` (see above)importkshingleasksmemo=ks.cews(db2,max_wildcards=1,min_samples_leaf=10,threshold=0.9)# ensure that certain shingles are in the memoization cache#memo = {k: db[k] for k in ["i.e.", "e.g."]}#memo = ks.cews(db2, memo=memo, max_wildcards=1, min_samples_leaf=10, threshold=0.9)# `min_samples_leaf` as pecentage#memo = ks.cews(db2, max_wildcards=1, min_samples_leaf=0.0005, threshold=0.9)# try to stop early when reaching a target vocab size#memo = ks.cews(db2, max_wildcards=1, vocab_size=500, min_samples_leaf='auto', threshold=0.9)# test rare edge cases first#memo = ks.cews(db2, max_wildcards=1, priority='rare', min_samples_leaf=1, threshold=0.9)# Build a pattern listPATTERNS=ks.shingles_to_patterns(memo,wildcard='\uFFFF')unkid=sum([len(pats)forpatsinPATTERNS.values()])Finally, we can start to encode data# Encode dataencoded,shingled=ks.encode_multi_match_text(text,k=5,PATTERNS=PATTERNS,num_matches=3,unkid=unkid)# is slower# encoded, shingled = ks.encode_multi_match_corpus(#     [text], k=5, PATTERNS=PATTERNS, num_matches=3, unkid=unkid)AppendixInstallationThekshinglegit repois
available asPyPi packagepip install kshingle
pip install git+ssh://git@github.com/ulf1/kshingle.gitCommandsInstall a virtual environmentpython3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt --no-cache-dir
pip install -r requirements-dev.txt --no-cache-dir(If your git repo is stored in a folder with whitespaces, then don’t use
the subfolder.venv. Use an absolute path without whitespaces.)Python commandsCheck syntax:flake8--ignore=F401--exclude=$(grep-v'^#'.gitignore | xargs | sed-e's//,/g')Run Unit Tests:pytestPublishpandocREADME.md--frommarkdown--torst-s-oREADME.rstpythonsetup.pysdisttwineupload-rpypidist/*Clean upfind . -type f -name "*.pyc" | xargs rm
find . -type d -name "__pycache__" | xargs rm -r
rm -r .pytest_cache
rm -r .venvSupportPleaseopen an issuefor support.ContributingPlease contribute usingGithub
Flow. Create a branch,
add commits, andopen a pull
request.