Package: esaxx-py
esaxx-pyPython wrapper of the Enhanced Suffix Array (ESA).This is a version adapted for Python, originating from the C++ implementation foundhere.Installationpipinstallesaxx-pyUsagefromesaimportesaxxdefprint_snippet(T,beg,length):foriinrange(length):c=T[beg+i]print("_"ifc.isspace()elsec,end="")T='abracadabra'SA=[0]*len(T)L=[0]*len(T)R=[0]*len(T)D=[0]*len(T)k=0x100node_num=0node_num=esaxx(T,SA,L,R,D,len(T),k,node_num)ifnode_num==-1:exit()print(f"node:{node_num}")foriinrange(node_num):print(f"{i}\t{R[i]-L[i]}\t{D[i]}\t",end="")print_snippet(T,SA[L[i]],D[i])print()node:5024abra151a223bra322ra4110Alternatively, you can use enumSubstring.py:ehcoabracadabra|pythonenumSubstring.pyFor the original implementation:g++enumSubstring.cppechoabracadabra|./a.outNoteIn the original implementation, the return value of esaxx was an error code, not node_num.
However, due to the constraints of Python and the difficulty in passing by reference, I've chosen to return node_num.Maximal SubstringsTo obtain Maximal Substrings:fromesaimportesaxxdefprint_snippet(T,beg,length):foriinrange(length):c=T[beg+i]print("_"ifc.isspace()elsec,end="")T='abracadabra'SA=[0]*len(T)L=[0]*len(T)R=[0]*len(T)D=[0]*len(T)k=0x100node_num=0node_num=esaxx(T,SA,L,R,D,len(T),k,node_num)ifnode_num==-1:exit()size=len(T)# Record changes in BWTrank=[0]*sizer=0foriinrange(size):ifi==0orT[(SA[i]+size-1)%size]!=T[(SA[i-1]+size-1)%size]:r+=1rank[i]=rprint("count\tlength\tstring")# Enumerate maximal partial stringsforiinrange(node_num):ifD[i]==0or(rank[R[i]-1]-rank[L[i]]==0):continueprint(f"{R[i]-L[i]}\t{D[i]}\t",end="")print_snippet(T,SA[L[i]],D[i])print()The first column represents the frequency of occurrence, and the second column represents the length of the string.countlengthstring24abra51aHere, even strings that appear more than once are listed, even if they are just one character. If you want to skip those, you can useif len < 2: continue.enumMaxSubstring.py:ehcoabracadabra|pythonenumMaxSubstring.pyC++ (enumMaxSubstring.cpp):g++ enumMaxSubstring.cpp
echo abracadabra | ./a.outUPDATE in 0.2.0Introduce a new function:get_maximal_substrings(str).This function allows for easier extraction of maximal substrings from a given string.Usage Example:fromesaimportget_maximal_substringsT='abracadabra'substrings=get_maximal_substrings(T)print("count\tlength\tstring")forsubstringinsubstrings:print(f'{substring.count}\t{substring.length}\t{substring.string})countlengthstring24abra51aUPDATE in 2.7.0Available unicode characterUsage Example:fromesaimportget_maximal_substrings_unicodeT='松島やああ松島や松島や'substrings=get_maximal_substrings_unicode(T)print(T)print("count\tlength\tstring")forsubstringinsubstrings:print(f'{substring.count}\t{substring.length}\t{substring.string})松島やああ松島や松島や
countlengthstring33松島や32島や31や21あAdditional InformationC++ Implementation:https://github.com/hillbig/esaxxhttps://code.google.com/archive/p/esaxx/https://github.com/TNishimoto/esaxxRust Version:https://github.com/Narsil/esaxx-rs/Software using esaxx:https://github.com/huggingface/tokenizershttps://github.com/google/sentencepiecehttps://github.com/shuyo/ldighttp://phontron.com/pialign/List of papers using esaxx:https://ipsj.ixsq.nii.ac.jp/ej/index.php?active_action=repository_view_main_item_detail&page_id=13&block_id=8&item_id=47681&item_no=1https://www.anlp.jp/proceedings/annual_meeting/2012/pdf_dir/A3-1.pdfhttps://www.anlp.jp/proceedings/annual_meeting/2012/pdf_dir/D5-2.pdfArticles aboutesaxx:https://tech.retrieva.jp/entry/2021/11/02/115816https://shogo82148.hatenablog.com/entry/20110916/1316172382https://takeda25.hatenablog.jp/entry/20101202/1291269994https://blog.shibayu36.org/entry/2017/01/28/150033