Package: jerome
JeromeA collection of functions that illustrate several techniques useful in text compression.Table of ContentsAbout The ProjectGetting StartedUsageAdditional DocumentationRoadmapContributingLicenseContactAbout The ProjectJerome is a library of string functions written in pure Python. The library's name is taken from St. Jerome of Stridon who is considered the patron saint of archivists.Zero dependencies[^1]100% test coverage[^1]: Examples use augustine_text to generate material to compressGetting StartedTo get a local copy up and running follow these simple steps.Installing with pippipinstalljeromeFor information about cloning and dev setup see:ContributingUsageHere is an example showing basic usage.fromdatetimeimportdatetimefromjeromeimport(SymbolKeeper,common,forward_bw,replacer,reverse_bw,runlength_decode,runlength_encode)fromjerome.sample_textimportwords# 75K words of procedurally generated text# This is about the length of novel.text=words(75000)text_length=len(text)compression_start=datetime.now()# SymbolKeeper is used to portion out un-used symbolsk=SymbolKeeper(reserved=set(list(text)))# These appear in our text so we don't want to use them as placeholders# common is a utility function for finding commonly occuring words# We're using k from above to create a dictionary where each key is a word#  and the value is a single symbol replacement for that wordreplacements={word:next(k)forwordincommon(text,min_length=4)}# {'dolore': '\x00', 'elit,': '\x02', 'labore': '\x03', ...# Run replacementsreplaced=replacer(text,replacements)# Burrows Wheeler transform the text to improve runlength resulttransformed=forward_bw(replaced)# Runlength encoderuncoded=runlength_encode(transformed)print(f"""| step | result || ---- | ------ || Original Text size |{text_length}|| With words replaced |{len(replaced)}|| Encoded |{(rlen:=len(runcoded))}|| Reasonable length |{(dlen:=len(str([(k,v)fork,vinreplacements.items()])))}|| Compressed size % |{round(((rlen+dlen)/text_length)*100,2)}|""")compression_end=datetime.now()# Reverse the whole thingassert(unruncoded:=runlength_decode(runcoded))==transformedassert(untransformed:=reverse_bw(unruncoded))==replacedassertreplacer(untransformed,replacements,reverse=True)==textprint(f"| Compression time |{round((compression_end-compression_start).total_seconds()*1000.0)}ms |")print(f"| Decompression time |{round((datetime.now()-compression_end).total_seconds()*1000.0)}ms |")print(f"| Total time |{round((datetime.now()-compression_start).total_seconds()*1000.0)}ms |")Example compression of randomized text:stepresultOriginal Text length402270With words replaced198228Encoded74160Reasonable length5724Compressed size %19.86Compression time1045 msDecompression time172 msTotal time1217 msNOTETime was taken on a Ryzen 3600x @ 3.9Ghz.This is only an example of how the text functions in Jerome work.
Python's built in bz2 is both many times faster and many time better at compressing than the above example.Additional DocumentationBurrows Wheeler TransformRoadmapIn place BWTHuffman CodingAdditional examplesSee theopen issuesfor a list of proposed features (and known issues).ContributingContributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make aregreatly appreciated.Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Add tests, we aim for 100% test coverageUsing CoverageCommit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull RequestCloning / Development setupClone the repo and installgitclonehttps://github.com/kajuberdut/Jerome.gitcdJerome
pipenvinstall--devRun testspipenvshell
py.testFor more about pipenv see:Pipenv GithubLicenseDistributed under the MIT License. SeeLICENSEfor more information.ContactPatrick Shechet -patrick.shechet@gmail.comProject Link:https://github.com/kajuberdut/Jerome