Package: cypunct
Cypunct is designed to solve the problem of quickly splitting a Unicode
string based on a set of characters.Cypunct is designed to work on Python 2.6, 2.7, and 3.3+. Because
Cypunct is a Cython extension, it will (probably) only work in the CPython
runtime.For Python versions 2.6 and 2.7, Cypunct will only run if these
CPython runtimes are compiled with the flag--enable-unicode=ucs4. Cypunct will throw an exception
if your Python 2 runtime was not compiled with UCS-4.InstallationInstallation is easiest with pip. Just runpipinstallcypunctUsageCypunct takes a Unicode string and afrozensetof delimiter characters,
and splits the string based on that set. Every delimiter character
should be a single Unicode code point –len(char)should be 1.A simple example, where we provide a smallfrozensetis below.>>>fromcypunctimportsplit>>>split("James Mishra is the... best human ever, or so I think.",frozenset({' ','.',','}))['James','Mishra','is','the','best','human','ever','or','so','I','think','']However, if you only need to split on whitespace characters,str.split()much
better performance. If you only need to split on one character,str.split(char)will also be much faster.Cypunct really shines when you need to split on many possible characters,
such as an entireUnicode character category.The below example splits on all Unicode punctuation, and nothing else.>>>fromcypunct.unicode_classesimportP>>>split("James Mishra is the... best human ever, or so I think.",P)['James Mishra is the',' best human ever',' or so I think','']The following Unicode classes are available as sets:CategoryDescriptionCOtherCcOther, FormatCfOther, Not AssignedCoOther, Private UseCsOther, SurrogateLLetterLlLetter, LowercaseLmLetter, ModifierLoLetter, OtherLtLetter, TitlecaseLuLetter, UppercaseMMarkMcMark, Space CombiningMeMark, EnclosingMnMark, NonspacingNNumberNdNumber, Decimal DigitNlNumber, LetterNoNumber, OtherPPunctuationPcPunctuation, ConnectorPdPunctuation, DashPePunctuation, ClosePfPunctuation, Final QuotePiPunctuation, Initial QuotePoPunctuation, OtherPsPunctuation, OpenSSymbolScSymbol, CurrencySkSymbol, ModifierSmSymbol, MathSoSymbol, OtherZSeparatorZlSeparator, LineZpSeparator, ParagraphZsSeparator, Spacecypunct.unicode_classes.COMMON_SEPARATORSis the union of theC,P,S, andZfrozensets. I have found it personally useful when splitting text for natural
language processing applications.If you don’t specify afrozensetfor Cypunct to use, then Cypunct will
default toCOMMON_SEPARATORS.Updating Unicode dataCurrently,cypunct.unicode_classesis a Python module autogenerated from aUnicodeData.txtfile. The autogeneration script exists inmake_punctuation_file.py.Most Cypunct users will not need to concern themselves with this, but this is important
to know if you are experiencing Unicode bugs or want to contribute to Cypunct.The currentUnicodeData.txtis fromftp://ftp.unicode.org/Public/10.0.0/ucd/UnicodeData.txt.Frequently Asked Questions (FAQ)Q: I got an installation error involving
“pkg_resources.VersionConflict (setuptools xx.xx.xx”.
How do I fix this?You have a very old version of setuptools, and we won’t be able to
compile our Cython extension with it. Runpip install--upgradesetuptoolsand try installing Cypunct again.Q: Wouldn’t this be way faster if it were written in Pure C?Yes, it would. I’m too lazy to hand-code a C CPython extension, but it’s on my todo list.
Right now, Cypunct is“fast enough”, and I can move onto other things in my
daily life.However, if you want to take on the challenge of rewriting Cypunct in C and having
the exact same functionality as the current Cython version, I’ll send you $100 USD.