Package: uniseg
A Python package to determine Unicode text segmentations.FeaturesThis package provides:Functions to get Unicode Character Database (UCD) properties concerned with
text segmentations.Functions to determine segmentation boundaries of Unicode strings.Classes that help implement Unicode-aware text wrapping on both console
(monospace) and graphical (monospace / proportional) font environments.Supporting segmentations are:code pointCode pointis“any value
in the Unicode codespace.”It is the basic unit for processing Unicode
strings.grapheme clusterGrapheme clusterapproximately represents“user-perceived character.”They may be made
up of single or multiple Unicode code points. e.g. “G” +acute-accentis
auser-perceived character.word breakWord boundaries are familiar segmentation in many common text operations.
e.g. Unit for text highlighting, cursor jumping etc. Note thatwordsare
not determinable only by spaces or punctuations in text in some languages.
Such languages like Thai or Japanese require dictionaries to determine
appropriate word boundaries. Though the package only provides simple word
breaking implementation which is based on the scripts and doesn’t use any
dictionaries, it also provides ways to customize its default behavior.sentence breakSentence breaks are also common in text processing but they are more
contextual and less formal. The sentence breaking implementation (which is
specified in UAX: Unicode Standard Annex) in the package is simple and
formal too. But it must be still useful in some usages.line breakImplementing line breaking algorithm is one of the key features of this
package. The feature is important in many general text presentations in
both CLI and GUI applications.RequirementsPython 3.8 or later.Installpip install unisegChanges0.8.0 (2024-02-08)Unicode 15.0.0.Regex-based grapheme cluster segmentation.Quit supporting Python versions < 3.8.0.7.2 (2022-09-20)Improve performance of Unicode lookups. PR by Max Bachmann.
<https://bitbucket.org/emptypage/uniseg-py/pull-requests/1>0.7.1 (2015-05-02)CHANGE: wrap.Wrapper.wrap(): returns the count of lines now.Separate LICENSE from README.txt for the packaging-related reason in some
environments.0.7.0 (2015-02-27)CHANGE: Quitted gathering all submodules’s members on the top, uniseg
module.CHANGE: Reformuniseg.wrapmodule and sample scripts.Maintained uniseg.wrap module, and sample scripts work again.0.6.4 (2015-02-10)Adduniseg-dbpathconsole command, which just print the path ofucd.sqlite3.Include sample scripts under the package’s subdirectory.0.6.3 (2015-01-25)Python 3.4Support modern setuptools, pip and wheel.0.6.2 (2013-06-09)Python 3.30.6.1 (2013-06-08)Unicode 6.2.0ReferencesUAX #29 Unicode Text Segmentation(15.0.0)<https://www.unicode.org/reports/tr29/tr29-41.html>UAX #14: Unicode Line Breaking Algorithm(15.0.0)<https://www.unicode.org/reports/tr14/tr14-49.html>Related / Similar ProjectsPyICU- Python extension wrapping the
ICU C++ APIPyICUis a Python extension wrapping International Components for
Unicode library (ICU). It also provides text segmentation supports and
they just perform richer and faster than those of ours. PyICU is an
extension library so it requires ICU dynamic library (binary files) and
compiler to build the extension. Our package is written in pure Python;
it runs slower but is more portable.pytextseg- Python module for text
segmentationpytextsegpackage focuses very similar goal to ours; it provides
Unicode-aware text wrapping features. They designed and uses their
original string class (not built-inunicode/strclasses) for the
purpose. We use strings as just ordinary built-inunicode/strobjects for text processing in our modules.