Package: wildebeest-nlp
wildebeestThe wildebeest scripts investigate, repair and normalize text for a wide range of issues at the character level.wb-ana(or wb_analysis.py)This script searches a tokenized text for a range of potential problems,
such as UTF-8 encoding violations, control characters, zero-with characters,
letters/numbers/punctuation/letter-modifiers from various scripts
(e.g. Latin and Cyrillic), tokens with letters from different scripts,
XML tokens, tokens with certain punctuation of interest, orphan letter modifiers,
non-canonical character combinations.wb-norm(or wb_normalize.py)This script automatically corrects some of the issues raised by wb-ana.
The script can repair common encoding errors, normalize characters into their UTF8-canonical form, map digits and some
punctuation to ASCII, delete many non-printable characters and perform other repair, normalization and cleaning steps.
A few steps are specific to Pashto, Farsi, or Devanagari (Hindi etc.).
Normalization steps can be activatedà la carte.InstallationClick here for installation info# Install from PyPi:pipinstallwildebeest-nlp# Alternatively, pip-install from GitHub master branch:pipinstallgit+https://github.com/uhermjakob/wildebeest.git# Alternatively, clone GitHub, which might be useful for editing/development:gitclonehttps://github.com/uhermjakob/wildebeest.git# or git clone git://github.com/uhermjakob/wildebeest.gitcdwildebeest
pipinstall--editable.# run it from dir having setup.pyA pip-install will provide commandswb-normandwb-anaas well as their alternate formswb_normalize.pyandwb_analysis.py.After a regulargit clone(without pip-install), in order to be able to call the Python scriptswb_normalize.pyandwb_analysis.py, make sure that:wb_normalize.pyandwb_analysis.pyare executable (i.e. 'x' mode bits are set)your $PYTHONPATH includes the directory in which this README file resides in ("outer wildebeest") andyour $PATH includes the directory that includeswb_normalize.pyandwb_analysis.py("inner wildebeest")wb-norm (or wb_normalize.py)The script repairs common encoding errors, normalizes characters into their canonical form,
deletes many non-printable characters and performs other repair, normalization and cleaning steps.
The script can be parameterized to include or exclude specific normalization steps (e.g. whether
or not to map non-ASCII digits and punctuation to ASCII).
A few steps are specific to Pashto, Farsi, or Devanagari (Hindi etc.).Usage   (click below for details)CLI to normalize a file:wb-normorwb_normalize.pyusage: wb-norm [-h] [-i INPUT-FILENAME] [-o OUTPUT-FILENAME] [--lc LANGUAGE-CODE] [--skip NORM-STEPS]
               [--add NORM-STEPS] [--all] [--all-except NORM-STEPS] [--only NORM-STEPS] [-v] [--version]
# or wb_normalize.py [-h] ...

Normalizes and cleans a given text

options:
  -h, --help            show this help message and exit
  -i INPUT-FILENAME, --input INPUT-FILENAME
                        (default: STDIN)
  -o OUTPUT-FILENAME, --output OUTPUT-FILENAME
                        (default: STDOUT)
  --lc LANGUAGE-CODE    ISO 639-3, e.g. 'fas' for Persian
  --skip NORM-STEPS     perform all default normalization/cleaning steps except those specified in comma-separated list
                        (default normalization/cleaning steps: repair-encoding-errors,del-surrogate,del-ctrl-char,
                        del-tatweel,core-compat,pres-form,hangul,repair-combining,combining-compose,combining-decompose,
                        repair-xml,repair-url-escapes)
  --add NORM-STEPS      perform all default normalization/cleaning steps plus those specified in comma-separated list 
                        (non-default normalization/cleaning steps: del-zero-width,del-arabic-diacr,del-hebrew-diacr,
                        ligatures,signs-and-symbols,cjk,width,font,small,vertical,enclosure,punct,punct-dash,punct-arabic,
                        punct-cjk,punct-greek,punct-misc-f,space,digit,arabic-char,farsi-char,pashto-char,georgian-char,
                        look-alike,repair-token)
  --all                 perform all normalization/cleaning steps, i.e. repair-encoding-errors,del-surrogate,
                        del-zero-width,del-ctrl-char,del-tatweel,del-arabic-diacr,del-hebrew-diacr,core-compat,pres-form,
                        ligatures,signs-and-symbols,cjk,width,font,small,vertical,enclosure,hangul,repair-combining,
                        combining-compose,combining-decompose,punct,punct-dash,punct-arabic,punct-cjk,punct-greek,
                        punct-misc-f,space,digit,arabic-char,farsi-char,pashto-char,georgian-char,look-alike,repair-xml,
                        repair-url-escapes,repair-token
  --all-except NORM-STEPS
                        perform all normalization/cleaning steps except those specified in comma-separated list
  --only NORM-STEPS     perform only normalization/cleaning steps specified in comma-separated list
  -v, --verbose         write change log etc. to STDERR
  --version             show program's version number and exitExamples:wb-norm -h  # for full usage info
wb-norm --version
cd `pip show wildebeest-nlp | grep ^Location | cut -d ' ' -f 2`  # go to directory where wildebeest-nlp is installed
cd wildebeest/test/data
wb-norm --lc fas -i wildebeest-test.txt -o wildebeest-test-norm.txt
wb-norm --lc fas --verbose --skip del-ctrl-char,del-tatweel < wildebeest-test.txt > wildebeest-test-norm-custom.txt
wb-norm --all < wildebeest-test.txt > wildebeest-test-norm-all.txt
wb-norm --all-except del-arabic-diacr,del-hebrew-diacr < wildebeest-test.txt
wb-norm --only del-arabic-diacr,del-hebrew-diacr < wildebeest-test.txt
wb-norm --add del-arabic-diacr,del-hebrew-diacr --skip del-ctrl-char,del-tatweel < wildebeest-test.txtSame for alternate script name wb_normalize.pywb_normalize.py -h  # for full usage info
wb_normalize.py --version
cd `pip show wildebeest-nlp | grep ^Location | cut -d ' ' -f 2`
cd wildebeest/test/data
wb_normalize.py --lc fas -i wildebeest-test.txt -o wildebeest-test-norm.txt
wb_normalize.py --lc fas --verbose --skip del-ctrl-char,del-tatweel < wildebeest-test.txt > wildebeest-test-norm-custom.txt
wb_normalize.py --all < wildebeest-test.txt > wildebeest-test-norm-all.txt
wb_normalize.py --all-except del-arabic-diacr,del-hebrew-diacr < wildebeest-test.txt
wb_normalize.py --only del-arabic-diacr,del-hebrew-diacr < wildebeest-test.txt
wb_normalize.py --add del-arabic-diacr,del-hebrew-diacr --skip del-ctrl-char,del-tatweel < wildebeest-test.txtNote: For robustness regarding input files that do not fully conform to UTF8, please use -i (rather than STDIN), as it includes UTF8-encoding error handling.norm_clean_string (Python function call to normalize a string)Note: When working on a clone (as opposed to a pip-install), please make sure that your $PYTHONPATH includes the directory in which this README file resides.fromwildebeest.wb_normalizeimportWildebeestwb=Wildebeest()ht=wb.build_norm_step_dict(base='ALL')# base values: 'NONE', 'DEFAULT', 'ALL' (normalization steps)# ht = wb.build_norm_step_dict()  # defaults: base = 'DEFAULT', skip = None, add = None# ht = wb.build_norm_step_dict(base='NONE', add=['digit', 'enclosure'])  # normalize only digits (to ASCII) and enclosures# ht = wb.build_norm_step_dict(base='DEFAULT', skip=['del-tatweel'], add=['digit', 'space'])# ht = wb.build_norm_step_dict(base='ALL', skip=['punct-dash', 'enclosure', 'del-arabic-diacr'])wb.load_look_alike_file()# optionalprint(wb.norm_clean_string('🄐…25kmÂ²',ht,lang_code='eng'))print(wb.norm_clean_string('೧೯೨೩',ht,lang_code='kan'))Normalization StepsThe script can perform a wide variety of normalization steps.12 normalization steps are performed by default, including basic character repair and UTF8 encoding normalization. The default is generally suitable for applications that largely need to preserve the original text.Another 25 normalization steps are available through options--add (list of steps),--all,--all-except (list of steps). The--alland--all-excpetsettings are suitable for many NLP applications.Default normalization steps can be disabled by option--skip (list of steps).Option--only (list of steps)applies only the normalization steps listed (without default normalization steps unless explicitly listed).Option--all-except (list of steps)is equivalent to--all --skip (list of steps)List of normalization steps included by defaultrepair-encoding-errorsThe script generally expects input encoded in UTF8. However, it will recognize and repair some common text encoding errors:(Some) text is still encoded in Windows1252 or Latin1. Any byte that is not part of a well-formed UTF8 character will be interpreted as a Windows1252 character (and mapped to UTF8). This includes printable Latin1 characters as a subset.Text in Windows1252 was incorrectly converted to UTF8 by a Latin1-to-UTF8 converter. This maps Windows1252 characters \x80-\x9F to \u0080-\uu009F, which is the Unicode block of C1 control characters. These C1 control characters are extremely rare, and so our script will interpret such C1 control characters as ill-converted Windows1252 characters, as do many major software applications such as Google Chrome, Microsoft Outlook, Github (text files) and PyCharm (where they are often displayed in a slightly different form).Text in Windows1252 or Latin1 was converted twice, using some combination of Latin1-to-UTF8 converter and Windows1252-to-UTF converter; or a file already in UTF8 was incorrectly subjected to another conversion. Samplewildebeestrepair:Input: Donât tell your âfiancÃ©â â SchÃ¶ne GrÃ¼Ãe aus MÃ¤hrenâ¦ â Ma sÅur trouve Ã§a Â«bÃªteÂ». Â¡CoÃ±o! â¬50 â¢ 25kmÂ² â¢ Â½ÂµmOutput: Don’t tell your “fiancé” — Schöne Grüße aus Mähren… – Ma sœur trouve ça «bête». ¡Coño! €50 • 25km² • ½µmdel-surrogatedeletes surrogate characters (representing non-UTF8 characters in input), alternative/backup to windows-1252del-ctrl-chardeletes control characters (expect tab and linefeed), some variation selectorsdel-tatweeldeletes Arabic tatweel (a text alignment character that increases the distance between Arabic letters)core-compatnormalizes Hangul Compatibility characters to Unicode standard Hangul characterspres-forme.g. maps from presentation form (isolated, initial, medial, final) to standard formhangulcombine Hangul jamos onto Hangul syllablesrepair-combininge.g. order of nukta/vowel-signcombining-composee.g. applies combining-modifiers to preceding character, e.g. ö (o +  ̈) -> öcombining-decomposee.g. for some Indian characters, splits off Nuktarepair-xmle.g. repairs multi-escaped tokens such as &quot; or &amp;#x200C;repair-url-escapese.g. repairs multi-escaped url substrings such as Jo%25C3%25ABlle_AubronList of additional normalization steps included by --all optiondel-zero-widthdeletes zero-width characters, byte order mark, directional marks, join marksarabic-charto Arabic canonical forms, e.g. maps Farsi kaf/yeh to Arabic versionsfarsi-charto Farsi canonical forms, e.g. maps Arabic yeh, kaf to Farsi versionspashto-charto Pashto canonical forms, e.g. maps Arabic kaf to Farsi versiongeorgian-charto Georgian canonical forms, e.g. to standard script, map archaic charactersligaturese.g. decomposes non-Arabic ligatures (e.g. ĳ, ﬃ, Ǆ, ﬓ)signs-and-symbolse.g. maps symbols (e.g. kappa symbol) and signs (e.g. micro sign µ)cjke.g. CJK square composites (e.g. ㋀㏾)widthe.g. maps fullwidth and halfwidth characters to ASCII, e.g. Ａ to Afontmaps font-variations characters such as ℂ, ℹ, 𝒜 to regular characterssmallmaps small versions of characters to normal versions, such as small ampersand ﹠ to regular &verticalmaps vertical versions of punctuation characters with normal horizontal version, such as vertical em-dash ︱ to horizontal em-dash —enclosuredecomposes circled, squared and parenthesized characters, e.g. 🄐 to (A)del-arabic-diacre.g. deletes optional Arabic diacritics such as fatha, damma, kasradel-hebrew-diacre.g. deletes Hebrew pointsdigite.g. maps decimal-system digits of 54 scripts to ASCII digitspuncte.g. maps ellipsis … to periods ... and two-dot-lead ‥ to ..; a few math symbols ∭; ⒛ 🄆punct-dashe.g. maps various dashes, hyphens, minus signs to ASCII hyphen-minuspunct-arabice.g. Arabic exclamation mark etc. to ASCII equivalentpunct-cjke.g. Chinese Ideographic Full Stop etc. to ASCII equivalentpunct-greeke.g. Greek question mark etc. to ASCII equivalentpunct-misc-fe.g. Tibetan punctuation to ASCII equivalentspacee.g. maps non-zero spaces to normal spacelook-alikenormalizes Latin/Cyrillic/Greek look-alike characters, e.g. Latin character A to Greek Α (capital alpha) in otherwise Greek wordrepair-tokene.g. splits +/-/*/digits off Arabic words; maps not-sign inside Arabic to token-separating hyphenwb-ana (or wb_analysis.py)Script searches a tokenized text for a range of potential problems,
such as UTF-8 encoding violations, control characters, zero-with characters,
letters/numbers/punctuation/letter-modifiers from various scripts,
tokens with letters from different scripts, XML tokens, tokens with certain
punctuation of interest, orphan letter modifiers, non-canonical character
combinations.UsageCLI to analyze a file:wb-anaorwb_analysis.pyusage: wb-ana  [-h] [-i INPUT-FILENAME] [--batch BATCH] [-s] [-o OUTPUT-FILENAME] [-j JSON-OUTPUT-FILENAME] [--file_id FILE_ID]
               [--lc LANGUAGE-CODE] [-v] [-pb] [-n MAX_CASES] [-x MAX_EXAMPLES] [-r REF-FILENAME] [--version]
# or wb_analysis.py  [-h] ... 
  
Analyzes a given text for a wide range of anomalies

options:
  -h, --help            show this help message and exit
  -i INPUT-FILENAME, --input INPUT-FILENAME
                        (default: STDIN)
  --batch BATCH_DIR     Directory with batch of input files (BATCH_DIR/*.txt)
  -s, --summary         single summary line per file
  -o OUTPUT-FILENAME, --output OUTPUT-FILENAME
                        (default: STDOUT)
  -j JSON-OUTPUT-FILENAME, --json JSON-OUTPUT-FILENAME
                        (default: None)
  --file_id FILE_ID
  --lc LANGUAGE-CODE    ISO 639-3, e.g. 'fas' for Persian
  -v, --verbose         write change log etc. to STDERR
  -pb, --progress_bar   Show progress bar
  -n MAX_CASES, --max_cases MAX_CASES
                        max number of cases per group
  -x MAX_EXAMPLES, --max_examples MAX_EXAMPLES
                        max number of examples per line
  -r REF-FILENAME, --ref_id_file REF-FILENAME
                        (optional file with sentence reference IDs)
  --version             show program's version number and exitExamples:wb-ana --help
echo 'Hеllο!' | wb-ana                  # 'Hеllο!' mischievously includes a Cyrillic and a Greek character
echo 'Hеllο!' | wb-norm --all | wb-ana  # different result
cd `pip show wildebeest-nlp | grep ^Location | cut -d ' ' -f 2`  # go to directory where wildebeest-nlp is installed
cd wildebeest/test/data
wb-ana -i hello.txt
wb-ana -i wildebeest-test.txt -o wildebeest-test-out
wb-ana --batch phrasebook -s -o phrasebook-dir-out
wb-ana -i phrasebook/deu.txt -r phrasebook/eng.txt -o phrasebook-deu-out
wb-ana -i wildebeest-test-invalid-utf8.txtSame for alternate script name wb_analysis.pywb_analysis.py --help
echo 'Hеllο!' | wb_analysis.py
echo 'Hеllο!' | wb_normalize.py --all | wb_analysis.py
cd `pip show wildebeest-nlp | grep ^Location | cut -d ' ' -f 2`
cd wildebeest/test/data
wb_analysis.py -i hello.txt
wb_analysis.py -i wildebeest-test.txt -o wildebeest-test-out
wb_analysis.py --batch phrasebook -s -o phrasebook-dir-out
wb_analysis.py -i phrasebook/deu.txt -r phrasebook/eng.txt -o phrasebook-deu-out
wb_analysis.py -i wildebeest-test-invalid-utf8.txtwildebeest.wb_analysis.process (Python function call to analyze a string, a list of strings, or a file)Note: When working on a clone (as opposed to a pip-install), please make sure that your $PYTHONPATH includes the directory in which this README file resides.importpprintimportsysimportwildebeest.wb_analysisaswb_anawb=wb_ana.process(string="Hеllο!")# "Hеllο!" mischievously includes a Cyrillic and a Greek characterwb.pretty_print(sys.stdout)# custom pretty-print with OVERVIEW and DETAIL sections to STDOUTpprint.pprint(wb.analysis)# generic pretty-printimportwildebeest.wb_analysisaswb_anawb=wb_ana.process(strings=["Hеllο!","Tschüß"])print(wb.analysis)# print analysis object (nested dictionary)Assuming an input filecorpus.txt, e.g. built by:printf'Hеllο!\nTschüß\n'>corpus.txtimportwildebeest.wb_analysisaswb_anawb=wb_ana.process(in_file='corpus.txt')print(wb.analysis)importwildebeest.wb_analysisaswb_anawithopen(f'out.txt','w')asout,open('out.json','w')asjson:wb_ana.process(in_file='corpus.txt',pp_output=out,json_output=json)wb-analysis.plOld Perl script searches a tokenized text for a range of potential problems,
such as UTF-8 encoding violations, control characters, non-ASCII punctuation,
characters from a variety of language groups, very long tokens, unsplit 's,
unsplit punctuation, script mixing; split URLs, email addresses, filenames,
XML tokens.Reports the number of instances in each category and give examples.
Currently available: wildebeest_analysis.pl (Perl) v2.6 (April 28, 2021)