Package: phrasal
PhrasalForewordsWhat is it ?Phrasal is a library of tools to help gather meaningful, proper sentences from websites.Well, at least if used together. Each tool has a value of its own.
For example, theNormalizer(my favorite!) is very useful for NLP, when you have a crappy text corpus you need to clean.
TheMocySplitteris a nice alternative to Moses when you need to cleverly split a stream of text into sentences, one per line.
Etc.Why was it developed ?I have been working on a project lately, calledSwissTextthat gathers Swiss German sentences from scraping the Internet (no kidding, see the LREC 2020 publication onarXiv).
To do so, I had to build upon existing tools and develop some of my own.
While they were initially for Swiss German, I figured that it would maybe be useful in other contexts, hence this repo which is a stripped-down version of some of the SwissText modules.How does it work ?This repo contains implementations of four types of tools, which constitute together a pipeline:converter: extract (main) text from raw HTML;normalizer: normalize the raw text, including the encoding, quotes, spaces, etc.;splitter: split the text into chunks (potential sentences);filterer: filter chunks to keep only "proper" sentences.For each step, I propose one or more implementations.Tools availableHtmlConvertersphrasal.BsConverterA converter built uponBeautifulSoupthat exact text found on the HTML.
Text from code blocks, scripts or styles is ignored.
It deals cleverly with encodings and always delivers text in UTF-8.phrasal.JustextConvertera converter based onjustext, that try to spot and remove boilerplate content.
By default, it only keeps "good" paragraph, that is text long enough to be a full sentences and with a low link density.Normalizersphrasal.Normalizer, or simplyphrasal.normalize_textNormalize some text (using a serie of homemade regexes), including: normalize spaces, replace combining diacritics by the accented letter codepoints and strip leftovers, normalize dashes and quotemarks, replace non-breakable spaces with regular ones, etc.It can also try to fix encoding errors (seeftfy) and strip most unicode emoji symbols.Splittersphrasal.MosesSplitterMoses' splittersplit-sentences.perlcompletely rewritten in Python. It thus perfectly mimics the behavior, while being 5x faster than calling perl from Python (approach taken byMosesTokenizerfor example).phrasal.MocySplitterAn improvement uponMosesSplitter, which: deals more efficiently with lowercase (people are lazy on the Web), try to preserve links, can split on:or;(optional), etc.Filterersphrasal.PatternSentenceFilterA filterer based on a list of simple rules a proper sentence should respect, such as "at least five words", "no S P E L L E Dwords", etc.What isawesome? The rules are expressed in a (homemade) YAML-based syntax and are highly customizable. If you don't like the behavior, have a look atpattern_sentence_filter.yamland try writing your own set of rules !link_utilsThephrasal.link_utilsmodule is a simple utility to process href links found on a page. It will resolve relative links
(given a base URL), remove duplicates, strip anchors and exclude non-HTTP/HTTPs links.To get the list of links from a URL (i.e.hreffound on the page main content), useextract_links:importphrasalall_links=phrasal.extract_links('https://github.com/derlin/phrasal')How to useInstall the library using:# regular install, one of:pythonsetup.pyinstallpipinstall.# for development, one of:pythonsetup.pydevelop
pipinstall-e.
pipinstall-e.[showcase]# for streamlitAs a libraryimportphrasalDone.From the command lineEach tool contains a command line interface with different arguments. Discover it by typing:python-mphrasal--helppython-mphrasal--help
Calloneofthetoolsfromthecommandline.Usage:classname[otherargumentsspecifictoclassname]|[-h]Allowedclassnamearguments:-BsConverter-JustextConverter-PatternSentenceFilter-MocySplitter-MosesSplitter-NormalizerHere are some examples:python-mphrasalJustextConverter-uhttps://icosys.ch/swisscrawl===fromURLhttps://icosys.ch/swisscrawl
AspartoftheSwissTranslationproject,SwissCrawlisacorpusof500,000+SwissGerman(GSW)[...][...]python-mphrasalPatternSentenceFilter-i<(echo'not-a-sentence\nYEAH !!!\nCet outil fonctionne très bien, je l’utilise tous les jours.')Cetoutilfonctionnetrèsbien,jel’utilisetouslesjours.python-mphrasalNormalizer-iraw_text.txt-oclean_text.txtI just need one tool...No problem, each tool is more or less independent.
You may want to simplify the code a bit (e.g. remove the interface inheritance, transform classes into static scripts, I don't know), but I hope the source code is self-explaining.Running testsTests are usingtoxandpytest. The easiest way to run them is:pipinstalltoxtox-venv
toxRunning the showcaseA showcase usingstreamlitis included.
It allows you to test the full pipeline straight from your browser and also play with the different tools and options
from theLive Customizer. Once you found what works for you, you can simply copy-paste the code snippet generated.Run the showcase locally by doing:pipinstallstreamlit
streamlitrunsrc/showcase/lit.pyLicenseThis work is licensed under Apache 2.0, so you can basically do anything with it.However, I wouldreally enjoyit if youcredit mesomehow, either by citing my name, send me an email to say hi (I get lonely sometime, may be nice to chat), leave a star on GitHub, or any other way you think may give me strength to keep doing open-source :blush:.Related resourcesget-htmlto get raw or renderer HTML (used in this repo)SwissTextSwissTranslation project page:octopus::octopus::octopus::octopus::octopus::octopus::octopus::octopus: (I just love octopuses)Personal websiteTODOadd some usecases, such as finding links, cleaning a text file, etc. add language support information