Package: nlpretext
NLPretextAll the goto functions you need to handle NLP use-cases, integrated in NLPretextTL;DRWorking on an NLP project and tired of always looking for the same silly preprocessing functions on the web?:tired_face:Need to efficiently extract email adresses from a document? Hashtags from tweets? Remove accents from a French post?:disappointed_relieved:NLPretext got you covered!:rocket:NLPretext packages in auniquelibrary all the textpreprocessingfunctions you need toeaseyour NLP project.:mag: Quickly explore below our preprocessing pipelines and individual functions referential.Default preprocessing pipelineCustom preprocessing pipelineReplacing phone numbersRemoving hashtagsExtracting emojisData augmentationCannot find what you were looking for? Feel free to open anissue.InstallationSupported Python VersionsMain version supported :3.8Other supported versions :3.9,3.10We strongly advise you to do the remaining steps in a virtual environnement.To install this library from PyPi, run the following command:pipinstallnlpretextor withPoetrypoetryaddnlpretextUsageDefault pipelineNeed to preprocess your text data but no clue about what function to use and in which order? The default preprocessing pipeline got you covered:fromnlpretextimportPreprocessortext="I just got the best dinner in my life @latourdargent !!! I  recommend 😀 #food #paris\n"preprocessor=Preprocessor()text=preprocessor.run(text)print(text)# "I just got the best dinner in my life!!! I recommend"Create your custom pipelineAnother possibility is to create your custom pipeline if you know exactly what function to apply on your data, here's an example:fromnlpretextimportPreprocessorfromnlpretext.basic.preprocessimport(normalize_whitespace,remove_punct,remove_eol_characters,remove_stopwords,lower_text)fromnlpretext.social.preprocessimportremove_mentions,remove_hashtag,remove_emojitext="I just got the best dinner in my life @latourdargent !!! I  recommend 😀 #food #paris\n"preprocessor=Preprocessor()preprocessor.pipe(lower_text)preprocessor.pipe(remove_mentions)preprocessor.pipe(remove_hashtag)preprocessor.pipe(remove_emoji)preprocessor.pipe(remove_eol_characters)preprocessor.pipe(remove_stopwords,args={'lang':'en'})preprocessor.pipe(remove_punct)preprocessor.pipe(normalize_whitespace)text=preprocessor.run(text)print(text)# "dinner life recommend"Take a look at all the functions that are availableherein thepreprocess.pyscripts in the different folders: basic, social, token.Load text dataPre-processing text data is useful only if you have loaded data to process! Importing text data as strings in your code can be really simple if you have short texts contained in a local .txt, but it can quickly become difficult if you want to load a lot of texts, stored in multiple formats and divided in multiple files. Hopefully, you can use NLPretext's TextLoader class to easily import text data.
while it is not mandatory our textLoader work best with dask, make sure to have the librairy installed if you want the best performances.fromnlpretext.textloaderimportTextLoaderfiles_path="local_folder/texts/text.txt"text_loader=TextLoader(use_dask=True)text_dataframe=text_loader.read_text(files_path)print(text_dataframe.text.values.tolist())# ["I just got the best dinner in my life!!!",  "I recommend", "It was awesome"]File path can be provided as string, list of strings, with or without wildcards. It also supports imports from cloud providers, if your machine is authentified on a project.text_loader=TextLoader(text_column="name_of_text_column_in_your_data")local_file_path="local_folder/texts/text.csv"# File from local folderlocal_corpus_path=["local_folder/texts/text_1.csv","local_folder/texts/text_2.csv","local_folder/texts/text_3.csv"]# Multiple files from local foldergcs_file_path="gs://my-bucket/texts/text.json"# File from GCSs3_file_path="s3://my-bucket/texts/text.json"# File from S3hdfs_file_path="hdfs://folder/texts/text.txt"# File from HDFSazure_file_path="az://my-bucket/texts/text.parquet"# File from Azuregcs_corpus_path="gs://my-bucket/texts/text_*.json"# Multiple files from GCS with wildcardtext_dataframe_1=text_loader.read_text(local_file_path)text_dataframe_2=text_loader.read_text(local_corpus_path)text_dataframe_3=text_loader.read_text(gcs_file_path)text_dataframe_4=text_loader.read_text(s3_file_path)text_dataframe_5=text_loader.read_text(hdfs_file_path)text_dataframe_6=text_loader.read_text(azure_file_path)text_dataframe_7=text_loader.read_text(gcs_corpus_path)You can also specify a Preprocessor if you want your data to be directly pre-processed when loaded.text_loader=TextLoader(text_column="text_col")preprocessor=Preprocessor()file_path="local_folder/texts/text.csv"# File from local folderraw_text_dataframe=text_loader.read_text(local_file_path)preprocessed_text_dataframe=text_loader.read_text(local_file_path,preprocessor=preprocessor)print(raw_text_dataframe.text_col.values.tolist())# ["These   texts are not preprocessed",  "This is bad ## "]print(preprocessed_text_dataframe.text_col.values.tolist())# ["These texts are not preprocessed",  "This is bad"]Individual FunctionsReplacing emailsfromnlpretext.basic.preprocessimportreplace_emailsexample="I have forwarded this email to obama@whitehouse.gov"example=replace_emails(example,replace_with="*EMAIL*")print(example)# "I have forwarded this email to *EMAIL*"Replacing phone numbersfromnlpretext.basic.preprocessimportreplace_phone_numbersexample="My phone number is 0606060606"example=replace_phone_numbers(example,country_to_detect=["FR"],replace_with="*PHONE*")print(example)# "My phone number is *PHONE*"Removing Hashtagsfromnlpretext.social.preprocessimportremove_hashtagexample="This restaurant was amazing #food #foodie #foodstagram #dinner"example=remove_hashtag(example)print(example)# "This restaurant was amazing"Extracting emojisfromnlpretext.social.preprocessimportextract_emojisexample="I take care of my skin 😀"example=extract_emojis(example)print(example)# [':grinning_face:']Data augmentationThe augmentation module helps you togenerate new textsbased on your given examples by modifying some words in the initial ones and tokeep associated entities unchanged, if any, in the case ofNER tasks. If you want words other than entities to remain unchanged, you can specify it within thestopwordsargument. Modifications depend on the chosen method, the ones currently supported by the module aresubstitutions with synonymsusing Wordnet or BERT from thenlpauglibrary.fromnlpretext.augmentation.text_augmentationimportaugment_textexample="I want to buy a small black handbag please."entities=[{'entity':'Color','word':'black','startCharIndex':22,'endCharIndex':27}]example=augment_text(example,method=”wordnet_synonym”,entities=entities)print(example)# "I need to buy a small black pocketbook please."📈 ReleasesYou can see the list of available releases on theGitHub Releasespage.We followSemantic Versionsspecification.We useRelease Drafter. As pull requests are merged, a draft release is kept up-to-date listing the changes, ready to publish when you’re ready. With the categories option, you can categorize pull requests in release notes using labels.For Pull Requests, these labels are configured, by default:LabelTitle in Releasesenhancement,feature🚀 Featuresbug,refactoring,bugfix,fix🔧 Fixes & Refactoringbuild,ci,testing📦 Build System & CI/CDbreaking💥 Breaking Changesdocumentation📝 Documentationdependencies⬆️ Dependencies updatesGitHub creates thebug,enhancement, anddocumentationlabels automatically. Dependabot creates thedependencieslabel. Create the remaining labels on the Issues tab of the GitHub repository, when needed.## 🛡 LicenseThis project is licensed under the terms of theApache Software License 2.0license. SeeLICENSEfor more details.## 📃 Citation@misc{nlpretext,
  author = {artefactory},
  title = {All the goto functions you need to handle NLP use-cases, integrated in NLPretext},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/artefactory/NLPretext}}}
}Project Organization├── LICENSE
├── CONTRIBUTING.md     <- Contribution guidelines
├── CODE_OF_CONDUCT.md  <- Code of conduct guidelines
├── Makefile
├── README.md           <- The top-level README for developers using this project.
├── .github/workflows   <- Where the CI and CD lives
├── datasets/external   <- Bash scripts to download external datasets
├── docker              <- All you need to build a Docker image from that package
├── docs                <- Sphinx HTML documentation
├── nlpretext           <- Main Package. This is where the code lives
│   ├── preprocessor.py <- Main preprocessing script
│   ├── text_loader.py  <- Main loading script
│   ├── augmentation    <- Text augmentation script
│   ├── basic           <- Basic text preprocessing
│   ├── cli             <- Command lines that can be used
│   ├── social          <- Social text preprocessing
│   ├── token           <- Token text preprocessing
│   ├── textloader      <- File loading
│   ├── _config         <- Where the configuration and constants live
│   └── _utils          <- Where preprocessing utils scripts lives
├── tests               <- Where the tests lives
├── pyproject.toml      <- Package configuration
├── poetry.lock         
└── setup.cfg           <- Configuration for plugins and other utilsCreditstextacyfor the following basic preprocessing functions:fix_bad_unicodenormalize_whitespaceunpack_english_contractionsreplace_urlsreplace_emailsreplace_numbersreplace_currency_symbolsremove_punctremove_accentsreplace_phone_numbers(with some modifications of our own)