Package: neattext
neattextNeatText:a simple NLP package for cleaning textual data and text preprocessing.
Simplifying Text Cleaning For NLP & MLProblemCleaning of unstructured text dataReduce noise [special characters,stopwords]Reducing repetition of using the same code for text preprocessingSolutionconvert the already known solution for cleaning text into a reuseable packageDocsCheck out the full docshereInstallationpipinstallneattextUsageThe OOP Way(Object Oriented Way)NeatText offers 5 main classes for working with text dataTextFrame : a frame-like object for cleaning textTextCleaner: remove or replace specificsTextExtractor: extract unwanted text dataTextMetrics: word stats and metricsTextPipeline: combine multiple functions in a pipelineOverall Components of NeatTextUsing TextFrameKeeps the text asTextFrameobject. This allows us to do more with our text.It inherits the benefits of the TextCleaner and the TextMetrics out of the box with some additional features for handling text data.This is the simplest way for text preprocessing with this library alternatively you can utilize the other classes too.>>>importneattextasnt>>mytext="This is the mail example@gmail.com ,our WEBSITE is https://example.com 😊.">>>docx=nt.TextFrame(text=mytext)>>>docx.text"This is the mail example@gmail.com ,our WEBSITE is https://example.com 😊.">>>>>>docx.describe()KeyValueLength:73vowels:21consonants:34stopwords:4punctuations:8special_char:8tokens(whitespace):10tokens(words):14>>>>>>docx.length73>>># Scan Percentage of Noise(Unclean data) in text>>>d.noise_scan(){'text_noise':19.17808219178082,'text_length':73,'noise_count':14}>>>>>>docs.head(16)'This is the mail'>>>docx.tail()>>>docx.count_vowels()>>>docx.count_stopwords()>>>docx.count_consonants()>>>docx.nlongest()>>>docx.nshortest()>>>docx.readability()Basic NLP Task (Tokenization,Ngram,Text Generation)>>>docx.word_tokens()>>>>>>docx.sent_tokens()>>>>>>docx.term_freq()>>>>>>docx.bow()Basic Text Preprocessing>>>docx.normalize()'this is the mail example@gmail.com ,our website is https://example.com 😊.'>>>docx.normalize(level='deep')'this is the mail examplegmailcom our website is httpsexamplecom '>>>docx.remove_puncts()>>>docx.remove_stopwords()>>>docx.remove_html_tags()>>>docx.remove_special_characters()>>>docx.remove_emojis()>>>docx.fix_contractions()Handling Files with NeatTextRead txt file directly into TextFrame>>>importneattextasnt>>>docx_df=nt.read_txt('file.txt')Alternatively you can instantiate a TextFrame and read a text file into it>>>importneattextasnt>>>docx_df=nt.TextFrame().read_txt('file.txt')Chaining Methods on TextFrame>>>t1="This is the mail example@gmail.com ,our WEBSITE is https://example.com 😊 and it will cost $100 to subscribe.">>>docx=TextFrame(t1)>>>result=docx.remove_emails().remove_urls().remove_emojis()>>>print(result)'This is the mail  ,our WEBSITE is   and it will cost $100 to subscribe.'Clean TextClean text by removing emails,numbers,stopwords,emojis,etcA simplified method for cleaning text by specifying as True/False what to clean from a text>>>fromneattext.functionsimportclean_text>>>>>>mytext="This is the mail example@gmail.com ,our WEBSITE is https://example.com 😊.">>>>>>clean_text(mytext)'mail example@gmail.com ,our website https://example.com .'You can remove punctuations,stopwords,urls,emojis,multiple_whitespaces,etc by setting them to True.You can choose to remove or not remove punctuations by setting to True/False respectively>>>clean_text(mytext,puncts=True)'mail example@gmailcom website https://examplecom '>>>>>>clean_text(mytext,puncts=False)'mail example@gmail.com ,our website https://example.com .'>>>>>>clean_text(mytext,puncts=False,stopwords=False)'this is the mail example@gmail.com ,our website is https://example.com .'>>>You can also remove the other non-needed items accordingly>>>clean_text(mytext,stopwords=False)'this is the mail example@gmail.com ,our website is https://example.com .'>>>>>>clean_text(mytext,urls=False)'mail example@gmail.com ,our website https://example.com .'>>>>>>clean_text(mytext,urls=True)'mail example@gmail.com ,our website .'>>>Removing Punctuations [A Very Common Text Preprocessing Step]You remove the most common punctuations such as fullstop,comma,exclamation marks and question marks by setting most_common=True which is the defaultAlternatively you can also remove all known punctuations from a text.>>>importneattextasnt>>>mytext="This is the mail example@gmail.com ,our WEBSITE is https://example.com 😊. Please don't forget the email when you enter !!!!!">>>docx=nt.TextFrame(mytext)>>>docx.remove_puncts()TextFrame(text="This is the mail example@gmailcom our WEBSITE is https://examplecom 😊 Please dont forget the email when you enter ")>>>docx.remove_puncts(most_common=False)TextFrame(text="This is the mail examplegmailcom our WEBSITE is httpsexamplecom 😊 Please dont forget the email when you enter ")Removing Stopwords [A Very Common Text Preprocessing Step]You can remove stopwords from a text by specifying the language. The default language is EnglishSupported Languages include English(en),Spanish(es),French(fr)|Russian(ru)|Yoruba(yo)|German(de)>>>importneattextasnt>>>mytext="This is the mail example@gmail.com ,our WEBSITE is https://example.com 😊. Please don't forget the email when you enter !!!!!">>>docx=nt.TextFrame(mytext)>>>docx.remove_stopwords(lang='en')TextFrame(text="mail example@gmail.com ,our WEBSITE https://example.com 😊. forget email enter !!!!!")Remove Emails,Numbers,Phone Numbers,Dates,Btc Address,VisaCard Address,etc>>>print(docx.remove_emails())>>>'This is the mail  ,our WEBSITE is https://example.com 😊.'>>>>>>print(docx.remove_stopwords())>>>'This mail example@gmail.com ,our WEBSITE https://example.com 😊.'>>>>>>print(docx.remove_numbers())>>>docx.remove_phone_numbers()>>>docx.remove_btc_address()Remove Special Characters>>>docx.remove_special_characters()Remove Emojis>>>print(docx.remove_emojis())>>>'This is the mail example@gmail.com ,our WEBSITE is https://example.com .'Remove Custom PatternYou can also specify your own custom pattern, incase you cannot find what you need in the functions using theremove_custom_pattern()function>>>importneattext.functionsasnfx>>>ex="Last !RT tweeter multiple &#7777">>>>>>nfx.remove_custom_pattern(e,r'&#\d+')'Last !RT tweeter multiple  'Replace Emails,Numbers,Phone Numbers>>>docx.replace_emails()>>>docx.replace_numbers()>>>docx.replace_phone_numbers()Chain Multiple Methods>>>t1="This is the mail example@gmail.com ,our WEBSITE is https://example.com 😊 and it will cost $100 to subscribe.">>>docx=TextCleaner(t1)>>>result=docx.remove_emails().remove_urls().remove_emojis()>>>print(result)'This is the mail  ,our WEBSITE is   and it will cost $100 to subscribe.'Using TextExtractorTo Extract emails,phone numbers,numbers,urls,emojis from text>>>fromneattextimportTextExtractor>>>docx=TextExtractor()>>>docx.text="This is the mail example@gmail.com ,our WEBSITE is https://example.com 😊.">>>docx.extract_emails()>>>['example@gmail.com']>>>>>>docx.extract_emojis()>>>['😊']Using TextMetricsTo Find the Words Stats such as counts of vowels,consonants,stopwords,word-stats>>>fromneattextimportTextMetrics>>>docx=TextMetrics()>>>docx.text="This is the mail example@gmail.com ,our WEBSITE is https://example.com 😊.">>>docx.count_vowels()>>>docx.count_consonants()>>>docx.count_stopwords()>>>docx.word_stats()>>>docx.memory_usage()UsageThe MOP(method/function oriented way) Way>>>fromneattext.functionsimportclean_text,extract_emails>>>t1="This is the mail example@gmail.com ,our WEBSITE is https://example.com .">>>clean_text(t1,puncts=True,stopwords=True)>>>'this mail examplegmailcom website httpsexamplecom'>>>extract_emails(t1)>>>['example@gmail.com']Alternatively you can also use this approach>>>importneattext.functionsasnfx>>>t1="This is the mail example@gmail.com ,our WEBSITE is https://example.com .">>>nfx.clean_text(t1,puncts=True,stopwords=True)>>>'this mail examplegmailcom website httpsexamplecom'>>>nfx.extract_emails(t1)>>>['example@gmail.com']ExplainerExplain an emoji or unicode for emojiemoji_explainer()emojify()unicode_2_emoji()>>>fromneattext.explainerimportemojify>>>emojify('Smiley')>>>'😃'>>>fromneattext.explainerimportemoji_explainer>>>emoji_explainer('😃')>>>'SMILING FACE WITH OPEN MOUTH'>>>fromneattext.explainerimportunicode_2_emoji>>>unicode_2_emoji('0x1f49b')'FLUSHED FACE'UsageThe Pipeline Way>>>fromneattext.pipelineimportTextPipeline>>>t1="""This is the mail example@gmail.com ,our WEBSITE is https://example.com 😊. This is visa 4111 1111 1111 1111 and bitcoin 1BvBMSEYstWetqTFn5Au4m4GFg7xJaNVN2 with mastercard 5500 0000 0000 0004. Send it to PO Box 555, KNU""">>>p=TextPipeline(steps=[remove_emails,remove_numbers,remove_emojis])>>>p.fit(t1)'This is the mail  ,our WEBSITE is https://example.com . This is visa     and bitcoin BvBMSEYstWetqTFnAumGFgxJaNVN with mastercard    . Send it to PO Box , KNU'Check For steps and named steps>>>p.steps>>>p.named_stepsAlternatively you can also use this approachDocumentationPlease read thedocumentationfor more information on what neattext does and how to use is for your needs.You can also check
out our readthedocs pagehereMore Features To Addbasic nlp taskcurrency normalizerAcknowledgementsInspired by packages likeclean-textfrom Johannes Fillter andtextifyby JCharisTechNBContributions Are WelcomedNotice a bug, please let us know.Thanks A lotByJesse E.Agbe(JCharis)Jesus Saves @JCharisTech