Package: textprepro
TextPrepro - Text PreprocessingTextPrepro- Everything Everyway All At Once Text Preprocessing: Allow you to preprocess both general and social media text with easy-to-use features. Help you gain insight from your data with analytical tools. Stand on the shoulders of various famous libraries (e.g., NLTK, Spacy, Gensim, etc.).Table of Contents⏳ Installation🚀 Quickstart🧹 Simply preprocess with the pipeline📂 Work with document or dataFrame🪐 Customize your own pipeline💡 Features & Guides📋 For General Text📱 For Social Media Text🌐 For Web Scraping Text📈 Analytical Tools⏳ InstallationSimply install viapip:pipinstalltextpreproor:pipinstall"git+https://github.com/umapornp/textprepro"🚀 Quickstart🧹 Simply preprocess with the pipelineYou can preprocess your textual data by using the functionpreprocess_text()with the default pipeline as follows:>>>importtextpreproaspre# Proprocess text.>>>text="ChatGPT is AI chatbot developed by OpenAI It is built on top of OpenAI GPT foundational large language models and has been fine-tuned an approach to transfer learning using both supervised and reinforcement learning techniques">>>text=pre.preprocess_text(text)>>>text"chatgpt ai chatbot developed openai built top openai gpt foundational large language model finetuned approach transfer learning using supervised reinforcement learning technique"📂 Work with document or dataFrameYou can preprocess your document or dataframe as follows:If you work with a list of strings, you can use the functionpreprocess_document()to preprocess each of them.importtextpreproaspre>>>document=["Hello123","World!@&"]>>>document=pre.preprocess_document(document)>>>document["hello","world"]If you work with a dataframe, you can use the functionapply()with the functionpreprocess_text()to apply the function to each row.importtextpreproaspreimportpandasaspd>>>document={"text":["Hello123","World!@&"]}>>>df=pd.DataFrame(document)>>>df["clean_text"]=df["text"].apply(pre.preprocess_text)>>>dftextclean_textHello123helloWorld!@&world🪐 Customize your own pipelineYou can customize your own preprocessing pipeline as follows:>>>importtextpreproaspre# Customize pipeline.>>>pipeline=[pre.lower,pre.remove_punctuations,pre.expand_contractions,pre.lemmatize]>>>text="ChatGPT is AI chatbot developed by OpenAI It is built on top of OpenAI GPT foundational large language models and has been fine-tuned an approach to transfer learning using both supervised and reinforcement learning techniques">>>text=pre.preprocess_text(text=text,pipeline=pipeline)>>>text"chatgpt is ai chatbot developed by openai it is built on top of openai gpt foundational large language model and ha been finetuned an approach to transfer learning using both supervised and reinforcement learning technique"💡 Features & GuidesTextPrep provides many easy-to-use features for preprocessing general text as well as social media text. Apart from preprocessing tools, TextPrep also provides useful analytical tools to help you gain insight from your data (e.g., word distribution graphs and word clouds).📋 For General Text👇 Misspelling CorrectionCorrect misspelled words:>>>importtextpreproaspre>>>text="she loves swiming">>>text=pre.correct_spelling(text)>>>text"she loves swimming"👇 Emoji & EmoticonRemove, replace, or decode emojis (e.g., 👍, 😊, ❤️):>>>importtextpreproaspre>>>text="very good 👍"# Remove.>>>text=pre.remove_emoji(text)>>>text"very good "# Replace.>>>text=pre.replace_emoji(text,"[EMOJI]")>>>text"very good [EMOJI]"# Decode.>>>text=pre.decode_emoji(text)>>>text"very good :thumbs_up:"Remove, replace, or decode emoticons (e.g., :-), (>_<), (^o^)):>>>importtextpreproaspre>>>text="thank you :)"# Remove.>>>text=pre.remove_emoticons(text)>>>text"thank you "# Replace.>>>text=pre.replace_emoticons(text,"[EMOTICON]")>>>text"thank you [EMOTICON]"# Decode.>>>text=pre.decode_emoticons(text)>>>text"thank you happy_face_or_smiley"👇 URLRemove or replace URLs:>>>importtextpreproaspre>>>text="my url https://www.google.com"# Remove.>>>text=pre.remove_urls(text)>>>text"my url "# Replace.>>>text=pre.replace_urls(text,"[URL]")>>>text"my url [URL]"👇 EmailRemove or replace emails.>>>importtextpreproaspre>>>text="my email name.surname@user.com"# Remove.>>>text=pre.remove_emails(text)>>>text"my email "# Replace.>>>text=pre.replace_emails(text,"[EMAIL]")>>>text"my email [EMAIL]"👇 Number & Phone NumberRemove or replace numbers.>>>importtextpreproaspre>>>text="my number 123"# Remove.>>>text=pre.remove_numbers(text)>>>text"my number "# Replace.>>>text=pre.replace_numbers(text)>>>text"my number 123"Remove or replace phone numbers.>>>importtextpreproaspre>>>text="my phone number +1 (123)-456-7890"# Remove.>>>text=pre.remove_phone_numbers(text)>>>text"my phone number "# Replace.>>>text=pre.replace_phone_numbers(text,"[PHONE]")>>>text"my phone number [PHONE]"👇 ContractionExpand contractions (e.g., can't, shouldn't, don't).>>>importtextpreproaspre>>>text="she can't swim">>>text=pre.expand_contractions(text)>>>text"she cannot swim"👇 StopwordRemove stopwords:
You can also specify stopwords:nltk,spacy,sklearn, andgensim.>>>importtextpreproaspre>>>text="her dog is so cute"# Default stopword is NLTK.>>>text=pre.remove_stopwords(text)>>>text"dog cute"# Use stopwords from Spacy.>>>text=pre.remove_stopwords(text,stpwords="spacy")>>>text"dog cute"👇 Punctuation & Special Character & WhitespaceRemove punctuations:>>>importtextpreproaspre>>>text="wow!!!">>>text=pre.remove_punctuations(text)>>>text"wow"Remove special characters:>>>importtextpreproaspre>>>text="hello world!! #happy">>>text=pre.remove_special_characters(text)>>>text"hello world happy"Remove whitespace:>>>importtextpreproaspre>>>text="  hello  world  ">>>text=pre.remove_whitespace(text)>>>text"hello world"👇 Non-ASCII Character (Accent Character)Standardize non-ASCII characters (accent characters):>>>importtextpreproaspre>>>text="latté café">>>text=pre.standardize_non_ascii(text)>>>text"latte cafe"👇 Stemming & LemmatizationStem text:>>>importtextpreproaspre>>>text="discover the truth">>>text=pre.stem(text)>>>text"discov the truth"Lemmatize text:>>>importtextpreproaspre>>>text="he works at a school">>>text=pre.lemmatize(text)>>>text"he work at a school"👇 Lowercase & UppercaseConvert text to lowercase & uppercase:>>>importtextpreproaspre>>>text="Hello World"# Lowercase>>>text=pre.lower(text)>>>text"hello world"# Uppercase>>>text=pre.upper(text)>>>text"HELLO WORLD"👇 TokenizationTokenize text: You can also specify types of tokenization:wordandtweet.>>>importtextpreproaspre>>>text="hello world @user #hashtag"# Tokenize word.>>>text=pre.tokenize(text,"word")>>>text["hello","world","@","user","#","hashtag"]# Tokenize tweet.>>>text=pre.upper(text,"tweet")>>>text["hello","world","@user","#hashtag"]📱 For Social Media Text👇 SlangRemove, replace, or expand slangs:>>>importtextpreproaspre>>>text="i will brb"# Remove>>>pre.remove_slangs(text)"i will "# Replace>>>pre.replace_slangs(text,"[SLANG]")"i will [SLANG]"# Expand>>>pre.expand_slangs(text)"i will be right back"👇 MentionRemove or replace mentions.>>>importtextpreproaspre>>>text="@user hello world"# Remove>>>text=pre.remove_mentions(text)>>>text"hello world"# Replace>>>text=pre.replace_mentions(text)>>>text"[MENTION] hello world"👇 HashtagRemove or replace hashtags.>>>importtextpreproaspre>>>text="hello world #twitter"# Remove>>>text=pre.remove_hashtags(text)>>>text"hello world"# Replace>>>text=pre.replace_hashtags(text,"[HASHTAG]")>>>text"hello world [HASHTAG]"👇 RetweetRemove retweet prefix.>>>importtextpreproaspre>>>text="RT @user: hello world">>>text=pre.remove_retweet_prefix(text)>>>text"hello world"🌐 For Web Scraping Text👇 HTML TagRemove HTML tags.>>>importtextpreproaspre>>>text="<head> hello </head> <body> world </body>">>>text=pre.remove_html_tags(text)>>>text"hello world"📈 Analytical Tools👇 Word DistributionFind word distribution.>>>importtextpreproaspre>>>document="love me love my dog">>>word_dist=pre.find_word_distribution(document)>>>word_distCounter({"love":2,"me":1,"my":1,"dog":1})Plot word distribution in a bar graph.>>>importtextpreproaspre>>>document="ChatGPT is AI chatbot developed by OpenAI It is built on top of OpenAI GPT foundational large language models and has been fine-tuned an approach to transfer learning using both supervised and reinforcement learning techniques">>>word_dist=pre.find_word_distribution(document)>>>pre.plot_word_distribution(word_dist)👇 Word CloudGenerate word cloud.>>>importtextpreproaspre>>>document="ChatGPT is AI chatbot developed by OpenAI It is built on top of OpenAI GPT foundational large language models and has been fine-tuned an approach to transfer learning using both supervised and reinforcement learning techniques">>>pre.generate_word_cloud(document)👇 Rare & Frequent WordRemove rare or frequent words.>>>importtextpreproaspre>>>document="love me love my dog"# Remove rare word>>>document=pre.remove_rare_words(document,num_words=2)"love me love"# Remove frequent word>>>document=pre.remove_freq_words(document,num_words=2)"my dog"