Package: tokenizer-xm
IntroductionThis package is an aggregation of several packages I found useful for text pre-processing including gensim and ntlk. I put them together to create a more comprehensive and convenient pipeline.Installationpip install tokenizer_xmUsageProcessing a single text stringfromtokenizer_xmimportTextPreProcessorimportstring# An example textexample_text="This is an amazing product! I've been using it for almost a year now and it's clearly better than any other products I've used."print("Original text:")print(example_text)print("---")print("Simple Preprocessed:")print("---")tk=TextPreProcessor(text=example_text,lemma_flag=False,stem_flag=False,stopwords=[])print(tk.process())print("---")print("Pre-processing with regular contractions (e.g. I've -> I have):")# In this package, I included a dictionary of regular contractions for your conveniencetk=TextPreProcessor(text=example_text,lemma_flag=False,stem_flag=False,\contractions=[],stopwords=[])print(tk.process())print("---")print("Pre-processing with lemmatization:")tk=TextPreProcessor(text=example_text,lemma_flag=True,stem_flag=False,\stopwords=[])print(tk.process())print("---")print("Pre-processing with lemmatization and stemming:")# This package uses the SnowballStemmer from ntlk.stem. I will try to make it customizable latertk=TextPreProcessor(text=example_text,lemma_flag=True,stem_flag=True,\stopwords=[])print(tk.process())print("---")print("Adding stop words")# This package uses the SnowballStemmer from ntlk.stem. I will try to make it customizable latertk=TextPreProcessor(text=example_text,lemma_flag=True,stem_flag=True,\stopwords=["this",'be',"an",'it'])print(tk.process())print("---")Original text:
This is an amazing product! I've been using it for almost a year now and it's clearly better than any other products I've used.
---
Simple Preprocessed:
---
['this', 'is', 'an', 'amazing', 'product', 'i', 'have', 'been', 'using', 'it', 'for', 'almost', 'a', 'year', 'now', 'and', 'it', 'has', 'it', 'is', 'clearly', 'better', 'than', 'any', 'other', 'products', 'i', 'have', 'used']
---
Pre-processing with regular contractions (e.g. I've -> I have):
['this', 'is', 'an', 'amazing', 'product', 'i', 'have', 'been', 'using', 'it', 'for', 'almost', 'a', 'year', 'now', 'and', 'it', 'has', 'it', 'is', 'clearly', 'better', 'than', 'any', 'other', 'products', 'i', 'have', 'used']
---
Pre-processing with lemmatization:
['this', 'be', 'an', 'amaze', 'product', 'i', 'have', 'be', 'use', 'it', 'for', 'almost', 'a', 'year', 'now', 'and', 'it', 'have', 'it', 'be', 'clearly', 'better', 'than', 'any', 'other', 'product', 'i', 'have', 'use']
---
Pre-processing with lemmatization and stemming:
['this', 'be', 'an', 'amaz', 'product', 'i', 'have', 'be', 'use', 'it', 'for', 'almost', 'a', 'year', 'now', 'and', 'it', 'have', 'it', 'be', 'clear', 'better', 'than', 'ani', 'other', 'product', 'i', 'have', 'use']
---
Adding stop words
['amaz', 'product', 'i', 'have', 'use', 'for', 'almost', 'a', 'year', 'now', 'and', 'have', 'clear', 'better', 'than', 'ani', 'other', 'product', 'i', 'have', 'use']
---The order of stop words removal and lemmatization/stemmingThe current algorithmperforms lemmatization and stem before stop-words removal. Thus,You need to be carefull when defining a list of stop words. For example, including the term "product" will also remove the term "production" if you set the stem_flag to True or the term "products" if you set lemma_flag to True.When the lemma_flag is set to True, terms like "is" and "are" will be lemmatized to "be". And if "be" is not in the list of stopwords, it will remain. It is recommended that you process the list of stop-words as well if you decide to perform lemmatization"""Example"""text="products, production, is"stop_words=['product','is']tk=TextPreProcessor(text=text,lemma_flag=False,stem_flag=False,\stopwords=stop_words)# Use the .txt_pre_pros_all method instead when the input is a corpusprint(tk.process())['products', 'production']tk=TextPreProcessor(text=text,lemma_flag=True,stem_flag=False,\stopwords=stop_words)# Use the .txt_pre_pros_all method instead when the input is a corpusprint(tk.process())['production', 'be']tk=TextPreProcessor(text=text,lemma_flag=True,stem_flag=True,\stopwords=stop_words)# Use the .txt_pre_pros_all method instead when the input is a corpusprint(tk.process())['be']