Package: tjtext-preproc-ds
This package is all in one text preprocessing pipeline and takes text input in the form of pandas series and outputs as pandas series object.All the text_processing sub-functions may be viewed herehttps://github.com/tariqjamil-bwp/pypi_projects/tree/main/tj_nlpInstalling:pip install .....Usage:(by example)
df2 = df1.copy()
df2['input_data'] = text_preprocessing_ds(df2['Phrase'])Note:It will not work with .apply(text_....) method, as .apply is item by item processing which may not be optimized at dataframe level.Details:
It has a variety of input flags, controlling the appropritae text preprocessing activity, and are by default set to True.
The details of flags is as under:** The function is optimized to take advantage of the multi threaded processors **Arguments:pdSeries: A positional argument, of pandas series object type, and should contain text phrases to be preprocessed** All following arguments are key-word type **NewLine removal:
Parameter: 'newlines_tabs'
This setting when enablesd causes the function to remove all the occurrences of newlines, tabs, and combinations like: \n, \.Example:
Input : This is her \ first day at this place.\n Please,\t Be nice to her.\n
Output : This is her first day at this place. Please, Be nice to her.HTML Tags
Parameter: 'remove_html'
This setting when enablesd causes the function to remove all the occurrences of html tags from the text.Example:
Input : This is a nice place to live.
Output : This is a nice place to live.Hyper Links removal
This setting when enablesd causes the function to remove all the occurrences of links.
Parameter: 'links'Example:
Input : To know more about cats and food & website: catster.com  visit:https://catster.com//how-to-feed-catsOutput : To know more about cats and food & website: visit:Removing extra white spaces
Parameter: 'extra_whitespace'
This setting when enablesd causes the function to remove extra whitespaces from the textExample:
Input : How   are   you   doing   ?
Output : How are you doing ?Accented Characters Removal:
Parameter: 'accented_chars'
This setting when enablesd causes the function to remove accented characters from the text contained within the Dataset.Example:
Input : Málaga, àéêöhello
Output : Malaga, aeeohelloLower Casing
Parameter: 'lowercase'
This setting when enablesd causes the function to convert text into lower case.Example:
Input : The World is Full of Surprises!
Output : the world is full of surprises!Removing Character Repetitions:
Parameter: 'repeatition'
This setting when enablesd causes the function to reduce repeatition to two characters for alphabets and to one character for punctuations.Example:
Input : Realllllllllyyyyy,        Greeeeaaaatttt   !!!!?....;;;;:)
Output : Reallyy, Greeaatt !?.;:)Expand Contractions:
Parameter: 'contractions'
This setting when enablesd causes the function to expand shortened words to the actual form.
e.g. don't to do notExample: 
Input : ain't, aren't, can't, cause, can't've
Output :  is not, are not, cannot, because, cannot haveRemoving Special Characters
Parameter: 'special_chars'
This setting when enablesd causes the function to remove all the special characters except a preset list of characters as they have imp meaning in the text provided.Example:
Input : Hello, T-a-r-i-q. Thi*s is $100.05 : the payment that you will recieve! (Is this okay?)
Output :  Hello, Tariq. This is $100.05 : the payment that you will recieve! Is this okay?Removong Stop Words
Parameter: 'stop_words'
This setting when enablesd causes the function to remove the stopwords which doesn't add much meaning to a sentence
& they can be remove safely without comprimising meaning of the sentence.Example:
Input : This is Tariq from karachi who came here to study.
Output : ["'This", 'Tariq', 'karachi', 'came', 'study', '.', "'"]Mis-spelled words Correction:
Parameter: 'mis_spell'
This setting when enablesd causes the function to correct spellings. Currently only English language is supported.Example:
Input : This is OAkbar from Londn who came heree to studdy.
Output : This is Akbar from London who came here to study.Lamentization:
Parameters: 'lemmatization_allow'
This setting when enablesd causes the function to converts word to their root words without explicitely cut down as done in stemming.Example:
Input : text reduced
Output :  text reduce