Package: preprocess-nlp
Preprocess NLP TextFramework DescriptionA simple and fast framework forPreprocessing or Cleaning of textExtracting top words or reduction of vocabularyFeature ExtractionWord VectorizationUses parallel execution by leveraging themultiprocessing libraryin Python for cleaning of text, extracting top words and feature extraction modules. Contains both sequential and parallel ways (For less CPU intensive processes) for preprocessing text with an option of user-defined number of processes.PS: There is no multi-processing support for word vectorizationCleaning Text- Clean text with various defined stages implemented using standardized techniques in Natural Language Processing (NLP)Vocab Reduction- Find the top words in the corpus, lets you choose a threshold to consider the words that can stay in the corpus and replaces the othersFeature Extraction- Extract features from corpus of text using SpaCyWord Vectorization- Simple code to convert words to vectors (TFIDF, Word2Vec, GloVe) usingScikit-learnandGensimPreprocess/Cleaning ModuleUsesnltkfor few of the stages defined below.
Various stages of cleaning include:StageDescriptionremove_tags_nonasciiRemove HTML tags, emails, URLs, non-ascii characters and converts accented characterslower_caseConverts the text to lower_caseexpand_contractionsExpands the word contractionsremove_punctuationRemove punctuation from text, but sentences are seperated by ' . 'remove_esacape_charsRemove escapse characters like \n, \t etcremove_stopwordsRemove stopwords using nltk pythonremove_numbersRemove all digits in the textlemmatizeUses WordNetLemmatizer to lemmatize textstemmingUses SnowballStemmer for stemming of textmin_word_lenMinimum word length to keep in textReduction of VocabularyShortlists top words based on the percentage as input. Replaces the words not shortlisted and replaces them efficienctly. Also, supports parallel and sequential processing.Feature Extraction ModuleUsesSpacy Pipemodule to avoid unnecessary parsing to increase speed.
Various stages of feature extraction include:StageDescriptionnounsExtract the list of Nouns from the given stringverbsExtract the list of Verbs from the given stringadjsExtract the list of Adjectives from the given stringnoun_phrasesExtract the list of Noun Phrases (Noun chunks) from the given stringkeywordsUsesYAKEfor extracting keywords from textnerExtracts Person, Location and Organization as named entitiesnumbersExtracts all digits in the textWord VectorizationFunctions written in python to convert words to vectors using libraries like Scikit-Learn and Gensim. Contains four vectorization techniques like CountVectorizer (Bag of Words Model), TFIDF-Vectorizer, Word2Vec and GloVe. Also contains others features to get the top words according to IDF Scores, similar words with similarity scores and average sentence-wise vectors.Code - ComponentsVarious Python files and their purposes are mentioned here:preprocess_nlp.py- Contains functions which are built around existing techniques for preprocessing or cleaning text. Defines both sequential and parallel ways of code execution for preprocessing.Preprocessing_Example_Notebook.ipynb- How-to-use example notebook for preprocessing or cleaning stagesrequirements.txt- Required libraries to run the projectvocab_elimination_nlp.py- Contains functions which are built around existing techniques for shortlisting top words and reducing vocab sizeVocab_Elimination_Example_Notebook.ipynb- How-to-use example notebook for vocabulary reduction/elimination or replacement.vectorization_nlp.py- Contains functions which are built around existing techniques for vectorizing words.Vectorization_Example_Notebook.ipynb- How-to-use example notebook for vectorization of words and additional functions or features.How to runpip install -r requirements.txtImportpreprocess_nlp.pyand use the functionspreprocess_nlp(for sequential) andasyn_call_preprocess(for parallel) as defined in notebookImportvocab_elimination_nlp.pyand use functions as defined in the notebookVocab_Elimination_Example_Notebook.ipynbImportfeature_extraction.pyand use functions as defined in notebookFeature_Extraction_Example_Notebook.ipynbImportvectorization_nlp.pyand use functions as defined in notebookVectorization_Example_Notebook.ipynbSequential & Parallel ProcessingSequential   - Processes records in a sequential order, does not consume a lot of CPU Memory but is slower compared to Parallel processingParallel     - Can create multiple processes (customizable/user-defined) to preprocess text parallelly, Memory intensive and fasterRefer the code for Docstrings and other function related documentation.Cheers :)