Package: text-processing-util-mds24
text_processing_util_mds24Welcome to the repository for text processing, a part of the DSCI-524 course by Group 10 in the MDS-V Cohort 8 at UBC.Empower your text analysis workflows with text processing package, a Python library designed for streamlined text processing tasks. This versatile package offers four key functions:text_cleanfor noise removal and text refinement,frequency_vectorizerto generate frequency-based vectors,tfidf_vectorizerfor TF-IDF vectorization, andtokenizer_paddingto assist in tokenization and padding of text sequences for usage in recurrent neural networks. By simplifying essential text preprocessing steps, this package facilitates efficient text-based analysis, providing an easy-to-use toolkit for natural language processing and text modeling endeavors.ContributorsOur team, in alphabetical order:Allan LeeJerry YuMo NorouziNasim Ghazanfari NasrabadiInstallationTo install the package from PyPI, please run the following command:pipinstalltext-processing-util-mds24Developer NotesNote:For those who are looking to develop and/or test this package, please follow the following instructions to install the package from this GitHub repository.Installation from RepositoryFirst, please make sure that you havepoetryandcondainstalled on your local computer. If not, please follow the official instructions for each respectively to install them. (poetry,conda)It is recommended to create a conda virtual environment to install the package by running the following commands:condacreate--nametext_processing_util_mds24python=3.9-y
condaactivatetext_processing_util_mds24Clone the repository to your local machine by running:gitclonegit@github.com:UBC-MDS/text_processing_util_mds24.gitFrom the root of this repository, install the package usingpoetryby running the following command. It is recommended to make sure that you are in the conda environment from Step 2 before running this command.poetryinstallTestingNote:Every function in this package except fortext_cleancallstext_cleanin the first line of the code. Hence, testing for errors that arise from unexpected inputs is only done fortext_clean. Integration testing is done for the other functions that calltext_clean.To test this package, please first make sure you have activated thetext_processing_util_mds24conda environment that was created in the previous section. Then, please run the following command from the root directory of the repository:pytesttests/If you would like to see line coverage, please run the following command from the root directory of the repository:pytest--cov=text_processing_util_mds24If you would like to see branch coverage, please run the following command from the root directory of the repository:pytest--cov-branch--cov=text_processing_util_mds24Functionstext_clean: Removes punctuation, turns all characters in each document lower case and removes numbers in documents.frequency_vectorizer: Calculates the frequency of each word in a list of text documents.tfidf_vectorizer: Calculates the Term Frequency-Inverse Document Frequency (TF-IDF) scores for a given list of documents, providing a numerical representation that highlights the importance of terms within the context of the entire document set.tokenizer_padding: Converts each word into an individual token represented by a number and pads shorter sequences, but keeps the order of the original sentence, which is important for RNNs.UsageHere are some examples of usage of the functions in this package.Example of usingtext_clean:fromtext_processing_util_mds24import(text_clean,tfidf_vectorizer,frequency_vectorizer,tokenizer_padding)docs=["Here is document one.","","we have document 2"]print(text_clean(docs))[['here', 'is', 'document', 'one'], [], ['we', 'have', 'document']]Example of usingfrequency_vectorizer:docs=["apple orange banana","apple banana banana"]result_tf_matrix,result_feature_names=frequency_vectorizer(docs)print(result_tf_matrix)print(result_feature_names)[[0.33333333 0.33333333 0.33333333]
 [0.33333333 0.66666667 0.        ]]
['apple', 'banana', 'orange']Example of usingtfidf_vectorizer:docs=["machine learning is interesting","machine learning is fascinating"]tfidf_matrix,feature_names=tfidf_vectorizer(docs)print(tfidf_matrix)print(feature_names)[[ 0.          0.         -0.10136628 -0.10136628 -0.10136628]
 [ 0.          0.         -0.10136628 -0.10136628 -0.10136628]]
['fascinating', 'interesting', 'is', 'learning', 'machine']Example of usingtokenizer_padding:docs=["one two three","one three four five"]tokenized_padded=tokenizer_padding(docs)print(tokenized_padded)[[1. 2. 3. 0.]
 [1. 3. 4. 5.]]For more demonstration of the functions intext_processing_util_mds24using a more real-life text sample, please see ourvignette.DocumentationThe official documentation for this package is hosted on Read the Docs:https://text-processing-util-mds24.readthedocs.io/en/latest/.EcosystemThis package is intended to clean and transform texts into different representations to feed into machine learning algorithms. Scikit-learn and Keras provide similar functionalities.frequency_vectorizer:https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.htmltfidf_vectorizer:https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.htmltokenizer_padding:https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizerhttps://www.tensorflow.org/api_docs/python/tf/keras/utils/pad_sequencesStill, our functions are different in some ways.frequency_vectorizercalculates the relative frequency of each word with regards to the total number of words in its document rather than the raw word counts.tokenizer_paddingcombines what would be an otherwise two-step process if one were to use Keras into one step. Bothtfidf_vectorizerandtokenizer_paddingoffer simpler functional APIs and implementations compared to the implementations from scikit-learn and Keras respectively.ContributingInterested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.Licensetext_processing_util_mds24was created by Jerry Yu, Nasim Ghazanfari Nasrabadi, Mohammad Norouzi, Allan Lee. It is licensed under the terms of the MIT license.Creditstext_processing_util_mds24was created withcookiecutterand thepy-pkgs-cookiecuttertemplate.