Package: words-n-fun
WORDS N FUN : Semantic analysis moduleThe purpose of this project is two folds:To normalize tools and how tos of semantic analysis projectsTo offer end to end pipelines to speed up the time to market of NLP servicesLicenceThis project is distributed under the GNU AFFERO GENERAL PUBLIC LICENSE V3.0. Please check the LICENSE file.Philosophy of this packageThis package used to contain two main parts:Preprocessing : to speed up the application of numerous preprocessing routines to text dataModelling : to speed up the developpement of NLP machine learning models
The later has been refactored into what is now called the NLP framework.PreprocessingOne of the main goals of the project was to build a pipeline that could accomodate several data types:strlistnp.arraypd.Seriespd.DataFramecsv file (where the supplied data is the path to the actual file)Whatever the supplied data type, it will kept along the pipeline, meaning that if a csv file is given, a new csv file will be created as the output. If the input is a pd.Series, another pd.Series of the same shape will be returned.
This feature is controled by the utils.data_agnostic function.Main scriptsutils.pyContains utils functions and tools, mainly the data agnostic feature.basic.pyContains the main data transformation functions.api.pyContains the main entry point of the package controlling how one could create an end to end transformation pipeline.Getting startedRequirements & setupComing soon: CI pipeline to push this package to pypiPython 3.X (tests are ran on python 3.8)Set up module dependencies :pip install -r requirements.txtSet up the actual module :pip install -e words_n_funorpython setup.py developDownload nltk stopwords data:importnltknltk.download('stopwords')How to use this moduleA notebook tutorial is provided in the tutorial directory.
Roughly speaking, her is what a basic use case should look like:# Module importfromwords_n_fun.preprocessingimportapi# Definition of the desired transformation pipelinepipeline=['remove_non_string','get_true_spaces','to_lower','remove_punct','remove_numeric','remove_stopwords','lemmatize','remove_accents','trim_string','remove_leading_and_ending_spaces']#### example 1 : csv file ##### Input fileinput_file="path/to/my/file.csv"# Column containing the text to preprocesscol="myText"# Separator charactersep=';'# Instanciation of a preprocessor objectpreprocessor=api.get_preprocessor(pipeline=pipeline,prefered_column=col,sep=sep)# Process dataoutput_file=preprocessor.transform(input_file)#### example 2 : list  ##### Input datainput_list=["First text to transform","Second text to pocess"]# Instanciation of a preprocessor objectpreprocessor=api.get_preprocessor(pipeline=pipeline)# Process dataoutput_list=preprocessor.transform(input_list)Best Practices & guidelinesContributors must try their best to follow these mainstream guidelines :Module managementhttps://docs.python.org/3.9/tutorial/modules.htmlPython developmenthttps://github.com/google/styleguide/blob/gh-pages/pyguide.md,https://www.python.org/dev/peps/pep-0008/All the preprocessing functions must take pd.Series data as input. The built-in utils.data_agnostic decorator allows the user to send different types of data but the actual function carrying out these transformation should be built around pd.Series.Numerous unit tests have been included intests:Please make sure to run them before trying to submit a merge requestNew features must be shipped with their corresponding tests