Package: data-purifier
Data-PurifierA Python library for Automated Exploratory Data Analysis, Automated Data Cleaning and Automated Data Preprocessing For Machine Learning and Natural Language Processing Applications in Python.Table of ContentsData-PurifierInstallationGet StartedAutomated EDA for NLPAutomated Data Preprocessing for NLPAutomated EDA for Machine LearningAutomated Report Generation for Machine LearningExampleInstallationPrerequsitesAnacondaTo use Data-purifier, it's recommended to create a new environment, and install the required dependencies:To install from PyPi:condacreate-n<your_env_name>python=3.6anaconda
condaactivate<your_env_name># ON WINDOWS: `source activate <your_env_name>`pipinstalldata-purifier
python-mspacydownloaden_core_web_smTo install from source:cd<Data-Purifier_Destination>
gitclonehttps://github.com/Elysian01/Data-Purifier.git# or download and unzip https://github.com/Elysian01/Data-Purifier/archive/master.zipcondacreate-n<your_env_name>python=3.6anaconda
condaactivate<your_env_name># ON WINDOWS: `source activate <your_env_name>`cdData-Purifier

pipinstall-rrequirements.txt
python-mspacydownloaden_core_web_smGet StartedLoad the moduleimportdatapurifierasdpfromdatapurifierimportMleda,Nleda,Nlpurifier,MlReportprint(dp.__version__)Get the list of the example datasetprint(dp.get_dataset_names())# to get all dataset namesprint(dp.get_text_dataset_names())# to get all text dataset namesLoad an example dataset, pass one of the dataset names from the example list as an argument.df=dp.load_dataset("womens_clothing_e-commerce_reviews")Automated EDA for NLPBasic NLPIt will check for null rows and drop them (if any) and then will perform following analysis row by row and will return dataframe containing those analysis:Word CountCharacter CountAverage Word LengthStop Word CountUppercase Word CountLater you can also observe distribution of above mentioned analysis just by selecting the column from the dropdown list, and our system will automatically plot it.It can also performsentiment analysison dataframe row by row, giving the polarity of each sentence (or row), later you can also view thedistribution of polarity.Word AnalysisCan find count ofspecific wordmentioned by the user in the textbox.Plotswordcloud plotPerformUnigram, Bigram, and Trigramanalysis, returning the dataframe of each and also showing its respective distribution plot.Code ImplementationFor Automated EDA and Automated Data Cleaning of NL dataset, load the dataset and pass the dataframe along with the targeted column containing textual data.nlp_df=pd.read_csv("./datasets/twitter16m.csv",header=None,encoding='latin-1')nlp_df.columns=["tweets","sentiment"]Basic AnalysisFor Basic EDA, pass the argumentbasicas argument in constructoreda=Nlpeda(nlp_df,"tweets",analyse="basic")eda.dfWord AnalysisFor Word based EDA, pass the argumentwordas argument in constructoreda=Nlpeda(nlp_df,"tweets",analyse="word")eda.unigram_df# for seeing unigram datfarameAutomated Data Preprocessing for NLPIn automated data preprocessing, it goes through the following pipeline, and return the cleaned data-frameDrop Null RowsConvert everything to lowercaseRemoves digits/numbersRemoves html tagsConvert accented chars to normal lettersRemoves special and punctuation charactersRemoves stop wordsRemoves multiple spacesCode ImplementationPass in the dataframe with the name of the column which you have to cleancleaned_df=NLAutoPurifier(df,target="tweets")Widget Based Data PreprocessingIt provides following cleaning techniques, where you have to just tick the checkbox and our system will automatically perform the operation for you.FeaturesFeaturesFeaturesDrop Null RowsLower all WordsContraction to ExpansionRemoval of emojisRemoval of emoticonsConversion of emoticons to wordsCount UrlsGet Word CountCount MailsConversion of emojis to wordsRemove Numbers and Alphanumeric wordsRemove Stop WordsRemove Special Characters and PunctuationsRemove MailsRemove Html TagsRemove UrlsRemove Multiple SpacesRemove Accented CharactersYou can convert word to its base form by selecting eitherstemmingorlemmatizationoption.Remove Top Common Word: By giving range of word, you canremove top common wordRemove Top Rare Word: By giving range of word, you canremove top rare wordAfter you are done, selecting your cleaning methods or techniques, click onStart Purifyingbutton to let the magic begins. Upon its completion you can access the cleaned dataframe by<obj>.dfCode Implementationpure=Nlpurifier(nlp_df,"tweets")View the processed and purified dataframepure.dfAutomated EDA for Machine LearningIt gives shape, number of categorical and numerical features, description of the dataset, and also the information about the number of null values and their respective percentage.For understanding the distribution of datasets and getting useful insights, there are many interactive plots generated where the user can select his desired column and the system will automatically plot it. Plot includesCount plotCorrelation plotJoint plotPair plotPie plotCode ImplementationLoad the dataset and let the magic of automated EDA begindf=pd.read_csv("./datasets/iris.csv")ae=Mleda(df)aeAutomated Report Generation for Machine LearningReport contains sample of data, shape, number of numerical and categorical features, data uniqueness information, description of data, and null information.df=pd.read_csv("./datasets/iris.csv")report=MlReport(df)ExampleColab NotebookOfficial Documentation:https://cutt.ly/CbFT5Dw