Package: mltronsAutoDataPrep
mltrons dptron: Dirty Data in, Clean Data Out!https://pypi.org/project/mltronsAutoDataPrep/IntroductionData is the most important element for data analysis. Real world data is unclean with a lot of spelling errors, missing values, formatting issues, skewness, no encoding or aggregation which makes it the most time-consuming & cumbersome task for analysts & scientists. As most of the scientists spend time around 80% of their time cleaning & preparing data, therefore weâ€™re introducing dptron to make that process extremely easier and faster!Dptron is an in-memory platform built for distributed & scalable data cleaning & preparation. DPtron is written in Python and is built on PySpark to deal with large amounts of data seamlessly. It uses an implementation of machine learning and deep learning algorithms to perform important data cleaning & preparation steps automatically. Dptron is extensible so that developers, analysts & scientists can streamline the process of data cleaning & preparation for better decision making while becoming more productive.Decision making is better & easier if the data is clean otherwise itâ€™s garbage-in and garbage-out.Important FeaturesSupports connection with AWS S3Supports upto 10TB of data sizeTreats spelling mistakes and other inconsistencies in URLsDetects & treats skewness in dataFeature engineering for time variableTreats & fills NULL values by using deep learning (next iteration)Treats spelling mistakes and other inconsistencies in other variables (next iteration)GETTING STARTED WITH DPTRON - AUTO DATA PREPInstalling On Mac OsOpen up your terminal and install Java8 required for pySpark:brewcaskinstalladoptopenjdk/openjdk/adoptopenjdk8**After installing Java8, set it as your default Java version:/usr/libexec/java_home-V**This will output thefollowing:Matching Java Virtual Machines (3):1.8.0_05, x86_64:   "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home
1.6.0_65-b14-462, x86_64:   "Java SE 6" /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
1.6.0_65-b14-462, i386: "Java SE 6" /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/HomePick the version you want to be the default (i.e 1.6.0_65-b14-462) then:exportJAVA_HOME=/usr/libexec/java_home-v1.8**After you've successfully install Java8, install dptron with the following command:pipinstallmltronsAutoDataPrepInstalling on WindowsIt's important that you replace all the paths that include the folder "Program Files" or "Program Files (x86)" to avoid future problems while running Spark.If you have Java already installed, you still need to fix the JAVA_HOME and PATH variables. To do that, you need to:1. Rename "Program Files" with "Progra~1"2. Rename "Program Files (x86)" with "Progra~2"Example: "C:\Program FIles\Java\jdk1.8.0_161" --> "C:\Progra~1\Java\jdk1.8.0_161"After renaming, make sure you have Java 8 installed and the environment variables correctly defined1:3. Download Java JDK 8 from [Java's official website]
(https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)After installing Java SDK 8, set the following environment variables:4. JAVA_HOME = C:\Progra~1\Java\jdk1.8.0_1615. PATH += C:\Progra~1\Java\jdk1.8.0_161\binAfter you've successfully installed and configured Java8, install dptron with the following command:pipinstallmltronsAutoDataPrepUsing dptron1. Reading data functionsaddresspath of the filelocallocation of the file exist (local pc or s3 bucket)file_formatformat of the file (csv,excel,parquet)s3s3 bucket credentials (applicable only if data on s3 bucket)frommltronsAutoDataPrep.lib.v2.Operations.readfileimportReadFileasrfres=rf.read(address="test.csv",local="yes",file_format="csv",s3={})2. Drop Features containing Null of certain thresholdprovide dataframe with threshold of null valuesreturn the list of columns containing null values more then the thresholdfrommltronsAutoDataPrep.lib.v2.Middlewares.drop_col_with_null_valimportDropNullValueColres=rf.read("test.csv",file_format='csv')drop_col=DropNullValueCol()columns_to_drop=drop_col.delete_var_with_null_more_than(res,threshold=30)df=res.drop(*columns_to_drop)3. Drop Features containing same valuesprovide dataframereturn the list of columns containing same valuesfrommltronsAutoDataPrep.lib.v2.Middlewares.drop_col_with_same_valimportDropSameValueColumndrop_same_val_col=DropSameValueColumn()columns_to_drop=drop_same_val_col.delete_same_val_com(res)df=res.drop(*columns_to_drop)4. Cleaned Url FeaturesAutomatically detects features containing UrlsPipeline structure to clean the urls usingNLPtechniquesfrommltronsAutoDataPrep.lib.v2.Pipelines.etl_pipelineimportEtlPipelineetl_pipeline=EtlPipeline()etl_pipeline.custom_url_transformer(res)res=etl_pipeline.transform(res)5. Split Date Time featuresAutomatically detects features containing date/timeSplit date time into usefull multiple feautures (day,month,year etc)frommltronsAutoDataPrep.lib.v2.Pipelines.etl_pipelineimportEtlPipelineetl_pipeline=EtlPipeline()etl_pipeline.custom_date_transformer(res)res=etl_pipeline.transform(res)6. Filling Missing ValuesUsing Deep Learning techniques Missing values are filledfrommltronsAutoDataPrep.lib.v2.Pipelines.etl_pipelineimportEtlPipelineetl_pipeline=EtlPipeline()etl_pipeline.custom_filling_missing_val(res)res=etl_pipeline.transform(res)7. Removing Skewness from featuresAutomatically detects which column contains skewnessMinimize skewness using statistical methodsfrommltronsAutoDataPrep.lib.v2.Pipelines.etl_pipelineimportEtlPipelineetl_pipeline=EtlPipeline()etl_pipeline.custom_skewness_transformer(res)res=etl_pipeline.transform(res)8. Remove Spelling mistakesProvide list of features in which contains spelling mistakesfrommltronsAutoDataPrep.lib.v2.Pipelines.etl_pipelineimportEtlPipelineetl_pipeline=EtlPipeline()etl_pipeline.custom_spell_transformer(res,['col1','col2'])res2=etl_pipeline.transform(res)DependenciesJava 8PySparkNumPypandaspython-dateutilpytzsee full list of dependicieshere