Package: pandas-data-cleaner
Pandas Data CleanerThis package is a data cleaning tool for Pandas DataFrames and other objects with a similar structure.The tool is designed to help clean data by providing a function onto which you can apply various cleaning methods.The main cleaning function can be found inpandas_data_cleaner.base.clean_data.The app also provides an abstract base classpandas_data_cleaner.base.CleaningStrategywhich can be used to implement custom cleaning strategies.InstallationTo install the application, run the following command:pipinstallpandas-data-cleanerCleaning DataIn order to clean data, you need:Pandas DataFrameList of strategies to applyAny additional arguments that you may need to pass to the cleaning function.Let's suppose we have the following DataFrame:importpandasaspddataframe=pd.DataFrame({"id":[1,2,1],"structure_value":["a","a","a"],"status":["ENABLED","ENABLED","DISABLED"],})As a table, this would look like this:idstructure_valuestatus1aENABLED2aENABLED1aDISABLEDIn this data frame, we can see that there are two rows with the same id but different values for status.As part of our cleaning exercise, we want to keep the latest row of data as this is the most up-to-date.Let's try to apply the RemoveDuplicates cleaning strategy to the data frame:importpandasaspdfrompandas_data_cleaner.baseimportclean_datafrompandas_data_cleaner.strategiesimportRemoveDuplicatesdataframe=pd.DataFrame({"id":[1,2,1],"structure_value":["a","a","a"],"status":["ENABLED","ENABLED","DISABLED"],})dataframe=clean_data(dataframe,[RemoveDuplicates])Running this will result in the following error:pandas_data_cleaner.exceptions.MissingOptionsError:Missingkwargs:
remove_duplicates_subset_fields
remove_duplicates_keepThis lets us that we need to provide additional arguments when calling the cleaning function, these are:remove_duplicates_subset_fieldsremove_dupplicates_keepTo find out more information about the additional arguments required, you can run:RemoveDuplicates.info()This will return some information on how the strategy works as well as additional information on the arguments that are required.For theRemoveDuplicatescleaning strategy,remove_duplicates_subset_fieldsis the fields we should perform the duplicate removal on andremove_duplicates_keepindicates given some duplicates are, which row should we keep.If we now tweak our earlier code:importpandasaspdfrompandas_data_cleaner.baseimportclean_datafrompandas_data_cleaner.strategiesimportRemoveDuplicatesdataframe=pd.DataFrame({"id":[1,2,1],"structure_value":["a","a","a"],"status":["ENABLED","ENABLED","DISABLED"],})dataframe=clean_data(dataframe,[RemoveDuplicates],remove_duplicates_subset_fields=["id"],remove_duplicates_keep="last")We will now get the following data frame:pd.DataFrame({"id":[2,1],"structure_value":["a","a"],"status":["ENABLED","DISABLED"],})As a table:idstructure_valuestatus2aENABLED1aDISABLEDAs we had setremove_duplicates_subset_fields=["id"], it found that there were two rows with the same ID. As we setremove_duplicates_keep="last", it kept the last row only.In our example, we used only one cleaning strategy, but we are free to use as many as we like, we simply need to add all the strategies to the list of cleaning strategies to apply.Creating Custom Cleaning StrategiesLet's suppose we intend to create a new cleaning strategy that removes certain columns.We would create a new class inheriting frombase.CleaningStrategy:frompandas_data_cleaner.baseimportCleaningStrategyclassRemoveColumns(CleaningStrategy):passWhen using this strategy, we need to know which column names to remove. We will therefore decide that, when using this class in theclean_datamethod, we need to provide aremove_columnsargument.To do this, we simply create a class attribute calledrequired_optionsand set it to["remove_columns"].We also will add some documentation to allow the end-user to receive some useful information when they runRemoveColumns.info().Our new strategy will now look like this:classRemoveColumns(CleaningStrategy):"""Removes columns from a dataframe.Required options:`remove_columns` - (_t.List[str]) A list of columns to remove."""required_options=["remove_columns"]Now, we need to create our cleaning method.
Once the cleaning method has been added, the class will look like the following:classRemoveColumns(CleaningStrategy):"""Removes columns from a dataframe.Required options:`remove_columns` - (List[str]) A list of columns to remove."""required_options=["remove_columns"]defclean(self):"""Executes the cleaning task."""self.dataframe.drop(self.remove_columns,axis=1,inplace=True)Let's discuss how this cleaning method works.
Firstly, whenever a user would use this strategy may run the following:clean_data(dataframe,[RemoveColumns],remove_columns=["id","status"])clean_datawill instantiate each cleaning strategy, in this case, justRemoveColumnsproviding the data frame as a required initial parameter as well as passing any keyword arguments to the function.Each strategy would then set both thedataframeand each keyword argument to the self object.This means that within the clean method, we would have access:self.dataframeself.remove_columns.If the command the user ran was instead:clean_data(dataframe,[RemoveColumns],remove_columns=["id","status"],foo="bar")Then within the clean method would have access:self.dataframeself.remove_columnsself.fooBy addingremove_columnsto therequired_optionslist, once this class is instantiated, we will be able to accessself.remove_columns.Now that we have built our cleaning strategy let's run it:dataframe=pd.DataFrame({"id":[1,2,3],"col1":[1,2,3],"col2":[1,2,3],"col3":[1,2,3],})dataframe=clean_data(dataframe,[RemoveColumns],remove_columns=["col1","col2"])print(dataframe)>>>pd.DataFrame({"id":[1,2,3],"col3":[1,2,3],})