Package: daffy
DAFFY DataFrame Column ValidatorDescriptionIn projects using Pandas, it's very common to have functions that take Pandas DataFrames as input or produce them as output.
It's hard to figure out quickly what these DataFrames contain. This library offers simple decorators to annotate your functions
so that they document themselves and that documentation is kept up-to-date by validating the input and output on runtime.For example,@df_in(columns=["Brand","Price"])# the function expects a DataFrame as input parameter with columns Brand and Price@df_out(columns=["Brand","Price"])# the function will return a DataFrame with columns Brand and Pricedeffilter_cars(car_df):# before this code is executed, the input DataFrame is validated according to the above decorator# filter some cars..returnfiltered_cars_dfTable of ContentsInstallationUsageContributingTestsLicenseChangelogInstallationInstall with your favorite Python dependency manager likepipinstalldaffyUsageStart by importing the needed decorators:fromdaffyimportdf_in,df_outTo check a DataFrame input to a function, annotate the function with@df_in. For example the following function expects to get
a DataFrame with columnsBrandandPrice:@df_in(columns=["Brand","Price"])defprocess_cars(car_df):# do stuff with carsIf your function takes multiple arguments, specify the field to be checked with it'sname:@df_in(name="car_df",columns=["Brand","Price"])defprocess_cars(year,style,car_df):# do stuff with carsYou can also check columns of multiple arguments if you specify the names@df_in(name="car_df",columns=["Brand","Price"])@df_in(name="brand_df",columns=["Brand","BrandName"])defprocess_cars(car_df,brand_df):# do stuff with carsTo check that a function returns a DataFrame with specific columns, use@df_outdecorator:@df_out(columns=["Brand","Price"])defget_all_cars():# get those carsreturnall_cars_dfIn case one of the listed columns is missing from the DataFrame, a helpful assertion error is thrown:AssertionError("Column Price missing from DataFrame. Got columns: ['Brand']")To check both input and output, just use both annotations on the same function:@df_in(columns=["Brand","Price"])@df_out(columns=["Brand","Price"])deffilter_cars(car_df):# filter some carsreturnfiltered_cars_dfIf you want to also check the data types of each column, you can replace the column array:columns=["Brand","Price"]with a dict:columns={"Brand":"object","Price":"int64"}This will not only check that the specified columns are found from the DataFrame but also that theirdtypeis the expected.
In case of a wrongdtype, an error message similar to following will explain the mismatch:AssertionError("Column Price has wrong dtype. Was int64, expected float64")You can enable strict-mode for both@df_inand@df_out. This will raise an error if the DataFrame contains columns
not defined in the annotation:@df_in(columns=["Brand"],strict=True)defprocess_cars(car_df):# do stuff with carswill, whencar_dfcontains columns["Brand", "Price"]raise an error:AssertionError: DataFrame contained unexpected column(s): PriceTo quickly check what the incoming and outgoing dataframes contain, you can add a@df_logannotation to the function. For
example adding@df_logto the abovefilter_carsfunction will product log lines:Function filter_cars parameters contained a DataFrame: columns: ['Brand', 'Price']
Function filter_cars returned a DataFrame: columns: ['Brand', 'Price']or with@df_log(include_dtypes=True)you get:Function filter_cars parameters contained a DataFrame: columns: ['Brand', 'Price'] with dtypes ['object', 'int64']
Function filter_cars returned a DataFrame: columns: ['Brand', 'Price'] with dtypes ['object', 'int64']ContributingContributions are accepted. Include tests in PR's.DevelopmentTo run the tests, clone the repository, install dependencies with Poetry and run tests with PyTest:poetryinstall
poetryshell
pytestTo enable linting on each commit, runpre-commit install. After that, your every commit will be checked withisort,blackandflake8.LicenseMITChangelog0.7.0Support Pandas 2.xDrop support for Python 3.7 and 3.8Build and test with Python 3.12 also0.6.0Make checking columns of multiple function parameters work also with positional arguments (thanks @latvanii)0.5.0Addedstrictparameter for@df_inand@df_out0.4.2Added docstrings for the decoratorsFix import of@df_log0.4.1Addinclude_dtypesparameter for@df_log.Fix handling of empty signature with@df_in.0.4.0Added@df_logfor logging.Improved assertion messages.0.3.0Added type hints.0.2.1Added Pypi classifiers.0.2.0Fixed decorator usage.Added functools wraps.0.1.0Initial release.