Package: tinytimmy
Tiny TimmyA dead simple and easy to use Data Quality (DQ) tool built for Dataframes and Files with Python.Tiny Timmyuses the Python bindings for Polars aRustbased DataFrame tool.Support includes ...polarspandaspysparkcsvfilesparquetfilesBothdataframeandfilesupport. Simply "point and shoot."InstallationInstall Tiny Timmy withpippip install tinytimmyUsageCreate an instance of Tiny Timmy.specifysource_typepolarspandaspysparkcsvparquetspecify eitherfile_pathordataframefrom tinytimmy.tinytim import TinyTim
tm = TinyTim(source_type="csv", file_path="202306-divvy-tripdata.csv")Then call either the default checks or a custom check.results = tm.default_checks()
results = tm.run_custom_check(["{SQL filter}", "{SQL filter}"])You can pass Tiny Timmy adataframewhile specifying what type it is (pandas,polars,pyspark)
and ask fordefault_checks, also you can simply pass a file uri to acsvorparquetfile.You can also pass custom DQ checks as a list ofSQLstatements in a normalWHEREclause.Tiny Timmyreturns check results as aPolarsdataframeby default, you can
request the results as apandasorpysparkdataframe.results = tm.default_checks(return_as='pandas')For example.┌───────────────────────────────────┬─────────────┐
│ check_type                        ┆ check_value │
│ ---                               ┆ ---         │
│ str                               ┆ i64         │
╞═══════════════════════════════════╪═════════════╡
│ null_check_start_station_name     ┆ 978         │
│ null_check_start_station_id       ┆ 978         │
│ …                                 ┆ …           │
│ started_at_whitespace_count       ┆ 1000        │
│ ended_at_whitespace_count         ┆ 1000        │
│ start_station_name_whitespace_co… ┆ 22          │
│ end_station_name_whitespace_coun… ┆ 22          │Current functionality ...default_checks()check all columns fornullvaluescheck if dataset is distinct or contains duplicatescheck if columns have whitespacecheck for leading or trailing whitespacerun_custom_check(["{some SQL WHERE clause}"])Example UsageCSVsupport.from tinytimmy.tinytim import TinyTim
tm = TinyTim(source_type="csv", file_path="202306-divvy-tripdata.csv")
results = tm.default_checks()
>> Column start_station_name has 978 null values
>> Column start_station_id has 978 null values
>> Column end_station_name has 978 null values
>> Column end_station_id has 978 null values
>> Your dataset has 45 duplicatesPandassupport.from tinytimmy.tinytim import TinyTim
df = pd.read_csv("202306-divvy-tripdata.csv")
tm = TinyTim(source_type="pandas", dataframe=df)
results = tm.default_checks()
>> Column start_station_name has 978 null values
>> Column start_station_id has 978 null values
>> Column end_station_name has 978 null values
>> Column end_station_id has 978 null values
>> Your dataset has no duplicatesCustomData Quality checks are supported as alistofSQLbased formats.
They are given as they would appear in aWHEREclause.
You can pass one or more checks in the list.from tinytimmy.tinytim import TinyTim
tm = TinyTim(source_type="csv", file_path="202306-divvy-tripdata.csv")
tm.default_checks()
results = tm.run_custom_check(["start_station_name IS NULL", "end_station_name IS NULL"])
Column start_station_name has 978 null values
Column start_station_id has 978 null values
Column end_station_name has 978 null values
Column end_station_id has 978 null values
Your dataset has no duplicates
Column started_at has 1000 whitespace values
Column ended_at has 1000 whitespace values
Column start_station_name has 22 whitespace values
Column end_station_name has 22 whitespace values
No leading or trailing whitespace values found
shape: (10, 2)
┌───────────────────────────────────┬─────────────┐
│ check_type                        ┆ check_value │
│ ---                               ┆ ---         │
│ str                               ┆ i64         │
╞═══════════════════════════════════╪═════════════╡
│ null_check_start_station_name     ┆ 978         │
│ null_check_start_station_id       ┆ 978         │
│ null_check_end_station_name       ┆ 978         │
│ null_check_end_station_id         ┆ 978         │
│ …                                 ┆ …           │
└───────────────────────────────────┴─────────────┘
Your custom check start_station_name IS NULL found 978 records that match your filter statement
Your custom check end_station_name IS NULL found 978 records that match your filter statementTests / Local Setup / Contributions.To develop and work on TinyTimmy locally, aDockerimage anddocker-composeis provided.First, build the imagedocker build --tag=tinytimmy .To run the local unit tests run ...docker-compose up testTo simply work inside the Docker container run ...docker run -it tinytimmy /bin/bash