Package: shmessy
ShmessyIf your data is messy - Use Shmessy!Shmessy designed to deal with messy pandas dataframes.
We all knows the frustrating times when we as analysts or data-engineers should handle messy dataframe and analyze them by ourselves.The goal of this tiny tool is to identify the physical / logical data type for each Dataframe column.
It based on fast validators that will validate the data (Based on a sample) against regex / pydantic types or any additional validation function that you want to implement.As you understand, this tool was designed to deal with dirty data,
ideally developed for Dataframes generated from CSV / Flat files or any source that doesn't contain strict schema.InstallationpipinstallshmessyUsageYou have two ways to use this toolIdentify the Dataframe schemaimportpandasaspdfromshmessyimportShmessydf=pd.read_csv('/tmp/file.csv')inferred_schema=Shmessy().infer_schema(df)Output (inferred_schema dump):{"infer_duration_ms":12,"columns":[{"field_name":"id","source_type":"Integer","inferred_type":"Integer"},{"field_name":"email_value","source_type":"String","inferred_type":"Email"},{"field_name":"date_value","source_type":"String","inferred_type":"Date","inferred_pattern":"%d-%m-%Y"},{"field_name":"datetime_value","source_type":"String","inferred_type":"Datetime","inferred_pattern":"%Y/%m/%d %H:%M:%S"},{"field_name":"yes_no_data","source_type":"String","inferred_type":"Boolean","inferred_pattern":["YES","NO"]},{"field_name":"unix_value","source_type":"Integer","inferred_type":"UnixTimestamp","inferred_pattern":"ms"},{"field_name":"ip_value","source_type":"String","inferred_type":"IPv4"}]}Identify and fix Pandas DataframeThis piece of code will change the column types of the input Dataframe according to Messy infer.importpandasaspdfromshmessyimportShmessydf=pd.read_csv('/tmp/file.csv')fixed_df=Shmessy().fix_schema(df)Original DataframeFixed DataframeRead Messy CSV filefromshmessyimportShmessydf=Shmessy().read_csv('/tmp/file.csv')Original fileFixed DataframeAPIConstructorshmessy=Shmessy(sample_size:Optional[int]=1000,reader_encoding:Optional[str]="UTF-8",locale_formatter:Optional[str]="en_US",use_random_sample:Optional[bool]=True,ignore_virtual_types:Optional[bool]=False,max_columns_num:Optional[int]=500)read_csvshmessy.read_csv(filepath_or_buffer:str|TextIO|BinaryIO,use_sniffer:Optional[bool]=True,# Use python sniffer to identify the dialect (seperator / quote-char / etc...)fixed_schema:Optional[ShmessySchema]=None,# Fix the given CSV according to this schemafix_column_names:Optional[bool]=False,# Replace non-alphabetic/numeric chars with underscorefallback_to_string:Optional[bool]=False,# Fallback to string in case of casting exceptionfallback_to_null:Optional[bool]=False,# Fallback to null in case of casting exception)->DataFrameinfer_schemashmessy.infer_schema(df:Dataframe# Input dataframe)->ShmessySchemafix_schemashmessy.fix_schema(df:Dataframe,fix_column_names:Optional[bool]=False,# Replace non-alphabetic/numeric chars with underscorefixed_schema:Optional[ShmessySchema]=None,# Fix the given DF according to this schemafallback_to_string:Optional[bool]=False,# Fallback to string in case of casting exceptionfallback_to_null:Optional[bool]=False,# Fallback to null in case of casting exception)->DataFrameget_inferred_schemashmessy.get_inferred_schema()->ShmessySchema