Package: pandandic
pandandicpandandic is a library for documenting dataset schemas in code, by inheriting from a base class and assigning attributes for columns and column sets.Installationpip install pandandicorpip install pandandic[extras]poetry add pandandicorpoetry add "pandandic[extras]"ExtrasparquetavroextrasprovidesparquetandavroallprovidesparquetandavroWhat Problem Does It Solve?Consider a project that reads data from several datasets, performs some preprocessing, runs a model and returns a result.
The preprocessing must act on certain columns and so the team rightfully add constants in order to perform slicing on the input dataframes.
Two of these datasets share a column name.
One of the datasets consists of time series data, and each time the dataset is refreshed the number of columns changes.
This scenario presents several challenges with how to structure the processing logic in a clear and adaptable manner whilst maintaining clear ownership.
Here is howpandandichelps:Schema ownership: withpandandic, each schema has a corresponding class.Shared variables: withpandandic, there are no shared constants. EachBaseFramesubclass is responsible for its own schema.Dynamic groups: withpandandicit is possible to define a set of columns with regular expressions. This schema will match dynamically on the data it is applied to, yet can still be accessed like an attribute.Group processing: withpandandicit is possible to define custom groups such as "all numeric", "all time-series", in order to easily apply processing tasks to groups of data in a self-documenting fashion.Other Things It DoesWrapsparquetreading:pip install pandandic[parquet],poetry add "pandandic[parquet]"Wrapsavroreading:pip install pandandic[avro],poetry add "pandandic[avro]"For both:pip install pandandic[all],poetry add "pandandic[all]"Wraps excel reading, although there are no extras configured for this due to the various output formats of excel and different packages providing them.What Doesn't It Do?Validation, save for what is built in to pandas. For validation of defined types, please see other libraries such as pandera, dataenforce, strictly-typed-pandas (apologies for any I have missed).Appending columns: if columns are appended to the object after callingread_xorfrom_dfthat should be captured by aColumnSet, they won't be captured. This can be solved by transforming to a dataframe and back again withto_dfandfrom_dfrespectively.Dask: although support may be added in future.Worked ExamplesBasicexamples/basic.csvfoobarbaza1oneb2twoc3threeexamples/basic_usage.pyfrompandandicimportBaseFrame,ColumnclassFooFrame(BaseFrame):"""Each column set below will be read with the given type. Columns can be accessed like attributes to return Seriesslices in the usual way."""foo=Column(type=str)bar=Column(type=int)data=FooFrame().read_csv("basic.csv")print(data.foo)print(data.bar)Intermediateexamples/intermediate.csvdatetemperature-0temperature-1temperature-2temperature-3temperature-4temperature-501/01/200123222120191802/01/200124232221201903/01/200125242322212004/01/2001262524232221examples/intermediate_usage.pyimportdatetimefrompandandicimportBaseFrame,Column,ColumnSetclassTemperatureFrame(BaseFrame):"""A ColumnGroup can use a list of column names or a regex to specify multiple columns at once.An exception is raised if members overlap, unless greedy_column_groups is set to True.In that case, the first member to match is assigned that group.A column group can be accessed like an attribute to provide a DataFrame view."""date=Column(type=datetime.date)temperature=ColumnSet(type=float,members=["temperature-\d+"],regex=True)df=TemperatureFrame().read_csv("intermediate.csv")df.set_index(TemperatureFrame.date.column_name,inplace=True)# name attribute also works here, but column_name is recommendedprint(df.temperature)As can be seen in the intermediate example, it is possible to access the defined TemperatureFrameColumndate from the class (notinstantiated object), and call.nameto refer to the constant, which in this case returns "date", the name of the attribute.This can be done as well with non-regexColumnSet, in that case accessing the.membersattribute.Advancedexamples/advanced.csvdatetemperature-0temperature-1temperature-2temperature-3door-open-0door-open-1door-open-2refcomment01/01/200123222120FalseFalseFalse75first observation02/01/200124232221FalseTrueFalse7603/01/200125242322TrueFalseFalse77left the door open04/01/200126252423FalseFalseTrue78final observationimportdatetimefrompandandicimportBaseFrame,Column,ColumnSet,ColumnGroupclassAdvancedFrame(BaseFrame):"""A Group can be used to group together multiple column groups and columns.It can be accessed like an attribute to provide a dataframe view."""date=Column(type=datetime.date)temperature=ColumnSet(type=float,members=["temperature-\d+"],regex=True)door_open=ColumnSet(type=bool,members=["door-open-0","door-open-1","door-open-2"],regex=False)ref=Column(type=int)comment=Column(type=str)numerical=ColumnGroup(members=[temperature,ref])time_series=ColumnGroup(members=[temperature,door_open])df=AdvancedFrame().read_csv("advanced.csv")df.set_index(AdvancedFrame.date.column_name,inplace=True)# name attribute also works here, but column_name is recommendedprint(df.time_series)ColumnGroupandColumnSetattributes can be accessed on the instantiated object, and will return aDataFrameview of their members.# examples/expert_usage.pyimportdatetimefrompandandicimportBaseFrame,Column,ColumnSet,ColumnGroup,DefinedLaterclassExpertFrame(BaseFrame):"""Aliasing can be used to dynamically set columns or column set members at runtime."""date=Column(type=datetime.date,alias=DefinedLater)metadata=ColumnSet(members=DefinedLater)temperature=ColumnSet(type=float,members=["temperature-\d+"],regex=True)door_open=ColumnSet(type=bool,members=["door-open-0","door-open-1","door-open-2"],regex=False)time_series=ColumnGroup(members=[temperature,door_open])# anything DefinedLater MUST be set before ExpertFrame reads or accesses a Column or ColumnSet via attributeExpertFrame.date.alias="date"ExpertFrame.metadata.members=["comment","ref"]df=ExpertFrame().read_csv("advanced.csv")df.set_index(ExpertFrame.date.column_name,inplace=True)# now sets index with the defined aliasprint(df.metadata)Columnalias can be set asDefinedLaterto clearly document that it is set dynamically at runtime.
The same is possible forColumnSetmembers. This has the benefit of adding a runtime check that the alias or members are set before being used.Warning: If aColumnalias is set, it will be usedregardlessof whether it exists in the data or not.Class DiagramclassDiagram
    
    DataFrame <|-- BaseFrame
    class BaseFrame {
        +int enforce_types
        +int enforce_columns
        +int allow_extra_columns
        +int greedy_column_sets
        +with_enforced_types()
        +with_enforced_columns()
        +with_allowed_extra_columns()
        +with_greedy_column_sets()
        +read_csv()
        +read_excel()
        +read_parquet()
        +read_avro()
        +from_df()
        +to_df()
        +read_csv_columns()
        +read_excel_columns()
        +read_parquet_columns()
        +read_avro_columns()
    }
    BaseFrame o-- Column
    class Column {
        +type
    }
    BaseFrame o-- ColumnSet
    class ColumnSet {
        +type
        +members
    }
    BaseFrame o-- ColumnGroup
    class ColumnGroup {
        +type
        +members
    }
    ColumnGroup *--	ColumnSet
    ColumnGroup *--	ColumnDefined Behavioursenforce_typesIf set to True (default), the types set inColumnandColumnSetattributes are enforced at read time (csv, excel) or cast after reading (parquet, avro, df).
No validation is done, so errorswillbe thrown by pandas if the data cannot be coerced to the schema.enforce_columnsIf set to True (default), definedColumnandColumnSetattributes define the mandatory columns of the frame.
Errorswillbe thrown by pandas if expected columns do not exist in the data.A regexColumnSetwill match only existing columns, and will not error if a match doesn't exist.allow_extra_columnsIf set to False (default), any extra columns will be removed.If set to True (not default), they will remain.greedy_column_setsIf set to False (default), there must be no overlap inColumnandColumnSetmembers.
If there is an overlap, aColumnSetExceptionwill be raised.If set to True (not default), aColumnSetwill "consume" columns, they will belong to thatColumnSetand inherit its defined type, and the system will not raise aColumnSetException.