Package: dictgest
DictGest - Python Dictionary IngestionDictGest - Python Dictionary IngestionDescriptionExamplesExample 1: Trivial Example - Handling Extra parametersExample 2: Data mapping renaming & reroutingExample 3: Data type enforcingExample 4: Custom Data extraction/conversion for a specific fieldExample 5: Custom Data conversion for a specific typeExample 6: Populating the same structure from multiple different dict formats (multiple APIs)Example 8: Populating from a 2D TableTransposing dataMapping one table row to target typeInstallingContributingSupportLicenseAcknowledgementsDescriptionWhen interacting with external REST APIs or with external configuration files we usually do not have control
over the received data structure/format.DictGestmakes ingesting dictionary data into python objects(dataclasss objects included) easy when the dictionary data doesn't match 1 to 1 with the Python class:The dictionary might have extra fields that are of no interestThe keys names in the dictionary do not match the class attribute namesThe structure of nested dictionaries does not match the class structureThe data types in the dictionary do not match data types of the target classExamplesExample 1: Trivial Example - Handling Extra parametersThe first most basic and trivial example is ingesting a dictionary that has extra data not of interestfromdictgestimportfrom_dictcar=from_dict(Car,dict_data)Example 2: Data mapping renaming & reroutingThe keys names in the source dictionary might not match the destionation class attribute names.
Also the source dictionary might have a nested structure different than our desired structure.fromtypingimportAnnotatedfromdataclassesimportdataclassfromdictgestimportfrom_dict,Patharticle=from_dict(Article,news_api_data)meta=from_dict(ArticleMeta,news_api_data)stats=from_dict(ArticleStats,news_api_data)The full working example can be found in theexamples folderThere can be cases where Annotating the type hints of the target class is not desired by the user or when mapping to multiple APIs might be required.
For these cases look at examples 6 & 7 for an alternate solution.Example 3: Data type enforcingSometimes the data coming from external sources might have different datatypes than what we desire.dictgencan do type conversion for you.fromdataclassesimportdataclassfromdictgestimportfrom_dict,typecast@typecast# Makes the class type convertable when encountered as typing hint@dataclass# The dataclass is just an example, it could have an normal classclassMeasurment:temp:floathumidity:floatclassSensor:def__init__(self,name:str,location:str,uptime:float,readings:list[Measurment]):...The conversions shown above were enabled by setting the@typecastdecorator for the targetted classes.The full working example can be found in theexamples folderExample 4: Custom Data extraction/conversion for a specific fieldSometimes we might want to apply custom transforms to some fields when extracting the data from the dictionary.
In this example we want to read the total number of votes, but in the dictionary source we only have two partial values: the positive and negative number of votes.We apply a custom transform to get our desired data, using theextractorargument ofdictgest.PathfromtypingimportAnnotatedfromdictgestimportPath,from_dictdefextract_votes(data):# creating a new value from two individual fields and converting themreturnint(data["positive"])+int(data["negative"])classVotes:def__init__(self,title,total_votes:Annotated[int,Path("details/votes",extractor=extract_votes)],):...article_data={"title":"Python 4.0 will...","details":{"votes":{"positive":"245","negative":"30"}},}votes=from_dict(Votes,article_data)The full working example can be found in theexamples folderExample 5: Custom Data conversion for a specific typeIn some cases we might want to employ a custom conversion for a certain datatype.fromdataclassesimportdataclassfromdictgestimportdefault_convertor,from_dict# Get any already registered bool convertordefault_bool_conv=default_convertor.get_convertor(bool)# create a custom converterdefcustom_bool_conv(val):ifval=="oups":returnFalse# Let the other cases be treated as beforereturndefault_bool_conv(val)# register the custom converter for booldefault_convertor.register(bool,custom_bool_conv)@dataclassclassResult:finished:boolnotified:boolresult=from_dict(Result,{"finished":True,"notified":"oups"})print(result)Example 6: Populating the same structure from multiple different dict formats (multiple APIs)There are cases where you might read information from multiple heterogenous APIs and you might want to convert them all to the same structure.Previously we have annotated fields( using typing.Annotation hint ) with Path eg:name: Annotated[str, Path('article')]. This works well for a single conversion mapping.For this current scenario we are going to decouple the class from the Routing.Previously single mapping scenario:@dataclassclassArticle:author:strtitle:Annotated[str,Path("headline")]content:Annotated[str,Path("details/content")]But now we have 2 API news sourcesdata_from_api1={"author":"H.O. Ward""headline":"Top 10 Python extensions","other_fields":...,"details":{"content":"Here are the top 10...","other_fields":...}}data_from_api2={"author":"G.O. Gu""news_title":"Vscode gets a new facelift","other_fields":...,"full_article":"Yesterday a new version ...",}}We are going to usedictgest.Routeto define multiple standalone routes.Our previous example becomes:@dataclassclassArticle:author:strtitle:str# Path annotations are decoupledcontent:str# Routing equivalent to previous examplearticle_api1=Route(title="headline",content="details/content")# New Routing for a new dict structurearticle_api2=Route(title="news_title",content="full_article")article1=from_dict(Article,data_from_api1,routing=article_api1)article2=from_dict(Article,data_from_api2,routing=article_api2)The full working example can be found in theexamples folderExample 8: Populating from a 2D TableSometimes when querying databases/external APIs the reponse might be in a form of a 2D Table (a list of lists)header=["humidity","temperatures","timestamps"]table_data=[[0.4,7.4,"1Dec2022"],...[0.6,5.4,"21Dec2022"],]And our desired target structure could look like this:@dataclassclassSenzorData:timestamps:list[datetime.datetime]temperatures:list[float]humidity:list[float]In this example we would like each data column to be  treated as a field of the target type.
To ingest our data into our target type we can usetable_to_itemfollowing:importdictgestasdgresult=dg.table_to_item(SenzorData,table_data,header)Transposing dataThe operation can be also be performed row wise by using thetranspose = Trueflag.
So givenheader=["humidity","temperatures","timestamps"]table_data_transposed=[# rows are switched with columns[0.4,...,0.6],[5.4,...,7.4]["1Dec2022",...,"21Dec2022"],]result=dg.table_to_item(SenzorData,table_data_transposed,header,transpose=True)Mapping one table row to target typeWe might not want to convert the whole table into a specific data type but map each row/column to a specific datatype.#Unlike SenzorData defined previously SenzorDataPoint holds information only for a single specific time.@dataclassclassSenzorDataPoint:timestamp:datetime.datetimetemperature:floathumidity:floatFor thistable_to_itemscan be usedresult = dg.table_to_item(SenzorDataPoint, table_data, header)

    result = dg.table_to_item(SenzorDataPoint, table_data_transposed, header, transpose=True)Installingpip install dictgestContributingFirst off, thanks for taking the time to contribute! Contributions are what makes the open-source community such an amazing place to learn, inspire, and create. Any contributions you make will benefit everybody else and aregreatly appreciated.SupportReach out to the maintainer at one of the following places:Github issuesDiscordLicenseThis project is licensed under theMIT license. Feel free to edit and distribute this template as you like.SeeLICENSEfor more information.AcknowledgementsThanksDan Oneatafor the discussions related to usecases and API.