Package: pandas-to-pydantic
pandas-to-pydanticWARNING: Library is currently unstable and in beta.This library provides functions for converting Pandas Dataframes to Pydantic Models. This allows you to easily transform data in a table-like format into a json-like format. Pydantic Model annotations are matched with Pandas Dataframe columns. Supports models nested in lists.Table of ContentsInstallationLicenseExample 1dataframe_to_pydanticExample 2Installationpip install pandas-to-pydanticExample 1This example will show how to convert data from a flat structure (.csv file, pandas dataframe) to a hierarchical structure (json file, pydantic models)Example Book DataBookIDTitleAuthorNameGenrePublishedYear1Harry Potter and the Philosopher's StoneJ.K. RowlingFantasy19972Harry Potter and the Chamber of SecretsJ.K. RowlingFantasy199831984George OrwellDystopian Fiction19494Animal FarmGeorge OrwellPolitical Satire19455Pride and PrejudiceJane AustenRomance18137Murder on the Orient ExpressAgatha ChristieMystery19349Adventures of Huckleberry FinnMark TwainAdventure188410The Adventures of Tom SawyerMark TwainAdventure187611The HobbitJ.R.R. TolkienFantasy193712The Lord of the RingsJ.R.R. TolkienFantasy1954importpandasaspdfrompydanticimportBaseModelfrompandas_to_pydanticimportdataframe_to_pydantic# Declare pydantic modelsclassBook(BaseModel):BookID:intTitle:strAuthorName:strGenre:strPublishedYear:int# Update this to your your file pathbook_data=pd.read_csv(FILE_PATH)# Convert pandas dataframe to a pydantic root modelbook_list_root=dataframe_to_pydantic(book_data,Book)dataframe_to_pydanticreturns a pydanticRootModel. Data can be accessed using its attributes and methods.https://docs.pydantic.dev/latest/api/root_model/For example:# Access data as a list of pydantic modelsbook_list_root.rootReturns (output shortened):[Book(BookID=1, Title="Harry Potter and the Philosopher's Stone", AuthorName='J.K. Rowling', Genre='Fantasy', PublishedYear=1997),
Book(BookID=2, Title='Harry Potter and the Chamber of Secrets', AuthorName='J.K. Rowling', Genre='Fantasy', PublishedYear=1998),
Book(BookID=3, Title='1984', AuthorName='George Orwell', Genre='Dystopian Fiction', PublishedYear=1949),
...]For example:# Access data as a list of dictbook_list_root.model_dump()Returns (output shortened):[{'BookID': 1,
  'Title': "Harry Potter and the Philosopher's Stone",
  'AuthorName': 'J.K. Rowling',
  'Genre': 'Fantasy',
  'PublishedYear': 1997},
 {'BookID': 2,
  'Title': 'Harry Potter and the Chamber of Secrets',
  'AuthorName': 'J.K. Rowling',
  'Genre': 'Fantasy',
  'PublishedYear': 1998},
 {'BookID': 3,
  'Title': '1984',
  'AuthorName': 'George Orwell',
  'Genre': 'Dystopian Fiction',
  'PublishedYear': 1949},
...]Example 2In this example, Pydantic models are nested using thelisttype annotation. When there are multiple layers of nesting, unique id fields should be provided for each list field with a child model usingid_column_map.Here, the unique id column for theGenremodel isGenre, and the unique id column for theAuthormodel isAuthorName. Keys inid_column_mapcan be the model name or field name. Values inid_column_mapare the unique column name.For example:classBook(BaseModel):BookID:intTitle:strPublishedYear:intclassAuthor(BaseModel):AuthorName:strBookList:list[Book]classGenre(BaseModel):Genre:strAuthorList:list[Author]dataframe_to_pydantic(data=bookData,model=Genre,id_column_map={"Genre":"Genre","AuthorList":"AuthorName"},).model_dump()Returns (output shortened)[{'Genre': 'Fantasy',
  'AuthorList': [{'AuthorName': 'J.K. Rowling',
    'BookList': [{'BookID': 1,
      'Title': "Harry Potter and the Philosopher's Stone",
      'PublishedYear': 1997},
     {'BookID': 2,
      'Title': 'Harry Potter and the Chamber of Secrets',
      'PublishedYear': 1998}]},
   {'AuthorName': 'J.R.R. Tolkien',
    'BookList': [{'BookID': 11, 'Title': 'The Hobbit', 'PublishedYear': 1937},
     {'BookID': 12,
      'Title': 'The Lord of the Rings',
      'PublishedYear': 1954}]}]},
 {'Genre': 'Dystopian Fiction',
  'AuthorList': [{'AuthorName': 'George Orwell',
    'BookList': [{'BookID': 3, 'Title': '1984', 'PublishedYear': 1949}]}]},
...]dataframe_to_pydanticArgsdata (pandas.DataFrame)Dataframe with columns matching fields in the pydantic modelWhen the pydantic model includes nested models, it is assumed that the first column is unique. SeeExample 2model (pydantic._internal._model_construction.ModelMetaClass)Accepts classes created with pydantic.BaseModelSupports nested models in listsAnnotation names must match columns in the dataframeid_column_map(dict[str,str])Required when nesting Pydantic modelsEach key corresponds with field name or model nameEach value corresponds with the unique id column for the nested Pydantic modelFor the parent level model, use the model name as keyReturnsmodel_list (pydantic.RootModel)Pydantic root model created as a list of the input modelhttps://docs.pydantic.dev/latest/api/root_model/Advanced ExampleThis example uses a larger data set with additional nesting.Example Library DataimportpandasaspdfrompydanticimportBaseModelfrompandas_to_pydanticimportdataframe_to_pydantic# Declare pydantic modelsclassLibaryDetail(BaseModel):LibraryName:strLocation:strEstablishedYear:intBookCollectionSize:intclassAuthor(BaseModel):AuthorID:intAuthorName:strAuthorBirthdate:strclassBook(BaseModel):BookID:intTitle:strGenre:strPublishedYear:intclassLibrary(BaseModel):LibraryID:intDetail:LibaryDetailAuthorList:list[Author]BookList:list[Book]# Input data is a pandas dataframedata=pd.read_csv(FILE_PATH)# Convert pandas dataframe to a pydantic root modellibrary_list_root=dataframe_to_pydantic(data,Library,{"Library":"LibraryID","BookList":"BookID","AuthorList":"AuthorID",},)# Access data as a list of pydantic modelslibrary_list_root.root# Access data as a list of dictlibrary_list_root.model_dump()