Package: ntv-pandas
NTV-pandas : A tabular analyzer and a semantic, compact and reversible JSON-pandas converterFor more information, see theuser guideor thegithub repository.NTV-pandas is referenced in thepandas ecosystem.Why a NTV-pandas converter ?pandas provide JSON converter but three limitations are present:the JSON-pandas converter take into account few data types,the JSON-pandas converter is not always reversible (conversion round trip)external data types (e.g. TableSchema types) are not includedpandas does not have a tool for analyzing tabular structures and detecting integrity errorsmain featuresThe NTV-pandas converter uses thesemantic NTV formatto include a large set of data types in a JSON representation.The converter integrates:all the pandasdtypeand the data-type associated to a JSON representation,an always reversible conversion,a full compatibility withTable Schema specificationThe NTV-pandas analyzer uses theTAB-analysistool to analyze and measure the relationships between Fields in DataFrame and theTAB-datasetto identify integrity errors (example).NTV-pandas was developped originally in thejson-NTV projectconverter exampleIn the example below, a DataFrame with multiple data types is converted to JSON (first to NTV format and then to Table Schema format).The DataFrame resulting from these JSON conversions are identical to the initial DataFrame (reversibility).With the existing JSON interface, these conversions are not possible.Data example:In[1]:fromshapely.geometryimportPointfromdatetimeimportdateimportpandasaspdimportntv_pandasasnpdIn[2]:data={'index':[100,200,300,400,500],'dates::date':[date(1964,1,1),date(1985,2,5),date(2022,1,21),date(1964,1,1),date(1985,2,5)],'value':[10,10,20,20,30],'value32':pd.Series([12,12,22,22,32],dtype='int32'),'res':[10,20,30,10,20],'coord::point':[Point(1,2),Point(3,4),Point(5,6),Point(7,8),Point(3,4)],'names':pd.Series(['john','eric','judith','mila','hector'],dtype='string'),'unique':True}In[3]:df=pd.DataFrame(data).set_index('index')df.index.name=NoneIn[4]:dfOut[4]:dates::datevaluevalue32rescoord::pointnamesunique1001964-01-01101210POINT(12)johnTrue2001985-02-05101220POINT(34)ericTrue3002022-01-21202230POINT(56)judithTrue4001964-01-01202210POINT(78)milaTrue5001985-02-05303220POINT(34)hectorTrueJSON-NTV representation:In[5]:df_to_json=df.npd.to_json()pprint(df_to_json,compact=True,width=120,sort_dicts=False)Out[5]:{':tab':{'index':[100,200,300,400,500],'dates::date':['1964-01-01','1985-02-05','2022-01-21','1964-01-01','1985-02-05'],'value':[10,10,20,20,30],'value32::int32':[12,12,22,22,32],'res':[10,20,30,10,20],'coord::point':[[1.0,2.0],[3.0,4.0],[5.0,6.0],[7.0,8.0],[3.0,4.0]],'names::string':['john','eric','judith','mila','hector'],'unique':True}}Reversibility:In[6]:print(npd.read_json(df_to_json).equals(df))Out[6]:TrueTable Schema representation:In[7]:df_to_table=df.npd.to_json(table=True)pprint(df_to_table['data'][0],sort_dicts=False)Out[7]:{'index':100,'dates':'1964-01-01','value':10,'value32':12,'res':10,'coord':[1.0,2.0],'names':'john','unique':True}In[8]:pprint(df_to_table['schema'],sort_dicts=False)Out[8]:{'fields':[{'name':'index','type':'integer'},{'name':'dates','type':'date'},{'name':'value','type':'integer'},{'name':'value32','type':'integer','format':'int32'},{'name':'res','type':'integer'},{'name':'coord','type':'geopoint','format':'array'},{'name':'names','type':'string'},{'name':'unique','type':'boolean'}],'primaryKey':['index'],'pandas_version':'1.4.0'}Reversibility:In[9]:print(npd.read_json(df_to_table).equals(df))Out[9]:True