Package: tafra
Thetafrabegan life as a thought experiment: how could we reduce the idea
of a dataframe (as expressed in libraries likepandasor languages
like R) to its useful essence, while carving away the cruft?
Theoriginal proof of conceptstopped at “group by”.This library expands on the proof of concept to produce a practically
usefultafra, which we hope you may find to be a helpful lightweight
substitute for certain uses ofpandas.Atafrais, more-or-less, a set of namedcolumnsordimensions.
Each of these is a typednumpyarray of consistent length, representing
the values for each column byrows.The library provides lightweight syntax for manipulating rows and columns,
support for managing data types, iterators for rows and sub-frames,pandas-like “transform” support and conversion frompandasDataframes,
and SQL-style “group by” and join operations.TafraTafraAggregationsUnion,GroupBy,Transform,IterateBy,InnerJoin,LeftJoin,CrossJoinAggregation Helpersunion,union_inplace,group_by,transform,iterate_by,inner_join,left_join,cross_joinConstructorsas_tafra,from_dataframe,from_series,from_recordsSQL Readersread_sql,read_sql_chunksDestructorsto_records,to_list,to_tuple,to_array,to_pandasPropertiesrows,columns,data,dtypes,size,ndim,shapeIter Methodsiterrows,itertuples,itercolsFunctional Methodsrow_map,tuple_map,col_map,pipeDict-like Methodskeys,values,items,get,update,update_inplace,update_dtypes,update_dtypes_inplaceOther Helper Methodsselect,head,copy,rename,rename_inplace,coalesce,coalesce_inplace,_coalesce_dtypes,delete,delete_inplacePrinter Methodspprint,pformat,to_htmlIndexing Methods_slice,_index,_ndindexGetting StartedInstall the library withpip:pipinstalltafraA short example>>>fromtafraimportTafra>>>t=Tafra({...'x':np.array([1,2,3,4]),...'y':np.array(['one','two','one','two'],dtype='object'),...})>>>t.pformat()Tafra(data={'x':array([1,2,3,4]),'y':array(['one','two','one','two'])},dtypes={'x':'int','y':'object'},rows=4)>>>print('List:','\n',t.to_list())List:[array([1,2,3,4]),array(['one','two','one','two'],dtype=object)]>>>print('Records:','\n',tuple(t.to_records()))Records:((1,'one'),(2,'two'),(3,'one'),(4,'two'))>>>gb=t.group_by(...['y'],{'x':sum}...)>>>print('Group By:','\n',gb.pformat())GroupBy:Tafra(data={'x':array([4,6]),'y':array(['one','two'])},dtypes={'x':'int','y':'object'},rows=2)FlexibilityHave some code that works withpandas, or just a way of doing things
that you prefer?tafrais flexible:>>>df=pd.DataFrame(np.c_[...np.array([1,2,3,4]),...np.array(['one','two','one','two'])...],columns=['x','y'])>>>t=Tafra.from_dataframe(df)And going back is just as simple:>>>df=pd.DataFrame(t.data)TimingsIn this case, lightweight also means performant. Beyond any additional
features added to the library,tafrashould provide the necessary
base for organizing data structures for numerical processing. One of the
most important aspects is fast access to the data itself. By minimizing
abstraction to access the underlyingnumpyarrays,tafraprovides
an order of magnitude increase in performance.Import noteIf you assign directly to theTafra.dataorTafra._dataattributes, youmustcallTafra._coalesce_dtypesafterwards in order to ensure the typing is consistent.Construct aTafraand aDataFrame:>>>tf=Tafra({...'x':np.array([1.,2.,3.,4.,5.,6.]),...'y':np.array(['one','two','one','two','one','two'],dtype='object'),...'z':np.array([0,0,0,1,1,1])...})>>>df=pd.DataFrame(t.data)Read OperationsDirect access:>>>%timemitx=t._data['x']55.3nsÂ±5.64nsperloop(meanÂ±std.dev.of7runs,10000000loopseach)Indirect with some penalty to supportTafraslicing andnumpy’s
advanced indexing:>>>%timemitx=t['x']219nsÂ±71.6nsperloop(meanÂ±std.dev.of7runs,1000000loopseach)pandastiming:>>>%timemitx=df['x']1.55ÂµsÂ±105nsperloop(meanÂ±std.dev.of7runs,1000000loopseach)This is the fastest methed for accessing the numpy array among alternatives ofdf.values(),df.to_numpy(), anddf.loc[].Assignment OperationsDirect access is not recommended as it avoids the validation steps, but it
does provide fast access to the data attribute:>>>x=np.arange(6)>>>%timeittf._data['x']=x65nsÂ±5.55nsperloop(meanÂ±std.dev.of7runs,10000000loopseach)Indidrect access has a performance penalty due to the validation checks to
ensure consistency of thetafra:>>>%timeittf['x']=x7.39ÂµsÂ±950nsperloop(meanÂ±std.dev.of7runs,100000loopseach)Even so, there is considerable performance improvement overpandas.pandastiming:>>>%timeitdf['x']=x47.8ÂµsÂ±3.53Âµsperloop(meanÂ±std.dev.of7runs,10000loopseach)Grouping Operationstafraalso excels at aggregation methods, the primary of which are a
SQL-likeGROUP BYand the split-apply-combine equivalent to a SQL-likeGROUP BYfollowing by aLEFT JOINback to the original table.>>>%timeittf.group_by(['y','z'],{'x':sum})138ÂµsÂ±4.03Âµsperloop(meanÂ±std.dev.of7runs,10000loopseach)>>>%timeittf.transform(['y','z'],{'sum_x':(sum,'x')})161ÂµsÂ±2.31Âµsperloop(meanÂ±std.dev.of7runs,10000loopseach)The equivalentpandasfunctions are given below. They require a chain
of several object methods to perform the same role, and the transform requires
a copy operation and assignment into the copiedDataFramein order to
preserve immutability.>>>%timeitdf.groupby(['y','z']).agg({'x':'sum'}).reset_index()2.5msÂ±177Âµsperloop(meanÂ±std.dev.of7runs,100loopseach)>>>%%timeit...tdf=df.copy()...tdf['x']=df.groupby(['y','z'])[['x']].transform(sum)2.81msÂ±143Âµsperloop(meanÂ±std.dev.of7runs,100loopseach)Version History1.0.10Addpipeand overload>>operator for Tafra objects1.0.9Add test files to build1.0.8Check rows in constructor to ensure equal data length1.0.7Handle missing or NULL values inread_csv().Cast empty elements to None when updating dtypes to avoid failure ofnp.astype().Update some typing, minor refactoring for performance1.0.6Additional validations in constructor, primary to evaluate Iterables of valuesSplitcol_maptocol_mapandkey_mapas the original function’s return signature depending upon an argument.Fix some documentation typos1.0.5Addtuple_mapmethodRefactor all iterators and..._mapfunctions to improve performanceUnpacknp.ndarrayif given as keys to constructorAddvalidate=Falsein__post_init__if inputs areknownto be valid to improve performance1.0.4Addread_csv,to_csvVarious refactoring and improvement in data validationAddtyping_extensionsto dependenciesChange method ofdtypestorage, extractstrrepresentation fromnp.dtype()1.0.3Addread_sqlandread_sql_chunksAddto_tupleandto_pandasCleanup constructor data validation1.0.2Add object_formatter to expose user formatting for dtype=objectImprovements to indexing and slicing1.0.1Add iter functionsAdd map functionsVarious constructor improvements1.0.0Initial Release