Package: pandas-tfrecords
This project was born under impression fromspark-tensorflow-connectorand implements similar functionality in order to save easy pandas dataframe to tfrecords and to restore tfrecords to pandas dataframe.It can work as with local files as with AWS S3 files. Please keep in mind, that here tensorflow works with local copies of remote files, which are synced vias3fswith S3. I did this workaround because my tensorflowv2.1.0didn’t work with S3 directly and raised authentication errorCredentials have expired attempting to repull from EC2 Metadata Service, but maybe it’s fixed already.Quick startpip install pandas-tfrecordsimportpandasaspdfrompandas_tfrecordsimportpd2tf,tf2pddf=pd.DataFrame({'A':[1,2,3],'B':['a','b','c'],'C':[[1,2],[3,4],[5,6]]})# localpd2tf(df,'./tfrecords')my_df=tf2pd('./tfrecords')# S3pd2tf(df,'s3://my-bucket/tfrecords')my_df=tf2pd('s3://my-bucket/tfrecords')boolean support (v0.1.6+):importpandasaspdfrompandas_tfrecordsimportpd2tf,tf2pddf=pd.DataFrame({'A':[True,False,True],'B':['a','b','c'],'C':[[1,False],[3,True],[5,False]]})pd2tf(df,'./tfrecords')my_df=tf2pd('./tfrecords',schema={'A':bool,'B':str,'C':[int,bool]})# or if needs 0/1 instead of True/Falsemy_df=tf2pd('./tfrecords',schema={'A':int,'B':str,'C':[int,int]})# or just skip schema, int used by default if bool isn't specifiedmy_df=tf2pd('./tfrecords')Converted typespandas -> tfrecordsbytes, str -> tf.string
int, np.integer, bool, np.bool_ -> tf.int64
float, np.floating -> tf.float32
list, np.ndarray of bytes, str, int, np.integer, bool, np.bool_, float, np.floating -> sequence of tf.string, tf.int64, tf.float32tfrecords -> pandastf.string -> bytes
tf.int64 -> int, bool
tf.float32 -> float
sequence of tf.string, tf.int64, tf.float32 -> list of bytes, int, bool, floatNB!Please pay attention it works only withone-dimentionalarrays. It means[1, 2, 3]will be converted to both sides, but[[1,2,3]]won’tbe converted to any side. It works that, becausespark-tensorflow-connectorworks similar, and I didn’t learn yet how to implement nested sequences. In order to work withnestedsequences you should usereshape.APIpandas_tfrecords.pandas_to_tfrecords(df,folder,compression_type='GZIP',compression_level=9,columns=None,max_mb=50)Arguments:df- pandas dataframe. Please keep in mind above info about nested sequences.folder- folder to save tfrecords, local or S3. Please be sure that it doesn’t contain other files or folders, if you want to read from this folder then.compression_type='GZIP'- compression types:'GZIP','ZLIB',None. IfNonenot compressed.compression_level=9- compression level 0…9.columns=None- list of columns to save, ifNoneall columns will be saved.max_mb=50- maximum size of uncompressed data to save in single file. If dataframe total size is bigger than this limit, then it will be splitted to several files. IfNoneit isn’t limited and single file will be saved.aliaspandas_tfrecords.pd2tfpandas_tfrecords.tfrecords_to_pandas(file_paths,schema=None,compression_type='auto',cast=True)Arguments:file_paths- One or sequence of file paths or folders, local or S3, to read tfrecords from.schema=None- IfNoneschema will be detected automatically. But you can specify which columns you want to read only. It should be a dict, which keys are column names and values are column data types:str(orbytes),int,float, and for sequences it should be wrapped tolist:[str](or[bytes]),[int],[float]. For example:df=pd.DataFrame({'A':[1,2,3],'B':['a','b','c'],'C':[[1,2],[3,4],[5,6]]})print(df)ABC01a[1,2]12b[3,4]23c[5,6]pd2tf(df,'./tfrecords')tf2pd('./tfrecords',schema={'A':int,'C':[int]})AC01[1,2]12[3,4]23[5,6]compression_type='auto'- compression type:'auto','GZIP','ZLIB',None.cast=True- ifTrueit castsbytesdata after converting fromtf.string. It tries to cast it toint,floatandstrsequentially. If it’s not possible, otherwise keeps as is.aliaspandas_tfrecords.tf2pd