Package: serialtime
Please visit  theGitHub repositoryfor more information.SerialTimeSerialTime is a Python serialization tool containing many serialization and deserialization shortcuts with timing. There are many modules that can serialize Python object such as:JoblibPyYAMLPython built-in pickle/cPicklePython built-in jsonPython built-in pickle/cPickle + gzipHowever, remembering their API is very difficult, and there are many differences in their API.
For example,json.dumponly accept file-like object as its argument, so we need to open the file first, and then give the function the file-like object, whilejoblib.dumponly accept file path as its argument.
Besides, there is no compression shortcut for Python built-in pickle/cPickle, so we also need to remember how to use something like gzip.This package aims to solve these problems and provides very simple and unified API shortcuts for some popular serialization methods.
In addition, we useBisTimingto calculate the execution time, so you can also easily know how quick the serialization is.InstallationInstallserialtimepipinstallserialtimeIf you want to usesave_joblib_pklorload_joblib_pkl:pipinstallscikit-learnscipyIf you want to usetry_load_yaml:pipinstallPyYAMLDocumentationShortcutsPython built-in pickle/cPickleserialtime.save_pkl(obj,path,log_description=None,logger=None,logging_level=logging.INFO,verbose_start=True,verbose_end=True,end_in_new_line=True,log_prefix="...")obj=serialtime.load_pkl(path,log_description=None,logger=None,logging_level=logging.INFO,verbose_start=True,verbose_end=True,end_in_new_line=True,log_prefix="...")Python built-in pickle/cPickle + gzipserialtime.save_pklgz(obj,path,log_description=None,logger=None,logging_level=logging.INFO,verbose_start=True,verbose_end=True,end_in_new_line=True,log_prefix="...")obj=serialtime.load_pklgz(path,log_description=None,logger=None,logging_level=logging.INFO,verbose_start=True,verbose_end=True,end_in_new_line=True,log_prefix="...")Joblibserialtime.save_joblib_pkl(obj,path,log_description=None,logger=None,logging_level=logging.INFO,verbose_start=True,verbose_end=True,end_in_new_line=True,log_prefix="...")obj=serialtime.load_joblib_pkl(path,log_description=None,logger=None,logging_level=logging.INFO,verbose_start=True,verbose_end=True,end_in_new_line=True,log_prefix="...")Interactive trying of loading YAMLSometimes we want to load the configuration file in the middle of a program.
If we run the program very long and the file format is incorrect, the program may directly raise an error and exit, so we don’t have any chance to fix the file.serialtime.try_load_yamlcan try to load the file, and pause when it encounter any error, and ask you whether to reload the file.
We can then fix the file and continue running the program.serialtime.try_load_yaml(yaml_path)Dataset wrapperPartialPreprocessedDatasetis used to transparrently reindex the data without moving or copying the original memory.Sometimes we want to reindex the data, for example:In[1]:importnumpyasnpIn[2]:dset=np.asarray([1,2,3])In[3]:dsetOut[3]:array([1,2,3])In[4]:idx=[2,0]In[5]:dset2=dset[idx]In[6]:dset2Out[6]:array([3,1])However, if the data is very large or it’s on disk, this may use too much memory.
We may not need all the convenient API innumpy.ndarrayorh5py.dataset, but some modules only accept a fullnumpy.ndarrayorh5py.dataset(i.e.,keras.image.ImageDataGenerator.flow()).
Our solution is to use an object to remember the new index, and translate the index while getting the value. For example:In[1]:importnumpyasnpIn[2]:fromserialtimeimportPartialPreprocessedDatasetIn[3]:dset=np.asarray([[0,1],[2,3],[4,5]])In[4]:dsetOut[4]:array([[0,1],[2,3],[4,5]])In[5]:idx=[2,0]In[6]:dset2=PartialPreprocessedDataset(dset,idx,shape=(2,),preprocess_func=lambdax:x*2)We can also use an optionalpreprocess_functo preprocess the instance while we are getting it.
In this example, we just double the values in the array.
Theshapewe give toPartialPreprocessedDatasetis the shape of one instance (the shape of the array that we can get afterpreprocess_func(dset[x])). Then we can do something like:In[7]:dset2.shapeOut[7]:(2,2)In[8]:len(dset2)Out[8]:2In[9]:dset2[0]Out[9]:array([8,10])In[10]:dset2[1]Out[10]:array([0,2])In[11]:dset2[2]IndexError:listindexoutofrangeTestingFor the current environment:python setup test.For Python 2.7, 3.4, 3.5, 3.6 and installation test:tox.