Package: data-vault
IPython data-vaultIPython magic for simple, organized, compressed and encrypted storage & transfer of files between notebooks.Background and demoRight tool for a simple jobThe%vaultmagic provides a reproducible caching mechanism for variables exchange between notebooks.
The cache is compressed, persistent and safe.Differently to the builtin%storemagic, the variables are stored in plain sight,
in a zipped archive, so that they can be easily accessed for manual inspection,
or for the use by other tools.Demonstration by usage:Let's open the vault (it will be created if not here yet):%open_vault-pdata/storage.zipGenerate some dummy dataset:frompandasimportDataFramefromrandomimportchoice,randintcities=['London','Delhi','Tokyo','Lagos','Warsaw','Chongqing']salaries=DataFrame([{'salary':randint(0,100),'city':choice(cities)}foriinrange(10000)])Store variable in a moduleAnd store it in the vault:%vaultstoresalariesindatasetsStored salaries (None → 40CA7812) at Sunday, 08. Dec 2019 11:58A short description is printed out (including a CRC32 hashsum and a timestamp) by default, but can be disabled by passing--timestamp Falseto%open_vaultmagic.
Even more information enhancing the reproducibility isstored in the cell metadata.Import variable from a moduleWe can now load the stored DataFrame in another (or the same) notebook:%vaultimportsalariesfromdatasetsImported salaries (40CA7812) at Sunday, 08. Dec 2019 12:02Thanks to (optional)memory optimizationswe saved some RAM (87% as compared to unoptimizedpd.read_csv()result).
To track how many MB were saved use--report_memory_gainsetting which will display memory optimization results below imports, for example:Reduced memory usage by 87.28%, from 0.79 MB to 0.10 MB.Import variable as something elseIf we already have the salaries variable, we can useas, just like in the Python import system.%vaultimportsalariesfromdatasetsassalaries_datasetStore or import with a custom functionfrompandasimportread_csvto_csv=lambdadf:df.to_csv()%vaultstoresalariesindatasetswithto_csvassalaries_csv%vaultimportsalaries_csvfromdatasetswithread_csvImport an arbitrary filefrompandasimportread_excel%vaultimport'cars.xlsx'ascars_datasetwithread_excelMore examples are available in theExamples.ipynbnotebook, which can berun interactively in the browser.GoalsSyntax:easy to understand in plain language (avoid abbreviations when possible),while intuitive for Python developers,...but sufficiently different so that it would not be mistaken with Python constructsfor example, we could have%from x import y, but this looks very like normal Python;
having%vault from x import ymakes it sufficiently easy to distinguishstar imports are better avoided, thus not supportedas imports may be confusing if there is more than oneReproducibility:promote good reproducible and traceable organization of files:promote storage in plain text files and the use of DataFramepickling is often an easy solution, but it can cause hurtful problems in prototyping phase (which is what notebooks are often used for): if you pickle you objects, then change the class definition and attempt to load your data again you are likely to fail severly; this is why the plain text files are the default option in this package (but pickling is supported too!).print out a short hashsum and human-readable datetime (always in UTC),while providing even more details in cell metadataallow to trace instances of the code being modified post executionSecurity:think of it as a tool to minimize the damage in case of accidentalgit addof data files (even if those should have been elsewhere and.gitignored in the first place),or, as an additional layer of security for already anonymized data,but this tool isnotaimed at facilitating the storage of highly sensitive datayou have to set a password, or explicitly set--secure Falseto get rid of a security warningFeatures overviewMetadata for storage operationsEach operation will print out the timestamp and the CRC32 short checksum of the files involved.
The timestamp of the operation is reported in the UTC timezone in a human-readable format.This can be disabled by setting-t Falseor--timestamp False, however for the sake of reproducibility
it is encouraged to keep this information visible in the notebook.More precise information including the SHA256 cheksum (with a lower probability of collisions),
and a full timestamp (to detect potential race condition errors in file write operations) are
embedded in the metadata of the cell. You can disable this by setting --metadata False.The exact command line is also stored in the metadata, so that if you accidentally modify the code cell
without re-running the code, the change can be tracked down.StorageIn order to enforce interoperability plain text files are used for pandas DataFrame and Series objects.
Other variables are stores as pickle objects. The location of the storage archive on the disk defaults
tostorage.zipin the current directory, and can changed using%open_vaultmagic:%open_vault-pcustom_storage.zipEncryptionThe encryption is not intended as a high security mechanism,
but only as an additional layer of protection for already anonymized data.The password to encrypt the storage archive is retrieved from the environmental variable,
using a name provided inencryption_variableduring the setup.%open_vault-eENV_STORAGE_KEYMemory optimizationsPandas DataFrames are by-default memory optimized by conversion of string variables to (ordered) categorical
columns (pandas equivalent of R's factors/levels). Each string column will be tested for the memory improvement
and the optimization will be only applied if it does reduce the memory usage.Why ZIP and not HDF?The storage archive is conceptually similar to Hierarchical Data Format (e.g. HDF5) object - it contains:a hierarchy of files, anda metadata filesI believe that HDF may be the future, but this future is not here yet - numerous issues with the packages handling
the HDF files, as well as low performance and compression rate prompted me to stay with a simple zip format now.ZIP is a popular file format with known features and limitations - files can be password encrypted, while the file
list is always accessible. This is okay given that the code of the project is assumed to be public, and only the
files in the storage area are assumed to be of encrypted, increasing the security in case of unauthorized access.As the limitations of the ZIP encryption are assumed to be a common knowledge, I hope that managing expectations
of the level of security offered by this package will be easier.Installation and requirementsPre-requirements:Python 3.6+7zip (16.02+) (seebelowfor Ubuntu and Mac commands)Installation:pip3installdata_vaultInstalling 7-zipYou can use p7zip packages from the default repositories:Ubuntusudoapt-getinstall-yp7zip-fullMacbrewinstallp7zipWindowsInstallers for Windows can be downloaded from the7-zip website.Windows is not supported as it has known issues.