Package: elliptio
AboutElliptIO is a small python library for storing and accessing files in data lakes in a data science context. It stores files including automatically generated metadata on any file system and inserts metada into a database. A lot of inspiration is drawn fromWeights & Biases.It is named after the Elliptio mussel genus which lives in freshwater lakes.Problems and solution approachParticular in data science you often find data lakes where...ProblemSolution approachdata cannot be reproducedAutomatically logrequired information.data lineage is unknownAutomatically tracklineage between files.data is accidentally modifiedLock files using S3 lock.data has no metadataUsers can specify custom metadata when saving files.directory structure is chaoticSimply save files by date and user. A good metadata search makes structure much less important.data is duplicatedAutomatically replace duplicated files with references (not yet implemented)Existing solutionsI find Weights and Biases a great app, from which a lot of inspiration is drawn. However, it can be rather expensive and focuses on a lot more things than just data storage, so can easily be an overkill.Object stores such as S3 or Ceph already provide the option to store metadata. However, this does not cover all required data for reproducibility. Also, querying metadata is not as efficient as querying a database.How to useimportjsonimportpandasaspdfromelliptioimportget_default_handler,ManualMetadata# setup manual metadata (optional) and handlermetadata=ManualMetadata(ticket="abc-123",project="my_project",config=json.dumps({"example":"value"}),description="lorem ipsum",)h=get_handler(dirpath="/tmp/my_data_lake",manual_metadata=metadata)# save file directly to remotedf=pd.DataFrame({"a":[1],"b":[2]})withh.create("train.txt")asf:df.to_csv(f.remote_url)# load file. Its file_id will be added to every new file in this session.train_file=h.load(f.file_id)# upload an existing new file# model.train(train_file)withh.create("model.pickle")asmodel:model.upload("/tmp/my_data_lake/best_model.pickle")assertmodel.based_on==[train_file.file_id]# querying the databasedf=h.query({"ticket":"abc-123"})How to installSimply runpip install elliptio.TipsYou can easily pass custom filesystem, database, tracker and id_creator classes toget_handlerThe current filesystem class is based on fsspec and thus should supportall their filesystem implementations (S3, Azure Blob service, Google Cloud Storage, etc.). See example below.To create a nice GUI for your database, I can recommendMetabase. Metabase, without the enterprise features, is APGL licensed. You have to be careful when modifying the code or incorporating it into your application, but running the app without modifications internally in "vanilla mode" seems to be fineaccording to them.Theterraform/directory contains example Terraform code to setup S3 and a free MongoDB on AWS. However, there's currently no MongoDB implementation for the DatabaseInterface.# Example for passing custom FileSystemInterfaces like S3fromelliptio.adaptersimportfs,dbfromelliptioimportget_handlerh=get_handler(fs=fs.FsspecFilesystem(prefix="some/prefix/",protocol="s3",storage_options={}),db=db.SqlDatabase("db.sqlite"),)TODOscompare with other data versioning tools fromhttps://github.com/EthicalML/awesome-production-machine-learning/(and other reproducibility tools)automatic metadataautomatically log git-hash andgit diff(also from new untracked files)doesargvwork with Jupyter notebooks?!