Package: cas-manifest
cas-manifest provides a means to store data that is always immutable, stable when you want it to be, and flexible when you need it to be.cas-manifest stores data artifacts via content addressible storage. It facilitates the use of CAS with standard, serializable wrappers that coexist with and support data.Why CAS?In short, CAS enforces immutability. When using CAS, a key's contents can never be changed. The following comes naturally:No moredata_final__2_newfiles - all objects are uniquely specifiedNo cache invalidation - cache objects freely, knowing that their contents will never change upstreamNo more provenance questions - models can be robustly linked to the datasets used to train themIn a CAS store, instead ofput-ing a Value at a Key, youputa Value and get back the Key uniquely determined by that value.Why manifests?It's all well and good to stuff some data into a binary artifact and keep a key that references it. The hard part is, when you're given a key, how do you know what's stored there? If you do settle on a standard serialization method, how do you let it evolve while maintaining backward compatibility?cas-manifest encourages the use of manifest classes to address these challenges. These manifest classes include code to serialize and deserialize artifacts. They provide a place to store metadata about the artifacts - this may be used for deserialization, used to indicate how the loaded data should be used, or may simply be informational. Finally, fields in the manifest class may reference other objects in CAS, allowing objects to be composed and reused. In combination with CAS, you can ensure that your metadata and underlying data never go out of sync, since your metadata will refer to an immutable reference to underlying data.ExampleImplementing SerializationLet's say that we wish to store datasets that we represent in memory as pandas Dataframes. We'll create a subclass ofcas_manifest.Serializableto represent what we wish to store:classCSVSerializable(Serializable[pd.DataFrame]):column_names:List[str]path:Ref@classmethoddefpack(cls,inst:pd.DataFrame,fs:HashFS)->CSVSerializable:withtempfile.TemporaryDirectory()astmpdir:tmp_path=Path(tmpdir)/'tmp.csv'withopen(tmp_path,mode='w')asf:inst.to_csv(f,header=False,index=False)csv_addr=fs.put(tmp_path)returnCSVSerializable(path=Ref(csv_addr.id),column_names=inst.columns.to_list())defunpack(self,fs:HashFS)->pd.DataFrame:addr=fs.get(self.path.hash_str)df=pd.read_csv(addr.abspath,names=self.column_names)returndfLet's break down the items one by one:classCSVSerializable(Serializable[pd.DataFrame]):The type parameter ofSerializableis the type that we use in memory. Whatever it is you use in application code, that's what you put here.column_names:List[str]path:Refcas-manifest uses pydantic to define its manifest classes. This allows you to specify class fields at the top level. In order to ensure that they can be serialized, these fields need to be simple types, or other pydantic classes.Refis a special wrapper class used to refer to other objects in cas.@classmethoddefpack(cls,inst:pd.DataFrame,fs:HashFS)->CSVSerializable:withtempfile.TemporaryDirectory()astmpdir:tmp_path=Path(tmpdir)/'tmp.csv'withopen(tmp_path,mode='w')asf:inst.to_csv(f,header=False,index=False)csv_addr=fs.put(tmp_path)returnCSVSerializable(path=Ref(csv_addr.id),column_names=inst.columns.to_list())packis a required method; this specifies how your data should be serialized. Take note of the types of the arguments.In the body of this method, we save our dataframe as a csv, without a header row. We thenputthat into theHashFSinstance thatpacktakes as an argument.HashFSprovides the implementation ofCASthat we use. Thatputoperation returns the address (or key) for our csv. We can then construct an instance of our wrapper class, which contains aRefto the csv file, and the column names as fields in the manifest class.defunpack(self,fs:HashFS)->pd.DataFrame:addr=fs.get(self.path.hash_str)df=pd.read_csv(addr.abspath,names=self.column_names)returndfunpackis another required method; this indicates how your serialized data should be deserialized. Again, note the types of the arguments and the return types. In this case, we get the location on the filesystem for the csv file that we saved. We then callpandas.read_csvto read it, supplying the column names stored in the manifest class.Note that, when this pattern is followed in the real world, it can often be confusing to keep track of whether column names are stored as a header row or kept elsewhere, whether there should be an index column, etc. This sounds silly, but it's a real problem - especially if you ever want to change your mind! cas-manifest standardizes these decisions by embedding the logic in the manifest class.Storing and retrieving dataNow that we've implemented all that, how do we use it?First, we need to put something into cas. Let's say that we have a DataFrame nameddf. We'll also need an instance ofHashFSfrom thehashfspackage. AWS users may wish to make use ofS3HashFSin this package, which provides an implementation ofHashFSbacked by S3. We can then do the followingimportpandasaspdfromhashfsimportHashFS,HashAddressdf:pd.DataFrame=...fs_instance:HashFS=...addr:HashAddress=CSVSerializable.dump(df,fs_instance)Given an instance ofHashFS,  we can calldumponCSVSerializableto serialize our DataFrame and store it infs_instance. The returned object is aHashAddress, which includes the immutable hash of the serialized object, as well as helper information like a path to its location on disk. If we wanted access to the serialized representation, we could also have calledCSVSerializable.pack(df, fs_instance)to get an instance ofCSVSerializable.Now, how do we retrieve our serialized object from storage? Again with our instance ofHashFS, we'll do the followinghash_str=addr.id# We create a Registry that knows what classes to expectregistry:SerializableRegistry[pd.DataFrame]=\SerializableRegistry(fs=fs_instance,classes=[CSVSerializable])# We can `open` a hash address to get access to the dataframewithregistry.open(hash_str)asdf:pass# df is the DataFrame that we saved before# Or we can get the serialized form directlyserialized:CSVSerializable=registry.load(addr.id)Why isopena context manager? Some implementations ofSerializablemay create temporary resources that need to be cleaned up, so we treatopenlike opening and closing a file.Evolving the serialization schemaNow, let's imagine that we decide we want to change our seralization format. Perhaps we'd like to make use of numpy's serialization methods to store the data in our dataframe. We can create another subclass ofSerializable, implementingpackandunpackas before:classNPYSerializable(Serializable[pd.DataFrame]):...We'll skip the implementation for brevity here, but one is available intests/dataset.py.We can serialize a dataframe in this new format just as we did withCSVSerializable. When we want to load data, that's where things get interesting:registry_2:SerializableRegistry[pd.DataFrame]=\SerializableRegistry(fs=fs_instance,classes=[CSVSerializable,NPYSerializable])registry_2now knows how to deserialize data stored in either format. You can pass it a hash string corresponding to either format, and it will correctly deserialize it into a DataFrame.This means that you won't have to implement code to sniff out how data is stored on disk and sprinkle it around your codebase. You can consolidate your serde logic in a class, and let cas-manifest sort out how to handle it from there.GotchasRegarding portability and schema evolution: keep in mind that your code isnotserialized. So, in order to load an object of typeX, you must still haveXavailable in your codebase. Instantiating your registry should make this part fairly clearRelated to the above, if you make changes to a class, you must ensure that they are backward-compatible (e.g. adding optional fields) in order to be able to load older data.Typing: I've done my best to supply correct type annotations, but mypy struggles to infer return types of some generic functions. Explicit type annotations can be helpful.