Package: on
onTools to make data access layersList, read, write, and delete data in a structured data source/target,
as if manipulating simple python builtins (dicts, lists), or through the interfaceyouwant to interact with,
with configuration or physical particularities out of the way.
Also, being able to change these particularities without having to change the business-logic code.Use casesInterfacing readsHow many times did someone share some data with you in the form of a zip of some nested folders
whose structure and naming choices are fascinatingly obscure? And how much time do you then spend to write code
to interface with that freak of nature? Well, one of the intents ofonis to make that easier to do.
You still need to understand the structure of the data store and how to deserialize these datas into python
objects you can manipulate. But with the proper tool, you shouldn't have to do much more than that.Changing where and how things are storedEver have to switch where you persist things (say from file system to S3), or change the way key into your data,
or the way that data is serialized? If you useontools to separate the different storage concerns,
it'll be quite easy to change, since change will be localized. And if you're dealing with code that was already
written, with concerns all mixed up,onshould still be able to help since you'll be able to
more easily give the new system a facade that makes it look like the old one.All of this can also be applied to data bases as well, in-so-far as the CRUD operations you're using
are covered by the base methods.Adapters: When the learning curve is in the way of learningShinny new storage mechanisms (DBs etc.) are born constantly, and some folks start using them, and we are eventually lead to use them
as well if we need to work with those folks' systems. And though we'd love to learn the wonderful new
capabilities the new kid on the block has, sometimes we just don't have time for that.Wouldn't it be nice if someone wrote an adapter to the new system that had an interface we were familiar with?
Talking to SQL as if it were mongo (or visa versa). Talking to S3 as if it were a file system.
Now it's not a long term solution: If we're really going to be using the new system intensively, we
should learn it. But when you just got to get stuff done, having a familiar facade to something new
is a life saver.onwould like to make it easier for you roll out an adapter to be able to talk
to the new system in the wayyouare familiar with.Thinking about storage later, if everYou have a new project or need to write a new app. You'll need to store stuff and read stuff back.
Stuff: Different kinds of resources that your app will need to function. Some people enjoy thinking
of how to optimize that aspect. I don't. I'll leave it to the experts to do so when the time comes.
Often though, the time is later, if ever. Few proof of concepts and MVPs ever make it to prod.So instead, I'd like to just get on with the business logic and write my program.
So what I need is an easy way to get some minimal storage functionality.
But when the time comes to optimize, I shouldn't have to change my code, but instead just change the way my
DAO does things. What I need ison.More examplesLooks like a dictBelow, we make a default store and demo a few basic operations on it.
The default store uses a dict as it's backend persister.
A dict is neither really a backend, nor a persister. But it helps to try things out with no
footprint.fromon.baseimportStores=Store()assertlist(s)==[]s['foo']='bar'# put 'bar' in 'foo'assert'foo'ins# check that 'foo' is in (i.e. a key of) sasserts['foo']=='bar'# see that the value that 'foo' contains is 'bar'assertlist(s)==['foo']# list all the keys (there's only one)assertlist(s.items())==[('foo','bar')]# list all the (key, value) pairsassertlist(s.values())==['bar']# list all the valuesassertlen(s)==1# Number of items in my stores['another']='item'# store another itemassertlen(s)==2# Now I have two!assertlist(s)==['foo','another']# here they areThere's nothing fantastic in the above code.
I've just demoed some operations on a dict.
But it's exactly this simplicity thatonaims for.
You can now replace thes = Store()withs = AnotherStore(...)whereAnotherStorenow uses some other backend that could be remote or local, could be a database, or any
system that can storesomething(the value)somewhere(the key).You can choose from an existing store (e.g. local files, for AWS S3, for MongoDB) or
quite easily make your own (more on that later).And yet, it will still look like you're talking to a dict. This not only means that you can
talk to various storage systems without having to actually learn how to, but also means
that the same business logic code you've written can be reused with no modification.Butonoffers more than just a simple consistent facade towhereyou store things,
but also provides means to definehowyou do it.In the case of key-value storage, the "how" is defined on the basis of the keys (how you reference)
the objects you're storing and the values (how you serialize and deserialize those objects).Converting keys: Relative paths and absolute pathsTake a look at the following example, that adds a layer of key conversion to a store.# defining the storefromon.baseimportStoreclassPrefixedKeyStore(Store):prefix=''def_id_of_key(self,key):returnself.prefix+key# prepend prefix before passing on to storedef_key_of_id(self,_id):ifnot_id.startswith(self.prefix):raiseValueError(f"_id{_id}wasn't prefixed with{self.prefix}")else:return_id[len(self.prefix):]# don't show the user the prefix# trying the store outs=PrefixedKeyStore()s.prefix='/ROOT/'assertlist(s)==[]s['foo']='bar'# put 'bar' in 'foo'assert'foo'ins# check that 'foo' is in (i.e. a key of) sasserts['foo']=='bar'# see that the value that 'foo' contains is 'bar'assertlist(s)==['foo']# list all the keys (there's only one)assertlist(s.items())==[('foo','bar')]# list all the (key, value) pairsassertlist(s.values())==['bar']# list all the valuesassertlen(s)==1# Number of items in my stores['another']='item'# store another itemassertlen(s)==2# Now I have two!assertlist(s)==['foo','another']# here they areQ: That wasn't impressive! It's just the same as the first Store. What's this prefix all about?A: The prefix thing is hidden, and that's the point. You want to talk the "relative" (i.e "prefix-free")
language, but may have the need for this prefix to be prepended to the key before persisting the data
and that prefix to be removed before being displayed to the user.
Think of working with files. Do you want to have to specify the root folder every time you store something
or retrieve something?Q: Prove it!A: Okay, let's look under the hood at what the underlying store (a dict) is dealing with:assertlist(s.store.items())==[('/ROOT/foo','bar'),('/ROOT/another','item')]You see? The keys that the "backend" is using are actually prefixed with"/ROOT/"Serialization/DeserializationLet's now demo serialization and deserialization.Say we want to deserialize any text we stored by appending"hello "to everything stored.# defining the storefromon.baseimportStoreclassMyFunnyStore(Store):def_obj_of_data(self,data):returnf'hello{data}'# trying the store outs=MyFunnyStore()assertlist(s)==[]s['foo']='bar'# put 'bar' in 'foo'assert'foo'ins# check that 'foo' is in (i.e. a key of) sasserts['foo']=='hello bar'# the value that 'foo' contains SEEMS to be 'hello bar'assertlist(s)==['foo']# list all the keys (there's only one)assertlist(s.items())==[('foo','hello bar')]# list all the (key, value) pairsassertlist(s.values())==['hello bar']# list all the valuesNote: This is an easy example to demo on-load transformation of data (i.e. deserialization),
but wouldn't be considered "deserialization" by all.
See theShould storage transform the data?discussion below.In the following, we want to serialize our text by upper-casing it (and see it as such)
when we retrieve the text.# defining the storefromon.baseimportStoreclassMyOtherFunnyStore(Store):def_data_of_obj(self,obj):returnobj.upper()# trying the store outs=MyOtherFunnyStore()assertlist(s)==[]s['foo']='bar'# put 'bar' in 'foo'assert'foo'ins# check that 'foo' is in (i.e. a key of) sasserts['foo']=='BAR'# see that the value that 'foo' contains is 'bar'assertlist(s)==['foo']# list all the keys (there's only one)assertlist(s.items())==[('foo','BAR')]# list all the (key, value) pairsassertlist(s.values())==['BAR']# list all the valuesIn the last to serialization examples, we only implemented one way transformations.
That's all fine if you just want to have a writer (so only need a serializer) or a reader (so only
need a deserializer).
In most cases though, you will need two way transformations, specifying how the object
should be serialized to be stored, and how it should be deserialized to get your object back.A pickle storeSay you wanted the store to pickle as your serializer. Here's how this could look like.# defining the storeimportpicklefromon.baseimportStoreclassPickleStore(Store):protocol=Nonefix_imports=Trueencoding='ASCII'def_data_of_obj(self,obj):# serializerreturnpickle.dumps(obj,protocol=self.protocol,fix_imports=self.fix_imports)def_obj_of_data(self,data):# deserializerreturnpickle.loads(data,fix_imports=self.fix_imports,encoding=self.encoding)# trying the store outs=PickleStore()assertlist(s)==[]s['foo']='bar'# put 'bar' in 'foo'asserts['foo']=='bar'# I can get 'bar' back# behind the scenes though, it's really a pickle that is stored:asserts.store['foo']==b'\x80\x03X\x03\x00\x00\x00barq\x00.'Again, it doesn't seem that impressive that you can get back a string that you stored in a dict.
For two reasons: (1) you don't really need to serialize strings to store them and (2) you don't need to serialize python
objects to store them in a dict.
But if you (1) were trying to store more complex types and (2) were actually persisting them in a file system or database,
then you'll need to serialize.
The point here is that the serialization and persisting concerns are separated from the storage and retrieval concern.
The code still looks like you're working with a dict.