Package: cabinets
Cabinetscabinetsis a Python library that provides a consistent interface for file operations
across multiple storage platforms. File extensions are dynamically detected to allow
automatic serialization and deserialization of Python objects.cabinetssupportsa variety of protocols and file
format parsers natively, and new protocols or parsers can be
easilyregistered.Table of contentsSample UsageRead a fileWrite a fileList files in a directoryReading and Writing with Other ProtocolsBuilt-in Protocols and ParsersProtocolsParsersProtocol ConfigurationCustom Protocols and ParsersAdding CabinetsAdding ParsersLoading PluginsContributingSample UsageRead a fileSet up a test file in your local filesystem:importjsonobj={'test':1}withopen('data.json','w')asfh:json.dump(obj,fh)Read back and parse the file usingcabinets:importcabinetsnew_obj=cabinets.read('test.json')That's it! The file isloadedandparsedin just one line.Write a filecabinetsalso supports creating files. We can rewrite the first example using
onlycabinets.importcabinetsobj={'test':1}cabinets.create('test.json',obj)new_obj=cabinets.read('test.json')assertnew_obj==objList files in a directoryIn some situations, you may need to know what files are in a directory before doing
any operations.cabinetsalso provides anlistfunction to assist with this.importcabinetsobj={'test':1}cabinets.create('example/test.json',obj)cabinets.create('example/test2.yaml',obj)cabinets.create('example/subdir/test3.txt',"test")assertcabinets.list('example/')==['test.json','test2.yaml']assertcabinets.list('example/subdir/')==['test3.txt']Important:For simplicity,cabinetsrestricts the output oflisttoonly file types.
Subdirectories are excluded, and must be queried separately.
Future versions may include a flag inlistfor returning subdirectories as well.Reading and Writing with Other ProtocolsUsingcabinetsallows you to interact with multiple file storage protocols depending
on the URI you specify. In the previous examples, we usedread()andwrite()to operate within our local file system; that's
becausecabinetsassumes we're using thefile://protocol by default. Luckily,
accessing other storage systems is just as easy!For example, operating on a file on AWS S3 is done exactly the same way:importcabinets# Read JSON file from your filesystemlocal_obj=cabinets.read('file://test.json')# Write that object to a file in AWS S3cabinets.create('s3://test.json',local_obj)# Read back the same file from AWS S3remote_obj=cabinets.read('s3://test.json')assertlocal_obj==remote_objThe above example will read a file from the local filesystem and create a new file
containing the same data, at the same path in S3.By prefixing thepathwith{protocol}://we specify how and wherecabinetsshould
look for a file. Usingfile://(default if none specified) tellscabinetsto use *
path* on the local filesystem. Usings3://on the other hand instructscabinetsto
perform operations against thatpathin AWS S3.NOTE: TheS3Cabinetmay require initial configuration for thes3protocol to
function properly. SeeProtocol Configurationfor details.See all the natively supported protocolsbelow.Built-in Protocols and ParsersProtocolsLocal File System (file://)S3 (s3://)ParsersYAML (.yml,.yaml)JSON (.json)Python Pickle (.pickle)CSV(beta)(.csv)TXT (.txt)importcabinets# .foo file in local filesystemlocal_foo_data=cabinets.read('file://test.foo')# .foo file in S3s3_foo_data=cabinets.read('s3://test.foo')Protocol ConfigurationSome storage platform protocols may require additional configuration parameters to be
set before they can be used. EachCabinetsubclass can expose
aset_configuration(**config)class method to take care of any required initial setup.fromcabinets.cabinet.s3_cabinetimportS3Cabinet# set the AWS S3 region to us-west-2 and specify an access keyS3Cabinet.set_configuration(region_name='us-west-2',aws_access_key_id=...)# use specific Cabinet to avoid protocol prefixS3Cabinet.read('bucket-in-us-west-2/test.json')# or use generic Cabinet with protocol prefiximportcabinetscabinets.read('s3://bucket-us-west-2/test.json')See the documentation of specificCabinetclasses for what configuration parameters
are available.Additionally, there is a top-levelset_configuration()function so that importing
specificCabinetsubclasses is not required. Simply pass the desired protocol as the
first argument.importcabinets# *OPTIONAL*: set the AWS S3 region to us-west-2 and specify an access keycabinets.set_configuration('s3',region_name='us-west-2',aws_access_key_id=...)# use generic Cabinet with protocol prefixcabinets.read('s3://bucket-us-west-2/test.json')Custom Protocols and Parserscabinetsis designed to allow complete extensibility in adding new protocols and
parsers. Just because your desired storage platform or file format is not listed above,
doesn't mean you can't use it withcabinets!Adding CabinetsNew protocol connections can be added by subclassing abstract base classCabinet, and
registering the class to one or more protocol identifiers:fromcabinetsimportCabinet,register_protocols@register_protocols('foo')classFooCabinet(Cabinet):@classmethoddefset_configuration(cls,**kwargs):# Set up any necessary configuration parameters for "foo" protocol...@classmethoddefread_content(cls,path:str)->bytes:# Custom logic for reading bytes from a path using "foo" protocol...@classmethoddefcreate_content(cls,path:str,content:bytes):# Custom logic for writing bytes to a path  using "foo" protocol...@classmethoddefdelete_content(cls,path):# Custom logic for deleting the object at a path  using "foo" protocol...Here we define aFooCabinet, and register it to the protocol identifierfoo. Once
this class is loaded, anycabinetsfunction calls using thefoo://prefix will be
processed with this class. This means if we called:importcabinetsfrom...importFooCabinet# ensure FooCabinet is loadedcabinets.read('foo://example.json')The first call that occurs will beFooCabinet.read_content('foo.json), and that result
is then parsed by theJSONParserbefore being returned.NOTE: In order for the protocols to be registered, the class definition must be
run at least once. Make sure the modules where your customCabinetclasses are defined
are imported somewhere before they are used, OR use the built inPluginsystem.Adding Parserscabinetsalso supports custom extension parsing in the exact same way:fromcabinets.parserimportParser,register_extensions@register_extensions('bar')classBarParser(Parser):@classmethoddefload_content(cls,content:bytes):# Parse bytes from "bar" file format into a Python object...@classmethoddefdump_content(cls,data:Any):# Dump a Python object into bytes in the "bar" file format...Now if we redo our above example using the.barextension:from...importFooCabinet,BarParser# ensure FooCabinet and BarParser are loadedcabinets.read('foo://example.bar')This statement is roughly equivalent to:BarParser.load_content(FooCabinet.read_content('foo.bar'))and should return a Python object from yourFoocabinet, using yourBarparser!Loading PluginsAs mentioned in the example above, your customCabinetandParserclasses must be
executed in order to be added to the internal cachecabinetsuses for protocol and
extension lookup. If your custom classes are imported before anycabinetsfunctions
are use then, this won't be an issue. However, in many use cases there is no reason to
import those classes aside from usage withcabinetsfunctions. Instead of requiring
each class to be imported manually at the start of your program,cabinetscan search a specified path for newCabinetandParserclasses, and load
them automatically.Specifying thePLUGIN_PATHenvironment variable will causecabinetsto search for
subdirectories calledcabinetandparserin that path. Modules residing within those
directories will be searched forCabinetandParsersubclasses respectively.└─ PLUGIN_PATH
    |
    └───cabinet
    │   │   foo_cabinet.py
    └───parser
    │   │   bar_parser.py
    │   │   baz_parser.pyIf the aboveFooCabinetandBarParserclasses are placed infoo_cabinet.pyandbar_parser.py, they will be loaded and registered to their specified cache without
needing to be referenced anywhere else in the program.ContributingThis package is open source (seeLICENSE), so please feel free
tocontributeby submitting a pull request, creating an issue, or contacting the authors directly.Authors and ContributorsLucas Lofaro(Co-Author):lucasmlofaro@gmail.comSam Hollenbach(
Co-Author):samhollenbach@gmail.com