Package: caterva2
Caterva2: On-demand access to Blosc2 data repositoriesWhat is it?Caterva2 is a distributed system written in Python meant for sharingBlosc2datasets among different hosts by using apublish–subscribemessaging pattern.  Here, publishers categorize datasets into root groups that are announced to the broker and propagated to subscribers.  Also, every subscriber exposes a REST interface that allows clients to access the datasets.Caterva2 subscribers perform on demand data access with local caching (fit for re-publishing), which can be particularly useful for the efficient sharing of remote datasets locally, thus optimizing communication and storage resources within work groups.Components of Caterva2A Caterva2 deployment includes:Onebrokerservice to enable the communication between publishers and subscribers.Severalpublishers, each one providing subscribers with access to one root and the datasets that it contains.Severalsubscribers, each one tracking changes in multiple roots and datasets from publishers, and caching them locally for efficient reuse.Severalclients, each one asking a subscriber to track roots and datasets, and provide access to their data and metadata.Publishers and subscribers may be apart, in different networks with limited or expensive connectivity between them, while subscribers and clients will usually be close enough to have fast and cheap connectivity (e.g. a local network).The Caterva2 package includes all the aforementioned components, although its main role is to provide a very simple and lightweight library to build your own Caterva2 clients.Use with cautionCurrently, this project is in early alpha stage, and it is not meant for production use yet.  In case you are interested in Caterva2, please contact us atcontact@blosc.org.InstallationYou may install Caterva2 in several ways:.Pre-built wheel from PyPI:python-mpipinstallcaterva2Wheel built from source code:gitclonehttps://github.com/Blosc/Caterva2cdCaterva2
python-mbuild
python-mpipinstalldist/caterva2-*.whlDeveloper setup:gitclonehttps://github.com/Blosc/Caterva2cdCaterva2
python-mpipinstall-e.In any case, if you intend to run Caterva2 services, client programs, or the test suite, you need to enable the proper extra features by appending[feature1,feature2...]to the last argument ofpipcommands above.  For instance, to enable all extras append[services,clients,tests].TestingAfter installing with the[tests]extra, you can quickly check that the package is sane by running the test suite (that comes with the package):python-mcaterva2.tests-vYou may also run tests from source code:cdCaterva2
python-mpytest-vTests will use a copy of Caterva2'sroot-exampledirectory.  After they finish, state files will be left under the_caterva2_testsdirectory for inspection (it will be re-created when tests are run again).In case you want to run the tests with your own running daemons, you can do:envCATERVA2_USE_EXTERNAL=1python-mcaterva2.tests-vNeitherroot-examplenor_caterva2_testswill be used in this case.Quick start(Find more detailed step-by-steptutorialsin Caterva2 documentation.)For the purpose of this quick start, let's use the datasets within theroot-examplefolder:cdCaterva2
ls-Froot-example/README.md  dir1/  dir2/  ds-1d-b.b2nd  ds-1d.b2nd  ds-hello.b2frameFirst, create a virtual environment and install Caterva2 with the[services,clients]extras (see above).  Then fire up the broker, start publishing a root namedfoowithroot-exampledatasets, and create a subscriber:cat2bro&# brokercat2pubfooroot-example&# publishercat2sub&# subscriber(To stop them later on, bring each one to the foreground withfgand press Ctrl+C.)The command line clientNow that the services are running, we can use thecat2cliclient to talk
to the subscriber. In another shell, let's list all the available roots in the system:cat2clirootsfooWe only have thefooroot that we started publishing. If other publishers were running,
we would see them listed here too.Let's ask our local subscriber to subscribe to thefooroot:cat2clisubscribefoo# -> OkNow, one can list the datasets in thefooroot:cat2clilistfoofoo/README.md
...
foo/ds-hello.b2frame
...
foo/dir2/ds-4d.b2ndLet's ask the subscriber for more info about thefoo/dir2/ds-4d.b2nddataset:cat2cliinfofoo/dir2/ds-4d.b2nd{
    'shape': [2, 3, 4, 5],
    'chunks': [1, 2, 3, 4],
    'blocks': [1, 2, 2, 2],
    'dtype': 'complex128',
    'schunk': {
        # ...
    }
}Let's print data from a specified dataset:cat2clishowfoo/ds-hello.b2frame[:12]# -> Hello world!It allows printing slices instead of the whole dataset too:cat2clishowfoo/dir2/ds-4d.b2nd[1,2,3][115.+115.j 116.+116.j 117.+117.j 118.+118.j 119.+119.j]Finally, we can tell the subscriber to download the dataset:cat2clidownloadfoo/dir2/ds-4d.b2ndDataset saved to foo/dir2/ds-4d.b2ndUsing a configuration fileAll the services mentioned above (and clients, to some limited extent) may get their configuration from acaterva2.tomlfile at the current directory (or an alternative file given with the--confoption).  Caterva2 source code includes a fully documentedcaterva2.sample.tomlfile (see alsocaterva2.tomlin Caterva2 tutorials).ToolsCaterva2 includes a simple script to export the full group and dataset hierarchy in an HDF5 file to a new Caterva2 root directory.  You may use it like:cat2importexisting-hdf5-file.h5new-caterva2-rootThe tool is still pretty limited in its supported input and generated output, please invoke it with--helpfor more information (see alsocat2importin Caterva2 utilities documentation).That's all folks!