Package: mcbackend
Where doyouwant to store your MCMC draws?
In memory?
On disk?
Or in a database running in a datacenter?No matter where you want to put them, or whichPPLgenerates them: McBackend takes care of your MCMC samples.QuickstartThemcbackendpackage consists of three parts:Part 1: A schema for MCMC run & chain metadataNo matter which programming language your favorite PPL is written in, theProtocolBuffersfrom McBackend can be used to generate code in languages like C++, C#, Python and many more to represent commonly used metadata about MCMC runs, chains and model variables.The definitions inprotobufs/meta.protoare designed to maximize compatibility withArviZobjects, making it easy to transform MCMC draws stored according to the McBackend schema toInferenceDataobjects for plotting & analysis.Part 2: A storage backend interfaceThedrawsandstatscreated by MCMC sampling algorithms at runtime need to be storedsomewhere.This "somewhere" is called the storagebackendin PPLs/MCMC frameworks likePyMCoremcee.Most storage backends must be initialized with metadata about the model variables so they can, for example, pre-allocated memory for thedrawsandstatsthey're about to receive.
After then receiving thousands ofdrawsandstatsthey must then provide methods by which thedraws/statscan be retrieved.Themcbackend.coremodule has classes such asBackend,Run, andChainto define these interfaces for any storage backend, no matter if it's an in-memory, filesystem or database storage.
Albeit this implementation is currently Python-only, the interface signature should be portable to e.g. C++.Viamcbackend.backendsthe McBackend package then provides backendimplementations.
Currently you may choose from:backend=mcbackend.NumPyBackend()backend=mcbackend.ClickHouseBackend(client=clickhouse_driver.Client("localhost"))# All that matters:isinstance(backend,mcbackend.Backend)# >>> TruePart 3: PPL adaptersAnything that is aBackendcan be wrapped by anadapterthat makes it compatible with your favorite PPL.In the example below, aClickHouseBackendis initialized to store MCMC draws from a PyMC model in aClickHousedatabase.
See below forhow to run it in Docker.importclickhouse_driverimportmcbackendimportpymcaspm# 1. Create _any_ kind of backendch_client=clickhouse_driver.Client("localhost")backend=mcbackend.ClickHouseBackend(ch_client)withpm.Model():# 2. Create your model...# 3. Hit the inference button ™ while passing the backend!pm.sample(trace=backend)In case of PyMC the adapter lives in the PyMC codebasesince version 5.1.1,
so all you need to do is pass anymcbackend.Backendvia thepm.sample(trace=...)parameter!Instead of using PyMC's built-in NumPy backend, the MCMC draws now end up in ClickHouse.Retrieving thedraws&statsContinuing the example from above we can now retrieve draws from the backend.Note that since this example wrote the draws to ClickHouse, we could run the code below on another machine, and even while the above model is still sampling!backend=mcbackend.ClickHouseBackend(ch_client)# Fetch the run from the database (downloads just metadata)run=backend.get_run(trace.run_id)# Get all draws from a chainchain=run.get_chains()[0]chain.get_draws("my favorite variable")# >>> array([ ... ])# Convert everything to `InferenceData`idata=run.to_inferencedata()print(idata)# >>> Inference data with groups:# >>> 	> posterior# >>> 	> sample_stats# >>> 	> observed_data# >>> 	> constant_data# >>># >>> Warmup iterations saved (warmup_*).Contributing what's nextMcBackend just started and is looking for contributions.
For example:Schema discussion: Which metadata is needed? (related:PyMC #5160)Interface discussion: How shouldBackend/Run/Chainevolve?Python Backends for disk storage (HDF5,*.proto, ...)C++Backend/Run/ChaininterfacesC++ ClickHouse backend (viaclickhouse-cpp)As the schema and API stabilizes a mid-term goal might be to replace PyMCBaseTrace/MultiTraceentirely to rely onmcbackend.Getting rid ofMultiTracewas along-term goalbehind makingpm.sample(return_inferencedata=True)the default.DevelopmentFirst clone the repository and installmcbackendlocally:pipinstall-e.To run the tests:pipinstall-rrequirements-dev.txt
pytest-vSome tests need a ClickHouse database server running locally.
To start one in Docker:dockerrun--detach--rm--namemcbackend-db-p9000:9000--ulimitnofile=262144:262144clickhouse/clickhouse-serverCompiling the ProtocolBuffersIf you don't already have it, first install the protobuf compiler:condainstallprotobuf
pipinstall--pre"betterproto[compiler]"To compile the*.protofiles for languages other than Python, check theProtocolBuffers documentation.The following script compiles them for Python using thebetterprotocompiler plugin to get nice-looking dataclasses.
It also copies the generated files to the right place inmcbackend.pythonprotobufs/generate.py