Package: xpublish-host
xpublish-hostA collection of tools and standards for deployingxpublishinstances.Why?With ~50 netCDF-based datasets to be published throughxpublish, Axiom needed a standard way to configure each of these deployments. We could have created single repository and defined each individualxpublishdeployment, we could have created individual repositories for each dataset, or we could have done something in the middle. We decided to abstract out the parts common to each deployment and put it here intoxpublish-host. This prevents the re-implementation of things like authentication (tbd), logging, metrics, and allows data engineers to focus on the data and not the deployment.GoalsStandardize the configuration of anxpublishdeployment (plugins, ports, cache, dask clusters, etc.) using config files and environmental variables, not python code.Standardize on a core set ofFastAPIobservability middleware (metrics, monitoring, etc.),Provide a plugin to definexpublishdatasets via configuration (DatasetsConfigPlugin).Provide a set of commonloaderfunctions for use as an argument in aDatasetsConfigto standardize common access patterns (xarray.open_mfdatasetis currently supported).Provide a pre-built Docker image to run an opinionated and performantxpublishinstance usinggunicorn.Thoughtsxpublish-hostmakes no assumptions about the datasets you want to publish throughxpublishand only requires the path to an importable python function that returns the object you want to be passed in as an argument toxpublish.Rest. This allowsxpublish-hostto support datasets in addition toxarray.Datasetin the future, such as Parquet files.As a compliment toxpublish-host, Axiom maintain a private repository of server and dataset YAML files that are compatible withxpublish-hostand theDatasetsConfigPlugin. On deploy we mount these files into thexpublish-hostcontainer and they represent the only customizations required to get a workingxpublish-hostinstance going.InstallationMost users will not need to installxpublish_hostdirectly as a library but instead will use the Docker image to deploy anxpublishinstance. If you want to use thexpublish_hosttools and config objects directly in python code, you can of course install it:Forcondausers you cancondainstall--channelconda-forgexpublish_hostor, if you are apipuserpipinstallxpublish_hostor, if you are usingdockerdockerrun--rm-p9000:9000axiom/xpublish-host:latestBatteries IncludedHost ConfigurationA top-level configuration for running anxpublish-hostinstance.The configuration is managed usingPydanticBaseSettingsandGoodConffor loading configuration from files.Thexpublish-hostconfiguration can be set in a few waysEnvironmental variables- prefixed withXPUB_, they map directly to thepydanticsettings classes,Environment files- Load environmental variables from a file. UsesXPUB_ENV_FILESto control the location of this file if it is defined. See thePydanticdocsfor more information,Configuration files (JSON and YAML)-GoodConfbasedconfiguration files. When using thexpublish_host.app.servehelper this file can be set by definingXPUB_CONFIG_FILE.Python arguments (API only)- When usingxpublish-hostas a library you can use the args/kwargs of each configuration object to control yourxpublishinstance.The best way to get familiar with which configuration options are available (until the documentation catches up) is to look at the actually configuration classes inxpublish_host/config.pyand the tests intests/test_config.pyandtests/utils.pyA feature-full configuration is as follows, which includes the defaults for each field.# These are passed into the `xpublish.Rest.serve` method to control how the# server is run. These are ignored if running through `gunicorn` in production mode# or using the Docker image. See the `CLI` section below for more details.publish_host:"0.0.0.0"publish_port:9000log_level:debug# Dask cluster configuration.# The `args` and `kwargs` arguments are passed directly into the `module`# Omitting cluster_config or setting to null will not use a cluster.cluster_config:module:dask.distributed.LocalClusterargs:[]kwargs:processes:truen_workers:2threads_per_worker:1memory_limit:1GiBhost:"0.0.0.0"scheduler_port:0# random portdashboard_address:0.0.0.0:0# random portworker_dashboard_address:0.0.0.0:0# random port# Should xpublish discover and load plugins?plugins_load_defaults:true# Define any additional plugins. This is where you can override# default plugins. These will replace any auto-discovered plugins.# The keys here (zarr, dconfig) are not important and are not used internallyplugins_config:zarr:module:xpublish.plugins.included.zarr.ZarrPlugindconfig:module:xpublish_host.plugins.DatasetsConfigPluginkwargs:# Define all of the datasets to load into the xpublish instance.datasets_config_file:datasets.yamldatasets_config:# The keys here (dataset_id_1) are not important and are not used internally# but it is good practice to make them equal to the dataset's id fielddataset_id_1:# The ID is used as the "key" of the dataset in `xpublish.Rest`# i.e. xpublish.Rest({ [dataset.id]: [loader_function_return] })id:dataset_idtitle:Dataset Titledescription:Dataset Description# Path to an importable python function that returns the dataset you want# to pass into `xpublish.Rest`loader:[python module path]# Arguments passed into the `loader` functionargs:-[loader arg1]-[loader arg2]# Keyword arguments passed into the `loader` function. See the `examples`# directory for more details on how this can be used.kwargs:keyword1:'keyword1'keyword2:false# After N seconds, invalidate the dataset and call the `loader` method againinvalidate_after:10# If true, defers the initial loading of the dataset until the first request# for the dataset comes in. Speeds up server load times but slows down the# first request (per-process) to each datasetskip_initial_load:true# Keyword arguments to pass into `xpublish.Rest` as app_kws# i.e. xpublish.Rest(..., app_kws=app_config)app_config:docs_url:/apiopenapi_url:/api.json# Keyword arguments to pass into `xpublish.Rest` as cache_kws# i.e. xpublish.Rest(..., cache_kws=cache_config)cache_config:available_bytes:1e11Metricsxpublish-hostprovides aprometheuscompatible metrics endpoint. The docker image supports multi-process metric generation throughgunicorn. By default the metrics endpoint is available at/metrics.The default labels formatxpublish-hostmetrics are:[XPUB_METRICS_PREFIX_NAME]_[metric_name]{app_name="[XPUB_METRICS_APP_NAME]",environment="[XPUB_METRICS_ENVIRONMENT]"}The metrics endpoint can be configured using environmental variables:XPUB_METRICS_APP_NAME(default:xpublish)XPUB_METRICS_PREFIX_NAME(default:xpublish_host)XPUB_METRICS_ENDPOINT(default:/metrics)XPUB_METRICS_ENVIRONMENT(default:development)XPUB_METRICS_DISABLE- disabled the metrics endpoint by setting this to any valueHealthA health check endpoint is available at/healthto be used by various health checkers (docker, load balancers, etc.). You can disable the heath check endpoint by settings the environmental variableXPUB_HEALTH_DISABLEto any value. To change the endpoint, setXPUB_HEALTH_ENDPOINTto the new value, i.e.export XPUB_HEALTH_ENDPOINT="/amiworking"DatasetsConfigPluginThis plugin is designed to load datasets intoxpublishfrom a mapping ofDatatsetConfigobjects. It can get the mapping directory from the plugin arguments or from ayamlfile.TheDatasetsConfigPluginplugin can take two parameters:datasets_config: dict[str, DatasetConfig]datasets_config_file: Path- File path to a YAML file defining the abovedatasets_configobject.Define datasets from anxpublish-hostconfiguration file:plugins_config:dconfig:module:xpublish_host.plugins.DatasetsConfigPluginkwargs:datasets_config:simple:id:statictitle:Staticdescription:Static dataset that is never reloadedloader:xpublish_host.examples.datasets.simpleOr from axpublish-hostconfiguration file referencing an external datasets configuration file# datasets.yamldatasets_config:simple:id:statictitle:Staticdescription:Static dataset that is never reloadedloader:xpublish_host.examples.datasets.simpleplugins_config:dconfig:module:xpublish_host.plugins.DatasetsConfigPluginkwargs:datasets_config_file:datasets.yamlYou can also mix and match between in-line configurations and file-based dataset configurations. Always be sure theidfield is unique for each defined dataset or they will overwrite each other, with config file definitions taking precedence.plugins_config:dconfig:module:xpublish_host.plugins.DatasetsConfigPluginkwargs:datasets_config_file:datasets.yamldatasets_config:simple_again:id:simple_againtitle:Simpledescription:Simple Datasetloader:xpublish_host.examples.datasets.simpleDatasetConfigTheDatasetConfigobject is a way to store information about how to load a dataset you want published throughxpublish. It supports dynamically loading datasets on request rather than requiring them to be loaded whenxpublishis started. It allows mixing together static datasets that do not change and dynamic datasets that you may want to reload periodically onto onexpublishinstance.Theloaderparameter should be a path to a python module that will return the dataset you want served throughxpublish. Theargsandkwargsparameters are passed into that function whenxpublishneeds to load or reload your dataset.Here is an example of how to configure anxpublishinstance that will serve astaticdataset that is loaded once on server start and adynamicdataset that is not reloaded on server start. It is loaded for the first time on first request and then reloaded every 10 seconds. It isn't reloaded on a schedule, it is reloaded on-request if the dataset has not been accessed afterinvalidate_afterseconds has elapsed.datasets_config:simple:id:statictitle:Staticdescription:Static dataset that is never reloadedloader:xpublish_host.examples.datasets.simpledynamic:id:dynamictitle:Dynamicdescription:Dynamic dataset re-loaded on request periodicallyloader:xpublish_host.examples.datasets.simpleskip_initial_load:trueinvalidate_after:10You can run the above config file and take a look at what is produced. There are (2) datasets:staticanddynamic. If you watch the logs and keep refreshing access to thedynamicdataset, it will re-load the dataset every10seconds.$xpublish-host-cxpublish_host/examples/dynamic.yaml

INFO:Uvicornrunningonhttp://0.0.0.0:9000(PressCTRL+Ctoquit)INFO:127.0.0.1:42808-"GET /datasets HTTP/1.1"200OK# The static dataset is already loadedINFO:127.0.0.1:41938-"GET /datasets/static/ HTTP/1.1"200OK# The dynamic dataset is loaded on first accessINFO:xpublish_host.plugins:Loadingdataset:dynamic
INFO:127.0.0.1:41938-"GET /datasets/dynamic/ HTTP/1.1"200OK# Subsequent access to dynamic before [invalidate_after] seconds uses# the already loaded datasetINFO:127.0.0.1:41938-"GET /datasets/dynamic/ HTTP/1.1"200OK
INFO:127.0.0.1:41938-"GET /datasets/dynamic/ HTTP/1.1"200OK
INFO:127.0.0.1:41938-"GET /datasets/dynamic/ HTTP/1.1"200OK
INFO:127.0.0.1:41938-"GET /datasets/dynamic/ HTTP/1.1"200OK
INFO:127.0.0.1:41938-"GET /datasets/dynamic/ HTTP/1.1"200OK# Eventually [invalidate_after] seconds elapses and the dynamic# dataset is reloaded when the request is madeINFO:xpublish_host.plugins:Loadingdataset:dynamic
INFO:127.0.0.1:41938-"GET /datasets/dynamic/ HTTP/1.1"200OK
INFO:127.0.0.1:41938-"GET /datasets/dynamic/ HTTP/1.1"200OK# The static dataset is never reloadedINFO:127.0.0.1:41938-"GET /datasets/static/ HTTP/1.1"200OK# This works when accessing datasets through other plugins as well (i.e. ZarrPlugin)INFO:xpublish_host.plugins:Loadingdataset:dynamic
INFO:127.0.0.1:48092-"GET /datasets/dynamic/zarr/.zmetadata HTTP/1.1"200OK
INFO:127.0.0.1:48092-"GET /datasets/dynamic/zarr/.zmetadata HTTP/1.1"200OKLoadersxpublish_host.loaders.mfdataset.load_mfdatasetA loader function to utilizexarray.open_mfdatasetto open a path of netCDF files. Common loading patterns have been abstracted into keyword arguments to standardize the function as much as possible.defload_mfdataset(root_path:str|Path,# root folder pathfile_glob:str,# a file glob to loadopen_mfdataset_kwargs:t.Dict={},#  any kwargs to pass directly to xarray.open_mfdatasetfile_limit:int|None=None,# limit the number of files to load, from the end after sorting ASCskip_head_files:int|None=0,# skip this number of files from the beginning of the file listskip_tail_files:int|None=0,# skip this number of files from the end of the file listcomputes:list[str]|None=None,# A list of variable names to call .compute() on to they are evaluated (useful for coordinates)chunks:dict[str,int]|None=None,# A dictionary of chunks to use for the datasetaxes:dict[str,str]|None=None,# A dictionary of axes mapping using the keys t, x, y, and zsort_by:dict[str,str]|None=None,# The field to sort the resulting dataset by (usually the time axis)isel:dict[str,slice]|None=None,# a list of isel slices to take after loading the datasetsel:dict[str,slice]|None=None,# a list of sel slices to take after loading the datasetrechunk:bool=False,# if we should re-chunk the data applying all sorting and slicingattrs_file_idx:int=-1,# the index into the file list to extract metadata fromcombine_by_coords:list[str|Path]=None,# a list of files to combine_by_coords with, useful for adding in grid definitions**kwargs)->xr.Dataset:Yeah, that is a lot. An Example may be better.# Select the last 24 indexes of ocean_time and the first Depth index# from the last 5 netCDF files found in a directory,# after sorting by the filename. Drop un-needed variables# and use a Dask cluster to load the files if one is available.# Compute the h and mask variables into memory so they are# not dask arrays, and finally, sort the resulting xarray# dataset by ocean_time and then Depth.datasets_config:sfbofs_latest:id:sfbofs_latesttitle:Last 24 hours of SFBOFS surface datadescription:Last 24 hours of SFBOFS surface dataloader:xpublish_host.loaders.mfdataset.load_mfdatasetkwargs:root_path:data/sfbofs/file_glob:"**/*.nc"file_limit:5axes:t:ocean_timez:Depthx:Longitudey:Latitudecomputes:-h-maskchunks:ocean_time:24Depth:1nx:277ny:165sort_by:-ocean_time-Depthisel:Depth:[0,1,null]ocean_time:[-24,null,null]open_mfdataset_kwargs:parallel:truedrop_variables:-forecast_reference_time-forecast_hourRunningThere are two main ways to runxpublish-host, one is suited for Development (xpublishby default usesuvicorn.run) and one suited for Production (xpublish-hostusesgunicorn). See theUvicorndocsfor more information.DevelopmentAPITo configure and deploy anxpublishinstance while pulling settings from a yaml file and environmental variables you can use theservefunction.Load config from a file>>>fromxpublish_host.appimportserve>>>serve('xpublish_host/examples/example.yaml')INFO:goodconf:Loadingconfigfromxpublish_host/examples/example.yaml...INFO:Uvicornrunningonhttp://0.0.0.0:9000(PressCTRL+Ctoquit)pythonLoad environmental variables from a custom .env file>>>importos>>>os.environ['XPUB_ENV_FILES']='xpublish_host/examples/example.env'>>>fromxpublish_host.appimportserve>>>serve()INFO:goodconf:Noconfigfilespecified.Loadingwithenvironmentvariables....INFO:Uvicornrunningonhttp://0.0.0.0:9000(PressCTRL+Ctoquit)pythonSet the default location to load a configuration file from>>>importos>>>os.environ['XPUB_CONFIG_FILE']='xpublish_host/examples/example.yaml'>>>fromxpublish_host.appimportserve>>>serve()INFO:goodconf:Loadingconfigfromxpublish_host/examples/example.yaml...INFO:Uvicornrunningonhttp://0.0.0.0:9000(PressCTRL+Ctoquit)pythonRestConfigYou can also use theRestConfigobjects directly to serve datasets through the API while mixing in configuration file as needed. If you using the API in this way without using a config file or environmental variables it is better to use thexpublishAPI directly instead.fromxpublish_host.configimportRestConfig,PluginConfigfromxpublish_host.pluginsimportDatasetsConfigPluginpc=PluginConfig(module=DatasetsConfigPlugin,kwargs=dict(datasets_config=dict(simple=dict(id='simple',title='title',description='description',loader='xpublish_host.examples.datasets.simple',),kwargs=dict(id='kwargs',title='title',description='description',loader='xpublish_host.examples.datasets.kwargs',args=('temperature',),kwargs=dict(values=[0,1,2,3,4,5,6,7,8])))))rc=RestConfig(load=True,plugins_config={'dconfig':pc})rest=rc.setup()# This returns an `xpublish.Rest` instancerest.serve(host='0.0.0.0',port=9000,log_level='debug',)DatasetConfigIf you are serving a single dataset there is a helper methodserveon theDatasetConfigobject.fromxpublish_host.pluginsimportDatasetConfigdc=DatasetConfig(id='id',title='title',description='description',loader='xpublish_host.examples.datasets.simple',)# Keyword arguments are passed into RestConfig and can include all of the# top level configuration options.dc.serve(host='0.0.0.0',port=9000,log_level='debug',)CLI (dev)When developing locally or in a non-production environment you can use helper CLI methods to run anxpublishserver and optionally pass in the path to a configuration file. Use the providedxpublish-hostcommand (when installed throughsetuptools) orpython xpublish_host/app.py, they are the same thing!Pass in a config file argument$xpublish-host-cxpublish_host/examples/example.yaml

INFO:goodconf:Loadingconfigfromxpublish_host/examples/example.yaml
...
INFO:Uvicornrunningonhttp://0.0.0.0:9000(PressCTRL+Ctoquit)Pull config file from an environmental variable$XPUB_CONFIG_FILE=xpublish_host/examples/example.yamlxpublish-host

INFO:goodconf:Loadingconfigfromxpublish_host/examples/example.yaml
...
INFO:Uvicornrunningonhttp://0.0.0.0:9000(PressCTRL+Ctoquit)Either way,xpublishwill be running on port 9000 with (2) datasets:simpleandkwargs. You can access the instance athttp://[host]:9000/datasets/.ProductionTo getxpublishto play nicely with async loops and processes being run bygunicornanddask, there is a custom worker class (xpublish_host.app.XpdWorker) and agunicornconfig file (xpublish_host/gunicorn.conf.py) that must be used. These are loaded automatically if you are using the provided Docker image.If you define acluster_configobject when running usinggunicorn, one cluster is spun up in the parent process and the scheduler_address for that cluster is passed to each  worker process. If you really want one cluster per process, you will have to implement it yourself and send a PR ;). Better integration withLocalClusterwould be nice, but the way it is done now allows a "bring your own" cluster configuration as well if you are managingdaskclusters outside of the scope of this project.Note:when usinggunicornthe host and port configurations can only be passed in using the-b/--bindarguments or in the configuration file. If set in any environmental variables they will be ignored!CLI (prod)You can rungunicornmanually (locally) to test how things will run inside of the Docker image.XPUB_CONFIG_FILE=xpublish_host/examples/example.yamlgunicornxpublish_host.app:app-cxpublish_host/gunicorn.conf.pyIf you would like the metrics endpoint (/metrics) to function correctly when running throughgunicorn, you need to create a temporary directory for metrics and pass it in as thePROMETHEUS_MULTIPROC_DIRdirectory. This is handled automatically in the provided Docker image.mkdir-p/tmp/xpub_metricsPROMETHEUS_MULTIPROC_DIR=/tmp/xpub_metricsXPUB_CONFIG_FILE=xpublish_host/examples/example.yamlgunicornxpublish_host.app:app-cxpublish_host/gunicorn.conf.pyEither way,xpublishwill be running on port 9000 with (2) datasets:simpleandkwargs. You can access the instance athttp://[host]:9000/datasets/. Metrics are available athttp://[host]:9000/metricsDockerThe Docker image by default loads anxpublish-hostconfiguration file from/xpd/config.yaml, a datasets configuration object from/xpd/datasets.yaml, and an environmental variable file from/xpd/.env. You can change the location of those files by setting the env variablesXPUB_CONFIG_FILE,XPUBDC_CONFIG_FILE, andXPUB_ENV_FILESrespectively.# Using default config pathdockerrun--rm-p9000:9000-v"$(pwd)/xpublish_host/examples/example.yaml:/xpd/config.yaml"axiom/xpublish-host:latest# Using ENV variablesdockerrun--rm-p9000:9000-e"XPUB_CONFIG_FILE=/xpd/xpublish_host/examples/example.yaml"axiom/xpublish-host:latestEither way,xpublishwill be running on port 9000 with (2) datasets:simpleandkwargs. You can access the instance athttp://[host]:9000/datasets/.