Package: hoborequest
HOBO-requestA Kubernetes-ready container application for integrating HOBOlink loggers with HSDS. This application
was developed in partnership with the HDF Group with NSF support for the UVA-ARC project (grant #2022639).Getting startedHOBO-request provides an interface to HOBOlink loggers via a public API that is provided by Onset.
It can be used with HSDS for the purposes of large-scale environmental data analysis, or as a stand-alone
tool for the purposes of testing. It can also be easily extended to be used with different APIs and sensor
networks.You can install the HOBO-request package or git clone this repository. For installing the pip package,
just create a virtual environmental and run:$python3-mvenvhobo-request
$sourcehobo-request/bin/activate
$pipinstallhoborequestYou can also take the development path way and work directly with the source code.In order to integrate HOBO-request with data loggers, a few libraries / modules need to be installed.
You can proceed by installing the dependencies described inrequirements.txt.$python3-mvenvhobo-request$sourcehobo-request/bin/activate
$pipinstall-rrequirements.txtHOBO-request is now ready to be used, but first it must be configured!ConfigurationFirst, copyconf/hobo-connect.conf-SAMPLEtoconf/hobo-connect.conf.
Then, edit 'conf/hobo-connect.conf' and set the following parameters:user_id: your Onset user IDclient_id: your Onset client IDclient_secret: API secret key obtained from Onsetstart_date_time: for precise time slicing, the start time in UTC, format: 2012-01-01 23:00:00end_date_time: same as above, the end time in UTC, format: 2021-01-12 11:00:00polling_interval_minutes: if you want to set a regular polling interval, leave start & end time fields empty.
Alternatively you can also set a start time, but leave end time emptyPlease observe that timestamps are always assumed to be in UTC. For the purposes of data storage, it is more
convenient to avoid timezones and daylight-saving differences, and leave to the analyst to manipulate
timestamps as he/she/they see fit.In addition to the configuration of your Onset credentials, you also need to set your HSDS parameters:hsds_filename: your HDF data store namehsds_endpoint: your HSDS endpointPlease observe that you can also use HOBO-request locally without HSDS for testing purposes. In that case, you only
need to sethsds_filenameto point to the h5 file in your local filesystem.Finally, you must set the repository where your logger and sensor metadata are registered. This is a crucial step
because HOBO-request depends on this config to update your metadata attributes for you. You must set the following
fields:meta_repo: git repository that contains your deployment metadata filesmeta_local_dir: local directory where you want to store the repo temporarilymeta_root_path: repository directory where the HDF root file metadata file ismeta_loggers_dir: repository directory where the loggers metadata files aremeta_sensors_dir: repository directory where the sensors metadata files areOnce this is done, we can move on to the generation of your data store.Creating your HDF data store with metadataHOBO-request reads YAML files with metadata organized in three levels:HDF root metadata(attributes)HOBO loggers(attributes)HOBO sensors(attributes)It is necessary, therefore, to create YAML files following the scheme we describe in theconf/andconf/sensors/templatesdirectories. For example:uva-arc.yaml: provides an example of how to set root attributes in your HDF fileloggers/: provides examples for the configuration of HOBOLink loggers / stationssensors/templates: provides examples for the configuration of HOBOLink sensorsOnce you are done creating your YAML files, you can load them in your newly created HDF file.
Change directory tosrc/, then runhobo-config.py:To generate your HDF file with root attributes:./hobo-config.pytest.h5../conf/uva-arc.yamlOr, if you are using the pip package, you can just run:$hoboconfigtest.h5../conf/uva-arc.yamlPlease note that you have to change the PATH for the YAML file in the command above.
You can find real examples for stations and sensors inthis repository.To load your logger and sensor metadata, you can type:./hobo-config.pytest.h5/tmp/deployment-metadata/alaska/loggers/*
./hobo-config.pytest.h5/tmp/deployment-metadata/alaska/sensors/*Or... if you are using the pip packagehoborequest:$hoboconfigtest.h5/tmp/deployment-metadata/alaska/loggers/*
$hoboconfigtest.h5/tmp/deployment-metadata/alaska/sensors/*Please note that it is necessary to run this step only once when creating the HDF data store.
After you load the HDF file onto HSDS, HOBO-request will automatically pull all the changes
from thegitrepository containing your YAML files and load all the metadata information
for you. There is no need to usehobo-configmanually again.Running HOBO-request to consume sensor dataNow that everything is in place, you have the choice of running HOBO-request locally for tests
using a regular HDF file (and h5py) or run the HDF data store on HSDS (with h5pyd). This is
done automagically for you, so you just have to point in your config file if you want to use
a regular file or interface with HSDS to feed data to your HDF data store.You can start HOBO-request with a simple command:$./hobo-connect.pyOr, if you are running the pip package:$hoboconnectEt... voilà! The application will return the following message as soon as it is done processing
the data:[HOBO-connect]:dataprocessingcompletedGenerating a container for HOBO-requestWe have provided scripts and deployment files to generate a container for HOBO-request, plus
documentation on how to run HSDS on k3s (or microk8s) for local testing.You can run the following commands to build your container and run it:$./build.sh
$./docker_run.shThe directoryk8s/contains the deployment files and scripts for running HOBO-request on k8s or microk8s.May the source be with you in your environmental studies!LicensingSee LICENSE and AUTHORS files for details.