Package: dtool-lookup-server-notification-plugin
GitHub:https://github.com/livMatS/dtool-lookup-server-notification-pluginPyPI:https://pypi.python.org/pypi/dtool-lookup-server-notification-pluginFree software: MIT LicenseFeaturesListen to elastic search notifications from an S3-compatible storage backendListen toS3 event notificationsfrom an S3-compatible storage backendIntroductiondtoolis a command line tool for packaging
data and metadata into a dataset. A dtool dataset manages data and metadata
without the need for a central database.However, if one has to manage more than a hundred datasets it can be helpful
to have the datasets’ metadata stored in a central server to enable one to
quickly find datasets of interest.Thedtool-lookup-serverprovides a web API for registering datasets’ metadata
and provides functionality to lookup, list and search for datasets.This plugin enables the dtool-lookup-server to listen to
notifications for the registration and deregistration of datasets.InstallationInstall the dtool lookup server dependency graph plugin$pipinstalldtool-lookup-server-notification-pluginSetup and configurationConfigure plugin behaviorThe plugin needs to know how to convert a bucket name into a base URI. The
environment variableDTOOL_LOOKUP_SERVER_NOTIFY_BUCKET_TO_BASE_URIis used
to specify that conversion, e.g.:DTOOL_LOOKUP_SERVER_NOTIFY_BUCKET_TO_BASE_URI={"bucket": "ecs://bucket"}It is also advisable to limit access to the notification listener to a certain
IP range. Use:DTOOL_LOOKUP_SERVER_NOTIFY_ALLOW_ACCESS_FROM=192.168.0.0/16to specify the allowed remote network. To specify a single IP, use:DTOOL_LOOKUP_SERVER_NOTIFY_ALLOW_ACCESS_FROM=192.168.1.1/32Configure elastic search integration in NetApp StorageGRIDCreate a new endpoint with URIhttps://myserver:myport/elastic-searchand URNarn:<mysite>:es:::<domain-name>/notify/allNote that<mysite>and<domain-name>can be chose arbitrarily.notify/allis appended to the URI and must point to the route of
the notify function.The bucket needs to be configured to support search integration. Use the
following XML template<MetadataNotificationConfiguration><Rule><ID>dtool</ID><Status>Enabled</Status><Prefix></Prefix><Destination><Urn>urn:mysite:es:::domain-name/notify/all</Urn></Destination></Rule></MetadataNotificationConfiguration>Configure webhook in minioThePublish Events to Webhook minio docswalks through the configuration for sending S3 event notifications to a webhook.
Assuming adtool-lookup-serverwith this plugin activated running athttp://dtool-lookup-server:5000, and your minio instance with a
buckettest-bucketathttps://s3server:9000, use# mc: minio clientmcconfighostadds3serverhttp://s3server:9000{admin_user}{admin_password}# Note that the endpoint must be reachable when configuring, otherwise minio will rejectmcadminconfigsets3server/notify_webhook:dtoolendpoint="http://dtool-lookup-server:5000/webhook/notify"mcadminservicerestarts3server# restart is necessary# Activate the actual notificationsmceventadds3server/test-bucketarn:minio:sqs::testbucket:dtool--event"put,delete"to configure a webhook endpoint identified bydtooland activateputanddeleteevent notification.
Choose the parameters for--event"put,delete"from minio’sSupported Bucket Evenets.Note that minio is very strict on whom they talk to. If yourdtool-lookup-servercommunicates viahttps, make sure that the server certificate usesSANsand that the
signing authority’s root certificate is available to minio. SeeInstall Certificates from Third-party CAsin the minio docs. Also assure all services are reachable by valid hostnames.
Within a containerized environment such as launched withdocker-compose,
host names containing underscores_may occur, but minio refuses to speak with
such.Configure webhook in NetApp StorageGRIDNetAPP StorageGRID is able to submit S3 event notifications when configured to
communicate with an SNS endpoint, refer to the according sections of theNetApp StorageGRID docsforcreating service endpointsandconfiguring event notifications.In short, create an endpoint`http://dtool-lookup-server:5000/webhook/notify`with a suitable URN, i.e.urn:dtool-lookup-server:sns:region:notify:all,
where you may pick all fields freely excepturnandsns.Next, enable event notifications for the desired bucket, i.e. for object creation events with a policy snippet like this:<NotificationConfiguration><TopicConfiguration><Id>Objectcreated</Id><Topic>urn:dtool-lookup-server:sns:region:notify:all</Topic><Event>s3:ObjectCreated:*</Event></TopicConfiguration></NotificationConfiguration>Querying server plugin configurationThe request$curl-H"$HEADER"http://localhost:5000/elastic-search/configwill return the current notification plugin configuration with all keys in lowercase{"allow_access_from":"0.0.0.0/0","bucket_to_base_uri":{"bucket":"s3://bucket"},"version":"0.1.0"}Seedtool_lookup_server_dependency_graph_plugin.config.Configfor more information.TestingLaunch a minimal mongodb instance with$cdtests/container&&docker-composeup-dand run tests from within repository root usingpytest--log-cli-level=DEBUGRefer to.github/workflows/test.ymlfor the recommended testing environment.Related repositoriesdtool-s3- storage broker interface to S3 object storagedtool-ecs- storage broker interface to ECS S3 object storage