Package: dtool-lookup-server-direct-mongo-plugin
GitHub:https://github.com/livMatS/dtool-lookup-server-direct-mongo-pluginPyPI:https://pypi.org/project/dtool-lookup-server-direct-mongo-plugin/Free software: MIT LicenseFeaturesQuery datasets via mongo languageFunnel datasets through aggregation pipelinesIntroductiondtoolis a command line tool for packaging
data and metadata into a dataset. A dtool dataset manages data and metadata
without the need for a central database.However, if one has to manage more than a hundred datasets it can be helpful
to have the datasets’ metadata stored in a central server to enable one to
quickly find datasets of interest.Thedtool-lookup-serverprovides a web API for registering datasets’ metadata
and provides functionality to lookup, list and search for datasets.This plugin allows to submit plain mongo queries and aggregation pipelines
directly to the lookup server.ConfigurationInform this plugin about the Mongo database to use by setting the environment
variables:export MONGO_URI="mongodb://localhost:27017/"
export MONGO_DB="dtool_lookup_server"
export MONGO_COLLECTION="metadata"If the Mongo search and retrieve plugins are used, then you may use the same
database, but must use a different collection.Useexport ALLOW_DIRECT_QUERY=true
export ALLOW_DIRECT_AGGREGATION=falseto enable or disable direct mongo query and aggregation on this plugin.ATTENTION: While direct queries respect user-wise access rights to database
entries on the lookup server level, there is no guarantee for aggregation
pipelines to do so per design. Don not enable direct aggregation in a production
environment.AuthenticationThe dtool lookup server makes use of the authorized header to pass through the
JSON web token for authorization. Below we create environment variables for the
token and the header used in the followingcurlcommand samples:$ TOKEN=$(flask user token test-user)
$ HEADER="Authorization: Bearer $TOKEN"Refer to the core dcumentation ofdtool-lookup-serverfor more information.Direct queryTo look for a sepcific fieldkey2: 42in a dataset’s README.yml (provided
the file is properly YAML-formatted), use$ curl -H “$HEADER” -H “Content-Type: application/json” -X POST-d ‘{“query”: {“readme.key2”: 42}}’http://localhost:5000/mongo/queryResponse content:[
  {
    "base_uri": "s3://test-bucket",
    "created_at": 1683797360.056,
    "creator_username": "jotelha",
    "dtoolcore_version": "3.18.2",
    "frozen_at": 1683797362.855,
    "name": "test_dataset_2",
    "number_of_items": 1,
    "size_in_bytes": 19347,
    "tags": [],
    "type": "dataset",
    "uri": "s3://test-bucket/26785c2a-e8f8-46bf-82a1-cec92dbdf28f",
    "uuid": "26785c2a-e8f8-46bf-82a1-cec92dbdf28f"
  }
]Direct aggregationThe following example of an aggregation pipeline identifies
and counts instances of the same dataset at different base URIs:$ curl -H "$HEADER" -H "Content-Type: application/json" -X POST \
    -d '{"aggregation": [
            {
                "$sort": {"base_uri": 1}
            }, {
                "$group":  {
                    "_id": "$name",
                    "count": {"$sum": 1},
                    "available_at": {"$push": "$base_uri"}
                }
            }, {
                "$project": {
                    "name": "$_id",
                    "count": true,
                    "available_at": true,
                    "_id": false
                }
            }, {
                "$sort": {"name": 1}
            }
        ]
    }' http://localhost:5000/mongo/aggregateResponse content:[
  {
    "available_at": [
      "s3://test-bucket"
    ],
    "count": 1,
    "name": "test_dataset_1"
  },
  {
    "available_at": [
      "s3://test-bucket",
      "smb://test-share"
    ],
    "count": 2,
    "name": "test_dataset_2"
  }
]TestingRunning unit tests withpytestrequires a healthy lookup server installation
and the availability of required services such as databases. Please refer to
the coredtool-lookup-serverfor setup instructions.