Package: dtool-lookup-server
GitHub:https://github.com/jic-dtool/dtool-lookup-serverPyPI:https://pypi.python.org/pypi/dtool-lookup-serverFree software: MIT LicenseFeaturesUse a dataset UUID to lookup one more URIs where the dataset is storedSearch for datasets of interest using free text searchBuilt in support to manage usersBuilt in support to manage base URIsBuild in support to manage a user permissions on base URIsBuilt in support for authentication using JSON web tokensIntroductiondtoolis a command line tool for packaging
data and metadata into a dataset. A dtool dataset manages data and metadata
without the need for a central database.However, if one has to manage more than a hundred datasets it can be helpful
to have the datasets’ metadata stored in a central server to enable one to
quickly find datasets of interest.The dtool-lookup-server provides a web API for registering datasets’ metadata
and provides functionality to lookup, list and search for datasets.When managing many groups data it can be useful to ensure that users can only
access metadata associated with datasets stored in base URI’s that they have
been given access to. The dtool-lookup-server therefore provides means to
manage users, base URIs and users’ permissions on those base URIs.The dtool-lookup-server is consumed by thedtool-lookup-client, and thedtool-lookup-webapp.
Third party applications making use of the dtool-lookup-server have also been
created, notably theIMTEK-Simulation/dtool-lookup-gui.InstallationInstall the dtool lookup server:$ pip install dtool-lookup-serverFor a minimal setup, the lookup server requires search and retrieve plugins.
Pick search and retrieve plugins of your choice and install those. Here, thedtool-lookup-server-search-plugin-mongoanddtool-lookup-server-retrieve-plugin-mongoserve as default for demonstration:$ pip install dtool-lookup-server-search-plugin-mongo
$ pip install dtool-lookup-server-retrieve-plugin-mongoSetup and configurationConfigure the Flask appThe dtool lookup server is a Flask app. Flask needs to know where to look for
the app. One therefore needs to define theFLASK_APPenvironment variable:$ export FLASK_APP=dtool_lookup_serverConfigure search and retrieve pluginsThe dtool lookup server is agnostic of how descriptive data is stored and
searched. The implementation is delegated to search and retrieve plugins.
Refer to their documentation for details on their configuration.In the sample case ofdtool-lookup-server-search-plugin-mongoanddtool-lookup-server-retrieve-plugin-mongo, the same Mongo database
can be used for both search and information retrieval.Create a directory where the MongoDB data will be stored:$ mkdir dataStart Mongo DB, for example using docker:$ docker run -d -p 27017:27017 -v `pwd`/data:/data/db mongoConfigure the search plugin with:export SEARCH_MONGO_URI="mongodb://localhost:27017/"
export SEARCH_MONGO_DB="dtool_lookup_server"
export SEARCH_MONGO_COLLECTION="datasets"and the retrieve plugin with:export RETRIEVE_MONGO_URI="mongodb://localhost:27017/"
export RETRIEVE_MONGO_DB="dtool_lookup_server"
export RETRIEVE_MONGO_COLLECTION="datasets"This must happen before issuing anyflaskcommands as below.Configure the SQL databaseThe dtool lookup server stores administrative metadata in a SQL database.
By default it uses a SQLite database. However, this can be configured by
setting theSQLALCHEMY_DATABASE_URI, i.e using something along the lines of:export SQLALCHEMY_DATABASE_URI=mysql://username:password@server/dbImportantly, you will need an according Python connector for whatever SQL database
you decide to use, e.g.psycopg2for PostgreSQL ormysqlclientfor MySQL.For more information seeflask-SQLAlchemy.Versioning of the relational database is handled usingflask-MigratePopulate the SQL database with tables using the commands below:$ flask db init
$ flask db migrate
$ flask db upgradeConfigure a public and private key pairThe dtool lookup server implements authentication using JSON Web Tokens.
Private and public key can for example be generated with:openssl genrsa -out /path/to/private/jwt_key 2048
openssl rsa -in /path/to/private/jwt_key -pubout -outform PEM -out /path/to/public/jwt_key.pubIt is possible to delegate the generation of JSON Web Tokens to a different
service as long as one has access to the public key:export JWT_PUBLIC_KEY_FILE=~/.ssh/id_rsa.pubIf one has access to the private key as well one can use theflask user tokencommand line utility to generate a token for the user. To enable this
one has to set theJWT_PRIVATE_KEY_FILEenvironment variable:export JWT_PRIVATE_KEY_FILE=~/.ssh/id_rsaMac users be warned that the Mac’s implementationssh-keygenmay result in
files that do not adhere to the RFC standard. As such you may get a warning
along the lines of:ValueError: Could not deserialize key data.In this case you need to find a version ofssh-keygenthat generates files
that adhere to the RFC standard, the easiest is probably to generate them in Linux.Making use of JSON Web Tokens from a different serverWhen making use of JSON Web Tokens from a different server it may be easier to
use configure the server using the pubic key directly rather than the public key
file:export JWT_PUBLIC_KEY="ssh-rsa XXXXXX user@localhost"Inspecting the flask app configurationInspect the Flask app configuration with:$ flask config show
{
  "env": "production",
  "debug": false,
  "testing": false,
  "propagate_exceptions": null,
  ...
  "search_mongo_collection": "datasets",
  "search_mongo_db": "dtool_info",
  "search_mongo_uri": "mongodb://localhost:27017/",
  "retrieve_mongo_collection": "datasets",
  "retrieve_mongo_db": "dtool_info",
  "retrieve_mongo_uri": "mongodb://localhost:27017/",
  ...
}The output is JSON-formatted with lower-case keys and will include plugin
configuration parameters as well.Inspect the installed dtool-lookup-server components with:$ flask config versions
{
  "dtool_lookup_server": "0.17.2",
  "dtool_lookup_server_retrieve_plugin_mongo": "0.1.0",
  "dtool_lookup_server_search_plugin_mongo": "0.1.0"
}Starting the flask appThe Flask web app can be started using the command below:$ flask runPopulating the dtool lookup server using the CLIIndexing a base URIDatasets can be stored on filesystem and in object storage such as AWS S3.  In
an AWS S3 bucket datasets are stored in a flat structure and the bucket itself
is the base URI. To index all the datasets in the S3 bucket, the base URI, one
first needs to register it in the dtool lookup server:flask base_uri add s3://dtool-demoOne can then index it using the command:$ flask base_uri index s3://dtool-demo
Registered: s3://dtool-demo/8ecd8e05-558a-48e2-b563-0c9ea273e71e
Registered: s3://dtool-demo/907e1b52-d649-476a-b0bc-643ef769a7d9
Registered: s3://dtool-demo/af6727bf-29c7-43dd-b42f-a5d7ede28337
Registered: s3://dtool-demo/ba92a5fa-d3b4-4f10-bcb9-947f62e652db
Registered: s3://dtool-demo/c58038a4-3a54-425e-9087-144d0733387f
Registered: s3://dtool-demo/faa44606-cb86-4877-b9ea-643a3777e021It is possible to list all the base URIs registered in the dtool lookup server:$ flask base_uri list
[
  {
    "base_uri": "s3://dtool-demo",
    "users_with_search_permissions": [],
    "users_with_register_permissions": []
  }
]In the output above it is worth noting that there are two types of permissions
associated with a base URI. “Search” permissions allow a user to search for
datasets in a base URI. “Register” permissions allow a user to register a
dataset in the dtool lookup server if it is stored in the specific base URI.Adding a user and managing permissionsThe command below adds the userolssontto the dtool lookup server:$ flask user add olssontThe command below gives the userolssontsearch permissions on thes3://dtool-demobase URI:$ flask user search_permission olssont s3://dtool-demoThe command below gives the userolssontregister permissions on thes3://dtool-demobase URI:$ flask user register_permission olssont s3://dtool-demoCreating an admin userThe command below adds the useroverlord, with admin privileges, to the
dtool lookup server:$ flask user add --is_admin overlordGenerating a JSON Web Token for a registered userThe command below can be used to generate a token for a user to authenticate
with when using the web API:$ flask user token olssont
eyJhbGciOiJSUzI1NiIsInR5... (truncated)Listing the registered usersThe command below lists the users registered in the dtool lookup server:$ flask user list
[
  {
    "username": "olssont",
    "is_admin": false,
    "register_permissions_on_base_uris": [
      "s3://dtool-demo"
    ],
    "search_permissions_on_base_uris": [
      "s3://dtool-demo"
    ]
  },
  {
    "username": "overlord",
    "is_admin": true,
    "register_permissions_on_base_uris": [],
    "search_permissions_on_base_uris": []
  }
]Deleting a userThe command below can be used to delete a user:$ flask user delete overlordAdding and removing admin privileges from an existing userThe command below can be used to give an existing user admin privileges:$ flask user update --is_admin olssontThe command below can be used to remove admin privileges from an existing user:$ flask user update olssontThe dtool lookup server APIThe dtool lookup server makes use of the authorized header to pass through the
JSON web token for authorization. Below we create environment variables for the
token and the header used in thecurlcommands:$ TOKEN=$(flask user token olssont)
$ HEADER="Authorization: Bearer $TOKEN"Standard user usageLooking up URIs based on a dataset’s UUIDA dataset can be uniquely identified by it’s UUID (Universally Unique
Identifier). Below we create an environment variable with the UUID of a dataset
in the s3://dtool-demo bucket:$ UUID=8ecd8e05-558a-48e2-b563-0c9ea273e71eIt is possible to list all the location a dataset is located in using the
command below:$ curl -H $HEADER http://localhost:5000/dataset/lookup/$UUIDResponse content:[
  {
    "base_uri": "s3://dtool-demo",
    "name": "Escherichia-coli-ref-genome",
    "uri": "s3://dtool-demo/8ecd8e05-558a-48e2-b563-0c9ea273e71e",
    "uuid": "8ecd8e05-558a-48e2-b563-0c9ea273e71e"
  }
]Note that it is possible for a dataset to be registered in more than one base
URI. As such looking up a dataset by UUID can result in multiple hits.Summary information about datasetsAn overall summary of datasets accessible to a user can be accessed using the request below:$ curl -H "$HEADER" http://localhost:5000/dataset/summaryThe response will contain JSON content along the lines of:{
    "number_of_datasets": 3,
    "creator_usernames": ["queen"],
    "base_uris": ["s3://mr-men", "s3://snow-white"],
    "datasets_per_creator": {"queen": 3},
    "datasets_per_base_uri": {"s3://mr-men": 1, "s3://snow-white": 2},
    "tags": ["fruit", "veg"],
    "datasets_per_tag": {"fruit": 2, "veg": 1}
}Listing all datasetsAll the dataset’s that a user has permissions to search for can be listed using
the request below:$ curl -H "$HEADER" http://localhost:5000/dataset/listSome of the output of the command above is displayed below:[
  {
    "base_uri": "s3://dtool-demo",
    "name": "Escherichia-coli-ref-genome",
    "uri": "s3://dtool-demo/8ecd8e05-558a-48e2-b563-0c9ea273e71e",
    "uuid": "8ecd8e05-558a-48e2-b563-0c9ea273e71e"
  },
  ... (truncated)
  {
    "base_uri": "s3://dtool-demo",
    "name": "Escherichia-coli-reads-ERR022075",
    "uri": "s3://dtool-demo/faa44606-cb86-4877-b9ea-643a3777e021",
    "uuid": "faa44606-cb86-4877-b9ea-643a3777e021"
  }
]Searching for specific datasetsThe command below does a full text search for the word “microscopy” in the descriptive metadata:$ curl -H "$HEADER" -H "Content-Type: application/json"  \
    -X POST -d '{"free_text": "microscopy"}'  \
    http://localhost:5000/dataset/searchBelow is the result of this search:[
  {
    "base_uri": "s3://dtool-demo",
    "created_at": "1530803916.74",
    "creator_username": "olssont",
    "dtoolcore_version": "3.3.0",
    "frozen_at": "1536749825.85",
    "name": "hypocotyl3",
    "type": "dataset",
    "uri": "s3://dtool-demo/ba92a5fa-d3b4-4f10-bcb9-947f62e652db",
    "uuid": "ba92a5fa-d3b4-4f10-bcb9-947f62e652db"
  }
]Below is a JSON string specifying a more complex query that will search for
datasets with “apples” in the “s3://snow-white” bucket created by either
“grumpy” or “dopey”, and has both of the tags “fruit” and “veg”:{
    "base_uris": ["s3://snow-white"],
    "creator_usernames": ["grumpy", "dopey"],
    "free_text": "apples",
    "tags": ["fruit", "veg"]
}NoteThe search engine make use of “OR” logic for the items inbase_urisandcreator_usernameslists, but uses
“AND” logic for filtering the search based on the items
in thetagslist.Accessing a dataset’s readme, annotations and manifestThe command below retrieves the readme for the dataset with the
URIs3://dtool-demo/ba92a5fa-d3b4-4f10-bcb9-947f62e652db:$ curl -H "$HEADER" -H "Content-Type: application/json"  \
    -X POST -d  \
    '{"uri": "s3://dtool-demo/ba92a5fa-d3b4-4f10-bcb9-947f62e652db"}'  \
    http://localhost:5000/dataset/readmeThe command below retrieves the annotations for the dataset with the
URIs3://dtool-demo/ba92a5fa-d3b4-4f10-bcb9-947f62e652db:$ curl -H "$HEADER" -H "Content-Type: application/json"  \
    -X POST -d  \
    '{"uri": "s3://dtool-demo/ba92a5fa-d3b4-4f10-bcb9-947f62e652db"}'  \
    http://localhost:5000/dataset/annotationsThe command below retrieves the manifest for the dataset with the
URIs3://dtool-demo/ba92a5fa-d3b4-4f10-bcb9-947f62e652db:$ curl -H "$HEADER" -H "Content-Type: application/json"  \
    -X POST -d  \
    '{"uri": "s3://dtool-demo/ba92a5fa-d3b4-4f10-bcb9-947f62e652db"}'  \
    http://localhost:5000/dataset/manifestGetting information about one’s own permissionsA user can find out about his/her own permissions using the command below:$ curl -H "$HEADER" http://localhost:5000/user/info/olssontResponse content:{
  "is_admin": false,
  "register_permissions_on_base_uris": [
    "s3://dtool-demo"
  ],
  "search_permissions_on_base_uris": [
    "s3://dtool-demo"
  ],
  "username": "olssont"
}Data champion user usageA data champion is different from a regular user in that he/she has
“register” permissions on a base URI. This means that a data champion
can register metadata about a data to the dtool lookup server.Registering a datasetBelow is an example of how to register a dataset:$ DATASET_INFO='{
  "annotations": {},
  "base_uri": "s3://dtool-demo",
  "created_at": 1537802877.62,
  "creator_username": "olssont",
  "dtoolcore_version": "3.7.0",
  "frozen_at": 1537916653.7,
  "name": "Escherichia-coli-ref-genome",
  "readme": {
    "accession_id": "U00096.3",
    "description": "U00096.3 genome with Bowtie2 indices",
    "index_build_cmd": "bowtie2-build U00096.3.fasta reference",
    "index_builder": "bowtie2-build version 2.3.3",
    "link": "https://www.ebi.ac.uk/ena/data/view/U00096.3",
    "organism": "Escherichia coli str. K-12 substr. MG1655"
  },
  "type": "dataset",
  "uri": "s3://dtool-demo/8ecd8e05-558a-48e2-b563-0c9ea273e71e",
  "uuid": "8ecd8e05-558a-48e2-b563-0c9ea273e71e"
}'
$ curl -H $HEADER -H "Content-Type: application/json"  \
    -X POST -d $DATASET_INFO  \
    http://localhost:5000/dataset/registerThe required keys are defined in the variabledtool_lookup_server.utils.DATASET_INFO_REQUIRED_KEYS.Admin user usageThe administrative user can register new users, base URIs and manage who has
permissions to search for and register datasets. Below we update the header
to use the token from theoverlordadmin user:$ TOKEN=$(flask user token overlord)
$ HEADER="Authorization: Bearer $TOKEN"Listing registered usersTo list all the registered users an admin user can use the below:$ curl -H "$HEADER" http://localhost:5000/admin/user/listResponse content:[
  {
    "is_admin": false,
    "register_permissions_on_base_uris": [
      "s3://dtool-demo"
    ],
    "search_permissions_on_base_uris": [
      "s3://dtool-demo"
    ],
    "username": "olssont"
  },
  {
    "is_admin": true,
    "register_permissions_on_base_uris": [],
    "search_permissions_on_base_uris": [],
    "username": "overlord"
  }
]Registering usersAn admin user can register other users in batch:$ curl -H "$HEADER" -H "Content-Type: application/json"  \
    -X POST -d '[{"username": "admin", "is_admin": true}, {"username": "joe"}]'  \
    http://localhost:5000/admin/user/registerRegistering a base URIAn admin user can register a new base URI:$ curl -H "$HEADER" -H "Content-Type: application/json"  \
    -X POST -d '{"base_uri": "s3://another-bucket"}'  \
    http://localhost:5000/admin/base_uri/registerListing registered base URIsAn admin user can list all registered base URIs:$ curl -H "$HEADER" http://localhost:5000/admin/base_uri/listResponse content:[
  {
    "base_uri": "s3://dtool-demo",
    "users_with_register_permissions": [
      "olssont"
    ],
    "users_with_search_permissions": [
      "olssont"
    ]
  },
  {
    "base_uri": "s3://another-bucket",
    "users_with_register_permissions": [],
    "users_with_search_permissions": []
  }
]Updating the permissions on a base URIAn admin user can update the permissions on a base URI:$ curl -H "$HEADER" -H "Content-Type: application/json"  \
    -X POST -d '{
      "base_uri": "s3://another-bucket",
      "users_with_register_permissions": [
        "olssont"
      ],
      "users_with_search_permissions": [
        "olssont"
      ]
    }'  \
    http://localhost:5000/admin/permission/update_on_base_uriNote that the request below can be used to clear all existing permissions:$ curl -H "$HEADER" -H "Content-Type: application/json"  \
    -X POST -d '{
      "base_uri": "s3://another-bucket",
      "users_with_register_permissions": [],
      "users_with_search_permissions": []}'  \
    http://localhost:5000/admin/permission/update_on_base_uriGetting information about the permissions on a base URIAn admin user can get information about the permissions on a base URI:$ curl -H "$HEADER" -H "Content-Type: application/json"  \
    -X POST -d '{"base_uri": "s3://another-bucket"}'  \
    http://localhost:5000/admin/permission/infoResponse content:{
  "base_uri": "s3://another-bucket",
  "users_with_register_permissions": [],
  "users_with_search_permissions": []
}Querying server configurationThe request:$ curl -H "$HEADER" http://localhost:5000/config/infowill return the current server configuration with all keys in lowercase, i.e.:{
  "env": "production",
  "debug": false,
  "testing": false,
  "propagate_exceptions": null,
  ...
  "search_mongo_collection": "datasets",
  "search_mongo_db": "dtool_info",
  "search_mongo_uri": "mongodb://localhost:27017/",
  "retrieve_mongo_collection": "datasets",
  "retrieve_mongo_db": "dtool_info",
  "retrieve_mongo_uri": "mongodb://localhost:27017/",
  ...
}The request:$ curl http://localhost:5000/config/versionswill return all components, i.e. server core, search, retrieve
and extension plugins with their versions, i.e.:{
  "dtool_lookup_server": "0.17.2",
  "dtool_lookup_server_retrieve_plugin_mongo": "0.1.0",
  "dtool_lookup_server_search_plugin_mongo": "0.1.0"
}This request does not require any authorization.Creating a pluginIt is possible to create add plugins to this system. This is achieved by
creating a separate Python package containing aFlask blueprint.A basic plugin could consist of a single__init__and asetup.pyfile
in the directory structure below:.
|-- my_plugin
|   `-- __init__.py
`-- setup.pyThe__init__.pyfile could contain the code below.fromflaskimportBlueprintmy_plugin_bp=Blueprint('my_plugin',__name__,url_prefix="/my_plugin")@my_plugin_bp.route('/',methods=["GET"])defshow(page):return"My plugin content"The Flask blueprint object(s) need to be associated with thedtool_lookup_server.blueprintsentrypoint in the Python packagesetup.pyfile. Thesetup.pyfile would need to look something along the
lines of the below.fromsetuptoolsimportsetupsetup(name="my-plugin",packages=["my_plugin"],install_requires=["flask",],entry_points={"dtool_lookup_server.blueprints":["my_plugin=my_plugin:my_plugin_bp",],})Scaffold code for implementing a plugin, created byJohannes L. Hoermanncan be found indtool-lookup-server-plugin-scaffolding.Examples of actual plugins include:dtool-lookup-server-dependency-graph-plugindtool-lookup-server-plugin-scaffolding