Package: dtool-lookup-api
Python API for interacting with dtool lookup server.This package offers a class-based asynchronous lookup API withindtool_lookup_api.core.LookupClient,
a simple class-less wrapper around it atdtool_lookup_api.asynchronous,
and a synchronous interface on top atdtool_lookup_api.synchronous.Direct imports of utility functions fromdtool_lookup_apiin the examples below forward to the synchronous API variant.InstallationTo install the dtool_lookup_api package.pipinstalldtool_lookup_apiThis package depends on adtool-lookup-serverinstance to talk to.ConfigurationThe API needs to know the URL of the lookup serverexportDTOOL_LOOKUP_SERVER_URL=https://localhost:5000You may also need specify an access token generated on the serverexportDTOOL_LOOKUP_SERVER_TOKEN=$(flaskusertokentestuser)Instead of specifying the access token directly, it is also possible to provideexportDTOOL_LOOKUP_SERVER_TOKEN_GENERATOR_URL=https://localhost:5001exportDTOOL_LOOKUP_SERVER_USERNAME=my-usernameexportDTOOL_LOOKUP_SERVER_PASSWORD=my-passwordfor the API to request a token. This, however, is intended only for testing
purposes and strongly discouraged in a production environment, as your password
would reside within environment variables or the dtool config file as clear text.Our recommended setup is a combination ofexportDTOOL_LOOKUP_SERVER_URL=https://localhost:5000exportDTOOL_LOOKUP_SERVER_TOKEN_GENERATOR_URL=https://localhost:5001in the config. If used interactively, the API will then ask for your
credentials at the first interaction and cache the provided values for this
session, i.e.In [1]: from dtool_lookup_api import query
   ...: res = query(
   ...:     {
   ...:         'readme.owners.name': {'$regex': '^Testing User$'},
   ...:     }
   ...: )
Authentication URL https://localhost:5001/token username:my-username
Authentication URL https://localhost:5001/token password:

In [2]: res
Out[2]:
[{'base_uri': 'smb://test-share',
  'created_at': 'Sun, 08 Nov 2020 18:38:40 GMT',
  'creator_username': 'jotelha',
  'dtoolcore_version': '3.17.0',
  'frozen_at': 'Wed, 11 Nov 2020 17:20:30 GMT',
  'name': 'simple_test_dataset',
  'tags': [],
  'type': 'dataset',
  'uri': 'smb://test-share/1a1f9fad-8589-413e-9602-5bbd66bfe675',
  'uuid': '1a1f9fad-8589-413e-9602-5bbd66bfe675'}]

In [3]: from dtool_lookup_api import all
   ...: all()
Out[4]:
[{'base_uri': 'smb://test-share',
  'created_at': 1604860720.736269,
  'creator_username': 'jotelha',
  'frozen_at': 1604921621.719575,
  'name': 'simple_test_dataset',
  'uri': 'smb://test-share/1a1f9fad-8589-413e-9602-5bbd66bfe675',
  'uuid': '1a1f9fad-8589-413e-9602-5bbd66bfe675'}]Credentials caching and interactive prompting are turned off withIn [1]: import dtool_lookup_api.core.config
   ...: dtool_lookup_api.core.config.Config.interactive = False
   ...: dtool_lookup_api.core.config.Config.cache = False

In [2]: from dtool_lookup_api import all
   ...: all()
...
RuntimeError: Authentication failedFor testing purposes, it is possible to disable SSL certificates validation withexportDTOOL_LOOKUP_SERVER_VERIFY_SSL=falseAs usual, these settings may be specified within the default dtool configuration
file as well, i.e. at~/.config/dtool/dtool.json{"DTOOL_LOOKUP_SERVER_TOKEN_GENERATOR_URL":"https://localhost:5001/token","DTOOL_LOOKUP_SERVER_URL":"https://localhost:5000"}List all datasetsTo list all registered datasetsIn [1]: from dtool_lookup_api import all
   ...: res = all()

In [2]: res
Out[2]:
[{'base_uri': 'smb://test-share',
'created_at': 1604860720.736269,
'creator_username': 'jotelha',
'frozen_at': 1604921621.719575,
'name': 'simple_test_dataset',
'uri': 'smb://test-share/1a1f9fad-8589-413e-9602-5bbd66bfe675',
'uuid': '1a1f9fad-8589-413e-9602-5bbd66bfe675'}]Looking up datasets by UUIDTo lookup URIs from a dataset UUID within PythonIn [1]: from dtool_lookup_api import lookup
   ...: uuid = "1a1f9fad-8589-413e-9602-5bbd66bfe675"
   ...: res = lookup(uuid)

In [2]: res
Out[2]:
[{'base_uri': 'smb://test-share',
  'created_at': 1604860720.736269,
  'creator_username': 'jotelha',
  'frozen_at': 1604921621.719575,
  'name': 'simple_test_dataset',
  'uri': 'smb://test-share/1a1f9fad-8589-413e-9602-5bbd66bfe675',
  'uuid': '1a1f9fad-8589-413e-9602-5bbd66bfe675'}]Full text searchingFull text search for the word “test”In [1]: from dtool_lookup_api import search
    ...: res = search("test")

In [2]: res
Out[2]:
[{'base_uri': 'smb://test-share',
  'created_at': 1604860720.736,
  'creator_username': 'jotelha',
  'dtoolcore_version': '3.17.0',
  'frozen_at': 1605027357.308,
  'name': 'simple_test_dataset',
  'tags': [],
  'type': 'dataset',
  'uri': 'smb://test-share/1a1f9fad-8589-413e-9602-5bbd66bfe675',
  'uuid': '1a1f9fad-8589-413e-9602-5bbd66bfe675'}]ManifestRequest the manifest of a particular dataset by URIIn [1]: from dtool_lookup_api import manifest
   ...: uri = 'smb://test-share/1a1f9fad-8589-413e-9602-5bbd66bfe675'
   ...: res = manifest(uri)

In [2]: res
Out[2]:
{'dtoolcore_version': '3.17.0',
 'hash_function': 'md5sum_hexdigest',
 'items': {'eb58eb70ebcddf630feeea28834f5256c207edfd': {'hash': '2f7d9c3e0cfd47e8fcab0c12447b2bf0',
   'relpath': 'simple_text_file.txt',
   'size_in_bytes': 17,
   'utc_timestamp': 1605027357.284966}}}ReadmeRequest the readme cotent of a particular dataset by URIIn [1]: from dtool_lookup_api import readme
    ..: res = readme('smb://test-share/1a1f9fad-8589-413e-9602-5bbd66bfe675')

In [2]: res
Out[2]:
{'creation_date': '2020-11-08',
'description': 'testing description',
'expiration_date': '2022-11-08',
'funders': [{'code': 'testing_code',
 'organization': 'testing_organization',
 'program': 'testing_program'}],
'owners': [{'email': 'testing@test.edu',
 'name': 'Testing User',
 'orcid': 'testing_orcid',
 'username': 'testing_user'}],
'project': 'testing project'}Direct mongo language queriesTo list all datasets at a certain base URI with their name matching some regular
expression pattern, send a direct mongo language query to the server withIn [15]: from dtool_lookup_api import query
    ...: res = query(
    ...:     {
    ...:         'base_uri': 'smb://test-share',
    ...:         'name': {'$regex': 'test'},
    ...:     }
    ...: )

In [16]: res
Out[16]:
[{'base_uri': 'smb://test-share',
'created_at': 'Sun, 08 Nov 2020 18:38:40 GMT',
'creator_username': 'jotelha',
'dtoolcore_version': '3.17.0',
'frozen_at': 'Tue, 10 Nov 2020 16:55:57 GMT',
'name': 'simple_test_dataset',
'tags': [],
'type': 'dataset',
'uri': 'smb://test-share/1a1f9fad-8589-413e-9602-5bbd66bfe675',
'uuid': '1a1f9fad-8589-413e-9602-5bbd66bfe675'}]It is possible to search readme content viaIn [21]: from dtool_lookup_api import query
    ...: res = query(
    ...:     {
    ...:         'readme.owners.name': {'$regex': '^Testing User$'},
    ...:     }
    ...: )

In [22]: res
Out[22]:
[{'base_uri': 'smb://test-share',
  'created_at': 'Sun, 08 Nov 2020 18:38:40 GMT',
  'creator_username': 'jotelha',
  'dtoolcore_version': '3.17.0',
  'frozen_at': 'Tue, 10 Nov 2020 16:55:57 GMT',
  'name': 'simple_test_dataset',
  'tags': [],
  'type': 'dataset',
  'uri': 'smb://test-share/1a1f9fad-8589-413e-9602-5bbd66bfe675',
  'uuid': '1a1f9fad-8589-413e-9602-5bbd66bfe675'}]This requires the server-sidedtool-lookup-server-direct-mongo-plugin.TODO: Response from server-side direct mongo plugin still yields dates as strings.
Fix withinhttps://github.com/IMTEK-Simulation/dtool-lookup-server-direct-mongo-plugin.Usage on Jupyter notebookThe current implementation viaasgiref.async_to_sync(https://github.com/django/asgiref)
hinders the use of the synchronous interface within Jupyter notebooks.
Directly use the asynchronous api insteadimportdtool_lookup_api.asynchronousasdlres=awaitdl.query({'base_uri':'smb://test-share','name':{'$regex':'test'},})The drawback of the above approach is that the same code doesn’t work in python and in jupyter (awaitoutsite of a function is a syntax error in non-interactive python context).
The code below can be executed in both contexts:importdtool_lookup_api.asynchronousasdlifasyncio.get_event_loop().is_running():# then we are in jupyter notebook# this allows nested event loops, i.e. calls to asyncio.run inside the notebook as well# This way, the same code works in notebook and pythonimportnest_asyncionest_asyncio.apply()defquery(query_dict):returnasyncio.run(dl.query(query_dict))query({'base_uri':'smb://test-share','name':{'$regex':'test'},})Seehttps://github.com/jupyter/notebook/issues/3397#issuecomment-419386811,https://ipython.readthedocs.io/en/stable/interactive/autoawait.htmlTestingTests require the presence of a working dtool lookup server ecosystem.
The testing workflow within.github/workflows/test.ymluses thedtool-lookup-server-container-compositionto provide a mock ecosystem. It is possible to run the workflow locally
with the help ofdockerandact.Afterinstalling and configuring act, runact-Pubuntu-latest=catthehacker/ubuntu:full-latest-sGITHUB_TOKEN=$GITHUB_TOKEN-W.github/workflows/test.yml--bindfrom within this repository.$GITHUB_TOKENmust hold a validaccess token.
The user must be member of thedockergroup.
The--bindoption avoids quirky permission errors by running
the test in the current directory. This will however result in the
local creation of two subdirectoriesdtool-lookup-server-container-compositionandworkflowduring testing, which may be removed withrm-rfdtool-lookup-server-container-compositionsudorm-rfworkfloweventually. All tests have been confirmed to work with thecatthehacker/ubuntu:full-20.04runner.