Package: intake-ids
intake-idsThis is anintakeplugin that provides drivers and a catalog forlocal artifactsin anInternational Data Spacesconnector. It automates contract negotiation, deprecation and re-negotiation processes needed for data access from connectors.A catalog provides a list of processable Resources in an IDS Connector. A Resource is only included (processable) in the catalog if it has at least one Representation with a supported mimetype. These are currently:text/csvFuture formats could include Parquet and JSON.Installationintake-idsis published onPyPI.
You can install it by running the following in your terminal:pipinstallintake-idsYou can test the functionality by opening the example notebook in theexamples/directory.UsageThe package can be imported usingfromintake_idsimportConnectorCatalogLoading a catalogYou can load from a remote IDS Connectorproviderby providing the URLs for both the local connector and the remote connectorproviderand the authentication tuple for the local connector:provider_url="https://provider:8080"consumer_url="https://consumer:8080"catalog=ConnectorCatalog(provider_url=provider_url,consumer_url=consumer_url,name="testcat",auth=("admin","password"))len(list(catalog))By default, ConnectorCatalog will combine all "IDS Catalog"s in the connector into one catalog. You can select the "IDS Catalog" usingcatalog_id.catalog_id="https://provider:8080/api/catalogs/eda0cda2-10f2-4b39-b462-5d4f2b1bb758"catalog=ConnectorCatalog(provider_url=provider_url,consumer_url=consumer_url,catalog_id=catalog_id,name="testcat",auth=("admin","password"))You can display the resources (items) in the catalogforentry_id,entryincatalog.items():display(entry)If the catalog has too many entries to comfortably print all at once,
you can narrow it by searching for a term (e.g. 'motion'):forentry_id,entryincatalog.search('motion').items():display(entry)Loading an artifactOnce you have identified a resource/representation you want to use, you can load it into a dataframe usingread()orread_chunked():df=pd.concat(entry.read_chunked())ordf=entry.read()This will automatically load that dataset into the specified container for the driver for the entry.Command line toolsThe plugin provides theintake-ids-periodic-cleanupscript for periodic validation and cleanup of the cache. You can use the following crontab to run this script every 5 minutes.*/5 * * * * $HOME/.local/bin/intake-ids-periodic-cleanupDetailsProcessable (Resource, Representation)-pair entries are then included in the catalog and matched to an available driver specialized for it's type.
Alongside the entries themselves are also metadata from the Representation and cursory information about the usage policy (Rules) and access rights (ContractOffer) of the Resource.Drivers and Agreement cachingDrivers for entries allow for reading Artifacts by sorting through all the ContractOffers available for the Resource and negotiating an Agreement using one of them. If no valid ContractOffer exists for the Resource, an error is thrown. If otherwise multiple valid ContractOffers exist for the Resource, a preferable (artifact-cacheable) offer is selected. ContractAgreements are cached on the system and reused the next time the Resource is read/requested by the user if they are still valid without negotiating a new Agreement. If the cached Agreement is not valid (eg. expired) at any point, it (and all other associated items, including it's artifacts, see the next section) is removed from the cache and the process for agreement negotiation takes place again.Currently following drivers exist:drivermimetypescontainerConnectorCSVtext/csvpandas.dataframeArtifact cachingDepending on the determined usage control pattern (found out by inspecting Rules in the Agreement) for the Resource they are part of, some Artifacts can be cached by the driver in the local filesystem and used directly the next time they are requested. Before each read from the cache, the driver checks the continued validity of the Agreement and evalutes the usage control restrictions, clearing the agreement/artifact from the cache if the results come negative.Cache support for usage patternsFollowing the specifications of theIDSA Position Paper about Usage Control, the IDS defines 21 policy classes.
The IDS Dataspace Connectorcurrently implements 9of these.
Of the remaining 9 usage patterns, intake-ids offers artifact- and agreement-caching (full caching) support for 4 patterns, remaining 5 are agreement-cached only.No.Titleartifact cachingagreement cachingsupport by IDS Dataspace Connectordescription1Allow the Usage of the Dataxxxprovides data usage without any restrictions2Connector-restricted Data Usagexxallows data usage for a specific connector3Application-restricted Data Usage4Interval-restricted Data Usagexxxprovides data usage within a specified time interval5Duration-restricted Data Usagexxxallows data usage for a specified time period6Location Restricted Policy7Perpetual Data Sale (Payment once)8Data Rental (Payment frequently)9Role-restricted Data Usage10Purpose-restricted Data Usage Policy11Event-restricted Usage Policy12Restricted Number of Usagesxxallows data usage for n times13Security Level Restricted Policyxxallows data access only for connectors with a specified security level14Use Data and Delete it Afterxxxallows data usage within a specified time interval with the restriction to delete it at a specified time stamp15Modify Data (in Transit)16Modify Data (in Rest)17Local Loggingxxallows data usage and sends logs to a specified Clearing House18Remote Notificationsxxallows data usage and sends notification message19Attach Policy when Distribute to a Third-party20Distribute only if Encrypted21State Restricted PolicyPeriodic policy validationThe plugin includes a console scriptintake-ids-periodic-cleanupthat evaluates all stored Agreements and usage control restrictions and removes invalid ones from the cache. It is provided as a command line tool and installed to your$PATHif the plugin was downloaded via pip/installed with setuptools. You can use this tool in a cronjob to make sure that usage policies are upheld even when the plugin is not in use.Requirementsinstall_requires =
    intake>=0.6.5
    pandas>=1.4.3
    requests>=2.28.1
    pydantic>=1.10.1
    isodate>=0.6.1
    appdirs>=1.4.4
python_requires = >=3.10