Package: oteapi-core
Open Translation Environment (OTE) API CoreFramework for accessing data resources, mapping data models, describing the data to ontologies and perform data transformationsWe highly recommend reading this page inthe official documentation.Important: As ofv0.6.0, OTEAPI Core is no longer compatible with pydantic v1, but only with pydantic v2.
For more information about migrating your plugin repository to pydantic v2, see thepydantic documentation's migration guide.Until the end of 2023, pydantic v1 will still be supported with security updates, but no new features will be added.
To keep using pydantic v1, one should use thev0.5.xversions of OTEAPI Core.About OTEAPI CoreOTEAPI Core provides the core functionality of OTEAPI, which stands for theOpen Translation Environment API.It uses thestrategysoftware design pattern to implement a simple and easy to extend access to a large range of data resources.
Semantic interoperability is supported via mapping of data models describing the data to ontologies.
A set of strategy interfaces that can be considered abstract classes for the implementation of strategies, and data models used in their configuration, are provided.
This repo also contains implementations for several standard strategies, e.g., downloading files, parsing Excel documents.
Transformations, mainly intended to transform data between representations, are also supported, but transformations can also be used for running simulations in a simple workflow.OTEAPI Core includes:A set of standard strategies;A plugin system for loading the standard strategies, as well as third party strategies;Data models for configuring the strategies;A Python library, through which the data can be accessed; andAn efficient data cache module that avoids downloading the same content several times.Types of strategiesDownload strategyDownload strategy patterns use a given protocol to download content into the data cache.
They are configured with theResourceConfigdata model, using the scheme of thedownloadUrlfield for strategy selection.
Theconfigurationfield can be used to configure how the downloaded content is stored in the cache using theDownloadConfigdata model.Standard downloaded strategies:file,https,http,sftp,ftpParse strategyParse strategy patterns convert content from the data cache to a Python dict.
Like download strategies, they are configured with theResourceConfigdata model, using themediaTypefield for strategy selection.
Additional strategy-specific configurations can be provided via theconfigurationfield.Standard parse strategies:application/json,image/jpg,image/jpeg,image/jp2,image/png,image/gif,image/tiff,image/eps,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,application/vnd.sqlite3Resource strategyResource strategy patterns can retrieve/upload data to external data services.
They are configured with theResourceConfigdata model, using the scheme of theaccessUrlandaccessServicefields.
The scheme of theaccessUrlis used for strategy selection.Mapping strategyStrategies for mapping fields/properties in data models to ontological concepts.Filter strategyFilter strategies can update the configuration of other strategies.
They can also update values in the data cache.Standard filter strategies:filter/crop,filter/sqlFunction strategyFunction strategies are synchronous transformations that (normally) run directly on the server hosting the OTE service.Transformation strategyTransformation strategies are a special form of a function strategy intended for long-running transformations.
In this sense, they represent asynchronous functions running in the background or on external resources.Standard transformation strategies:celery/remoteThe transformation strategy has consolidated the execution of the
transformation with theget()method to unify the strategy interfaces.get()is intended to start an asynchronous process and return atask_idwhich can be queried using thestatus()method (outside of a pipeline).Entry points for pluginsThe way strategies are registered and found is throughentry points.Special group names allow understanding the strategy type and the entry point values allow understanding of what kind of strategy a specific class implements.
A full overview of recognized entry point group names can be seen inTable of entry point strategies.Defining entry pointsIn the following examples, let's imagine we have a package importable in Python throughmy_pluginand contains two download strategies and a single parse strategy:A peer-2-peer download strategy, implemented in a class namedPeer2PeerDownloadimportable frommy_plugin.strategies.download.peer_2_peer.A MongoDB download strategy, implemented in a class namedMongoRetrieveimportable frommy_plugin.strategies.mongo.A MongoDB parse strategy, implemented in a class namedMongoParseimportable frommy_plugin.strategies.mongo.There are now various different ways to let the Python environment know of these strategies through entry points.setup.pyIn the package'ssetup.pyfile, one can specify entry points.
Here, an example snippet is shown usingsetuptools:# setup.pyfromsetuptoolsimportsetupsetup(# ...,entry_points={"oteapi.download":["my_plugin.p2p = my_plugin.strategies.download.peer_2_peer:Peer2PeerDownload","my_plugin.mongo = my_plugin.strategies.mongo:MongoRetrieve",],"oteapi.parse":["my_plugin.application/vnd.mongo+json = my_plugin.strategies.mongo:MongoParse",]},)YAML/JSON custom filesUse custom files that are later parsed and used in asetup.pyfile.entry_points:oteapi.download:-"my_plugin.p2p=my_plugin.strategies.download.peer_2_peer:Peer2PeerDownload"-"my_plugin.mongo=my_plugin.strategies.mongo:MongoRetrieve"oteapi.parse:-"my_plugin.application/vnd.mongo+json=my_plugin.strategies.mongo:MongoParse"{"entry_points":{"oteapi.download":["my_plugin.p2p = my_plugin.strategies.download.peer_2_peer:Peer2PeerDownload","my_plugin.mongo = my_plugin.strategies.mongo:MongoRetrieve"],"oteapi.parse":["my_plugin.application/vnd.mongo+json = my_plugin.strategies.mongo:MongoParse"]}}setup.cfg/pyproject.tomlA more modern approach is to usesetup.cfgorpyproject.toml.[options.entry_points]oteapi.download=my_plugin.p2p=my_plugin.strategies.download.peer_2_peer:Peer2PeerDownloadmy_plugin.mongo=my_plugin.strategies.mongo:MongoRetrieveoteapi.parse=my_plugin.application/vnd.mongo+json=my_plugin.strategies.mongo:MongoParseSyntax and semanticsAs seen above, there are a few different syntactical flavors of how to list the entry points.
However, the "value" stays the same throughout.General Python entry pointsThe general syntax for entry points is based oninifiles and parsed using the built-inconfigparsermodule describedhere.
Specifically for entry points the nomenclature is the following:[options.entry_points]GROUP=NAME=VALUETheVALUEis then further split into:PACKAGE.MODULE:OBJECT.ATTRIBUTE [EXTRA1, EXTRA2].OTEAPI strategy entry pointsFrom the general syntax outlined above, OTEAPI Core then implements rules and requirements regarding the syntax for strategies.A classMUSTbe specified (as anOBJECT).TheNAMEMUSTconsist of exactly two parts:PACKAGEand strategy type value in the form ofPACKAGE.STRATEGY_TYPE_VALUE.TheGROUPMUSTbe a valid OTEAPI entry point group, seeTable of entry point strategiesfor a full list of valid OTEAPI entry point group values.To understand what the strategy type value should be, seeTable of entry point strategies.Table of entry point strategiesStrategy Type NameStrategy Type ValueEntry Point GroupDocumentation ReferenceDownloadschemeoteapi.downloadDownload strategyFilterfilterTypeoteapi.filterFilter strategyFunctionfunctionTypeoteapi.functionFunction strategyMappingmappingTypeoteapi.mappingMapping strategyParsemediaTypeoteapi.parseParse strategyResourceaccessServiceoteapi.resourceResource strategyTransformationtransformationTypeoteapi.transformationTransformation strategyOther OTEAPI-related repositoriesOTEAPI Services- a RESTful interface to OTEAPI CoreOTELib- a Python interface to OTEAPI ServicesOTEAPI Plugin Template- acookiecuttertemplate for OTEAPI PluginsInstallationOTEAPI Core can be installed with:pipinstalloteapi-coreFor developersIf you want to install OTEAPI Core to have a developer environment, please clone down the repository from GitHub and install:gitclonehttps://github.com/EMMC-ASBL/oteapi-core/path/to/oteapi-core
pipinstall-U--upgrade-strategy=eager-e/path/to/oteapi-core[dev]Note,/path/to/oteapi-corecan be left out of the first line, but then it must be updated in the second line, either to./oteapi-core/oteapi-coreor.if youcdinto the generated folder wherein the repository has been cloned.The--upgrade-strategy=eagerpart can be left out.
We recommend installing within a dedicated virtual environment.To test the installation, you can run:cd/path/to/oteapi-core
pytestIf you run into issues at this stage, pleaseopen an issue.Using Docker with PostgreSQLDocker is an effective tool for creating portable, isolated environments for your applications.
Here's an example of setting up a PostgreSQL instance using Docker:Create a Docker volume: Docker volumes enable data to persist across uses of Docker containers.
In this context, we create a volume called pgdata to store database data.dockervolumecreatepgdataStart a Docker container: Use thedocker runcommand to initiate a new Docker container using the postgres image.
Here's a breakdown of the options used in the command:-d: Runs the container in the background (detached mode), freeing up your terminal.--name postgres: Names the container postgres, allowing it to be referenced in future Docker commands.-e POSTGRES_PASSWORD=postgres: Sets an environment variable in the container to specify the PostgreSQL database password as postgres.-p 5432:5432: Maps port 5432 of the container to port 5432 of the host machine, letting applications on the host connect to the PostgreSQL database in the container.-v pgdata:/var/lib/postgresql/data: Mounts the pgdata volume at the path /var/lib/postgresql/data inside the container, which is the storage location for PostgreSQL data files.--restart always: Ensures the container restarts whenever it stops, unless it is manually stopped, in which case it only restarts when the Docker daemon starts, usually on system boot.dockerrun-d--namepostgres\-ePOSTGRES_PASSWORD=postgres\-p5432:5432\-vpgdata:/var/lib/postgresql/data\--restartalwayspostgresLicenseOTEAPI Core is released under theMIT licensewith copyright © SINTEF.AcknowledgmentOTEAPI Core has been supported by the following projects:OntoTrans(2020-2024) that receives funding from the European Union’s Horizon 2020 Research and Innovation Programme, under Grant Agreement no. 862136.VIPCOAT(2021-2025) receives funding from the European Union’s Horizon 2020 Research and Innovation Programme - DT-NMBP-11-2020 Open Innovation Platform for Materials Modelling, under Grant Agreement no: 952903.OpenModel(2021-2025) receives funding from the European Union’s Horizon 2020 Research and Innovation Programme - DT-NMBP-11-2020 Open Innovation Platform for Materials Modelling, under Grant Agreement no: 953167.