Package: cognite-extractor-utils-rest
Cogniteextractor-utilsREST extensionThe REST extension forCogniteextractor-utilsprovides a way
to easily write your own extractors for RESTful source systems.The library is currently under development, and should not be used in production environments yet.OverviewThe REST extension for extractor utils templetizes how the extractor will make HTTP requests to the source,
automatically serializes the response into user-defined DTO classes, and handles uploading of data to CDF.The only part of the extractor necessary to for a user to implement areDescribing how HTTP requests should be constructed using pre-built function decoratorsDescribing the response schema using PythondataclassesImplementing a mapping from the source data model to the CDF data modelFor example, considerCDF's Events APIas a source. We could
describe the response schema as anEventsListdataclass:@dataclassclassRawEvent:externalId:Optional[str]dataSetId:Optional[int]startTime:Optional[int]endTime:Optional[int]type:Optional[str]subtype:Optional[str]description:Optional[str]metadata:Optional[Dict[str,str]]assetIds:Optional[List[Optional[int]]]source:Optional[str]id:Optional[int]lastUpdatedTime:Optional[int]createdTime:Optional[int]@dataclassclassEventsList:items:List[RawEvent]nextCursor:Optional[str]We can then write a handler that takes in one of theseEventLists, and returns CDF Events, as represented by instances
of theEventclass from thecognite.extractorutils.rest.typingmodule.extractor=RestExtractor(name="Event extractor",description="Extractor from CDF events to CDF events",version="1.0.0",base_url=f"https://api.cognitedata.com/api/v1/projects/{os.environ['COGNITE_PROJECT']}/",headers={"api-key":os.environ["COGNITE_API_KEY"]},)@extractor.get("events",response_type=EventsList)defget_events(events:EventsList)->Generator[Event,None,None]:foreventinevents.items:yieldEvent(external_id=f"testy-{event.id}",description=event.description,start_time=event.startTime,end_time=event.endTime,type=event.type,subtype=event.subtype,metadata=event.metadata,source=event.source,)withextractor:extractor.run()A full example is provided in theexample.pyfile.The return typeIf the return type is set tocognite.extractorutils.rest.http.JsonBodythen the raw json payload will be passed to the handler.
This is useful for cases where the payload is hard or impossible to describe with data classes.If the return type is set torequests.Response, the raw response message itself is passed to the handler.Lists at the rootUsing Python dataclasses we're not able to express JSON structures where the root element
is a list. To get around that responses of this nature will be automatically converted to something which can be modeled with Python dataclasses.A JSON structure containing a list as it's root element will be converted to an object containing a single key, "items", which has the original JSON list as it's value, as in the example below.[{"object_id": 1}, {"object_id": 2}, {"object_id": 3}]will be converted to{
    "items": [{"object_id": 1}, {"object_id": 2}, {"object_id": 3}]
}This does not apply if the return type is set toJsonBody.ContributingWe usepoetryto manage dependencies and to administrate virtual environments. To developextractor-utils, follow the following steps to set up your local environment:Install poetry: (add--userif desirable)$ pip install poetryClone repository:$ git clone git@github.com:cognitedata/python-extractor-utils-rest.gitMove into the newly created local repository:$ cd python-extractor-utils-restCreate virtual environment and install dependencies:$ poetry installAll code must pass typing and style checks to be merged. It is recommended to install pre-commit hooks to ensure that
these checks pass before commiting code:$ poetry run pre-commit installThis project adheres to theContributor Covenant v2.0as a code of conduct.