Package: hiro-graph-client
HIRO Graph API ClientThis is a client library to access data of theHIRO Graph.This library also contains classes for handling theWebSocketsevent-wsandaction-wsAPI.To process large batches of data, take a look at "hiro-batch-client". - (
PyPI:hiro-batch-client,
GitHub:hiro-batch-client-python)For more information about HIRO Automation, look athttps://www.arago.co/For more information about the APIs this library covers, seehttps://developer.hiro.arago.co/7.0/api/Currently, implemented areHiroAppforappHiroAuthforauthHiroGraphforgraphHiroIamforiamHiroKiforkiHiroAuthzforauthzHiroVariablesforvariablesand for the websocketsAbstractEventWebSocketHandlerforevent-wsAbstractActionWebSocketHandlerforaction-wsQuickstartTo use this library, you will need an account athttps://id.arago.co/and access to an OAuth Client-Id and Client-Secret
to access the HIRO Graph. See alsohttps://developer.hiro.arago.co.Most of the documentation is done in the sourcecode.HiroGraph ExampleExample to use the straightforward graph api client:fromhiro_graph_clientimportPasswordAuthTokenApiHandler,HiroGraphhiro_client:HiroGraph=HiroGraph(api_handler=PasswordAuthTokenApiHandler(root_url="https://core.arago.co",username='',password='',client_id='',client_secret=''))# The commands of the Graph API are methods of the class HIROGraph.# The next line executes a vertex query for instance.query_result=hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')print(query_result)TokenApiHandlerAuthorization against the HIRO Graph is done via tokens. These tokens are handled by classes of
typeAbstractTokenApiHandlerin this library. Each of the Hiro-Client-Object (HiroGraph, ,HiroApp, etc.) need to
have some kind of TokenApiHandler at construction.This TokenApiHandler is also responsible to determine the most up-to-date endpoints for the API calls. You can supply a
custom list of endpoints by using the dict parametercustom_endpoints=on construction.A custom list of headers can also be set via the dict parameterheaders=in the constructor. These would update the
internal headers. Header names can be supplied in any upper/lower-case.This library supplies the following TokenApiHandlers:FixedTokenApiHandlerA simple TokenApiHandler that is generated with a preset-token at construction. Cannot update its token.EnvironmentTokenApiHandlerA TokenApiHandler that reads an environment variable (default isHIRO_TOKEN) from the runtime environment. Will only
update its token when the environment variable changes externally.PasswordAuthTokenApiHandlerThis TokenApiHandler logs into the HiroAuth backend and obtains a token from login credentials. This is also the only
TokenApiHandler (so far) that automatically tries to renew a token from the backend when it has expired.All code examples in this documentation can use these TokenApiHandlers interchangeably, depending on how such a token is
provided.The HiroGraph example from above with another customized TokenApiHandler:fromhiro_graph_clientimportEnvironmentTokenApiHandler,HiroGraphhiro_client:HiroGraph=HiroGraph(api_handler=EnvironmentTokenApiHandler(root_url="https://core.arago.co"))# The commands of the Graph API are methods of the class HIROGraph.# The next line executes a vertex query for instance.query_result=hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')print(query_result)Example with additional parameters:fromhiro_graph_clientimportEnvironmentTokenApiHandler,HiroGraphhiro_client:HiroGraph=HiroGraph(api_handler=EnvironmentTokenApiHandler(root_url="https://core.arago.co",env_var='_TOKEN',headers={'X-Custom-Header':'My custom value'},custom_endpoints={"graph":"/api/graph/7.2","auth":"/api/auth/6.2"},client_name="HiroGraph (testing)"# Will be used in the header 'User-Agent'))# The commands of the Graph API are methods of the class HIROGraph.# The next line executes a vertex query for instance.query_result=hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')print(query_result)Token Handler sharingWhen you need to access multiple APIs of HIRO, share the TokenApiHandler between the API objects to avoid unnecessary
requests for token- and version-information against HIRO. The TokenApiHandler will share arequests.Session, token-
and version-request-handling, and the token string itself between them.fromhiro_graph_clientimportHiroGraph,HiroApp,PasswordAuthTokenApiHandlerhiro_api_handler=PasswordAuthTokenApiHandler(root_url="https://core.arago.co",username='',password='',client_id='',client_secret='')hiro_client:HiroGraph=HiroGraph(api_handler=hiro_api_handler)hiro_app_client:HiroApp=HiroApp(api_handler=hiro_api_handler)Connection sharingYou can also let TokenApiHandlers share a common connection session instead of letting each of them create their own.
This might prove useful in a multithreading environment where tokens have to be set externally or change often (i.e.
one token per user per thread). This also ensures, that version-requests happen only once when the connection is
initialized.Use the parameterspool_maxsizeandpool_blockto further tune the connection parameters for parallel access to
the backend. Seerequests Session Objectsand the Python documentation ofrequests.adapters.HTTPAdapterandGraphConnectionHandler.fromhiro_graph_clientimportHiroGraph,HiroApp,FixedTokenApiHandler,GraphConnectionHandlerconnection_handler=GraphConnectionHandler(root_url="https://core.arago.co",pool_maxsize=200,# Optional: Max pool of cached connections for this connection sessionpool_block=True,# Optional: Do not allow more parallel connections than pool_maxsizeclient_name="Your Graph Client 0.0.1"# Optional: Will be used in the header 'User-Agent')# Work with token of user 1user1_client:HiroGraph=HiroGraph(api_handler=FixedTokenApiHandler(connection_handler=connection_handler,token='token user 1'))# Work with token of user 2 (Shared Token Handler)user2_api_handler=FixedTokenApiHandler(connection_handler=connection_handler,token='token user 2')user2_client:HiroGraph=HiroGraph(api_handler=user2_api_handler)hiro_app_client:HiroApp=HiroApp(api_handler=user2_api_handler)Everything written inToken Handler Sharingstill applies.SSL ConfigurationSSL parameters are configured using the classSSLConfig. This class translates the parameters given to the required
fields for therequestslibrary of Python (parameterscertandverifythere). This configuration is given to the
TokenApiHandlers and will be used by the clients attached to it as well.If this is not set, the default settings of the libraryrequestswill be used, which is to verify any server
certificates by using system defaults.Example: Disable verificationfromhiro_graph_clientimportEnvironmentTokenApiHandler,HiroGraph,SSLConfighiro_client:HiroGraph=HiroGraph(api_handler=EnvironmentTokenApiHandler(root_url="https://core.arago.co",# Disable any verification.ssl_config=SSLConfig(verify=False)))query_result=hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')print(query_result)Example: Set custom SSL certificatesfromhiro_graph_clientimportEnvironmentTokenApiHandler,HiroGraph,SSLConfighiro_client:HiroGraph=HiroGraph(api_handler=EnvironmentTokenApiHandler(root_url="https://core.arago.co",# Set custom certification files. If any of them are omitted, system defaults will be used.ssl_config=SSLConfig(cert_file="<path to client certificate file>",key_file="<path to key file for the client certificate>",ca_bundle_file="<path to the ca_bundle to verify the server certificate>")))query_result=hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')print(query_result)Graph Client "HiroGraph"The Graph Client is mostly straightforward to use, since all public methods of this class represent an API call in theGraph API. Documentation is available in source code as
well. Some calls are a bit more complicated though and explained in more detail below:AttachmentsTo upload data to such a vertex, useHiroGraph.post_attachment(data=...). The parameterdata=will be given directly
to the call of the Python libraryrequestsasrequests.post(data=...). To stream data, setdatato an object of
typeIO. See the documentation of the Python libraryrequestsfor more details.Downloading an attachment is done in chunks, so huge blobs of data in memory can be avoided when streaming this data.
Each chunk is 64k by default.To stream such an attachment to a file, see the example below:ogit_id='<ogit/_id of a vertex>'data_iter=hiro_client.get_attachment(ogit_id)withio.start("attachment.bin","wb")asfile:forchunkindata_iter:file.write(chunk)To read the complete data in memory, see this example:ogit_id='<ogit/_id of a vertex>'data_iter=hiro_client.get_attachment(ogit_id)attachment=b''.join(data_iter)WebSocketsThis library contains classes that make using HIRO WebSocket protocols easier. They handle authentication, exceptions
and much more.The classes do not handle buffering of messages, so it is the duty of the programmer to ensure, that incoming messages
are either handled quickly or being buffered to avoid clogging the websocket. The classes are thread-safe, so it is
possible to handle each incoming message asynchronously in its own thread and have those threads send results back if
needed (See multithreaded example inAction WebSocket).Closing WebSocketsTo shut these WebSockets (ws) down cleanly, please consider these scenarios:Default behaviourThe library reacts onKeyboardInterrupt(SIGINT) and closes the WebSocket cleanly with closing message to the sever
when such an interrupt is received. If another signal (like SIGTERM) is received, the program will stop immediately
without any closing message back to the server.Signal handlingWhen installing signal handlers, you need to usews.signal_stop()to shut the WebSocket down. Do NOT usews.stop()or the closing process will deadlock. This is because the signal interrupt is executed in the same thread
asws.run_forever().Example:importsignal[...]withActionWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN'))asws:defsignal_handler(signum,handler):ws.signal_stop()signal.signal(signal.SIGINT,signal_handler)signal.signal(signal.SIGTERM,signal_handler)signal.signal(signal.SIGHUP,signal_handler)# This signal might not be available on MS Windowsws.run_forever()Closing from a separate threadWhen closing the WebSocket from another thread than the one runningws.run_forever(), you should usews.stop(). This
ensures, that the shutdown is synchronized andws.stop()will return after the WebSocket has been closed.Event WebSocketThis websocket receives notifications about changes to vertices that match a certain filter.See alsoAPI description of event-wsExample:importthreadingfromhiro_graph_client.clientlibimportFixedTokenApiHandlerfromhiro_graph_client.eventswebsocketimportAbstractEventsWebSocketHandler,EventMessage,EventsFilterclassEventsWebSocket(AbstractEventsWebSocketHandler):defon_create(self,message:EventMessage):""" Vertex has been created """print("Create:\n"+str(message))defon_update(self,message:EventMessage):""" Vertex has been updated """print("Update:\n"+str(message))defon_delete(self,message:EventMessage):""" Vertex has been removed """print("Delete:\n"+str(message))events_filter=EventsFilter(filter_id='testfilter',filter_content="(element.ogit/_type=ogit/MARS/Machine)")withEventsWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN'),events_filters=[events_filter],query_params={"allscopes":"false","delta":"false"})asws:ws.run_forever()# Use KeyboardInterrupt (Ctrl-C) to exit.If you do not set the parameterscope=, the default scope of your account will be used. If you need to set the scope
by hand, use the following:[...]api_handler=FixedTokenApiHandler('HIRO_TOKEN')default_scope=api_handler.decode_token()['data']['default-scope']withEventsWebSocket(api_handler=api_handler,events_filters=[events_filter],scopes=[default_scope],query_params={"allscopes":"false","delta":"false"})asws:ws.run_forever()# Use KeyboardInterrupt (Ctrl-C) to exit.Action WebSocketThis websocket receives notifications about actions that have been triggered within a KI. Use this to write your own
custom action handler.See alsoAPI description of action-wsSimple example:importthreadingfromhiro_graph_client.actionwebsocketimportAbstractActionWebSocketHandlerfromhiro_graph_client.clientlibimportFixedTokenApiHandlerclassActionWebSocket(AbstractActionWebSocketHandler):defon_submit_action(self,action_id:str,capability:str,parameters:dict):""" Message *submitAction* has been received """# Handle the messageprint(f"ID:{action_id}, Capability:{capability}, Parameters:{str(parameters)}")# Send back message *sendActionResult*self.send_action_result(action_id,"Everything went fine.")defon_config_changed(self):""" The configuration of the ActionHandler has changed """passwithActionWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN'))asws:ws.run_forever()# Use KeyboardInterrupt (Ctrl-C) to exit.Multithreading example using a thread executor:importthreadingimportconcurrent.futuresfromhiro_graph_client.actionwebsocketimportAbstractActionWebSocketHandlerfromhiro_graph_client.clientlibimportFixedTokenApiHandler,AbstractTokenApiHandlerclassActionWebSocket(AbstractActionWebSocketHandler):def__init__(self,api_handler:AbstractTokenApiHandler):""" Initialize properties """super().__init__(api_handler)self._executor=Nonedefstart(self)->None:""" Initialize the executor """super().start()self._executor=concurrent.futures.ThreadPoolExecutor()defstop(self,timeout:int=None)->None:""" Shut the executor down """ifself._executor:self._executor.shutdown()self._executor=Nonesuper().stop(timeout)defhandle_submit_action(self,action_id:str,capability:str,parameters:dict):""" Runs asynchronously in its own thread. """print(f"ID:{action_id}, Capability:{capability}, Parameters:{str(parameters)}")self.send_action_result(action_id,"Everything went fine.")defon_submit_action(self,action_id:str,capability:str,parameters:dict):""" Message *submitAction* has been received. Message is handled in thread executor. """ifnotself._executor:raiseRuntimeError('ActionWebSocket has not been started.')self._executor.submit(ActionWebSocket.handle_submit_action,self,action_id,capability,parameters)defon_config_changed(self):""" The configuration of the ActionHandler has changed """passwithActionWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN'))asws:ws.run_forever()# Use KeyboardInterrupt (Ctrl-C) to exit.CHANGELOGv5.3.0Decode any token via staticdecode_token_ext().Changes for /api/auth/6.6:Ability to revoke a token.Recognizeaccess_tokenand_TOKENin token results.Remove obsolete methodfresh()from class TokenInfo because each token refresh issues a new refresh_token by default
now.v5.2.5[bugfix] Calculation withinclientlib.TokenInfo.expired()now returnsFalseas expected when noexpires_atis available for a token. Previous versions would raise an exception.v5.2.4Merge headers in AbstractAPI instead of replacing them.v5.2.3Remove accept_all_certs for good. Directly use the flag in SSLConfig.v5.2.2Debugged python code documentation.Fixed copyright notices.v5.2.1Update because of wrong documentation onhttps://pypi.org/project/hiro-graph-client/v5.2.0Separate connection sessions from token handlers. Introducing classGraphConnectonHandlerwhich can be shared
betweenTokenApiHandlers and API-Classes.Updated documentation.v5.1.0Use connection pooling via requests.Session.Setpool_connections=1,pool_maxsize=10by default (the latter can be changed).Optionpool_blockto block any connections that would exceed thepool_maxsize.Removed obsoleteaccept_all_certs().Fixed pytest in Makefile target.Added information aboutrequests.Sessionand thepool_maxsizeto documentation.v5.0.0Externalizebatchclient.pyas its own project athttps://github.com/arago/hiro-batch-client-python.v4.9.0Added batch handling of issues.handle_vertices_combinedrecognizes_issue_datanow for issues directly linked to
vertices.Updated documentation.v4.8.0Refactoring ofwebsocketlib.py:Removed unnecessary extra thread.Debugged shutdown of WebSockets via signals and threads.Updated documentation:On signal handling.On clean shutdown of WebSockets.Adjusted the example code for WebSockets.Allow-or_as build number separator on test tags, i.e.t4.8.0_0andt4.8.0-0are handled identical now.v4.7.5[budgfix] Handleself.submitStoreandself.resultStoreproperly on shutdown in AbstractActionWebSocketHandler.
They might not have been initialized at all when startup fails.v4.7.4[bugfix] Catch 401 when using PasswordAuthTokenHandler and refresh_token. Raise TokenUnauthorizedError when this
happens to trigger a retry.v4.7.3Refactoring ofHiroGraphBatch. Removed obsolete methods.v4.7.2Fixed bad call of patch() methods in IAM API.v4.7.1Updated some code documentation inHiroGraphthat became obsolete.Updated graph queries inHiroGraph:For vertex and gremlin queries: Skip unnecessary fields in query payload.Add parametercountto vertex query.Renamed default _client_name tohiro-graph-client(name of the lib on PyPI).v4.7.0Content- / Media-Type handlingThrowWrongContentTypeErrorwhen an unexpected (or missing) Content-Type is returned.Check for Content-Type of error results for constructing HTTPError.Error handlingSeparate error message extraction, so it can be overwritten per API.Try to handle different message formats by guessing, where their error message might be.Intercept special error messages for APIki.Flaglog_communication_on_errorto enable logging of communication when an error is detected in the response.Add parametermax_triesto TokenHandlers.v4.6.2Added documentation of new APIsv4.6.1Updated CHANGELOG.mdv4.6.0Added the following APIS:KIAuthZVariablesAdjust return value typing with API methods that return lists of dicts.v4.5.2BUGFIXPrevious versions incorrectly translated True and False to "True" and
"False" whereas "true" and "false" (lowercase) are needed in URL queries. Fixed.v4.5.1Adddecode_token()to decode the information embedded in a HIRO token.v4.5.0GitHub repository got renamed frompython-hiro-clientstohiro-client-python. No other technical changes.v4.4.0Added IAM clientUpdated Graph client and Auth clientput_binary is allowed to return simple strings now. (i.e. for avatar image updates).v4.3.0Adding SSL configurationv4.2.14Removed bug with reversed operator in websocketlib.Updated installation instructions in README.md.v4.2.13You need to explicitly setquery_params={'allscopes': 'true'}if you want to enable it for EventWebSockets. If this
is left out of the query_params, it will be added as 'allscopes': 'false'.v4.2.12Use typing to make sure, thatquery_params=for WebSockets is of typeDict[str, str].Setquery_params={'allscopes': 'false'}as default for the EventWebSocket.v4.2.11Debugged EventWebSocket handling.Abort connection when setting scope or filters failed.v4.2.10Adding scopes to EventWebSockets.v4.2.9Documentation of feature in v4.2.8v4.2.8WebSockets have new optionquery_paramsto add arbitrary query parameters to the initial websocket request.v4.2.7Changes toAbstractAuthenticatedWebSocketHandler:Introducingrun_forever()which will return after the reader thread has been joined. This can happen when another
thread callsstop()(normal exit) or an internal error occurs, either directly when the connection is attempted, a
token got invalid and could not be refreshed, or any other exception that has been thrown in the internal reader
thread.This ensures, that the main thread does not continue when the websocket reader thread is not there.Enable parallel executions ofsend()across multiple threads.Make sure, that only one thread triggers a restart by a call torestart().Check for active websocket reader thread viais_active().Update examples for websockets in README.md.GenericUpdate README.md to show usage ofclient_name.v4.2.6Do not require package uuid - it is already supplied with pythonv4.2.5Send valid close messages to backend.Introduced parameterclient_nameto give connections a name and also set headerUser-Agentmore easily.v4.2.4Updated CHANGELOG.md.v4.2.3Hardening of clientlib. Removed some None-Value-Errors.v4.2.2Introduce parameterremote_exit_codestoAbstractAuthenticatedWebSocketHandler.v4.2.1Avoid blocking thread in_backoff()by not usingsleep()butthreading.Condition.wait().v4.2.0Implement websocket protocolsevent-wsaction-wsv4.1.3Use yield from instead of returnv4.1.2Removed a bug with double yields on binary datav4.1.1Only log request/responses when logging.DEBUG is enabledv4.1.0Added timeseries handling to commandhandle_vertices_combinedv4.0.0AbstractTokenApiHandlerBetter token handling.Resolve graph api endpoints via calling /api/version.Ability to customize headers. Headers are handled case-insensitively and are submitted to requests
capitalized.Ability to override internal endpoints.AbstractIOCarrier works withwithstatements now.AddedBasicFileIOCarrier.RemovedApiConfig.Renamed several internal classes.Better error messages.HTTP secure protocol logging.Fixed timestamp creation for tokens.Joe's suggestions - thanks Joe!v3.1.0Separation of APIs in client libraries. Currently, supported APIs are:HiroGraph:https://core.arago.co/help/specs/?url=definitions/graph.yamlHiroAuth:https://core.arago.co/help/specs/?url=definitions/auth.yamlHiroApp:https://core.arago.co/help/specs/?url=definitions/app.yamlUse correct headers with binary transfers.Added gremlin and multi-id queries to HiroGraph.v3.0.0Renamed classes to match documentation elsewhere (i.e. Graphit -> HiroGraph, GraphitBatch -> HiroGraphBatch).Catch token expired error when refresh_token has expired.Documentation with examplesv2.4.2Added VERSION to package_data in setup.pyv2.4.1Added documentation for PyPIv2.4.0Initial release after split fromhttps://github.com/arago/hiro-clients