Package: vantiqconnectorsdk
Repository ContentsThis repository contains the SDK for building Vantiq connectors (also known as extension sources)
using Python. At a high level, a connector is configured and then run, responding or sending messages to messages the Vantiq server. Each connector acts as asourcein a Vantiq namespace.You will need credentials to a Vantiq server to use this SDK.  Please contact your Vantiq system administrator to obtain these credentials.Overview of a ConnectorFor details about the overall structure of a connector and the protocol used between the connector and the Vantiq source, please seehere.Each running instance of this code represents a set of connectors that connect to a single Vantiq namespace. The set of sources for the connector & the connection information is found in the server config file.  We will describe the details later.Using the Python VantiqConnector SDK, the overall operation is as follows:Create aVantiqConnectorSetthat manages the set of connections (one per source listed in theserver.configfile.Establish a set ofhandlersor callbacks that will allow the connectors to respond to messages from the Vantiq server. There are callbacks for the following situations:connect-- when your connection to the server is established, the source configuration is provided to the connector. This can be used to tailor the behavior or local work.publish-- when the Vantiq server sends an event to the source, that event is delivered to thepublishhandlerquery-- when the Vantiq server queries the source, that query is delivered to thequeryhandler.  The query handler can return information or errors to the application running on the server.close-- when the connection to the Vantiq server ends, theclosehandler is called, giving the connector a chance to clean up resources.The connector runtime will maintain the connection, reconnecting as needed.The following sections describe the use of this SDK to construct a connector.Building a ConnectorRunning this code required Python 3.10 or later.To use the SDK, install it into your Python environment.  Usingpip,pip install vantiqconnectorsdkOther Python environment tools may use different commands.Once it is installed, you can import it.fromvantiqconnectorsdkimportVantiqConnector,VantiqConnectorSet,VantiqConnectorException,VantiqSourceConnectionThe SDK is built usingasyncio.  For details about working withasyncio, please seePythonasynciodocumentation.The Vantiq Connector SDK includes the following items:VantiqConnector -- a set of constants defined as part of the interface.a set of constants to help in the use of these two classes:For construction errors:VantiqConnector.ERROR_CODE -- the short name for an errorVantiqConnector.ERROR_TEMPLATE -- the template for the message itselfVantiqConnector.ERROR_PARAMETERS -- parameters for the error messageFor context reference during callbacks:VantiqConnector.SOURCE_NAME -- the name of the source for which this callback is intendedVantiqConnector.RESPONSE_ADDRESS -- routing information to ensure that a query response is delivered to the correct queryStatus codes for query responses:VantiqConnector.QUERY_COMPLETE -- the last response for this queryVantiqConnector.QUERY_PARTIAL -- a partial response to a queryVantiqConnector.QUERY_EMPTY -- an indication that there is no response available for this queryVantiqConnectorSet -- a class that manages the connections for this set of connectorsVantiqSourceConnection -- a class that manages the message flow and state for a single connectorLoggingThe SDK uses the standard Python logging. Logger names are constructed from the class name.
If you need to setup logging, you can call theVantiqConnector.setup_logging()call.  This
will look for alogger.inifile in theserverConfigdirectory.  This is the same place we look
for theserver.configfile (seebelow).Connector Startup ConfigurationConnectors need a minimum of three configuration properties at startup:targetServer: The URL of the Vantiq server to which the connector should connect.authToken: The access token that the connector will use to authenticate to the desired namespace.sources: The name of the source or sources (comma-separated list) managed by this set of connectors.In addition to the configuration properties shown above, theserver.configfile can also include the following property:sendPings: A boolean property that, if set totrue, enables the SDK to send ping messages to the Vantiq Server.
The ping messages are handled by the underlying websockets library.The system expects theserver.configfile to be located in a directory namedserverConfigin the working directory of the connector.For users who may not want to write theauthTokenproperty to a file because of its sensitive nature, set the environment variableCONNECTOR_AUTH_TOKENto its value. If theauthTokenis specified in theserver.configdocument, that value will take precedence.
Otherwise, if theauthTokenis not set in the configuration file, the system will retrieve whatever value is provided in the environment variable.The server config file is written asproperty=value, with each property on its own line. The following is an example
of a validserver.configfile (including theauthToken, but that can be omitted and specified as an environment
variable instead):authToken=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
sources=MySourceName
targetServer=https://dev.vantiq.com/
sendPings=trueProgram FlowCreate aVantiqConnectorSet. This will read the config file and establish (but not start) aVantiqSourceConnectionfor each source listed in the config file.Define the handlers to be used.  You can use a single set of handlers for all sources (usingVantiqConnectorSet.configure_handlers_for_all(), or you can establish handlers for each source (by iterating through the connections provided byVantiqConnectorSet.get_connections()and then callingVantiqSourceConnection.configure_handlers()for each connection.Once the environment is set, you can start the connector operations.  This is done by callingVantiqConnectorSet.run_connectors().  Once this call is made (and awaited), the SDK will take over management of the connections.Each source connection can perform its work by responding to calls to thepublishandqueryhandlers.If you need to send an event to the server, useVantiqSourceConnection.send_notification().If the connection closes, the SDK will automatically re-initiate the connection.  This will result in calls theconnecthandlers again, etc.  Such a reconnection can happen due to network or connection issues.  It can also happen if the source configuration on the Vantiq side is changed.  A changed configuration requires a that the connector obtain the new configuration, and that is done by simply making a new connection.Using VantiqSourceConnectionEveryVantiqSourceConnectiondeals with a single source via its own WebSocket connection.SetupTo operate, eachVantiqSourceConnectionmust be configured with a set of handlers.  These handlers configured as follows.  (Note that a similar call can be made using theVantiqConnectorSetif all connections should use the same handler.CallVantiqSourceConnection.configure_handlers(close_handler, connect_handler, publish_handler, query_handler).  These handlers are defined as follows.handle_close : Callable[[dict], Awaitable[None]]handle_connect : Callable[[dict, dict], Awaitable[None]handle_publish : Callable[[dict, dict], Awaitable[None]handle_query : Callable[[dict, dict], Awaitable[None]These parameters specify the callbacks to handle close, connect, publish, and query calls, respectively. For each, the first parameter is a dict providing the context for the operation.  The context will contain an entry keyed withVantiqConnector.SOURCE_NAMEwhose value is the name of the source for which this is a callback.  For query calls, there will be an additional entry keyed withVantiqConnector.RESPONSE_ADDRESS. The value here is used to route the query response back to the caller in the Vantiq server.This context parameter is to be used in the various query response calls.For the handle_connect, handle_query, and handle_publish calls, there will be a second dict parameter. This will contain the message sent from Vantiq to the connector (source).Note that the handlers all return anAwaitable.  This means that they can operate usingasyncioto do other things.  However,asynciooperations should be careful not to block the event loop. If an handler is going to take a long time to run, you will want to set it running separately.Connection (Configuration)Thehandle_connectcallback is called as part of the Vantiq connection sequence, delivering the source configuration to the connector. receives the message containing a configuration document created on the Vantiq client.This callback will be called with two parameter: the context and the configuration document.
Thecontextparameter was outlined in the previous section.
Theconfigurationdocument is adictobject containing the source's configuration document. There is no format specification of the configuration document -- its content is defined by the connector.  Generally, the configuration document allows the connector to be tailored to specific purposes.  It may contain database connection information, machine learning models to run, or operational information like cache sizes, etc.ReconnectionUnder certain circumstances (e.g., a change to the source's configuration document or some network connection issue), the Vantiq server may disconnect
you from the source with the intent that you will immediately reconnect.This operation is unavoidable, but is generally transparent to the connector.  When this happens, thehandle_closecallbackwill be called, and then thehandle_connectcallback.  Since this may be an indication that the source configuration has changed, eachconnectcall should be treated as a new connection/configuration.This process is automatic.  No special work by the connector is required.Processing MessagesMessages involving a source come in three (3) flavors:Messages (events) sent by a sourceQuery messages sent to a source. Query messages, in turn, have responses.Messages published to a sourceThe following sections outline the work involved here.NotificationsNotifications are messages that the source will pass on to Vantiq rules that includeWHEN EVENT OCCURS ON "/sources/<source name>". To send one, simply callVantiqSourceConnection.send_notification(<message to be sent>), which will translate the message into JSON and add everything Vantiq needs to recognize the message.  Themessage to be sentshoud be adictobject.QueriesQueries are sent to the source via VAIL code of the formrow = SELECT * FROM SOURCE someSource with prop1 = value1, prop2 = value2, ...A query like this will result in a query message to the source (and, thus, to the connector via thehandle_querycallback). The parameter value passed to the callback will be a dict object{ prop1: value1, prop2: value2, ... }.  That is, the parameter values are those passed from thewithclause in the VAIL SELECT statement.Once the query message is received (i.e., delivered to the callback), the connector to which the
query was delivered must respond. It responds by making calls to return a query response or a query error.Query ContextQuery responses are responses to aSELECTrequest from Vantiq that targets a source, and can either be adictor a series ofdictobjects in the form of alistorarray. They only mean anything in relation to an initial Query message received from Vantiq, and thus should
only be sent as part response to a query via the callback.To enable this behavior, the callback is passed acontextparameter that provides information about the context of the query. This parameter will take the form of adictobject with two (2)
items: aVantiqConnector.SOURCE_NAMEitem that contains a string specifying the source toward which this query is
sent, and aVantiqConnector.RESPONSE_ADDRESSitem that contains the routing information to allow the query resposnes (or error) to find their way back the the Vantiq code that issued the query.Query ResponsesTo send a query response, callVantiqSourceConnection.send_query_response(ctx, code, message)Thectxor context parameter is passed in to thehandle_querycallback.  Simply pass that back unchanged.The code is an HTTP status code. You can use the following:VantiqConnector.QUERY_EMPTYto indicate that this query has an empty response.  In this case, the message parameter must be None.VantiqConnector.QUERY_PARTIALto indicate that this is one of a number of responses to this query. This is used if you are dynamically producing results to return.VantiqConnector.QUERY_COMPLETEto indicate that this is the last (or only) response to this query.For the latter two (2) codes, themessageparameter must be either adictor alistofdictobjects.  These will be sent to the Vantiq server in a JSON format.Query ErrorsQuery errors are sent when a Query cannot be completed successfully. To send a Query error, callVantiqSourceConnection.send_query_error(ctx, message).Thectxor query context was passed in to thehandle_querycallback.
Themessage(dict) parameter must contain the following items.VantiqConnector.ERROR_CODE-- the short name for the errorVantiqConnector.ERROR_TEMPLATE-- template for the error message.  Templates contain zero-based indexed references for parameters list (e.g.,{0}to refer to the first parameter,{1}the second, etc.).VantiqConnector.ERROR_PARAMETERS-- a list of parameters for the template.Published MesssagesWhen the Vantiq server publishes an event to a source that is implemented using a connector, it is delivered via thehandle_publishcallback. As with a query message, the handler will be passed two (2) parameter:  A context and the message.The context in this case will contain theVantiqConnector.SOURCE_NAMEitem.  It need not contain theVantiqConnector.RESPONSE_ADDRESSitem.  You can not respond to messages delivered via thehandle_publishcallback.If the Vantiq server executes VAIL code likePUBLISH { prop1: value1, prop2: value2, ... } TO SOURCE connectorSourcethe connector will receive ahandle_callbackcall containing the context parameter{'source_name':'connectorSource'}and a message parameter containing{'prop1':value1,'prop2':value2}The connector can then act on the message as it deems appropriate.Kubernetes Operation and TCP ProbeIn the case that the connector is deployed within a Kubernetes cluster, theVantiqConnectorSetoffers support for TCP Startup/Liveness/Readiness probes. The class defines two methods:declare_healthy()anddeclare_unhealthy(), that can be used to make the respective declarations.Internally,declare_healthy()anddeclare_unhealthy()open and close a TCP socket, respectively; the Kubernetes probes attempt to connect to that port. If connection is successful, the probe is considered successful; if not, the probe is unsuccessful.The socket is opened on port 8000 by default, but this can be changed by including tcpProbePort: in the connector'sserver.configdocument. It is the responsibility of the connector developer to manage when the connector is healthy and when it is not.Note thatdeclare_healthy()anddeclare_unhealthy()are available from theVantiqSourceConnectionas well.  These make calls to the containingVantiqConnectorSet. Health status is available only on aVantiqConnectorSetbasis.TheVantiqConnectorSetandVantiqSourceConnectionclasses both offer theis_healthy()method.  This will returnTrueif the connector set has been declared healthy,Falseif it has been declared unhealthy, andNoneif no health declaration has been made.CloseThe closure handler does not deal with a specific message or type of message, but instead is called when either your
code callsVantiqSourceConnection.close()or the WebSocket connection is forced to close, most likely due to a problem with the connection.Thehandle_closeis passed a context containing the source name (seepublish). At the time the handler is called, the connection has already been closed.  No interaction with the Vantiq server is possible;  this handler call allows the connector to perform any resource deallocation or other cleanup necessary.LicenseThe source code in this project is licensed under theMIT License.The MIT License (MIT)Copyright (c) 2022 Vantiq, Inc.Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.