Package: daml-dit-if
daml-dit-ifDaml Hubintegrationsare loadable Python modules that mediate the relationship between a Daml Hub
ledger and various external systems. Because of their special role within
Daml Hub, integrations have the ability to issue and receive external
network requests in addition to the usual ledger interactions supported
bybotsandtriggers.
This gives integrations the ability to issue network requests based on
ledger activity as well as issue ledger commands based on network
activity. To allow monitoring and control by Daml Hub, integrations must
also follow a set of interface conventions. This repository
contains a framework that builds on Digital Asset'sDAZL Ledger Clientto simplify development of common types of integrations.This framework makes it possible to develop custom integration
types, but due to their access to the network, integrations have
privileged status within Daml Hub and require elevated permissions to
install. Please contactDigital Assetfor
more information on deploying custom integration types into Daml Hub.For examples of fully constructed integrations built with the
framework, there are several open source examples available in GitHub
repositories. These also correspond to some of the default integrations
available to all Daml Hub users via theBrowse Integrationstab in theconsole's ledger view.Core PackCoindeskSlackExberryIntegration Packaging and DeploymentIntegrations are packaged inDIT files,
built using thedditbuild tool,
and can be deployed into Daml Hub using the same upload mechanism as other
artifacts (Daml Models, bots, etc.). Daml Hub also has an 'arcade' facility
that uses a publicGitHub repositoryto maintain a list of integrations and sample apps that can be deployed through
a single click on theconsole user interface.Logically speaking, Daml Hub integrations package their Python
implementation code alongside metadata describing available
integration types and any other resources required to make the
integration operate. Integrations usually require Daml models to
represent their data model, and often require external Python
dependencies specified through arequirements.txtfile. Both of
these are examples of additional resources that might be bundled into
a DIT file.  Note that whiledaml-dit-ifanddazlare both Python
dependencies required by integrations, they are exceptions to this
rule.  Daml Hub provides both of these by default, so they should not
be listed inrequirements.txtand should not be included in the DIT
file.Integration Design GuidelinesBuilding integrations for Daml Hub, we've found the following to be good
guidance for designing reliable and usable integrations:Rather than require a number of distinct integration types or integration instances, favor designs that reduce the number of types and instances. This can simplify both configuration and deployment.When possible, consider using contracts on ledger to allow configuration of multiple activities of the same integration rather than multiple instances of the same integration with different sets of configuration parameters.Store private API tokens and keys on-ledger in contracts. This restricts visibility of secrets to the integration itself.Try to ensure that as much retry/handshaking logic is managed directly in integration code, rather than in Daml. This makes it easier to ensure that the Daml logic is more purely focused on the business processes being modeled rather than the details of the integration protocol.Prefer level sensitive logic to edge sensitive logic. Rather than triggering an external interaction based on the occurrence of an event, trigger based on the presence of a contract and archive the contract when the interaction is complete. This can improve reliability and error recovery in the event of failed integrations, and help keep technical integration details out of the business logic in Daml.The Integration Framework APIThe integration framework API has two parts:Metadata describing the available integration types.A Python API for registering event handlers.The metadata for an integration is stored in an additionalintegration_typessection withindabl-meta.yaml. This section
lists and describes the integration types defined within the DIT file.
This metadata section includes the name of the entry point function
for the integration type, some descriptive text, and a list of the
configuration arguments accepted by the integration:Theledger event log integrationis defined like this:catalog:... elided ...integration_types:... elided ...- id:com.projectdabl.integrations.core.ledger_event_logname:Ledger Event Logdescription:>Writes a log message for all ledger events.entrypoint:core_int.integration_ledger_event_log:integration_ledger_event_log_mainenv_class:core_int.integration_ledger_event_log:IntegrationLedgerEventLogEnvfields:-id:historyBoundname:Transaction History Bounddescription:>Bound on the length of the history maintained by the integrationfor the purpose of the log fetch endpoint. -1 can be used to removethe bound entirely.field_type:textdefault_value:"1024"id- The symbolic identifier used to select the integration type within the DIT.name- A user friendly name for the integration.description- A description of what the integration does.entrypoint- The package qualified name of the entrypoint function.env_class- The package qualified name of the class used to contain the integration'senvironment.fields- A list ofconfiguration fieldsthat users will be able to enter through the console when configuring an integration. These will be passed into the integration instance at runtime via an instance ofenv_class.Theentrypointandenv_classfields identify by name the two
Python structures that represent the runtime definition of an
integration type's implemenation.The first,entrypoint, is required for all integration types and
names a function the framework calls when starting a new instance of
an integration. During the entrypoint function, integrations are able
to register handlers for various sort of events (ledger, web, and
otherwise), start coroutines, and access integration configuration
arguments specified through the Daml Hub console UI. The syntax forentrypointis$PYTHON_PACKAGE_NAME:$FUNCTION_NAME, and the function
itself must have the following signature.fromdaml_dit_if.apiimportIntegrationEventsdefintegration_ledger_event_log_main(env:'IntegrationLedgerEventLogEnv',events:'IntegrationEvents'):Theeventsargument is an instance ofIntegrationEventscontaining a number offunction decoratorsthat can be used to registerevent handlers.
This represents the bulk of the integration call API, and contains
means to register handlers for various DAZL ledger events, HTTPS
endpoints, timers, and internal message queues. Based on the event
handlers that the integration registers, the integration framwork
configures itself appropriately to make those events known to the
integration while it is running.Theenvargument is the environment in which the integration is
running. This contains named fields with all of the integration
configuration parameters, as well as access to various other features
of the integration framework. These include in-memory queuing and
metadata for the Daml model associated with the integration. Because
the configuration parameters for an integration can vary from
one integration type to the next, the type ofenvcan be specialized
to the given integration type. For the example above, the specific
environment typeIntegrationLedgerEventLogEnv, is defined as
follows.fromdaml_dit_if.apiimportIntegrationEnvironment@dataclassclassIntegrationLedgerEventLogEnv(IntegrationEnvironment):historyBound:intThe environment class for an integration type is named indabl-meta.yamlwith the fieldenv_class. This class must derive
fromIntegrationEnvironment,
and if a specific subclass is not specified, the integration will
receive an instance ofIntegrationEnvironmentas its environment
when it starts up.  The syntax forenv_classis the same as the
syntax forentrypoint:$PYTHON_PACKAGE_NAME:$FUNCTION_NAME.LoggingDaml Hub integrations use the default Python logging package, and the
framework provides support for controlling log level at runtime. To
integrate properly with this logic, it is important that integrations
use the standard mechanism for accessing the integration logger. This
logger is switched fromINFOlevel toDEBUGlevel at aDABL_LOG_LEVELsetting of 10 or above.fromdaml_dit_if.apiimportgetIntegrationLoggerLOG=getIntegrationLogger()Integration coroutinesFor integrations that need to maintain ongoing processing independent
of event handlers, a coroutine can be returned from the
entrypoint. The framework will arrange for this coroutine to be
scheduled for execution alongside the other coroutines managed by the
framework itself. For an example of this, see theExberry integration.Integration Event HandlersIntegrations are purely event driven and may only take action in
response to an event notification from the framework. Integrations
register their interest in given events by decorating custom functions
with decorators provided to the integration when it is starting
up. These decorators are found in theIntegrationEventsinstance
passed to the entrypoint function. As an example, theSlack integrationlistens for outbound messages as follows.defintegration_slack_main(env:'IntegrationEnvironment',events:'IntegrationEvents'):...elided...@events.ledger.contract_created('SlackIntegration.OutboundMessage:OutboundMessage')asyncdefon_contract_created(event):...elided...Once the integration has started,on_contract_createdwill be called
for each DAZL event corresponding to anOutboundMessagecontract
being created on the ledger.In addition to ledger events, the framework provides a range of other
types of integration event handlers:@dataclassclassIntegrationEvents:queue:'IntegrationQueueEvents'time:'IntegrationTimeEvents'ledger:'IntegrationLedgerEvents'webhook:'IntegrationWebhookRoutes'Ledger-DAZLledger events. (Contract Archived, Contract Created, Transaction Boundaries, etc.)Webhook- Inbound HTTPS requests from the outside world. (GET or POST)Time- Periodic timer events. (Useful to poll an external system, etc.)Queue- In-memory message queue events. (Useful when none of the other types apply.)Ledger EventsThe framework provides ledger event handlers for ledger initializtion
events, transaction boundaries, and contract create and archived
events.  These events are all subject to the Daml ledger visiblity
model. An integration runs as a specific ledger party with a specific
set of rights to the ledger. The integration will only see contract
events visible to that party.All ledger event handlers can return a list of DAZL ledger commands to
be issued by the framework when the event handler returns.fromdazlimportexercise...elided...@events.ledger.contract_created('SlackIntegration.OutboundMessage:OutboundMessage')asyncdefon_contract_created(event):...elided...return[exercise(event.cid,'Archive')]The contract created decorator takes a few of arguments that
control how it presents events to the framework.@abc.abstractmethoddefcontract_created(self,template:Any,match:'Optional[ContractMatch]'=None,sweep:bool=True,flow:bool=True):templateis the DAZL template query string for event handler.  The
event handler will be called only for contracts that match this
query. It can be*to subscribe to all templates, or it can be a
qualified template name:@events.ledger.contract_created('SlackIntegration.OutboundMessage:OutboundMessage')asyncdefon_contract_created(eveIf the template name does not specify a full package ID, the framework
will assume that the template name refers to a template in the
integration's package and automatically qualify that name with the
package ID. This eliminates ambiguity if there are multiple templates
with the same symbolic name and eliminates a DAZL error that occurs
when subscribing to contract template that the ledger has not yet seen
instantiated.sweepandflowcontrol how the event handler sees historical and
newly created contracts on the ledger. IfsweepisTrue, the
framework will sweep the ledger for contracts that already exist when
the integration is starting up and call the event handler for each
such contract when starting up. WhenflowisTrue, the integration
will receive events corresponding to new contracts that are created
while it is running. Note that the framework provides no corresponding
control over contract archived events. If an archived event handler is
registered for a contract template, it will receive all visible
archive events for the template, regardless of whether or not the
framework called a created event handler corresponding to the
template.Webhook EventsEach integration instance maintains anaiohttpweb endpoint that's
used to accept inbound HTTPS requests from external systems. Integrations
can register to receive bothGETandPOSTrequests from external
systems using thewebhookevent decorators.@events.webhook.post(label='Slack Event Webhook Endpoint')asyncdefon_webhook_post(request):body=awaitrequest.json()Each integration is assigned a base Daml Hub URL based on the name of
its enclosing ledger and its integration ID. All webhook URL's for a
given integration are relative to that assigned base URL, and are
presented to the user via the integration status display.Due to their nature, webhook handlers have to have the ability to
return both a set of ledger commands and an HTTP response. This is
accomodated with theIntegrationWebhookResponseclass that contains
both acommandsfield and aresponsefield.fromdaml_dit_if.apiimportIntegrationWebhookResponse@events.webhook.get(url_suffix='/json',label='JSON Table',auth=AuthorizationLevel.PUBLIC)asyncdefon_get_table_json(request):row_data=get_formatted_table_data()returnIntegrationWebhookResponse(response=json_response({'rows':row_data}))To populate theresponseof anIntegrationWebhookResponse, there
are also several utility functions for generating standardaiohttpresponses:fromdaml_dit_if.apiimport\json_response,\empty_success_response,\blob_success_response,\unauthorized_response,\forbidden_response,\not_found_response,\bad_request,\internal_server_errorWebhook Event ConfigurationThe full definition for an integration decorator allows several
parameters to control the event handler.@abc.abstractmethoddefget(self,url_suffix:'Optional[str]'=None,label:'Optional[str]'=None,auth:'Optional[AuthorizationLevel]'=AuthorizationLevel.PUBLIC):labelis a user friendly description of the event handler URL. It is
used to label the status presented to the event handler as it is
displayed in the console.url_suffixis the URL suffix for this event handler relative to the
integration's base URL. It can be used to distinguish multiple
endpoints within the same integration if necessary, or left out
entirely.aiohttppattern matching works in these suffixes as well.authis the authorization mode of the webhook endpoint. By default,
all webhook endpoints are publically visible, but the framework has
two options for stricter access controls.ANY_PARTY- Requests must be presented with a valid Daml Hub JWT corresponding to the integration's ledger.INTEGRATION_PARTY- Requests must be presented with a valid Daml Hub JWT corresponding to the integration's ledger and party.A JWT is considered to be valid for a given party, only if that party
is listed in both thereadAsandactAsclaims.ANY_PARTYis intended to be used in scenarios where the integration
might wish to enforce its own access controls based on an
authenticated user identity. To support this, the framework has two
functions for extracting the user's identify from an inbound
request. Note that these functions do not return results onPUBLICendpoints, due to the fact that there is no authentication checking
done for these endpoints and no notion of request identity.fromdaml_dit_if.apiimportget_request_parties,\get_single_request_party...elided...@events.webhook.get(label='CSV Table',auth=AuthorizationLevel.INTEGRATION_PARTY)asyncdefon_get_table_csv(request):row_data=get_formatted_table_data()LOG.info('>>>%r/%r',get_request_parties(request),get_single_request_party(request))Timer EventsTo support time-based activities (polling, etc.), the integration
framework provides support for periodic timer events. These are events
that the framework schedules to be invoked at a repeating schedule at
a fixed interval. The interval is specified in seconds and the
decorator contains an optional label argument used to populate the
descriptive text on the integration status display.  As with other
event handlers, the handler for timer events can return a list of
ledger commands to be issued by the framework.@events.time.periodic_interval(env.interval,label='Periodic Timer')asyncdefinterval_timer_elapsed():LOG.debug('Timer elapsed:%r',active_cids)return[exercise(cid,env.templateChoice,{})forcidinactive_cids]The integration framework makes a best effort attempt to call the
timer event handler at the requested periodicity, but no guarantees
are made about exact timing. The requested periodicity should be
considered a minimum. The event handler for a given timer will not be
called re-entrantly.Queue EventsThe integration framework also provides in-memory queues and will call
queue event handlers for message placed on those queues. The intent of
this capability is to provide a way for integrations to respond to
types of events that the other event handlers do not cover.  An
example of this is theSymphony integration,
which uses queue events to accept and process incoming messages
received from the Symphony client library. Because there is no
framework event handler specific to Symphony, the client library
connection is opened as part of initialization and is written to place
incoming messages from the socket onto an internal messaging
queue. The queue handler can then take appropriate action on the
ledger for inbound events. This is also the preferred integration
strategy for websockets - open the connection when the integration
is initialized and have the connection post events to a framework queue
for integration processing.This is how the Sympnony integration registers the handler:defintegration_symphony_receive_dm_main(env:'IntegrationSymphonyReceiveDMEnv',events:'IntegrationEvents'):...elided...@events.queue.message()asyncdefhandle_message(message):return[create(message['type'],message['payload'])]Inbound messages are placed into the queue usingenv.queue.put(...):asyncdefon_im_message(self,im_message):...elided...awaitself.env.queue.put(msg_data)Both the event handler decorator and theputcall accept an optionalqueue_namethat allows messages to be divided into multiple channels
for separate handling.classIntegrationQueueSink:@abc.abstractmethodasyncdefput(self,message:'Any',queue_name:'str'='default'):classIntegrationQueueEvents:@abc.abstractmethoddefmessage(self,queue_name:'str'='default'):There is neither a persistence guarantee nor any retry logic in the
internal queuing mechanism. If a message is placed on an internal
queue and the integration fails or is stopped before the event handler
is invoked, the message will not be processed.Integration Configuration ArgumentsIntegrations can be paramterized using multiple mechanisms, each with
pros and cons. By default, every integration (and Daml Hub automation
in general) is configured with a ledger party and a label. The
integration is connected to the ledger as that party, and the label is
essentially a comment that can be used to describe the purpose of the
automation.Integrations may also receive configuration information via ledger
contracts. TheCoinDesk Integrationuses contracts to describe the number of BTC exchange rates to
maintain on the ledger. The integration accepts this configuration
via the ledger directly. This is also the preferred way to communicate
API keys, tokens etc. to an integration. The Daml ledger privacy model
prevents any private data communicated via contract from exposure to
unauthorized parties.For other sorts of configurations, there is also a mechanism by which
integrations can define configuration fields. These are presented to
the user in the integration configurator and communicated to the
integration via an argument file (int_args.yaml) that's parsed by
the framework and passed to the entrypoint function via theenvargument. These configuration parameters can contain descriptions and
be of data types that are specifically useful integrations. In additon
to numbers and strings, there is also support for configuration fields
that are enumerations, party identifiers, contract template ID's,
contract choice names, etc. They are configured in thefieldsblock
of the integration type definition ofdabl_meta.yaml.Here is an example field list definition from the Ledger Event Log
integration.fields:-id:historyBoundname:Transaction History Bounddescription:>Bound on the length of the history maintained by the integrationfor the purpose of the log fetch endpoint. -1 can be used to removethe bound entirely.field_type:textdefault_value:"1024"id- The machine readable ID of the field. This corresponds to the field name in theenv_classclass instance used to store the environment.name- The user friendly name of the configuration field.description- Long form text describing the purpose of the configuration field.field_type- The type definition of the configuration field.default_value- An optional default value for the field.required- An optional boolean that can be explicitly set tofalseis the field  is optionalType NameDescriptiondefaultortextPlain single line text.numberA number, decimals allowed.integerAn integer. Presented as a field with arrow up/down.partyThe name of a ledger party. Presented as a dropdown.contract_templateThe name of a contract template on the ledger. Presented as a dropdowncontract_choiceThe name of a choice on a contract template specified by anothercontract_templatefield. Specified with a JSON reference to the template field:contract_choice:{"templateNameField": "targetTemplate"}enumAn enumeration. Specified with a JSON list of choices:enum:["Create And Execute", "Trigger Contract"]clobLong form text, presented as multiple lines.