github.com/conduitio/conduit-connector-sdk (go): Package sdk implements utilities for implementing a Conduit connector.  Conduit connectors can be thought of as the edges of a Conduit pipeline. They are responsible for reading records from and writing records to third party systems. Conduit uses connectors as plugins that hide the intricacies of working with a particular third party system, so that Conduit itself can focus on efficiently processing records and moving them safely from sources to destinations.  To implement a connector, start by defining a global variable of type Connector, preferably in connector.go at the root of your project to make it easy to discover.  Connector will be used as the starting point for accessing three main connector components that you need to provide:  General advice for implementing connectors:  A Source is responsible for continuously reading data from a third party system and returning it in form of a Record.  Every Source implementation needs to include an UnimplementedSource to satisfy the interface. This allows us to potentially change the interface in the future while remaining backwards compatible with existing Source implementations.  You need to implement the functions required by Source and provide your own implementations. Please look at the documentation of Source for further information about individual functions.  You should also create a constructor function for your source struct. Note that this is the same function that should be set as the value of Connector.NewSource. The constructor should be used to wrap your source in the default middleware. You can add additional middleware, but unless you have a very good reason, you should always include the default middleware.  Additional tips for implementing a source:  A Destination is responsible for writing Record to third party systems.  Every Destination implementation needs to include an UnimplementedDestination to satisfy the interface. This allows us to potentially change the interface in the future while remaining backwards compatible with existing Destination implementations.  You need to implement the functions required by Destination and provide your own implementations. Please look at the documentation of Destination for further information about individual functions.  You should also create a constructor function for your destination struct. Note that this is the same function that should be set as the value of Connector.NewDestination. The constructor should be used to wrap your destination in the default middleware. You can add additional middleware, but unless you have a very good reason, you should always include the default middleware.  Additional tips for implementing a destination:  The SDK provides acceptance tests that can be run in a simple Go test.¹  To run acceptance tests you should create a test file, preferably named acceptance_test.go at the root of your project to make it easy to discover. Inside create a Go test where you trigger the function AcceptanceTest.  AcceptanceTest uses the AcceptanceTestDriver for certain operations. The SDK already provides a default implementation for the driver with ConfigurableAcceptanceTestDriver, although you can supply your own implementation if you need to adjust the behavior of acceptance tests for your connector.  Some acceptance tests will try to write data using the destination and then read the same data using the source. Because of that you need to make sure that the configurations point both to the same exact data store (e.g. in case of the file connector the source and destination need to read and write to the same file).  If your connector does not implement both sides of the connector (a source and a destination) you will need to write a custom driver that knows how to read or write, depending on which side of the connector is not implemented. Here is an example how to do that:  For more information about what behavior can be customized please refer to the AcceptanceTestDriver interface.  ¹Acceptance tests are currently still experimental.
Keywords: conduit|go|golang