Package: pipelinewise-tap-kafka
pipelinewise-tap-kafkaThis is aSingertap that reads data from Kafka topic and produces JSON-formatted data following theSinger spec.This is aPipelineWisecompatible target connector.How to use itThe recommended method of running this tap is to use it fromPipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation atKafkaIf you want to run thisSinger Tapindependently please read further.Install and RunFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.It's recommended to use a virtualenv:python3-mvenvvenvpipinstallpipelinewise-tap-kafkaorpython3-mvenvvenv.venv/bin/activatepipinstall--upgradepippipinstall.ConfigurationCreate a config.json{
  "bootstrap_servers": "foo.com,bar.com",
  "group_id": "my_group",
  "topic": "my_topic",
  "primary_keys": {
    "id": "/path/to/primary_key"
  }
}Full list of options inconfig.json:PropertyTypeRequired?Descriptionbootstrap_serversStringYeshost[:port]string (or list of comma separatedhost[:port]strings) that the consumer should contact to bootstrap initial cluster metadata.group_idStringYesThe name of the consumer group to join for dynamic partition assignment (if enabled), and to use for fetching and committing offsets.topicStringYesName of kafka topic to subscribe topartitionsList(Default: [] (all)) Partition(s) of topic to consume, example[0,4]primary_keysObjectOptionally you can define primary key for the consumed messages. It requires a column name and/slashed/pathsala xpath selector to extract the value from the kafka messages. The extracted column will be added to every output singer message.use_message_keyBool(Default: true) Defines whether to use Kafka message key as a primary key for the record. Note: if a custom primary key(s) has been defined, it will be used instead of the message_key.initial_start_timeString(Default: latest) Start time reference of the message consumption if no bookmarked position instate.json. One of:beginning,earliest,latestor an ISO-8601 formatted timestamp string.max_runtime_msInteger(Default: 300000) The maximum time for the tap to collect new messages from Kafka topic. If this time exceeds it will flush the batch and close kafka connection.commit_interval_msInteger(Default: 5000) Number of milliseconds between two commits. This is different than the kafka auto commit feature. Tap-kafka sends commit messages automatically but only when the data consumed successfully and persisted to local store.consumer_timeout_msInteger(Default: 10000) KafkaConsumer setting. Number of milliseconds to block during message iteration before raising StopIterationsession_timeout_msInteger(Default: 30000) KafkaConsumer setting. The timeout used to detect failures when using Kafka’s group management facilities.heartbeat_interval_msInteger(Default: 10000) KafkaConsumer setting. The expected time in milliseconds between heartbeats to the consumer coordinator when using Kafka’s group management facilities.max_poll_interval_msInteger(Default: 300000) KafkaConsumer setting. The maximum delay between invocations of poll() when using consumer group management.message_formatString(Default: json) Supported message formats arejsonandprotobuf.proto_schemaStringProtobuf message format in.protosyntax. Required if themessage_formatisprotobuf.proto_classes_dirString(Default: current working dir)debug_contextsStringcomma separated list of debug contexts to enable for the consumersee librkafkaThis tap reads Kafka messages and generating singer compatible SCHEMA and RECORD messages in the following format.Property NameDescriptionMESSAGE_TIMESTAMPTimestamp extracted from the kafka metadataMESSAGE_OFFSETOffset extracted from the kafka metadataMESSAGE_PARTITIONPartition extracted from the kafka metadataMESSAGEThe original Kafka messageMESSAGE_KEY(Optional) Added by default (can be overridden) in case no custom keys definedDYNAMIC_PRIMARY_KEY(S)(Optional) Dynamically added primary key values, extracted from the Kafka messageRun the tap in Discovery Modetap-kafka --config config.json --discover                # Should dump a Catalog to stdout
tap-kafka --config config.json --discover > catalog.json # Capture the CatalogAdd Metadata to the CatalogEach entry under the Catalog's "stream" key will need the following metadata:{
  "streams": [
    {
      "stream_name": "my_topic"
      "metadata": [{
        "breadcrumb": [],
        "metadata": {
          "selected": true,
        }
      }]
    }
  ]
}Run the tap in Sync Modetap-kafka --config config.json --properties catalog.jsonThe tap will write bookmarks to stdout which can be captured and passed as an optional--state state.jsonparameter to the tap for the next sync.To run tests:Install python test dependencies in a virtual env and run nose unit and integration testspython3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install -e .[test]To run unit tests:make unit_testTo run integration test:make integration_testTo run pylint:Install python dependencies and run python linterpython3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install -e .[test]
  pylint tap_kafka -d C,W,unexpected-keyword-arg,duplicate-code