Package: kafka_influxdb
A Kafka consumer for InfluxDB written in Python.Supports InfluxDB 0.9.x and up. For InfluxDB 0.8.x support, check out the0.3.0 tag.Use casesKafka will serve as a buffer for your metric data during high load.Also it’s useful for sending metrics from offshore data centers with unreliable connections to your monitoring backend.QuickstartFor a quick test, run kafka-influxdb inside a container alongside Kafka and InfluxDB. Some sample messages are generated automatically on startup (using kafkacat).Python 2:make
docker exec -it kafkainfluxdb
python -m kafka_influxdb -c config_example.yaml -sPython 3:make RUNTIME=py3
docker exec -it kafkainfluxdb
python -m kafka_influxdb -c config_example.yaml -sPyPy 5.xmake RUNTIME=pypy
docker exec -it kafkainfluxdb
pypy3 -m kafka_influxdb -c config_example.yaml -s --kafka_reader=kafka_influxdb.reader.kafka_python(Note that one additional flag is given:--kafka_reader=kafka_influxdb.reader.kafka_python. This is because PyPy is incompabile with the confluent kafka consumer which is a C-extension to librdkafka. Therefore we use the kafka_python library here, which is compatible with PyPy but a bit slower.)Installationpip install kafka_influxdb
kafka_influxdb -c config-example.yamlPerformanceThe following graph shows the number of messages/s read from Kafka for various Python versions and Kafka consumer plugins.This is testing against a Kafka topic with 10 partitions and five message brokers.As you can see the best performance is achieved on Python 3 using the-Oflag for bytecode optimization in combination with theconfluent-kafkareader (default setup). Note that encoding and sending the data to InfluxDB might lower this maximum performance although you should still see a significant performance boost compared to logstash.BenchmarkFor a quick benchmark, you can start a completekafkacat->Kafka->kafka_influxdb->Influxdbsetup with the following command:makeThis will immediately start reading messages from Kafka and write them into InfluxDB. To see the output, you can use the InfluxDB cli.docker exec -it docker_influxdb_1 bash # Double check your container name
influx
use metrics
show measurementsSupported formatsYou can write a custom encoder to support any input and output format (even fancy things like Protobuf). Look at the examples inside theencoderdirectory to get started. The following formats are officially supported:Input formatsCollectd Graphite ASCII format: :mydatacenter.myhost.load.load.shortterm 0.45 1436357630Collectd JSON format:[{"values":[0.6],"dstypes":["gauge"],"dsnames":["value"],"time":1444745144.824,"interval":10.000,"host":"xx.example.internal","plugin":"cpu","plugin_instance":"1","type":"percent","type_instance":"system"}]Raw InfluxDB line protocol (e.g. for Telegraf support):Output formatsInfluxDB 0.9.2+ line protocol format: :load_load_shortterm,datacenter=mydatacenter,host=myhost value="0.45" 1436357630InfluxDB 0.8.x JSON format(deprecated)ConfigurationTake a look at theconfig-example.yamlto find out how to create a config file.You can overwrite the settings from the commandline. The following parameters are allowed:OptionDescription-h,--helpShow help message and exit--kafka_hostKAFKA_HOSTHostname or IP of Kafka message broker (default: localhost)--kafka_portKAFKA_PORTPort of Kafka message broker (default: 9092)--kafka_topicKAFKA_TOPICTopic for metrics (default: my_topic)--kafka_groupKAFKA_GROUPKafka consumer group (default: my_group)--kafka_readerKAFKA_READERKafka client library to use (kafka_python or confluent) (default: kafka_influxdb.reader.confluent)--influxdb_hostINFLUXDB_HOSTInfluxDB hostname or IP (default: localhost)--influxdb_portINFLUXDB_PORTInfluxDB API port (default: 8086)--influxdb_userINFLUXDB_USERInfluxDB username (default: root)--influxdb_passwordINFLUXDB_PASSWORDInfluxDB password (default: root)--influxdb_dbnameINFLUXDB_DBNAMEInfluxDB database to write metrics into (default: metrics)--influxdb_use_sslUse SSL connection for InfluxDB (default: False)--influxdb_verify_sslVerify the SSL certificate before connecting (default: False)--influxdb_timeoutINFLUXDB_TIMEOUTMax number of seconds to establish a connection to InfluxDB (default: 5)--influxdb_use_udpUse UDP connection for InfluxDB (default: False)--influxdb_retention_policyINFLUXDB_RETENTION_POLICYRetention policy for incoming metrics (default: autogen)--influxdb_time_precisionINFLUXDB_TIME_PRECISIONPrecision of incoming metrics. Can be one of ‘s’, ‘m’, ‘ms’, ‘u’ (default: s)--encoderENCODERInput encoder which converts an incoming message to dictionary (default: collectd_graphite_encoder)--buffer_sizeBUFFER_SIZEMaximum number of messages that will be collected before flushing to the backend (default: 1000)-cCONFIGFILE,--configfileCONFIGFILEConfigfile path (default: None)-s,--statisticsShow performance statistics (default: True)-v,--verboseSet verbosity level. Increase verbosity by adding a v: -v -vv -vvv (default: 0)--versionShow versionComparison with other toolsThere is a Kafka input plugin and an InfluxDB output plugin forlogstash. It supports Influxdb 0.9+. We’ve achieved a message throughput of around5000 messages/secondwith that setup. Check out the configuration at docker/logstash/config.conf. You can run the benchmark yourself:make RUNTIME=logstash
docker exec -it logstash
logstash -f config.confPlease send a Pull Request if you know of other tools that can be mentioned here.