rsyslog/rsyslog — pull request #186 by oldmantaiter (2014-12-15 18:06:14+00:00)
Title: omkafka: add stats and dynamic topic

stats:
- submitted: producer submitted message to broker.
- failures: producer failed to submit message to broker.

Topic is able to be a template, the topics are stored in the instanceData and the kafka_rd_t object has been moved into instanceData. This is alright because librdkafka is threadsafe and to acheive the dynamic topic results it is required.
The topics are stored in a hashtable due to librdkafka leaking memory when passing in a conf object, plus us keeping a local topic "list" makes the subsequent topic conf and rd_kafka_topic_new calls unnecessary. The move of the topic down the writeKafka is for dynamic parsing of the topic for each message that comes through.

Additional items:
- Added safety to rd_kafka_t destruction, where it flushes the queued messages before object destruction.
- Dynamic topic is enabled with dyntopic="on" in the configuration.
- On HUP the active topics are closed, since we only give the topic hashtable 50 items, this could cause issues when rsyslog is running over a long period of time with lots of dynamically parsed topics from templates.
- Not sure if 50 is an acceptable number of items for the hashtable, I'm open to either increasing or decreasing it.
- After the HUP, writeKafka should instantiate a new topic and add it back to the hashtable at a minimal performance hit.

These changes have been run through valgrind and I can confirm there is no leaking (at least within my setup).