Repository: Parsely/pykafka

Issue #449: synchronous producer hangs when kafka is unreachable
Created: 2016-02-05T20:37:39Z
This issue was brought to my attention by Michael Kowalchik in the mailing list
Steps to reproduce

Run a synchronous producer pointing at a kafka cluster with a single node
Kill the single kafka node while the producer is running
Observe that the producer's main thread hangs and must be killed externally

The expected result is that the producer's main thread should raise a SocketDisconnectedError from produce().
The following sequence of events causes this bug. When a broker is unavailable, SocketDisconnectedError is raised and caught. If there are no brokers available, this call to _update() raises an exception. This in turn kills the worker thread, meaning that code after the _update call, including pushing the original SocketDisconnectedError onto the delivery reports queue, does not execute. On the main thread, produce() is still waiting for the delivery report, which will never come because the worker has died.
Possible solutions include ignoring certain exception types from the _update() call, or catching any exception that happens between to_retry being populated and the delivery report being enqueued. I'd lean toward simply catching and warning about this specific exception, maybe giving it a more specific name in the process.

Duplicate issue #456: If connectivity to Kafka is lost the async producer dies and get_delivery_report() fails to report any errors
Created: 2016-02-08T18:43:27Z
Setup:

OS: Ubuntu 14.04 (x86_64)
Kafka: 2.11-0.9.0.0 (2 brokers)
Zookeeper: 3.4.6 (3 servers)
Python: 2.7.6
pip freeze:

argparse==1.2.1
gevent==1.1b6
greenlet==0.4.9
kazoo==2.2.1
pykafka==2.2.0
python-snappy==0.5
six==1.10.0
tabulate==0.7.5
wsgiref==0.1.2

What I'm doing: https://gist.github.com/gg7/07dbc112fa25ed8dcbf6
After IPython command number 12 I executed iptables -I INPUT -p tcp --sport 9092 -j DROP && iptables -I OUTPUT -p tcp --dport 9092 -j DROP on my machine.
I was hoping that producer.get_delivery_report() would report that some/all of my messages were not ack-ed. Is this the intended behaviour?
Potentially a second issue: if I delete my iptables rule and try producing more messages here's what happens:
ERROR:pykafka.producer:Exception encountered in worker thread:
  File "/home/george/dev/lib/kafka_log/env/lib/python2.7/site-packages/pykafka/producer.py", line 438, in queue_reader
    self.producer._send_request(batch, self)
  File "/home/george/dev/lib/kafka_log/env/lib/python2.7/site-packages/pykafka/producer.py", line 366, in _send_request
    self._update()
  File "/home/george/dev/lib/kafka_log/env/lib/python2.7/site-packages/pykafka/producer.py", line 200, in _update
    self._cluster.update()
  File "/home/george/dev/lib/kafka_log/env/lib/python2.7/site-packages/pykafka/cluster.py", line 410, in update
    metadata = self._get_metadata()
  File "/home/george/dev/lib/kafka_log/env/lib/python2.7/site-packages/pykafka/cluster.py", line 269, in _get_metadata
    metadata = self._request_metadata(broker_connects, topics)
  File "/home/george/dev/lib/kafka_log/env/lib/python2.7/site-packages/pykafka/cluster.py", line 228, in _request_metadata
    for host, port in broker_connects:

produce() threw an exception
Traceback (most recent call last):
  File "<ipython-input-12-621d76f18fda>", line 4, in <module>
    producer.produce(message=value, partition_key=b'hello')
  File "/home/george/dev/lib/kafka_log/env/lib/python2.7/site-packages/pykafka/producer.py", line 271, in produce
    self._raise_worker_exceptions()
  File "/home/george/dev/lib/kafka_log/env/lib/python2.7/site-packages/pykafka/producer.py", line 168, in _raise_worker_exceptions
    raise ex
TypeError: 'NoneType' object is not iterable
PyKafka doesn't seem to make any attempts to reconnect to Kafka after such a major failure. What's the best way to detect its "death" and restart it? I'd like to maintain a long-lived connection to Kafka and am interested in doing things correctly.

Closing comment (2016-02-08T20:58:57Z): Thanks for the detailed report @gg7. The traceback  you posted has been fixed on master, but it is not yet in a released version. This fix may also address the delivery report issue you're seeing.
I'm going to mark this as a duplicate of #449 - please comment or reopen if you think I've misevaluated.