Repository: wurstmeister/kafka-docker

Issue #374: Zookeeper "zxid"  error when restarting zookeeper container on a different swarm machine 
Created: 2018-08-01T10:33:54Z
Hello,
If I'm using your docker-compose-swarm.yml

with 2 docker hosts in the swarm (1 master 1 worker) and scale the kafka to 2.
when zookeeper is moving from one host to the other. (i.e by restarting the container)
I'm getting these error in zookeeper:
2018-08-01 10:24:13,735 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /10.0.5.3:53856
2018-08-01 10:24:13,736 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@901] - Refusing session request for client /10.0.5.3:53856 as it has seen zxid 0x7a our last zxid is 0x0 client must try another server
2018-08-01 10:24:13,737 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /10.0.5.3:53856 (no session established for client)



and kafka stops working. (until I restart kafka as mentionned here : #315 )
Is something missing in this configuration ?

Duplicate issue #378: Cannot connect to zookeeper after restarting zookeeper pod
Created: 2018-08-02T15:48:31Z
I am running Kafka and Zookeeper on a Kubernetes setup. I launch both kafka and zookeeper pod. They are both working fine. I bring down zookeeper pod and kafka has the following logs:
java.net.NoRouteToHostException: Host is unreachable
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2018-08-02 15:29:10,380] INFO Opening socket connection to server zookeeper.casb-deployment.svc.cluster.local/10.103.187.203:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-08-02 15:29:16,384] WARN Client session timed out, have not heard from server in 7184ms for sessionid 0x164fb39384c0000 (org.apache.zookeeper.ClientCnxn)
[2018-08-02 15:29:16,385] INFO Client session timed out, have not heard from server in 7184ms for sessionid 0x164fb39384c0000, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-08-02 15:29:18,230] INFO Opening socket connection to server zookeeper.casb-deployment.svc.cluster.local/10.103.187.203:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-08-02 15:29:24,230] WARN Client session timed out, have not heard from server in 7745ms for sessionid 0x164fb39384c0000 (org.apache.zookeeper.ClientCnxn)
[2018-08-02 15:29:24,231] INFO Client session timed out, have not heard from server in 7745ms for sessionid 0x164fb39384c0000, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-08-02 15:29:25,681] INFO Opening socket connection to server zookeeper.casb-deployment.svc.cluster.local/10.103.187.203:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-08-02 15:29:31,689] WARN Client session timed out, have not heard from server in 7358ms for sessionid 0x164fb39384c0000 (org.apache.zookeeper.ClientCnxn)
[2018-08-02 15:29:31,689] INFO Client session timed out, have not heard from server in 7358ms for sessionid 0x164fb39384c0000, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-08-02 15:29:33,488] INFO Opening socket connection to server zookeeper.casb-deployment.svc.cluster.local/10.103.187.203:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

Even after restarting zookeeper, Kafka cannot connect to the zookeeper pod and keeps on throwing the same exceptions.
Kafka.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: casb-deployment
  name: kafkaservice
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kafkaservice
    spec:
      hostname: kafkaservice
      containers:
      - name: kafkaservice
        imagePullPolicy: IfNotPresent
        image: wurstmeister/kafka:1.1.0
        env:
         - name: KAFKA_BROKER_ID
           value: "1"
         - name: KAFKA_ADVERTISED_HOST_NAME
           value: "kafkaservice"
         - name: KAFKA_CREATE_TOPICS
           value: "topic1:1:1"
         - name: KAFKA_MESSAGE_TIMESTAMP_TYPE
           value: "LogAppendTime"
         - name: KAFKA_LOG_MESSAGE_TIMESTAMP_TYPE
           value: "LogAppendTime"   
         - name: KAFKA_ZOOKEEPER_CONNECT
           value: "zookeeper:2181"
        ports: 
        - name: port9092
          containerPort: 9092          
        volumeMounts:
        - mountPath: /kafka/kafka-logs-kafkaservice
          name: kafka-volume
      volumes:
      - name: kafka-volume
        hostPath:
          path: /var/lib/kafka-logs

apiVersion: v1
kind: Service
metadata:
  namespace: casb-deployment
  name: kafkaservice
  labels:
    app: kafkaservice
spec:
  selector:
    app: kafkaservice
  ports:
  - name: port9092
    port: 9092
    targetPort: 9092
    protocol: TCP


Zookeeper.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: casb-deployment
  name: zookeeper
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      hostname: zookeeper
      containers:
      - name: zookeeper
        imagePullPolicy: IfNotPresent
        image: zookeeper:3.3.6
        env:
         - name: ZOO_MY_ID
           value: "1"
        ports: 
        - name: port2181
          containerPort: 2181
        volumeMounts:
        - mountPath: /data
          name: zookeeper-data
        - mountPath: /datalogs
          name: zookeeper-logs
      volumes:
      - name: zookeeper-data
        hostPath:
          path: /var/lib/zookeeper/data
      - name: zookeeper-logs
        hostPath:
          path: /var/lib/zookeeper/logs

apiVersion: v1
kind: Service
metadata:
  namespace: casb-deployment
  name: zookeeper
  labels:
    app: zookeeper
spec:
  selector:
    app: zookeeper
  ports:
  - name: port2181
    port: 2181
    targetPort: 2181
    protocol: TCP

I am guessing its a Kubernetes related issue. I also looked into this issue but I am not sure if its related to mine.

Closing comment (2018-08-02T16:33:58Z): Looking at the logs, I'd guess it was something like state being maintained in the zookeeper client (similar to #374).
A few basic things to check:

zookeeper is configured correctly (https://hub.docker.com/_/zookeeper/ would indicate you'd need to specify the ZOO_SERVERS environment var or disable replicated mode)
zookeeper is healthy after restart
restarting kafka recovers from this situation

It may also be known behaviour, normally zookeeper would be run as a quorum in replicated mode so that the cluster can tolerate failure / restarts. Only having one zookeeper node is very fragile.
However, it seems as though this issue is either Kafka / Zookeeper / Kubernetes related - not to do with the kafka-docker image itself. I'd recommend trying to ask on the respective mailing lists / forums for more specific help.
I don't have any free bandwidth at the moment to dig into this so I'll leave the issue open for a while to see if anyone else can provide help / assistance.
Thanks