bitly/go-nsq — pull request #95 by mreiferson (2014-11-22 18:08:08+00:00)
Title: nsq_to_nsq: panic on shutdown

My steps:

Using the following run file for nsq_to_nsq, have a local nsqd running. Do not have a destination nsqd running. `sv stop` nsq_to_nsq once (this hangs for 10 seconds). nsq_to_nsq continues backing off that it cannot connect to the remote nsqd. `sv stop` again, panic. If you need more details, please ask.

Run file:
```
#!/bin/sh
exec 2>&1

export GOMAXPROCS=2

exec chpst -u nsq:nsq /opt/nsq/bin/nsq_to_nsq \
-topic=stuff \
-destination-topic=stuff \
-destination-nsqd-tcp-address=nsq-instance-1:1910 \
-nsqd-tcp-address=localhost:1910 \
-channel=nsq_to_nsq \
-consumer-opt=tls_root_ca_file,/opt/nsq/certs/nsq-ca-cert.pem \
-consumer-opt=tls_cert,/opt/nsq/certs/nsq-client-cert.pem \
-consumer-opt=tls_key,/opt/nsq/certs/nsq-client-key.pem \
-consumer-opt=tls_v1,true \
-producer-opt=tls_root_ca_file,/opt/nsq/certs/nsq-ca-cert.pem \
-producer-opt=tls_cert,/opt/nsq/certs/nsq-client-cert.pem \
-producer-opt=tls_key,/opt/fnsq/certs/nsq-client-key.pem \
-producer-opt=tls_v1,true
```

Log trace on panic:
```
2014-11-21_08:55:07.68147 2014/11/21 08:55:07 INF 2 (nsq-instance-1:1910) connecting to nsqd
2014-11-21_08:55:08.68159 2014/11/21 08:55:08 ERR 2 (nsq-instance-1:1910) error connecting to nsqd - dial tcp 1.2.3.4:5678: i/o timeout
2014-11-21_08:55:08.68162 2014/11/21 08:55:08 ERR 1 [stuff/nsq_to_nsq] Handler returned error (dial tcp 1.2.3.4:5678: i/o timeout) for msg 076f3d041e875000
2014-11-21_08:55:08.68163 2014/11/21 08:55:08 INF 2 (nsq-instance-1:1910) connecting to nsqd
2014-11-21_08:55:09.68180 2014/11/21 08:55:09 ERR 2 (nsq-instance-1:1910) error connecting to nsqd - dial tcp 1.2.3.4:5678: i/o timeout
2014-11-21_08:55:09.68182 2014/11/21 08:55:09 ERR 1 [stuff/nsq_to_nsq] Handler returned error (dial tcp 1.2.3.4:5678: i/o timeout) for msg 076f3d041e875001
2014-11-21_08:55:09.68183 2014/11/21 08:55:09 INF 2 (nsq-instance-1:1910) connecting to nsqd
2014-11-21_08:55:10.59484 2014/11/21 08:55:10 INF 1 [stuff/nsq_to_nsq] stopping handlers
2014-11-21_08:55:10.59814 panic: runtime error: send on closed channel
2014-11-21_08:55:10.59815
2014-11-21_08:55:10.59816 goroutine 25 [running]:
2014-11-21_08:55:10.59816 runtime.panic(0x6fe900, 0x8eea1e)
2014-11-21_08:55:10.59816	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/runtime/panic.c:279 +0xf5
2014-11-21_08:55:10.59817 github.com/bitly/go-nsq.(*Consumer).onConnMessage(0xc20802e6c0, 0xc208003c20, 0xc2080c18b0)
2014-11-21_08:55:10.59817	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/consumer.go:490 +0x90
2014-11-21_08:55:10.59818 github.com/bitly/go-nsq.(*consumerConnDelegate).OnMessage(0xc208036040, 0xc208003c20, 0xc2080c18b0)
2014-11-21_08:55:10.59818	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/delegates.go:119 +0x3e
2014-11-21_08:55:10.59819 github.com/bitly/go-nsq.(*Conn).readLoop(0xc208003c20)
2014-11-21_08:55:10.59819	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/conn.go:495 +0xb5e
2014-11-21_08:55:10.59819 created by github.com/bitly/go-nsq.(*Conn).Connect
2014-11-21_08:55:10.59820	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/conn.go:168 +0x65c
2014-11-21_08:55:10.59821
2014-11-21_08:55:10.59821 goroutine 16 [select]:
2014-11-21_08:55:10.59821 main.main()
2014-11-21_08:55:10.59822	/Users/mreiferson/dev/src/github.com/bitly/nsq/apps/nsq_to_nsq/nsq_to_nsq.go:396 +0x1586
2014-11-21_08:55:10.59822
2014-11-21_08:55:10.59823 goroutine 19 [finalizer wait, 1 minutes]:
2014-11-21_08:55:10.59823 runtime.park(0x417500, 0x8f3788, 0x8f1a09)
2014-11-21_08:55:10.59823	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/runtime/proc.c:1369 +0x89
2014-11-21_08:55:10.59824 runtime.parkunlock(0x8f3788, 0x8f1a09)
2014-11-21_08:55:10.59824	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/runtime/proc.c:1385 +0x3b
2014-11-21_08:55:10.59825 runfinq()
2014-11-21_08:55:10.59825	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/runtime/mgc0.c:2644 +0xcf
2014-11-21_08:55:10.59825 runtime.goexit()
2014-11-21_08:55:10.59826	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/runtime/proc.c:1445
2014-11-21_08:55:10.59826
2014-11-21_08:55:10.59826 goroutine 20 [syscall]:
2014-11-21_08:55:10.59827 os/signal.loop()
2014-11-21_08:55:10.59827	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/os/signal/signal_unix.go:21 +0x1e
2014-11-21_08:55:10.59828 created by os/signal.init·1
2014-11-21_08:55:10.59828	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/os/signal/signal_unix.go:27 +0x32
2014-11-21_08:55:10.59828
2014-11-21_08:55:10.59829 goroutine 21 [select]:
2014-11-21_08:55:10.59829 github.com/bitly/go-nsq.(*Consumer).rdyLoop(0xc20802e6c0)
2014-11-21_08:55:10.59831	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/consumer.go:625 +0xee1
2014-11-21_08:55:10.59831 created by github.com/bitly/go-nsq.NewConsumer
2014-11-21_08:55:10.59832	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/consumer.go:155 +0x5ef
2014-11-21_08:55:10.59832
2014-11-21_08:55:10.59833 goroutine 22 [IO wait]:
2014-11-21_08:55:10.59833 net.runtime_pollWait(0x7f08e63c7b20, 0x77, 0x0)
2014-11-21_08:55:10.59834	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/runtime/netpoll.goc:146 +0x66
2014-11-21_08:55:10.59834 net.(*pollDesc).Wait(0xc208132f40, 0x77, 0x0, 0x0)
2014-11-21_08:55:10.59834	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/fd_poll_runtime.go:84 +0x46
2014-11-21_08:55:10.59835 net.(*pollDesc).WaitWrite(0xc208132f40, 0x0, 0x0)
2014-11-21_08:55:10.59835	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/fd_poll_runtime.go:93 +0x42
2014-11-21_08:55:10.59836 net.(*netFD).connect(0xc208132ee0, 0x0, 0x0, 0x7f08e63c6c88, 0xc2080e5f60, 0xecc00f4ee, 0x28a53e51, 0x8f4de0, 0x0, 0x0)
2014-11-21_08:55:10.59836	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/fd_unix.go:114 +0x22e
2014-11-21_08:55:10.59837 net.(*netFD).dial(0xc208132ee0, 0x7f08e63c6c38, 0x0, 0x7f08e63c6c38, 0xc208010c60, 0xecc00f4ee, 0x28a53e51, 0x8f4de0, 0x7d8440, 0x0, ...)
2014-11-21_08:55:10.59837	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/sock_posix.go:115 +0x321
2014-11-21_08:55:10.59838 net.socket(0x744b90, 0x3, 0x2, 0x1, 0x0, 0xc208010c00, 0x7f08e63c6c38, 0x0, 0x7f08e63c6c38, 0xc208010c60, ...)
2014-11-21_08:55:10.59839	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/sock_posix.go:91 +0x424
2014-11-21_08:55:10.59839 net.internetSocket(0x744b90, 0x3, 0x7f08e63c6c38, 0x0, 0x7f08e63c6c38, 0xc208010c60, 0xecc00f4ee, 0x28a53e51, 0x8f4de0, 0x1, ...)
2014-11-21_08:55:10.59839	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/ipsock_posix.go:137 +0x161
2014-11-21_08:55:10.59840 net.dialTCP(0x744b90, 0x3, 0x0, 0xc208010c60, 0xecc00f4ee, 0x28a53e51, 0x8f4de0, 0x0, 0x0, 0x0)
2014-11-21_08:55:10.59840	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/tcpsock_posix.go:156 +0x13a
2014-11-21_08:55:10.59841 net.dialSingle(0x744b90, 0x3, 0x7fff12835d6f, 0x12, 0x0, 0x0, 0x7f08e63c6ba8, 0xc208010c60, 0xecc00f4ee, 0x28a53e51, ...)
2014-11-21_08:55:10.59841	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/dial.go:238 +0x3ef
2014-11-21_08:55:10.59842 net.func·016(0xecc00f4ee, 0x28a53e51, 0x8f4de0, 0x0, 0x0, 0x0, 0x0)
2014-11-21_08:55:10.59842	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/dial.go:164 +0x132
2014-11-21_08:55:10.59843 net.dial(0x744b90, 0x3, 0x7f08e63c6ba8, 0xc208010c60, 0x7f08e6238978, 0xecc00f4ee, 0x28a53e51, 0x8f4de0, 0x0, 0x0, ...)
2014-11-21_08:55:10.59843	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/fd_unix.go:40 +0x75
2014-11-21_08:55:10.59844 net.(*Dialer).Dial(0xc2080eb9c0, 0x744b90, 0x3, 0x7fff12835d6f, 0x12, 0x0, 0x0, 0x0, 0x0)
2014-11-21_08:55:10.59844	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/dial.go:171 +0x44f
2014-11-21_08:55:10.59845 net.DialTimeout(0x744b90, 0x3, 0x7fff12835d6f, 0x12, 0x3b9aca00, 0x0, 0x0, 0x0, 0x0)
2014-11-21_08:55:10.59845	/usr/local/Cellar/go/1.3.1/libexec/src/pkg/net/dial.go:151 +0xda
2014-11-21_08:55:10.59846 github.com/bitly/go-nsq.(*Conn).Connect(0xc20802dc20, 0x9, 0x0, 0x0)
2014-11-21_08:55:10.59846	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/conn.go:135 +0x7f
2014-11-21_08:55:10.59847 github.com/bitly/go-nsq.(*Producer).connect(0xc20809a000, 0x0, 0x0)
2014-11-21_08:55:10.59847	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/producer.go:220 +0x3f4
2014-11-21_08:55:10.59848 github.com/bitly/go-nsq.(*Producer).sendCommandAsync(0xc20809a000, 0xc2080c1860, 0xc2080d0900, 0xc208010ae0, 0x3, 0x3, 0x0, 0x0)
2014-11-21_08:55:10.59849	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/producer.go:182 +0xd7
2014-11-21_08:55:10.59850 github.com/bitly/go-nsq.(*Producer).PublishAsync(0xc20809a000, 0x7fff12835d47, 0x9, 0xc20807e000, 0xea, 0x600, 0xc2080d0900, 0xc208010ae0, 0x3, 0x3, ...)
2014-11-21_08:55:10.59850	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/producer.go:128 +0x1bb
2014-11-21_08:55:10.59851 main.(*PublishHandler).HandleMessage(0xc2080335e0, 0xc2080ef1d0, 0x0, 0x0)
2014-11-21_08:55:10.59851	/Users/mreiferson/dev/src/github.com/bitly/nsq/apps/nsq_to_nsq/nsq_to_nsq.go:244 +0x36e
2014-11-21_08:55:10.59852 github.com/bitly/go-nsq.(*Consumer).handlerLoop(0xc20802e6c0, 0x7f08e63c6ae8, 0xc2080335e0)
2014-11-21_08:55:10.59852	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/consumer.go:926 +0x1c5
2014-11-21_08:55:10.59853 created by github.com/bitly/go-nsq.(*Consumer).AddConcurrentHandlers
2014-11-21_08:55:10.59853	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/consumer.go:908 +0xf1
2014-11-21_08:55:10.59854
2014-11-21_08:55:10.59854 goroutine 23 [chan receive, 1 minutes]:
2014-11-21_08:55:10.59855 main.(*PublishHandler).responder(0xc2080335e0)
2014-11-21_08:55:10.59856	/Users/mreiferson/dev/src/github.com/bitly/nsq/apps/nsq_to_nsq/nsq_to_nsq.go:92 +0xb9
2014-11-21_08:55:10.59856 created by main.main
2014-11-21_08:55:10.59856	/Users/mreiferson/dev/src/github.com/bitly/nsq/apps/nsq_to_nsq/nsq_to_nsq.go:382 +0x12f2
2014-11-21_08:55:10.59857
2014-11-21_08:55:10.59857 goroutine 26 [select]:
2014-11-21_08:55:10.59858 github.com/bitly/go-nsq.(*Conn).writeLoop(0xc208003c20)
2014-11-21_08:55:10.59858	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/conn.go:523 +0x7bc
2014-11-21_08:55:10.59859 created by github.com/bitly/go-nsq.(*Conn).Connect
2014-11-21_08:55:10.59859	/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/godep/rev/ac/221df5bdb6d5bfc624a297b5b00b59d7065be2/src/github.com/bitly/go-nsq/conn.go:169 +0x677
```