bitly/nsq — pull request #236 by mreiferson (2013-09-02 04:25:38+00:00)
Title: nsqd: high CPU when idle w/ lots (thousands) of topics/consumers

Hi,

I ran a simple test to create thousands of topics with a single consumer each which led to nsqd bombing with the following error:

```
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x7fff8cd2f212
```

and then a stack trace for, I guess, every running goroutine.

```
goroutine 1 [chan receive]:
main.main()
/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/nsqgopath.TQxJw0U2/src/github.com/bitly/nsq/nsqd/main.go:145 +0xde6

goroutine 2 [syscall]:

goroutine 4 [syscall]:
os/signal.loop()
/usr/local/Cellar/go/1.1.1/src/pkg/os/signal/signal_unix.go:21 +0x1c
created by os/signal.init·1
/usr/local/Cellar/go/1.1.1/src/pkg/os/signal/signal_unix.go:27 +0x2f

goroutine 6 [chan receive]:
main.func·009()
/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/nsqgopath.TQxJw0U2/src/github.com/bitly/nsq/nsqd/main.go:102 +0x39
created by main.main
/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/nsqgopath.TQxJw0U2/src/github.com/bitly/nsq/nsqd/main.go:104 +0x697

goroutine 7 [select]:
main.(*NSQd).idPump(0xc20013a000)
/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/nsqgopath.TQxJw0U2/src/github.com/bitly/nsq/nsqd/nsqd.go:353 +0x2bd
main.func·010()
/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/nsqgopath.TQxJw0U2/src/github.com/bitly/nsq/nsqd/nsqd.go:92 +0x27
github.com/bitly/nsq/util.func·001()
/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/nsqgopath.TQxJw0U2/src/github.com/bitly/nsq/util/wait_group_wrapper.go:14 +0x2c
created by github.com/bitly/nsq/util.(*WaitGroupWrapper).Wrap
/var/folders/4q/k4wfn18j7v79zj7l2tqr_5p80000gn/T/nsqgopath.TQxJw0U2/src/github.com/bitly/nsq/util/wait_group_wrapper.go:16 +0xa6

[ ... etc ... ]
```
Here's the ruby program that triggered it:

```ruby
require 'nsq'
reader = NSQ::Reader.new(:nsqd_tcp_addresses => '127.0.0.1:4150')
5000.times do |i|
reader.subscribe("foo-#{i)}", "bar") do
end
end
reader.run
```

This may not be a valid use case for nsq, but perhaps some throttling of topics may be necessary.