tidepool-org/go-common — pull request #3 by cheddar (2014-07-29 15:10:28+00:00)
Title: Ensure coordinatorManager.Close() will eventually complete

In local development, I noticed that tide-whisperer was hanging on shutdown. I killed it again and got the following panic:

```
panic: runtime error: send on closed channel

goroutine 34 [running]:
runtime.panic(0x34ee00, 0x56e89e)
/usr/local/go/src/pkg/runtime/panic.c:279 +0xf5
github.com/tidepool-org/go-common.(*Server).Close(0xc2080018b0, 0x0, 0x0)
/Users/cheddar/github/tidepool-org/tide-whisperer/src/github.com/tidepool-org/go-common/server.go:25 +0x75
main.func·006()
/Users/cheddar/github/tidepool-org/tide-whisperer/tide-whisperer.go:178 +0x1a8
created by main.main
/Users/cheddar/github/tidepool-org/tide-whisperer/tide-whisperer.go:182 +0x10f7

goroutine 16 [chan send, 14546 minutes]:
github.com/tidepool-org/go-common/clients/hakken.(*coordinatorManager).Close(0xc20807c108, 0x0, 0x0)
/Users/cheddar/github/tidepool-org/tide-whisperer/src/github.com/tidepool-org/go-common/clients/hakken/coordinator.go:137 +0xfe
github.com/tidepool-org/go-common/clients/hakken.(*HakkenClient).Close(0xc20807c0e0, 0x0, 0x0)
/Users/cheddar/github/tidepool-org/tide-whisperer/src/github.com/tidepool-org/go-common/clients/hakken/hakken.go:99 +0x45
main.main()
/Users/cheddar/github/tidepool-org/tide-whisperer/tide-whisperer.go:185 +0x1120

goroutine 19 [finalizer wait, 14549 minutes]:
runtime.park(0x175e0, 0x573190, 0x571c89)
/usr/local/go/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x573190, 0x571c89)
/usr/local/go/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
/usr/local/go/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1445

goroutine 20 [syscall]:
os/signal.loop()
/usr/local/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
/usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x32

goroutine 25 [select]:
github.com/tidepool-org/go-common/clients/shoreline.func·001()
/Users/cheddar/github/tidepool-org/tide-whisperer/src/github.com/tidepool-org/go-common/clients/shoreline/shoreline.go:127 +0x20f
created by github.com/tidepool-org/go-common/clients/shoreline.(*ShorelineClient).Start
/Users/cheddar/github/tidepool-org/tide-whisperer/src/github.com/tidepool-org/go-common/clients/shoreline/shoreline.go:137 +0xaa

goroutine 17 [syscall, 14546 minutes]:
runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1445

goroutine 33 [semacquire, 14546 minutes]:
sync.runtime_Semacquire(0xc20807c18c)
/private/var/folders/00/0sdwh000h01000cxqpysvccm0035qk/T/makerelease530016500/go/src/pkg/runtime/sema.goc:199 +0x30
sync.(*Mutex).Lock(0xc20807c188)
/usr/local/go/src/pkg/sync/mutex.go:66 +0xd6
github.com/tidepool-org/go-common/clients/hakken.(*coordinatorManager).getClients(0xc20807c108, 0x0, 0x0, 0x0)
/Users/cheddar/github/tidepool-org/tide-whisperer/src/github.com/tidepool-org/go-common/clients/hakken/coordinator.go:59 +0x55
github.com/tidepool-org/go-common/clients/hakken.func·003()
/Users/cheddar/github/tidepool-org/tide-whisperer/src/github.com/tidepool-org/go-common/clients/hakken/hakken.go:161 +0xf6
created by github.com/tidepool-org/go-common/clients/hakken.(*HakkenClient).Publish
/Users/cheddar/github/tidepool-org/tide-whisperer/src/github.com/tidepool-org/go-common/clients/hakken/hakken.go:167 +0x12b
```

Which seems to indicate that there was a deadlock between coordinatorManager.Close() and coordinatorManager.getClients().

It appears that coordinatorManager.Close() was waiting on a receive on the close channel, but nothing was on the other end of it. So this PR introduces a timeout to that wait and returns an error if it wasn't able to successfully close.