Unexpected client disconnections can cause a panic and crash the server

Ran into an odd edgecase, detailed in [this gist](https://gist.github.com/rubyist/9e3c089142945c27867e). When a client disconnects before the Copy operation is finished, the Close() method will be called twice. This causes wg.Done() to be called one too many times, making Go panic down in the sync code as it goes negative.

This change switches to using a conn pointer and keeps track of when the Conn is Close()d so it only calls Done() once.