Add in listener removal

Fix for issue #18 

Have made two (unsuccessful) attempts at testing this. First was a run up a large number of connections, close then and then measure memory difference. Second was to count number of 'shutdown' listeners in place. Neither gave expected result.

Anyway here is the fix for @edpelesh's issue and I suggest we come back to this later.