#24 patch to remove existing layer clients

Somewhere the end port connection+port isn't being removed. This is a
patch for this, so a client is force shutdown so the new connection can
be added to the dictionary.

I'd like some one to test they can also connect/disconnect to a layer with this.