Properly close the socket and remove the watcher