actually remove client if it is disconnected