Move the server removal to after the user removal so things responding to user disconnect that care about the server (like whowas) can still work