Remove the user after a ban