Do not KeyError when told to leave a channel the bot is not in