remove_from_game: no-op if player not in game

Otherwise, we mistakenly send "X has left the game" to the channel
despite them never being in the game