Fix weird case of playlist skipping when removing leader