More optimization: defer computing non_leaders, it's usually not used