Handle case of sparse segments on shards