Whatever we do heuristically based solely on the field values will be wrong some of the time.  I think to better solve the "actual" problem, we should take the provenance of the detail itself into account -- if the provenance was the sync source, then we can feel free to treat it as a modification, otherwise we should always treat it as an add/remove.  Do we expose that provenance data in the partial aggregate at all?