Remove patches only after processing all specs

Patches that are applied in several specs have to be removed only after
all spec files have been cleaned up.