Fix #158: Remember which fields were removed and when rebuilding the set of fields by means of rebuildCachedFields, remove all fields that were removed before.