Use set to track overrides instead of list

Lookups will be slightly faster this way and it will dedupe.
Re-order some properties to ensure we handle edge cases correctly