Refactor unassigning of contacts