Properly removing assigned contact from an activity