Simplified `as_of` logic, handle non-id pk fields
Double check for deleted items since history_date can match