Handle delete_by_keys for association to clean up unnecessary entries.