Remove from_ids lookup in attach_foreignkey