Delegate attribute_set to attribute_operator for associations

[#56005352]