Don't calculate all in-place changes to determine if attribute_changed?

Calling `changed_attributes` will ultimately check if every mutable
attribute has changed in place. Since this gets called whenever an
attribute is assigned, it's extremely slow. Instead, we can avoid this
calculation until we actually need it.

Fixes #18029