Instead of checking if whole object changed we should check if the foreign key attribute changed with:

```ruby
record.attribute_changed?(reflection.foreign_key)
```