Only running after_remove callback for removed records.

If a record that does not belong to a collection is removed from an association, after_remove is still called.