(Fix)  #13681 Crossreferences trigger warnings too now.

There is only a single line of code that actually changed.
We do not add all sources of deleted references to the objects to
be deleted. References also get deleted if only the target is to
be removed.