Refactor to eliminate access to xref structure in goroutine because it wasn't safe.
Also clean up handling of pre-existing Indirect objects.