I think you could clean up a bunch of the code, rather than storing a boolean for `use_locations`, you stored the reference type base class (as a class), and then used `issubclass` elsewhere in the code. Then you could basically just ask whether the current type is a subclass of the reference type, and if so, skip any conversion, otherwise, pick the appropriate conversion.

However, I don't think that needs to happen before this PR gets merged.