Move all references into the ReferenceFactory