Circumvent pandas dtype inference when indexing coordinates