TST: more test fixes for dtype checking