ENH: default on size when we have things that cannot be nan (e.g., ints)