astropy/astropy — pull request #919 by taldcroft (2013-05-31 19:36:33+00:00)
Title: Interpret blank field values as missing in io.ascii.read

Based on discussion in #900 and email to astropy-dev ("Treatment of blank fields in io.ascii.read") it was decided that by default blank fields in ASCII tables should be treated as missing instead of as literal "" (empty string) values. This matches the behavior of other table readers such as pandas and numpy.genfromtxt.

The main design question is what to do for string-valued columns. Both pandas and genfromtxt treat a blank field in a string-valued column as missing. On one hand that is a consistent behavior for all data types, and it would make implementation trivial. In this case most of the change could be done by having `fill_values=('', '0')` as a default in `io.ascii.read`. Then users could disable this default behavior entirely by setting `fill_values=False`.

The downside is that it represents backward-incompatible behavior for string columns. For numeric columns there is no real problem because they would have required explicit `fill_values` to parse.

Keeping empty strings as valid values will likely require the addition of a new keyword arg to the `read()` function. Also (from initial thoughts) the implementation in this case is probably a lot harder. This is because a blank field is converted to a 0, then an attempt is made to convert to a numeric column. Then if that fails all the blank fields have to be put back to a "" before converting to a string column. Some memory of where the blank fields will be needed, which is new.