Repository: numpy/numpy

Issue:
recfromtext bytes delimiter
I had existing code that uses BytesIO and a byte delimiter. It worked in <=1.13 and breaks after.  The code works if the delimiter is switched to a string. All tests were performed with Python 3.6.5 from Anaconda. The failure can be reproduced using the excerpt below.

Fails using 1.14.0, 1.14.1, and 1.14.2
```
python -c "import numpy as np; import io; print(np.version.version); data = np.recfromtxt(io.BytesIO(b'SPC:MW\nNO:30\nNO2:36'), delimiter=b':', dtype = ['S3', 'f4']); print(data)"
1.14.2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/work/ROMO/anaconda3/envs/testnp113/lib/python3.6/site-packages/numpy/lib/npyio.py", line 2230, in recfromtxt
    output = genfromtxt(fname, **kwargs)
  File "/work/ROMO/anaconda3/envs/testnp113/lib/python3.6/site-packages/numpy/lib/npyio.py", line 2116, in genfromtxt
    rows = np.array(data, dtype=[('', _) for _ in dtype_flat])
ValueError: could not assign tuple of length 1 to structure with 2 fields.
```

Passes with 1.13.3
```
python -c "import numpy as np; import io; print(np.version.version); data = np.recfromtxt(io.BytesIO(b'SPC:MW\nNO:30\nNO2:36'), delimiter=b':', dtype = ['S3', 'f4']); print(data)"
1.13.3
[(b'SPC',  nan) (b'NO',  30.) (b'NO2',  36.)]
```



Discussion:
Quite likely caused by #10054. 

In particular, it seems like the delimiter is not being processed properly: In 1.13 when `split` is called on `SPC:MW\n` it results in `('SPC', 'MW')`, but in 1.14 it is `(SPC:MW\n,'')`. I didn't see what the exact problem is, but I think it has something to do withthe `isinstance(delimiter, basestring)` check in `LineSplitter.__init__`.
@juliantaylor Thoughts?
I'm taking another look. It's definiely a LineSplitter bytes issue.

```python
>>> np.__version__
'1.15.0.dev0+6721890'
>>> from numpy.lib._iotools import LineSplitter
>>> LineSplitter(delimiter=':')(b'SPC:MW')
['SPC', 'MW']
>>> LineSplitter(delimiter=b':')(b'SPC:MW')
['SPC:MW']
```

So we used to support bytes objects as the delimiter in 1.13, but now we don't. It happens because `isinstance(b":", basestring)` is `False`.

I need to refresh myself on what `numpy.compat.basestring` is supposed to do. Right now is is just `str` (not `(str, bytes)`).

It looks like a lot of code in `npyio` now assumes strings instead of bytes, eg all the string literals used there as arguments to `strip` and `split`, after #10054. 

Maybe it is our intention to no longer support bytes?
OK, sorry for the noise, I just read #10054 properly. 

I think the fix is quite simple: The delimiter needs to be encoded as latin1 if it is bytes. There is even a new docstring that says this happens, but it seems it wasn't implemented. I'll give a shot at a fix.

Resolving patch:
diff --git a/numpy/lib/_iotools.py b/numpy/lib/_iotools.py
--- a/numpy/lib/_iotools.py
+++ b/numpy/lib/_iotools.py
@@ -205,7 +205,11 @@ def autostrip(self, method):
     #
 
     def __init__(self, delimiter=None, comments='#', autostrip=True, encoding=None):
+        delimiter = _decode_line(delimiter)
+        comments = _decode_line(comments)
+
         self.comments = comments
+
         # Delimiter is a character
         if (delimiter is None) or isinstance(delimiter, basestring):
             delimiter = delimiter or None


Test patch:
diff --git a/numpy/lib/tests/test__iotools.py b/numpy/lib/tests/test__iotools.py
--- a/numpy/lib/tests/test__iotools.py
+++ b/numpy/lib/tests/test__iotools.py
@@ -53,6 +53,11 @@ def test_other_delimiter(self):
         test = LineSplitter(',')(strg)
         assert_equal(test, ['1', '2', '3', '4', '', '5'])
 
+        # gh-11028 bytes comment/delimiters should get decoded
+        strg = b" 1,2,3,4,,5 % test"
+        test = LineSplitter(delimiter=b',', comments=b'%')(strg)
+        assert_equal(test, ['1', '2', '3', '4', '', '5'])
+
     def test_constant_fixed_width(self):
         "Test LineSplitter w/ fixed-width fields"
         strg = "  1  2  3  4     5   # test"