Repository: numpy/numpy

Issue:
`numpy.dtype()` issue warning when run python3 with `-b` switch
See [here](https://github.com/pandas-dev/pandas/issues/16776)

1. Run Python using
`python3 -b`
2. Run 
```
import numpy as np
np.dtype('int32')
```
I get output
```
>>> import numpy as np
>>> np.dtype('int32')
__main__:1: BytesWarning: Comparison between bytes and string
dtype('int32')

>>> print(sys.version)
3.5.3 |Continuum Analytics, Inc.| (default, Mar  6 2017, 12:15:08) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]

>>> np.version.version
'1.13.0'
```


Discussion:
Hmm. All numpy is doing is accessing a dictionary with a 'bytes' key. The offending line  essentially does`np.sctypeDict[b'int32']`.

You can get the same error in plain python3 by accessing a dict:
```python
>>> d = {'int32': 1}
>>> d[b'int32']
__main__:1: BytesWarning: Comparison between bytes and string
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: b'int32'
```

This doesn't seem like a numpy problem to me. If anything it may be a python3 problem, but it also seems possible to me it is the "expected" behavior of the `-b` flag, I'm not sure.
I suspect the problem is the opposite - namely `d = {b'int32': 1}; d['int32']`
You would think :)  My reading is that in python3 the string is first converted to a bytes object [here](https://github.com/numpy/numpy/blob/bf12213a1835d42acbfc0bd5f5587d30372fc50e/numpy/core/src/multiarray/descriptor.c#L1440), then the dict lookup happens [here](https://github.com/numpy/numpy/blob/bf12213a1835d42acbfc0bd5f5587d30372fc50e/numpy/core/src/multiarray/descriptor.c#L1612), then it is converted back to unicode [here](https://github.com/numpy/numpy/blob/bf12213a1835d42acbfc0bd5f5587d30372fc50e/numpy/core/src/multiarray/descriptor.c#L1616).
I'm able to reproduce the `BytesWarning` simply by importing `numpy`. I bisected. For me this first started in commit 2c14e47834262de85673c0b71e94494a5f642268.
I have isolated the issue to the following code:

https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/descriptor.c#L1667-L1677

The bytes object `b'int64'` is passed. The code first indexes the dict by the bytes object and and then by the str object. But all the keys in the dict are str. I tried changing this to:

```c
            if (PyBytes_Check(obj)) {
                PyObject *tmp;
                tmp = PyUnicode_FromEncodedObject(obj, "ascii", "strict");
                if (tmp != NULL) {
                    item = PyDict_GetItem(typeDict, tmp);
                    Py_DECREF(tmp);
                }
            }
            else {
                item = PyDict_GetItem(typeDict, obj);
            }
```

But after that change, the tests segfault. Not quite sure where to go from there.
Sounds like the whole thing could use some cleanup with the knowledge that we are python3 maybe. @jdufresne, I think the problem with your patch is, that it is a bit obscure that `tmp == NULL` results in a `goto fail` immediately. I am not actually immediately sure why that shows up after your change and not before, but I think that explainst he segfaults you are seeing.
I'd note again that this seems possibly not a bug. The `-b` option is meant to help people port from py2 to py3 by pointing out operations which changed behavior. But in this case, we have already accounted for the py2 vs py3 behavior since that code is inside a special py3 block, so there is no problem.

Perhaps the correct thing to do is to suppress the BytesWarning. 

Then again, perhaps there is a way to reorganize the code to avoid the warning, by testing for bytes *before* doing the key lookup. Totally untested idea:
```python
        if (typeDict != NULL) {        
#if defined(NPY_PY3K)
            if (PyBytes_Check(obj)) {
                PyObject *tmp;
                tmp = PyUnicode_FromEncodedObject(obj, "ascii", "strict");
                if (tmp != NULL) {
                    item = PyDict_GetItem(typeDict, tmp);
                    Py_DECREF(tmp);
                }
            }
            else {
                item = PyDict_GetItem(typeDict, obj);
            }
#else
           item = PyDict_GetItem(typeDict, obj); 
#endif
Oh, I see that's exactly what you tried...

I think in that case the segfault is because item is uninitialized. Set it to NULL at the top of the function, maybe.
Well, I guess it is not a bug, but I am happy with putting in a patch as an enhancement, since it makes the task more difficult for users trying to move to python3 and using numpy.
Thanks for the tip @ahaldane. You were right about initializing `item`.
Can we instead just remove the current `str -> bytes` conversion that @ahaldane links to within numpy on python3? I think `dtype(b'int64')` _should_ emit this warning.
I agree that would be a better solution. I'll take a look at implementing this.

Resolving patch:
diff --git a/numpy/core/arrayprint.py b/numpy/core/arrayprint.py
--- a/numpy/core/arrayprint.py
+++ b/numpy/core/arrayprint.py
@@ -1470,7 +1470,11 @@ def array_repr(arr, max_line_width=None, precision=None, suppress_small=None):
         arr, max_line_width, precision, suppress_small)
 
 
-_guarded_str = _recursive_guard()(str)
+@_recursive_guard()
+def _guarded_repr_or_str(v):
+    if isinstance(v, bytes):
+        return repr(v)
+    return str(v)
 
 
 def _array_str_implementation(
@@ -1488,7 +1492,7 @@ def _array_str_implementation(
         # obtain a scalar and call str on it, avoiding problems for subclasses
         # for which indexing with () returns a 0d instead of a scalar by using
         # ndarray's getindex. Also guard against recursive 0d object arrays.
-        return _guarded_str(np.ndarray.__getitem__(a, ()))
+        return _guarded_repr_or_str(np.ndarray.__getitem__(a, ()))
 
     return array2string(a, max_line_width, precision, suppress_small, ' ', "")
 
diff --git a/numpy/f2py/__init__.py b/numpy/f2py/__init__.py
--- a/numpy/f2py/__init__.py
+++ b/numpy/f2py/__init__.py
@@ -109,6 +109,7 @@ def compile(source,
             output = ''
         else:
             status = 0
+            output = output.decode()
         if verbose:
             print(output)
     finally:


Test patch:
diff --git a/numpy/lib/tests/test_io.py b/numpy/lib/tests/test_io.py
--- a/numpy/lib/tests/test_io.py
+++ b/numpy/lib/tests/test_io.py
@@ -1871,7 +1871,7 @@ def test_inconsistent_dtype(self):
         data = ["1, 1, 1, 1, -1.1"] * 50
         mdata = TextIO("\n".join(data))
 
-        converters = {4: lambda x: "(%s)" % x}
+        converters = {4: lambda x: "(%s)" % x.decode()}
         kwargs = dict(delimiter=",", converters=converters,
                       dtype=[(_, int) for _ in 'abcde'],)
         assert_raises(ValueError, np.genfromtxt, mdata, **kwargs)