Repository: numpy/numpy

Issue:
New in NumPy 1.14: failures in PyTables 
In the following snippet:

```python
import numpy as np
data_proper = [(456, b'dbe', 1.2), (2, b'ded', 1.3)]
data_improper = [[456, b'dbe', 1.2], [2, b'ded', 1.3]]
descr = [('f0', '<i4'), ('f1', 'S3'), ('f2', '<f8')]
dtype = np.dtype((np.record, descr))

np.array(data_improper, dtype=dtype)
```

Execution of the last line in 1.13 used to raise:
```
TypeError: a bytes-like object is required, not 'int'
```
which allowed the logic in [numpy/core/records.py#L676](https://github.com/numpy/numpy/blob/master/numpy/core/records.py#L676) to recover and reinterpret the line as `np.array(data_proper, dtype=dtype)`.

The trouble is that in NumPy 1.14
```
ValueError: invalid literal for int() with base 10: b'dbe'
```
is being raised instead, which results in the failure. 

This is a gist of the issue behind numerous test failures in pyTables when used with NumPy 1.14.
  


Discussion:
The error comes from cPython, but where it actually originates is hard to tell, the traceback has been truncated too far. I think we will need a bisect on this.
Perhaps the easy fix is to just to catch `ValueError` in `rec.array` as well?
@eric-wieser I'd worry that that might not be the only place affected. I think we need to know just what changed.
I think the simplest reproduction is:
```python
>>> np.array(456, [('i', int)])
TypeError: a bytes-like object is required, not 'int'  # 1.13
array((456,), dtype=[('i', '<i4')])  # 1.14
```
However, the new behavior seems kinda consistent, when compared to:
```python
>>> np.array(456).astype([('i', int)])
array((456,), dtype=[('i', '<i4')])  # 1.13 and 1.14
```
and
```python
>>> np.array(np.array(456), [('i', int)])
array((456,), dtype=[('i', '<i4')])  # 1.13 and 1.14
```
  
For reference, the 1.13.3 error is `TypeError: expected a readable buffer object`.
I do think the catch in `records.py` is too narrow, too dependent on underlying implementation details, so that needs a fix.
> so that needs a fix.

Frankly, I think we should deprecate that code path entirely. If you pass malformed data, expecting numpy to guess what you meant seems undesirable
The change came in with #6053. I don't think this is so much a bug, as a change in behavior.
I think we need a bigger discussion on the way forward, so pinging @ahaldane . IIRC, h5py was also affected by this change.
I'm reading along. Following on what @eric-wieser said, the change in behavior was to allow scalars to broadcast to all fields during array creation, not just during assignment. The difference in behavior of these two lines seemed inconsistent in 1.13:
```python
>>> a = np.zeros(3, dtype='i4,i4') # no scalar assignment problem
>>> a[:] = 456  # no scalar assignment problem
>>> np.array([456], dtype='i4,i4')
TypeError: expected a readable buffer object  # can't initialize to scalar in 1.13
```

Furthermore, in 1.13:
```python
>>> np.array([456, 'dbe'], dtype='i4,S3')
TypeError: expected a readable buffer object
>>> np.rec.array([456, 'dbe'], dtype='i4,S3')
rec.array((456, 'dbe'), 
          dtype=[('f0', '<i4'), ('f1', 'S3')])
```
Why should the first line fail, but the second succeed?

So my feeling is that the special casing in `records.py` isn't ideal, but maybe there is away to keep things back-compatible, I'm not sure yet.

By the way, I will be at a conference later this week, so may be away from my computer.
  
The problem seems isolated to the `record.py` module, so for 1.14 adding `ValueError` to the list of caught errors is probably an adequate fix. @eric-wieser's point about trying to fix up malformed input is the long term problem, and we probably need more time than 1.14 allows to deal with that.
@charris Thank you. The improper record array object input to `numpy.record.array` was all coming from hard-coded list of lists in pyTable's test files, so it is easy, albeit tedious, to modify. I am prepping a PR for pyTable now.
  
@charris I have opened two PRs for PyTables repo, one addressing errors caused by the current issues, and [the other](https://github.com/PyTables/PyTables/pull/663) dealing with deprecation warnings.
>  so for 1.14 adding ValueError to the list of caught errors is probably an adequate fix

Agreed, although we might want to see if we can squeeze any other new errors out of the conversion too. Maybe even a catchall `Exception` would be worthwhile.

> and we probably need more time than 1.14 allows to deal with that.

Although if all our CI-testing users are already fixing this, this might be an easy time to slip in a deprecation warning too.



Resolving patch:
diff --git a/numpy/core/records.py b/numpy/core/records.py
--- a/numpy/core/records.py
+++ b/numpy/core/records.py
@@ -38,6 +38,7 @@
 
 import sys
 import os
+import warnings
 
 from . import numeric as sb
 from . import numerictypes as nt
@@ -677,7 +678,7 @@ def fromrecords(recList, dtype=None, shape=None, formats=None, names=None,
 
     try:
         retval = sb.array(recList, dtype=descr)
-    except TypeError:  # list of lists instead of list of tuples
+    except (TypeError, ValueError):
         if (shape is None or shape == 0):
             shape = len(recList)
         if isinstance(shape, (int, long)):
@@ -687,6 +688,12 @@ def fromrecords(recList, dtype=None, shape=None, formats=None, names=None,
         _array = recarray(shape, descr)
         for k in range(_array.size):
             _array[k] = tuple(recList[k])
+        # list of lists instead of list of tuples ?
+        # 2018-02-07, 1.14.1
+        warnings.warn(
+            "fromrecords expected a list of tuples, may have received a list "
+            "of lists instead. In the future that will raise an error",
+            FutureWarning, stacklevel=2)
         return _array
     else:
         if shape is not None and retval.shape != shape:


Test patch: