Repository: pandas-dev/pandas

Issue:
bad arguments to dataframe constructor causes segfault
#### Code Sample, a copy-pastable example if possible

```python
pandas.DataFrame(data=(range(10), range(10, 20)), columns=('ones', 'twos'))
```

```
# python
Python 2.7.14 (default, Dec 12 2017, 16:55:09)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> pandas.DataFrame(data=(range(10), range(10, 20)), columns=('ones', 'twos'))
Segmentation fault
```
#### Problem description

Segmentation fault on bad inputs to DataFrame constructor.

#### Expected Output

Raise an exception:

```AssertionError: 2 columns passed, passed data had 10 columns```

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.14.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.125-linuxkit
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: None.None

pandas: 0.24.2
pytest: 3.8.1
pip: 9.0.1
setuptools: 38.2.4
Cython: None
numpy: 1.16.3
scipy: None
pyarrow: None
xarray: None
IPython: 5.8.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: 2.2.6
xlrd: None
xlwt: 1.3.0
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: 1.0.1
sqlalchemy: 1.1.11
pymysql: None
psycopg2: 2.7.3.2 (dt dec pq3 ext lo64)
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

</details>



Discussion:
Can you try on master? I think this was fixed recently.

On Thu, May 16, 2019 at 1:43 PM Kurt Rose <notifications@github.com> wrote:

> Code Sample, a copy-pastable example if possible
>
> pandas.DataFrame(data=(range(10), range(10, 20)), columns=('ones', 'twos'))
>
> # python
> Python 2.7.14 (default, Dec 12 2017, 16:55:09)
> [GCC 4.9.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import pandas
> >>> pandas.DataFrame(data=(range(10), range(10, 20)), columns=('ones', 'twos'))
> Segmentation fault
>
> Problem description
>
> Segmentation fault on bad inputs to DataFrame constructor.
> Expected Output
>
> Raise an exception:
>
> AssertionError: 2 columns passed, passed data had 10 columns
> Output of pd.show_versions()
>
> [paste the output of pd.show_versions() here below this line]
> INSTALLED VERSIONS
>
> commit: None
> python: 2.7.14.final.0
> python-bits: 64
> OS: Linux
> OS-release: 4.9.125-linuxkit
> machine: x86_64
> processor:
> byteorder: little
> LC_ALL: None
> LANG: C.UTF-8
> LOCALE: None.None
>
> pandas: 0.24.2
> pytest: 3.8.1
> pip: 9.0.1
> setuptools: 38.2.4
> Cython: None
> numpy: 1.16.3
> scipy: None
> pyarrow: None
> xarray: None
> IPython: 5.8.0
> sphinx: None
> patsy: None
> dateutil: 2.6.0
> pytz: 2017.2
> blosc: None
> bottleneck: None
> tables: None
> numexpr: None
> feather: None
> matplotlib: None
> openpyxl: 2.2.6
> xlrd: None
> xlwt: 1.3.0
> xlsxwriter: None
> lxml.etree: None
> bs4: None
> html5lib: 1.0.1
> sqlalchemy: 1.1.11
> pymysql: None
> psycopg2: 2.7.3.2 (dt dec pq3 ext lo64)
> jinja2: 2.9.6
> s3fs: None
> fastparquet: None
> pandas_gbq: None
> pandas_datareader: None
> gcsfs: None
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/26429?email_source=notifications&email_token=AAKAOIS26CLHS42RWSEXMULPVWTMJA5CNFSM4HNPGCM2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GUHYDFA>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOITUHQ4EK2TUUQMNWETPVWTMJANCNFSM4HNPGCMQ>
> .
>

oh, great!  this isn't really a blocker for me; is there a doc or anything on how to install from source?  (does pip install)
The contributing docs have all that info. Pip may work, but you need a compiler. 

> On May 16, 2019, at 17:50, Kurt Rose <notifications@github.com> wrote:
> 
> oh, great! this isn't really a blocker for me; is there a doc or anything on how to install from source? (does pip install)
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or mute the thread.

> I think this was fixed recently.

xref #25691

> Can you try on master?

```python-traceback
Python 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> pandas.__version__
'0.25.0.dev0+567.g1263e1a9c'
>>> pandas.DataFrame(data=(range(10), range(10, 20)), columns=('ones', 'twos'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\simon\OneDrive\code\pandas-simonjayhawkins\pandas\core\frame.py", line 434, in __init__
    arrays, columns = to_arrays(data, columns, dtype=dtype)
  File "C:\Users\simon\OneDrive\code\pandas-simonjayhawkins\pandas\core\internals\construction.py", line 418, in to_arrays
    dtype=dtype)
  File "C:\Users\simon\OneDrive\code\pandas-simonjayhawkins\pandas\core\internals\construction.py", line 428, in _list_to_arrays
    coerce_float=coerce_float)
  File "C:\Users\simon\OneDrive\code\pandas-simonjayhawkins\pandas\core\internals\construction.py", line 484, in _convert_object_arr
ay
    con=len(content)))
AssertionError: 2 columns passed, passed data had 10 columns
>>>
```

doesn't segfault, but should `AssertionError` be raised?


hmm we do check construction errors; iirc they should be ValueError

so yes this should change

Resolving patch:
diff --git a/pandas/core/internals/construction.py b/pandas/core/internals/construction.py
--- a/pandas/core/internals/construction.py
+++ b/pandas/core/internals/construction.py
@@ -424,8 +424,13 @@ def _list_to_arrays(data, columns, coerce_float=False, dtype=None):
     else:
         # list of lists
         content = list(lib.to_object_array(data).T)
-    return _convert_object_array(content, columns, dtype=dtype,
-                                 coerce_float=coerce_float)
+    # gh-26429 do not raise user-facing AssertionError
+    try:
+        result = _convert_object_array(content, columns, dtype=dtype,
+                                       coerce_float=coerce_float)
+    except AssertionError as e:
+        raise ValueError(e) from e
+    return result
 
 
 def _list_of_series_to_arrays(data, columns, coerce_float=False, dtype=None):


Test patch:
diff --git a/pandas/tests/frame/test_constructors.py b/pandas/tests/frame/test_constructors.py
--- a/pandas/tests/frame/test_constructors.py
+++ b/pandas/tests/frame/test_constructors.py
@@ -435,6 +435,11 @@ def test_constructor_error_msgs(self):
         with pytest.raises(ValueError, match=msg):
             DataFrame(np.random.rand(2, 3), columns=['A', 'B'], index=[1, 2])
 
+        # gh-26429
+        msg = "2 columns passed, passed data had 10 columns"
+        with pytest.raises(ValueError, match=msg):
+            DataFrame((range(10), range(10, 20)), columns=('ones', 'twos'))
+
         msg = ("If using all scalar "
                "values, you must pass "
                "an index")
diff --git a/pandas/tests/io/json/test_pandas.py b/pandas/tests/io/json/test_pandas.py
--- a/pandas/tests/io/json/test_pandas.py
+++ b/pandas/tests/io/json/test_pandas.py
@@ -353,7 +353,7 @@ def test_frame_from_json_bad_data(self):
                         '"index":["1","2","3"],'
                         '"data":[[1.0,"1"],[2.0,"2"],[null,"3"]]}')
         msg = "3 columns passed, passed data had 2 columns"
-        with pytest.raises(AssertionError, match=msg):
+        with pytest.raises(ValueError, match=msg):
             read_json(json, orient="split")
 
         # bad key