Repository: pandas-dev/pandas

Issue:
BUG: Unexpected behaviour when reading large text files with mixed datatypes
read_csv gives unexpected behaviour with large files if a column contains both strings and integers. eg

``` python

>>> df=DataFrame({'colA':range(500000-1)+['apple', 'pear']+range(500000-1)})
len(set(df.colA))
500001

>>> df.to_csv('testpandas2.txt')
>>> df2=read_csv('testpandas2.txt')
>>> len(set(df2.colA))
762143

 >>> pandas.__version__
'0.11.0'
```

It seems some of the integers are parsed as integers and others as strings.

``` python
>>> list(set(df2.colA))[-10:]
['282248', '282249', '282240', '282241', '282242', '15679', '282244', '282245', '282246', '282247']
>>> list(set(df2.colA))[:10]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
```

BUG: Unexpected behaviour when reading large text files with mixed datatypes
read_csv gives unexpected behaviour with large files if a column contains both strings and integers. eg

``` python

>>> df=DataFrame({'colA':range(500000-1)+['apple', 'pear']+range(500000-1)})
len(set(df.colA))
500001

>>> df.to_csv('testpandas2.txt')
>>> df2=read_csv('testpandas2.txt')
>>> len(set(df2.colA))
762143

 >>> pandas.__version__
'0.11.0'
```

It seems some of the integers are parsed as integers and others as strings.

``` python
>>> list(set(df2.colA))[-10:]
['282248', '282249', '282240', '282241', '282242', '15679', '282244', '282245', '282246', '282247']
>>> list(set(df2.colA))[:10]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
```



Discussion:
```
In [1]: df2=read_csv('testpandas2.txt',index_col=0)

In [2]: df2
Out[2]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000000 entries, 0 to 999999
Data columns (total 1 columns):
colA    1000000  non-null values
dtypes: object(1)

In [3]: from collections import Counter

In [4]: Counter(df2.colA.apply(lambda x: type(x)))
Out[4]: Counter({<type 'int'>: 737856, <type 'str'>: 262144})
```

So the way parsing works (when you don't specify a specifc dtyp) is that on a particular column you
loop over all dtypes, and try to convert to an actual type; if something breaks you go to the next dtype. The
data is left is modified in-place, so the rows before the strings are converted to integers; when it hits the strings the parsing stops and the column is marked object

so the end result is the correct dtype.

you essentially want _downcasting back to strings for object dtype_; easy enough, specify _object_ as the dtype for this column.

If you want this automatic I think we'd have to provide an option to do it, because that would be inefficient from a parsing speed as you have to copy the column for every dtype you try

can you explain why this actually matters?

I'm not sure I understand. Why aren't there 500K integers and 500K+2
strings if everything after the string is encountered is parsed as a string?

This matters because if you try and aggregate using the object type column
as a key, the results will be incorrect. You get twice as many keys as you
actually intended. Thus, even trying to find the number of unique keys in a
table, a fairly basic task, will not work as expected.

On Wed, Jun 12, 2013 at 6:14 PM, jreback notifications@github.com wrote:

> In [1]: df2=read_csv('testpandas2.txt',index_col=0)
> 
> In [2]: df2
> Out[2]:
> <class 'pandas.core.frame.DataFrame'>
> Int64Index: 1000000 entries, 0 to 999999
> Data columns (total 1 columns):
> colA    1000000  non-null values
> dtypes: object(1)
> 
> In [3]: from collections import Counter
> 
> In [4]: Counter(df2.colA.apply(lambda x: type(x)))
> Out[4]: Counter({<type 'int'>: 737856, <type 'str'>: 262144})
> 
> So the way parsing works (when you don't specify a specifc dtyp) is that
> on a particular column you
> loop over all dtypes, and try to convert to an actual type; if something
> breaks you go to the next dtype. The
> data is left is modified in-place, so the rows before the strings are
> converted to integers; when it hits the strings the parsing stops and the
> column is marked object
> 
> so the end result is the correct dtype.
> 
> you essentially want _downcasting back to strings for object dtype_; easy
> enough, specify _object_ as the dtype for this column.
> 
> If you want this automatic I think we'd have to provide an option to do
> it, because that would be inefficient from a parsing speed as you have to
> copy the column for every dtype you try
> 
> can you explain why this actually matters?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3866#issuecomment-19340528
> .

@wesm pls take a look

so the int conversion stops at 262144, which is exactly 2**16 \* 4...weird must be something odd going on

I can repro, but fix is eluding me :)

```
In [1]: df2=read_csv('testpandas2.txt',index_col=0)

In [2]: df2
Out[2]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000000 entries, 0 to 999999
Data columns (total 1 columns):
colA    1000000  non-null values
dtypes: object(1)

In [3]: from collections import Counter

In [4]: Counter(df2.colA.apply(lambda x: type(x)))
Out[4]: Counter({<type 'int'>: 737856, <type 'str'>: 262144})
```

So the way parsing works (when you don't specify a specifc dtyp) is that on a particular column you
loop over all dtypes, and try to convert to an actual type; if something breaks you go to the next dtype. The
data is left is modified in-place, so the rows before the strings are converted to integers; when it hits the strings the parsing stops and the column is marked object

so the end result is the correct dtype.

you essentially want _downcasting back to strings for object dtype_; easy enough, specify _object_ as the dtype for this column.

If you want this automatic I think we'd have to provide an option to do it, because that would be inefficient from a parsing speed as you have to copy the column for every dtype you try

can you explain why this actually matters?

I'm not sure I understand. Why aren't there 500K integers and 500K+2
strings if everything after the string is encountered is parsed as a string?

This matters because if you try and aggregate using the object type column
as a key, the results will be incorrect. You get twice as many keys as you
actually intended. Thus, even trying to find the number of unique keys in a
table, a fairly basic task, will not work as expected.

On Wed, Jun 12, 2013 at 6:14 PM, jreback notifications@github.com wrote:

> In [1]: df2=read_csv('testpandas2.txt',index_col=0)
> 
> In [2]: df2
> Out[2]:
> <class 'pandas.core.frame.DataFrame'>
> Int64Index: 1000000 entries, 0 to 999999
> Data columns (total 1 columns):
> colA    1000000  non-null values
> dtypes: object(1)
> 
> In [3]: from collections import Counter
> 
> In [4]: Counter(df2.colA.apply(lambda x: type(x)))
> Out[4]: Counter({<type 'int'>: 737856, <type 'str'>: 262144})
> 
> So the way parsing works (when you don't specify a specifc dtyp) is that
> on a particular column you
> loop over all dtypes, and try to convert to an actual type; if something
> breaks you go to the next dtype. The
> data is left is modified in-place, so the rows before the strings are
> converted to integers; when it hits the strings the parsing stops and the
> column is marked object
> 
> so the end result is the correct dtype.
> 
> you essentially want _downcasting back to strings for object dtype_; easy
> enough, specify _object_ as the dtype for this column.
> 
> If you want this automatic I think we'd have to provide an option to do
> it, because that would be inefficient from a parsing speed as you have to
> copy the column for every dtype you try
> 
> can you explain why this actually matters?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3866#issuecomment-19340528
> .

@wesm pls take a look

so the int conversion stops at 262144, which is exactly 2**16 \* 4...weird must be something odd going on

I can repro, but fix is eluding me :)


Resolving patch:
diff --git a/pandas/io/common.py b/pandas/io/common.py
--- a/pandas/io/common.py
+++ b/pandas/io/common.py
@@ -36,10 +36,15 @@ def urlopen(*args, **kwargs):
 _VALID_URLS = set(uses_relative + uses_netloc + uses_params)
 _VALID_URLS.discard('')
 
+
 class PerformanceWarning(Warning):
     pass
 
 
+class DtypeWarning(Warning):
+    pass
+
+
 def _is_url(url):
     """Check to see if a URL has a valid protocol.
 


Test patch:
diff --git a/pandas/io/tests/test_parsers.py b/pandas/io/tests/test_parsers.py
--- a/pandas/io/tests/test_parsers.py
+++ b/pandas/io/tests/test_parsers.py
@@ -11,6 +11,7 @@
 
 from numpy import nan
 import numpy as np
+from pandas.io.common import DtypeWarning
 
 from pandas import DataFrame, Series, Index, MultiIndex, DatetimeIndex
 from pandas.compat import(
@@ -1865,6 +1866,24 @@ def test_parse_integers_above_fp_precision(self):
 
         self.assertTrue(np.array_equal(result['Numbers'], expected['Numbers']))
 
+    def test_chunks_have_consistent_numerical_type(self):
+        integers = [str(i) for i in range(499999)]
+        data = "a\n" + "\n".join(integers + ["1.0", "2.0"] + integers)
+
+        with tm.assert_produces_warning(False):
+            df = self.read_csv(StringIO(data))
+        self.assertTrue(type(df.a[0]) is np.float64)  # Assert that types were coerced.
+        self.assertEqual(df.a.dtype, np.float)
+
+    def test_warn_if_chunks_have_mismatched_type(self):
+        # See test in TestCParserLowMemory.
+        integers = [str(i) for i in range(499999)]
+        data = "a\n" + "\n".join(integers + ['a', 'b'] + integers)
+
+        with tm.assert_produces_warning(False):
+            df = self.read_csv(StringIO(data))
+        self.assertEqual(df.a.dtype, np.object)
+
 
 class TestPythonParser(ParserTests, unittest.TestCase):
 
@@ -2301,7 +2320,6 @@ def test_usecols_dtypes(self):
         self.assertTrue((result.dtypes == [object, np.int, np.float]).all())
         self.assertTrue((result2.dtypes == [object, np.float]).all())
 
-
     def test_usecols_implicit_index_col(self):
         # #2654
         data = 'a,b,c\n4,apple,bat,5.7\n8,orange,cow,10'
@@ -2528,16 +2546,22 @@ def test_tokenize_CR_with_quoting(self):
 
     def test_raise_on_no_columns(self):
         # single newline
-        data = """
-"""
+        data = "\n"
         self.assertRaises(ValueError, self.read_csv, StringIO(data))
 
         # test with more than a single newline
-        data = """
+        data = "\n\n\n"
+        self.assertRaises(ValueError, self.read_csv, StringIO(data))
 
+    def test_warn_if_chunks_have_mismatched_type(self):
+        # Issue #3866 If chunks are different types and can't
+        # be coerced using numerical types, then issue warning.
+        integers = [str(i) for i in range(499999)]
+        data = "a\n" + "\n".join(integers + ['a', 'b'] + integers)
 
-"""
-        self.assertRaises(ValueError, self.read_csv, StringIO(data))
+        with tm.assert_produces_warning(DtypeWarning):
+            df = self.read_csv(StringIO(data))
+        self.assertEqual(df.a.dtype, np.object)
 
 
 class TestParseSQL(unittest.TestCase):