Repository: dask/dask

Issue:
read_csv falsely detect NA in integer column
It seems that NA's are mistakenly detected in my second integer column, thus whole column is read as float64.
When I load data I get my integer column converted to float64, specifying dtypes will raise exception on NAs in integer. The problem is there are no NA in data.

``` py
x = dd.read_csv("hdfs://path/X1e7_2c.csv")
x.dtypes
#KEY      int64
#X2     float64
#dtype: object
x[x.KEY.isnull()].compute()
#Empty DataFrame
#Columns: [KEY, X2]
#Index: []
x[x.X2.isnull()].compute()
#Empty DataFrame
#Columns: [KEY, X2]
#Index: []
```

I manage to workaround this using the following, let me know if it can be improved/simplified.

``` py
x = dd.read_csv("hdfs://path/X1e7_2c.csv", na_filter=False)
x.dtypes
#KEY     int64
#X2     object
#dtype: object
x[x.KEY.isnull()].compute()
#Empty DataFrame
#Columns: [KEY, X2]
#Index: []
x[x.X2.isnull()].compute()
#Empty DataFrame
#Columns: [KEY, X2]
#Index: []
x[['X2']] = x[['X2']].astype(int)
x.dtypes
#KEY    int64
#X2     int64
#dtype: object
```



Discussion:
Does the CSV file fit in pandas?  If so what does it say are the dtypes?

``` python
import hdfs3
hdfs = hdfs3.HDFileSystem()

with hdfs.open("/path/X1e7_2c.csv") as f:
    df = pd.read_hdf(f)

print(df.dtypes)
```

I'm getting the following error on `pd.read_hdf`.

``` py
#.../pandas/io/pytables.py", line 320, in read_hdf
#    raise NotImplementedError('Support for generic buffers has not been '
#NotImplementedError: Support for generic buffers has not been implemented.
```

`hdfs.ls` lists files well.

I have the same csv locally, and which pandas reads properly.

``` py
xx = pd.read_csv("X1e7_2c.csv")
xx.dtypes
#KEY    int64
#X2     int64
#dtype: object
```

Ah, sorry, I was dumb and listed read_hdf rather than read_csv

My main question was really "How does Pandas handle this data?  Does it make the same mistake?"

Looks good with `pd.read_csv` from hadoop

``` py
#df.dtypes               
#KEY    int64
#X2     int64
#dtype: object
```

I suspect this is the same issue that was fixed in https://github.com/dask/dask/pull/1495. Can you try installing from git and see if that fixes things?

`pip install -U git+https://github.com/dask/dask.git`

@jcorbin just tried and got the same behaviour (float64), but not sure if update was made correctly (used miniconda/bin/pip because using miniconda path for python). Should I also upgrade it on nodes? Is there any way to get git commit hash of currently running dask?

The fix should only matter locally (on the head node), but generally it's a good idea to have the same versions of things running everywhere. Note that the dask code that runs read_csv distributed is the same that runs when on a single machine, so you can test on your machine using a local csv as well.

If you're sure you ran it on master and got the same issue, then I'm not sure what's going on. I'd need a reproducible example to help debug. 

I have another 1e7 dataset from same random number generator and this one loads well as int64, so this heavily depends on the data, will provide dataset soon, via email.

If you can please try to keep things public.

@mrocklin sure [mega.nz X1e7_2c.tar.gz](https://mega.nz/#!JpgXHCTA!fIntjGl-XHcea-xc_6P8idt_MkylnOEczTxSs7fRSdM)


Resolving patch:
diff --git a/dask/bytes/local.py b/dask/bytes/local.py
--- a/dask/bytes/local.py
+++ b/dask/bytes/local.py
@@ -31,7 +31,7 @@ def read_bytes(path, delimiter=None, not_zero=False, blocksize=2**27,
     if '*' in path:
         filenames = list(map(os.path.abspath, sorted(glob(path))))
         sample, first = read_bytes(filenames[0], delimiter, not_zero,
-                                   blocksize, sample=True,
+                                   blocksize, sample=sample,
                                    compression=compression)
         rest = [read_bytes(f, delimiter, not_zero, blocksize, sample=False,
                            compression=compression)[1]
@@ -65,7 +65,7 @@ def read_bytes(path, delimiter=None, not_zero=False, blocksize=2**27,
                 nbytes = sample
             else:
                 nbytes = 10000
-            sample = read_block_from_file(path, 0, nbytes, None, compression)
+            sample = read_block_from_file(path, 0, nbytes, delimiter, compression)
 
         return sample, values
 
diff --git a/dask/bytes/s3.py b/dask/bytes/s3.py
--- a/dask/bytes/s3.py
+++ b/dask/bytes/s3.py
@@ -80,7 +80,7 @@ def read_bytes(path, s3=None, delimiter=None, not_zero=False, blocksize=2**27,
         if not filenames:
             raise IOError("No such files: '%s'" % s3_path)
         sample, first = read_bytes(filenames[0], s3, delimiter, not_zero,
-                                   blocksize, sample=True,
+                                   blocksize, sample=sample,
                                    compression=compression)
         rest = [read_bytes(f, s3, delimiter, not_zero, blocksize,
                        sample=False, compression=compression)[1]
diff --git a/dask/dataframe/csv.py b/dask/dataframe/csv.py
--- a/dask/dataframe/csv.py
+++ b/dask/dataframe/csv.py
@@ -232,12 +232,7 @@ def read_csv(urlpath, blocksize=AUTO_BLOCKSIZE, chunkbytes=None,
     else:
         header = sample.split(b_lineterminator)[0] + b_lineterminator
 
-    try:
-        head = pd.read_csv(BytesIO(sample), **kwargs)
-    except:
-        # Remove the last row from the sample due to buffer row split:
-        sample = sample[:(sample.rfind(b_lineterminator))]
-        head = pd.read_csv(BytesIO(sample), **kwargs)
+    head = pd.read_csv(BytesIO(sample), **kwargs)
 
     df = read_csv_from_bytes(values, header, head, kwargs,
                              collection=collection, enforce=enforce)


Test patch:
diff --git a/dask/bytes/tests/test_local.py b/dask/bytes/tests/test_local.py
--- a/dask/bytes/tests/test_local.py
+++ b/dask/bytes/tests/test_local.py
@@ -30,6 +30,7 @@ def test_read_bytes():
         sample, values = read_bytes('.test.accounts.*')
         assert isinstance(sample, bytes)
         assert sample[:5] == files[sorted(files)[0]][:5]
+        assert sample.endswith(b'\n')
 
         assert isinstance(values, (list, tuple))
         assert isinstance(values[0], (list, tuple))
@@ -40,6 +41,19 @@ def test_read_bytes():
         assert set(results) == set(files.values())
 
 
+def test_read_bytes_sample_delimiter():
+    with filetexts(files, mode='b'):
+        sample, values = read_bytes('.test.accounts.*',
+                                    sample=80, delimiter=b'\n')
+        assert sample.endswith(b'\n')
+        sample, values = read_bytes('.test.accounts.1.json',
+                                    sample=80, delimiter=b'\n')
+        assert sample.endswith(b'\n')
+        sample, values = read_bytes('.test.accounts.1.json',
+                                    sample=2, delimiter=b'\n')
+        assert sample.endswith(b'\n')
+
+
 def test_read_bytes_blocksize_none():
     with filetexts(files, mode='b'):
         sample, values = read_bytes('.test.accounts.*', blocksize=None)
@@ -103,6 +117,7 @@ def test_compression(fmt, blocksize):
         sample, values = read_bytes('.test.accounts.*.json',
                 blocksize=blocksize, delimiter=b'\n', compression=fmt)
         assert sample[:5] == files[sorted(files)[0]][:5]
+        assert sample.endswith(b'\n')
 
         results = compute(*concat(values))
         assert (b''.join(results) ==
diff --git a/dask/bytes/tests/test_s3.py b/dask/bytes/tests/test_s3.py
--- a/dask/bytes/tests/test_s3.py
+++ b/dask/bytes/tests/test_s3.py
@@ -90,6 +90,7 @@ def test_read_bytes(s3):
     sample, values = read_bytes(test_bucket_name+'/test/accounts.*', s3=s3)
     assert isinstance(sample, bytes)
     assert sample[:5] == files[sorted(files)[0]][:5]
+    assert sample.endswith(b'\n')
 
     assert isinstance(values, (list, tuple))
     assert isinstance(values[0], (list, tuple))
@@ -99,10 +100,24 @@ def test_read_bytes(s3):
     results = compute(*concat(values))
     assert set(results) == set(files.values())
 
+
+def test_read_bytes_sample_delimiter(s3):
+    sample, values = read_bytes(test_bucket_name+'/test/accounts.*', s3=s3,
+                                sample=80, delimiter=b'\n')
+    assert sample.endswith(b'\n')
+    sample, values = read_bytes(test_bucket_name+'/test/accounts.1.json', s3=s3,
+                                sample=80, delimiter=b'\n')
+    assert sample.endswith(b'\n')
+    sample, values = read_bytes(test_bucket_name+'/test/accounts.1.json', s3=s3,
+                                sample=2, delimiter=b'\n')
+    assert sample.endswith(b'\n')
+
+
 def test_read_bytes_non_existing_glob(s3):
     with pytest.raises(IOError):
         read_bytes(test_bucket_name+'/non-existing/*', s3=s3)
 
+
 def test_read_bytes_blocksize_none(s3):
     _, values = read_bytes(test_bucket_name+'/test/accounts.*', blocksize=None,
                            s3=s3)
@@ -204,6 +219,7 @@ def test_compression(s3, fmt, blocksize):
         sample, values = read_bytes('compress/test/accounts.*', s3=s3,
                                     compression=fmt, blocksize=blocksize)
         assert sample.startswith(files[sorted(files)[0]][:10])
+        assert sample.endswith(b'\n')
 
         results = compute(*concat(values))
         assert b''.join(results) == b''.join([files[k] for k in sorted(files)])
diff --git a/dask/dataframe/tests/test_csv.py b/dask/dataframe/tests/test_csv.py
--- a/dask/dataframe/tests/test_csv.py
+++ b/dask/dataframe/tests/test_csv.py
@@ -248,6 +248,19 @@ def test_auto_blocksize_csv(monkeypatch):
 
 
 def test_head_partial_line_fix():
-    with filetexts({'.overflow.csv': 'a,b\n0,"abcdefghijklmnopqrstuvwxyz"\n1,"abcdefghijklmnopqrstuvwxyz"'}):
+    files = {'.overflow1.csv': ('a,b\n'
+                                '0,"abcdefghijklmnopqrstuvwxyz"\n'
+                                '1,"abcdefghijklmnopqrstuvwxyz"'),
+             '.overflow2.csv': ('a,b\n'
+                                '111111,-11111\n'
+                                '222222,-22222\n'
+                                '333333,-33333\n')}
+    with filetexts(files):
         # 64 byte file, 52 characters is mid-quote; this should not cause exception in head-handling code.
-        read_csv('.overflow.csv', sample=52)
+        read_csv('.overflow1.csv', sample=52)
+
+        # 35 characters is cuts off before the second number on the last line
+        # Should sample to end of line, otherwise pandas will infer `b` to be
+        # a float dtype
+        df = read_csv('.overflow2.csv', sample=35)
+        assert (df.dtypes == 'i8').all()