Repository: dask/dask

Issue:
`dd.read_csv` fails when the server doesn't support HTTP HEAD requests
Reading a CSV from a URL with `dask.dataframe` fails when the HTTP server does not support HEAD requests. The corresponding `pd.read_csv` succeeds, since it gets the whole file in one go and does not try to chunk the download.

Here is a MWE:
```python
import dask.dataframe as dd
dd.read_csv('https://data.lacity.org/resource/hpbr-2ant.csv', blocksize=None)
```

I had hoped that by setting `blocksize=None` I might be able to download the whole CSV and avoid the HEAD request (GET works fine), but it still tries to read a sample from `b_sample`:
https://github.com/dask/dask/blob/cb58d97f9452f67a922ba332b7a9471289ba94ee/dask/dataframe/io/csv.py#L341-L355
Relevant xrefs: #3160, intake/intake#302

Also, the relevant server (in this case a Socrata data portal) *should* be better about HTTP verbs, but in the absence of that, we can be more fault tolerant.

With a bit of guidance, I'd be happy to submit a PR for this.


Discussion:
The trouble is, that Dask wants to know the metadata of the dataframe before reading all of the data, so that you can do symbolic operations on it in a lazy way. When you load with pandas, you are up-front committed to reading the whole thing.

The version of HTTPFileSystem in [fsspec](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.implementations.http.HTTPFileSystem) is a little smarter than the previous version still used by Dask, in that it will still read the starts of files where less data is fetched than the readahead size. 
However, it still doesn't work, because the "sample" part of read_csv looks for the delimiter around the end of the sample range. This involves seeking to the end of the file, and we don't know where that it. Unfortunately, the server also ignores "range" requests...

In short, the process is geared towards finding line-delimiters in short parts of much bigger files, and then using that information to load the whole thing in relatively large chunks - this is certainly not the case here. At the very best, it would mean downloading all the data twice, once for the metadata and once when actually processing.
If we were given a list of many (hopefully small) CSV files presumably we
would only have to read in the first file.  Is this correct?

On Tue, Mar 26, 2019 at 10:53 AM Martin Durant <notifications@github.com>
wrote:

> The trouble is, that Dask wants to know the metadata of the dataframe
> before reading all of the data, so that you can do symbolic operations on
> it in a lazy way. When you load with pandas, you are up-front committed to
> reading the whole thing.
>
> The version of HTTPFileSystem in fsspec
> <https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.implementations.http.HTTPFileSystem>
> is a little smarter than the previous version still used by Dask, in that
> it will still read the starts of files where less data is fetched than the
> readahead size.
> However, it still doesn't work, because the "sample" part of read_csv
> looks for the delimiter around the end of the sample range. This involves
> seeking to the end of the file, and we don't know where that it.
> Unfortunately, the server also ignores "range" requests...
>
> In short, the process is geared towards finding line-delimiters in short
> parts of much bigger files, and then using that information to load the
> whole thing in relatively large chunks - this is certainly not the case
> here. At the very best, it would mean downloading all the data twice, once
> for the metadata and once when actually processing.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/4633#issuecomment-476772101>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszK8-mBGDoARtzO_iVL8EdnE5Qnxtks5val6UgaJpZM4cL7gV>
> .
>

To be sure, if the server doesn't support grabbing header information or range requests, we can't do any of the chunking or lazy operations (at least until it is downloaded).

But is it possible to make this more fault-tolerant for those cases? It would still be nice to be able to read these CSVs, even if the process is not very smart or high-performance.
https://github.com/martindurant/filesystem_spec/pull/38 would help int he fsspec implementation. This could be ported to dask, but is still ugly and hacky. We can be smarter, e.g., have "sample" be the whole of the first file and not try to seek delimiters anywhere. Should be possible with a little work.
I think that @ian-r-rose is volunteering to do some of that work.  @martindurant if you're able to lay out steps that a newcomer to the project (though generally an experienced dev otherwise) could execute then I suspect that you could get Ian to help you with this.  It may be that this gets too deep into the project though, in which case perhaps it's better not to do so.

(for context, Ian is a JupyterLab dev (among other things) the creator of dask-labextension, and generally a good catch if we can snag his time :) )
I think basically it comes down to being able to pass `sample=False` in [this line](https://github.com/dask/dask/blob/master/dask/dataframe/io/csv.py#L341), in which case the first of the `values` after line 360 would, once computed, play the part of the `b_sample`. Currently, the input `sample` arg to read_csv is treated explicitly as a number further up the function.
Another thing we could consider to shortcut the whole issue, is for the user to be able to provide `meta=` and not bother analysing the header at all.

Resolving patch:
diff --git a/dask/bytes/core.py b/dask/bytes/core.py
--- a/dask/bytes/core.py
+++ b/dask/bytes/core.py
@@ -48,7 +48,7 @@ def read_bytes(urlpath, delimiter=None, not_zero=False, blocksize="128 MiB",
         Chunk size in bytes, defaults to "128 MiB"
     compression : string or None
         String like 'gzip' or 'xz'.  Must support efficient random access.
-    sample : int or string
+    sample : int, string, or boolean
         Whether or not to return a header sample.
         Values can be ``False`` for "no sample requested"
         Or an integer or string value like ``2**20`` or ``"1 MiB"``
diff --git a/dask/bytes/http.py b/dask/bytes/http.py
--- a/dask/bytes/http.py
+++ b/dask/bytes/http.py
@@ -112,7 +112,16 @@ def __init__(self, url, session=None, block_size=None, **kwargs):
                                   **self.kwargs)
         except ValueError:
             # No size information - only allow read() and no seek()
-            self.size = None
+            self.size = None  # pragma: no cover
+        except requests.HTTPError as err:
+            # If we got an HTTP error, it may be due to HEAD being unsupported,
+            # or it may be due to server/permissions/not-found errors. In the former
+            # case we disable read() and seek(), in the latter we re-raise.
+            code = err.response.status_code
+            if code >= 500 or code in (401, 403, 404):
+                raise err
+            self.size = None  # pragma: no cover
+
         self.cache = None
         self.closed = False
         self.start = None
diff --git a/dask/dataframe/io/csv.py b/dask/dataframe/io/csv.py
--- a/dask/dataframe/io/csv.py
+++ b/dask/dataframe/io/csv.py
@@ -332,7 +332,7 @@ def read_pandas(reader, urlpath, blocksize=AUTO_BLOCKSIZE, collection=True,
     if compression not in seekable_files and compression not in cfiles:
         raise NotImplementedError("Compression format %s not installed" %
                                   compression)
-    if blocksize and blocksize < sample and lastskiprow != 0:
+    if blocksize and sample and blocksize < sample and lastskiprow != 0:
         warn("Unexpected behavior can result from passing skiprows when\n"
              "blocksize is smaller than sample size.\n"
              "Setting ``sample=blocksize``")
@@ -356,6 +356,10 @@ def read_pandas(reader, urlpath, blocksize=AUTO_BLOCKSIZE, collection=True,
 
     if not isinstance(values[0], (tuple, list)):
         values = [values]
+    # If we have not sampled, then use the first row of the first values
+    # as a representative sample.
+    if b_sample is False and len(values[0]):
+        b_sample = values[0][0].compute()
 
     # Get header row, and check that sample is long enough. If the file
     # contains a header row, we need at least 2 nonempty rows + the number of
@@ -367,7 +371,7 @@ def read_pandas(reader, urlpath, blocksize=AUTO_BLOCKSIZE, collection=True,
     # If the last partition is empty, don't count it
     nparts = 0 if not parts else len(parts) - int(not parts[-1])
 
-    if nparts < lastskiprow + need and len(b_sample) >= sample:
+    if sample is not False and nparts < lastskiprow + need and len(b_sample) >= sample:
         raise ValueError("Sample is not large enough to include at least one "
                          "row of data. Please increase the number of bytes "
                          "in `sample` in the call to `read_csv`/`read_table`")


Test patch:
diff --git a/dask/bytes/tests/test_local.py b/dask/bytes/tests/test_local.py
--- a/dask/bytes/tests/test_local.py
+++ b/dask/bytes/tests/test_local.py
@@ -158,6 +158,12 @@ def test_parse_sample_bytes():
         assert len(sample) == 40
 
 
+def test_read_bytes_no_sample():
+    with filetexts(files, mode='b'):
+        sample, _ = read_bytes('.test.accounts.1.json', sample=False)
+        assert sample is False
+
+
 def test_read_bytes_blocksize_none():
     with filetexts(files, mode='b'):
         sample, values = read_bytes('.test.accounts.*', blocksize=None)
diff --git a/dask/dataframe/io/tests/test_csv.py b/dask/dataframe/io/tests/test_csv.py
--- a/dask/dataframe/io/tests/test_csv.py
+++ b/dask/dataframe/io/tests/test_csv.py
@@ -621,6 +621,12 @@ def test_empty_csv_file():
         assert list(df.columns) == ['a', 'b']
 
 
+def test_read_csv_no_sample():
+    with filetexts(csv_files, mode='b') as fn:
+        df = dd.read_csv(fn, sample=False)
+        assert list(df.columns) == ['name', 'amount', 'id']
+
+
 def test_read_csv_sensitive_to_enforce():
     with filetexts(csv_files, mode='b'):
         a = dd.read_csv('2014-01-*.csv', enforce=True)