Repository: dask/dask

Issue:
request: HTTP file system
It would be convenient to be able to do `open_files` on a list of HTTP URLs. This cannot be a complete file-system like the other backends, because there is no possibility of `glob()`, but I still think it would be useful for accessing remote data. 

In general, the size of a given URL's data can be found through a HEAD command, and chunks read using the `Range` header keyword; some servers do not support that, however, and then downloads could only be parallelised across multiple files.


Discussion:
I agree that this would be a useful thing to have

On Sun, Feb 11, 2018 at 4:35 PM, Martin Durant <notifications@github.com>
wrote:

> It would be convenient to be able to do open_files on a list of HTTP
> URLs. This cannot be a complete file-system like the other backends,
> because there is no possibility of glob(), but I still think it would be
> useful for accessing remote data.
>
> In general, the size of a given URL's data can be found through a HEAD
> command, and chunks read using the Range header keyword; some servers do
> not support that, however, and then downloads could only be parallelised
> across multiple files.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3158>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszPzvmmiRwRVdCUlvFZ0SJE63eX0Oks5tT107gaJpZM4SBhJV>
> .
>

I'm curious, is there a particular use case that you had in mind?

On Sun, Feb 11, 2018 at 5:22 PM, Matthew Rocklin <mrocklin@anaconda.com>
wrote:

> I agree that this would be a useful thing to have
>
> On Sun, Feb 11, 2018 at 4:35 PM, Martin Durant <notifications@github.com>
> wrote:
>
>> It would be convenient to be able to do open_files on a list of HTTP
>> URLs. This cannot be a complete file-system like the other backends,
>> because there is no possibility of glob(), but I still think it would be
>> useful for accessing remote data.
>>
>> In general, the size of a given URL's data can be found through a HEAD
>> command, and chunks read using the Range header keyword; some servers do
>> not support that, however, and then downloads could only be parallelised
>> across multiple files.
>>
>> —
>> You are receiving this because you are subscribed to this thread.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/dask/dask/issues/3158>, or mute the thread
>> <https://github.com/notifications/unsubscribe-auth/AASszPzvmmiRwRVdCUlvFZ0SJE63eX0Oks5tT107gaJpZM4SBhJV>
>> .
>>
>
>

Yes, I am looking at archival astro data in FITS format, where the extensions can be addressed independently, and, for uncompressed data, you can slice the binary data directly.

Also, in the case of Intake, many data resources (and possibly their schemas) are available via a URI that may be HTTP or something else, and we'd like to use just one function to generate file-like objects for them.

It is worth noting that some loading functions like `pd.read_csv` already support HTTP (I don't know if that is done chunk-wise, though).

Resolving patch:
diff --git a/dask/bytes/core.py b/dask/bytes/core.py
--- a/dask/bytes/core.py
+++ b/dask/bytes/core.py
@@ -449,6 +449,15 @@ def get_fs(protocol, storage_options=None):
     elif protocol == 'hdfs':
         cls = get_hdfs_driver(_globals.get("hdfs_driver", "auto"))
 
+    elif protocol in ['http', 'https']:
+        import_required('requests',
+                        "Need to install `requests` for HTTP support\n"
+                        "   conda install requests\n"
+                        "    or\n"
+                        "   pip install requests")
+        import dask.bytes.http  # noqa, registers HTTP backend
+        cls = _filesystems[protocol]
+
     else:
         raise ValueError("Unknown protocol %s" % protocol)
 
diff --git a/dask/bytes/http.py b/dask/bytes/http.py
new file mode 100644
--- /dev/null
+++ b/dask/bytes/http.py
@@ -0,0 +1,283 @@
+from __future__ import print_function, division, absolute_import
+
+import requests
+import uuid
+
+from . import core
+
+DEFAULT_BLOCK_SIZE = 5 * 2 ** 20
+
+
+class HTTPFileSystem(core.FileSystem):
+    """
+    Simple File-System for fetching data via HTTP(S)
+
+    Unlike other file-systems, HTTP is limited in that it does not provide glob
+    or write capability.
+    """
+    sep = '/'
+
+    def __init__(self, **storage_options):
+        """
+        Parameters
+        ----------
+        storage_options: key-value
+            May be credentials, e.g., `{'auth': ('username', 'pword')}` or any
+            other parameters for requests
+        """
+        self.block_size = storage_options.pop('block_size', DEFAULT_BLOCK_SIZE)
+        self.kwargs = storage_options
+        self.session = requests.Session()
+
+    def glob(self, url):
+        """For a template path, return matching files"""
+        raise NotImplementedError
+
+    def mkdirs(self, url):
+        """Make any intermediate directories to make path writable"""
+        raise NotImplementedError
+
+    def open(self, url, mode='rb', block_size=None, **kwargs):
+        """Make a file-like object
+
+        Parameters
+        ----------
+        url: str
+            Full URL with protocol
+        mode: string
+            must be "rb"
+        kwargs: key-value
+            Any other parameters, passed to requests calls
+        """
+        if mode != 'rb':
+            raise NotImplementedError
+        block_size = block_size if block_size is not None else self.block_size
+        return HTTPFile(url, self.session, block_size, **self.kwargs)
+
+    def ukey(self, url):
+        """Unique identifier, implied file might have changed every time"""
+        return uuid.uuid1().hex
+
+    def size(self, url):
+        """Size in bytes of the file at path"""
+        return file_size(url, session=self.session, **self.kwargs)
+
+
+core._filesystems['http'] = HTTPFileSystem
+core._filesystems['https'] = HTTPFileSystem
+
+
+class HTTPFile(object):
+    """
+    A file-like object pointing to a remove HTTP(S) resource
+
+    Supports only reading, with read-ahead of a predermined block-size.
+
+    In the case that the server does not supply the filesize, only reading of
+    the complete file in one go is supported.
+
+    Parameters
+    ----------
+    url: str
+        Full URL of the remote resource, including the protocol
+    session: requests.Session or None
+        All calls will be made within this session, to avoid restarting
+        connections where the server allows this
+    block_size: int or None
+        The amount of read-ahead to do, in bytes. Default is 5MB, or the value
+        configured for the FileSystem creating this file
+    kwargs: all other key-values are passed to reqeuests calls.
+    """
+
+    def __init__(self, url, session=None, block_size=None, **kwargs):
+        self.url = url
+        self.kwargs = kwargs
+        self.loc = 0
+        self.session = session if session is not None else requests.Session()
+        self.blocksize = (block_size if block_size is not None
+                          else DEFAULT_BLOCK_SIZE)
+        try:
+            self.size = file_size(url, self.session, allow_redirects=True,
+                                  **self.kwargs)
+        except ValueError:
+            # No size information - only allow read() and no seek()
+            self.size = None
+        self.cache = None
+        self.closed = False
+        self.start = None
+        self.end = None
+
+    def seek(self, where, whence=0):
+        """Set file position
+
+        Parameters
+        ----------
+        where: int
+            Location to set
+        whence: int (default 0)
+            If zero, set from start of file (value should be positive); if 1,
+            set relative to current position; if 2, set relative to end of file
+            (value shoulf be negative)
+
+        Returns the position.
+        """
+        if self.size is None:
+            raise ValueError('Cannot seek since size of file is not known')
+        if whence == 0:
+            nloc = where
+        elif whence == 1:
+            nloc += where
+        elif whence == 2:
+            nloc = self.size + where
+        else:
+            raise ValueError('Whence must be in [1, 2, 3], but got %s' % whence)
+        if nloc < 0:
+            raise ValueError('Seek before start of file')
+        self.loc = nloc
+        return nloc
+
+    def tell(self):
+        """Get current file byte position"""
+        return self.loc
+
+    def read(self, length=-1):
+        """Read bytes from file
+
+        Parameters
+        ----------
+        length: int
+            Read up to this many bytes. If negative, read all content to end of
+            file. If the server has not supplied the filesize, attempting to
+            read only part of the data will raise a ValueError.
+        """
+        if self.size is None:
+            if length >= 0:
+                raise ValueError('File size is unknown, must read all data')
+            else:
+                return self._fetch_all()
+        if length < 0 or self.loc + length > self.size:
+            end = self.size
+        else:
+            end = self.loc + length
+        if self.loc >= self.size:
+            return b''
+        self. _fetch(self.loc, end)
+        data = self.cache[self.loc - self.start:end - self.start]
+        self.loc = end
+        return data
+
+    def _fetch(self, start, end):
+        """Set new bounds for data cache and fetch data, if required"""
+        if self.start is None and self.end is None:
+            # First read
+            self.start = start
+            self.end = end + self.blocksize
+            self.cache = self._fetch_range(start, self.end)
+        elif start < self.start:
+            if self.end - end > self.blocksize:
+                self.start = start
+                self.end = end + self.blocksize
+                self.cache = self._fetch_range(self.start, self.end)
+            else:
+                new = self._fetch_range(start, self.start)
+                self.start = start
+                self.cache = new + self.cache
+        elif end > self.end:
+            if self.end > self.size:
+                return
+            if end - self.end > self.blocksize:
+                self.start = start
+                self.end = end + self.blocksize
+                self.cache = self._fetch_range(self.start, self.end)
+            else:
+                new = self._fetch_range(self.end, end + self.blocksize)
+                self.end = end + self.blocksize
+                self.cache = self.cache + new
+
+    def _fetch_all(self):
+        """Read whole file in one shot, without caching
+
+        This is only called when size is None and read() is called without a
+        byte-count.
+        """
+        r = self.session.get(self.url, **self.kwargs)
+        r.raise_for_status()
+        return r.content
+
+    def _fetch_range(self, start, end):
+        """Download a block of data
+
+        The expectation is that the server returns only the requested bytes,
+        with HTTP code 206. If this is not the case, we first check the headers,
+        and then stream the output - if the data size is bigger than we
+        requested, an exception is raised.
+        """
+        kwargs = self.kwargs.copy()
+        headers = self.kwargs.pop('headers', {})
+        headers['Range'] = 'bytes=%i-%i' % (start, end - 1)
+        r = self.session.get(self.url, headers=headers, stream=True, **kwargs)
+        r.raise_for_status()
+        if r.status_code == 206:
+            # partial content, as expected
+            return r.content
+        if 'Content-Length' in r.headers:
+            cl = int(r.headers['Content-Length'])
+            if cl <= end - start:
+                # data size OK
+                return r.content
+            else:
+                raise ValueError('Got more bytes (%i) than requested (%i)' % (
+                    cl, end - start))
+        cl = 0
+        out = []
+        for chunk in r.iter_content(chunk_size=2 ** 20):
+            # data size unknown, let's see if it goes too big
+            if chunk:
+                out.append(chunk)
+                cl += len(chunk)
+                if cl > end - start:
+                    raise ValueError(
+                        'Got more bytes so far (>%i) than requested (%i)' % (
+                            cl, end - start))
+            else:
+                break
+        return b''.join(out)
+
+    def __enter__(self):
+        self.loc = 0
+        return self
+
+    def __exit__(self, *args):
+        self.close()
+
+    def __iter__(self):
+        # no text lines here, use TextIOWrapper
+        raise NotImplementedError
+
+    def write(self):
+        raise NotImplementedError
+
+    def flush(self):
+        pass
+
+    def close(self):
+        self.closed = True
+
+    def seekable(self):
+        return True
+
+    def writable(self):
+        return False
+
+    def readable(self):
+        return True
+
+
+def file_size(url, session, **kwargs):
+    """Call HEAD on the server to get file size"""
+    r = session.head(url, **kwargs)
+    r.raise_for_status()
+    if 'Content-Length' in r.headers:
+        return int(r.headers['Content-Length'])
+    else:
+        raise ValueError("Server did not supply size of %s" % url)
diff --git a/dask/bytes/utils.py b/dask/bytes/utils.py
--- a/dask/bytes/utils.py
+++ b/dask/bytes/utils.py
@@ -51,6 +51,10 @@ def infer_storage_options(urlpath, inherit_storage_options=None):
         if windows_path:
             path = '%s:%s' % windows_path.groups()
 
+    if protocol in ['http', 'https']:
+        # for HTTP, we don't want to parse, as requests will anyway
+        return {'protocol': protocol, 'path': urlpath}
+
     options = {
         'protocol': protocol,
         'path': path,


Test patch:
diff --git a/dask/bytes/tests/test_bytes_utils.py b/dask/bytes/tests/test_bytes_utils.py
--- a/dask/bytes/tests/test_bytes_utils.py
+++ b/dask/bytes/tests/test_bytes_utils.py
@@ -91,7 +91,8 @@ def test_infer_storage_options():
     assert so.pop('username') == 'User-name'
     assert so.pop('host') == 'Node-name.com'
 
-    assert infer_storage_options('http://127.0.0.1:8080/test.csv')['host'] == '127.0.0.1'
+    u = 'http://127.0.0.1:8080/test.csv'
+    assert infer_storage_options(u) == {'protocol': 'http', 'path': u}
 
     # For s3 and gcs the netloc is actually the bucket name, so we want to
     # include it in the path. Test that:
diff --git a/dask/bytes/tests/test_http.py b/dask/bytes/tests/test_http.py
new file mode 100644
--- /dev/null
+++ b/dask/bytes/tests/test_http.py
@@ -0,0 +1,135 @@
+import os
+import pytest
+import requests
+import subprocess
+import time
+
+from dask.bytes.core import open_files
+from dask.compatibility import PY2
+from dask.utils import tmpdir
+
+files = ['a', 'b']
+
+
+@pytest.fixture(scope='module')
+def dir_server():
+    with tmpdir() as d:
+        for fn in files:
+            with open(os.path.join(d, fn), 'wb') as f:
+                f.write(b'a' * 10000)
+
+        if PY2:
+            cmd = ['python', '-m', 'SimpleHTTPServer', '8999']
+        else:
+            cmd = ['python', '-m', 'http.server', '8999']
+        p = subprocess.Popen(cmd, cwd=d)
+        timeout = 10
+        while True:
+            try:
+                requests.get('http://localhost:8999')
+                break
+            except requests.exceptions.ConnectionError:
+                time.sleep(0.1)
+                timeout -= 0.1
+                if timeout < 0:
+                    raise RuntimeError('Server did not appear')
+        yield d
+        p.terminate()
+
+
+def test_simple(dir_server):
+    root = 'http://localhost:8999/'
+    fn = files[0]
+    f = open_files(root + fn)[0]
+    with f as f:
+        data = f.read()
+    assert data == open(os.path.join(dir_server, fn), 'rb').read()
+
+
+@pytest.mark.parametrize('block_size', [None, 99999])
+def test_ops(dir_server, block_size):
+    root = 'http://localhost:8999/'
+    fn = files[0]
+    f = open_files(root + fn)[0]
+    data = open(os.path.join(dir_server, fn), 'rb').read()
+    with f as f:
+        # these pass because the default
+        assert f.read(10) == data[:10]
+        f.seek(0)
+        assert f.read(10) == data[:10]
+        assert f.read(10) == data[10:20]
+        f.seek(-10, 2)
+        assert f.read() == data[-10:]
+
+
+def test_ops_blocksize(dir_server):
+    root = 'http://localhost:8999/'
+    fn = files[0]
+    f = open_files(root + fn, block_size=2)[0]
+    data = open(os.path.join(dir_server, fn), 'rb').read()
+    with f as f:
+        # it's OK to read the whole file
+        assert f.read() == data
+
+    # note that if we reuse f from above, because it is tokenized, we get
+    # the same open file - where is this cached?
+    fn = files[1]
+    f = open_files(root + fn, block_size=2)[0]
+    with f as f:
+        # fails becasue we want only 12 bytes
+        with pytest.raises(ValueError):
+            assert f.read(10) == data[:10]
+
+
+def test_errors(dir_server):
+    f = open_files('http://localhost:8999/doesnotexist')[0]
+    with pytest.raises(requests.exceptions.RequestException):
+        with f:
+            pass
+    f = open_files('http://nohost/')[0]
+    with pytest.raises(requests.exceptions.RequestException):
+        with f:
+            pass
+    root = 'http://localhost:8999/'
+    fn = files[0]
+    f = open_files(root + fn, mode='wb')[0]
+    with pytest.raises(NotImplementedError):
+        with f:
+            pass
+    f = open_files(root + fn)[0]
+    with f as f:
+        with pytest.raises(ValueError):
+            f.seek(-1)
+
+
+def test_files(dir_server):
+    root = 'http://localhost:8999/'
+    fs = open_files([root + f for f in files])
+    for f, f2 in zip(fs, files):
+        with f as f:
+            assert f.read() == open(os.path.join(dir_server, f2), 'rb').read()
+
+
+@pytest.mark.network
+def test_parquet():
+    dd = pytest.importorskip('dask.dataframe')
+    pytest.importorskip('fastparquet')  # no pyarrow compatability FS yet
+    df = dd.read_parquet([
+        'https://github.com/Parquet/parquet-compatibility/raw/'
+        'master/parquet-testdata/impala/1.1.1-NONE/'
+        'nation.impala.parquet']).compute()
+    assert df.n_nationkey.tolist() == list(range(25))
+    assert df.columns.tolist() == ['n_nationkey', 'n_name', 'n_regionkey',
+                                   'n_comment']
+
+
+@pytest.mark.network
+def test_bag():
+    # This test pulls from different hosts
+    db = pytest.importorskip('dask.bag')
+    urls = ['https://raw.githubusercontent.com/weierophinney/pastebin/'
+            'master/public/js-src/dojox/data/tests/stores/patterns.csv',
+            'https://en.wikipedia.org']
+    b = db.read_text(urls)
+    assert b.npartitions == 2
+    b.compute()