Repository: dask/dask

Issue:
Bag.from_url
Similar to #241 we might want to construct bags from resources found on the web.  The following lines from githubarchive.org show a possible api using unix-style options with ranges.
- Activity for 1/1/2015 @ 3PM UTC   `wget http://data.githubarchive.org/2015-01-01-15.json.gz`
- Activity for 1/1/2015 `wget http://data.githubarchive.org/2015-01-01-{0..23}.json.gz`
- Activity for all of January 2015  `wget http://data.githubarchive.org/2015-01-{01..30}-{0..23}.json.gz`

Is there a simple python way to duplicate this behavior?  Is `urllib2.urlopen` the best way to pull down data?



Discussion:
This is also something we do all the time. A tricky aspect of this is that it is quick easy to hit limits when querying URLs (especially when done concurrently) -- you really need some retry/backoff logic. Also, it's quite often handy to be able to cache results on disk locally.

I wonder if we can handle retry/backoff logic outside of the dask graph.  Presumably this could be wrapped up in some robust `urlopen` function.  Under what cases does urlopen fail?

Wouldn't the urllib or requests handle the retrying? http://stackoverflow.com/a/15431343

Should we check if each URL exists before making the bag? We could do this by requesting a HEAD from each URL.

I would do the simplest thing until a practical problem or user guides us to make a change.


Resolving patch:
diff --git a/dask/bag/__init__.py b/dask/bag/__init__.py
--- a/dask/bag/__init__.py
+++ b/dask/bag/__init__.py
@@ -1,5 +1,5 @@
 from __future__ import absolute_import, division, print_function
 
 from .core import (Bag, Item, from_sequence, from_filenames, from_hdfs,
-    to_textfiles, concat)
+        from_url, to_textfiles, concat)
 from ..context import set_options
diff --git a/dask/bag/core.py b/dask/bag/core.py
--- a/dask/bag/core.py
+++ b/dask/bag/core.py
@@ -9,6 +9,11 @@
 import bz2
 import os
 
+try:
+    from urllib2 import urlopen
+except ImportError:
+    from urllib.request import urlopen
+
 from glob import glob
 from collections import Iterable, Iterator, defaultdict
 from functools import wraps, partial
@@ -846,6 +851,36 @@ def from_sequence(seq, partition_size=None, npartitions=None):
     return Bag(d, name, len(d))
 
 
+def from_url(urls):
+    """Create a dask.bag from a url
+
+    >>> a = from_url('http://raw.githubusercontent.com/ContinuumIO/dask/master/README.rst')
+    >>> a.npartitions
+    1
+    >>> a.take(8)
+    ('Dask\n',
+     '====\n',
+     '\n',
+     '|Build Status| |Coverage| |Doc Status| |Gitter|\n',
+     '\n',
+     'Dask provides multi-core execution on larger-than-memory datasets using blocked\n',
+     'algorithms and task scheduling.  It maps high-level NumPy and list operations\n',
+     'on large datasets on to graphs of many operations on small in-memory datasets.\n')
+
+    >>> b = from_url(['http://github.com', 'http://google.com'])
+    >>> b.npartions
+    2
+
+    """
+    if isinstance(urls, str):
+        urls = [urls]
+    name = next(load_names)
+    dsk = {}
+    for i, u in enumerate(urls):
+        dsk[(name, i)] = (list, (urlopen, u))
+    return Bag(dsk, name, len(urls))
+
+
 def dictitems(d):
     """ A pickleable version of dict.items
 


Test patch:
diff --git a/dask/bag/tests/test_bag.py b/dask/bag/tests/test_bag.py
--- a/dask/bag/tests/test_bag.py
+++ b/dask/bag/tests/test_bag.py
@@ -190,6 +190,16 @@ def test_can_use_dict_to_make_concrete():
     assert isinstance(dict(b.frequencies()), dict)
 
 
+def test_from_url():
+    a = db.from_url(['http://google.com', 'http://github.com'])
+    assert a.npartitions == 2
+    a.compute()
+
+    b = db.from_url('http://raw.githubusercontent.com/ContinuumIO/dask/master/README.rst')
+    assert b.npartitions == 1
+    assert b'Dask\n' in b.compute()
+
+
 def test_from_filenames():
     with filetexts({'a1.log': 'A\nB', 'a2.log': 'C\nD'}) as fns:
         assert set(line.strip() for line in db.from_filenames(fns)) == \