Repository: blaze/blaze

Issue:
Allow dynamic registering of new data source 
1. Allowing different data source to be associated to different url path. 
   For example, given a set of data:

``` yaml
iris:
  source: sqlite:///../examples/data/iris.db
accounts:
  source: ../examples/data/accounts.json.gz
```

access each part separately via:
http://localhost:6363/iris,  
and
http://localhost:6363/accounts

 2. Further, allowing dynamic registering of new data source after the original construction of server, the server may start with empty data source, and grow as different data source become available. 

For example, adding a new piece

``` yaml
cities:
  source: ../examples/data/cities.csv
```

to the server above on the run, and addressing the newly added via: http://localhost:6363/cities



Discussion:
In fact I think the API should change to something like this:

```
# compute (mostly unchanged, with query parameters)
- /compute?format=<typ>?.....

# resources (to facilitate user uploaded)
- /resource/upload?uri=... (local -> remote resource, e.g. uploading)
- /resource/remove?uri= (remove remote)
- /resource/download?uri= (remote -> local resource, e.g. downloading)
- /resource/discover?uri= (rename of ``spider``, discover remote resources)

# also need a way to say compute and directly add as another resource (with another name)
/compute?......,upload=uri_that_I_want

# data (prefix all actual resources)
- /data
```

so this would add the interactions of local/remote resource management.

There are a lot of things going on here. Comments below.
- After spending the morning playing with this and having to read a bit of the HTTP spec, I think that's too much information specified in the URL. For example, format is now passed in the header of the request as a vendor specific mimetype like this: `application/vnd.blaze+msgpack`. We should look around for existing solutions and idiomatic ways of doing the things we need to do rather than making up our own scheme and ignoring years of good work.
- In general, the client shouldn't know things like where the uploaded resource will be stored on the server and I don't think it should be able to specify where it should go either. It really only knows resource names.
- Requests that remove things should be a delete operation not a post with a delete url, like this `DELETE /my_table` and not like this `POST /my_table/delete`
- I'm -1 on adding umpteen query parameters to the compute route to achieve behaviors like "run this computation and then store it as a named resource". That route should do one thing and one thing only. I could imagine something like this `POST /compute/<name_of_result>` or `POST /<name_of_result>/compute` but I'm not sure how to use the existing compute implementation for that.
- I'm not sure what `/data` would do, but if you want to look at what data you have, you simple call the `/datashape` route.

@cpcloud you post what the API is atm. (e.g. the endpoints and what they do). no idea that `/datashape` exists.

good point! there's definitely some documentation needed

HTTP methods:

``` http
POST /compute
Content-Type: application-vnd.blaze+[format]
```
- `/compute` takes a serialized expression tree as its `data` argument
- `/compute` looks at the value of the `Content-Type` header in the request to determine what format to return the result in

``` http
GET /datashape
```
- `/datashape` takes no arguments

A user would do something like this (assume I have a blaze server running):

``` python
from blaze import *
c = Client('blaze://localhost')
result = float(c.iris.petal_length.sum())
```

that expression gets serialized to something like this:

``` python
{'args': [{'args': [{'args': ['t', 'var * {petal_length: float64}', None],
     'op': 'Symbol'},
    'petal_length'],
   'op': 'Field'},
  [0],
  False],
 'op': 'sum'}
```

which is sent to the server and reconstructed and then computed against and the response is returned according to the specified format (json, msgpack, or pickle at the moment).

this doc page spells out the above process in more detail: http://blaze.pydata.org/en/latest/server.html#using-the-python-requests-library

@cpcloud ok, though we would support direct URL access (e.g. something like ``blaze://localhost/compute?query=.....), but I see that's prob too complicated ATM (as you necessary have to have a string repr of the query, so not that useful), but something to think about.

But, having RESTful put/get/delete of remote resources seems to be quite useful.

My understanding is the API as two part:

For data consumer, it's pretty much the same as existing API, except that the user can access the root of the data piece he cares about via a url_prefix, for example:

```
t = Client('blaze://localhost/city')
```

once with the handle t, further dive into the detail of data is just as now, there's no need for the user to use complicated query string on url

(if "Moving down the stack" to see things at HTTP request level, it's the same as today except for the url_prefix)

For data provider, to register or deregister a data source, they should use HTTP POST/DELETE  method to address the data source rooted at certain url_prefix as a whole, again no diving into the detail of data.  For register, the actual source string and datashape will be in the payload of POST.


Resolving patch:
diff --git a/blaze/server/server.py b/blaze/server/server.py
--- a/blaze/server/server.py
+++ b/blaze/server/server.py
@@ -4,6 +4,7 @@
 import functools
 import re
 from warnings import warn
+import collections
 
 import flask
 from flask import Blueprint, Flask, request, Response
@@ -19,14 +20,14 @@ def wrapped(*a, **k):
             return wrapped
         return wrapper
 
-from toolz import assoc
+from toolz import assoc, valmap
 
 from datashape import Mono, discover
 from datashape.predicates import iscollection, isscalar
 from odo import odo
 
 import blaze
-from blaze import compute
+from blaze import compute, resource
 from blaze.expr import utils as expr_utils
 from blaze.compute import compute_up
 
@@ -114,6 +115,32 @@ def authorized(*args, **kwargs):
     return authorized
 
 
+def check_request(f):
+    @functools.wraps(f)
+    def check():
+        content_type = request.headers['content-type']
+        matched = mimetype_regex.match(content_type)
+
+        if matched is None:
+            return 'Unsupported serialization format %s' % content_type, 415
+
+        try:
+            serial = _get_format(matched.groups()[0])
+        except KeyError:
+            return (
+                "Unsupported serialization format '%s'" % matched.groups()[0],
+                415,
+            )
+
+        try:
+            payload = serial.loads(request.data)
+        except ValueError:
+            return ("Bad data.  Got %s " % request.data, 400)  # 400: Bad Request
+
+        return f(payload, serial)
+    return check
+
+
 class Server(object):
 
     """ Blaze Data Server
@@ -150,8 +177,6 @@ class Server(object):
     >>> server = Server({'accounts': df})
     >>> server.run() # doctest: +SKIP
     """
-    __slots__ = 'app', 'data', 'port'
-
     def __init__(self, data=None, formats=None, authorization=None):
         app = self.app = Flask('blaze.server.server')
         if data is None:
@@ -376,26 +401,8 @@ def from_tree(expr, namespace=None):
 @api.route('/compute', methods=['POST', 'HEAD', 'OPTIONS'])
 @crossdomain(origin='*', methods=['POST', 'HEAD', 'OPTIONS'])
 @authorization
-def compserver():
-    content_type = request.headers['content-type']
-    matched = mimetype_regex.match(content_type)
-
-    if matched is None:
-        return 'Unsupported serialization format %s' % content_type, 415
-
-    try:
-        serial = _get_format(matched.groups()[0])
-    except KeyError:
-        return (
-            "Unsupported serialization format '%s'" % matched.groups()[0],
-            415,
-        )
-
-    try:
-        payload = serial.loads(request.data)
-    except ValueError:
-        return ("Bad data.  Got %s " % request.data, 400)  # 400: Bad Request
-
+@check_request
+def compserver(payload, serial):
     ns = payload.get('namespace', dict())
     dataset = _get_data()
     ns[':leaf'] = symbol('leaf', discover(dataset))
@@ -423,3 +430,29 @@ def compserver():
         'data': result,
         'names': expr.fields
     })
+
+@api.route('/add', methods=['POST', 'HEAD', 'OPTIONS'])
+@crossdomain(origin='*', methods=['POST', 'HEAD', 'OPTIONS'])
+@authorization
+@check_request
+def addserver(payload, serial):
+    """Add a data resource to the server.
+
+    The reuest should contain serialized MutableMapping (dictionary) like
+    object, and the server should already be hosting a MutableMapping
+    resource.
+    """
+    data = _get_data.cache[flask.current_app]
+
+    data_not_mm_msg = ("Cannot update blaze server data since its current data"
+                       " is a %s and not a mutable mapping (dictionary like).")
+    if not isinstance(data, collections.MutableMapping):
+        return (data_not_mm_msg % type(data), 422)
+
+    payload_not_mm_msg = ("Cannot update blaze server with a %s payload, since"
+                          " it is not a mutable mapping (dictionary like).")
+    if not isinstance(payload, collections.MutableMapping):
+        return (payload_not_mm_msg % type(payload), 422)
+
+    data.update(valmap(resource, payload))
+    return 'OK'


Test patch:
diff --git a/blaze/server/tests/test_server.py b/blaze/server/tests/test_server.py
--- a/blaze/server/tests/test_server.py
+++ b/blaze/server/tests/test_server.py
@@ -4,6 +4,8 @@
 pytest.importorskip('flask')
 
 from base64 import b64encode
+from contextlib import contextmanager
+from copy import copy
 
 import datashape
 import numpy as np
@@ -13,7 +15,8 @@
 
 from odo import odo
 from blaze.utils import example
-from blaze import discover, symbol, by, CSV, compute, join, into, resource
+from blaze import (discover, symbol, by, CSV, compute, join, into, resource,
+                   Data)
 from blaze.server.client import mimetype
 from blaze.server.server import Server, to_tree, from_tree
 from blaze.server.serialization import all_formats
@@ -43,6 +46,12 @@ def server():
     s.app.testing = True
     return s
 
+@contextmanager
+def temp_server(data=None):
+    """For when we want to mutate the server"""
+    s = Server(copy(data), formats=all_formats)
+    s.app.testing = True
+    yield s.app.test_client()
 
 @pytest.yield_fixture
 def test(server):
@@ -50,6 +59,23 @@ def test(server):
         yield c
 
 
+@pytest.yield_fixture
+def empty_server():
+    s = Server(formats=all_formats)
+    s.app.testing = True
+    with s.app.test_client() as c:
+        yield c
+
+
+@pytest.yield_fixture
+def iris_server():
+    iris = CSV(example('iris.csv'))
+    s = Server(iris, all_formats)
+    s.app.testing = True
+    with s.app.test_client() as c:
+        yield c
+
+
 def test_datasets(test):
     response = test.get('/datashape')
     assert response.data.decode('utf-8') == str(discover(data))
@@ -183,15 +209,6 @@ def dont_test_compute_with_namespace(test, serial):
     assert data['names'] == ['name']
 
 
-@pytest.yield_fixture
-def iris_server():
-    iris = CSV(example('iris.csv'))
-    s = Server(iris, all_formats)
-    s.app.testing = True
-    with s.app.test_client() as c:
-        yield c
-
-
 iris = CSV(example('iris.csv'))
 
 
@@ -504,3 +521,103 @@ def test_isin(test, serial):
     }
     assert result.status_code == 200
     assert expected == serial.loads(result.data)
+
+
+@pytest.mark.parametrize('serial', all_formats)
+def test_add_data_to_empty_server(empty_server, serial):
+    # add data
+    with temp_server() as test:
+        iris_path = example('iris.csv')
+        blob = serial.dumps({'iris': iris_path})
+        response1 = empty_server.post(
+            '/add',
+            headers=mimetype(serial),
+            data=blob,
+        )
+        assert 'OK' in response1.status
+        assert response1.status_code == 200
+
+        # check for expected server datashape
+        response2 = empty_server.get('/datashape')
+        expected2 = str(discover({'iris': resource(iris_path)}))
+        assert response2.data.decode('utf-8') == expected2
+
+        # compute on added data
+        t = Data({'iris': resource(iris_path)})
+        expr = t.iris.petal_length.sum()
+
+        response3 = empty_server.post(
+            '/compute',
+            data=serial.dumps({'expr': to_tree(expr)}),
+            headers=mimetype(serial)
+        )
+
+        result3 = serial.loads(response3.data)['data']
+        expected3 = compute(expr, {'iris': resource(iris_path)})
+        assert result3 == expected3
+
+
+@pytest.mark.parametrize('serial', all_formats)
+def test_add_data_to_server(serial):
+    with temp_server(data) as test:
+        # add data
+        initial_datashape = test.get('/datashape').data.decode('utf-8')
+        iris_path = example('iris.csv')
+        blob = serial.dumps({'iris': iris_path})
+        response1 = test.post(
+            '/add',
+            headers=mimetype(serial),
+            data=blob,
+        )
+        assert 'OK' in response1.status
+        assert response1.status_code == 200
+
+        # check for expected server datashape
+        new_datashape = test.get('/datashape').data.decode('utf-8')
+        data2 = data.copy()
+        data2.update({'iris': resource(iris_path)})
+        expected2 = str(discover(data2))
+        from pprint import pprint as pp
+        #import ipdb; ipdb.set_trace()
+        assert new_datashape == expected2
+        a = new_datashape != initial_datashape
+        assert new_datashape != initial_datashape
+
+        # compute on added data
+        t = Data({'iris': resource(iris_path)})
+        expr = t.iris.petal_length.sum()
+
+        response3 = test.post(
+            '/compute',
+            data=serial.dumps({'expr': to_tree(expr)}),
+            headers=mimetype(serial)
+        )
+
+        result3 = serial.loads(response3.data)['data']
+        expected3 = compute(expr, {'iris': resource(iris_path)})
+        assert result3 == expected3
+
+
+@pytest.mark.parametrize('serial', all_formats)
+def test_cant_add_data_to_server(iris_server, serial):
+    # try adding more data to server
+    iris_path = example('iris.csv')
+    blob = serial.dumps({'iris': iris_path})
+    response1 = iris_server.post(
+        '/add',
+        headers=mimetype(serial),
+        data=blob,
+    )
+    assert response1.status_code == 422
+
+
+@pytest.mark.parametrize('serial', all_formats)
+def test_bad_add_payload(empty_server, serial):
+    # try adding more data to server
+    blob = serial.dumps('This is not a mutable mapping.')
+    response1 = empty_server.post(
+        '/add',
+        headers=mimetype(serial),
+        data=blob,
+    )
+    assert response1.status_code == 422