Repository: runwayml/model-sdk

Issue:
POSTing a missing value for a category data_type throws a 500
POSTing a variable for a category data type that isn't in the `choices` list results in a bombed request. See code notes below.

```python
import runway
from runway.data_types import category

# if no default value is specified, the first element in the choices
# list will be used
cat = category(choices=["rgb", "bgr", "rgba", "bgra"], default="rgba")
@runway.setup(options={ "pixel_order": cat })
def setup(opts):
    print("The selected pixel order is {}".format(opts["pixel_order"]))

runway.run()
```

```bash
curl -H "content-type: application/json" -d '{"pixel_order": "bgr"}' http://localhost:8000/setup # 200 OK
curl -H "content-type: application/json" -d '{"pixel_order": "not_found"}' # 500 Error
```

```
The selected pixel order is bgr
127.0.0.1 - - [2019-04-09 10:06:19] "POST /setup HTTP/1.1" 200 165 0.001573
[2019-04-09 10:06:28,616] ERROR in app: Exception on /setup [POST]
Traceback (most recent call last):
  File "/Users/brannon/Documents/code/runway/rw_python_sdk/venv/lib/python3.7/site-packages/Flask-1.0.2-py3.7.egg/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/brannon/Documents/code/runway/rw_python_sdk/venv/lib/python3.7/site-packages/Flask-1.0.2-py3.7.egg/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/brannon/Documents/code/runway/rw_python_sdk/venv/lib/python3.7/site-packages/Flask_Cors-3.0.7-py3.7.egg/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/Users/brannon/Documents/code/runway/rw_python_sdk/venv/lib/python3.7/site-packages/Flask-1.0.2-py3.7.egg/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/brannon/Documents/code/runway/rw_python_sdk/venv/lib/python3.7/site-packages/Flask-1.0.2-py3.7.egg/flask/_compat.py", line 35, in reraise
    raise value
  File "/Users/brannon/Documents/code/runway/rw_python_sdk/venv/lib/python3.7/site-packages/Flask-1.0.2-py3.7.egg/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/brannon/Documents/code/runway/rw_python_sdk/venv/lib/python3.7/site-packages/Flask-1.0.2-py3.7.egg/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/brannon/Documents/code/runway/rw_python_sdk/venv/lib/python3.7/site-packages/runway_python-0.0.55-py3.7.egg/runway/model.py", line 44, in setup_route
    self.setup_model(opts)
  File "/Users/brannon/Documents/code/runway/rw_python_sdk/venv/lib/python3.7/site-packages/runway_python-0.0.55-py3.7.egg/runway/model.py", line 153, in setup_model
    deserialized_opts[name] = opt.deserialize(opts[name])
  File "/Users/brannon/Documents/code/runway/rw_python_sdk/venv/lib/python3.7/site-packages/runway_python-0.0.55-py3.7.egg/runway/data_types.py", line 133, in deserialize
    raise InvalidInputError(self.name)
  File "/Users/brannon/Documents/code/runway/rw_python_sdk/venv/lib/python3.7/site-packages/runway_python-0.0.55-py3.7.egg/runway/exceptions.py", line 32, in __init__
    super(MissingInputError, self).__init__()
TypeError: super(type, obj): obj must be an instance or subtype of type
127.0.0.1 - - [2019-04-09 10:06:28] "POST /setup HTTP/1.1" 500 443 0.014482
```

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/runwayml/model-sdk/pull/46?src=pr&el=h1) Report
> Merging [#46](https://codecov.io/gh/runwayml/model-sdk/pull/46?src=pr&el=desc) into [master](https://codecov.io/gh/runwayml/model-sdk/commit/97340ced834a040547e4fb72430224ef9bcf34f9?src=pr&el=desc) will **increase** coverage by `1.44%`.
> The diff coverage is `100%`.


```diff
@@            Coverage Diff             @@
##           master      #46      +/-   ##
==========================================
+ Coverage   91.98%   93.42%   +1.44%     
==========================================
  Files           5        5              
  Lines         424      426       +2     
==========================================
+ Hits          390      398       +8     
+ Misses         34       28       -6
```


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c3a0dae..d749298 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,18 @@
 
 ## v0.0.70
 
+- Move `/` meta endpoint to `/meta` and specify `GET` as the only allowed method (its still available via `/` for backwards compatibility).
+- `/healthcheck` endpoint now returns JSON (e.g. `{ "status": "RUNNING" }`)
+- Limit `/healthcheck` to only allow HTTP methods.
+- Add `modelSDKVersion` to manifest at `GET /meta`.
+- Add `millisRunning` to manifest at `GET /meta`.
+- Add `millisSinceLastCommand` to manifest at `GET /meta`.
+- Add `GPU` to manifest at `GET /meta` so that a model's GPU accessibility can be determined at runtime.
+- Add undocumented `millis_running()` and `millis_since_last_command()` to `RunwayModel`.
+- Add `default` parameter to `vector` data type.
+- Model server now wraps common server error codes in JSON responses (e.g. 401, 403, 404, 405, 500).
+- Model server now sniffs the body of `POST` for JSON even if `content-type: application/json` is not sent in the request header. [#1](https://github.com/runwayml/model-sdk/issues/1)
+- Model server now returns `content-type: application/json`. [#6](https://github.com/runwayml/model-sdk/issues/6)
 - Add `RW_NO_SERVE` environment variable and `no_serve` keyword argument to `runway.run()`. These settings prevent `runway.run()` from starting the Flask server so that mock HTTP requests can be made via `app.test_client()`. See [Testing Flask Applications](http://flask.pocoo.org/docs/1.0/testing/) for more details.
 - Add model tests in [`tests/test_model.py`](tests/test_model.py)
 - Minor change to `docs/` so that JavaScript HTTP -> HTTPS redirect only occurs when the protocol is actually `http:`.
diff --git a/runway/data_types.py b/runway/data_types.py
index 3af9939..02eeb14 100644
--- a/runway/data_types.py
+++ b/runway/data_types.py
@@ -284,7 +284,8 @@ class category(object):
 
     def deserialize(self, value):
         if value not in self.choices:
-            raise InvalidArgumentError(self.name)
+            msg = 'category value "%s" does not appear in choices list.' % value
+            raise InvalidArgumentError(self.name, msg)
         return value
 
     def serialize(self, value):
diff --git a/runway/exceptions.py b/runway/exceptions.py
index 19cae20..617e932 100644
--- a/runway/exceptions.py
+++ b/runway/exceptions.py
@@ -85,9 +85,10 @@ class InvalidArgumentError(RunwayError):
     :ivar code: An HTTP error code, set to 400
     :type code: number
     """
-    def __init__(self, name):
+    def __init__(self, name, message=None):
         super(InvalidArgumentError, self).__init__()
         self.message = 'Invalid argument: %s.' % name
+        if message is not None: self.message += ' %s' % message
         self.code = 400
 
 
diff --git a/runway/model.py b/runway/model.py
index 9eba787..832e0f0 100644
--- a/runway/model.py
+++ b/runway/model.py
@@ -5,13 +5,15 @@ import datetime
 import traceback
 import json
 from six import reraise
-from flask import Flask, request
+from flask import Flask, request, jsonify
 from flask_cors import CORS
 from gevent.pywsgi import WSGIServer
 from .exceptions import RunwayError, MissingInputError, MissingOptionError, \
     InferenceError, UnknownCommandError, SetupError
 from .data_types import *
-from .utils import gzipped, serialize_command, cast_to_obj
+from .utils import gzipped, serialize_command, cast_to_obj, timestamp_millis, \
+        validate_post_request_body_is_json, get_json_or_none_if_invalid
+from .__version__ import __version__ as model_sdk_version
 
 class RunwayModel(object):
     """A Runway Model server. A singleton instance of this class is created automatically
@@ -19,6 +21,8 @@ class RunwayModel(object):
     """
 
     def __init__(self):
+        self.millis_run_started_at = None
+        self.millis_last_command = None
         self.options = []
         self.setup_fn = None
         self.commands = {}
@@ -27,35 +31,75 @@ class RunwayModel(object):
         self.running_status = 'STARTING'
         self.app = Flask(__name__)
         CORS(self.app)
+        self.define_error_handlers()
         self.define_routes()
 
+    def define_error_handlers(self):
+
+        # not yet implemented, but if and when it is lets make sure its returned
+        # as JSON
+        @self.app.errorhandler(401)
+        def unauthorized(e):
+            msg = 'Unauthorized (well... '
+            msg += 'really unauthenticated but hey I didn\'t write the spec).'
+            return jsonify(dict(error=msg)), 401
+
+        # not yet implemented, but if and when it is lets make sure its returned
+        # as JSON
+        @self.app.errorhandler(403)
+        def forbidden(e):
+            return jsonify(dict(error='Forbidden.')), 403
+
+        @self.app.errorhandler(404)
+        def page_not_found(e):
+            return jsonify(dict(error='Not found.')), 404
+
+        @self.app.errorhandler(405)
+        def method_not_allowed(e):
+            return jsonify(dict(error='Method not allowed.')), 405
+
+        # we shouldn't have any of these as we are wrapping errors in
+        # RunwayError objects and returning stacktraces, but it can't hurt
+        # to be safe.
+        @self.app.errorhandler(500)
+        def internal_server_error(e):
+            return jsonify(dict(error='Internal server error.')), 500
+
     def define_routes(self):
-        @self.app.route('/')
+
+        @self.app.route('/', methods=['GET'])
+        @self.app.route('/meta', methods=['GET'])
         def manifest():
-            return json.dumps(dict(
+            return jsonify(dict(
+                modelSDKVersion=model_sdk_version,
+                millisRunning=self.millis_running(),
+                millisSinceLastCommand=self.millis_since_last_command(),
+                GPU=os.environ.get('GPU') == '1',
                 options=[opt.to_dict() for opt in self.options],
                 commands=[serialize_command(cmd) for cmd in self.commands.values()]
             ))
 
-        @self.app.route('/healthcheck')
+        @self.app.route('/healthcheck', methods=['GET'])
         def healthcheck_route():
-            return self.running_status
+            return jsonify(dict(status=self.running_status))
 
         @self.app.route('/setup', methods=['POST'])
+        @validate_post_request_body_is_json
         def setup_route():
-            opts = request.json
+            opts = get_json_or_none_if_invalid(request)
             try:
                 self.setup_model(opts)
-                return json.dumps(dict(success=True))
+                return jsonify(dict(success=True))
             except RunwayError as err:
                 err.print_exception()
-                return json.dumps(err.to_response()), err.code
+                return jsonify(err.to_response()), err.code
 
         @self.app.route('/setup', methods=['GET'])
         def setup_options_route():
-            return json.dumps(self.options)
+            return jsonify(self.options)
 
         @self.app.route('/<command_name>', methods=['POST'])
+        @validate_post_request_body_is_json
         def command_route(command_name):
             try:
                 try:
@@ -64,7 +108,7 @@ class RunwayModel(object):
                     raise UnknownCommandError(command_name)
                 inputs = self.commands[command_name]['inputs']
                 outputs = self.commands[command_name]['outputs']
-                input_dict = request.json
+                input_dict = get_json_or_none_if_invalid(request)
                 deserialized_inputs = {}
                 for inp in inputs:
                     name = inp.name
@@ -73,6 +117,7 @@ class RunwayModel(object):
                     value = input_dict[name] or getattr(inp, 'default', None)
                     deserialized_inputs[name] = inp.deserialize(value)
                 try:
+                    self.millis_last_command = timestamp_millis()
                     results = command_fn(self.model, deserialized_inputs)
                     if type(results) != dict:
                         name = outputs[0].name
@@ -86,11 +131,11 @@ class RunwayModel(object):
                 for out in outputs:
                     name = out.to_dict()['name']
                     serialized_outputs[name] = out.serialize(results[name])
-                return json.dumps(serialized_outputs).encode('utf8')
+                return jsonify(serialized_outputs)
 
             except RunwayError as err:
                 err.print_exception()
-                return json.dumps(err.to_response()), err.code
+                return jsonify(err.to_response()), err.code
 
         @self.app.route('/<command_name>', methods=['GET'])
         def usage_route(command_name):
@@ -99,10 +144,18 @@ class RunwayModel(object):
                     command = self.commands[command_name]
                 except KeyError:
                     raise UnknownCommandError(command_name)
-                return json.dumps(serialize_command(command))
+                return jsonify(serialize_command(command))
             except RunwayError as err:
                 err.print_exception()
-                return json.dumps(err.to_response())
+                return jsonify(err.to_response()), err.code
+
+    def millis_running(self):
+        if self.millis_run_started_at is None: return None
+        return timestamp_millis() - self.millis_run_started_at
+
+    def millis_since_last_command(self):
+        if self.millis_last_command is None: return None
+        return timestamp_millis() - self.millis_last_command
 
     def setup(self, decorated_fn=None, options=None):
         """This decorator is used to wrap your own ``setup()`` (or equivalent)
@@ -388,6 +441,9 @@ class RunwayModel(object):
         except RunwayError as err:
             err.print_exception()
             sys.exit(1)
+
+        # start the run started at millis timer even if we don't actually serve
+        self.millis_run_started_at = timestamp_millis()
         if no_serve:
             print('Not starting model server because "no_serve" directive is present.')
         else:
diff --git a/runway/utils.py b/runway/utils.py
index 486f197..ed28151 100644
--- a/runway/utils.py
+++ b/runway/utils.py
@@ -7,12 +7,13 @@ import os
 import functools
 import sys
 import gzip
+import datetime
 if sys.version_info[0] < 3:
     from cStringIO import StringIO as IO
 else:
     from io import BytesIO as IO
 import numpy as np
-from flask import after_this_request, request
+from flask import after_this_request, request, jsonify
 
 
 URL_REGEX = re.compile(
@@ -23,6 +24,19 @@ URL_REGEX = re.compile(
         r'(?::\d+)?' # optional port
         r'(?:/?|[/?]\S+)$', re.IGNORECASE)
 
+def validate_post_request_body_is_json(f):
+    @functools.wraps(f)
+    def wrapped(*args, **kwargs):
+        json = get_json_or_none_if_invalid(request)
+        if json is not None:
+            return f(*args, **kwargs)
+        else:
+            err_msg = 'The body of all POST requests must contain JSON'
+            return jsonify(dict(error=err_msg)), 400
+    return wrapped
+
+def get_json_or_none_if_invalid(request):
+    return request.get_json(force=True, silent=True)
 
 def serialize_command(cmd):
     ret = {}
@@ -96,3 +110,7 @@ def cast_to_obj(cls_or_obj):
     if inspect.isclass(cls_or_obj):
         return cls_or_obj()
     return cls_or_obj
+
+def timestamp_millis():
+    offset = datetime.datetime.utcnow() - datetime.datetime(1970, 1, 1)
+    return int(offset.total_seconds() * 1000)


Test patch:
diff --git a/tests/test_model.py b/tests/test_model.py
index b3b2a60..25cfd91 100644
--- a/tests/test_model.py
+++ b/tests/test_model.py
@@ -7,11 +7,14 @@ sys.path.insert(0, '.')
 import os
 import json
 import pytest
+from time import sleep
 from runway.model import RunwayModel
+from runway.__version__ import __version__ as model_sdk_version
 from runway.data_types import category, text, number, array, image, vector, file, any as any_type
 from runway.exceptions import *
-from utils import get_test_client
+from utils import get_test_client, get_manifest
 from deepdiff import DeepDiff
+from flask import abort
 
 os.environ['RW_NO_SERVE'] = '1'
 
@@ -23,6 +26,10 @@ def test_model_setup_and_command():
     closure = dict(setup_ran = False, command_ran = False)
 
     expected_manifest = {
+        'modelSDKVersion': model_sdk_version,
+        'millisRunning': None,
+        'millisSinceLastCommand': None,
+        'GPU': os.environ.get('GPU', False),
         'options': [{
             'type': 'category',
             'name': 'size',
@@ -66,13 +73,37 @@ def test_model_setup_and_command():
 
     client = get_test_client(rw)
 
-    # check the manifest via a GET /
+    response = client.get('/meta')
+    assert response.is_json
+
+    manifest = json.loads(response.data)
+
+    # unset millisRunning as we can't reliably predict this value.
+    # testing that it is an int should be good enough.
+    assert type(manifest['millisRunning']) == int
+    manifest['millisRunning'] = None
+
+    assert manifest == expected_manifest
+
+    # TEMPORARILY CHECK / PATH IN ADDITION TO /meta ----------------------------
+    # ... sorry for the gross dupe code ;)
     response = client.get('/')
+    assert response.is_json
+
     manifest = json.loads(response.data)
+
+    # unset millisRunning as we can't reliably predict this value.
+    # testing that it is an int should be good enough.
+    assert type(manifest['millisRunning']) == int
+    manifest['millisRunning'] = None
+
     assert manifest == expected_manifest
+    # --------------------------------------------------------------------------
 
     # check the input/output manifest for GET /test_command
     response = client.get('/test_command')
+    assert response.is_json
+
     command_manifest = json.loads(response.data)
     assert command_manifest == expected_manifest['commands'][0]
 
@@ -80,8 +111,14 @@ def test_model_setup_and_command():
         'input': 'test input'
     }
     response = client.post('/test_command', json=post_data)
+    assert response.is_json
     assert json.loads(response.data) == { 'output' : 100 }
 
+    # now that we've run a command lets make sure millis since last command is
+    # a number
+    manifest_after_command = get_manifest(client)
+    assert type(manifest_after_command['millisSinceLastCommand']) == int
+
     assert closure['command_ran'] == True
     assert closure['setup_ran'] == True
 
@@ -96,7 +133,8 @@ def test_model_healthcheck():
     rw.run(debug=True)
     client = get_test_client(rw)
     response = client.get('/healthcheck')
-    assert response.data == b'RUNNING'
+    assert response.is_json
+    assert response.json == { 'status': 'RUNNING' }
 
 def test_model_setup_no_arguments():
 
@@ -144,6 +182,50 @@ def test_model_options_missing():
         with pytest.raises(MissingOptionError):
             rw.run(debug=True)
 
+def test_setup_invalid_category():
+
+    rw = RunwayModel()
+    @rw.setup(options={'category': category(choices=['Starks', 'Lannisters'])})
+    def setup(opts):
+        pass
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.post('/setup', json={ 'category': 'Tyrells' })
+
+    assert response.status_code == 400
+    json_response = json.loads(response.data)
+    assert 'error' in json_response
+    # ensure the user is displayed an error that indicates the category option
+    # is problematic
+    assert 'Invalid argument: category' in json_response['error']
+    # ensure the user is displayed an error that indicates the problematic value
+    assert 'Tyrells' in json_response['error']
+
+def test_command_invalid_category():
+
+    rw = RunwayModel()
+    inputs = {'category': category(choices=['Starks', 'Lannisters'])}
+    outputs = {'reflect': text }
+    @rw.command('test_command', inputs=inputs, outputs=outputs)
+    def test_command(opts):
+        return opts['category']
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.post('/test_command', json={ 'category': 'Targaryen' })
+
+    assert response.status_code == 400
+    json_response = json.loads(response.data)
+    assert 'error' in json_response
+    # ensure the user is displayed an error that indicates the category option
+    # is problematic
+    assert 'Invalid argument: category' in json_response['error']
+    # ensure the user is displayed an error that indicates the problematic value
+    assert 'Targaryen' in json_response['error']
+
 def test_meta(capsys):
 
     rw = RunwayModel()
@@ -263,6 +345,213 @@ def test_meta(capsys):
 
     os.environ['RW_META'] = '0'
 
+def test_post_setup_json_no_mime_type():
+
+    rw = RunwayModel()
+
+    @rw.setup(options={'input': text})
+    def setup(opts):
+        pass
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.post('/setup', data='{"input": "test input"}')
+    assert response.is_json
+    assert json.loads(response.data) == { 'success': True }
+
+def test_post_setup_invalid_json_no_mime_type():
+
+    rw = RunwayModel()
+
+    @rw.setup(options={'input': text})
+    def setup(opts):
+        pass
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.post('/setup', data='{"input": test input"}')
+
+    assert response.is_json
+    assert response.status_code == 400
+
+    expect = { 'error': 'The body of all POST requests must contain JSON' }
+    assert json.loads(response.data) == expect
+
+
+def test_post_setup_json_mime_type():
+
+    rw = RunwayModel()
+
+    @rw.setup(options={'input': text})
+    def setup(opts):
+        pass
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.post('/setup', json={ 'input': 'test input' })
+    assert response.is_json
+    assert json.loads(response.data) == { 'success': True }
+
+def test_post_setup_form_encoding():
+
+    rw = RunwayModel()
+
+    @rw.setup(options={'input': text})
+    def setup(opts):
+        pass
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+
+    content_type='application/x-www-form-urlencoded'
+    response = client.post('/setup', data='input=test', content_type=content_type)
+
+    assert response.is_json
+    assert response.status_code == 400
+
+    expect = { 'error': 'The body of all POST requests must contain JSON' }
+    assert json.loads(response.data) == expect
+
+def test_post_command_json_no_mime_type():
+
+    rw = RunwayModel()
+
+    @rw.command('times_two', inputs={ 'input': number }, outputs={ 'output': number })
+    def times_two(model, args):
+        return args['input'] * 2
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.post('/times_two', data='{ "input": 5 }')
+    assert response.is_json
+    assert json.loads(response.data) == { 'output': 10 }
+
+def test_post_command_json_mime_type():
+
+    rw = RunwayModel()
+
+    @rw.command('times_two', inputs={ 'input': number }, outputs={ 'output': number })
+    def times_two(model, args):
+        return args['input'] * 2
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.post('/times_two', json={ 'input': 5 })
+    assert response.is_json
+    assert json.loads(response.data) == { 'output': 10 }
+
+def test_post_command_form_encoding():
+
+    rw = RunwayModel()
+
+    @rw.command('times_two', inputs={ 'input': number }, outputs={ 'output': number })
+    def times_two(model, args):
+        return args['input'] * 2
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+
+    content_type='application/x-www-form-urlencoded'
+    response = client.post('/times_two', data='input=5', content_type=content_type)
+    assert response.is_json
+    assert response.status_code == 400
+
+    expect = { 'error': 'The body of all POST requests must contain JSON' }
+    assert json.loads(response.data) == expect
+
+def test_405_method_not_allowed():
+
+    rw = RunwayModel()
+
+    @rw.setup(options={'input': text})
+    def setup(opts):
+        pass
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.put('/setup', json= { 'input': 'test input'})
+
+    assert response.is_json
+    assert response.status_code == 405
+    assert response.json == { 'error': 'Method not allowed.' }
+
+def test_404_not_found():
+
+    rw = RunwayModel()
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.get('/asfd')
+
+    assert response.is_json
+    assert response.status_code == 404
+
+def test_401_unauthorized():
+
+    rw = RunwayModel()
+
+    @rw.app.route('/test/unauthorized')
+    def unauthorized():
+        abort(401)
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.get('/test/unauthorized')
+
+    assert response.is_json
+    assert response.status_code == 401
+
+    expect = { 'error': 'Unauthorized (well... really unauthenticated but hey I didn\'t write the spec).' }
+    assert response.json == expect
+
+def test_403_forbidden():
+
+    rw = RunwayModel()
+
+    @rw.app.route('/test/forbidden')
+    def unauthorized():
+        abort(403)
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.get('/test/forbidden')
+
+    assert response.is_json
+    assert response.status_code == 403
+
+    expect = { 'error': 'Forbidden.' }
+    assert response.json == expect
+
+def test_500_internal_server_error():
+
+    rw = RunwayModel()
+
+    @rw.app.route('/test/internal_server_error')
+    def unauthorized():
+        abort(500)
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+    response = client.get('/test/internal_server_error')
+
+    assert response.is_json
+    assert response.status_code == 500
+
+    expect = { 'error': 'Internal server error.' }
+    assert response.json == expect
+
 def test_setup_error_setup_no_args():
 
     rw = RunwayModel()
@@ -300,4 +589,76 @@ def test_inference_error():
     rw.run(debug=True)
 
     response = client.post('test_command', json={ 'input': 5 })
+    assert response.is_json
     assert 'InferenceError' in str(response.data)
+
+def test_millis_since_run_increases_over_time():
+
+    rw = RunwayModel()
+    client = get_test_client(rw)
+    rw.run(debug=True)
+
+    last_time = get_manifest(client)['millisRunning']
+    assert type(last_time) == int
+    for i in range(3):
+        sleep(0.01)
+        millis_running = get_manifest(client)['millisRunning']
+        assert millis_running > last_time
+        last_time = millis_running
+
+def test_millis_since_last_command_resets_each_command():
+
+    rw = RunwayModel()
+
+    @rw.command('test_command', inputs={ 'input': number }, outputs = { 'output': text })
+    def test_command(model, inputs):
+        pass
+
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+
+    assert get_manifest(client)['millisSinceLastCommand'] is None
+    client.post('test_command', json={ 'input': 5 })
+
+    first_time = get_manifest(client)['millisSinceLastCommand']
+    assert type(first_time) == int
+
+    for i in range(5):
+        sleep(0.02)
+        millis_since_last_command = get_manifest(client)['millisSinceLastCommand']
+        assert millis_since_last_command > first_time
+        client.post('test_command', json={ 'input': 5 })
+        assert get_manifest(client)['millisSinceLastCommand'] < millis_since_last_command
+
+def test_gpu_in_manifest_no_env_set():
+
+    rw = RunwayModel()
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+
+    if os.environ.get('GPU') is not None:
+        del os.environ['GPU']
+
+    assert get_manifest(client)['GPU'] == False
+
+def test_gpu_in_manifest_gpu_env_true():
+
+    rw = RunwayModel()
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+
+    os.environ['GPU'] = '1'
+    assert get_manifest(client)['GPU'] == True
+
+def test_gpu_in_manifest_gpu_env_false():
+
+    rw = RunwayModel()
+    rw.run(debug=True)
+
+    client = get_test_client(rw)
+
+    os.environ['GPU'] = '0'
+    assert get_manifest(client)['GPU'] == False
diff --git a/tests/utils.py b/tests/utils.py
index 5b48430..c2df38a 100644
--- a/tests/utils.py
+++ b/tests/utils.py
@@ -1,6 +1,11 @@
+import json
 from runway import RunwayModel
 
 def get_test_client(rw_model):
     assert isinstance(rw_model, RunwayModel)
     rw_model.app.config['TESTING'] = True
     return rw_model.app.test_client()
+
+def get_manifest(client):
+    response = client.get('/')
+    return json.loads(response.data)