Repository: boto/botocore
Pull request #741 (2015-12-15 21:37:05)

Support stubbing out operations.
This will allow customers to stub out operations, providing their own responses so that they don't have to hit an endpoint to test their code. The interface for this is the `Stubber` class, which will manage the response queue and and handle validating the provided response stubs.

Usage example:

```python
import botocore.session
import botocore.stub
import datetime


s3_client = botocore.session.get_session().create_client('s3')
stubber = botocore.stub.Stubber(s3_client)
list_objects_response = {
   'IsTruncated': False,
   'EncodingType': 'url',
   'Marker': '',
   'Prefix': '',
   'Name': 'test-bucket',
   'Contents': [
      {
         'Size': 123,
         'Owner': {
            'DisplayName': 'display_name',
            'ID': '123abc'
         },
         'Key': 'file.txt',
         'StorageClass': 'STANDARD',
         'ETag': '"abc123"',
         'LastModified': datetime.datetime(2015, 12, 7, 18, 5, 52)
      },
      {
         'Size': 456,
         'Owner': {
            'DisplayName': 'display_name',
            'ID': '123abc'
         },
         'Key': 'file.py',
         'StorageClass': 'STANDARD',
         'ETag': '"abc123"',
         'LastModified': datetime.datetime(2015, 12, 7, 17, 38, 49)
      }
   ],
   'MaxKeys': 1000
}
stubber.add_response('list_objects', list_objects_response)
stubber.activate()
response = s3_client.list_objects(Bucket='test-bucket')
assert(response == list_objects_response)
```

cc @rayluo @kyleknap @mtdowling @jamesls 

garnaat: Nice!  +1
jamesls: Awesome.  I'd like to look at this in more depth, but I'm curious if there's a reason using an existing event like `before-call` was insufficient?  Are there additional capabilities you get from hooking deeper down into endpoint vs. `before-call`?  Just want to ensure there's not a pre-existing event we can use before we go down the path of adding another event.  I'm also hesitant to expose the requests's `Response` class to users.
mtdowling: Looks good. Some high level feedback:

1. With these types of abstractions, it's often nice to be able to get the requests that were sent through the stubber class. Do you think it would be useful to have some way to be able to ask the stubber what requests it stubbed, and then in turn some way to clear the history? In the PHP SDK, we just track the last request and last response -- https://github.com/aws/aws-sdk-php/blob/master/src/MockHandler.php
2. What do you think about adding support for stubbing errors so that when invoked, the class raises a pre-created exception?
3. Do you think there would be any value in allowing different strategies for determining which response to use when stubbing? For example, one could just pop from a queue (what you've implemented), while another strategy might be to just call a function where a user could do various things in a test each time a call is stubbed. Another use case might be a strategy that has some predetermined expectation on what requests will be sent and the response associated with them. We didn't include this type of functionality in the PHP SDK because it is really easy to create anonymous functions that act as the HTTP client.
4. Instead of my third bullet-point (and possibly in addition to...), what about adding functions to the queue as well? When a function is popped of the queue the function is invoked with the request params and the return value is the result of the function (or an exception is raised)? Using functions might be required for raising exceptions if the exception cannot be created in advance (because it needs a reference the request or something).
garnaat: I did something similar to this is https://github.com/garnaat/placebo but I ended up having to rely on the client's private ``_endpoint`` attribute to mock out the call and get my data in there.  Maybe there is another way but I couldn't find it.  Something like this would make things much easier.
JordonPhillips: @jamesls There are several events in `Endpoint` that `before-call` would trample if that were used. I didn't want to interfere with anything else that we (or a customer) might be doing. Further, intercepting at that point allows for the possibility of testing out retry logic. As far as `Response` is concerned, how about letting them pass in the status code and reason of the response, and we create it for them? Or maybe even a `ClientError`?

@mtdowling 

1. I would be +1 on something like this. We could potentially have a map of requests to responses and the user would pass in the history size they want on creation of the `Stubber` class (probably defaulting to 1).
2. Right now you can stub errors by passing in an error response. I think letting them pass a status code / reason would probably suffice though.
3. I think these would be cases for subclasses. We could encourage this by making `Stubber` an abstract class and providing some number of implementations.
4. I think this could potentially be useful, but it skirts around validation. Additionally, they would need to provide this for every single call where they want their function to run, and I'm not sure that's a wonderful interface.

@garnaat in the current state, there's not really a better way to go about it since you can't short-circuit `Endpoint`. With this PR you would be able to short-circuit with an event handler.
JordonPhillips: After much discussion, this is going to be changed and broken up a bit. The first part will be addressed in #743. We will then build on top of that.
JordonPhillips: The tests on this PR will not pass until #743 is merged and I can safely rebase.
JordonPhillips: It is now safe
jamesls: Overall I think the API's nice and easy to use.  I like the validation of responses.  I think customers are going to find this really useful.  I have some small comments/question above but overall looks nice.
JordonPhillips: :+1: 
kyleknap: Awesome. Had that one suggestion that we could add later, but otherwise looks good. :ship:

Patch:
diff --git a/botocore/endpoint.py b/botocore/endpoint.py
index 623a4a40e..ebd8a5ab6 100644
--- a/botocore/endpoint.py
+++ b/botocore/endpoint.py
@@ -193,8 +193,8 @@ class Endpoint(object):
                                                  operation_model)
         parser = self._response_parser_factory.create_parser(
             operation_model.metadata['protocol'])
-        return ((http_response, parser.parse(response_dict,
-                                             operation_model.output_shape)),
+        return ((http_response,
+                 parser.parse(response_dict, operation_model.output_shape)),
                 None)
 
     def _looks_like_dns_error(self, e):
diff --git a/botocore/exceptions.py b/botocore/exceptions.py
index 48cedc9dc..0c555832d 100644
--- a/botocore/exceptions.py
+++ b/botocore/exceptions.py
@@ -353,6 +353,10 @@ class InvalidS3AddressingStyleError(BotoCoreError):
     )
 
 
+class StubResponseError(BotoCoreError):
+    fmt = 'Error getting response stub for operation {operation_name}: {reason}'
+
+
 class InvalidConfigError(BotoCoreError):
     fmt = '{error_msg}'
 
diff --git a/botocore/stub.py b/botocore/stub.py
new file mode 100644
index 000000000..47f4f282a
--- /dev/null
+++ b/botocore/stub.py
@@ -0,0 +1,201 @@
+# Copyright 2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+# http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+import copy
+from collections import deque
+from botocore.validate import validate_parameters
+from botocore.exceptions import ParamValidationError, StubResponseError
+from botocore.vendored.requests.models import Response
+
+
+class Stubber(object):
+    """
+    This class will allow you to stub out requests so you don't have to hit
+    an endpoint to write tests. Responses are returned first in, first out.
+    If operations are called out of order, or are called with no remaining
+    queued responses, an error will be raised.
+
+    Example::
+        import datetime
+        import botocore.session
+        from botocore.stub import Stubber
+
+
+        s3 = botocore.session.get_session().create_client('s3')
+        stubber = Stubber(s3)
+
+        response = {
+            'IsTruncated': False,
+            'Name': 'test-bucket',
+            'MaxKeys': 1000, 'Prefix': '',
+            'Contents': [{
+                'Key': 'test.txt',
+                'ETag': '"abc123"',
+                'StorageClass': 'STANDARD',
+                'LastModified': datetime.datetime(2016, 1, 20, 22, 9),
+                'Owner': {'ID': 'abc123', 'DisplayName': 'myname'},
+                'Size': 14814
+            }],
+            'EncodingType': 'url',
+            'ResponseMetadata': {
+                'RequestId': 'abc123',
+                'HTTPStatusCode': 200,
+                'HostId': 'abc123'
+            },
+            'Marker': ''
+        }
+
+        stubber.add_response('list_objects', response)
+        stubber.activate()
+
+        service_response = s3.list_objects(Bucket='test-bucket')
+        assert service_response == response
+    """
+    def __init__(self, client):
+        """
+        :param client: The client to add your stubs to.
+        """
+        self.client = client
+        self._event_id = 'boto_stubber'
+        self._queue = deque()
+
+    def activate(self):
+        """
+        Activates the stubber on the client
+        """
+        self.client.meta.events.register(
+                'before-call.*.*',
+                self._get_response_handler,
+                unique_id=self._event_id)
+
+    def deactivate(self):
+        """
+        Deactivates the stubber on the client
+        """
+        self.client.meta.events.unregister(
+                'before-call.*.*',
+                self._get_response_handler,
+                unique_id=self._event_id)
+
+    def add_response(self, method, service_response):
+        """
+        Adds a service response to the response queue. This will be validated
+        against the service model to ensure correctness. It should be noted,
+        however, that while missing attributes are often considered correct,
+        your code may not function properly if you leave them out. Therefore
+        you should always fill in every value you see in a typical response for
+        your particular request.
+
+        :param method: The name of the client method to stub.
+        :type method: str
+
+        :param service_response: A dict response stub. Provided parameters will
+            be validated against the service model.
+        :type service_response: dict
+        """
+        self._add_response(method, service_response)
+
+    def _add_response(self, method, service_response, http_response=None):
+        if not hasattr(self.client, method):
+            raise ValueError(
+                "Client %s does not have method: %s"
+                % (self.client.meta.service_model.service_name, method))
+
+        if http_response is None:
+            http_response = Response()
+            http_response.status_code = 200
+            http_response.reason = 'OK'
+
+        operation_name = self.client.meta.method_to_api_mapping.get(method)
+        self._validate_response(operation_name, service_response)
+
+        response = (operation_name, (http_response, service_response))
+        self._queue.append(response)
+
+    def add_client_error(self, method, service_error_code='',
+                         service_message='', http_status_code=400):
+        """
+        Adds a ClientError to the response queue.
+
+        :param method: The name of the service method to return the error on.
+        :type method: str
+
+        :param service_error_code: The service error code to return,
+                                   e.g. NoSuchBucket
+        :type service_error_code: str
+
+        :param service_message: The service message to return, e.g.
+                        'The specified bucket does not exist.'
+        :type service_message: str
+
+        :param http_status_code: The HTTP status code to return, e.g. 404, etc
+        :type http_status_code: int
+        """
+        http_response = Response()
+        http_response.status_code = http_status_code
+
+        # We don't look to the model to build this because the caller would need
+        # to know the details of what the HTTP body would need to look like.
+        parsed_response = {
+            'ResponseMetadata': {'HTTPStatusCode': http_status_code},
+            'Error': {
+                'Message': service_message,
+                'Code': service_error_code
+            }
+        }
+
+        operation_name = self.client.meta.method_to_api_mapping.get(method)
+        response = (operation_name, (http_response, parsed_response))
+        self._queue.append(response)
+
+    def assert_no_pending_responses(self):
+        """
+        Asserts that all expected calls were made.
+        """
+        remaining = len(self._queue)
+        if remaining != 0:
+            raise AssertionError("%d responses remaining in queue." % remaining)
+
+    def _get_response_handler(self, model, params, **kwargs):
+        if not self._queue:
+            raise StubResponseError(
+                operation_name=model.name,
+                reason='Unexpected API Call: called with parameters %s' %
+                       params)
+
+        name, response = self._queue.popleft()
+        if name != model.name:
+            raise StubResponseError(
+                operation_name=model.name,
+                reason='Operation mismatch: found response for %s.' % name)
+
+        return response
+
+    def _validate_response(self, operation_name, service_response):
+        service_model = self.client.meta.service_model
+        operation_model = service_model.operation_model(operation_name)
+        output_shape = operation_model.output_shape
+
+        # Remove ResponseMetadata so that the validator doesn't attempt to
+        # perform validation on it.
+        response = service_response
+        if 'ResponseMetadata' in response:
+            response = copy.copy(service_response)
+            del response['ResponseMetadata']
+
+        if output_shape is not None:
+            validate_parameters(response, output_shape)
+        elif response:
+            # If the output shape is None, that means the response should be
+            # empty apart from ResponseMetadata
+            raise ParamValidationError(
+                report="Service response should only contain ResponseMetadata.")


Test patch:
diff --git a/tests/functional/test_stub.py b/tests/functional/test_stub.py
new file mode 100644
index 000000000..c11571a74
--- /dev/null
+++ b/tests/functional/test_stub.py
@@ -0,0 +1,60 @@
+# Copyright 2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+# http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+
+from tests import unittest
+
+import botocore
+import botocore.session
+from botocore.stub import Stubber
+from botocore.exceptions import StubResponseError, ClientError
+import botocore.client
+import botocore.retryhandler
+import botocore.translate
+
+
+class TestStubber(unittest.TestCase):
+    def setUp(self):
+        session = botocore.session.get_session()
+        config = botocore.client.Config(signature_version=botocore.UNSIGNED)
+        self.client = session.create_client('s3', config=config)
+
+        self.stubber = Stubber(self.client)
+
+    def test_stubber_returns_response(self):
+        service_response = {'ResponseMetadata': {'foo': 'bar'}}
+        self.stubber.add_response('list_objects', service_response)
+        self.stubber.activate()
+        response = self.client.list_objects(Bucket='foo')
+        self.assertEqual(response, service_response)
+
+    def test_activated_stubber_errors_with_no_registered_stubs(self):
+        self.stubber.activate()
+        with self.assertRaises(StubResponseError):
+            self.client.list_objects(Bucket='foo')
+
+    def test_stubber_errors_when_stubs_are_used_up(self):
+        self.stubber.add_response('list_objects', {})
+        self.stubber.activate()
+        self.client.list_objects(Bucket='foo')
+
+        with self.assertRaises(StubResponseError):
+            self.client.list_objects(Bucket='foo')
+
+    def test_client_error_response(self):
+        error_code = "AccessDenied"
+        error_message = "Access Denied"
+        self.stubber.add_client_error('list_objects', error_code, error_message)
+        self.stubber.activate()
+
+        with self.assertRaises(ClientError):
+            self.client.list_objects(Bucket='foo')
diff --git a/tests/unit/test_stub.py b/tests/unit/test_stub.py
new file mode 100644
index 000000000..9c18da9d9
--- /dev/null
+++ b/tests/unit/test_stub.py
@@ -0,0 +1,144 @@
+# Copyright 2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+# http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+
+from tests import unittest
+import mock
+
+from botocore.stub import Stubber
+from botocore.exceptions import ParamValidationError, StubResponseError
+from botocore.model import ServiceModel
+from botocore import hooks
+
+
+class TestStubber(unittest.TestCase):
+    def setUp(self):
+        self.event_emitter = hooks.HierarchicalEmitter()
+        self.client = mock.Mock()
+        self.client.meta.events = self.event_emitter
+        self.client.meta.method_to_api_mapping.get.return_value = 'foo'
+        self.stubber = Stubber(self.client)
+        self.validate_parameters_mock = mock.Mock()
+        self.validate_parameters_patch = mock.patch(
+            'botocore.stub.validate_parameters', self.validate_parameters_mock)
+        self.validate_parameters_patch.start()
+
+    def tearDown(self):
+        self.validate_parameters_patch.stop()
+
+    def emit_get_response_event(self, model=None, request_dict=None,
+                                signer=None, context=None):
+        if model is None:
+            model = mock.Mock()
+            model.name = 'foo'
+
+        handler, response = self.event_emitter.emit_until_response(
+            event_name='before-call.myservice.foo', model=model,
+            params=request_dict, request_signer=signer, context=context)
+
+        return response
+
+    def test_stubber_registers_events(self):
+        self.event_emitter = mock.Mock()
+        self.client.meta.events = self.event_emitter
+        self.stubber.activate()
+        self.assertTrue(self.event_emitter.register.called)
+
+    def test_stubber_unregisters_events(self):
+        self.event_emitter = mock.Mock()
+        self.client.meta.events = self.event_emitter
+        self.stubber.activate()
+        self.stubber.deactivate()
+        self.assertTrue(self.event_emitter.unregister.called)
+
+    def test_add_response(self):
+        response = {'foo': 'bar'}
+        self.stubber.add_response('foo', response)
+
+        with self.assertRaises(AssertionError):
+            self.stubber.assert_no_pending_responses()
+
+    def test_add_response_fails_when_missing_client_method(self):
+        del self.client.foo
+        with self.assertRaises(ValueError):
+            self.stubber.add_response('foo', {})
+
+    def test_validates_service_response(self):
+        self.stubber.add_response('foo', {})
+        self.assertTrue(self.validate_parameters_mock.called)
+
+    def test_validate_ignores_response_metadata(self):
+        service_response = {'ResponseMetadata': {'foo': 'bar'}}
+        service_model = ServiceModel({
+            'documentation': '',
+            'operations': {
+                'foo': {
+                    'name': 'foo',
+                    'input': {'shape': 'StringShape'},
+                    'output': {'shape': 'StringShape'}
+                }
+            },
+            'shapes': {
+                'StringShape': {'type': 'string'}
+            }
+        })
+        op_name = service_model.operation_names[0]
+        output_shape = service_model.operation_model(op_name).output_shape
+
+        self.client.meta.service_model = service_model
+        self.stubber.add_response('TestOperation', service_response)
+        self.validate_parameters_mock.assert_called_with(
+            {}, output_shape)
+
+        # Make sure service response hasn't been mutated
+        self.assertEqual(service_response, {'ResponseMetadata': {'foo': 'bar'}})
+
+    def test_validates_on_empty_output_shape(self):
+        service_model = ServiceModel({
+            'documentation': '',
+            'operations': {
+                'foo': {
+                    'name': 'foo'
+                }
+            }
+        })
+        self.client.meta.service_model = service_model
+
+        with self.assertRaises(ParamValidationError):
+            self.stubber.add_response('TestOperation', {'foo': 'bar'})
+
+    def test_get_response(self):
+        service_response = {'bar': 'baz'}
+        self.stubber.add_response('foo', service_response)
+        self.stubber.activate()
+        response = self.emit_get_response_event()
+        self.assertEqual(response[1], service_response)
+        self.assertEqual(response[0].status_code, 200)
+
+    def test_get_client_error_response(self):
+        error_code = "foo"
+        service_message = "bar"
+        self.stubber.add_client_error('foo', error_code, service_message)
+        self.stubber.activate()
+        response = self.emit_get_response_event()
+        self.assertEqual(response[1]['Error']['Message'], service_message)
+        self.assertEqual(response[1]['Error']['Code'], error_code)
+
+    def test_get_response_errors_with_no_stubs(self):
+        self.stubber.activate()
+        with self.assertRaises(StubResponseError):
+            self.emit_get_response_event()
+
+    def test_assert_no_responses_remaining(self):
+        self.stubber.add_response('foo', {})
+        with self.assertRaises(AssertionError):
+            self.stubber.assert_no_pending_responses()