Package: lobotomy
LobotomyInstallationTl;dr UsageUsageConfiguration FilesTest PatchingYAML IO Modifiers!lobotomy.to_json!lobotomy.inject_stringSpecial Casess3.download_fileobjCommand Line InterfaceAdvanced UsageKey PrefixesPatching TargetsSession ConfigurationClient OverridesError HandlingCallable ResponsesThelo•boto•mylibrary allows one to mock the low-level boto3
client libraries efficiently, especially in more complex scenario testing
situations, using configuration-based response definitions. The benefit is a
separation of the configuration from the test execution, which cleans up the
test invocation process.Installationlobotomy is available on pypi and installable via pip:$pipinstalllobotomyor via poetry as a development dependency in a project:$poetryaddlobotomy-DTl;dr UsageCreate a configuration file in YAML, TOML, or JSON format with a rootclientskey. Beneath that key add the desired client calls mocked responses by service
as shown here for asession.client('s3').get_object()mocked response:clients:s3:get_object:Body:'ThecontentsofmyS3file.'LastModified:'2020-12-01T01:02:03Z'Then in the test notice that the lobotomy patching process handles the
rest, including casting the configuration values specified in the call
above into the more complex data types returned for the specific call.importlobotomyimportpathlibimportboto3importdatetimemy_directory=pathlib.Path(__file__).parent@lobotomy.patch(my_directory.joinpath("test_lobotomy.yaml"))deftest_lobotomy(lobotomized:lobotomy.Lobotomy):"""Should return the mocked get_object response generated from theconfiguration data specified in the lobotomy patch above. By defaultthe patch(...) applies to the boto3.Session object, so callingboto3.Session() will create a lobotomy.Session instead of a normalboto3.Session. From there the low-level client interface is designedto match normal usage."""s3_client=boto3.Session().client("s3")# Lobotomy will validate that you have specified the required keys# in your request, so Bucket and Key have to be supplied here even though# they are not meaningful values in this particular test scenario.response=s3_client.get_object(Bucket="foo",Key="bar")expected=b"The contents of my S3 file."assertresponse["Body"].read()==expected,"""Expected the mocked response body data to be returned as aStreamingBody object with blob/bytes contents. The lobotomylibrary introspects boto to properly convert the string bodyvalue in the configuration file into the expected return formatfor the particular call."""expected=datetime.datetime(2020,12,1,1,2,3,0,datetime.timezone.utc)assertresponse["LastModified"]==expected,"""Expected the mocked response last modified value to be a timezone-awaredatetime value generated from the string timestamp value in theconfiguration file to match how it would be returned by boto inan actual response."""call=lobotomized.get_service_call("s3","get_object")assertcall.request["Bucket"]=="foo","""Expected the s3.get_object method call arguments to have specified the bucketas "foo"."""assertcall.request["Key"]=="bar","""Expected the s3.get_object method call argumets to have specified the key as"bar"."""UsageConfiguration FilesTest scenarios can be written in YAML, TOML, or JSON formats. YAML or TOML are
recommended unless copying output responses from JSON calls is easier for a
given use-case. In the following example, we'll define the calls using YAML:clients:sts:get_caller_identity:Account:'987654321's3:get_object:Body:'ThecontentsofmyS3file.'LastModified:'2020-11-01T12:23:34Z'All call responses are stored within the rootclientsattribute with service
names as sub-attributes and then the service method call responses defined
beneath the service name attributes. Multiple services and multiple methods
per service are specified in this way by the hierarchical key lists.If the contents of a method response definition are not a list, the same
response will be returned for each call to that client method during the
test. Specifying a list of responses will alter the behavior such that each
successive call will iterate through that list of configured responses.clients:s3:get_object:-Body:'ThecontentsofmyS3file.'LastModified:'2020-11-01T12:23:34Z'-Body:'AnotherS3file.'LastModified:'2020-11-03T12:23:34Z'The lobotomy library dynamically inspects boto for its response structure
and data types, such that they match the types for the normal boto response
for each client method call. In the case of thesession.client('s3').get_object()calls above, theBodywould return
aStreamingBodyobject with the string converted to bytes to match the
normal output. Similarly, theLastModifiedwould be converted to a
timezone-aware datetime object by lobotomy as well. This makes it easy
to specify primitive data types in the configuration file that are transformed
into their more complex counterparts when returned within the execution of
the actual code.Test PatchingOnce the configuration file has been specified, it is used within a test
via alobotomy.patchas shown below.importlobotomyimportpathlibimportboto3importdatetimemy_directory=pathlib.Path(__file__).parent@lobotomy.patch(my_directory.joinpath("test_lobotomy.yaml"))deftest_lobotomy(lobotomized:lobotomy.Lobotomy):"""Should return the mocked get_object response generated from theconfiguration data specified in the lobotomy patch above. By defaultthe patch(...) applies to the boto3.Session object, so callingboto3.Session() will create a lobotomy.Session instead of a normalboto3.Session. From there the low-level client interface is designedto match normal usage."""s3_client=boto3.Session().client("s3")# Lobotomy will validate that you have specified the required keys# in your request, so Bucket and Key have to be supplied here even though# they are not meaningful values in this particular test scenario.response=s3_client.get_object(Bucket="foo",Key="bar")expected=b"The contents of my S3 file."assertresponse['Body'].read()==expected,"""Expect the mocked response body data to be returned as aStreamingBody object with blob/bytes contents. The lobotomylibrary introspects boto to properly convert the string bodyvalue in the configuration file into the expected return formatfor the particular call."""expected=datetime.datetime(2020,12,1,1,2,3,0,datetime.timezone.utc)assertresponse['LastModified']==expected,"""Expect the mocked response last modified value to be a timezeon-awaredatetime value generated from the string timestamp value in theconfiguration file to match how it would be returned by boto inan actual response."""The patching process replaces theboto3.Sessionclass with alobotomy.Lobotomyobject that contains the loaded configuration data.
When patched in this fashion,boto3.Session()calls will actually belobotomy.Lobotomy()calls that returnlobotomy.Sessionobjects. These
sessions have the interface of theboto3.Sessionobject, but behave in
a way such that client responses are returned from the configuration data
instead of through interactivity with AWS.For simple cases with little configuration, it is also possible to patch
data stored directly within the Python code. The above test could be rewritten
in this way as:importlobotomyconfiguration={"clients":{"s3":{"get_object":[{"Body":"The contents of my S3 file.","LastModified":"2020-11-01T12:23:34Z",},{"Body":"Another S3 file.","LastModified":"2020-11-03T12:23:34Z",},],},},}@lobotomy.patch(data=configuration)deftest_lobotomy(lobotomized:lobotomy.Lobotomy):"""..."""Although one of the benefits of lobotomy is the ability to streamline the
tests files by reducing the response configuration, which can be a bit
verbose inside Python files and that is the highly recommended approach.A third option for simpler cases is to use theLobotomy.add_call()method
to register calls and responses.importlobotomy@lobotomy.patch()deftest_lobotomy(lobotomized:lobotomy.Lobotomy):"""..."""lobotomized.add_call(service_name="s3",method_name="get_object",response={"Body":"The contents of my S3 file.","LastModified":"2020-11-01T12:23:34Z",},)lobotomized.add_call(service_name="s3",method_name="get_object",response={"Body":"Another S3 file.","LastModified":"2020-11-03T12:23:34Z",},)In the case above no data or path was supplied to thelobotomy.patch()and instead
call responses were registered within the test function itself. The response argument
in theLobotomy.add_call()method is optional. If omitted, a default one will be
created instead in the same fashion as one would be created via the CLI (see below).Note that it is also possible to mix loading data and adding calls within the test
function.YAML IO ModifiersWhen using YAML files, lobotomy comes with custom YAML classes that can provide
even more flexibility and ease in defining data. The following are the available
modifiers and how to use them:to_jsonThis modifier will convert YAML data into a JSON string in the creation of the
response, which makes it easier to represent complex JSON data in the scenario
data.clients:secretsmanager:get_secret_value:SecretString:!lobotomy.to_jsonfirst:first_valuesecond:second_valueIn this example the!lobotomy.to_jsonYAML modifier instructs the lobotomy to
converts the object data beneath theSecretStringattribute into a JSON string
as part of the response object. In this case then in the associated Python test:importboto3importlobotomyimportjson@lobotomy.patch(path="scenario.yaml")deftest_showing_to_json(lobotomized:lobotomy.Lobotomy):"""Should expect a JSON string for the 'SecretString' value."""client=boto3.Session().client("secretsmanager")response=client.get_secret_value(SecretId="fake")expected={"first":"first_value","second":"second_value"}assertexpected==json.loads(response["SecretValue"])the value is returned as the expected JSON string.inject_stringThis modifier is used to inject the string contents of another file into the value
of the associated attribute.clients:s3:get_object:Body:!lobotomy.inject_string'./body.txt'Here the!lobotomy.inject_stringYAML modifier instructs lobotomy to load the
contents of the external file./body.txtinto theBodyattribute value where
the external file path is defined relative to the defining YAML file.So in this case, if there's abody.txtfile with the contentsHello lobotomy!,
the Python test would find this in reading the body:importboto3importlobotomy@lobotomy.patch(path="scenario.yaml")deftest_showing_inject_string(lobotomized:lobotomy.Lobotomy):"""Should expect the body.txt to be injected into the Body response attribute."""client=boto3.Session().client("s3")response=client.get_object(Bucket="fake",Key="fake")assertresponse["Body"].read()==b"Hello lobotomy!"Special CasesThe following are special case operations that are handled slightly differently than
the regular lobotomy behaviors.s3.download_fileobjThis is a special action that is not actually included in the botocore API. It's a
convenience wrapper that has the special behavior of written data to a byte-buffer
object instead of returning a value. To handle that, the"Fileobj"response should
be populated with either a string or a bytes object. Lobotomy will internally handle
writing that to the specified buffer like this:importboto3importlobotomyimportio@lobotomy.Patch()deftest_s3_download_fileobj(lobotomized:lobotomy.Lobotomy):"""Should write s3.download_fileobj outputs to the Fileobj argument buffer."""lobotomized.add_call("s3","download_fileobj",{"Fileobj":"Hello World!"})client=boto3.Session().client("s3")data=io.BytesIO()client.download_fileobj(Bucket="bar",Key="baz",Fileobj=data)call=lobotomized.get_service_calls("s3","download_fileobj")[0]assertcall.request["Fileobj"]==dataassertcall.request["Bucket"]=="bar"assertcall.request["Key"]=="baz"assertdata.getvalue().decode()=="Hello World!"Command Line InterfaceThe lobotomy library also has a command line interface to help streamline
the process of creating configuration files. The CLI has anaddcommand
that can be used to auto-generate method call response configurations to
a new or existing configuration file. The values are meant to be replaced
and unused keys to be removed to streamline for testing, but it helps a lot
to get the full structure of the response in place and work from there instead
of having to look it up yourself.For example, creating the file:clients:sts:get_caller_identity:Account:'987654321's3:get_object:-Body:'ThecontentsofmyS3file.'LastModified:'2020-11-01T12:23:34Z'-Body:'AnotherS3file.'LastModified:'2020-11-03T12:23:34Z'could be done first through the CLI commands:$lobotomyaddsts.get_caller_identityexample.yamlAfter that command is executed, theexample.yamlfile will be
created and populated initially with:clients:sts:get_caller_identity:Account:'...'Arn:'...'UserId:'...'Notice the values are placeholders. We can adjust the values to what we want
and remove the unnecessary keys for our particular case such that the file
contents are then:clients:sts:get_caller_identity:Account:'987654321'Next add the firsts3.get_objectcall:$lobotomyadds3.get_objectexample.yamlThe configuration file now looks like:clients:s3:get_object:AcceptRanges:'...'Body:'...'CacheControl:'...'ContentDisposition:'...'ContentEncoding:'...'ContentLanguage:'...'ContentLength:1ContentRange:'...'ContentType:'...'DeleteMarker:nullETag:'...'Expiration:'...'Expires:'2020-11-04T14:37:18.042821Z'LastModified:'2020-11-04T14:37:18.042821Z'Metadata:nullMissingMeta:1ObjectLockLegalHoldStatus:'...'ObjectLockMode:'...'ObjectLockRetainUntilDate:'2020-11-04T14:37:18.042821Z'PartsCount:1ReplicationStatus:'...'RequestCharged:'...'Restore:'...'SSECustomerAlgorithm:'...'SSECustomerKeyMD5:'...'SSEKMSKeyId:'...'ServerSideEncryption:'...'StorageClass:'...'TagCount:1VersionId:'...'WebsiteRedirectLocation:'...'sts:get_caller_identity:Account:'987654321'Of course, this is a simple case because we don't need much of the response
structure in our simplified use-case, but hopefully you can see the value
of being able to add the response structure so easily for more complex cases.
Once again, the new call is adjusted to fit our particular needs:clients:s3:get_object:Body:'ThecontentsofmyS3file.'LastModified:'2020-11-01T12:23:34Z'sts:get_caller_identity:Account:'987654321'Adding the seconds3.get_objectcall is identical:$lobotomyadds3.get_objectexample.yamlHowever, lobotomy notices the existing call there and so converts theget_objectresponse configuration to a list of responses for you:clients:s3:Body:'ThecontentsofmyS3file.'LastModified:'2020-11-01T12:23:34Z'get_object:AcceptRanges:'...'Body:'...'CacheControl:'...'ContentDisposition:'...'ContentEncoding:'...'ContentLanguage:'...'ContentLength:1ContentRange:'...'ContentType:'...'DeleteMarker:nullETag:'...'Expiration:'...'Expires:'2020-11-04T14:42:51.077364Z'LastModified:'2020-11-04T14:42:51.077364Z'Metadata:nullMissingMeta:1ObjectLockLegalHoldStatus:'...'ObjectLockMode:'...'ObjectLockRetainUntilDate:'2020-11-04T14:42:51.077364Z'PartsCount:1ReplicationStatus:'...'RequestCharged:'...'Restore:'...'SSECustomerAlgorithm:'...'SSECustomerKeyMD5:'...'SSEKMSKeyId:'...'ServerSideEncryption:'...'StorageClass:'...'TagCount:1VersionId:'...'WebsiteRedirectLocation:'...'sts:get_caller_identity:Account:'987654321'Finally, edit the new call response configuration and we end up with the
configuration we were looking for:clients:sts:get_caller_identity:Account:'987654321's3:get_object:-Body:'ThecontentsofmyS3file.'LastModified:'2020-11-01T12:23:34Z'-Body:'AnotherS3file.'LastModified:'2020-11-03T12:23:34Z'Advanced UsageKey PrefixesBy default configuration files are rooted at theclientskey within the
file. However, it is possible to specify a different root key prefix, which
is useful when co-locating lobotomy test configuration with other test
configuration data in the same file, or when co-locating multiple lobotomy
test configurations within the same file. To achieve that a prefix must be
specified during patching.As an example, consider the configuration file:lobotomy:test_a:clients:sts:get_caller_identity:Account:'987654321'test_b:clients:sts:get_caller_identity:Account:'123456678'UserId:'AIFASDJWISJAVHXME'In this case the prefixes arelobotomy.test_aandlobotomy.test_b.
To use these in a test theprefixmust be specified in the patch:importpathlibimportlobotomyconfig_path=pathlib.Path(__file__).parent.joinpath("validation.yaml")@lobotomy.patch(config_path,prefix="lobotomy.test_a")deftest_a(lobotomized:lobotomy.Lobotomy):"""..."""@lobotomy.patch(config_path,prefix="lobotomy.test_b")deftest_b(lobotomized:lobotomy.Lobotomy):"""..."""The prefix can be specified as a.delimited string, or as a list/tuple.
The list/tuple is needed if the keys themselves contains..Patching TargetsBy default, lobotomy will patchboto3.Session. There are scenarios where
a different patch would be desired either to limit the scope of the patch or
to patch another library wrapping theboto3.Sessioncall. In those cases,
specify the patch path argument:importpathlibimportlobotomyconfig_path=pathlib.Path(__file__).parent.joinpath("test.yaml")@lobotomy.patch(config_path,patch_path="something.else.Session")deftest_another_patch_path(lobotomized:lobotomy.Lobotomy):"""..."""Session ConfigurationIn addition to the clients configurations described above, it is also possible
to configure the session values as well with thesession:attribute. The
available configuration settings for the session are:session:profile_name:some-profileregion_name:us-west-2available_profiles:-some-profile-some-other-profilecredentials:access_key:A123KEYsecret_key:somesecretkeyvaluetoken:theaccesstokenifsetmethod:how-the-credentials-were-loadedclients:...All of these values are optional and in cases where they are omitted, but the session
requires having them, they will be defaulted.Client OverridesThere are cases where it is desirable to replace one or more service clients with
non-lobotomy objects. Lobotomy supports that by adding client overrides to the patched
lobotomy object:fromunittest.mockimportMagicMockimportlobotomy@lobotomy.patch()deftest_example(lobotomized:lobotomy.Lobotomy):"""Should do something..."""mock_dynamo_db_client=MagicMock()lobotomized.add_client_override("dynamodb",mock_dynamo_db_client)# continue testing...Error HandlingThe lobotomy library mimics client error handling with a lobotomized version of the
same interface used by the live clients. As such, handling and capturing errors works
transparently. For example,importboto3importpytestimportlobotomy@lobotomy.patch()deftest_client_errors(lobotomized:lobotomy.Lobotomy):"""Should raise the specified error."""lobotomized.add_error_call(service_name="s3",method_name="list_objects",error_code="NoSuchBucket",error_message="Hello...",)session=boto3.Session()client=session.client("s3")withpytest.raises(client.exceptions.NoSuchBucket):client.list_objects(Bucket="foo")Or the generic client error handling with response codes can be used as well:importboto3importpytestimportlobotomy@lobotomy.patch()deftest_client_errors(lobotomized:lobotomy.Lobotomy):"""Should raise the specified error."""lobotomized.add_call(service_name="s3",method_name="list_objects",error_code="NoSuchBucket",error_message="Hello...",)session=boto3.Session()client=session.client("s3")withpytest.raises(lobotomy.ClientError)asexception_info:client.list_objects(Bucket="foo")assertexception_info.value.response["Error"]["Code"]=="NoSuchBucket"These errors can also be specified in YAML files like this:clients:s3:list_objects:!lobotomy.errorcode:NoSuchBucketmessage:Hello...The!lobotomy.errorwill load this as an error response with the same behavior as the
examples shown above.Callable ResponsesFor more advanced cases it is also possible to use any callable as the source for
generating the response. For example, if I want the response to be different based
on the arguments specified when making the call, I can create a function to return
a result that reflects the inputs.importtypingimportboto3importlobotomydef_respond(*args,**kwargs)->typing.Dict[str,typing.Any]:return{"Body":"{}.{}".format(kwargs["Bucket"],kwargs["Key"]).encode()}@lobotomy.patch()deftest_callable_responses(lobotomized:"lobotomy.Lobotomy"):"""Should return the expected body value from the callable response."""lobotomized.add_call("s3","get_object",_respond)session=boto3.Session()client=session.client("s3")response=client.get_object(Bucket="foo",Key="bar")assertresponse["Body"].read()==b"foo.bar"Any callable is supported as long as it accepts the*args, **kwargsappropriate to
the calling client service request.