Repository: boto/botocore
Pull request #1027 (2016-09-09 05:38:04)

Added partion to client meta object
I found that I needed access to the partition being used for a particular client, so that I can write tools that work in both normal AWS regions as well as GovCloud and China.
I view this as a feature similar to the region_name property of the client meta object.

I have added the partition as a property on the client meta object.
I have added unit tests for new meta property.

The one part I am not sure about to what to do if no partition is set for an endpoint_config.
For the moment I have set it to be None, which seems like a reasonable fallback, though I am happy for advice on a better default value. Possibly just setting 'aws' as the fallback partition.

codecov-io: ## [Current coverage](https://codecov.io/gh/boto/botocore/pull/1027?src=pr) is 97.56% (diff: 75.00%)
> Merging [#1027](https://codecov.io/gh/boto/botocore/pull/1027?src=pr) into [develop](https://codecov.io/gh/boto/botocore/branch/develop?src=pr) will decrease coverage by **0.01%**

```diff
@@            develop      #1027   diff @@
==========================================
  Files            44         44          
  Lines          6884       6888     +4   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           6717       6720     +3   
- Misses          167        168     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [40510b3...f80c92c](https://codecov.io/gh/boto/botocore/compare/40510b3469e5f7e05e97f012e97ed097275c6df9...f80c92c9ed34cd7462e85226d07ec1f32027cb21?src=pr)
jamesls: Can you give me a little background info on how you'd use the partition information for a given client?  I'd just like to understand the use case a little better.
Posnet: This is specifically for a stage in our tooling where we enhance cloudtrails information with data we manually pull from the API.

At the moment we have a hard coded mapping from regions to partition so that our scraping tool (similar to netflix edda) is able to populate the cloudtrails record with more data before forwarding it to our log aggregation platform.

This works fine for us, but it seemed like a cleaner solution to have it available from the client object. Our current mapping code actually uses the botocore data files to lookup the region names and work out the partition, very similar to my submitted pull request.
jamesls: Seems reasonable to me.  Our `ClientMeta` has `region_name` and `endpoint_url`.  Exposing `partition` seems fine.  We'll need to move the tests over to `tests/functional`.  I think that's why codecov is complaining.  We don't run integration tests during our travis builds.

@kyleknap @JordonPhillips any concerns?
kyleknap: @jamesls I am fine with this and yes let's get these moved over from integration tests to unit or functional.
Posnet: I have moved the tests to the unit tests
jamesls: Looks good to me.  I think I'd still like to also see a functional test that uses a real client via `session.create_client(...)` to ensure everything's hooked up properly.
Posnet: I have added functional tests as requested.
Posnet: Any update on whether this can be merged?
Posnet: Is there anything else I need to do for this pull request?
jamesls: Could you remove the `tests/coverage.xml` file?  Looks like it was added by accident.

Otherwise, the test updates look good to me, let's just get one more reviewer to look at it. 

cc @kyleknap @JordonPhillips 

Patch:
diff --git a/botocore/args.py b/botocore/args.py
index 6a5cd31e0..2b9366d8a 100644
--- a/botocore/args.py
+++ b/botocore/args.py
@@ -50,6 +50,7 @@ class ClientArgsCreator(object):
         protocol = final_args['protocol']
         config_kwargs = final_args['config_kwargs']
         s3_config = final_args['s3_config']
+        partition = endpoint_config['metadata'].get('partition', None)
 
         event_emitter = copy.copy(self._event_emitter)
         signer = RequestSigner(
@@ -83,7 +84,8 @@ class ClientArgsCreator(object):
             'request_signer': signer,
             'service_model': service_model,
             'loader': self._loader,
-            'client_config': new_config
+            'client_config': new_config,
+            'partition': partition
         }
 
     def compute_client_args(self, service_model, client_config,
diff --git a/botocore/client.py b/botocore/client.py
index 18e713226..294ebcecc 100644
--- a/botocore/client.py
+++ b/botocore/client.py
@@ -387,7 +387,7 @@ class BaseClient(object):
 
     def __init__(self, serializer, endpoint, response_parser,
                  event_emitter, request_signer, service_model, loader,
-                 client_config):
+                 client_config, partition):
         self._serializer = serializer
         self._endpoint = endpoint
         self._response_parser = response_parser
@@ -397,7 +397,7 @@ class BaseClient(object):
         self._client_config = client_config
         self.meta = ClientMeta(event_emitter, self._client_config,
                                endpoint.host, service_model,
-                               self._PY_TO_OP_NAME)
+                               self._PY_TO_OP_NAME, partition)
         self._register_handlers()
 
     def _register_handlers(self):
@@ -660,12 +660,13 @@ class ClientMeta(object):
     """
 
     def __init__(self, events, client_config, endpoint_url, service_model,
-                 method_to_api_mapping):
+                 method_to_api_mapping, partition):
         self.events = events
         self._client_config = client_config
         self._endpoint_url = endpoint_url
         self._service_model = service_model
         self._method_to_api_mapping = method_to_api_mapping
+        self._partition = partition
 
     @property
     def service_model(self):
@@ -686,3 +687,7 @@ class ClientMeta(object):
     @property
     def method_to_api_mapping(self):
         return self._method_to_api_mapping
+
+    @property
+    def partition(self):
+        return self._partition


Test patch:
diff --git a/tests/functional/test_client_metadata.py b/tests/functional/test_client_metadata.py
new file mode 100644
index 000000000..38dbb5443
--- /dev/null
+++ b/tests/functional/test_client_metadata.py
@@ -0,0 +1,44 @@
+# Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You
+# may not use this file except in compliance with the License. A copy of
+# the License is located at
+#
+# http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
+# ANY KIND, either express or implied. See the License for the specific
+# language governing permissions and limitations under the License.
+from tests import unittest
+
+import botocore.session
+
+class TestClientMeta(unittest.TestCase):
+    def setUp(self):
+        self.session = botocore.session.get_session()
+
+    def test_region_name_on_meta(self):
+        client = self.session.create_client('s3', 'us-west-2')
+        self.assertEqual(client.meta.region_name, 'us-west-2')
+
+    def test_endpoint_url_on_meta(self):
+        client = self.session.create_client('s3', 'us-west-2',
+                                            endpoint_url='https://foo')
+        self.assertEqual(client.meta.endpoint_url, 'https://foo')
+
+    def test_client_has_standard_partition_on_meta(self):
+        client = self.session.create_client('s3', 'us-west-2')
+        self.assertEqual(client.meta.partition, 'aws')
+
+    def test_client_has_china_partition_on_meta(self):
+        client = self.session.create_client('s3', 'cn-north-1')
+        self.assertEqual(client.meta.partition, 'aws-cn')
+
+    def test_client_has_gov_partition_on_meta(self):
+        client = self.session.create_client('s3', 'us-gov-west-1')
+        self.assertEqual(client.meta.partition, 'aws-us-gov')
+
+    def test_client_has_no_partition_on_meta_if_custom_region(self):
+        client = self.session.create_client('s3', 'myregion')
+        self.assertEqual(client.meta.partition, None)
diff --git a/tests/unit/test_args.py b/tests/unit/test_args.py
index 86b224206..576130a70 100644
--- a/tests/unit/test_args.py
+++ b/tests/unit/test_args.py
@@ -114,7 +114,8 @@ class TestCreateClientArgs(unittest.TestCase):
         bridge.resolve.return_value = {
             'region_name': 'us-west-2', 'signature_version': 'v4',
             'endpoint_url': 'https://ec2/',
-            'signing_name': 'ec2', 'signing_region': 'us-west-2'}
+            'signing_name': 'ec2', 'signing_region': 'us-west-2',
+            'metadata': {}}
         with mock.patch('botocore.args.EndpointCreator') as m:
             args_create.get_client_args(
                 service_model, 'us-west-2', True, 'https://ec2/', True,
diff --git a/tests/unit/test_client.py b/tests/unit/test_client.py
index 2f3f2748f..6f74adfee 100644
--- a/tests/unit/test_client.py
+++ b/tests/unit/test_client.py
@@ -248,6 +248,26 @@ class TestAutoGeneratedClient(unittest.TestCase):
         self.assertEqual(service_client.meta.endpoint_url,
                          'https://foo.bar')
 
+    def test_client_has_standard_partition_on_meta(self):
+        creator = self.create_client_creator()
+        service_client = creator.create_client(
+            'myservice', 'us-west-2', credentials=self.credentials)
+        self.assertEqual(service_client.meta.partition,
+                         'aws')
+
+    def test_client_has_non_standard_partition_on_meta(self):
+        creator = self.create_client_creator()
+        self.resolver.construct_endpoint.return_value = {
+            'partition': 'aws-cn',
+            'hostname': 'foo',
+            'endpointName': 'cn-north-1',
+            'signatureVersions': ['v4'],
+        }
+        service_client = creator.create_client(
+            'myservice', 'cn-north-1', credentials=self.credentials)
+        self.assertEqual(service_client.meta.partition,
+                         'aws-cn')
+
     def test_api_version_is_passed_to_loader_if_provided(self):
         creator = self.create_client_creator()
         self.endpoint.host = 'https://foo.bar'