Repository: boto/botocore

Issue:
Mixing pythonic variable_names with CamelCase
I've been writing a wrapper around botocore for interfacing with DynamoDB, and have discovered that it requires a weird mix of parameters names. When passing in key word arguments, it seems that top level keys must be pythonic, while lower level keys must be camel cased. 

For example, these arguments are _not_ valid when used with UpdateTable:

``` python
update_table_operation({
    'provisioned_throughput': {
        'write_capacity_units': 2, 
        'read_capacity_units': 2
    }, 
    'table_name': 'test-table'
})
botocore.exceptions.UnknownKeyError: Unknown key 'read_capacity_units' for param 'ProvisionedThroughput'.  Must be one of: ReadCapacityUnits, WriteCapacityUnits
```

This mixed syntax works:

``` python
update_table_operation({
    'provisioned_throughput': {
        'WriteCapacityUnits': 2, 
        'ReadCapacityUnits': 2
    }, 
    'table_name': 'test-table'
})
```

Note that changing the top level key word to camel case does not work either.

The exception is raised [here](https://github.com/boto/botocore/blob/develop/botocore/parameters.py#L470):

``` python
def _validate_known_keys(self, value):
        valid_keys = [p.name for p in self.members]
        for key in value:
            if key not in valid_keys:
                raise UnknownKeyError(
                    value=key, choices=', '.join(valid_keys), param=self.name)
```

The `valid_keys` list is constructed based on the camel case names for parameters, but each object in `self.members` also has a `py_name` attribute. In fact, on the ReadCapacityUnits member object the attribute is defined as `read_capacity_units`. It would be simple enough to build the list using `[p.py_name for p in self.members]`, but I'm guessing there are many other places where a similar list is constructed.

If a purely pythonic (or camel case) API is desired, I am willing to work on it and send a PR. 



Discussion:
I've also found this behavior unintuitive in the extreme; to me, it's a gross design flaw that will doubtless bite user after user.

Considering my MWS/FPS code, it's clear that my preference is to make our "thin layer" match the Amazon API as closely as possible, including choice of parameter name and case.  In fact, it was a almost a concession on my part to name the call methods in  in snake_case versus the Amazon convention of CapWords (and it of course meant additional needless complexity in the code as well).

Despite the fact that MWS/FPS likely differs from every other boto module in this regard, I don't recall fielding a single question about argument case; once users understand that the API offers a 1:1 translation, they can rely upon that fact across the entire module and leverage both Amazon and boto documentation.  There's some real value to this convention that should not be overlooked; it may appear to be a trivial style change, but to the user, if it looks the same, it's probably the same -- and if it looks different, perhaps it _is_ different.  Don't make me think!

In this case, I feel that the slavish adoption of an apparent pythonic convention has already meant a significant sacrifice to the translucency of the API, but the fact that the convention varies inexplicably inside input data structures is, frankly, mind-boggling.

As far as method argument case goes, PEP8 offers no specific recommendation on convention, but to quote the most relevant section of that document specifically:

> ## Naming Conventions
> 
> ### Overriding Principle
> 
> Names that are visible to the user as public parts of the API should follow conventions that reflect usage rather than implementation.

I don't know what else needs to be said, except to restate that any convention that's followed consistently would be preferable to what we have to work with at present.

I've written my wrapper around botocore now, available [here](https://github.com/jlafon/PynamoDB). Because of this mix of camel case and snake case variable names I have resorted to using a [regex to transform the names](https://github.com/jlafon/PynamoDB/blob/devel/pynamodb/connection/util.py). 

Speaking specifically about the DynamoDB API, there are quite a few strings used throughout it ([here is a list of them](https://github.com/jlafon/PynamoDB/blob/devel/pynamodb/constants.py)). That leaves me with the choice of defining a snake case and a camel case version of every single constant, or using a regex to go back and forth. I don't really like either, because it decreases the readability of the code, and I have to remember when to use snake case and when to use camel case. Although I'm using DynamoDB as an example, the same can be said about any of the services wrapped by botocore. 

In summary, in order to build something _consistent_ for our users, we're stuck awkwardly adopting either camel case or snake case ourselves and translating where appropriately in our own abstractions. Boto and botocore are immensely valuable resources to the Python community, being amongst the most popular packages. You should take advantage of your reach to demonstrate, by example, how to write beautiful Python.

@jlafon Thanks for the feedback and sorry for the delayed response.

I think you make some valid points.  The original motivation for the snake_casing of top level parameters was that if users are using botocore like this:

```
s3 = session.get_service('s3')
list_objects = s3.get_operation('ListObjects')
list_objects.call(endpoint, bucket='jamesls-test-sync', max_keys=1)
```

Then the API does not mix any casing.  The idea was to treat the arguments as `.call(endpoint, arg_name1=<anything>, arg_name2=<anything>, arg_name3=<anything>)`.

However, as you point out when `<anything>` happens to be a dict, then the casing reflects that used by the API, and for AWS APIs, that always means camelCase or CamelCase.

I would not be opposed to changing this, but with a few caveats:
- We'd like to avoid breaking existing users of botocore (or at least minimal breakage)
- I think we'd still like to support the exact casing used by the service API (the aws-cli, for example, uses this service API casing, e.g. `--provisioned-throughput ReadCapacityUnits=10,WriteCapacityUnits=10`

It might be as straightforward as relaxing the validation on the `_known_keys` check, as you suggest, but I'd have to look more in depth to be sure.

btw, pynamodb looks like a cool project.


Resolving patch:
diff --git a/botocore/operation.py b/botocore/operation.py
--- a/botocore/operation.py
+++ b/botocore/operation.py
@@ -18,7 +18,7 @@
 from botocore.exceptions import UnknownParameterError
 from botocore.paginate import Paginator
 from botocore.payload import Payload, XMLPayload, JSONPayload
-from botocore import BotoCoreObject
+from botocore import BotoCoreObject, xform_name
 
 logger = logging.getLogger(__name__)
 
@@ -156,6 +156,7 @@ def build_parameters(self, **kwargs):
         """
         built_params = self._get_built_params()
         missing = []
+        kwargs = self._camel_to_snake_case(kwargs)
         self._check_for_unknown_params(kwargs)
         for param in self.params:
             if param.required:
@@ -172,6 +173,15 @@ def build_parameters(self, **kwargs):
                                          object_name=self)
         return built_params
 
+    def _camel_to_snake_case(self, kwargs):
+        """
+        Converts top level keys in `kwargs` from camel case to snake case
+        """
+        params = {}
+        for key, value in kwargs.items():
+            params[xform_name(key)] = value
+        return params
+
     def _check_for_unknown_params(self, kwargs):
         valid_names = [p.py_name for p in self.params]
         for key in kwargs:
diff --git a/botocore/utils.py b/botocore/utils.py
--- a/botocore/utils.py
+++ b/botocore/utils.py
@@ -11,6 +11,7 @@
 # ANY KIND, either express or implied. See the License for the specific
 # language governing permissions and limitations under the License.
 
+
 from .exceptions import InvalidExpressionError
 
 


Test patch:
diff --git a/tests/unit/test_parameters.py b/tests/unit/test_parameters.py
--- a/tests/unit/test_parameters.py
+++ b/tests/unit/test_parameters.py
@@ -332,6 +332,53 @@ def test_struct_unknown_keys_validation(self):
                              'group_name': 'unknown',
                              'Tenancy': 'baz'})
 
+    def test_snake_case_parameters(self):
+        """
+        Test support for snake case parameters
+        """
+        session = botocore.session.get_session()
+        service = session.get_service('dynamodb')
+        op = service.get_operation('UpdateTable')
+
+        # All CamelCase
+        params = {
+            'ProvisionedThroughput': {
+                'WriteCapacityUnits': 2,
+                'ReadCapacityUnits': 2
+            },
+            'TableName': 'test-table'
+        }
+        built_params = op.build_parameters(**params)
+        self.assertEqual(
+            built_params,
+            {
+                'ProvisionedThroughput': {
+                    'WriteCapacityUnits': 2,
+                    'ReadCapacityUnits': 2
+                },
+                'TableName': 'test-table'
+            }
+        )
+
+        # Mixed case
+        params = {
+            'provisioned_throughput': {
+                'WriteCapacityUnits': 2,
+                'ReadCapacityUnits': 2
+            },
+            'table_name': 'test-table'
+        }
+        built_params = op.build_parameters(**params)
+        self.assertEqual(
+            built_params,
+            {
+                'ProvisionedThroughput': {
+                    'WriteCapacityUnits': 2,
+                    'ReadCapacityUnits': 2
+                },
+                'TableName': 'test-table'
+            }
+        )
 
 
 if __name__ == "__main__":