Repository: cloud-custodian/cloud-custodian

Issue:
emr set default server side query to (waiting, bootstrapping, running)
Policy looks like so
  - name: weekend_emr_terminate
    resource: emr
    comments:
      weekend emr termination
    filters:
      - "tag:custodian_opt_out": absent
      - "ClusterStates": waiting
    actions:
      - terminate

Getting a throttling exception

ClientError: An error occurred (ThrottlingException) when calling the DescribeCluster operation (reached max retries: 4): Rate exceeded
2017-02-23 12:54:56,986: custodian.commands:ERROR Error while executing policy weekend_emr_terminate, continuing



Discussion:
emr keeps full history, so its important to use a server side query filter. we should probably make this a default (for pending, waiting, running).

```
  - name: weekend_emr_terminate
    resource: emr
    comments:
      weekend emr termination
    query:
      ClusterStates: 
         - waiting
    filters:
      - "tag:custodian_opt_out": absent
    actions:
      - terminate
```

@kapilt  thanks for your help on this..  still not able to get it going...is there something i am missing?  

policies:
  - name: weekend_emr_terminate
    resource: emr
    comments:
      weekend emr termination
    query:
      ClusterStates:
        - waiting  
    filters:
      - "tag:custodian_opt_out": absent
      - "Status.State": WAITING
    actions:
      - terminate

Exception: Failed to validate on policy weekend_emr_terminate
 {'ClusterStates': ['waiting']} is not of type 'array'

Failed validating 'type' in schema[17]['allOf'][0]['properties']['query']:
    {'items': {'maxProperties': 1, 'minProperties': 1, 'type': 'object'},
     'type': 'array'}

On instance['query']:
    {'ClusterStates': ['waiting']}



Booya....figured it out....
Changed it to 
    query:
        - ClusterStates: [RUNNING, WAITING]

Resolving patch:
diff --git a/c7n/resources/emr.py b/c7n/resources/emr.py
--- a/c7n/resources/emr.py
+++ b/c7n/resources/emr.py
@@ -90,6 +90,14 @@ def consolidate_query_filter(self):
             else:
                 names.add(query_filter['Name'])
                 result.append(query_filter)
+        if 'ClusterStates' not in names:
+            # include default query
+            result.append(
+                {
+                    'Name': 'ClusterStates',
+                    'Values': ['waiting', 'running', 'bootstrapping'],
+                }
+            )
         return result
 
     def augment(self, resources):


Test patch:
diff --git a/tests/test_emr.py b/tests/test_emr.py
--- a/tests/test_emr.py
+++ b/tests/test_emr.py
@@ -55,7 +55,34 @@ def test_consolidate_query_filter(self):
             mgr.consolidate_query_filter(),
             [
                 {'Values': ['val1', 'val2'], 'Name': 'tag:foo'},
-                {'Values': ['val3'], 'Name': 'tag:bar'}
+                {'Values': ['val3'], 'Name': 'tag:bar'},
+                # default query
+                {
+                    'Values': ['waiting', 'running', 'bootstrapping'],
+                    'Name': 'ClusterStates',
+                },
+            ]
+        )
+
+        query = {
+            'query': [
+                {'tag:foo': 'val1'},
+                {'tag:foo': 'val2'},
+                {'tag:bar': 'val3'},
+                {'ClusterStates': 'terminated'},
+            ]
+        }
+        mgr = emr.EMRCluster(ctx, query)
+        self.assertEqual(
+            mgr.consolidate_query_filter(),
+            [
+                {'Values': ['val1', 'val2'], 'Name': 'tag:foo'},
+                {'Values': ['val3'], 'Name': 'tag:bar'},
+                # verify default is overridden
+                {
+                    'Values': ['terminated'],
+                    'Name': 'ClusterStates',
+                },
             ]
         )