Repository: cloud-custodian/cloud-custodian

Issue:
SQS policy in cloudtrail mode
I got the log below when running a SQS policy in cloudtrail mode:
```
START RequestId: <request_id> Version: $LATEST
[INFO]  2018-04-19T17:10:42.252Z  <request_id>  Processing event
[DEBUG] 2018-04-19T17:10:42.967Z  <request_id>  Disabling cache
[INFO]  2018-04-19T17:10:42.968Z  <request_id>  Found resource ids: [u'https://sqs.us-east-1.amazonaws.com/account_id/sqs_name']
[DEBUG] 2018-04-19T17:10:43.269Z  <request_id>  retrieving details for 0 queues
[INFO]  2018-04-19T17:10:43.269Z  <request_id>  Resources []
END RequestId: <request_id>
```

Although there is no error message, custodian is not able to get any SQS so that no filter/action can be further taken. Further found two issues related:
1. **Various formats of the SQS `queueUrl`**:
The format we got from CW event and on the SQS console is:
`https://sqs.<region>.amazonaws.com/<account_id>/<sqs_name>`
Yet the format from boto3/cli is:
`http://queue.amazonaws.com/<account_id>/<sqs_name>` (for us-east-1)
`http://<region>.queue.amazonaws.com/<account_id>/<sqs_name>` (for all the other regions)
We can also see different formats from the test data: [format_2_1](https://github.com/capitalone/cloud-custodian/blob/master/tests/data/placebo/test_sqs_set_retention/sqs.ListQueues_1.json#L5), [format_2_2](https://github.com/capitalone/cloud-custodian/blob/master/tests/data/placebo/test_sqs_tags/sqs.ListQueues_1.json#L5) <br>
This is making the resource id from CW event always different from the id custodian gets when querying SQS resources
Confirmed with AWS that format 2 is an old one, but AWS doesn't have plan to retire the old format because of the backwards compatibility (like python 2.6). There is also [an related issue in aws-cli](https://github.com/aws/aws-cli/issues/1715)

2. **`filter_name` SQS is using**:
The `filter_name` for SQS is `QueueNamePrefix`
CW event doesn't give us the id of the queue but whole url, yet the only parameter [`SQS.Client.list_queues`](http://boto3.readthedocs.io/en/latest/reference/services/sqs.html#SQS.Client.list_queues) has is `
QueueNamePrefix`. 
`list_queues(QueueNamePrefix='queue_url')` will always get empty response.

I can think of two ways to solve this problem:
1. (a)Add the endpoint url when creating `client` for sqs so that the queueUrl format is unified
(b)Change `filter_name` to `None` so that custodian can get all the resources and [let custodian filter it](https://github.com/capitalone/cloud-custodian/blob/master/c7n/query.py#L100-L107).
2. Change the identities from the whole url to only the queue name for sqs before custodian send `list_queues(QueueNamePrefix='...')` 

Solution 2 doesn't require unifying the format, but that can still be added.
Would like to check which one is preferred or if there is any concern on the solutions before creating the PR.

Thanks.



Discussion:
i suspect at least for a cloudtrail policy the issue is just specifying ids to be the queue name from the event. Still worth resolving holistically, but passing the queue name as the id would suffice to get a working cloudtrail policy.

Resolving patch:
diff --git a/c7n/resources/sqs.py b/c7n/resources/sqs.py
--- a/c7n/resources/sqs.py
+++ b/c7n/resources/sqs.py
@@ -63,6 +63,15 @@ def get_permissions(self):
     def get_arns(self, resources):
         return [r['QueueArn'] for r in resources]
 
+    def get_resources(self, ids, cache=True):
+        ids_normalized = []
+        for i in ids:
+            if not i.startswith('https://'):
+                ids_normalized.append(i)
+                continue
+            ids_normalized.append(i.rsplit('/', 1)[-1])
+        return super(SQS, self).get_resources(ids_normalized, cache)
+
     def augment(self, resources):
         client = local_session(self.session_factory).client('sqs')
 


Test patch:
diff --git a/tests/data/placebo/test_sqs_get_resources/sqs.GetQueueAttributes_1.json b/tests/data/placebo/test_sqs_get_resources/sqs.GetQueueAttributes_1.json
new file mode 100644
--- /dev/null
+++ b/tests/data/placebo/test_sqs_get_resources/sqs.GetQueueAttributes_1.json
@@ -0,0 +1,31 @@
+{
+    "status_code": 200,
+    "data": {
+        "Attributes": {
+            "QueueArn": "arn:aws:sqs:us-east-2:644160558196:BrickHouse",
+            "ApproximateNumberOfMessages": "0",
+            "ApproximateNumberOfMessagesNotVisible": "0",
+            "ApproximateNumberOfMessagesDelayed": "0",
+            "CreatedTimestamp": "1525276033",
+            "LastModifiedTimestamp": "1525276033",
+            "VisibilityTimeout": "30",
+            "MaximumMessageSize": "262144",
+            "MessageRetentionPeriod": "345600",
+            "DelaySeconds": "0",
+            "ReceiveMessageWaitTimeSeconds": "0"
+        },
+        "ResponseMetadata": {
+            "RequestId": "3383aa7f-a419-56d5-a5c9-f91c45969536",
+            "HTTPStatusCode": 200,
+            "HTTPHeaders": {
+                "server": "Server",
+                "date": "Wed, 02 May 2018 15:54:55 GMT",
+                "content-type": "text/xml",
+                "content-length": "1164",
+                "connection": "keep-alive",
+                "x-amzn-requestid": "3383aa7f-a419-56d5-a5c9-f91c45969536"
+            },
+            "RetryAttempts": 0
+        }
+    }
+}
\ No newline at end of file
diff --git a/tests/data/placebo/test_sqs_get_resources/sqs.GetQueueAttributes_2.json b/tests/data/placebo/test_sqs_get_resources/sqs.GetQueueAttributes_2.json
new file mode 100644
--- /dev/null
+++ b/tests/data/placebo/test_sqs_get_resources/sqs.GetQueueAttributes_2.json
@@ -0,0 +1,31 @@
+{
+    "status_code": 200,
+    "data": {
+        "Attributes": {
+            "QueueArn": "arn:aws:sqs:us-east-2:644160558196:BrickHouse",
+            "ApproximateNumberOfMessages": "0",
+            "ApproximateNumberOfMessagesNotVisible": "0",
+            "ApproximateNumberOfMessagesDelayed": "0",
+            "CreatedTimestamp": "1525276033",
+            "LastModifiedTimestamp": "1525276033",
+            "VisibilityTimeout": "30",
+            "MaximumMessageSize": "262144",
+            "MessageRetentionPeriod": "345600",
+            "DelaySeconds": "0",
+            "ReceiveMessageWaitTimeSeconds": "0"
+        },
+        "ResponseMetadata": {
+            "RequestId": "987e05be-6ef7-5e7b-8d82-56dd8f674ae4",
+            "HTTPStatusCode": 200,
+            "HTTPHeaders": {
+                "server": "Server",
+                "date": "Wed, 02 May 2018 15:54:56 GMT",
+                "content-type": "text/xml",
+                "content-length": "1164",
+                "connection": "keep-alive",
+                "x-amzn-requestid": "987e05be-6ef7-5e7b-8d82-56dd8f674ae4"
+            },
+            "RetryAttempts": 0
+        }
+    }
+}
\ No newline at end of file
diff --git a/tests/data/placebo/test_sqs_get_resources/sqs.ListQueues_1.json b/tests/data/placebo/test_sqs_get_resources/sqs.ListQueues_1.json
new file mode 100644
--- /dev/null
+++ b/tests/data/placebo/test_sqs_get_resources/sqs.ListQueues_1.json
@@ -0,0 +1,21 @@
+{
+    "status_code": 200,
+    "data": {
+        "QueueUrls": [
+            "https://us-east-2.queue.amazonaws.com/644160558196/BrickHouse"
+        ],
+        "ResponseMetadata": {
+            "RequestId": "c88063db-d0b5-5569-96e9-30b8313a0175",
+            "HTTPStatusCode": 200,
+            "HTTPHeaders": {
+                "server": "Server",
+                "date": "Wed, 02 May 2018 15:54:55 GMT",
+                "content-type": "text/xml",
+                "content-length": "328",
+                "connection": "keep-alive",
+                "x-amzn-requestid": "c88063db-d0b5-5569-96e9-30b8313a0175"
+            },
+            "RetryAttempts": 0
+        }
+    }
+}
\ No newline at end of file
diff --git a/tests/data/placebo/test_sqs_get_resources/sqs.ListQueues_2.json b/tests/data/placebo/test_sqs_get_resources/sqs.ListQueues_2.json
new file mode 100644
--- /dev/null
+++ b/tests/data/placebo/test_sqs_get_resources/sqs.ListQueues_2.json
@@ -0,0 +1,21 @@
+{
+    "status_code": 200,
+    "data": {
+        "QueueUrls": [
+            "https://us-east-2.queue.amazonaws.com/644160558196/BrickHouse"
+        ],
+        "ResponseMetadata": {
+            "RequestId": "33a8cdad-f762-5ddd-af78-9ece16f2d7bf",
+            "HTTPStatusCode": 200,
+            "HTTPHeaders": {
+                "server": "Server",
+                "date": "Wed, 02 May 2018 15:54:56 GMT",
+                "content-type": "text/xml",
+                "content-length": "328",
+                "connection": "keep-alive",
+                "x-amzn-requestid": "33a8cdad-f762-5ddd-af78-9ece16f2d7bf"
+            },
+            "RetryAttempts": 0
+        }
+    }
+}
\ No newline at end of file
diff --git a/tests/data/placebo/test_sqs_get_resources/tagging.GetResources_1.json b/tests/data/placebo/test_sqs_get_resources/tagging.GetResources_1.json
new file mode 100644
--- /dev/null
+++ b/tests/data/placebo/test_sqs_get_resources/tagging.GetResources_1.json
@@ -0,0 +1,18 @@
+{
+    "status_code": 200,
+    "data": {
+        "PaginationToken": "",
+        "ResourceTagMappingList": [],
+        "ResponseMetadata": {
+            "RequestId": "28addbb7-4e21-11e8-941b-1d203a71fd52",
+            "HTTPStatusCode": 200,
+            "HTTPHeaders": {
+                "x-amzn-requestid": "28addbb7-4e21-11e8-941b-1d203a71fd52",
+                "content-type": "application/x-amz-json-1.1",
+                "content-length": "50",
+                "date": "Wed, 02 May 2018 15:54:55 GMT"
+            },
+            "RetryAttempts": 0
+        }
+    }
+}
\ No newline at end of file
diff --git a/tests/data/placebo/test_sqs_get_resources/tagging.GetResources_2.json b/tests/data/placebo/test_sqs_get_resources/tagging.GetResources_2.json
new file mode 100644
--- /dev/null
+++ b/tests/data/placebo/test_sqs_get_resources/tagging.GetResources_2.json
@@ -0,0 +1,18 @@
+{
+    "status_code": 200,
+    "data": {
+        "PaginationToken": "",
+        "ResourceTagMappingList": [],
+        "ResponseMetadata": {
+            "RequestId": "292dba31-4e21-11e8-993e-27f77e0a8c2b",
+            "HTTPStatusCode": 200,
+            "HTTPHeaders": {
+                "x-amzn-requestid": "292dba31-4e21-11e8-993e-27f77e0a8c2b",
+                "content-type": "application/x-amz-json-1.1",
+                "content-length": "50",
+                "date": "Wed, 02 May 2018 15:54:56 GMT"
+            },
+            "RetryAttempts": 0
+        }
+    }
+}
\ No newline at end of file
diff --git a/tests/test_sqs.py b/tests/test_sqs.py
--- a/tests/test_sqs.py
+++ b/tests/test_sqs.py
@@ -357,3 +357,15 @@ def test_sqs_set_retention(self):
             QueueUrl=resources[0]['QueueUrl'],
             AttributeNames=['MessageRetentionPeriod'])['Attributes']
         self.assertEqual(int(retention['MessageRetentionPeriod']), 86400)
+
+    def test_sqs_get_resources(self):
+        factory = self.replay_flight_data('test_sqs_get_resources')
+        p = self.load_policy({
+            'name': 'sqs-reduce',
+            'resource': 'sqs'}, session_factory=factory)
+        url1 = 'https://us-east-2.queue.amazonaws.com/644160558196/BrickHouse'
+        url2 = 'https://sqs.us-east-2.amazonaws.com/644160558196/BrickHouse'
+        resources = p.resource_manager.get_resources([url1])
+        self.assertEqual(resources[0]['QueueUrl'], url1)
+        resources = p.resource_manager.get_resources([url2])
+        self.assertEqual(resources[0]['QueueUrl'], url1)