Repository: cloud-custodian/cloud-custodian

Issue:
sentry log relay as lambda function
The sentry logger in c7n_sentry has some logic to work as a log subscription filter, but thats not completed. The goal would be to attach it as a long subscription filter to all log groups given a prefix.



Discussion:
From private slack:

> sentry indexer as lambda would be helpful, we're currently doing it as an hrly cron, getting to real time would be helpful for our ops side
> 
>> I find https://github.com/capitalone/cloud-custodian/blob/38248fcfd524f1d829f296fbf7c6890ee07ecf65/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L39-L46.
>
> there's parts of it already done
> i never finished it after getting the cron indexer working though
> the delta to get it working should hopefully be small, the provisioning of lambdas and attaching to log groups is already done.
> and used by other items in active use (log sub error emails)
I'm not finding very much lambda wiring. I'm working on bringing over code from `c7n_mailer`. Looks like this case is simpler because we don't have any external deps so we don't need an archive.
... which is probably what this means:

> the provisioning of lambdas and attaching to log groups is already done.
and used by other items in active use (log sub error emails)


> I'm not finding very much lambda wiring.

Scratch that. [Here it is](https://github.com/capitalone/cloud-custodian/blob/b6200acd1408e06e7c90014cb1199214b6e4eafa/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L320-L363).
Looks like a copy/paste [from](https://github.com/capitalone/cloud-custodian/blob/b6200acd1408e06e7c90014cb1199214b6e4eafa/c7n/ufuncs/logsub.py#L91-L132) `logsub.py`.
Except that [`cwl2sentry.py`](https://github.com/capitalone/cloud-custodian/blob/b6200acd1408e06e7c90014cb1199214b6e4eafa/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L351) doesn't exist (that I can `find`).
So am I writing `cwl2sentry.py` ... or is that basically [`orgreplay`](https://github.com/capitalone/cloud-custodian/blob/b6200acd1408e06e7c90014cb1199214b6e4eafa/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L366-L445) already?
I've set up a Sentry testbed and have got the cli minimally working against it with:

```
(cloud-custodian) $ c7n-sentry orgreplay -c config.json --sentry-dsn=foo --sentry-org=bar --sentry-token=deadbeef
```

Now I guess I need to populate `config.json` [such](https://github.com/capitalone/cloud-custodian/blob/b6200acd1408e06e7c90014cb1199214b6e4eafa/tools/c7n_sentry/c7n_sentry/common.py#L38-L50) that [`accounts`](https://github.com/capitalone/cloud-custodian/blob/b6200acd1408e06e7c90014cb1199214b6e4eafa/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L368) becomes non-empty.
From reading [code](https://github.com/capitalone/cloud-custodian/blob/2cced870fd657e88dddffbedec15b7e82791c86a/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L385-L436) I infer this basic structure for `config.json`:

```json
{
    "foo": {
        "name": "foo",
        "account_id": "bar",
        "config_files": [],
        "role": ""
    }
}
```
Hmm ...

```
(cloud-custodian) $ c7n-sentry orgreplay -c config.json --sentry-dsn=foo --sentry-org=c537n --sentry-token=deadbeef
2017-01-16 06:58:16,339: cwl2sentry:INFO sentry endpoint: https://sentry.io/api/0/
2017-01-16 06:58:17,042: cwl2sentry:INFO creating org team foo
2017-01-16 06:58:17,318: cwl2sentry:INFO creating account project foo
2017-01-16 06:58:17,872: cwl2sentry:ERROR Error processing account foo: 0
(cloud-custodian) $
```
[`f.exception()`](https://github.com/capitalone/cloud-custodian/blob/2cced870fd657e88dddffbedec15b7e82791c86a/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L445) is zero? That's unhelpful.
Throwing in a `print futures` suggests otherwise:

```
{<Future at 0x1022d7a50 state=finished raised KeyError>: {u'config_files': [], u'role': u'', u'name': u'c537n-whit537s-c7n-testbed', u'account_id': u'bar'}}
```
Ah!

```
(Pdb) l
441             for f in as_completed(futures):
442                 exc = f.exception()
443                 print futures
444                 if exc:
445                     import pdb; pdb.set_trace()
446  ->                 log.error("Error processing account %s: %s",
447                               a['name'], exc)
448     
449     
450     def setup_parser():
451         from common import setup_parser as orgparser
(Pdb) pp exc
KeyError(0,)
(Pdb)
```
Hmm ... still rather unhelpful. No traceback?
[No traceback](http://stackoverflow.com/questions/19309514/getting-original-line-number-for-exception-in-concurrent-futures).
[Fixed](https://bugs.python.org/issue21817) in Python 3.5?
/me drops back to single-threaded for now ...
There we go, proceeding ...

```
2017-01-16 07:45:04,728: cwl2sentry:INFO sentry endpoint: https://sentry.io/api/0/
2017-01-16 07:45:05,426: cwl2sentry:DEBUG processing c537n-whit537s-c7n-testbed
2017-01-16 07:45:05,426: cwl2sentry:INFO creating org team c537n-whit537s-c7n
2017-01-16 07:45:05,676: cwl2sentry:INFO creating account project c537n-whit537s-c7n-testbed
Traceback (most recent call last):
  File "/Users/whit537/workbench/capitalone/cloud-custodian/bin/c7n-sentry", line 11, in <module>
    load_entry_point('c7n-sentry', 'console_scripts', 'c7n-sentry')()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 4n
    options.command(options)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 4y
    process_account(a)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 4t
    dsn = result[0]['dsn']['secret']
KeyError: 0
(cloud-custodian) $
```
What, conceptually, is an "account"? That is an AWS account, ya?
But it looks like the name is arbitrary? It's used to create Sentry containers (team, project)?

```json
{
    "": {
        "name": "testing",
        "account_id": "custodian-skunk-works",
        "config_files": [],
        "role": ""
    }
}
```

```
2017-01-17 04:35:04,783: cwl2sentry:INFO sentry endpoint: https://sentry.io/api/0/
2017-01-17 04:35:05,424: cwl2sentry:DEBUG processing testing
2017-01-17 04:35:05,424: cwl2sentry:INFO creating org team testing
2017-01-17 04:35:05,697: cwl2sentry:INFO creating account project testing
> /Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py(404)process_ac)
-> dsn = result[0]['dsn']['secret']
(Pdb) result
{u'detail': u''}
(Pdb) result[0]
*** KeyError: 0
(Pdb)
```
Actually, looks like name is not arbitrary, but rather is set in AWS, along with an account id.

![screen shot 2017-01-17 at 4 40 41 am](https://cloud.githubusercontent.com/assets/134455/22015225/248176b4-dc6f-11e6-8b55-7fec52e33832.png)

![screen shot 2017-01-17 at 4 42 23 am](https://cloud.githubusercontent.com/assets/134455/22015279/5cfa0182-dc6f-11e6-85bf-47956f87317e.png)

```
2017-01-17 04:39:19,440: cwl2sentry:DEBUG processing custodian-skunk-works
2017-01-17 04:39:19,440: cwl2sentry:INFO creating org team custodian-skunk
2017-01-17 04:39:20,374: cwl2sentry:INFO creating account project custodian-skunk-works
```
Alright, so why are we [trying to get a key](https://github.com/capitalone/cloud-custodian/blob/2cced870fd657e88dddffbedec15b7e82791c86a/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L402-L404) from a ([potentially](https://github.com/capitalone/cloud-custodian/blob/2cced870fd657e88dddffbedec15b7e82791c86a/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L396)) [newly created project](https://github.com/capitalone/cloud-custodian/blob/2cced870fd657e88dddffbedec15b7e82791c86a/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L398-L400) rather than using the one that we [had to provide](https://github.com/capitalone/cloud-custodian/blob/2cced870fd657e88dddffbedec15b7e82791c86a/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L375) in order to create the project in the first place?
Reusing the existing `dsn` seems promising. Not sure why we're recreating team/project in sentry rather than reusing, tho ...

```
2017-01-17 05:00:53,845: cwl2sentry:INFO sentry endpoint: https://sentry.io/api/0/
2017-01-17 05:00:54,470: cwl2sentry:DEBUG processing custodian-skunk-works
2017-01-17 05:00:54,470: cwl2sentry:INFO creating org team custodian-skunk
2017-01-17 05:00:54,894: cwl2sentry:INFO creating account project custodian-skunk-works
2017-01-17 05:00:55,543: cwl2sentry:DEBUG Fetching hub instance policy errors for custodian-skunk-works
2017-01-17 05:00:55,951: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Tr}
2017-01-17 05:00:57,265: cwl2sentry:WARNING Could not process custodian-skunk-works region us-east-1 error: A.
2017-01-17 05:00:57,265: cwl2sentry:DEBUG Fetching spoke lambda policy errors for custodian-skunk-works
Traceback (most recent call last):
  File "/Users/whit537/workbench/capitalone/cloud-custodian/bin/c7n-sentry", line 11, in <module>
    load_entry_point('c7n-sentry', 'console_scripts', 'c7n-sentry')()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 4n
    options.command(options)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 4y
    process_account(a)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 4t
    for fname, config in a['config_files'].items():
AttributeError: 'list' object has no attribute 'items'
(cloud-custodian) $
```

```diff
diff --git a/tools/c7n_sentry/c7n_sentry/c7nsentry.py b/tools/c7n_sentry/c7n_sentry/c7nsentry.py
index 362381e..96fa22f 100644
--- a/tools/c7n_sentry/c7n_sentry/c7nsentry.py
+++ b/tools/c7n_sentry/c7n_sentry/c7nsentry.py
@@ -398,10 +398,6 @@ def orgreplay(options):
                 options.sentry_org, team_name),
                   json={'name': a['name']})
 
-        result = sget(endpoint + "projects/%s/%s/keys/" % (
-            options.sentry_org, a['name'])).json()
-        dsn = result[0]['dsn']['secret']
-
         bagger = partial(
             Bag,
             profile=options.profile, role=None, log_streams=None,
@@ -434,6 +430,10 @@ def orgreplay(options):
                                     a['name'], r, b.log_group)
                             continue
 
+    for a in accounts:
+        process_account(a)
+    return
+
     with ThreadPoolExecutor(max_workers=3) as w:
         futures = {}
         for a in accounts:

```
Whoa! First successful execution! :-)

```
(cloud-custodian) $ c7n-sentry orgreplay -c config.json --sentry-dsn=foo --sentry-org=c537n --sentry-token=deadbeef
2017-01-17 05:07:47,322: cwl2sentry:INFO sentry endpoint: https://sentry.io/api/0/
2017-01-17 05:07:49,385: cwl2sentry:DEBUG processing custodian-skunk-works
2017-01-17 05:07:49,386: cwl2sentry:INFO creating org team custodian-skunk
2017-01-17 05:07:49,698: cwl2sentry:INFO creating account project custodian-skunk-works
2017-01-17 05:07:49,965: cwl2sentry:DEBUG Fetching hub instance policy errors for custodian-skunk-works
2017-01-17 05:07:50,199: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Tr}
2017-01-17 05:07:50,914: cwl2sentry:WARNING Could not process custodian-skunk-works region us-east-1 error: A.
2017-01-17 05:07:50,915: cwl2sentry:DEBUG Fetching spoke lambda policy errors for custodian-skunk-works
2017-01-17 05:07:50,915: cwl2sentry:DEBUG Fetching hub instance policy errors for custodian-skunk-works
2017-01-17 05:07:51,076: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Tr}
2017-01-17 05:07:52,036: cwl2sentry:WARNING Could not process custodian-skunk-works region us-west-2 error: A.
2017-01-17 05:07:52,036: cwl2sentry:DEBUG Fetching spoke lambda policy errors for custodian-skunk-works
(cloud-custodian) $ cat config.json
```

```json
{
    "": {
        "name": "custodian-skunk-works",
        "account_id": "xxxxxxxxxx",
        "config_files": {},
        "role": ""
    }
}
```

```diff
diff --git a/tools/c7n_sentry/c7n_sentry/c7nsentry.py b/tools/c7n_sentry/c7n_sentry/c7nsentry.py
index 362381e..96fa22f 100644
--- a/tools/c7n_sentry/c7n_sentry/c7nsentry.py
+++ b/tools/c7n_sentry/c7n_sentry/c7nsentry.py
@@ -398,10 +398,6 @@ def orgreplay(options):
                 options.sentry_org, team_name),
                   json={'name': a['name']})
 
-        result = sget(endpoint + "projects/%s/%s/keys/" % (
-            options.sentry_org, a['name'])).json()
-        dsn = result[0]['dsn']['secret']
-
         bagger = partial(
             Bag,
             profile=options.profile, role=None, log_streams=None,
@@ -434,6 +430,10 @@ def orgreplay(options):
                                     a['name'], r, b.log_group)
                             continue
 
+    for a in accounts:
+        process_account(a)
+    return
+
     with ThreadPoolExecutor(max_workers=3) as w:
         futures = {}
         for a in accounts:
```
Works with `ThreadPoolExecutor`, too.

```
2017-01-17 05:09:47,513: cwl2sentry:INFO sentry endpoint: https://sentry.io/api/0/
2017-01-17 05:09:48,821: cwl2sentry:DEBUG processing custodian-skunk-works
2017-01-17 05:09:48,821: cwl2sentry:INFO creating org team custodian-skunk
2017-01-17 05:09:49,084: cwl2sentry:INFO creating account project custodian-skunk-works
2017-01-17 05:09:49,339: cwl2sentry:DEBUG Fetching hub instance policy errors for custodian-skunk-works
2017-01-17 05:09:49,568: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Tr}
2017-01-17 05:09:50,253: cwl2sentry:WARNING Could not process custodian-skunk-works region us-east-1 error: A.
2017-01-17 05:09:50,253: cwl2sentry:DEBUG Fetching spoke lambda policy errors for custodian-skunk-works
2017-01-17 05:09:50,254: cwl2sentry:DEBUG Fetching hub instance policy errors for custodian-skunk-works
2017-01-17 05:09:50,428: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Tr}
2017-01-17 05:09:51,313: cwl2sentry:WARNING Could not process custodian-skunk-works region us-west-2 error: A.
2017-01-17 05:09:51,314: cwl2sentry:DEBUG Fetching spoke lambda policy errors for custodian-skunk-works
```

```diff
diff --git a/tools/c7n_sentry/c7n_sentry/c7nsentry.py b/tools/c7n_sentry/c7n_sentry/c7nsentry.py
index 362381e..d014639 100644
--- a/tools/c7n_sentry/c7n_sentry/c7nsentry.py
+++ b/tools/c7n_sentry/c7n_sentry/c7nsentry.py
@@ -398,10 +398,6 @@ def orgreplay(options):
                 options.sentry_org, team_name),
                   json={'name': a['name']})
 
-        result = sget(endpoint + "projects/%s/%s/keys/" % (
-            options.sentry_org, a['name'])).json()
-        dsn = result[0]['dsn']['secret']
-
         bagger = partial(
             Bag,
             profile=options.profile, role=None, log_streams=None,

```
Alright! So what are config files? Do I need them? Now that it works for an empty case, how do I get it to work for a _non_-empty case? I want to see a log message in Sentry!
Presumably they are c7n config files?
Ouch. Looks like we [don't actually use](https://github.com/capitalone/cloud-custodian/blob/2cced870fd657e88dddffbedec15b7e82791c86a/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L424-L436) the filename of the config file (and I'm not sure we [ever intended](https://github.com/capitalone/cloud-custodian/pull/852/commits/300f00615ec82908ad2535095b54dccc7580f1e0#diff-f85c25418e2f1f560460c7f2cf6140e6L51) to), but rather config stored in `config.json` under `config_files`. 😳
Found a working `config_files` format:

```json
{
    "": {
        "name": "custodian-skunk-works",
        "account_id": "xxxxxxxxxx",
        "config_files": {
            "": {
                "policies": [
                    {
                        "mode": "foo",
                        "name": "bar"
                    }
                ]
            }
        },
        "role": ""
    }
}
```

```
(cloud-custodian) $ export SENTRY_DSN=foo
(cloud-custodian) $ export SENTRY_TOKEN=bar
(cloud-custodian) $ c7n-sentry --verbose orgreplay -c config.json --sentry-org=c537n
2017-01-17 05:57:51,476: cwl2sentry:INFO sentry endpoint: https://sentry.io/api/0/
2017-01-17 05:57:52,587: cwl2sentry:DEBUG processing custodian-skunk-works
2017-01-17 05:57:52,587: cwl2sentry:INFO creating org team custodian-skunk
2017-01-17 05:57:52,849: cwl2sentry:INFO creating account project custodian-skunk-works
2017-01-17 05:57:53,114: cwl2sentry:DEBUG Fetching hub instance policy errors for custodian-skunk-works
2017-01-17 05:57:53,249: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Traceback', 'logGroupName': u'/cloud-custodian/custodian-skunk-works/us-east-1'}
2017-01-17 05:57:53,831: cwl2sentry:WARNING Could not process custodian-skunk-works region us-east-1 error: An error occurred (ResourceNotFoundException) when calling the FilterLogEvents operation: The specified log group does not exist.
2017-01-17 05:57:53,832: cwl2sentry:DEBUG Fetching spoke lambda policy errors for custodian-skunk-works
2017-01-17 05:57:53,898: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Traceback', 'logGroupName': u'/aws/lambda/custodian-bar'}
2017-01-17 05:57:54,070: cwl2sentry:INFO account: custodian-skunk-works region: us-east-1 group: /aws/lambda/custodian-bar not found
2017-01-17 05:57:54,070: cwl2sentry:DEBUG Fetching hub instance policy errors for custodian-skunk-works
2017-01-17 05:57:54,244: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Traceback', 'logGroupName': u'/cloud-custodian/custodian-skunk-works/us-west-2'}
2017-01-17 05:57:55,236: cwl2sentry:WARNING Could not process custodian-skunk-works region us-west-2 error: An error occurred (ResourceNotFoundException) when calling the FilterLogEvents operation: The specified log group does not exist.
2017-01-17 05:57:55,236: cwl2sentry:DEBUG Fetching spoke lambda policy errors for custodian-skunk-works
2017-01-17 05:57:55,335: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Traceback', 'logGroupName': u'/aws/lambda/custodian-bar'}
2017-01-17 05:57:55,797: cwl2sentry:INFO account: custodian-skunk-works region: us-west-2 group: /aws/lambda/custodian-bar not found
(cloud-custodian) $
```
I guess now it's time to make a Lambda that periodically raises heck.
Okay! Now to replay it ...

![screen shot 2017-01-17 at 6 06 47 am](https://cloud.githubusercontent.com/assets/134455/22018114/2b0a45d6-dc7b-11e6-8107-61fa037b7f34.png)

Progress!

```
(cloud-custodian) $ c7n-sentry --verbose orgreplay -c config.json --sentry-org=c537n
2017-01-17 06:07:23,554: cwl2sentry:INFO sentry endpoint: https://sentry.io/api/0/
2017-01-17 06:07:24,712: cwl2sentry:DEBUG processing custodian-skunk-works
2017-01-17 06:07:24,713: cwl2sentry:INFO creating org team custodian-skunk
2017-01-17 06:07:24,973: cwl2sentry:INFO creating account project custodian-skunk-works
2017-01-17 06:07:25,227: cwl2sentry:DEBUG Fetching hub instance policy errors for custodian-skunk-works
2017-01-17 06:07:25,410: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Traceback', 'logGroupName': u'/cloud-custodian/custodian-skunk-works/us-east-1'}
2017-01-17 06:07:26,102: cwl2sentry:WARNING Could not process custodian-skunk-works region us-east-1 error: An error occurred (ResourceNotFoundException) when calling the FilterLogEvents operation: The specified log group does not exist.
2017-01-17 06:07:26,102: cwl2sentry:DEBUG Fetching spoke lambda policy errors for custodian-skunk-works
2017-01-17 06:07:26,168: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Traceback', 'logGroupName': u'/aws/lambda/custodian-bar'}
2017-01-17 06:07:26,312: cwl2sentry:INFO account: custodian-skunk-works region: us-east-1 group: /aws/lambda/custodian-bar not found
2017-01-17 06:07:26,312: cwl2sentry:DEBUG Fetching hub instance policy errors for custodian-skunk-works
2017-01-17 06:07:26,376: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Traceback', 'logGroupName': u'/cloud-custodian/custodian-skunk-works/us-west-2'}
2017-01-17 06:07:27,354: cwl2sentry:WARNING Could not process custodian-skunk-works region us-west-2 error: An error occurred (ResourceNotFoundException) when calling the FilterLogEvents operation: The specified log group does not exist.
2017-01-17 06:07:27,354: cwl2sentry:DEBUG Fetching spoke lambda policy errors for custodian-skunk-works
2017-01-17 06:07:27,455: cwl2sentry:DEBUG Querying log events with {'interleaved': True, 'filterPattern': 'Traceback', 'logGroupName': u'/aws/lambda/custodian-bar'}
Traceback (most recent call last):
  File "/Users/whit537/workbench/capitalone/cloud-custodian/bin/c7n-sentry", line 11, in <module>
    load_entry_point('c7n-sentry', 'console_scripts', 'c7n-sentry')()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 479, in main
    options.command(options)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 433, in orgreplay
    return [process_account(a) for a in accounts]
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 426, in process_account
    process_log_group(b)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 161, in process_log_group
    'logGroup': config.log_group}, client)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 258, in get_sentry_message
    'culprit': culprit,
UnboundLocalError: local variable 'culprit' referenced before assignment
(cloud-custodian) $
```
Interesting ... we [assume](https://github.com/capitalone/cloud-custodian/blob/2cced870fd657e88dddffbedec15b7e82791c86a/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L229-L232) a traceback with `c7n` in it.
💃

![screen shot 2017-01-17 at 6 21 12 am](https://cloud.githubusercontent.com/assets/134455/22018511/2c46a3d4-dc7d-11e6-868f-3150f726e5b2.png)
That is with:

```python
def lambda_handler(event, context):
    raise heck
```

```json
{
    "": {
        "name": "custodian-skunk-works",
        "account_id": "xxxxxxxxxx",
        "config_files": {
            "": {
                "policies": [
                    {
                        "mode": "foo",
                        "name": "bar"
                    }
                ]
            }
        },
        "role": ""
    }
}
```

```diff
diff --git a/tools/c7n_sentry/c7n_sentry/c7nsentry.py b/tools/c7n_sentry/c7n_sentry/c7nsentry.py
index 362381e..5c8ebc2 100644
--- a/tools/c7n_sentry/c7n_sentry/c7nsentry.py
+++ b/tools/c7n_sentry/c7n_sentry/c7nsentry.py
@@ -227,8 +227,8 @@ def get_sentry_message(config, data, log_client=None, is_lambda=True):
         level, logger = 'ERROR', None
 
     for f in reversed(error['stacktrace']['frames']):
+        culprit = "%s.%s" % (f['module'], f['function'])
         if f['module'].startswith('c7n'):
-            culprit = "%s.%s" % (f['module'], f['function'])
             break
 
     breadcrumbs = None
@@ -371,7 +371,8 @@ def orgreplay(options):
     sget = partial(requests.get, headers=auth_headers)
     spost = partial(requests.post, headers=auth_headers)
 
-    dsn = urlparse.urlparse(options.sentry_dsn)
+    unparsed_dsn = options.sentry_dsn
+    dsn = urlparse.urlparse(unparsed_dsn)
     endpoint = "%s://%s/api/0/" % (
         dsn.scheme,
         "@" in dsn.netloc and dsn.netloc.rsplit('@', 1)[1] or dsn.netloc)
@@ -398,14 +399,10 @@ def orgreplay(options):
                 options.sentry_org, team_name),
                   json={'name': a['name']})
 
-        result = sget(endpoint + "projects/%s/%s/keys/" % (
-            options.sentry_org, a['name'])).json()
-        dsn = result[0]['dsn']['secret']
-
         bagger = partial(
             Bag,
             profile=options.profile, role=None, log_streams=None,
-            start=options.start, end=options.end, sentry_dsn=dsn,
+            start=options.start, end=options.end, sentry_dsn=unparsed_dsn,
             account_id=a['account_id'],
             account_name=a['name'])
 
@@ -434,6 +431,8 @@ def orgreplay(options):
                                     a['name'], r, b.log_group)
                             continue
 
+    return [process_account(a) for a in accounts]
+
     with ThreadPoolExecutor(max_workers=3) as w:
         futures = {}
         for a in accounts:
```
Alright, now to deploy [`process_log_event`](https://github.com/capitalone/cloud-custodian/blob/2cced870fd657e88dddffbedec15b7e82791c86a/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L95-L114) as a lambda itself.
Well, maybe a PR first to clean up these bugs ...
re culprit, its basically used by sentry to highlight the interesting line, original use here was limited to custodian, allowing for specification of app/package module might be more appropriate for general purpose.
Looks like the DSN get is a 404. Not sure what that was there for.

```
(Pdb) requests.get('https://sentry.io/api/0/projects/c537n/custodian-skunk-works/keys/', headers={'Authorization': 'Bearer ' + options.sentry_token})
<Response [404]>
(Pdb) requests.get('https://sentry.io/api/0/projects/c537n/custodian-skunk-works/keys/', headers={'Authorization': 'Bearer ' + options.sentry_token}).json()
{u'detail': u''}
(Pdb)
```
I'm having a bit of trouble getting a second error to log. Could be due to aggregation in Sentry, or it could simply be because of insufficient timezone math. :)

![screen shot 2017-01-18 at 4 58 09 am](https://cloud.githubusercontent.com/assets/134455/22059467/e0b3e5f8-dd3a-11e6-9698-05128cc2da58.png)
The "5 hours ago — 5 hours old" just happened. I'm in UTC-5.
I've made a couple more commits to #852 with fixes for the culprit and dsn issues.
... and a README update.
Okay! Let's see if I can deploy `process_log_event` as a Lambda ...
😳

```
(cloud-custodian) $ c7n-sentry deploy -c config.json 
2017-01-19 06:17:16,571: custodian.lambda:INFO Publishing custodian policy lambda function cloud-custodian-sentry
(cloud-custodian) $
```
Hmm ... where did it go? 😆 
There it is! In `us-east-1`. ;-)
Easy enough to change, though a little confusing that `--region` appends to `.regions` vs. setting `.region`.

```
(cloud-custodian) $ c7n-sentry deploy -c config.json --region=us-west-2
2017-01-19 07:16:20,871: custodian.lambda:INFO Publishing custodian policy lambda function cloud-custodian-sentry
(cloud-custodian) $
```
Okay! I am into `process_log_event` in a Lambda with [this diff](https://gist.github.com/whit537/d3b737de4bdb7122f36738534724474f) (against ffe61e1cd85402d019bbb98b4152b140065569ab). 👍

Now to debug the function itself ...

```
'awslogs': KeyError
Traceback (most recent call last):
File "/var/task/c7n_sentry/c7nsentry.py", line 102, in process_log_event
serialized = event['awslogs'].pop('data')
KeyError: 'awslogs'
```
 Looks like maybe that's a result of ... running the function from test w/o going through CWE?
Is `process_log_event` the correct function to mount, or does it need a wrapper?
Do I have the plumbing right between the `custodian-bar` and `cloud-custodian-sentry` lambdas?
How do CloudWatch Events work? The `rate(1 minute)` thing makes it sound like we're polling instead of pushing? We don't receive one call per event, or ...?
http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html
> CloudWatch Events does not provide second-level precision in schedule expressions. The finest resolution using a Cron expression is a minute.

http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html

I'm not sure we're in a "schedule expression" situation, tho ...
Yeah, that's describing configuring an Event via CloudWatch, whereas we're adding a Trigger on the Lambda side. Still a schedule expression tho:

![screen shot 2017-01-20 at 5 12 19 am](https://cloud.githubusercontent.com/assets/134455/22145368/0a883678-decf-11e6-8726-da129c165aa8.png)

I mean, `process_log_event` has a [signature and docstring](https://github.com/capitalone/cloud-custodian/blob/2cced870fd657e88dddffbedec15b7e82791c86a/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L95-L96) that suggest that it _should_ be mountable as a lambda.
So the `custodian-bar` generates exceptions, which it logs to CloudWatch.

And then `cloud-custodian-sentry` is subscribed to CloudWatch Events (with a one-minute refresh). Is it picking up what `custodian-bar` is laying down?
It seems to be picking up _something_ ... what _is_ it picking up?
```python
{ u'account': u'644160558196'
, u'region': u'us-west-2'
, u'detail': {}
, u'detail-type': u'Scheduled Event'
, u'source': u'aws.events'
, u'version': u'0'
, u'time': u'2017-01-20T10:52:41Z'
, u'id': u'371ffaa0-f307-487e-896c-31c8f5f35d48'
, u'resources': [u'arn:aws:events:us-west-2:644160558196:rule/cloud-custodian-sentry']
 }
its really hard to comment on this stream of conscience ;-)  nutshell sentry func wants an event from cloud watch logs when its put in place as a log subscription filter, it looks like you've set it up as a cwe scheduled func which isn't really the same thing. let me know if would be helpful to catch up for a few minutes on gitter or hangout.
@kapilt just sorted me on a quick video call. `CloudWatchLogSubscription` should give us realtime, there's already code to use it in [`get_function`](https://github.com/capitalone/cloud-custodian/blob/2cced870fd657e88dddffbedec15b7e82791c86a/tools/c7n_sentry/c7n_sentry/c7nsentry.py#L320-L363).
Started reconciling my local changes w/ `get_function` as:

https://gist.github.com/whit537/92dd432917e31bfa07dd36b3885e4cf2

Hitting:

```
(cloud-custodian) $ c7n-sentry deploy -c config.json --region=us-west-2
Traceback (most recent call last):
  File "/Users/whit537/workbench/capitalone/cloud-custodian/bin/c7n-sentry", line 11, in <module>
    load_entry_point('c7n-sentry', 'console_scripts', 'c7n-sentry')()
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 508, in main
    options.command(options)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 464, in deploy
    account_id=None
  File "/Users/whit537/workbench/capitalone/cloud-custodian/tools/c7n_sentry/c7n_sentry/c7nsentry.py", line 373, in get_function
    return LambdaFunction(config, archive)
  File "/Users/whit537/workbench/capitalone/cloud-custodian/c7n/mu.py", line 508, in __init__
    raise ValueError("Missing required keys %s" % " ".join(missing))
ValueError: Missing required keys handler
(cloud-custodian) $
```

Will pick up there in a bit ...

Resolving patch:
diff --git a/tools/c7n_sentry/c7n_sentry/c7nsentry.py b/tools/c7n_sentry/c7n_sentry/c7nsentry.py
--- a/tools/c7n_sentry/c7n_sentry/c7nsentry.py
+++ b/tools/c7n_sentry/c7n_sentry/c7nsentry.py
@@ -76,7 +76,7 @@
 
 VERSION = "0.1"
 
-log = logging.getLogger("cwl2sentry")
+log = logging.getLogger("c7n-sentry")
 
 
 def init():
@@ -102,12 +102,12 @@ def process_log_event(event, context):
     serialized = event['awslogs'].pop('data')
     data = json.loads(zlib.decompress(
         base64.b64decode(serialized), 16+zlib.MAX_WBITS))
-    msg = get_sentry_message(data)
+    msg = get_sentry_message(config, data)
     if msg is None:
         return
     if config['sentry_dsn']:
         # Deliver directly to sentry
-        send_sentry_message(config, msg)
+        send_sentry_message(config['sentry_dsn'], msg)
     elif config['sentry_sqs']:
         # Delivery indirectly via sqs
         sqs.send_message(
@@ -317,24 +317,22 @@ def parse_traceback(msg, site_path="site-packages", in_app_prefix="c7n"):
         'stacktrace': data}
 
 
-def get_function(session_factory, name, role,
+def get_function(session_factory, name, handler, role,
                  log_groups,
                  project, account_name, account_id,
+                 sentry_dsn,
                  pattern="Traceback"):
     """Lambda function provisioning.
 
     Self contained within the component, to allow for easier reuse.
     """
-
     # Lazy import to avoid runtime dependency
-    import inspect
-
-    import c7n
     from c7n.mu import (
         LambdaFunction, PythonPackageArchive, CloudWatchLogSubscription)
 
     config = dict(
         name=name,
+        handler=handler,
         runtime='python2.7',
         memory_size=512,
         timeout=15,
@@ -345,18 +343,22 @@ def get_function(session_factory, name, role,
                 session_factory, log_groups, pattern)])
 
     archive = PythonPackageArchive(
-        # Directory to lambda file
-        os.path.join(
-            os.path.dirname(inspect.getabsfile(c7n)), 'ufuncs', 'cwl2sentry.py'),
-        # Don't include virtualenv deps
+        os.path.dirname(__file__),
+        skip='*.pyc',
         lib_filter=lambda x, y, z: ([], []))
+
     archive.create()
     archive.add_contents(
         'config.json', json.dumps({
             'project': project,
             'account_name': account_name,
-            'account_id': account_id
+            'account_id': account_id,
+            'sentry_dsn': sentry_dsn,
         }))
+    archive.add_contents(
+        'handler.py',
+        'from c7n_sentry.c7nsentry import process_log_event'
+    )
     archive.close()
 
     return LambdaFunction(config, archive)
@@ -430,6 +432,9 @@ def process_account(a):
                                     a['name'], r, b.log_group)
                             continue
 
+    return [process_account(a) for a in accounts]
+
+
     with ThreadPoolExecutor(max_workers=3) as w:
         futures = {}
         for a in accounts:
@@ -440,15 +445,48 @@ def process_account(a):
                 log.error("Error processing account %s: %r", a['name'], exc)
 
 
+def deploy(options):
+    from common import get_accounts
+    for account in get_accounts(options):
+        for region_name in options.regions:
+            for fname, config in account['config_files'].items():
+                for policy in config.get('policies', ()):
+                    if policy.get('mode'):
+                        deploy_one(
+                            region_name, account, policy, options.sentry_dsn)
+
+
+def deploy_one(region_name, account, policy, sentry_dsn):
+    from c7n.mu import LambdaManager
+    session_factory = lambda: boto3.Session(region_name=region_name)
+    log_group_name = '/aws/lambda/custodian-{}'.format(policy['name'])
+    arn = 'arn:aws:logs:{}:{}:log-group:{}:*'.format(
+        region_name, account['account_id'], log_group_name)
+    function = get_function(
+        session_factory=session_factory,
+        name='cloud-custodian-sentry',
+        handler='handler.process_log_event',
+        role=account['role'],
+        log_groups=[{'logGroupName': log_group_name, 'arn': arn}],
+        project=None,
+        account_name=account['name'],
+        account_id=account['account_id'],
+        sentry_dsn=sentry_dsn,
+    )
+    log.info("Deploying lambda for {} in {}".format(
+        log_group_name, region_name))
+    LambdaManager(session_factory).publish(function)
+
+
 def setup_parser():
-    from common import setup_parser as orgparser
+    from common import setup_parser as common_parser
 
     parser = argparse.ArgumentParser()
     parser.add_argument('--verbose', default=False, action="store_true")
     subs = parser.add_subparsers()
 
     cmd_orgreplay = subs.add_parser('orgreplay')
-    orgparser(cmd_orgreplay)
+    common_parser(cmd_orgreplay)
     cmd_orgreplay.set_defaults(command=orgreplay)
     cmd_orgreplay.add_argument('--profile')
     #cmd_orgreplay.add_argument('--role')
@@ -460,6 +498,12 @@ def setup_parser():
     cmd_orgreplay.add_argument('--sentry-token',
                                default=os.environ.get('SENTRY_TOKEN'))
 
+    cmd_deploy = subs.add_parser('deploy')
+    common_parser(cmd_deploy)
+    cmd_deploy.add_argument('--sentry-dsn',
+                            default=os.environ.get('SENTRY_DSN'))
+    cmd_deploy.set_defaults(command=deploy)
+
     return parser
 
 


Test patch: