Package: c7n-mailer
c7n-mailer: Custodian Mailer% [comment]: # (         !!! IMPORTANT !!!                    )
% [comment]: # (This file is moved during document generation.)
% [comment]: # (Only edit the original document at ./tools/c7n_mailer/README.md)A mailer implementation for Custodian. Outbound mail delivery is still somewhat
organization-specific, so this at the moment serves primarily as an example
implementation.The Cloud Custodian Mailer can now be easily run in a Docker container. Clickherefor details.Message RelayCustodian Mailer subscribes to an SQS queue, looks up users, and sends emails
via SES, Slack messages, and/or notifications to DataDog or Splunk. Custodian Lambda and instance policies can send messages to the SQS queue. SQS queues
should be cross-account enabled for sending between accounts.TutorialOur goal in starting out with the Custodian mailer is to install the mailer,
and run a policy that triggers an email to your inbox.Installthe mailer on your laptop (if you are not running as aDocker containeror usepip install c7n-mailerIn your text editor, create amailer.ymlfile to hold your mailer config.In the AWS console, create a new standard SQS queue (quick create is fine).
Copy the queue URL toqueue_urlinmailer.yml.In AWS, locate or create a role that has read access to the queue. Grab the
role ARN and set it asroleinmailer.yml.There are different notification endpoints options, you can combine both.Email:Make sure your email address is verified in SES, and set it asfrom_addressinmailer.yml. By default SES is in sandbox mode where you
mustverifyevery individual recipient of emails. If need be, make an AWS support ticket to
be taken out of SES sandbox mode.Yourmailer.ymlshould now look something like this:queue_url:https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-testrole:arn:aws:iam::123456790:role/c7n-mailer-testfrom_address:you@example.comYou can also setregionif you are in a region other thanus-east-1as well aslambda_tagsto give the mailer tags.region:us-east-2lambda_tags:owner:opsNow let's make a Custodian policy to populate your mailer queue. Create atest-policy.ymlfile with this content (updatetoandqueueto match your
environment)policies:-name:c7n-mailer-testresource:sqsfilters:-"tag:MailerTest":absentactions:-type:notifytemplate:defaultpriority_header:'2'subject:testing the c7n mailerto:-you@example.comtransport:type:sqsqueue:https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-testDataDog:The standard way to do a DataDog integration is use the
c7n integration with AWS CloudWatch and use theDataDog integration with AWSto collect CloudWatch metrics. The mailer/messenger integration is only
for the case you don't want or you can't use AWS CloudWatch, e.g. in Azure or GCP.Note this integration requires the additional dependency of Datadog Python bindings:pip install datadogYourmailer.ymlshould now look something like this:queue_url:https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-testrole:arn:aws:iam::123456790:role/c7n-mailer-testdatadog_api_key:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXdatadog_application_key:YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY(Also setregionif you are in a region other thanus-east-1.)Now, let's make a Custodian policy to populate your mailer queue. Create atest-policy.yml:policies:-name:c7n-mailer-testresource:ebsfilters:-Attachments:[]actions:-type:notifyto:-datadog://?metric_name=datadog.metric.name&metric_value_tag=Sizetransport:type:sqsqueue:https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-testThere is a specialtoformat that specifies Datadog delivery and it includes the Datadog configuration via URL parameters:metric_name: is the name of the metrics send to DataDog.metric_value_tag: by default the metric value to send to DataDog is1, but if you want to use one of the tags returned in the policy, you can set it with the attributemetric_value_tag. For example, in thetest-policy.ymlpolicy, the value used is the size of the EBS volume. The value must be a number and it will be transformed to a float value.Slack:The Custodian mailer supports Slack messaging as a separate notification mechanism for the SQS transport method. To enable Slack integration, you must specify a Slack token in theslack_tokenfield in yourmailer.ymlfile. For example:queue_url:https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-testrole:arn:aws:iam::123456790:role/c7n-mailer-testslack_token:xoxo-token123To enable Slack messaging, several unique fields are evaluated in the policy, as shown in the below example:policies:
  - name: c7n-mailer-test
    resource: ebs
    filters:
     - Attachments: []
    actions:
      - type: notify
        slack_template: slack
        slack_msg_color: danger
        to:
          - slack://owners
          - slack://foo@bar.com
          - slack://#custodian-test
          - slack://webhook/#c7n-webhook-test
          - slack://tag/resource_tag
          - https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-testSlack messages support the use of a unique template field specified inslack_template. This field is unique and usage will not break
existing functionality for messages also specifying an email template in thetemplatefield. This field is optional, however,
and if not specified, the mailer will use the default valueslack_default.The unique template fieldslack_msg_colorcan be used to specify a color
border for the Slack message. This accepts the Slack presets ofdanger(red),warning(yellow) andgood(green). It can also accept an HTML hex code. See
theSlack documentationfor details.Note:if you are using a hex color code, it will need to be wrapped in quotes
like so:slack_msg_color: '#4287f51'. Otherwise, the YAML interpreter will consider it acomment.The Slack integration for c7n-mailer supports several flavors of messaging, listed below. These are not mutually exclusive and any combination of the types can be used, but the preferred method isincoming webhooks.Requiresslack_tokenKeyTypeNotesNohttps://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXXstring(PREFERRED)Send to anincoming webhook(the channel is defined in the webhook)Yesslack://ownersstringSend to the recipient list generated within email delivery logicYesslack://foo@bar.comstringSend to the recipient specified by email addressfoo@bar.comYesslack://#custodian-teststringSend to the Slack channel indicated in string, i.e. #custodian-testNoslack://webhook/#c7n-webhook-teststring(DEPRECATED)Send to a Slack webhook; appended with the target channel.IMPORTANT:This requires aslack_webhookvalue defined in themailer.yml.Yesslack://tag/resource-tagstringSend to target found in resource tag. Example of value in tag:https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXXSlack delivery can also be set via a resource's tag name. For example, usingslack://tag/slack_channelwill look for a tag name ofslack_channel, and if matched on a resource will deliver the message to the value of that resource's tag:slack_channel:https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXXslack_channel:custodian-testowner:foo@barDelivery via tag has been tested with webhooks, but it should support all delivery methods.Splunk HTTP Event Collector (HEC)The Custodian mailer supports delivery to the HTTP Event Collector (HEC) endpoint of a Splunk instance as a separate notification mechanism for the SES transport method. To enable Splunk HEC integration, you must specify the URL to the HEC endpoint as well as a valid username and token:queue_url:https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-testrole:arn:aws:iam::123456790:role/c7n-mailer-testsplunk_hec_url:https://http-inputs-foo.splunkcloud.com/services/collector/eventsplunk_hec_token:268b3cc2-f32e-4a19-a1e8-aee08d86ca7fTo send events for a policy to the Splunk HEC endpoint, add atoaddress notify action specifying the name of the Splunk index to send events to in the formsplunkhec://indexName:policies:
  - name: c7n-mailer-test
    resource: ebs
    filters:
     - Attachments: []
    actions:
      - type: notify
        to:
          - splunkhec://myIndexName
        transport:
          type: sqs
          queue: https://sqs.us-east-1.amazonaws.com/1234567890/c7n-mailer-testThesplunkhec://indexNameaddress type can be combined in the same notify action with other destination types (e.g. email, Slack, DataDog, etc).Now run:c7n-mailer --config mailer.yml --update-lambda && custodian run -c test-policy.yml -s .Note: You can set the profile via environment variable e.g.export AWS_DEFAULT_PROFILE=fooYou should see output similar to the following:(env) $ c7n-mailer --config mailer.yml --update-lambda && custodian run -c test-policy.yml -s .
DEBUG:custodian.lambda:Created custodian lambda archive size: 3.01mb
2017-01-12 07:55:16,227: custodian.policy:INFO Running policy c7n-mailer-test resource: sqs region:default c7n:0.8.22.0
2017-01-12 07:55:16,229: custodian.policy:INFO policy: c7n-mailer-test resource:sqs has count:1 time:0.00
2017-01-12 07:55:18,017: custodian.actions:INFO sent message:dead-beef policy:c7n-mailer-test template:default count:1
2017-01-12 07:55:18,017: custodian.policy:INFO policy: c7n-mailer-test action: notify resources: 1 execution_time: 1.79
(env) $Check the AWS console for a new Lambda namedcloud-custodian-mailer. The
mailer runs every five minutes, so wait a bit and then look for an email in
your inbox. If it doesn't appear, look in the lambda's logs for debugging
information. If it does, congratulations! You are off and running with the
Custodian mailer.Usage & ConfigurationOnceinstalledyou should have ac7n-mailerexecutable on your path:(env) $ c7n-mailer
usage: c7n-mailer [-h] -c CONFIG
c7n-mailer: error: argument -c/--config is required
(env) $Fundamentally, whatc7n-mailerdoes is it deploys a Lambda function (usingMu) based on
the configuration you specify in a YAML file.  Here isthe
schemato which the file must conform
and here is a description of the options:Required?KeyTypeNotes✅queue_urlstringthe queue to listen to for messagesfrom_addressstringdefault from addressendpoint_urlstringSQS API URL (for use with VPC Endpoints)contact_tagsarray of stringstags that we should look at for address informationemail_base_urlstringBase URL to construct a valid email address from a resource owneradditional_email_headersobjectAdditional email headers.org_domainstringDomain to add to AWS usernames to construct email addresses.Standard Lambda Function ConfigRequired?KeyType✅rolestringdead_letter_configobjectmemoryintegerregionstringruntimestringsecurity_groupsarray of stringssubnetsarray of stringstimeoutintegerlambda_namestringlambda_descriptionstringlambda_tagsobjectlambda_schedulestringStandard Azure Functions ConfigRequired?KeyTypeNotesfunction_propertiesobjectContainsappInsights,storageAccountandservicePlanobjectsappInsightsobjectContainsname,locationandresourceGroupNamepropertiesstorageAccountobjectContainsname,locationandresourceGroupNamepropertiesservicePlanobjectContainsname,location,resourceGroupName,skuTierandskuNamepropertiesnamestringlocationstringDefault:west us 2resourceGroupNamestringDefaultcloud-custodianskuTierstringDefault:BasicskuNamestringDefault:B1Mailer Infrastructure ConfigRequired?KeyTypeNotescache_enginestringcache engine; either sqlite or rediscross_accountsobjectaccount to assume back into for sending to SNS topicsdebugbooleandebug on/offldap_bind_dnstringeg: ou=people,dc=example,dc=comldap_bind_userstringeg: FOO\BARldap_bind_passwordsecured stringldap bind passwordldap_bind_password_in_kmsbooleandefaults to true, most people (except capone) want to set this to false. If set to true, make sureldap_bind_passwordcontains your KMS encrypted ldap bind password as a base64-encoded string.ldap_email_attributestringldap_email_keystringeg 'mail'ldap_manager_attributestringeg 'manager'ldap_uid_attributestringldap_uid_regexstringldap_uid_tagsstringldap_uristringeg 'ldaps://example.com:636'redis_hoststringredis host if cache_engine == redisredis_portintegerredis port, default: 6369ses_regionstringAWS region that handles SES API callsses_rolestringARN of the role to assume to send email with SESSMTP ConfigRequired?KeyTypeNotessmtp_serverstringto configure your lambda role to talk to smtpd in your private vpc, seeheresmtp_portintegersmtp port (default is 25)smtp_sslbooleanthis defaults to Truesmtp_usernamestringsmtp_passwordsecured stringIfsmtp_serveris unset,c7n_mailerwill use AWS SES or Azure SendGrid.DataDog ConfigRequired?KeyTypeNotesdatadog_api_keystringDataDog API key.datadog_application_keystringDatadog application key.These fields are not necessary if c7n_mailer is run in a instance/lambda/etc with the DataDog agent.Slack ConfigRequired?KeyTypeNotesslack_tokenstringSlack API tokenSendGrid ConfigRequired?KeyTypeNotessendgrid_api_keysecured stringSendGrid API tokenMicrosoft Graph ConfigRequired?KeyTypeNotesgraph_token_endpointstringGraph token endpointgraph_sendmail_endpointstringGraph sendmail endpointgraph_client_idstringClient IDgraph_client_secretsecured stringClient secretSplunk HEC ConfigThe following configuration items arealloptional. The ones marked "Required for Splunk" are only required if you're sending notifications tosplunkhec://destinations.Required for Splunk?KeyTypeNotes✅splunk_hec_urlstringURL to your Splunk HTTP Event Collector endpoint✅splunk_hec_tokenstringSplunk HEC authentication token for specified usernamesplunk_remove_pathsarray of stringsList ofRFC6901JSON Pointers to remove from the event, if present, before sending to Splunksplunk_actions_listbooleanIf true, add anactionslist to the top-level message sent to Splunk, containing the names of all non-notify actions takensplunk_max_attemptsintegerMaximum number of times to try POSTing data to Splunk HEC (default 4)splunk_hec_max_lengthintegerMaximum data length that Splunk HEC accepts; an error will be logged for any message sent over this lengthsplunk_hec_sourcetypestringConfigure sourcetype of the payload sent to Splunk HEC. (default is '_json')SDK ConfigRequired?KeyTypeNoteshttp_proxystringhttps_proxystringprofilestringSecured StringIn order to ensure sensitive data is not stored plaintext in a policy,c7n-mailersupports secured
strings. You can treat it as a regularstringor usesecured stringfeatures.AWSYou can use KMS to encrypt your secrets and use encrypted secret in mailer policy.
Custodian tries to decrypt the string using KMS, if it fails c7n treats it as a plaintext secret.plaintext_secret:<raw_secret>secured_string:<encrypted_secret>AzureYou can store your secrets in Azure Key Vault secrets and reference them from the policy.plaintext_secret:<raw_secret>secured_string:type:azure.keyvaultsecret:https://your-vault.vault.azure.net/secrets/your-secretNote:secrets.getpermission on the KeyVault for the Service Principal is required.GCPYou can store your secrets as GCP Secret Manager secrets and reference them from the policy.plaintext_secret:<raw_secret>secured_string:type:gcp.secretmanagersecret:projects/12345678912/secrets/your-secretAn example of an SMTP password set as a secured string:smtp_password:type:gcp.secretmanagersecret:projects/59808015552/secrets/smtp_pwNote: If you do not specify a version,/versions/latestwill be appended to your secret location.Configuring a policy to send emailOutbound email can be added to any policy by including thenotifyaction.policies:-name:bad-applesresource:asgfilters:-"tag:ASV":absentactions:-type:notifytemplate:defaulttemplate_format:'html'priority_header:'1'subject:fix your tagsto:-resource-ownerowner_absent_contact:-foo@example.comtransport:type:sqsqueue:https://sqs.us-east-1.amazonaws.com/80101010101/cloud-custodian-message-relaySo breaking it down, you add an action of typenotify. You can specify a
template that's used to format the email; customizing templates is describedbelow.Thetolist specifies the intended recipient for the email. You can specify
either an email address, an SNS topic, a Datadog Metric, or a special value. The special values
are eitherresource-owner, in which case the email will be sent to the listedOwnerContacttag on the resource that matched the policy, orevent-ownerfor push-based/realtime policies that will send to the user
that was responsible for the underlying event.priority_headerto indicate the importance of an email withheaders. Different emails clients will display stars, exclamation points or flags depending on the value. Should be an string from 1 to 5.Both of these special values are best effort, i.e., if noOwnerContacttag is
specified thenresource-owneremail will not be delivered, and in the case ofevent-owneran instance role or system account will not result in an email.The optionalowner_absent_contactlist specifies email addresses to notify only if
theresource-ownerspecial option was unable to find any matching owner contact
tags.In addition, you may choose to use a custom tag instead of the defaultOwnerContact.  In order to configure this, the mailer.yaml must be modified to include the contact_tags and the custom tag.  Theresource-ownerwill now email the custom tag instead ofOwnerContact.contact_tags:-"custom_tag"For reference purposes, the JSON Schema of thenotifyaction:{"type":"object","required":["type","transport","to"],"properties":{"type":{"enum":["notify"]},"to":{"type":"array","items":{"type":"string"}},"owner_absent_contact":{"type":"array","items":{"type":"string"}},"subject":{"type":"string"},"priority_header":{"type":"string"},"template":{"type":"string"},"transport":{"type":"object","required":["type","queue"],"properties":{"queue":{"type":"string"},"region":{"type":"string"},"type":{"enum":["sqs"]}}}}}Using on AzureRequires:c7n_azurepackage.  SeeInstalling Azure PluginSendGrid account. SeeUsing SendGrid with AzureAzure Storage QueueThe mailer supports an Azure Storage Queue transport and SendGrid delivery on Azure.
Configuration for this scenario requires only minor changes from AWS deployments.You will need to grantStorage Queue Data Contributorrole on the Queue for the identity
mailer is running under.The notify action in your policy will reflect transport typeasqwith the URL
to an Azure Storage Queue.  For example:policies:-name:azure-notifyresource:azure.resourcegroupdescription:send a message to a mailer instanceactions:-type:notifytemplate:defaultpriority_header:'2'subject:Hello from C7N Mailerto:-you@youremail.comtransport:type:asqqueue:https://storageaccount.queue.core.windows.net/queuenameIn your mailer configuration, you'll need to provide your SendGrid API key as well as
prefix your queue URL withasq://to let mailer know what type of queue it is:queue_url:asq://storageaccount.queue.core.windows.net/queuenamefrom_address:you@youremail.comsendgrid_api_key:SENDGRID_API_KEYThe mailer will transmit all messages found on the queue on each execution, and will retry
sending 3 times in the event of a failure calling SendGrid.  After the retries the queue
message will be discarded.In addition, SendGrid delivery on Azure supports using resource tags to send emails. For example, in thetofield:to:-tag:OwnerEmailThis will find the email address associated with the resource'sOwnerEmailtag, and send an email to the specified address.
If no tag is found, or the associated email address is invalid, no email will be sent.Deploying Azure FunctionsThe--update-lambdaCLI option will also deploy Azure Functions if you have an Azure
mailer configuration.c7n-mailer --config mailer.yml --update-lambdawhere a simplemailer.ymlusing Consumption functions may look like:queue_url:asq://storage.queue.core.windows.net/custodianfrom_address:foo@mail.comsendgrid_api_key:<key>function_properties:servicePlan:name:'testmailer1'Configuring Function IdentityYou can configure the service principal used for api calls made by the
mailer azure function by specifying an identity configuration under
function properties. Mailer supports User Assigned Identities, System
Managed Identities, defaulting to an embedding of the cli user's
service principals credentials.When specifying a user assigned identity, unlike in a custodian
function policy where simply providing an name is sufficient, the
uuid/id and client id of the identity must be provided. You can
retrieve this information on the cli using theaz identity list.function_properties:identity:type:UserAssignedid:"/subscriptions/333fd504-7f11-2270-88c8-7325a27f7222/resourcegroups/c7n/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mailer"client_id:"b9cb06fa-dfb8-4342-add3-aab5acb2abbc"A system managed identity can also be used, and the Azure platform will
create an identity when the function is provisoned, however the function's identity
then needs to be retrieved and mapped to rbac permissions post provisioning, this
user management activity must be performed manually.function_properties:identity:type:SystemAssignedUsing on GCPRequires:c7n_gcppackage.  SeeGCP Getting Startedgoogle-cloud-secret-managerpackage, for pulling in secured string values.A working SMTP Account.GCP Pubsub SubscriptionThe mailer supports GCP Pubsub transports and SMTP/Email delivery, as well as Datadog and Splunk.
Configuration for this scenario requires only minor changes from AWS deployments.The notify action in your policy will reflect transport typeprojectswith the URL
to a GCP Pub/Sub Topic.  For example:policies:-name:gcp-notifyresource:gcp.computedescription:example policyactions:-type:notifytemplate:defaultpriority_header:'2'subject:Hello from C7N Mailerto:-you@youremail.comtransport:type:pubsubtopic:projects/myproject/topics/mytopicIn your mailer configuration, you'll need to provide your SMTP account information
as well as your topic subscription path in the queue_url variable. Please note that the
subscription you specify should be subscribed to the topic you assign in your policies'
notify action for GCP resources.queue_url:projects/myproject/subscriptions/mysubscriptionfrom_address:you@youremail.com# c7n-mailer currently requires a role be present, even if it's emptyrole:""smtp_server:my.smtp.add.resssmtp_port:25smtp_ssl:truesmtp_username:smtpusersmtp_password:type:gcp.secretmanagersecret:projects/12345678912/secrets/smtppasswordThe mailer will transmit all messages found on the queue on each execution using SMTP/Email delivery.Deploying GCP FunctionsGCP Cloud Functions for c7n-mailer are currently not supported.Writing an email templateTemplates are authored injinja2.
Drop a file with the.j2extension into the a templates directory, and send a pull request to this
repo. You can then reference it in thenotifyaction as thetemplatevariable by file name minus extension. Templates ending with.html.j2are
sent as HTML-formatted emails, all others are sent as plain text.You can use-tor--templatescli argument to pass custom folder with your templates.The following variables are available when rendering templates:variablevaluerecipientemail addressresourceslist of resources that matched the policy filterseventfor CWE-push-based lambda policies, the event that triggeredactionnotifyaction that generated this SQS messagepolicypolicy that triggered this notify actionaccountshort name of the aws accountregionregion the policy was executing inexecution_startThe time policy started executingThe following extra global functions are available:signaturebehaviorformat_struct(struct)pretty print a json structureresource_tag(resource, key)retrieve a tag value from a resource or return an empty string, aliased as get_resource_tag_valueformat_resource(resource, resource_type)renders a one line summary of a resourcedate_time_format(utc_str, tz_str='US/Eastern', format='%Y %b %d %H:%M %Z')customize rendering of an utc datetime stringsearch(expression, value)jmespath search value using expressionyaml_safe(value)yaml dumperThe following extra jinja filters are available:filterbehaviorutc_string|date_time_format(tz_str='US/Pacific', format='%Y %b %d %H:%M %Z')prettyformatthe date / time30|get_date_time_deltaConvert a timedeltalike '30' days in the future, to a datetime string. You can also use negative values for the past.Developer Install (OS X El Capitan)Clone the repository:$ git clone https://github.com/cloud-custodian/cloud-custodianInstall dependencies (with virtualenv):$ virtualenv c7n_mailer
$ source c7n_mailer/bin/activate
$ cd tools/c7n_mailer
$ pip install -r requirements.txtInstall the extensions:python setup.py developTesting Templates and RecipientsAc7n-mailer-replayentrypoint is provided to assist in testing email notifications
and templates. This script operates on an actual SQS message from cloud-custodian itself,
which you can either retrieve from the SQS queue or replicate locally. By default it expects
the message file to be base64-encoded, gzipped JSON, just like c7n sends to SQS. With the-p|--plainargument, it will expect the message file to contain plain JSON.c7n-mailer-replayhas three main modes of operation:With no additional arguments, it will render the template specified by the policy the
message is for, and actually send mail from the local machine asc7n-mailerwould.
This only works with SES, not SMTP.With the-T|--template-printargument, it will log the email addresses that would
receive mail, and print the rendered message body template to STDOUT.With the-d|--dry-runargument, it will print the actual email body (including headers)
that would be sent, for each message that would be sent, to STDOUT.Testing Templates for AzureThec7n-mailer-replayentrypoint can be used to test templates for Azure with either of the arguments:-T|--template-print-d|--dry-runRunningc7n-mailer-replaywithout either of these arguments will throw an error as it will attempt
to authorize with AWS.The following is an example for retrieving a sample message to test against templates:Run a policy with the notify action, providing the name of the template to test, to populate the queue.Using the azure cli, save the message locally:$ az storage message get --queue-name <queuename> --account-name <storageaccountname> --query '[].content' > test_message.gzThe example message can be provided toc7n-mailer-replayby running:$ c7n-mailer-replay test_message.gz -T --config mailer.yml