Repository: cloud-custodian/cloud-custodian

Issue:
Azure: c7n-mailer deployment issue.
Using `pip install c7n-mailer`

```
root@8b79684d85a3:/# c7n-mailer
Traceback (most recent call last):
  File "/usr/local/bin/c7n-mailer", line 5, in <module>
    from c7n_mailer.cli import main
  File "/usr/local/lib/python3.8/dist-packages/c7n_mailer/cli.py", line 9, in <module>
    from c7n_mailer import deploy, utils
  File "/usr/local/lib/python3.8/dist-packages/c7n_mailer/deploy.py", line 19, in <module>
    from c7n.mu import (
ModuleNotFoundError: No module named 'c7n'
```

Using docker image for `docker pull cloudcustodian/mailer`:

with python 3.6 and 3.8:

```
Traceback (most recent call last):
  File "/usr/local/bin/c7n-mailer", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/c7n_mailer/cli.py", line 233, in main
    azure_deploy.provision(mailer_config)
  File "/usr/local/lib/python3.6/site-packages/c7n_mailer/azure_mailer/deploy.py", line 129, in provision
    function_app_name=function_app_name)
TypeError: __init__() got an unexpected keyword argument 'function_app_resource_group_name'
```

Older version from pip and docker seems to work. After doing an install:

```
c7n_1  | 2020-08-05 01:21:43,817 - custodian.azure.session - INFO - Authenticated [Principal | 12345]
c7n_1  | 2020-08-05 01:21:44,370 - custodian.azure.deployment_unit.DeploymentUnit - INFO - Function Application "c7n-mailer-mailer-12345" not found.
c7n_1  | 2020-08-05 01:21:46,171 - custodian.azure.deployment_unit.DeploymentUnit - INFO - Application Service Plan "c7n-mailer" not found.
c7n_1  | 2020-08-05 01:21:46,171 - custodian.azure.deployment_unit.DeploymentUnit - INFO - Creating Application Service Plan "c7n-mailer"
c7n_1  | 2020-08-05 01:21:46,342 - custodian.azure.deployment_unit.DeploymentUnit - INFO - Found Resource Group "cloud-custodian".
c7n_1  | 2020-08-05 01:21:55,289 - custodian.azure.deployment_unit.DeploymentUnit - INFO - Application Service Plan "c7n-mailer" successfully created
c7n_1  | 2020-08-05 01:21:55,514 - custodian.azure.deployment_unit.DeploymentUnit - INFO - Found Application Insights "c7n-mailer".
c7n_1  | 2020-08-05 01:21:56,189 - custodian.azure.deployment_unit.DeploymentUnit - INFO - Found Storage Account "custodian12345".
c7n_1  | 2020-08-05 01:21:56,412 - custodian.azure.deployment_unit.DeploymentUnit - INFO - Creating Function Application "c7n-mailer-mailer-12345"
c7n_1  | 2020-08-05 01:22:28,013 - custodian.azure.deployment_unit.DeploymentUnit - INFO - Function Application "c7n-mailer-mailer-12345" successfully created
c7n_1  | 2020-08-05 01:22:28,013 - c7n_mailer.azure.deploy - INFO - Building function package for c7n-mailer-mailer-12345
c7n_1  | 2020-08-05 01:22:28,186 - c7n_mailer.azure.deploy - INFO - Function package built, size is 0MB
c7n_1  | 2020-08-05 01:22:28,186 - custodian.azure.function_app_utils - INFO - Publishing Function application
c7n_1  | 2020-08-05 01:22:30,016 - custodian.azure.function_package.FunctionPackage - INFO - Publishing Function package from /tmp/tmpj9auz92f
c7n_1  | 2020-08-05 01:22:31,597 - custodian.azure.function_package.FunctionPackage - INFO - Function publish result: 202
c7n_1  | 2020-08-05 01:22:31,598 - custodian.azure.function_app_utils - INFO - Finished publishing Function application
```

I am not able to see the function in Azure portal. Any pointers, please.

<img width="838" alt="Screen Shot 2020-08-04 at 6 30 14 PM" src="https://user-images.githubusercontent.com/11890959/89361673-b5a70000-d680-11ea-9415-49179b2cb783.png">





Discussion:
looks like an issue here, based on my understanding, it takes 4 args:

```
    class FunctionAppInfrastructureParameters:
        def __init__(self, app_insights, service_plan, storage_account,
                     function_app):
            self.app_insights = app_insights
            self.service_plan = service_plan
            self.storage_account = storage_account
            self.function_app = function_app
```

5 are getting passed:
```
    params = FunctionAppUtilities.FunctionAppInfrastructureParameters(
        app_insights=app_insights,
        service_plan=service_plan,
        storage_account=storage_account,
        function_app_resource_group_name=service_plan['resource_group_name'],
        function_app_name=function_app_name)
```

@logachev 

this is my docker-compose for mailer:

```
version: "3.4"
services:
  c7n:
    env_file:
      - .env
    image: "cloudcustodian/mailer:0.9.1.0" 
    entrypoint: 
    - /bin/bash
    - -c
    - |
      /usr/local/bin/c7n-mailer --config /tmp/mailer.yml --update-lambda
    volumes:
        - "mailer.yml:/tmp/mailer.yml"
        - "templates:/tmp/templates"
```

When I am trying to run it:
```
c7n_1  | 2020-08-05 05:14:29,226 - custodian.azure.session - INFO - Authenticated [Principal | 12345]
c7n_1  | 2020-08-05 05:14:29,813 - custodian.azure.deployment_unit.DeploymentUnit - INFO - Found Function Application "c7n-mailer-mailer-f12345".
c7n_1  | 2020-08-05 05:14:29,988 - c7n_mailer.azure.deploy - INFO - Building function package for c7n-mailer-mailer-12345
c7n_1  | 2020-08-05 05:14:30,168 - c7n_mailer.azure.deploy - INFO - Function package built, size is 0MB
c7n_1  | 2020-08-05 05:14:30,168 - custodian.azure.function_app_utils - INFO - Publishing Function application
c7n_1  | 2020-08-05 05:14:31,239 - custodian.azure.function_package.FunctionPackage - INFO - Publishing Function package from /tmp/tmp5f27m1fe
c7n_1  | 2020-08-05 05:14:33,939 - custodian.azure.function_package.FunctionPackage - INFO - Function publish result: 202
c7n_1  | 2020-08-05 05:14:33,940 - custodian.azure.function_app_utils - INFO - Finished publishing Function application
```
If I just do: `--run --debug`, it works, so why is it not getting packaged and pushed.
 
It shows package size is 0MB, any pointers?
thanks for the detailed bug report, this is a regression in 0.9.4 from changes to azure functions to support the more secure MSI/CSI authn/authz to provide api access to functions. the mailer needs an update to redress, using the previous c7n/c7n azure/mailer releases from 0.9.3 for the moment would be the recommend.
```
docker run \
-e AZURE_CLIENT_ID="123" \
-e AZURE_CLIENT_SECRET="123" \
-e AZURE_SUBSCRIPTION_ID="123" \
-e AZURE_TENANT_ID="123" \
-v c7n-mailer/environments:/tmp/environments
-it cloudcustodian/mailer:0.9.3.0 -c /tmp/environments/mailer.yml --update-lambda
```

App service is getting deployed no issues, it created Storage account if it does not exists, Insight.. etc..
but the function is coming as blank.

Looking at the `App Service` -> `Settings` -> `Configuration`.  I don't see `WEBSITE_RUN_FROM_PACKAGE` variable.

<img width="684" alt="Screen-Shot-2020-08-11-at-2 29 32-PM" src="https://user-images.githubusercontent.com/11890959/90065089-9c194000-dca0-11ea-9ef9-d6760d728a6a.png">


Tried in two different subscriptions.
Faced the same issue with c7n-mailer deployment to Azure.
Caught the zipped mailer project during the deployment, unzipped it, and tried to deploy the mailer using Visual Studio Code. The deployment process failed with the error:
`ERROR: Could not find a version that satisfies the requirement pkg_resources==0.0.0 (from -r requirements.txt (line 27)) (from versions: none)`
Then I commented out the line # 36 of c7n_maile/c7n_mailer/deploy.py (    # 'pkg_resources',) and was able to successfully deploy the mailer in Azure.
fix merged
Discussion on Gitter that there are still lingering issues here.  I'm reopening this issue to investigate.

![image](https://user-images.githubusercontent.com/4087345/106157160-442fa200-6137-11eb-9d72-79d3f5b143f1.png)


Resolving patch:
diff --git a/tools/c7n_azure/setup.py b/tools/c7n_azure/setup.py
--- a/tools/c7n_azure/setup.py
+++ b/tools/c7n_azure/setup.py
@@ -49,6 +49,7 @@
  'azure-mgmt-logic>=3.0.0,<4.0.0',
  'azure-mgmt-managementgroups>=0.2.0,<0.3.0',
  'azure-mgmt-monitor>=0.7.0,<0.8.0',
+ 'azure-mgmt-msi==1.0.0',
  'azure-mgmt-network>=9.0.0,<10.0.0',
  'azure-mgmt-policyinsights>=0.4.0,<0.5.0',
  'azure-mgmt-rdbms>=1.9.0,<2.0.0',
diff --git a/tools/c7n_mailer/c7n_mailer/azure_mailer/deploy.py b/tools/c7n_mailer/c7n_mailer/azure_mailer/deploy.py
--- a/tools/c7n_mailer/c7n_mailer/azure_mailer/deploy.py
+++ b/tools/c7n_mailer/c7n_mailer/azure_mailer/deploy.py
@@ -27,8 +27,9 @@ def cache_path():
 
 
 def get_mailer_requirements():
-    deps = ['azure-keyvault', 'azure-storage-queue',
-            'azure-storage-blob', 'sendgrid'] + list(CORE_DEPS)
+    deps = ['azure-mgmt-managementgroups', 'azure-mgmt-web',
+            'azure-graphrbac', 'azure-keyvault', 'azure-storage-queue',
+            'azure-storage-blob', 'netaddr', 'sendgrid', 'pyyaml'] + list(CORE_DEPS)
     requirements = generate_requirements(
         deps, ignore=['boto3', 'botocore', 'pywin32'],
         exclude=['pkg_resources'],
@@ -121,13 +122,15 @@ def provision(config):
     function_app_name = FunctionAppUtilities.get_function_name(
         '-'.join([service_plan['name'], function_name]), suffix)
     FunctionAppUtilities.validate_function_name(function_app_name)
+    identity = jmespath.search('function_properties.identity', config) or {
+        'type': AUTH_TYPE_EMBED}
 
     params = FunctionAppUtilities.FunctionAppInfrastructureParameters(
         app_insights=app_insights,
         service_plan=service_plan,
         storage_account=storage_account,
         function_app={'resource_group_name': service_plan['resource_group_name'],
-                      'identity': {'type': AUTH_TYPE_EMBED},
+                      'identity': identity,
                       'name': function_app_name})
 
     FunctionAppUtilities.deploy_function_app(params)
diff --git a/tools/c7n_mailer/c7n_mailer/utils.py b/tools/c7n_mailer/c7n_mailer/utils.py
--- a/tools/c7n_mailer/c7n_mailer/utils.py
+++ b/tools/c7n_mailer/c7n_mailer/utils.py
@@ -10,10 +10,14 @@
 
 import jinja2
 import jmespath
-from botocore.exceptions import ClientError
 from dateutil import parser
 from dateutil.tz import gettz, tzutc
 
+try:
+    from botocore.exceptions import ClientError
+except ImportError:  # pragma: no cover
+    pass  # Azure provider
+
 
 class Providers:
     AWS = 0


Test patch:
diff --git a/tools/c7n_mailer/tests/test_azure.py b/tools/c7n_mailer/tests/test_azure.py
--- a/tools/c7n_mailer/tests/test_azure.py
+++ b/tools/c7n_mailer/tests/test_azure.py
@@ -108,11 +108,15 @@ def test_azure_mailer_requirements(self):
         self.assertIn('adal', reqs)
         self.assertIn('azure-storage-common', reqs)
         self.assertIn('azure-common', reqs)
+        self.assertIn('azure-mgmt-managementgroups', reqs)
+        self.assertIn('azure-mgmt-web', reqs)
+        self.assertIn('azure-graphrbac', reqs)
         self.assertIn('msrestazure', reqs)
         self.assertIn('jmespath', reqs)
         self.assertIn('jinja2', reqs)
         self.assertIn('sendgrid', reqs)
         self.assertIn('ldap3', reqs)
+        self.assertIn('netaddr', reqs)
 
     @patch('c7n_mailer.azure_mailer.deploy.FunctionPackage')
     def test_build_function_package(self, package_mock):
@@ -130,6 +134,102 @@ def test_build_function_package(self, package_mock):
         package_mock.return_value.pkg.add_contents.assert_any_call(
             "test_mailer_sub/function.json", contents=ANY)
 
+    @patch('c7n_mailer.azure_mailer.deploy.build_function_package')
+    @patch('c7n_mailer.azure_mailer.deploy.FunctionAppUtilities')
+    @patch('c7n_mailer.azure_mailer.deploy.Session')
+    @patch('c7n_mailer.azure_mailer.deploy.local_session')
+    def test_provision_embedded_auth(self, mock_local_session, mock_session,
+                                     mock_func_utils, mock_build_pkg):
+        mock_session.get_subscription_id.return_value = 'mock-id'
+        mock_local_session.return_value = mock_session
+        mock_func_utils.get_function_name.return_value = 'mock-func-name'
+        mock_build_pkg.return_value = MagicMock()
+
+        deploy.provision(MAILER_CONFIG_AZURE)
+
+        mock_func_utils.FunctionAppInfrastructureParameters.assert_called_with(
+            app_insights=ANY,
+            service_plan=ANY,
+            storage_account=ANY,
+            function_app={
+                'resource_group_name': 'cloud-custodian',
+                'identity': {
+                    'type': 'Embedded'
+                },
+                'name': 'mock-func-name'
+            })
+
+        mock_func_utils.deploy_function_app.assert_called_once()
+        mock_func_utils.publish_functions_package.assert_called_once()
+
+    @patch('c7n_mailer.azure_mailer.deploy.build_function_package')
+    @patch('c7n_mailer.azure_mailer.deploy.FunctionAppUtilities')
+    @patch('c7n_mailer.azure_mailer.deploy.Session')
+    @patch('c7n_mailer.azure_mailer.deploy.local_session')
+    def test_provision_msi_auth(self, mock_local_session, mock_session,
+                                mock_func_utils, mock_build_pkg):
+        mock_session.get_subscription_id.return_value = 'mock-id'
+        mock_local_session.return_value = mock_session
+        mock_func_utils.get_function_name.return_value = 'mock-func-name'
+        mock_build_pkg.return_value = MagicMock()
+        system_assigned = {"identity": {"type": "SystemAssigned"}}
+
+        with patch.dict(MAILER_CONFIG_AZURE, {"function_properties": system_assigned}):
+            deploy.provision(MAILER_CONFIG_AZURE)
+
+            mock_func_utils.FunctionAppInfrastructureParameters.assert_called_with(
+                app_insights=ANY,
+                service_plan=ANY,
+                storage_account=ANY,
+                function_app={
+                    'resource_group_name': 'cloud-custodian',
+                    'identity': {
+                        'type': 'SystemAssigned'
+                    },
+                    'name': 'mock-func-name'
+                })
+
+            mock_func_utils.deploy_function_app.assert_called_once()
+            mock_func_utils.publish_functions_package.assert_called_once()
+
+    @patch('c7n_mailer.azure_mailer.deploy.build_function_package')
+    @patch('c7n_mailer.azure_mailer.deploy.FunctionAppUtilities')
+    @patch('c7n_mailer.azure_mailer.deploy.Session')
+    @patch('c7n_mailer.azure_mailer.deploy.local_session')
+    def test_provision_uai_auth(self, mock_local_session, mock_session,
+                                mock_func_utils, mock_build_pkg):
+        mock_session.get_subscription_id.return_value = 'mock-id'
+        mock_local_session.return_value = mock_session
+        mock_func_utils.get_function_name.return_value = 'mock-func-name'
+        mock_build_pkg.return_value = MagicMock()
+        user_assigned = {
+            "identity": {
+                "type": "UserAssigned",
+                "id": "mock-id",
+                "client_id": "mock-client-id"
+            }
+        }
+
+        with patch.dict(MAILER_CONFIG_AZURE, {"function_properties": user_assigned}):
+            deploy.provision(MAILER_CONFIG_AZURE)
+
+            mock_func_utils.FunctionAppInfrastructureParameters.assert_called_with(
+                app_insights=ANY,
+                service_plan=ANY,
+                storage_account=ANY,
+                function_app={
+                    'resource_group_name': 'cloud-custodian',
+                    'identity': {
+                        'type': 'UserAssigned',
+                        'id': 'mock-id',
+                        'client_id': 'mock-client-id'
+                    },
+                    'name': 'mock-func-name'
+                })
+
+            mock_func_utils.deploy_function_app.assert_called_once()
+            mock_func_utils.publish_functions_package.assert_called_once()
+
     @patch('c7n_mailer.azure_mailer.azure_queue_processor.SmtpDelivery')
     def test_smtp_delivery(self, mock_smtp):
         smtp_mailer_config = {