Repository: ansible/ansible

Issue:
lambda_policy is complaining about invalid characters that should be accepted.
 - Bug Report

##### COMPONENT NAME
lambda_policy

##### ANSIBLE VERSION
```
ansible 2.5.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/etc/ansible/custom_modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Feb 20 2018, 09:19:12) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
```

##### CONFIGURATION
<!---
ANSIBLE_PIPELINING(/etc/ansible/ansible.cfg) = True
ANSIBLE_SSH_RETRIES(/etc/ansible/ansible.cfg) = 3
CACHE_PLUGIN(/etc/ansible/ansible.cfg) = memory
DEFAULT_ACTION_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible/plugins/action']
DEFAULT_CALLBACK_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible/plugins/callback']
DEFAULT_CONNECTION_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible/plugins/connection']
DEFAULT_FILTER_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible/plugins/filter']
DEFAULT_FORKS(/etc/ansible/ansible.cfg) = 5
DEFAULT_GATHERING(/etc/ansible/ansible.cfg) = implicit
DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = [u'/etc/ansible']
DEFAULT_LOG_PATH(/etc/ansible/ansible.cfg) = /var/log/ansible.log
DEFAULT_LOOKUP_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible/plugins/lookup']
DEFAULT_MANAGED_STR(/etc/ansible/ansible.cfg) = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
DEFAULT_MODULE_PATH(/etc/ansible/ansible.cfg) = [u'/etc/ansible/custom_modules']
DEFAULT_POLL_INTERVAL(/etc/ansible/ansible.cfg) = 15
DEFAULT_ROLES_PATH(/etc/ansible/ansible.cfg) = [u'/etc/ansible/roles']
DEFAULT_SQUASH_ACTIONS(/etc/ansible/ansible.cfg) = ['apk', 'apt', 'dnf', 'package', 'pacman', 'pkgng', 'yum', 'zypper']
DEFAULT_STRATEGY_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible/plugins/strategy']
DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 20
DEFAULT_TRANSPORT(/etc/ansible/ansible.cfg) = smart
DEFAULT_VARS_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible/plugins/vars']
DEFAULT_VAULT_PASSWORD_FILE(/etc/ansible/ansible.cfg) = /etc/ansible/.ssh/vault.key
-->

##### OS / ENVIRONMENT
Ansible is on Redhat

##### SUMMARY
lambda_policy module is being called and I'm passing in a ARN to the function_name attribute.  Getting an error:
Function name arn:aws:lambda:us-east-1:999999999999:function:startinstance:1 is invalid. Names must contain only alphanumeric characters and hyphens.

##### STEPS TO REPRODUCE
Here is the function definition and the policy definition:
```
- name: Create lambda function for starting and stopping EC2 instances
  lambda:
    name: "{{ item.name }}"
    state: present
    zip_file: "/tmp/ec2_cycle/ec2_cycle.zip"
    runtime: python2.7
    role: "{{ lambdarole.iam_role.arn }}"
    handler: "{{ item.handler }}"
    timeout: 10
  with_items:
    - name: start_instance
      handler: ec2_cycle.ec2_start
    - name: start_rds_instance
      handler: ec2_cycle.rds_start
    - name: stop_instance
      handler: ec2_cycle.stop
  register: lambdafunction

- name: Create Lambda policy to allow CloudWatch event to trigger startEC2Instance function
  lambda_policy:
    function_name: "{{ lambdafunction.results[0].configuration.function_arn }}"
    statement_id: "startInstanceStatementId"
    action: "lambda:InvokeFunction"
    principal: "events.amazonaws.com"
    region: "{{ region }}"
    state: present
```

##### EXPECTED RESULTS
This configuration has worked for many runs in the past.  It broke after moving to Ansible 2.5.

##### ACTUAL RESULTS
```
See above error description.  Command output:
Function name arn:aws:lambda:us-east-1:999999999999:function:startinstance:1 is invalid. Names must contain only alphanumeric characters and hyphens.
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/cloud/amazon/lambda_policy.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/lambda_policy.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @mikedlr @pjodouin @willthames
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I've seen this before - it's usually the trailing `:1` on the ARN which is the version. I think I use something like `function_arn.split(':')[:7]|join(':')`. This will work whether or not the version is returned.

Basically the function_arn  varies based on different scenarios - whether it's an update or a create or no change to a lambda - we just provide the results from calling AWS's API but we should endeavour to make them a little more consistent where possible. If we're going to do that though, returning the version might make sense (or we could separate the version from the arn)

More investigation is needed to clarify the exact behaviour in the various scenarios and work out what we want to do.

So, are you saying this should work:

`function_name: "{{ lambdafunction.results[0].configuration.function_arn.split(':')[:7]|join(':') }}"`

instead of:

`function_name: "{{ lambdafunction.results[0].configuration.function_arn }}"`

In looking at the code for lambda_python, it appears that the ":" was removed from the regex string evaluating the function_arn.  If ":" is not allowed, then this would always fail.

Line 191 currently:
`        if not re.search(r'^[\w\-]+$', function_name):'`

Should be:
`       if not re.search(r'^[\w\-:]+$', function_name):`
Sorry, that code in lambda_policy is completely wrong - it will never work if you pass in an ARN!

But this always works:

```
function_name: "{{ lambdafunction.results[0].configuration.function_name }}"
```

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/lambda_policy.py b/lib/ansible/modules/cloud/amazon/lambda_policy.py
--- a/lib/ansible/modules/cloud/amazon/lambda_policy.py
+++ b/lib/ansible/modules/cloud/amazon/lambda_policy.py
@@ -188,6 +188,13 @@ def validate_params(module):
 
     # validate function name
     if function_name.startswith('arn:'):
+        if not re.search(r'^[\w\-:]+$', function_name):
+            module.fail_json(
+                msg='ARN {0} is invalid. ARNs must contain only alphanumeric characters, hyphens and colons.'.format(function_name)
+            )
+        if len(function_name) > 140:
+            module.fail_json(msg='ARN name "{0}" exceeds 140 character limit'.format(function_name))
+    else:
         if not re.search(r'^[\w\-]+$', function_name):
             module.fail_json(
                 msg='Function name {0} is invalid. Names must contain only alphanumeric characters and hyphens.'.format(
@@ -196,13 +203,6 @@ def validate_params(module):
         if len(function_name) > 64:
             module.fail_json(
                 msg='Function name "{0}" exceeds 64 character limit'.format(function_name))
-    else:
-        if not re.search(r'^[\w\-:]+$', function_name):
-            module.fail_json(
-                msg='ARN {0} is invalid. ARNs must contain only alphanumeric characters, hyphens and colons.'.format(function_name)
-            )
-        if len(function_name) > 140:
-            module.fail_json(msg='ARN name "{0}" exceeds 140 character limit'.format(function_name))
 
 
 def get_qualifier(module):


Test patch:
diff --git a/test/integration/targets/lambda_policy/files/minimal_trust_policy.json b/test/integration/targets/lambda_policy/files/minimal_trust_policy.json
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/lambda_policy/files/minimal_trust_policy.json
@@ -0,0 +1,12 @@
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Principal": {
+        "Service": "lambda.amazonaws.com"
+      },
+      "Action": "sts:AssumeRole"
+    }
+  ]
+}
diff --git a/test/integration/targets/lambda_policy/tasks/main.yml b/test/integration/targets/lambda_policy/tasks/main.yml
--- a/test/integration/targets/lambda_policy/tasks/main.yml
+++ b/test/integration/targets/lambda_policy/tasks/main.yml
@@ -5,6 +5,15 @@
 
 - block:
 
+    - name: set up AWS credentials
+      set_fact:
+        aws_connection_info: &aws_connection_info
+          aws_region: '{{ aws_region }}'
+          aws_access_key: '{{ aws_access_key }}'
+          aws_secret_key: '{{ aws_secret_key }}'
+          security_token: '{{ security_token }}'
+      no_log: yes
+
     # ============================================================
     - name: test with no parameters
       lambda_policy:
@@ -54,7 +63,6 @@
     - name: test exceptions generated by forcing bad ec2 url
       lambda_policy:
         function_name: "{{ lambda_function_name }}"
-        region: "{{ec2_region}}"
         state: present
         statement_id: api-gateway-invoke-lambdas
         action: lambda:InvokeFunction
@@ -89,38 +97,43 @@
         dest: "{{output_dir}}/mini_http_lambda.zip"
       register: zip_res
 
+    # This should exist, but there's no expectation that the test user should be able to
+    # create/update this role, merely validate that it's there.
+    # Use ansible -m iam_role -a 'name=ansible_lambda_role
+    # assume_role_policy_document={{ lookup("file", "test/integration/targets/lambda_policy/files/minimal_trust_policy.json", convert_data=False) }}
+    # ' -vvv localhost
+    # to create this through more privileged credentials before running this test suite.
+    - name: create minimal lambda role
+      iam_role:
+        name: ansible_lambda_role
+        assume_role_policy_document: "{{ lookup('file', 'minimal_trust_policy.json', convert_data=False) }}"
+        create_instance_profile: no
+        <<: *aws_connection_info
+      register: iam_role
+
+    - name: wait 10 seconds for role to become available
+      pause:
+        seconds: 10
+      when: iam_role.changed
+
     - name: test state=present - upload the lambda
       lambda:
-        name="{{lambda_function_name}}"
-        runtime="python2.7"
-        handler="mini_http_lambda.handler"
-        role="ansible_lambda_role"
-        ec2_region='{{ec2_region}}'
-        aws_access_key='{{aws_access_key}}'
-        aws_secret_key='{{aws_secret_key}}'
-        security_token='{{security_token}}'
-        zip_file="{{zip_res.dest}}"
+        name: "{{lambda_function_name}}"
+        runtime: "python2.7"
+        handler: "mini_http_lambda.handler"
+        role: "ansible_lambda_role"
+        zip_file: "{{zip_res.dest}}"
+        <<: *aws_connection_info
       register: lambda_result
 
-    - name: install aws cli - FIXME temporary this should go for a lighterweight solution
-      command: pip install awscli
-      register: result
-
     - name: get the aws account ID for use in future commands
-      command: aws sts get-caller-identity --output text --query 'Account'
-      environment:
-          AWS_ACCESS_KEY_ID: '{{aws_access_key}}'
-          AWS_SECRET_ACCESS_KEY: '{{aws_secret_key}}'
-          AWS_SESSION_TOKEN: '{{security_token}}'
-      register: result
-
-    - name: register account id
-      set_fact:
-        aws_account_id: "{{ result.stdout | replace('\n', '') }}"
+      aws_caller_facts:
+        <<: *aws_connection_info
+      register: aws_caller_facts
 
     - name: register lambda uri for use in template
       set_fact:
-        mini_lambda_uri: "arn:aws:apigateway:{{ec2_region}}:lambda:path/2015-03-31/functions/arn:aws:lambda:{{ec2_region}}:{{aws_account_id}}:function:{{ lambda_result.configuration.function_name }}/invocations"
+        mini_lambda_uri: "arn:aws:apigateway:{{ aws_region }}:lambda:path/2015-03-31/functions/arn:aws:lambda:{{ aws_region }}:{{ aws_caller_facts.account }}:function:{{ lambda_result.configuration.function_name }}/invocations"
 
     - name: build API file
       template:
@@ -131,19 +144,16 @@
       aws_api_gateway:
         api_file: "{{output_dir}}/endpoint-test-swagger-api.yml.j2"
         stage: "lambdabased"
-        region: '{{ec2_region}}'
-        aws_access_key: '{{aws_access_key}}'
-        aws_secret_key: '{{aws_secret_key}}'
-        security_token: '{{security_token}}'
+        <<: *aws_connection_info
       register: create_result
 
-
     - name: register api id for later
       set_fact:
         api_id: "{{ create_result.api_id }}"
 
     - name: check API fails with permissions failure
-      uri: url="https://{{create_result.api_id}}.execute-api.{{ec2_region}}.amazonaws.com/lambdabased/mini/Mr_Ansible_Tester"
+      uri:
+        url: "https://{{create_result.api_id}}.execute-api.{{aws_region}}.amazonaws.com/lambdabased/mini/Mr_Ansible_Tester"
       register: unauth_uri_result
       ignore_errors: true
 
@@ -156,18 +166,26 @@
     - name: give api gateway execute permissions on lambda
       lambda_policy:
         function_name: "{{ lambda_function_name }}"
-        region: "{{ec2_region}}"
         state: present
         statement_id: api-gateway-invoke-lambdas
         action: lambda:InvokeFunction
         principal: apigateway.amazonaws.com
-        source_arn: "arn:aws:execute-api:{{ ec2_region }}:{{ aws_account_id }}:*/*"
-        aws_access_key: '{{aws_access_key}}'
-        aws_secret_key: '{{aws_secret_key}}'
-        security_token: '{{security_token}}'
+        source_arn: "arn:aws:execute-api:{{ aws_region }}:{{ aws_caller_facts.account }}:*/*"
+        <<: *aws_connection_info
+
+    - name: try again but with ARN
+      lambda_policy:
+        function_name: "{{ lambda_result.configuration.function_arn }}"
+        state: present
+        statement_id: api-gateway-invoke-lambdas
+        action: lambda:InvokeFunction
+        principal: apigateway.amazonaws.com
+        source_arn: "arn:aws:execute-api:{{ aws_region }}:{{ aws_caller_facts.account }}:*/*"
+        <<: *aws_connection_info
 
     - name: check API works with execute permissions
-      uri: url="https://{{create_result.api_id}}.execute-api.{{ec2_region}}.amazonaws.com/lambdabased/mini/Mr_Ansible_Tester"
+      uri:
+        url: "https://{{create_result.api_id}}.execute-api.{{aws_region}}.amazonaws.com/lambdabased/mini/Mr_Ansible_Tester"
       register: uri_result
 
     - name: assert API works success
@@ -180,10 +198,7 @@
       aws_api_gateway:
         api_file: "{{output_dir}}/endpoint-test-swagger-api.yml.j2"
         stage: "lambdabased"
-        region: '{{ec2_region}}'
-        aws_access_key: '{{aws_access_key}}'
-        aws_secret_key: '{{aws_secret_key}}'
-        security_token: '{{security_token}}'
+        <<: *aws_connection_info
       register: create_result
       ignore_errors: true
 
@@ -193,26 +208,16 @@
     # ============================================================
     - name: destroy lambda for test cleanup if created
       lambda:
-        name="{{lambda_function_name}}"
-        ec2_region='{{ec2_region}}'
-        ec2_access_key='{{ec2_access_key}}'
-        ec2_secret_key='{{ec2_secret_key}}'
-        security_token='{{security_token}}'
-        state=absent
+        name: "{{lambda_function_name}}"
+        <<: *aws_connection_info
+        state: absent
       register: result
+      ignore_errors: yes
 
     - name: destroy API for test cleanup if created
       aws_api_gateway:
         state: absent
         api_id: '{{api_id}}'
-        region: '{{ec2_region}}'
-        aws_access_key: '{{ec2_access_key}}'
-        aws_secret_key: '{{ec2_secret_key}}'
-        security_token: '{{security_token}}'
+        <<: *aws_connection_info
       register: destroy_result
-
-    - name: assert destroy statements succeeded
-      assert:
-        that:
-           - 'destroy_result.changed == True'
-           - 'result is not failed'
+      ignore_errors: yes