Repository: ansible/ansible

Issue:
fortios_ipv4_policy.py state=absent
##### General Notes
Issue 1: Clarity
ID is not clear. ID refers to the "edit #" in the config, but not the seq.# that is used in GUI (also the sequence in which they appear in the config).
  EX:
```
  I use this module to configure a new policy for "id: 2", it will either replace or configure a new policy with beginning "edit 2"
  I use the module again to configure a new policy for "id: 3", it will either replace or configure a new policy with beginning "edit 3"
  I use the module again to delete the policy for "id: 2", it will remove policy with beginning "edit 2"
  I use the module again to configure a new policy for "id: 2", it will configure a new policy with beginning "edit 2"
  If I look at the order of the rules in the GUI, "edit 3" will have a sequence number of 2, and "edit 2" will have a sequence number of 3, which corresponds to the order of the rules in both the GUI and CLI.
```

Issue 2 (main issue): Module Error
When setting the "state" to "absent", the module errors because the fortios_ipv4_policy.py file has a line that is trying to pass a "path" variable, which does not exist. I got this to work by replacing with "path" with "firewall policy" (I believe this is what is intended, but please verify this meets all expected behavior).
```python
    #Absent State
    if module.params['state'] == 'absent':
        fortigate.candidate_config[path].del_block(policy_id)
```
to
```python
    #Absent State
    if module.params['state'] == 'absent':
        fortigate.candidate_config['firewall policy'].del_block(policy_id)
```

Issue 3: Update Documentation
The state command's comments read "Specifies if address need to be added or deleted." It seems that the actual intent of this setting is to decide whether a policy "id" (I have chosen "id" as this seems to determining factor, please see Issue 4 below) is to be created/modified or deleted. Suggest to word something like "Specifies if  policy "id" needs to be updated or deleted."

Issue 4: Module Requirements
The module requires unused parameters: "dst_addr", "policy_action", "service", and "src_addr". These are required when state=present, but are completely ignored when state=absent (using my workaround in Issue 2). If these are required, then they should check the policy "id" that is passed in and validate that the policy "id" configuration matches the  "dst_addr", "policy_action", "service", and "src_addr" parameters. Otherwise, these should only be required when setting state=present.
  EX: 
```  
  I have policy id 2 configured for dst_addr=web_servers, policy_action=accept, service=HTTP, and src_addr=users
  I run state=absent (using my fix in Issue 2) with id=2, dst_addr=all, policy_action=deny, service=FTP, and src_addr=all: the result is that policy id 2 is removed from the configuration (even though all 4 parameters do not match the configured policy).
```
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
fortios_ipv4_policy
##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/etc/ntc/ansible-modules/']
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
configured module search path = [u'/etc/ntc/ansible-modules/']

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
Ubuntu -> Fortios
##### SUMMARY
<!--- Explain the problem briefly -->
Invalid variable name "path" in fortios_ipv4_policy.py when setting state=absent

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Issue is when setting "state" to absent (used to delete a policy id)

<!--- Paste example playbooks or commands between quotes below -->
```yaml
  - name: IPv4 ACL CONFIG
    hosts: fortinet
    connection: local
    gather_facts: False
    tags: core

    tasks:
      - name: Add ACL
        fortios_ipv4_policy:
          host: "{{ inventory_hostname }}"
          username: "{{ username }}"
          password: "{{ password }}"
          id: 2
          src_addr: users
          dst_addr: web_servers
          service: HTTP
          policy_action: accept
          state: absent
        tags: acl_conf
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Policy ID 2 to be removed from Fortigate
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
(ans23) #ansible-playbook fpb.yml -t acl_conf -vvvv
Using /etc/ansible/ansible.cfg as config file
Loading callback plugin default of type stdout, v2.0 from /home/user/ans23/local/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: fpb.yml ****************************************************************************************************************************************************************
3 plays in fpb.yml

PLAY [GETTERS] *******************************************************************************************************************************************************************
META: ran handlers
META: ran handlers
META: ran handlers

PLAY [IPv4 ACL CONFIG] ***********************************************************************************************************************************************************
META: ran handlers

TASK [Add ACL] *******************************************************************************************************************************************************************
task path: /home/user/fpb.yml:28
Using module file /home/user/ans23/local/lib/python2.7/site-packages/ansible/modules/network/fortios/fortios_ipv4_policy.py
<x.x.x.x> ESTABLISH LOCAL CONNECTION FOR USER: user
<x.x.x.x> EXEC /bin/sh -c 'echo ~ && sleep 0'
<x.x.x.x> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660 `" && echo ansible-tmp-1491250419.7-34981597923660="` echo /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660 `" ) && sleep 0'
<x.x.x.x> PUT /tmp/tmp12VgU1 TO /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660/fortios_ipv4_policy.py
<x.x.x.x> EXEC /bin/sh -c 'chmod u+x /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660/ /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660/fortios_ipv4_policy.py && sleep 0'
<x.x.x.x> EXEC /bin/sh -c 'python /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660/fortios_ipv4_policy.py; rm -rf "/home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660/" > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_HKDLrk/ansible_module_fortios_ipv4_policy.py", line 287, in <module>
    main()
  File "/tmp/ansible_HKDLrk/ansible_module_fortios_ipv4_policy.py", line 228, in main
    fortigate.candidate_config[path].del_block(policy_id)
NameError: global name 'path' is not defined

fatal: [x.x.x.x]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_HKDLrk/ansible_module_fortios_ipv4_policy.py\", line 287, in <module>\n    main()\n  File \"/tmp/ansible_HKDLrk/ansible_module_fortios_ipv4_policy.py\", line 228, in main\n    fortigate.candidate_config[path].del_block(policy_id)\nNameError: global name 'path' is not defined\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 0
}
	to retry, use: --limit @/home/user/fpb.retry

PLAY RECAP ***********************************************************************************************************************************************************************
x.x.x.x               : ok=0    changed=0    unreachable=0    failed=1 

<!--- Paste verbatim command output between quotes below -->
```

```

fortios_ipv4_policy.py state=absent
##### General Notes
Issue 1: Clarity
ID is not clear. ID refers to the "edit #" in the config, but not the seq.# that is used in GUI (also the sequence in which they appear in the config).
  EX:
```
  I use this module to configure a new policy for "id: 2", it will either replace or configure a new policy with beginning "edit 2"
  I use the module again to configure a new policy for "id: 3", it will either replace or configure a new policy with beginning "edit 3"
  I use the module again to delete the policy for "id: 2", it will remove policy with beginning "edit 2"
  I use the module again to configure a new policy for "id: 2", it will configure a new policy with beginning "edit 2"
  If I look at the order of the rules in the GUI, "edit 3" will have a sequence number of 2, and "edit 2" will have a sequence number of 3, which corresponds to the order of the rules in both the GUI and CLI.
```

Issue 2 (main issue): Module Error
When setting the "state" to "absent", the module errors because the fortios_ipv4_policy.py file has a line that is trying to pass a "path" variable, which does not exist. I got this to work by replacing with "path" with "firewall policy" (I believe this is what is intended, but please verify this meets all expected behavior).
```python
    #Absent State
    if module.params['state'] == 'absent':
        fortigate.candidate_config[path].del_block(policy_id)
```
to
```python
    #Absent State
    if module.params['state'] == 'absent':
        fortigate.candidate_config['firewall policy'].del_block(policy_id)
```

Issue 3: Update Documentation
The state command's comments read "Specifies if address need to be added or deleted." It seems that the actual intent of this setting is to decide whether a policy "id" (I have chosen "id" as this seems to determining factor, please see Issue 4 below) is to be created/modified or deleted. Suggest to word something like "Specifies if  policy "id" needs to be updated or deleted."

Issue 4: Module Requirements
The module requires unused parameters: "dst_addr", "policy_action", "service", and "src_addr". These are required when state=present, but are completely ignored when state=absent (using my workaround in Issue 2). If these are required, then they should check the policy "id" that is passed in and validate that the policy "id" configuration matches the  "dst_addr", "policy_action", "service", and "src_addr" parameters. Otherwise, these should only be required when setting state=present.
  EX: 
```  
  I have policy id 2 configured for dst_addr=web_servers, policy_action=accept, service=HTTP, and src_addr=users
  I run state=absent (using my fix in Issue 2) with id=2, dst_addr=all, policy_action=deny, service=FTP, and src_addr=all: the result is that policy id 2 is removed from the configuration (even though all 4 parameters do not match the configured policy).
```
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
fortios_ipv4_policy
##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.4.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/etc/ntc/ansible-modules/']
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
configured module search path = [u'/etc/ntc/ansible-modules/']

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
Ubuntu -> Fortios
##### SUMMARY
<!--- Explain the problem briefly -->
Invalid variable name "path" in fortios_ipv4_policy.py when setting state=absent

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Issue is when setting "state" to absent (used to delete a policy id)

<!--- Paste example playbooks or commands between quotes below -->
```yaml
  - name: IPv4 ACL CONFIG
    hosts: fortinet
    connection: local
    gather_facts: False
    tags: core

    tasks:
      - name: Add ACL
        fortios_ipv4_policy:
          host: "{{ inventory_hostname }}"
          username: "{{ username }}"
          password: "{{ password }}"
          id: 2
          src_addr: users
          dst_addr: web_servers
          service: HTTP
          policy_action: accept
          state: absent
        tags: acl_conf
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Policy ID 2 to be removed from Fortigate
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
(ans23) #ansible-playbook fpb.yml -t acl_conf -vvvv
Using /etc/ansible/ansible.cfg as config file
Loading callback plugin default of type stdout, v2.0 from /home/user/ans23/local/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: fpb.yml ****************************************************************************************************************************************************************
3 plays in fpb.yml

PLAY [GETTERS] *******************************************************************************************************************************************************************
META: ran handlers
META: ran handlers
META: ran handlers

PLAY [IPv4 ACL CONFIG] ***********************************************************************************************************************************************************
META: ran handlers

TASK [Add ACL] *******************************************************************************************************************************************************************
task path: /home/user/fpb.yml:28
Using module file /home/user/ans23/local/lib/python2.7/site-packages/ansible/modules/network/fortios/fortios_ipv4_policy.py
<x.x.x.x> ESTABLISH LOCAL CONNECTION FOR USER: user
<x.x.x.x> EXEC /bin/sh -c 'echo ~ && sleep 0'
<x.x.x.x> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660 `" && echo ansible-tmp-1491250419.7-34981597923660="` echo /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660 `" ) && sleep 0'
<x.x.x.x> PUT /tmp/tmp12VgU1 TO /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660/fortios_ipv4_policy.py
<x.x.x.x> EXEC /bin/sh -c 'chmod u+x /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660/ /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660/fortios_ipv4_policy.py && sleep 0'
<x.x.x.x> EXEC /bin/sh -c 'python /home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660/fortios_ipv4_policy.py; rm -rf "/home/user/.ansible/tmp/ansible-tmp-1491250419.7-34981597923660/" > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_HKDLrk/ansible_module_fortios_ipv4_policy.py", line 287, in <module>
    main()
  File "/tmp/ansible_HKDLrk/ansible_module_fortios_ipv4_policy.py", line 228, in main
    fortigate.candidate_config[path].del_block(policy_id)
NameError: global name 'path' is not defined

fatal: [x.x.x.x]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_HKDLrk/ansible_module_fortios_ipv4_policy.py\", line 287, in <module>\n    main()\n  File \"/tmp/ansible_HKDLrk/ansible_module_fortios_ipv4_policy.py\", line 228, in main\n    fortigate.candidate_config[path].del_block(policy_id)\nNameError: global name 'path' is not defined\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 0
}
	to retry, use: --limit @/home/user/fpb.retry

PLAY RECAP ***********************************************************************************************************************************************************************
x.x.x.x               : ok=0    changed=0    unreachable=0    failed=1 

<!--- Paste verbatim command output between quotes below -->
```

```



Discussion:
@bjolivot Could you please take a look

It would be good to get this fixed for 2.3 final (Wednesday 12th April) Thanks
@bjolivot Could you please take a look

It would be good to get this fixed for 2.3 final (Wednesday 12th April) Thanks

Resolving patch:
diff --git a/lib/ansible/modules/network/fortios/fortios_ipv4_policy.py b/lib/ansible/modules/network/fortios/fortios_ipv4_policy.py
--- a/lib/ansible/modules/network/fortios/fortios_ipv4_policy.py
+++ b/lib/ansible/modules/network/fortios/fortios_ipv4_policy.py
@@ -36,11 +36,17 @@
 options:
   id:
     description:
-      - Policy ID.
+      - "Policy ID.
+        Warning: policy ID number is different than Policy sequence number.
+        The policy ID is the number assigned at policy creation.
+        The sequence number represents the order in which the Fortigate will evaluate the rule for policy enforcement,
+        and also the order in which rules are listed in the GUI and CLI.
+        These two numbers do not necessarily correlate: this module is based off policy ID.
+        TIP: policy ID can be viewed in the GUI by adding 'ID' to the display columns"
     required: true
   state:
     description:
-      - Specifies if address need to be added or deleted.
+      - Specifies if policy I(id) need to be added or deleted.
     choices: ['present', 'absent']
     default: present
   src_intf:
@@ -53,8 +59,7 @@
     default: any
   src_addr:
     description:
-      - Specifies source address (or group) object name(s).
-    required: true
+      - Specifies source address (or group) object name(s). Required when I(state=present).
   src_addr_negate:
     description:
       - Negate source address param.
@@ -62,8 +67,7 @@
     choices: ["true", "false"]
   dst_addr:
     description:
-      - Specifies destination address (or group) object name(s).
-    required: true
+      - Specifies destination address (or group) object name(s). Required when I(state=present).
   dst_addr_negate:
     description:
       - Negate destination address param.
@@ -71,14 +75,12 @@
     choices: ["true", "false"]
   policy_action:
     description:
-      - Specifies accept or deny action policy.
+      - Specifies accept or deny action policy. Required when I(state=present).
     choices: ['accept', 'deny']
-    required: true
     aliases: ['action']
   service:
     description:
-      - "Specifies policy service(s), could be a list (ex: ['MAIL','DNS'])."
-    required: true
+      - "Specifies policy service(s), could be a list (ex: ['MAIL','DNS']). Required when I(state=present)."
     aliases:
       - services
   service_negate:
@@ -180,12 +182,12 @@ def main():
         src_intf                  = dict(default='any'),
         dst_intf                  = dict(default='any'),
         state                     = dict(choices=['present', 'absent'], default='present'),
-        src_addr                  = dict(required=True, type='list'),
-        dst_addr                  = dict(required=True, type='list'),
+        src_addr                  = dict(type='list'),
+        dst_addr                  = dict(type='list'),
         src_addr_negate           = dict(type='bool', default=False),
         dst_addr_negate           = dict(type='bool', default=False),
-        policy_action             = dict(choices=['accept','deny'], required=True, aliases=['action']),
-        service                   = dict(aliases=['services'], required=True, type='list'),
+        policy_action             = dict(choices=['accept','deny'], aliases=['action']),
+        service                   = dict(aliases=['services'], type='list'),
         service_negate            = dict(type='bool', default=False),
         schedule                  = dict(type='str', default='always'),
         nat                       = dict(type='bool', default=False),
@@ -200,15 +202,22 @@ def main():
     #merge global required_if & argument_spec from module_utils/fortios.py
     argument_spec.update(fortios_argument_spec)
 
+    ipv4_policy_required_if = [
+        ['state', 'present', ['src_addr', 'dst_addr', 'policy_action', 'service']],
+    ]
+
     module = AnsibleModule(
         argument_spec=argument_spec,
         supports_check_mode=True,
-        required_if=fortios_required_if,
+        required_if=fortios_required_if + ipv4_policy_required_if ,
     )
 
     #init forti object
     fortigate = AnsibleFortios(module)
 
+    #Security policies root path
+    config_path = 'firewall policy'
+
     #test params
     #NAT related
     if not module.params['nat']:
@@ -221,11 +230,11 @@ def main():
     policy_id = str(module.params['id'])
 
     #load config
-    fortigate.load_config('firewall policy')
+    fortigate.load_config(config_path)
 
     #Absent State
     if module.params['state'] == 'absent':
-        fortigate.candidate_config[path].del_block(policy_id)
+        fortigate.candidate_config[config_path].del_block(policy_id)
 
     #Present state
     elif module.params['state'] == 'present':


Test patch: