Repository: ansible/ansible

Issue:
nxos_bgp* modules fail to apply for both nxapi and cli transports
##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
- nxos_bgp, nxos_bgp_af, nxos_bgp_neighbor, nxos_bgp_neighbor_af

##### NXOS VERSION
7.0(3)I5(1)

##### ANSIBLE VERSION
```
ansible 2.3.0.0 (detached HEAD d56ba0945d) last updated 2017/04/24 16:39:23 (GMT -400)
 config file = 
 configured module search path = Default w/o overrides
 python version = 2.7.13 (default, Apr  4 2017, 08:47:57) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)]
```

##### OS / ENVIRONMENT
running on OSX

##### SUMMARY

##### STEPS TO REPRODUCE
```yaml
---
cli:
 host: "{{ inventory_hostname }}"
 username: "{{ nxos_cli_user | default('admin') }}"
 password: "{{ nxos_cli_pass | default('admin') }}"

nxapi:
 host: "{{ inventory_hostname }}"
 username: "{{ nxos_nxapi_user | default('admin') }}"
 password: "{{ nxos_nxapi_pass | default('admin') }}"
 transport: nxapi
 user_ssl: no

---
- hosts: nxos 
 connection: local 
 gather_facts: no

 tasks:
 - name: Configure a simple ASN
   nxos_bgp:
     asn: 65535
     #vrf: test
     router_id: 1.1.1.1
     state: present
     provider: "{{ cli }}"
     #provider: "{{ nxapi }}"
```

##### EXPECTED RESULTS
Expect bgp to be configured using playbook properties.


##### ACTUAL RESULTS
**NOTE:** All nxos_bgp* modules fail with the same error.
```
$ansible-playbook -i /etc/ansible/hosts ../ap/test_nxos_bgp.yaml -vvvv
No config file found; using defaults
Loading callback plugin default of type stdout, v2.0 from /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/plugins/callback/__init__.pyc

PLAYBOOK: test_nxos_bgp.yaml ********************************************************************************************************************************************************************
1 plays in ../ap/test_nxos_bgp.yaml

PLAY [nxos] *************************************************************************************************************************************************************************************
META: ran handlers

TASK [Configure a simple ASN] *******************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/ap/test_nxos_bgp.yaml:7
<n9k> connection transport is cli
<n9k> socket_path: /Users/mwiebe/.ansible/pc/30512dd245
open_shell() returned 0 ok 
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/network/nxos/nxos_bgp.py
<n9k> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n9k> EXEC /bin/sh -c 'echo ~ && sleep 0'
<n9k> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1493066974.43-188762941701890 `" && echo ansible-tmp-1493066974.43-188762941701890="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1493066974.43-188762941701890 `" ) && sleep 0'
<n9k> PUT /var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/tmpX30gv1 TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1493066974.43-188762941701890/nxos_bgp.py
<n9k> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1493066974.43-188762941701890/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1493066974.43-188762941701890/nxos_bgp.py && sleep 0'
<n9k> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1493066974.43-188762941701890/nxos_bgp.py; rm -rf "/Users/mwiebe/.ansible/tmp/ansible-tmp-1493066974.43-188762941701890/" > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
 File "/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible_wTRkbb/ansible_module_nxos_bgp.py", line 874, in <module>
   main()
 File "/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible_wTRkbb/ansible_module_nxos_bgp.py", line 824, in main
   existing = invoke('get_existing', module, args)
 File "/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible_wTRkbb/ansible_module_nxos_bgp.py", line 468, in invoke
   return func(*args, **kwargs)
 File "/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible_wTRkbb/ansible_module_nxos_bgp.py", line 569, in get_existing
   config = netcfg.get_section(parents)
AttributeError: 'str' object has no attribute 'get_section'

fatal: [n9k]: FAILED! => {
   "changed": false, 
   "failed": true, 
   "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible_wTRkbb/ansible_module_nxos_bgp.py\", line 874, in <module>\n    main()\n  File \"/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible_wTRkbb/ansible_module_nxos_bgp.py\", line 824, in main\n    existing = invoke('get_existing', module, args)\n  File \"/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible_wTRkbb/ansible_module_nxos_bgp.py\", line 468, in invoke\n    return func(*args, **kwargs)\n  File \"/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible_wTRkbb/ansible_module_nxos_bgp.py\", line 569, in get_existing\n    config = netcfg.get_section(parents)\nAttributeError: 'str' object has no attribute 'get_section'\n", 
   "module_stdout": "", 
   "msg": "MODULE FAILURE", 
   "rc": 0
}
 to retry, use: --limit @/Users/mwiebe/Projects/nxos_ansible/ap/test_nxos_bgp.retry

PLAY RECAP **************************************************************************************************************************************************************************************
n9k          : ok=0    changed=0    unreachable=0    failed=1   

```


Discussion:
cc @dagwieers
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
bot_broken
I've made a pass though all the bgp modules, can you let me know if there are still issues in devel please?
@Qalthos Yes, I will re-test them today and let you know.

**Summary:**
  * nxos_bgp basic test working fine now in latest devel
  * nxos_bgp_af **still has issues**: Summary and potential fix listed below
  * nxos_bgp_neighbor basic test working fine now in latest devel
  * nxos_bgp_neighbor_af **still has issues**: Summary below. I did not look at the code to identify a potential fix.

@Qalthos Let me know if you want me to open PR's for the candidate fixes below or if it's just easier for you to fix them.


Resolving patch:
diff --git a/lib/ansible/modules/network/nxos/nxos_bgp_af.py b/lib/ansible/modules/network/nxos/nxos_bgp_af.py
--- a/lib/ansible/modules/network/nxos/nxos_bgp_af.py
+++ b/lib/ansible/modules/network/nxos/nxos_bgp_af.py
@@ -334,85 +334,52 @@
 ]
 
 
-def get_custom_list_value(config, arg, module):
-    value_list = []
-    splitted_config = config.splitlines()
+def get_value(arg, config, module):
+    command = PARAM_TO_COMMAND_KEYMAP[arg]
+    command_val_re = re.compile(r'(?:{0}\s)(?P<value>.*)$'.format(command), re.M)
+    has_command_val = command_val_re.search(config)
+
     if arg == 'inject_map':
-        REGEX_INJECT = r'.*inject-map\s(?P<inject_map>\S+)\sexist-map\s(?P<exist_map>\S+)-*'
-
-        for line in splitted_config:
-            value = []
-            inject_group = {}
-            try:
-                match_inject = re.match(REGEX_INJECT, line, re.DOTALL)
-                inject_group = match_inject.groupdict()
-                inject_map = inject_group['inject_map']
-                exist_map = inject_group['exist_map']
-                value.append(inject_map)
-                value.append(exist_map)
-            except AttributeError:
-                value = []
+        inject_re = r'.*inject-map\s(?P<inject_map>\S+)\sexist-map\s(?P<exist_map>\S+)-*'
+
+        value = []
+        match_inject = re.match(inject_re, config, re.DOTALL)
+        if match_inject:
+            inject_group = match_inject.groupdict()
+            inject_map = inject_group['inject_map']
+            exist_map = inject_group['exist_map']
+            value.append(inject_map)
+            value.append(exist_map)
+
+            inject_map_command = ('inject-map {0} exist-map {1} '
+                                  'copy-attributes'.format(
+                                      inject_group['inject_map'],
+                                      inject_group['exist_map']))
+
+            inject_re = re.compile(r'\s+{0}\s*$'.format(inject_map_command), re.M)
+            if inject_re.search(config):
+                value.append('copy_attributes')
+
+    elif arg in ['networks', 'redistribute']:
+        value = []
+        if has_command_val:
+            value = has_command_val.group('value').split()
 
             if value:
-                copy_attributes = False
-                inject_map_command = ('inject-map {0} exist-map {1} '
-                                      'copy-attributes'.format(
-                                          inject_group['inject_map'],
-                                          inject_group['exist_map']))
-
-                REGEX = re.compile(r'\s+{0}\s*$'.format(inject_map_command), re.M)
-                try:
-                    if REGEX.search(config):
-                        copy_attributes = True
-                except TypeError:
-                    copy_attributes = False
-
-                if copy_attributes:
-                    value.append('copy_attributes')
-                value_list.append(value)
-
-    elif arg == 'networks':
-        REGEX_NETWORK = re.compile(r'(?:network\s)(?P<value>.*)$')
-
-        for line in splitted_config:
-            value = []
-            if 'network' in line:
-                value = REGEX_NETWORK.search(line).group('value').split()
-
-                if value:
-                    if len(value) == 3:
-                        value.pop(1)
-                    value_list.append(value)
-
-    elif arg == 'redistribute':
-        RED_REGEX = re.compile(r'(?:{0}\s)(?P<value>.*)$'.format(
-            PARAM_TO_COMMAND_KEYMAP[arg]), re.M)
-        for line in splitted_config:
-            value = []
-            if 'redistribute' in line:
-                value = RED_REGEX.search(line).group('value').split()
-                if value:
-                    if len(value) == 3:
-                        value.pop(1)
-                    elif len(value) == 4:
-                        value = ['{0} {1}'.format(
-                            value[0], value[1]), value[3]]
-                    value_list.append(value)
-    return value_list
-
-
-def get_custom_string_value(config, arg, module):
-    value = ''
-    if arg.startswith('distance'):
-        REGEX_DISTANCE = ('.*distance\s(?P<d_ebgp>\w+)\s(?P<d_ibgp>\w+)'
-                          '\s(?P<d_local>\w+)')
-        try:
-            match_distance = re.match(REGEX_DISTANCE, config, re.DOTALL)
+                if len(value) == 3:
+                    value.pop(1)
+                elif arg == 'redistribute' and len(value) == 4:
+                    value = ['{0} {1}'.format(
+                        value[0], value[1]), value[3]]
+
+    elif command == 'distance':
+        distance_re = r'.*distance\s(?P<d_ebgp>\w+)\s(?P<d_ibgp>\w+)\s(?P<d_local>\w+)'
+        match_distance = re.match(distance_re, config, re.DOTALL)
+
+        value = ''
+        if match_distance:
             distance_group = match_distance.groupdict()
-        except AttributeError:
-            distance_group = {}
 
-        if distance_group:
             if arg == 'distance_ebgp':
                 value = distance_group['d_ebgp']
             elif arg == 'distance_ibgp':
@@ -420,22 +387,17 @@ def get_custom_string_value(config, arg, module):
             elif arg == 'distance_local':
                 value = distance_group['d_local']
 
-    elif arg.startswith('dampening'):
-        REGEX = re.compile(r'(?:{0}\s)(?P<value>.*)$'.format(
-            PARAM_TO_COMMAND_KEYMAP[arg]), re.M)
+    elif command.split()[0] == 'dampening':
+        value = ''
         if arg == 'dampen_igp_metric' or arg == 'dampening_routemap':
-            value = ''
-            if PARAM_TO_COMMAND_KEYMAP[arg] in config:
-                value = REGEX.search(config).group('value')
+            if command in config:
+                value = has_command_val.group('value')
         else:
-            REGEX_DAMPENING = r'.*dampening\s(?P<half>\w+)\s(?P<reuse>\w+)\s(?P<suppress>\w+)\s(?P<max_suppress>\w+)'
-            try:
-                match_dampening = re.match(REGEX_DAMPENING, config, re.DOTALL)
+            dampening_re = r'.*dampening\s(?P<half>\w+)\s(?P<reuse>\w+)\s(?P<suppress>\w+)\s(?P<max_suppress>\w+)'
+            match_dampening = re.match(dampening_re, config, re.DOTALL)
+            if match_dampening:
                 dampening_group = match_dampening.groupdict()
-            except AttributeError:
-                dampening_group = {}
 
-            if dampening_group:
                 if arg == 'dampening_half_time':
                     value = dampening_group['half']
                 elif arg == 'dampening_reuse_time':
@@ -446,34 +408,17 @@ def get_custom_string_value(config, arg, module):
                     value = dampening_group['max_suppress']
 
     elif arg == 'table_map_filter':
-        TMF_REGEX = re.compile(r'\s+table-map.*filter$', re.M)
+        tmf_regex = re.compile(r'\s+table-map.*filter$', re.M)
         value = False
-        try:
-            if TMF_REGEX.search(config):
-                value = True
-        except TypeError:
-            value = False
+        if tmf_regex.search(config):
+            value = True
+
     elif arg == 'table_map':
-        TM_REGEX = re.compile(r'(?:table-map\s)(?P<value>\S+)(\sfilter)?$', re.M)
+        tm_regex = re.compile(r'(?:table-map\s)(?P<value>\S+)(\sfilter)?$', re.M)
+        has_tablemap = tm_regex.search(config)
         value = ''
-        if PARAM_TO_COMMAND_KEYMAP[arg] in config:
-            value = TM_REGEX.search(config).group('value')
-    return value
-
-
-def get_value(arg, config, module):
-    custom = [
-        'inject_map',
-        'networks',
-        'redistribute'
-    ]
-
-    if arg in custom:
-        value = get_custom_list_value(config, arg, module)
-
-    elif (arg.startswith('distance') or arg.startswith('dampening') or
-          arg.startswith('table_map')):
-        value = get_custom_string_value(config, arg, module)
+        if has_tablemap:
+            value = has_tablemap.group('value')
 
     elif arg in BOOL_PARAMS:
         command_re = re.compile(r'\s+{0}\s*'.format(command), re.M)
@@ -483,12 +428,10 @@ def get_value(arg, config, module):
             value = True
 
     else:
-        command_val_re = re.compile(r'(?:{0}\s)(?P<value>.*)$'.format(PARAM_TO_COMMAND_KEYMAP[arg]), re.M)
         value = ''
 
-        has_command = command_val_re.search(config)
-        if has_command:
-            value = has_command.group('value')
+        if has_command_val:
+            value = has_command_val.group('value')
 
     return value
 
@@ -526,10 +469,10 @@ def get_existing(module, args, warnings):
 
 def apply_key_map(key_map, table):
     new_dict = {}
-    for key in table:
+    for key, value in table.items():
         new_key = key_map.get(key)
         if new_key:
-            new_dict[new_key] = table.get(key)
+            new_dict[new_key] = value
 
     return new_dict
 
@@ -741,7 +684,7 @@ def state_present(module, existing, proposed, candidate):
             candidate.add(commands, parents=parents)
 
 
-def state_absent(module, existing, proposed, candidate):
+def state_absent(module, candidate):
     commands = []
     parents = ["router bgp {0}".format(module.params['asn'])]
     if module.params['vrf'] != 'default':
@@ -830,12 +773,10 @@ def main():
     proposed_args = dict((k, v) for k, v in module.params.items()
                          if v is not None and k in args)
 
-    if proposed_args.get('networks'):
-        if proposed_args['networks'][0] == 'default':
-            proposed_args['networks'] = 'default'
-    if proposed_args.get('inject_map'):
-        if proposed_args['inject_map'][0] == 'default':
-            proposed_args['inject_map'] = 'default'
+    for arg in ['networks', 'inject_map']:
+        if proposed_args.get(arg):
+            if proposed_args[arg][0] == 'default':
+                proposed_args[arg] = 'default'
 
     proposed = {}
     for key, value in proposed_args.items():
@@ -849,12 +790,13 @@ def main():
     if state == 'present':
         state_present(module, existing, proposed, candidate)
     elif state == 'absent' and existing:
-        state_absent(module, existing, proposed, candidate)
+        state_absent(module, candidate)
 
     if candidate:
+        candidate = candidate.items_text()
         load_config(module, candidate)
         result['changed'] = True
-        result['commands'] = candidate.items_text()
+        result['commands'] = candidate
     else:
         result['commands'] = []
 
diff --git a/lib/ansible/modules/network/nxos/nxos_bgp_neighbor_af.py b/lib/ansible/modules/network/nxos/nxos_bgp_neighbor_af.py
--- a/lib/ansible/modules/network/nxos/nxos_bgp_neighbor_af.py
+++ b/lib/ansible/modules/network/nxos/nxos_bgp_neighbor_af.py
@@ -352,7 +352,8 @@ def get_value(arg, config, module):
     ]
     command = PARAM_TO_COMMAND_KEYMAP[arg]
     has_command = re.search(r'\s+{0}\s*'.format(command), config, re.M)
-    has_command_val = command_val_re.search(r'(?:{0}\s)(?P<value>.*)$'.format(command), config, re.M)
+    has_command_val = re.search(r'(?:{0}\s)(?P<value>.*)$'.format(command), config, re.M)
+    value = ''
 
     if arg in custom:
         value = get_custom_value(arg, config, module)
@@ -363,7 +364,6 @@ def get_value(arg, config, module):
             value = True
 
     elif command.split()[0] in ['filter-list', 'prefix-list', 'route-map']:
-        value = ''
         direction = arg.rsplit('_', 1)[1]
         if has_command_val:
             params = has_command_val.group('value').split()
@@ -376,11 +376,8 @@ def get_value(arg, config, module):
             if has_command_val:
                 value = has_command_val.group('value')
 
-    else:
-        value = ''
-
-        if has_command_val:
-            value = has_command_val.group('value')
+    elif has_command_val:
+        value = has_command_val.group('value')
 
     return value
 
@@ -747,7 +744,7 @@ def main():
     if state == 'present':
         state_present(module, existing, proposed, candidate)
     elif state == 'absent' and existing:
-        state_absent(module, existing, proposed, candidate)
+        state_absent(module, existing, candidate)
 
     if candidate:
         candidate = candidate.items_text()


Test patch:
diff --git a/test/units/modules/network/nxos/fixtures/nxos_bgp_config.cfg b/test/units/modules/network/nxos/fixtures/nxos_bgp_config.cfg
--- a/test/units/modules/network/nxos/fixtures/nxos_bgp_config.cfg
+++ b/test/units/modules/network/nxos/fixtures/nxos_bgp_config.cfg
@@ -5,4 +5,7 @@ router bgp 65535
   event-history cli size medium
   event-history detail
   vrf test2
+    address-family ipv4 unicast
     timers bgp 1 10
+  neighbor 3.3.3.5
+    address-family ipv4 unicast
diff --git a/test/units/modules/network/nxos/test_nxos_bgp_af.py b/test/units/modules/network/nxos/test_nxos_bgp_af.py
--- a/test/units/modules/network/nxos/test_nxos_bgp_af.py
+++ b/test/units/modules/network/nxos/test_nxos_bgp_af.py
@@ -48,17 +48,21 @@ def load_fixtures(self, commands=None):
     def test_nxos_bgp_af(self):
         set_module_args(dict(asn=65535, afi='ipv4', safi='unicast'))
         self.execute_module(
-            changed=True,
+            changed=True, sort=False,
             commands=['router bgp 65535', 'address-family ipv4 unicast']
         )
 
     def test_nxos_bgp_af_vrf(self):
         set_module_args(dict(asn=65535, vrf='test', afi='ipv4', safi='unicast'))
         self.execute_module(
-            changed=True,
+            changed=True, sort=False,
             commands=['router bgp 65535', 'vrf test', 'address-family ipv4 unicast']
         )
 
+    def test_nxos_bgp_af_vrf_exists(self):
+        set_module_args(dict(asn=65535, vrf='test2', afi='ipv4', safi='unicast'))
+        self.execute_module(changed=False, commands=[])
+
     def test_nxos_bgp_af_dampening_routemap(self):
         set_module_args(dict(asn=65535, afi='ipv4', safi='unicast',
                              dampening_routemap='route-map-a'))
diff --git a/test/units/modules/network/nxos/test_nxos_bgp_neighbor_af.py b/test/units/modules/network/nxos/test_nxos_bgp_neighbor_af.py
--- a/test/units/modules/network/nxos/test_nxos_bgp_neighbor_af.py
+++ b/test/units/modules/network/nxos/test_nxos_bgp_neighbor_af.py
@@ -53,3 +53,14 @@ def test_nxos_bgp_neighbor_af(self):
             'router bgp 65535', 'neighbor 3.3.3.3', 'address-family ipv4 unicast',
             'route-reflector-client'
         ])
+
+    def test_nxos_bgp_neighbor_af_exists(self):
+        set_module_args(dict(asn=65535, neighbor='3.3.3.5', afi='ipv4', safi='unicast'))
+        self.execute_module(changed=False, commands=[])
+
+    def test_nxos_bgp_neighbor_af_absent(self):
+        set_module_args(dict(asn=65535, neighbor='3.3.3.5', afi='ipv4', safi='unicast', state='absent'))
+        self.execute_module(
+            changed=True, sort=False,
+            commands=['router bgp 65535', 'neighbor 3.3.3.5', 'no address-family ipv4 unicast']
+        )