Repository: ansible/ansible
Pull request: fix reload smu issue
##### SUMMARY
<!--- Describe the change, including rationale and design decisions -->
Fixes #41268 
<!--- If you are fixing an existing issue, please include "Fixes #nnn" in your
commit message and your description; but you should still explain what
the change does.-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bugfix Pull Request
##### COMPONENT NAME
<!--- Name of the module, plugin, module or task -->
nxos_rpm

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.7.0.dev0 (devel 1a70681630) last updated 2018/06/06 16:06:26 (GMT -400)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /root/agents-ci/ansible/lib/ansible
  executable location = /root/agents-ci/ansible/bin/ansible
  python version = 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4]

```


##### ADDITIONAL INFORMATION
<!--- Include additional information to help people understand the change here.
For bugs that don't have a linked bug report, a step-by-step reproduction
of the problem is helpful. -->
* This PR fixes #41268 
* The module now checks for return code and exits gracefully.
* In addition to this, the patch needs to be committed after the reload of the device, this needs to be taken care in the playbook and the integration test cases are modified for this case. 
* For removal of the reload patch too, the device reboots after deactivation and this is taken care as well.

<!--- Paste verbatim command output below, e.g. before and after your change -->
```

```


Problem statement:
nxos_rpm does not properly work for reload patches
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
nxos_rpm

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.7.0.dev0 (devel 1a70681630) last updated 2018/06/06 16:06:26 (GMT -400)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /root/agents-ci/ansible/lib/ansible
  executable location = /root/agents-ci/ansible/bin/ansible
  python version = 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4]

```

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->
Ansible Server : Ubuntu 14.04
Device: N9K running 7.0(3)I6(1)

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->
* Currently, nxos_rpm works well for feature RPMs and nxos patches which do not need reload of the device. However, if the patch is a reload type, the device reboots after the activation of the patch and this is causing ansible server to report connection error even though the operation itself is successful. 
This needs to be addressed and a graceful exit is needed from the module.

Note: I will open a PR shortly to address these issues.

Hints from issue discussion:

Commit message: fix reload smu issue

Patch to review:
diff --git a/lib/ansible/modules/network/nxos/nxos_rpm.py b/lib/ansible/modules/network/nxos/nxos_rpm.py
index fc8e49985bb018..16999ccecf084d 100644
--- a/lib/ansible/modules/network/nxos/nxos_rpm.py
+++ b/lib/ansible/modules/network/nxos/nxos_rpm.py
@@ -37,6 +37,8 @@
     - For patches, the minimum platform version needed is 7.0(3)I2(5)
     - For feature rpms, the minimum platform version needed is 7.0(3)I6(1)
     - The module manages the entire RPM lifecycle (Add, activate, commit, deactivate, remove)
+    - For reload patches, this module is NOT idempotent until the patch is
+      committed.
 options:
     pkg:
         description:
@@ -49,7 +51,8 @@
               their default values.
         default: bootflash
     aggregate:
-      description: List of RPM/patch definitions.
+        description:
+            - List of RPM/patch definitions.
     state:
         description:
             - If the state is present, the rpm will be installed,
@@ -113,7 +116,7 @@ def config_cmd_operation(module, cmd):
     while iteration < 10:
         msg = load_config(module, [cmd], True)
         if msg:
-            if 'another install operation is in progress' in msg[0].lower():
+            if 'another install operation is in progress' in msg[0].lower() or 'failed' in msg[0].lower():
                 time.sleep(2)
                 iteration += 1
             else:
@@ -153,6 +156,25 @@ def activate_operation(module, show_cmd, pkg):
     return cmd
 
 
+def activate_reload(module, pkg, flag):
+    iteration = 0
+    if flag:
+        cmd = 'install activate {0} forced'.format(pkg)
+    else:
+        cmd = 'install deactivate {0} forced'.format(pkg)
+    opts = {'ignore_timeout': True}
+    while iteration < 10:
+        msg = load_config(module, [cmd], True, opts)
+        if msg:
+            if isinstance(msg[0], int):
+                if msg[0] == -32603:
+                    return cmd
+            elif isinstance(msg[0], str):
+                if 'another install operation is in progress' in msg[0].lower() or 'failed' in msg[0].lower():
+                    time.sleep(2)
+                    iteration += 1
+
+
 def commit_operation(module, show_cmd, pkg):
     cmd = 'install commit {0}'.format(pkg)
     config_cmd_operation(module, cmd)
@@ -176,6 +198,7 @@ def remove_operation(module, show_cmd, pkg):
 
 def install_remove_rpm(module, full_pkg, file_system, state):
     commands = []
+    reload_patch = False
 
     splitted_pkg = full_pkg.split('.')
     pkg = '.'.join(splitted_pkg[0:-1])
@@ -184,6 +207,7 @@ def install_remove_rpm(module, full_pkg, file_system, state):
     show_active = 'show install active'
     show_commit = 'show install committed'
     show_patches = 'show install patches'
+    show_pkg_info = 'show install pkg-info {0}'.format(pkg)
 
     if state == 'present':
         inactive_body = execute_show_command(show_inactive, module)
@@ -192,14 +216,23 @@ def install_remove_rpm(module, full_pkg, file_system, state):
         if pkg not in inactive_body and pkg not in active_body:
             commands.append(add_operation(module, show_inactive, file_system, full_pkg, pkg))
 
+        patch_type_body = execute_show_command(show_pkg_info, module)
+        if patch_type_body and 'Patch Type    :  reload' in patch_type_body:
+            # This is reload smu/patch rpm
+            reload_patch = True
+
         if pkg not in active_body:
-            commands.append(activate_operation(module, show_active, pkg))
+            if reload_patch:
+                commands.append(activate_reload(module, pkg, True))
+                return commands
+            else:
+                commands.append(activate_operation(module, show_active, pkg))
 
         commit_body = execute_show_command(show_commit, module)
         if pkg not in commit_body:
             patch_body = execute_show_command(show_patches, module)
             if pkg in patch_body:
-                # this is an smu
+                # This is smu/patch rpm
                 commands.append(commit_operation(module, show_active, pkg))
             else:
                 err = 'Operation "install activate {0} forced" Failed'.format(pkg)
@@ -209,13 +242,22 @@ def install_remove_rpm(module, full_pkg, file_system, state):
         commit_body = execute_show_command(show_commit, module)
         active_body = execute_show_command(show_active, module)
 
+        patch_type_body = execute_show_command(show_pkg_info, module)
+        if patch_type_body and 'Patch Type    :  reload' in patch_type_body:
+            # This is reload smu/patch rpm
+            reload_patch = True
+
         if pkg in commit_body and pkg in active_body:
-            commands.append(deactivate_operation(module, show_active, pkg, True))
-            commit_body = execute_show_command(show_commit, module)
-            if pkg in commit_body:
-                # This is smu/patch rpm
-                commands.append(commit_operation(module, show_inactive, pkg))
-            commands.append(remove_operation(module, show_inactive, pkg))
+            if reload_patch:
+                commands.append(activate_reload(module, pkg, False))
+                return commands
+            else:
+                commands.append(deactivate_operation(module, show_active, pkg, True))
+                commit_body = execute_show_command(show_commit, module)
+                if pkg in commit_body:
+                    # This is smu/patch rpm
+                    commands.append(commit_operation(module, show_inactive, pkg))
+                commands.append(remove_operation(module, show_inactive, pkg))
 
         elif pkg in commit_body:
             # This is smu/patch rpm
@@ -224,8 +266,12 @@ def install_remove_rpm(module, full_pkg, file_system, state):
 
         elif pkg in active_body:
             # This is smu/patch rpm
-            commands.append(deactivate_operation(module, show_inactive, pkg, False))
-            commands.append(remove_operation(module, show_inactive, pkg))
+            if reload_patch:
+                commands.append(activate_reload(module, pkg, False))
+                return commands
+            else:
+                commands.append(deactivate_operation(module, show_inactive, pkg, False))
+                commands.append(remove_operation(module, show_inactive, pkg))
 
         else:
             inactive_body = execute_show_command(show_inactive, module)
diff --git a/test/integration/targets/nxos_rpm/tests/common/sanity.yaml b/test/integration/targets/nxos_rpm/tests/common/sanity.yaml
index d570249f5b66a7..2c70c02a9cb459 100644
--- a/test/integration/targets/nxos_rpm/tests/common/sanity.yaml
+++ b/test/integration/targets/nxos_rpm/tests/common/sanity.yaml
@@ -22,7 +22,7 @@
 - block:
   - name: Install smu RPM
     nxos_rpm: &tsmurpm
-      pkg: "nxos.sample-n9k_ALL-1.0.0-7.0.3.I7.3.lib32_n9000.rpm"
+      pkg: "nxos.sample-n9k_ALL-1.0.0-7.0.3.I6.1.lib32_n9000.rpm"
     register: result
 
   - assert: &true1
@@ -39,7 +39,7 @@
 
   - name: Remove smu RPM
     nxos_rpm: &rsmurpm
-      pkg: "nxos.sample-n9k_ALL-1.0.0-7.0.3.I7.3.lib32_n9000.rpm"
+      pkg: "nxos.sample-n9k_ALL-1.0.0-7.0.3.I6.1.lib32_n9000.rpm"
       state: absent
     register: result
 
@@ -92,4 +92,62 @@
 
   when: sdk_run
 
+- block:
+  - name: Install reload smu RPM
+    nxos_rpm: &trsmurpm
+      pkg: "nxos.CSCve91311-n9k_ALL-1.0.0-7.0.3.I6.1.lib32_n9000.rpm"
+    register: result
+
+  - assert: *true1
+
+  - name: Wait for device to come back up
+    wait_for:
+      port: 22
+      state: started
+      timeout: 600
+      delay: 60
+      host: "{{ inventory_hostname }}"
+
+  - name: Continue with install reload smu RPM
+    nxos_rpm: *trsmurpm
+    register: result
+
+  - assert: *true1
+
+  - name: Check Idempotence
+    nxos_rpm: *trsmurpm
+    register: result
+
+  - assert: *false1
+
+  - name: Remove reload smu RPM
+    nxos_rpm: &rrsmurpm
+      pkg: "nxos.CSCve91311-n9k_ALL-1.0.0-7.0.3.I6.1.lib32_n9000.rpm"
+      state: absent
+    register: result
+
+  - assert: *true1
+
+  - name: Wait for device to come back up
+    wait_for:
+      port: 22
+      state: started
+      timeout: 600
+      delay: 60
+      host: "{{ inventory_hostname }}"
+
+  - name: Continue with remove reload smu RPM
+    nxos_rpm: *rrsmurpm
+    register: result
+
+  - assert: *true1
+
+  - name: Check Idempotence
+    nxos_rpm: *rrsmurpm
+    register: result
+
+  - assert: *false1
+
+  when: smu_run
+
 - debug: msg="END connection={{ ansible_connection }} nxos_rpm sanity test"


Code review comments:
diff_hunk: @@ -92,4 +92,62 @@
 
   when: sdk_run
 
+- block:
+  - name: Install reload smu RPM
+    nxos_rpm: &trsmurpm
+      pkg: "nxos.CSCve91311-n9k_ALL-1.0.0-7.0.3.I6.1.lib32_n9000.rpm"
+    register: result
+
+  - assert: *true1
+
+  - name: Wait for device to come back up
+    wait_for:
+      port: 22
+      state: started
+      timeout: 600
+      delay: 60
+      host: "{{ inventory_hostname }}"
path: test/integration/targets/nxos_rpm/tests/common/sanity.yaml
text: @user1:
please use `inventory_hostname_short` instead

diff_hunk: @@ -92,4 +92,62 @@
 
   when: sdk_run
 
+- block:
+  - name: Install reload smu RPM
+    nxos_rpm: &trsmurpm
+      pkg: "nxos.CSCve91311-n9k_ALL-1.0.0-7.0.3.I6.1.lib32_n9000.rpm"
+    register: result
+
+  - assert: *true1
+
+  - name: Wait for device to come back up
+    wait_for:
+      port: 22
+      state: started
+      timeout: 600
+      delay: 60
+      host: "{{ inventory_hostname }}"
+
+  - name: Continue with install reload smu RPM
+    nxos_rpm: *trsmurpm
+    register: result
+
+  - assert: *true1
+
+  - name: Check Idempotence
+    nxos_rpm: *trsmurpm
+    register: result
+
+  - assert: *false1
+
+  - name: Remove reload smu RPM
+    nxos_rpm: &rrsmurpm
+      pkg: "nxos.CSCve91311-n9k_ALL-1.0.0-7.0.3.I6.1.lib32_n9000.rpm"
+      state: absent
+    register: result
+
+  - assert: *true1
+
+  - name: Wait for device to come back up
+    wait_for:
+      port: 22
+      state: started
+      timeout: 600
+      delay: 60
+      host: "{{ inventory_hostname }}"
path: test/integration/targets/nxos_rpm/tests/common/sanity.yaml
text: @user1:
use `inventory_hostname_short` instead.