Repository: ansible/ansible

Issue:
nxos control socket does not work properly following nxos upgrade or reload
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
nxos_install_os

##### ANSIBLE VERSION
```
ansible 2.6.0 (devel 33f358383a) last updated 2018/04/26 08:59:33 (GMT -400)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/Users/mwiebe/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible
  executable location = /Users/mwiebe/Projects/nxos_ansible/fix_ansible/bin/ansible
  python version = 2.7.13 (default, Apr  4 2017, 08:47:57) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)]
```

##### CONFIGURATION
```
DEFAULT_ROLES_PATH(env: ANSIBLE_ROLES_PATH) = [u'/Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets']
PARAMIKO_HOST_KEY_AUTO_ADD(env: ANSIBLE_PARAMIKO_HOST_KEY_AUTO_ADD) = True
PERSISTENT_COMMAND_TIMEOUT(env: ANSIBLE_PERSISTENT_COMMAND_TIMEOUT) = 600
PERSISTENT_CONNECT_TIMEOUT(env: ANSIBLE_PERSISTENT_CONNECT_TIMEOUT) = 600
```

##### OS / ENVIRONMENT
MacOS

##### SUMMARY
When running nxos playbooks that result in an reboot of the nxos device, and already existing control socket will be used following the upgrade, but the socket appears to be in a bad state.

##### STEPS TO REPRODUCE
I can provide the playbooks if needed, but the important thing to note is the workflow.

1. Call `nxos_install_os` to upgrade or downgrade the device.
2. After the upgrade, any tasks that follow the upgrade attempt to use the existing socket don't work properly.

**NOTE 1:** If I clear the persistent sockets after the upgrade but before the next task after the upgrade things work ok, but this will not scale since I have to clear all of the existing sockets and potentially impact other ansible  runs that are already in progress.

**NOTE 2:** I tried using the meta: reset_connection instead of cleaning up the sockets, but it does not work when multiple hosts are specified in the inventory file.

##### EXPECTED RESULTS
Any task that follows an upgrade should succeed if the existing socket is re-used.  In this specific example, the task that gets executed after the upgrade is to check the new OS version.

##### ACTUAL RESULTS
###### Logs when the persistent sockets are **NOT** Cleared.

The socket used for the upgrade and task after the upgrade is the same:

Upgrade:
<n3k.example.com> local domain socket path is /Users/mwiebe/.ansible/pc/5e93d558b8

After Upgrade:
<n3k.example.com> socket_path: /Users/mwiebe/.ansible/pc/5e93d558b8


```
TASK [nxos_install_os : Install OS image n3000-uk9.6.0.2.U6.3a.bin] *********************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_with_kick.yaml:2
<n3k.example.com> connection transport is cli
<n3k.example.com> using connection plugin network_cli (was local)
<n3k.example.com> starting connection from persistent connection plugin
<n3k.example.com> found existing local domain socket, using it!
<n3k.example.com> updating play_context for connection
<n3k.example.com> 
<n3k.example.com> local domain socket path is /Users/mwiebe/.ansible/pc/5e93d558b8
<n3k.example.com> socket_path: /Users/mwiebe/.ansible/pc/5e93d558b8
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/network/nxos/nxos_install_os.py
<n3k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n3k.example.com> EXEC /bin/sh -c 'echo ~mwiebe && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524751908.97-48848500574857 `" && echo ansible-tmp-1524751908.97-48848500574857="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524751908.97-48848500574857 `" ) && sleep 0'
<n3k.example.com> PUT /Users/mwiebe/.ansible/tmp/ansible-local-16124AMsjy7/tmprxRDNY TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1524751908.97-48848500574857/nxos_install_os.py
<n3k.example.com> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1524751908.97-48848500574857/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1524751908.97-48848500574857/nxos_install_os.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1524751908.97-48848500574857/nxos_install_os.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-tmp-1524751908.97-48848500574857/ > /dev/null 2>&1 && sleep 0'
changed: [n3k.example.com] => {
    "changed": true, 
    "install_state": [
        "Compatibility check is done:", 
        "Module  bootable          Impact  Install-type  Reason", 
        "------  --------  --------------  ------------  ------", 
        "     1       yes      disruptive         reset  Forced by the user", 
        "Images will be upgraded according to following table:", 
        "Module             Image         Running-Version             New-Version  Upg-Required", 
        "------  ----------------  ----------------------  ----------------------  ------------", 
        "     1            system            6.0(2)U6(2a)            6.0(2)U6(3a)           yes", 
        "     1         kickstart            6.0(2)U6(2a)            6.0(2)U6(3a)           yes", 
        "     1              bios      v4.5.0(11/09/2017)      v1.2.0(08/25/2011)            no"
    ], 
    "invocation": {
        "module_args": {
            "host": "n3k.example.com", 
            "issu": "no", 
            "kickstart_image_file": "n3000-uk9-kickstart.6.0.2.U6.3a.bin", 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": null, 
            "provider": {
                "host": "n3k.example.com", 
                "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "port": null, 
                "ssh_keyfile": null, 
                "timeout": 500, 
                "transport": "cli", 
                "use_proxy": true, 
                "use_ssl": null, 
                "username": "admin", 
                "validate_certs": null
            }, 
            "ssh_keyfile": null, 
            "system_image_file": "n3000-uk9.6.0.2.U6.3a.bin", 
            "timeout": 500, 
            "transport": "cli", 
            "use_proxy": true, 
            "use_ssl": null, 
            "username": "admin", 
            "validate_certs": null
        }
    }
}

TASK [nxos_install_os : debug] **********************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_with_kick.yaml:11
ok: [n3k.example.com] => {
    "msg": " [u'Compatibility check is done:', u'Module  bootable          Impact  Install-type  Reason', u'------  --------  --------------  ------------  ------', u'     1       yes      disruptive         reset  Forced by the user', u'Images will be upgraded according to following table:', u'Module             Image         Running-Version             New-Version  Upg-Required', u'------  ----------------  ----------------------  ----------------------  ------------', u'     1            system            6.0(2)U6(2a)            6.0(2)U6(3a)           yes', u'     1         kickstart            6.0(2)U6(2a)            6.0(2)U6(3a)           yes', u'     1              bios      v4.5.0(11/09/2017)      v1.2.0(08/25/2011)            no']"
}

TASK [nxos_install_os : Wait for device to come back up with new image] *****************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_with_kick.yaml:13
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/utilities/logic/wait_for.py
<n3k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n3k.example.com> EXEC /bin/sh -c 'echo ~mwiebe && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752346.88-154324758837876 `" && echo ansible-tmp-1524752346.88-154324758837876="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752346.88-154324758837876 `" ) && sleep 0'
<n3k.example.com> PUT /Users/mwiebe/.ansible/tmp/ansible-local-16124AMsjy7/tmpquvNWs TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752346.88-154324758837876/wait_for.py
<n3k.example.com> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752346.88-154324758837876/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752346.88-154324758837876/wait_for.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752346.88-154324758837876/wait_for.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752346.88-154324758837876/ > /dev/null 2>&1 && sleep 0'
ok: [n3k.example.com] => {
    "changed": false, 
    "elapsed": 154, 
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED", 
                "FIN_WAIT1", 
                "FIN_WAIT2", 
                "SYN_RECV", 
                "SYN_SENT", 
                "TIME_WAIT"
            ], 
            "connect_timeout": 5, 
            "delay": 60, 
            "exclude_hosts": null, 
            "host": "n3k.example.com", 
            "msg": null, 
            "path": null, 
            "port": 22, 
            "search_regex": null, 
            "sleep": 1, 
            "state": "started", 
            "timeout": 500
        }
    }, 
    "path": null, 
    "port": 22, 
    "search_regex": null, 
    "state": "started"
}

TASK [nxos_install_os : include] ********************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:23
skipping: [n3k.example.com] => {
    "changed": false, 
    "skip_reason": "Conditional result was False"
}

TASK [nxos_install_os : Check installed OS for newly installed version {{ tv }}] ********************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:28
<n3k.example.com> connection transport is cli
<n3k.example.com> using connection plugin network_cli (was local)
<n3k.example.com> starting connection from persistent connection plugin
<n3k.example.com> found existing local domain socket, using it!
<n3k.example.com> updating play_context for connection
<n3k.example.com> 
<n3k.example.com> local domain socket path is /Users/mwiebe/.ansible/pc/5e93d558b8
<n3k.example.com> socket_path: /Users/mwiebe/.ansible/pc/5e93d558b8
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/network/nxos/nxos_command.py
<n3k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n3k.example.com> EXEC /bin/sh -c 'echo ~mwiebe && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752503.97-224069650570435 `" && echo ansible-tmp-1524752503.97-224069650570435="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752503.97-224069650570435 `" ) && sleep 0'
<n3k.example.com> PUT /Users/mwiebe/.ansible/tmp/ansible-local-16124AMsjy7/tmpaXHXeu TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752503.97-224069650570435/nxos_command.py
<n3k.example.com> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752503.97-224069650570435/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752503.97-224069650570435/nxos_command.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752503.97-224069650570435/nxos_command.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752503.97-224069650570435/ > /dev/null 2>&1 && sleep 0'
ok: [n3k.example.com] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "commands": [
                "show version | json"
            ], 
            "host": "n3k.example.com", 
            "interval": 1, 
            "match": "all", 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": null, 
            "provider": {
                "host": "n3k.example.com", 
                "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "port": null, 
                "ssh_keyfile": null, 
                "timeout": null, 
                "transport": "cli", 
                "use_proxy": true, 
                "use_ssl": null, 
                "username": "admin", 
                "validate_certs": null
            }, 
            "retries": 10, 
            "ssh_keyfile": null, 
            "timeout": null, 
            "transport": "cli", 
            "use_proxy": true, 
            "use_ssl": null, 
            "username": "admin", 
            "validate_certs": null, 
            "wait_for": null
        }
    }, 
    "stdout": [
        "None"
    ], 
    "stdout_lines": [
        [
            "None"
        ]
    ]
}

TASK [nxos_install_os : debug] **********************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:34
fatal: [n3k.example.com]: FAILED! => {
    "msg": "The task includes an option with an undefined variable. The error was: 'list object' has no attribute 'kickstart_ver_str'\n\nThe error appears to have been in '/Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml': line 34, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- debug: msg=\"Version detected {{ output['stdout_lines'][0]['kickstart_ver_str'] }}\"\n  ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes.  Always quote template expression brackets when they\nstart a value. For instance:\n\n    with_items:\n      - {{ foo }}\n\nShould be written as:\n\n    with_items:\n      - \"{{ foo }}\"\n"
}
	to retry, use: --limit @/Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/nxos.retry

PLAY RECAP ******************************************************************************************************************************************************************************************************************
n3k.example.com          : ok=47   changed=3    unreachable=0    failed=1   

(py2-ansible) mwiebe@MWIEBE-M-D2TB:~/Projects/nxos_ansible/fix_ansible/test/integration$
```



###### Logs when the persistent sockets **ARE** Cleared.

The socket used for the upgrade and task after the upgrade are different since they are cleared by the playbook.

Upgrade:
<n3k.example.com> local domain socket path is /Users/mwiebe/.ansible/pc/0f03b52e8a

After Upgrade:
<n3k.example.com> local domain socket path is /Users/mwiebe/.ansible/pc/0f03b52e8a

```
TASK [nxos_install_os : Install OS image n3000-uk9.6.0.2.U6.2a.bin] *********************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_with_kick.yaml:2
<n3k.example.com> connection transport is cli
<n3k.example.com> using connection plugin network_cli (was local)
<n3k.example.com> starting connection from persistent connection plugin
<n3k.example.com> found existing local domain socket, using it!
<n3k.example.com> updating play_context for connection
<n3k.example.com> 
<n3k.example.com> local domain socket path is /Users/mwiebe/.ansible/pc/0f03b52e8a
<n3k.example.com> socket_path: /Users/mwiebe/.ansible/pc/0f03b52e8a
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/network/nxos/nxos_install_os.py
<n3k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n3k.example.com> EXEC /bin/sh -c 'echo ~mwiebe && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752881.16-149956170048470 `" && echo ansible-tmp-1524752881.16-149956170048470="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752881.16-149956170048470 `" ) && sleep 0'
<n3k.example.com> PUT /Users/mwiebe/.ansible/tmp/ansible-local-21475eCJekn/tmpyUcTch TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752881.16-149956170048470/nxos_install_os.py
<n3k.example.com> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752881.16-149956170048470/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752881.16-149956170048470/nxos_install_os.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752881.16-149956170048470/nxos_install_os.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-tmp-1524752881.16-149956170048470/ > /dev/null 2>&1 && sleep 0'
changed: [n3k.example.com] => {
    "changed": true, 
    "install_state": [
        "Compatibility check is done:", 
        "Module  bootable          Impact  Install-type  Reason", 
        "------  --------  --------------  ------------  ------", 
        "     1       yes      disruptive         reset  Forced by the user", 
        "Images will be upgraded according to following table:", 
        "Module             Image         Running-Version             New-Version  Upg-Required", 
        "------  ----------------  ----------------------  ----------------------  ------------", 
        "     1            system            6.0(2)U6(3a)            6.0(2)U6(2a)           yes", 
        "     1         kickstart            6.0(2)U6(3a)            6.0(2)U6(2a)           yes", 
        "     1              bios      v4.5.0(11/09/2017)      v1.2.0(08/25/2011)            no"
    ], 
    "invocation": {
        "module_args": {
            "host": "n3k.example.com", 
            "issu": "no", 
            "kickstart_image_file": "n3000-uk9-kickstart.6.0.2.U6.2a.bin", 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": null, 
            "provider": {
                "host": "n3k.example.com", 
                "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "port": null, 
                "ssh_keyfile": null, 
                "timeout": 500, 
                "transport": "cli", 
                "use_proxy": true, 
                "use_ssl": null, 
                "username": "admin", 
                "validate_certs": null
            }, 
            "ssh_keyfile": null, 
            "system_image_file": "n3000-uk9.6.0.2.U6.2a.bin", 
            "timeout": 500, 
            "transport": "cli", 
            "use_proxy": true, 
            "use_ssl": null, 
            "username": "admin", 
            "validate_certs": null
        }
    }
}

TASK [nxos_install_os : debug] **********************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_with_kick.yaml:11
ok: [n3k.example.com] => {
    "msg": " [u'Compatibility check is done:', u'Module  bootable          Impact  Install-type  Reason', u'------  --------  --------------  ------------  ------', u'     1       yes      disruptive         reset  Forced by the user', u'Images will be upgraded according to following table:', u'Module             Image         Running-Version             New-Version  Upg-Required', u'------  ----------------  ----------------------  ----------------------  ------------', u'     1            system            6.0(2)U6(3a)            6.0(2)U6(2a)           yes', u'     1         kickstart            6.0(2)U6(3a)            6.0(2)U6(2a)           yes', u'     1              bios      v4.5.0(11/09/2017)      v1.2.0(08/25/2011)            no']"
}

TASK [nxos_install_os : Wait for device to come back up with new image] *****************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_with_kick.yaml:13
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/utilities/logic/wait_for.py
<n3k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n3k.example.com> EXEC /bin/sh -c 'echo ~mwiebe && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753433.28-173545017697864 `" && echo ansible-tmp-1524753433.28-173545017697864="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753433.28-173545017697864 `" ) && sleep 0'
<n3k.example.com> PUT /Users/mwiebe/.ansible/tmp/ansible-local-21475eCJekn/tmpU3pUKg TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753433.28-173545017697864/wait_for.py
<n3k.example.com> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753433.28-173545017697864/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753433.28-173545017697864/wait_for.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753433.28-173545017697864/wait_for.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753433.28-173545017697864/ > /dev/null 2>&1 && sleep 0'
ok: [n3k.example.com] => {
    "changed": false, 
    "elapsed": 148, 
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED", 
                "FIN_WAIT1", 
                "FIN_WAIT2", 
                "SYN_RECV", 
                "SYN_SENT", 
                "TIME_WAIT"
            ], 
            "connect_timeout": 5, 
            "delay": 60, 
            "exclude_hosts": null, 
            "host": "n3k.example.com", 
            "msg": null, 
            "path": null, 
            "port": 22, 
            "search_regex": null, 
            "sleep": 1, 
            "state": "started", 
            "timeout": 500
        }
    }, 
    "path": null, 
    "port": 22, 
    "search_regex": null, 
    "state": "started"
}

TASK [nxos_install_os : include] ********************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:23
skipping: [n3k.example.com] => {
    "changed": false, 
    "skip_reason": "Conditional result was False"
}

TASK [nxos_install_os : include] ********************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:26
included: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/clear_persistent_sockets.yaml for n3k.example.com

TASK [nxos_install_os : Clean up sockets with file module] ******************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/clear_persistent_sockets.yaml:2
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/files/file.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<127.0.0.1> EXEC /bin/sh -c 'echo ~mwiebe && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.33-241997183142611 `" && echo ansible-tmp-1524753582.33-241997183142611="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.33-241997183142611 `" ) && sleep 0'
<127.0.0.1> PUT /Users/mwiebe/.ansible/tmp/ansible-local-21475eCJekn/tmpHVEBOW TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.33-241997183142611/file.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.33-241997183142611/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.33-241997183142611/file.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.33-241997183142611/file.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.33-241997183142611/ > /dev/null 2>&1 && sleep 0'
changed: [n3k.example.com -> 127.0.0.1] => {
    "changed": true, 
    "diff": {
        "after": {
            "path": "/Users/mwiebe/.ansible/pc/", 
            "state": "absent"
        }, 
        "before": {
            "path": "/Users/mwiebe/.ansible/pc/", 
            "state": "directory"
        }
    }, 
    "invocation": {
        "module_args": {
            "attributes": null, 
            "backup": null, 
            "content": null, 
            "delimiter": null, 
            "diff_peek": null, 
            "directory_mode": null, 
            "follow": true, 
            "force": false, 
            "group": null, 
            "mode": null, 
            "original_basename": null, 
            "owner": null, 
            "path": "/Users/mwiebe/.ansible/pc/", 
            "recurse": false, 
            "regexp": null, 
            "remote_src": null, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": null, 
            "state": "absent", 
            "unsafe_writes": null, 
            "validate": null
        }
    }, 
    "path": "/Users/mwiebe/.ansible/pc/", 
    "state": "absent"
}

TASK [nxos_install_os : Display socket info after delete] *******************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/clear_persistent_sockets.yaml:8
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/commands/command.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<127.0.0.1> EXEC /bin/sh -c 'echo ~mwiebe && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.85-181762354849647 `" && echo ansible-tmp-1524753582.85-181762354849647="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.85-181762354849647 `" ) && sleep 0'
<127.0.0.1> PUT /Users/mwiebe/.ansible/tmp/ansible-local-21475eCJekn/tmpHb7qCH TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.85-181762354849647/command.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.85-181762354849647/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.85-181762354849647/command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.85-181762354849647/command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753582.85-181762354849647/ > /dev/null 2>&1 && sleep 0'
changed: [n3k.example.com -> 127.0.0.1] => {
    "changed": true, 
    "cmd": "/bin/ls /Users/mwiebe/.ansible", 
    "delta": "0:00:00.007333", 
    "end": "2018-04-26 10:39:43.075096", 
    "invocation": {
        "module_args": {
            "_raw_params": "/bin/ls /Users/mwiebe/.ansible", 
            "_uses_shell": true, 
            "chdir": null, 
            "creates": null, 
            "executable": "/bin/bash", 
            "removes": null, 
            "stdin": null, 
            "warn": true
        }
    }, 
    "rc": 0, 
    "start": "2018-04-26 10:39:43.067763", 
    "stderr": "", 
    "stderr_lines": [], 
    "stdout": "cp\ntmp", 
    "stdout_lines": [
        "cp", 
        "tmp"
    ]
}

TASK [nxos_install_os : debug] **********************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/clear_persistent_sockets.yaml:15
ok: [n3k.example.com] => {
    "msg": "Local Socket Info [u'cp', u'tmp']"
}

TASK [nxos_install_os : Check installed OS for newly installed version {{ tv }}] ********************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:28
<n3k.example.com> connection transport is cli
<n3k.example.com> using connection plugin network_cli (was local)
<n3k.example.com> starting connection from persistent connection plugin
<n3k.example.com> local domain socket does not exist, starting it
<n3k.example.com> control socket path is /Users/mwiebe/.ansible/pc/0f03b52e8a
<n3k.example.com> <n3k.example.com> ESTABLISH CONNECTION FOR USER: admin on PORT 22 TO n3k.example.com
<n3k.example.com> <n3k.example.com> ssh connection done, setting terminal
<n3k.example.com> <n3k.example.com> loaded terminal plugin for network_os nxos
<n3k.example.com> <n3k.example.com> loaded cliconf plugin for network_os nxos
<n3k.example.com> <n3k.example.com> firing event: on_open_shell()
<n3k.example.com> <n3k.example.com> ssh connection has completed successfully
<n3k.example.com> connection to remote device started successfully
<n3k.example.com> local domain socket listeners started successfully
<n3k.example.com> 
<n3k.example.com> local domain socket path is /Users/mwiebe/.ansible/pc/0f03b52e8a
<n3k.example.com> socket_path: /Users/mwiebe/.ansible/pc/0f03b52e8a
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/network/nxos/nxos_command.py
<n3k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n3k.example.com> EXEC /bin/sh -c 'echo ~mwiebe && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753586.66-250480194756863 `" && echo ansible-tmp-1524753586.66-250480194756863="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753586.66-250480194756863 `" ) && sleep 0'
<n3k.example.com> PUT /Users/mwiebe/.ansible/tmp/ansible-local-21475eCJekn/tmpEVz2ta TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753586.66-250480194756863/nxos_command.py
<n3k.example.com> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753586.66-250480194756863/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753586.66-250480194756863/nxos_command.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753586.66-250480194756863/nxos_command.py && sleep 0'
<n3k.example.com> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-tmp-1524753586.66-250480194756863/ > /dev/null 2>&1 && sleep 0'
ok: [n3k.example.com] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "commands": [
                "show version | json"
            ], 
            "host": "n3k.example.com", 
            "interval": 1, 
            "match": "all", 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": null, 
            "provider": {
                "host": "n3k.example.com", 
                "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "port": null, 
                "ssh_keyfile": null, 
                "timeout": null, 
                "transport": "cli", 
                "use_proxy": true, 
                "use_ssl": null, 
                "username": "admin", 
                "validate_certs": null
            }, 
            "retries": 10, 
            "ssh_keyfile": null, 
            "timeout": null, 
            "transport": "cli", 
            "use_proxy": true, 
            "use_ssl": null, 
            "username": "admin", 
            "validate_certs": null, 
            "wait_for": null
        }
    }, 
    "stdout": [
        {
            "bios_cmpl_time": "11/09/2017", 
            "bios_ver_str": "4.5.0", 
            "bootflash_size": "2007040", 
            "chassis_id": "Nexus 3048 Chassis", 
            "cpu_name": "Intel(R) Celeron(R) CPU        P450", 
            "header_str": "Cisco Nexus Operating System (NX-OS) Software\nTAC support: http://www.cisco.com/tac\nDocuments: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html\nCopyright (c) 2002-2016, Cisco Systems, Inc. All rights reserved.\nThe copyrights to certain works contained herein are owned by\nother third parties and are used and distributed under license.\nSome parts of this software are covered under the GNU Public\nLicense. A copy of the license is available at\nhttp://www.gnu.org/licenses/gpl.html.", 
            "host_name": "n3k-205", 
            "isan_cmpl_time": "2/14/2016 4:00:00", 
            "isan_file_name": "bootflash:///n3000-uk9.6.0.2.U6.2a.bin", 
            "isan_tmstmp": "02/14/2016 13:23:29", 
            "kern_uptm_days": "0", 
            "kern_uptm_hrs": "0", 
            "kern_uptm_mins": "2", 
            "kern_uptm_secs": "0", 
            "kick_cmpl_time": "2/14/2016 4:00:00", 
            "kick_file_name": "bootflash:///n3000-uk9-kickstart.6.0.2.U6.2a.bin", 
            "kick_tmstmp": "02/14/2016 12:57:46", 
            "kickstart_ver_str": "6.0(2)U6(2a)", 
            "loader_ver_str": "N/A", 
            "mem_type": "kB", 
            "memory": "3665492", 
            "module_id": "48x1GE + 4x10G Supervisor", 
            "power_seq_ver_str": "Module 1: version v4.4", 
            "proc_board_id": "FOC164121K6", 
            "rr_ctime": "Thu Apr 26 14:17:36 2018", 
            "rr_reason": "Disruptive upgrade", 
            "rr_service": null, 
            "rr_sys_ver": "6.0(2)U6(3a)", 
            "rr_usecs": "914976", 
            "sys_ver_str": "6.0(2)U6(2a)"
        }
    ], 
    "stdout_lines": [
        {
            "bios_cmpl_time": "11/09/2017", 
            "bios_ver_str": "4.5.0", 
            "bootflash_size": "2007040", 
            "chassis_id": "Nexus 3048 Chassis", 
            "cpu_name": "Intel(R) Celeron(R) CPU        P450", 
            "header_str": "Cisco Nexus Operating System (NX-OS) Software\nTAC support: http://www.cisco.com/tac\nDocuments: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html\nCopyright (c) 2002-2016, Cisco Systems, Inc. All rights reserved.\nThe copyrights to certain works contained herein are owned by\nother third parties and are used and distributed under license.\nSome parts of this software are covered under the GNU Public\nLicense. A copy of the license is available at\nhttp://www.gnu.org/licenses/gpl.html.", 
            "host_name": "n3k-205", 
            "isan_cmpl_time": "2/14/2016 4:00:00", 
            "isan_file_name": "bootflash:///n3000-uk9.6.0.2.U6.2a.bin", 
            "isan_tmstmp": "02/14/2016 13:23:29", 
            "kern_uptm_days": "0", 
            "kern_uptm_hrs": "0", 
            "kern_uptm_mins": "2", 
            "kern_uptm_secs": "0", 
            "kick_cmpl_time": "2/14/2016 4:00:00", 
            "kick_file_name": "bootflash:///n3000-uk9-kickstart.6.0.2.U6.2a.bin", 
            "kick_tmstmp": "02/14/2016 12:57:46", 
            "kickstart_ver_str": "6.0(2)U6(2a)", 
            "loader_ver_str": "N/A", 
            "mem_type": "kB", 
            "memory": "3665492", 
            "module_id": "48x1GE + 4x10G Supervisor", 
            "power_seq_ver_str": "Module 1: version v4.4", 
            "proc_board_id": "FOC164121K6", 
            "rr_ctime": "Thu Apr 26 14:17:36 2018", 
            "rr_reason": "Disruptive upgrade", 
            "rr_service": null, 
            "rr_sys_ver": "6.0(2)U6(3a)", 
            "rr_usecs": "914976", 
            "sys_ver_str": "6.0(2)U6(2a)"
        }
    ]
}

TASK [nxos_install_os : debug] **********************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:34
ok: [n3k.example.com] => {
    "msg": "Version detected 6.0(2)U6(2a)"
}
META: ran handlers
META: ran handlers

PLAY RECAP ******************************************************************************************************************************************************************************************************************
n3k.example.com          : ok=53   changed=5    unreachable=0    failed=0   

(py2-ansible) mwiebe@MWIEBE-M-D2TB:~/Projects/nxos_ansible/fix_ansible/test/integration$
```


Discussion:
Files identified in the description:
* [lib/ansible/modules/network/nxos/nxos_install_os.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/nxos/nxos_install_os.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @GGabriele @jedelman8 @privateip @rahushen @rcarrillocruz @trishnaguha
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@trishnaguha After updating to the latest `devel` on Friday, I still see the problem for connection local if I use `meta: reset_connection`

```shell
ansible 2.6.0 (rel260/nxos_upgrade 7bb4da05b6) last updated 2018/05/11 09:14:42 (GMT -400)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /root/agents-ci/ansible/lib/ansible
  executable location = /root/agents-ci/ansible/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
agent-lab9-pm:integration:2014> 
```

Here is the toplevel playbook that is controlling the upgrade workflow:

```yaml
---
- include: targets/nxos_install_os/tasks/upgrade/install_with_kick.yaml
  when: ki is defined and combined is undefined

- include: targets/nxos_install_os/tasks/upgrade/install_system.yaml
  when: combined is defined

#- include: targets/nxos_install_os/tasks/upgrade/clear_persistent_sockets.yaml

- meta: reset_connection

- name: "Check installed OS for newly installed version {{ tv }}"
  nxos_command:
    commands: ['show version | json']
    provider: "{{ connection }}"
  register: output

- debug: msg="Version detected {{ output['stdout_lines'][0]['kickstart_ver_str'] }}"
```

```shell
TASK [nxos_install_os : include] **************************************************************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:23
included: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml for n9k.example.com

TASK [nxos_install_os : Install OS image nxos.7.0.3.I7.2.bin] *********************************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:2
<n9k.example.com> connection transport is cli
<n9k.example.com> using connection plugin network_cli (was local)
<n9k.example.com> starting connection from persistent connection plugin
<n9k.example.com> local domain socket does not exist, starting it
<n9k.example.com> control socket path is /root/.ansible/pc/dc96894460
<n9k.example.com> <n9k.example.com> ESTABLISH CONNECTION FOR USER: admin on PORT 22 TO n9k.example.com
<n9k.example.com> <n9k.example.com> ssh connection done, setting terminal
<n9k.example.com> <n9k.example.com> loaded terminal plugin for network_os nxos
<n9k.example.com> <n9k.example.com> loaded cliconf plugin for network_os nxos
<n9k.example.com> <n9k.example.com> firing event: on_open_shell()
<n9k.example.com> <n9k.example.com> ssh connection has completed successfully
<n9k.example.com> connection to remote device started successfully
<n9k.example.com> local domain socket listeners started successfully
<n9k.example.com> 
<n9k.example.com> local domain socket path is /root/.ansible/pc/dc96894460
<n9k.example.com> socket_path: /root/.ansible/pc/dc96894460
Using module file /root/agents-ci/ansible/lib/ansible/modules/network/nxos/nxos_install_os.py
<n9k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
<n9k.example.com> EXEC /bin/sh -c 'echo ~root && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1526305091.81-175724497864052 `" && echo ansible-tmp-1526305091.81-175724497864052="` echo /root/.ansible/tmp/ansible-tmp-1526305091.81-175724497864052 `" ) && sleep 0'
<n9k.example.com> PUT /root/.ansible/tmp/ansible-local-27423CtSNQw/tmp0EfG7h TO /root/.ansible/tmp/ansible-tmp-1526305091.81-175724497864052/nxos_install_os.py
<n9k.example.com> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1526305091.81-175724497864052/ /root/.ansible/tmp/ansible-tmp-1526305091.81-175724497864052/nxos_install_os.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1526305091.81-175724497864052/nxos_install_os.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1526305091.81-175724497864052/ > /dev/null 2>&1 && sleep 0'
changed: [n9k.example.com] => {
    "changed": true, 
    "install_state": [
        "Compatibility check is done:", 
        "Module  bootable          Impact  Install-type  Reason", 
        "------  --------  --------------  ------------  ------", 
        "     1       yes      disruptive         reset  default upgrade is not hitless", 
        "Images will be upgraded according to following table:", 
        "Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required", 
        "------  ----------  ----------------------------------------  --------------------  ------------", 
        "     1        nxos                               7.0(3)I7(4)           7.0(3)I7(2)           yes", 
        "     1        bios     v07.59(08/26/2016):v07.06(03/02/2014)    v07.59(08/26/2016)            no", 
        "--------------------------------------", 
        "Module 1: Refreshing compact flash and upgrading bios/loader/bootrom."
    ], 
    "invocation": {
        "module_args": {
            "auth_pass": null, 
            "authorize": null, 
            "host": "n9k.example.com", 
            "issu": "desired", 
            "kickstart_image_file": null, 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": null, 
            "provider": {
                "auth_pass": null, 
                "authorize": null, 
                "host": "n9k.example.com", 
                "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "port": null, 
                "ssh_keyfile": null, 
                "timeout": 600, 
                "transport": "cli", 
                "use_proxy": true, 
                "use_ssl": null, 
                "username": "admin", 
                "validate_certs": null
            }, 
            "ssh_keyfile": null, 
            "system_image_file": "nxos.7.0.3.I7.2.bin", 
            "timeout": 600, 
            "transport": "cli", 
            "use_proxy": true, 
            "use_ssl": null, 
            "username": "admin", 
            "validate_certs": null
        }
    }
}

TASK [nxos_install_os : debug] ****************************************************************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:10
ok: [n9k.example.com] => {
    "msg": " [u'Compatibility check is done:', u'Module  bootable          Impact  Install-type  Reason', u'------  --------  --------------  ------------  ------', u'     1       yes      disruptive         reset  default upgrade is not hitless', u'Images will be upgraded according to following table:', u'Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required', u'------  ----------  ----------------------------------------  --------------------  ------------', u'     1        nxos                               7.0(3)I7(4)           7.0(3)I7(2)           yes', u'     1        bios     v07.59(08/26/2016):v07.06(03/02/2014)    v07.59(08/26/2016)            no', u'--------------------------------------', u'Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.']"
}

TASK [nxos_install_os : Wait for device to come back up with new image] ***********************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:12
Using module file /root/agents-ci/ansible/lib/ansible/modules/utilities/logic/wait_for.py
<n9k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
<n9k.example.com> EXEC /bin/sh -c 'echo ~root && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1526305299.81-157048354075752 `" && echo ansible-tmp-1526305299.81-157048354075752="` echo /root/.ansible/tmp/ansible-tmp-1526305299.81-157048354075752 `" ) && sleep 0'
<n9k.example.com> PUT /root/.ansible/tmp/ansible-local-27423CtSNQw/tmp6GKHF6 TO /root/.ansible/tmp/ansible-tmp-1526305299.81-157048354075752/wait_for.py
<n9k.example.com> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1526305299.81-157048354075752/ /root/.ansible/tmp/ansible-tmp-1526305299.81-157048354075752/wait_for.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1526305299.81-157048354075752/wait_for.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1526305299.81-157048354075752/ > /dev/null 2>&1 && sleep 0'
ok: [n9k.example.com] => {
    "changed": false, 
    "elapsed": 145, 
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED", 
                "FIN_WAIT1", 
                "FIN_WAIT2", 
                "SYN_RECV", 
                "SYN_SENT", 
                "TIME_WAIT"
            ], 
            "connect_timeout": 5, 
            "delay": 60, 
            "exclude_hosts": null, 
            "host": "n9k.example.com", 
            "msg": null, 
            "path": null, 
            "port": 22, 
            "search_regex": null, 
            "sleep": 1, 
            "state": "started", 
            "timeout": 500
        }
    }, 
    "path": null, 
    "port": 22, 
    "search_regex": null, 
    "state": "started"
}
META: reset connection

TASK [nxos_install_os : Check installed OS for newly installed version {{ tv }}] **************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:30
<n9k.example.com> connection transport is cli
<n9k.example.com> using connection plugin network_cli (was local)
<n9k.example.com> starting connection from persistent connection plugin
<n9k.example.com> found existing local domain socket, using it!
<n9k.example.com> updating play_context for connection
<n9k.example.com> 
<n9k.example.com> local domain socket path is /root/.ansible/pc/dc96894460
<n9k.example.com> socket_path: /root/.ansible/pc/dc96894460
Using module file /root/agents-ci/ansible/lib/ansible/modules/network/nxos/nxos_command.py
<n9k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
<n9k.example.com> EXEC /bin/sh -c 'echo ~root && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1526305445.88-128631414888627 `" && echo ansible-tmp-1526305445.88-128631414888627="` echo /root/.ansible/tmp/ansible-tmp-1526305445.88-128631414888627 `" ) && sleep 0'
<n9k.example.com> PUT /root/.ansible/tmp/ansible-local-27423CtSNQw/tmpY51XA9 TO /root/.ansible/tmp/ansible-tmp-1526305445.88-128631414888627/nxos_command.py
<n9k.example.com> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1526305445.88-128631414888627/ /root/.ansible/tmp/ansible-tmp-1526305445.88-128631414888627/nxos_command.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1526305445.88-128631414888627/nxos_command.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1526305445.88-128631414888627/ > /dev/null 2>&1 && sleep 0'
ok: [n9k.example.com] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "auth_pass": null, 
            "authorize": null, 
            "commands": [
                "show version | json"
            ], 
            "host": "n9k.example.com", 
            "interval": 1, 
            "match": "all", 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": null, 
            "provider": {
                "auth_pass": null, 
                "authorize": null, 
                "host": "n9k.example.com", 
                "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "port": null, 
                "ssh_keyfile": null, 
                "timeout": 600, 
                "transport": "cli", 
                "use_proxy": true, 
                "use_ssl": null, 
                "username": "admin", 
                "validate_certs": null
            }, 
            "retries": 10, 
            "ssh_keyfile": null, 
            "timeout": 600, 
            "transport": "cli", 
            "use_proxy": true, 
            "use_ssl": null, 
            "username": "admin", 
            "validate_certs": null, 
            "wait_for": null
        }
    }, 
    "stdout": [
        "None"
    ], 
    "stdout_lines": [
        [
            "None"
        ]
    ]
}

TASK [nxos_install_os : debug] ****************************************************************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:37
fatal: [n9k.example.com]: FAILED! => {
    "msg": "The task includes an option with an undefined variable. The error was: 'list object' has no attribute 'kickstart_ver_str'\n\nThe error appears to have been in '/root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml': line 37, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n#- debug: msg=\"Version detected {{ output }}\"\n- debug: msg=\"Version detected {{ output['stdout_lines'][0]['kickstart_ver_str'] }}\"\n  ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes.  Always quote template expression brackets when they\nstart a value. For instance:\n\n    with_items:\n      - {{ foo }}\n\nShould be written as:\n\n    with_items:\n      - \"{{ foo }}\"\n"
}
	to retry, use: --limit @/root/agents-ci/ansible/test/integration/nxos.retry

PLAY RECAP ************************************************************************************************************************************************************************************************************
n9k.example.com        : ok=43   changed=1    unreachable=0    failed=1   
```
@trishnaguha I also re-tested using `network_cli` plugin using two testbeds.  The task that executed right after the upgrade worked on one of the testbeds, but not the other.

Here are the logs:

```shell
TASK [nxos_install_os : Install OS image nxos.7.0.3.I7.2.bin] *********************************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:2
<n9kone.example.com> attempting to start connection
<n9kone.example.com> using connection plugin network_cli
<n9ktwo.example.com> attempting to start connection
<n9ktwo.example.com> using connection plugin network_cli
<n9kone.example.com> found existing local domain socket, using it!
<n9kone.example.com> updating play_context for connection
<n9kone.example.com> 
<n9kone.example.com> local domain socket path is /root/.ansible/pc/f76310c0b9
<n9ktwo.example.com> found existing local domain socket, using it!
<n9ktwo.example.com> updating play_context for connection
<n9ktwo.example.com> 
<n9ktwo.example.com> local domain socket path is /root/.ansible/pc/ead08cb852
Using module file /root/agents-ci/ansible/lib/ansible/modules/network/nxos/nxos_install_os.py
Using module file /root/agents-ci/ansible/lib/ansible/modules/network/nxos/nxos_install_os.py
<n9kone.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
<n9kone.example.com> EXEC /bin/sh -c 'echo ~admin && sleep 0'
<n9ktwo.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
<n9ktwo.example.com> EXEC /bin/sh -c 'echo ~admin && sleep 0'
<n9ktwo.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~admin/.ansible/tmp/ansible-tmp-1526306819.77-27003405220624 `" && echo ansible-tmp-1526306819.77-27003405220624="` echo ~admin/.ansible/tmp/ansible-tmp-1526306819.77-27003405220624 `" ) && sleep 0'
<n9kone.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~admin/.ansible/tmp/ansible-tmp-1526306819.77-160388227123893 `" && echo ansible-tmp-1526306819.77-160388227123893="` echo ~admin/.ansible/tmp/ansible-tmp-1526306819.77-160388227123893 `" ) && sleep 0'
<n9kone.example.com> PUT /root/.ansible/tmp/ansible-local-28575RTf6Vw/tmprXg9o2 TO ~admin/.ansible/tmp/ansible-tmp-1526306819.77-160388227123893/nxos_install_os.py
<n9kone.example.com> EXEC /bin/sh -c 'chmod u+x '"'"'~admin/.ansible/tmp/ansible-tmp-1526306819.77-160388227123893/'"'"' '"'"'~admin/.ansible/tmp/ansible-tmp-1526306819.77-160388227123893/nxos_install_os.py'"'"' && sleep 0'
<n9ktwo.example.com> PUT /root/.ansible/tmp/ansible-local-28575RTf6Vw/tmp9y12BI TO ~admin/.ansible/tmp/ansible-tmp-1526306819.77-27003405220624/nxos_install_os.py
<n9ktwo.example.com> EXEC /bin/sh -c 'chmod u+x '"'"'~admin/.ansible/tmp/ansible-tmp-1526306819.77-27003405220624/'"'"' '"'"'~admin/.ansible/tmp/ansible-tmp-1526306819.77-27003405220624/nxos_install_os.py'"'"' && sleep 0'
<n9kone.example.com> EXEC /bin/sh -c '/usr/bin/python '"'"'~admin/.ansible/tmp/ansible-tmp-1526306819.77-160388227123893/nxos_install_os.py'"'"' && sleep 0'
<n9ktwo.example.com> EXEC /bin/sh -c '/usr/bin/python '"'"'~admin/.ansible/tmp/ansible-tmp-1526306819.77-27003405220624/nxos_install_os.py'"'"' && sleep 0'
<n9kone.example.com> EXEC /bin/sh -c 'rm -f -r '"'"'~admin/.ansible/tmp/ansible-tmp-1526306819.77-160388227123893/'"'"' > /dev/null 2>&1 && sleep 0'
changed: [n9kone.example.com] => {
    "changed": true, 
    "install_state": [
        "Compatibility check is done:", 
        "Module  bootable          Impact  Install-type  Reason", 
        "------  --------  --------------  ------------  ------", 
        "     1       yes      disruptive         reset  default upgrade is not hitless", 
        "Images will be upgraded according to following table:", 
        "Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required", 
        "------  ----------  ----------------------------------------  --------------------  ------------", 
        "     1        nxos                               7.0(3)I7(4)           7.0(3)I7(2)           yes", 
        "     1        bios     v07.59(08/26/2016):v07.06(03/02/2014)    v07.59(08/26/2016)            no", 
        "--------------------------------------", 
        "Module 1: Refreshing compact flash and upgrading bios/loader/bootrom."
    ], 
    "invocation": {
        "module_args": {
            "auth_pass": null, 
            "authorize": null, 
            "host": null, 
            "issu": "desired", 
            "kickstart_image_file": null, 
            "password": null, 
            "port": null, 
            "provider": {
                "auth_pass": null, 
                "authorize": null, 
                "host": null, 
                "password": null, 
                "port": null, 
                "ssh_keyfile": null, 
                "timeout": null, 
                "transport": "cli", 
                "use_proxy": true, 
                "use_ssl": null, 
                "username": null, 
                "validate_certs": null
            }, 
            "ssh_keyfile": null, 
            "system_image_file": "nxos.7.0.3.I7.2.bin", 
            "timeout": null, 
            "transport": "cli", 
            "use_proxy": true, 
            "use_ssl": null, 
            "username": null, 
            "validate_certs": null
        }
    }
}
<n9ktwo.example.com> EXEC /bin/sh -c 'rm -f -r '"'"'~admin/.ansible/tmp/ansible-tmp-1526306819.77-27003405220624/'"'"' > /dev/null 2>&1 && sleep 0'
changed: [n9ktwo.example.com] => {
    "changed": true, 
    "install_state": [
        "Compatibility check is done:", 
        "Module  bootable          Impact  Install-type  Reason", 
        "------  --------  --------------  ------------  ------", 
        "     1       yes      disruptive         reset  default upgrade is not hitless", 
        "    22       yes      disruptive         reset  default upgrade is not hitless", 
        "    24       yes      disruptive         reset  default upgrade is not hitless", 
        "    26       yes      disruptive         reset  default upgrade is not hitless", 
        "    28       yes      disruptive         reset  default upgrade is not hitless", 
        "    29       yes      disruptive         reset  default upgrade is not hitless", 
        "    30       yes      disruptive         reset  default upgrade is not hitless", 
        "Images will be upgraded according to following table:", 
        "Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required", 
        "------  ----------  ----------------------------------------  --------------------  ------------", 
        "     1       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes", 
        "     1        bios                       v01.48(00:v01.40(00             v01.48(00            no", 
        "    22       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes", 
        "    22        bios                       v01.48(00:v01.42(00             v01.48(00            no", 
        "    24       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes", 
        "    24        bios                       v01.48(00:v01.42(00             v01.48(00            no", 
        "    26       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes", 
        "    26        bios                       v01.48(00:v01.42(00             v01.48(00            no", 
        "    28        nxos                               7.0(3)I7(4)           7.0(3)I7(2)           yes", 
        "    28        bios     v08.32(10/18/2016):v08.06(09/10/2014)    v08.32(10/18/2016)            no", 
        "    29       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes", 
        "    29        bios                       v01.48(00:v01.42(00             v01.48(00            no", 
        "    30       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes", 
        "    30        bios                       v01.48(00:v01.42(00             v01.48(00            no", 
        "--------------------------------------", 
        "Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 22: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 24: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 26: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 28: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 29: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 30: Refreshing compact flash and upgrading bios/loader/bootrom."
    ], 
    "invocation": {
        "module_args": {
            "auth_pass": null, 
            "authorize": null, 
            "host": null, 
            "issu": "desired", 
            "kickstart_image_file": null, 
            "password": null, 
            "port": null, 
            "provider": {
                "auth_pass": null, 
                "authorize": null, 
                "host": null, 
                "password": null, 
                "port": null, 
                "ssh_keyfile": null, 
                "timeout": null, 
                "transport": "cli", 
                "use_proxy": true, 
                "use_ssl": null, 
                "username": null, 
                "validate_certs": null
            }, 
            "ssh_keyfile": null, 
            "system_image_file": "nxos.7.0.3.I7.2.bin", 
            "timeout": null, 
            "transport": "cli", 
            "use_proxy": true, 
            "use_ssl": null, 
            "username": null, 
            "validate_certs": null
        }
    }
}

TASK [nxos_install_os : debug] ****************************************************************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:10
<n9kone.example.com> attempting to start connection
<n9kone.example.com> using connection plugin network_cli
<n9ktwo.example.com> attempting to start connection
<n9ktwo.example.com> using connection plugin network_cli
<n9kone.example.com> found existing local domain socket, using it!
<n9kone.example.com> updating play_context for connection
<n9kone.example.com> 
<n9kone.example.com> local domain socket path is /root/.ansible/pc/f76310c0b9
ok: [n9kone.example.com] => {
    "msg": " [u'Compatibility check is done:', u'Module  bootable          Impact  Install-type  Reason', u'------  --------  --------------  ------------  ------', u'     1       yes      disruptive         reset  default upgrade is not hitless', u'Images will be upgraded according to following table:', u'Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required', u'------  ----------  ----------------------------------------  --------------------  ------------', u'     1        nxos                               7.0(3)I7(4)           7.0(3)I7(2)           yes', u'     1        bios     v07.59(08/26/2016):v07.06(03/02/2014)    v07.59(08/26/2016)            no', u'--------------------------------------', u'Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.']"
}
<n9ktwo.example.com> found existing local domain socket, using it!
<n9ktwo.example.com> updating play_context for connection
<n9ktwo.example.com> 
<n9ktwo.example.com> local domain socket path is /root/.ansible/pc/ead08cb852
ok: [n9ktwo.example.com] => {
    "msg": " [u'Compatibility check is done:', u'Module  bootable          Impact  Install-type  Reason', u'------  --------  --------------  ------------  ------', u'     1       yes      disruptive         reset  default upgrade is not hitless', u'    22       yes      disruptive         reset  default upgrade is not hitless', u'    24       yes      disruptive         reset  default upgrade is not hitless', u'    26       yes      disruptive         reset  default upgrade is not hitless', u'    28       yes      disruptive         reset  default upgrade is not hitless', u'    29       yes      disruptive         reset  default upgrade is not hitless', u'    30       yes      disruptive         reset  default upgrade is not hitless', u'Images will be upgraded according to following table:', u'Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required', u'------  ----------  ----------------------------------------  --------------------  ------------', u'     1       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes', u'     1        bios                       v01.48(00:v01.40(00             v01.48(00            no', u'    22       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes', u'    22        bios                       v01.48(00:v01.42(00             v01.48(00            no', u'    24       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes', u'    24        bios                       v01.48(00:v01.42(00             v01.48(00            no', u'    26       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes', u'    26        bios                       v01.48(00:v01.42(00             v01.48(00            no', u'    28        nxos                               7.0(3)I7(4)           7.0(3)I7(2)           yes', u'    28        bios     v08.32(10/18/2016):v08.06(09/10/2014)    v08.32(10/18/2016)            no', u'    29       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes', u'    29        bios                       v01.48(00:v01.42(00             v01.48(00            no', u'    30       lcn9k                               7.0(3)I7(4)           7.0(3)I7(2)           yes', u'    30        bios                       v01.48(00:v01.42(00             v01.48(00            no', u'--------------------------------------', u'Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 22: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 24: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 26: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 28: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 29: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 30: Refreshing compact flash and upgrading bios/loader/bootrom.']"
}

TASK [nxos_install_os : Wait for device to come back up with new image] ***********************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:12
<n9kone.example.com> attempting to start connection
<n9kone.example.com> using connection plugin network_cli
<n9ktwo.example.com> attempting to start connection
<n9ktwo.example.com> using connection plugin network_cli
<n9kone.example.com> found existing local domain socket, using it!
<n9kone.example.com> updating play_context for connection
<n9kone.example.com> 
<n9kone.example.com> local domain socket path is /root/.ansible/pc/f76310c0b9
<n9ktwo.example.com> found existing local domain socket, using it!
<n9ktwo.example.com> updating play_context for connection
<n9ktwo.example.com> 
<n9ktwo.example.com> local domain socket path is /root/.ansible/pc/ead08cb852
Using module file /root/agents-ci/ansible/lib/ansible/modules/utilities/logic/wait_for.py
<n9kone.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
<n9kone.example.com> EXEC /bin/sh -c 'echo ~admin && sleep 0'
Using module file /root/agents-ci/ansible/lib/ansible/modules/utilities/logic/wait_for.py
<n9ktwo.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
<n9ktwo.example.com> EXEC /bin/sh -c 'echo ~admin && sleep 0'
<n9kone.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~admin/.ansible/tmp/ansible-tmp-1526307104.97-89297678787937 `" && echo ansible-tmp-1526307104.97-89297678787937="` echo ~admin/.ansible/tmp/ansible-tmp-1526307104.97-89297678787937 `" ) && sleep 0'
<n9ktwo.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~admin/.ansible/tmp/ansible-tmp-1526307104.97-165276444711933 `" && echo ansible-tmp-1526307104.97-165276444711933="` echo ~admin/.ansible/tmp/ansible-tmp-1526307104.97-165276444711933 `" ) && sleep 0'
<n9kone.example.com> PUT /root/.ansible/tmp/ansible-local-28575RTf6Vw/tmp2kAncp TO ~admin/.ansible/tmp/ansible-tmp-1526307104.97-89297678787937/wait_for.py
<n9kone.example.com> EXEC /bin/sh -c 'chmod u+x '"'"'~admin/.ansible/tmp/ansible-tmp-1526307104.97-89297678787937/'"'"' '"'"'~admin/.ansible/tmp/ansible-tmp-1526307104.97-89297678787937/wait_for.py'"'"' && sleep 0'
<n9ktwo.example.com> PUT /root/.ansible/tmp/ansible-local-28575RTf6Vw/tmp2lvLUu TO ~admin/.ansible/tmp/ansible-tmp-1526307104.97-165276444711933/wait_for.py
<n9ktwo.example.com> EXEC /bin/sh -c 'chmod u+x '"'"'~admin/.ansible/tmp/ansible-tmp-1526307104.97-165276444711933/'"'"' '"'"'~admin/.ansible/tmp/ansible-tmp-1526307104.97-165276444711933/wait_for.py'"'"' && sleep 0'
<n9kone.example.com> EXEC /bin/sh -c '/usr/bin/python '"'"'~admin/.ansible/tmp/ansible-tmp-1526307104.97-89297678787937/wait_for.py'"'"' && sleep 0'
<n9ktwo.example.com> EXEC /bin/sh -c '/usr/bin/python '"'"'~admin/.ansible/tmp/ansible-tmp-1526307104.97-165276444711933/wait_for.py'"'"' && sleep 0'
<n9kone.example.com> EXEC /bin/sh -c 'rm -f -r '"'"'~admin/.ansible/tmp/ansible-tmp-1526307104.97-89297678787937/'"'"' > /dev/null 2>&1 && sleep 0'
ok: [n9kone.example.com] => {
    "changed": false, 
    "elapsed": 72, 
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED", 
                "FIN_WAIT1", 
                "FIN_WAIT2", 
                "SYN_RECV", 
                "SYN_SENT", 
                "TIME_WAIT"
            ], 
            "connect_timeout": 5, 
            "delay": 60, 
            "exclude_hosts": null, 
            "host": "n9kone.example.com", 
            "msg": null, 
            "path": null, 
            "port": 22, 
            "search_regex": null, 
            "sleep": 1, 
            "state": "started", 
            "timeout": 500
        }
    }, 
    "path": null, 
    "port": 22, 
    "search_regex": null, 
    "state": "started"
}
<n9ktwo.example.com> EXEC /bin/sh -c 'rm -f -r '"'"'~admin/.ansible/tmp/ansible-tmp-1526307104.97-165276444711933/'"'"' > /dev/null 2>&1 && sleep 0'
ok: [n9ktwo.example.com] => {
    "changed": false, 
    "elapsed": 189, 
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED", 
                "FIN_WAIT1", 
                "FIN_WAIT2", 
                "SYN_RECV", 
                "SYN_SENT", 
                "TIME_WAIT"
            ], 
            "connect_timeout": 5, 
            "delay": 60, 
            "exclude_hosts": null, 
            "host": "n9ktwo.example.com", 
            "msg": null, 
            "path": null, 
            "port": 22, 
            "search_regex": null, 
            "sleep": 1, 
            "state": "started", 
            "timeout": 500
        }
    }, 
    "path": null, 
    "port": 22, 
    "search_regex": null, 
    "state": "started"
}
META: reset connection

TASK [nxos_install_os : Check installed OS for newly installed version {{ tv }}] **************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:30
<n9kone.example.com> attempting to start connection
<n9kone.example.com> using connection plugin network_cli
<n9ktwo.example.com> attempting to start connection
<n9ktwo.example.com> using connection plugin network_cli
<n9ktwo.example.com> found existing local domain socket, using it!
<n9ktwo.example.com> updating play_context for connection
<n9ktwo.example.com> 
<n9ktwo.example.com> local domain socket path is /root/.ansible/pc/ead08cb852
Using module file /root/agents-ci/ansible/lib/ansible/modules/network/nxos/nxos_command.py
<n9ktwo.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
<n9ktwo.example.com> EXEC /bin/sh -c 'echo ~admin && sleep 0'
<n9ktwo.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~admin/.ansible/tmp/ansible-tmp-1526307295.35-256304375313199 `" && echo ansible-tmp-1526307295.35-256304375313199="` echo ~admin/.ansible/tmp/ansible-tmp-1526307295.35-256304375313199 `" ) && sleep 0'
<n9ktwo.example.com> PUT /root/.ansible/tmp/ansible-local-28575RTf6Vw/tmpOuoAku TO ~admin/.ansible/tmp/ansible-tmp-1526307295.35-256304375313199/nxos_command.py
<n9ktwo.example.com> EXEC /bin/sh -c 'chmod u+x '"'"'~admin/.ansible/tmp/ansible-tmp-1526307295.35-256304375313199/'"'"' '"'"'~admin/.ansible/tmp/ansible-tmp-1526307295.35-256304375313199/nxos_command.py'"'"' && sleep 0'
<n9ktwo.example.com> EXEC /bin/sh -c '/usr/bin/python '"'"'~admin/.ansible/tmp/ansible-tmp-1526307295.35-256304375313199/nxos_command.py'"'"' && sleep 0'
<n9ktwo.example.com> EXEC /bin/sh -c 'rm -f -r '"'"'~admin/.ansible/tmp/ansible-tmp-1526307295.35-256304375313199/'"'"' > /dev/null 2>&1 && sleep 0'
ok: [n9ktwo.example.com] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "auth_pass": null, 
            "authorize": null, 
            "commands": [
                "show version | json"
            ], 
            "host": null, 
            "interval": 1, 
            "match": "all", 
            "password": null, 
            "port": null, 
            "provider": {
                "auth_pass": null, 
                "authorize": null, 
                "host": null, 
                "password": null, 
                "port": null, 
                "ssh_keyfile": null, 
                "timeout": null, 
                "transport": "cli", 
                "use_proxy": true, 
                "use_ssl": null, 
                "username": null, 
                "validate_certs": null
            }, 
            "retries": 10, 
            "ssh_keyfile": null, 
            "timeout": null, 
            "transport": "cli", 
            "use_proxy": true, 
            "use_ssl": null, 
            "username": null, 
            "validate_certs": null, 
            "wait_for": null
        }
    }, 
    "stdout": [
        "None"
    ], 
    "stdout_lines": [
        [
            "None"
        ]
    ]
}
<n9kone.example.com> local domain socket does not exist, starting it
<n9kone.example.com> control socket path is /root/.ansible/pc/f76310c0b9
<n9kone.example.com> <n9kone.example.com> ESTABLISH CONNECTION FOR USER: admin on PORT 22 TO n9kone.example.com
<n9kone.example.com> <n9kone.example.com> ssh connection done, setting terminal
<n9kone.example.com> <n9kone.example.com> loaded terminal plugin for network_os nxos
<n9kone.example.com> <n9kone.example.com> loaded cliconf plugin for network_os nxos
<n9kone.example.com> <n9kone.example.com> firing event: on_open_shell()
<n9kone.example.com> <n9kone.example.com> ssh connection has completed successfully
<n9kone.example.com> connection to remote device started successfully
<n9kone.example.com> local domain socket listeners started successfully
<n9kone.example.com> 
<n9kone.example.com> local domain socket path is /root/.ansible/pc/f76310c0b9
Using module file /root/agents-ci/ansible/lib/ansible/modules/network/nxos/nxos_command.py
<n9kone.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
<n9kone.example.com> EXEC /bin/sh -c 'echo ~admin && sleep 0'
<n9kone.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~admin/.ansible/tmp/ansible-tmp-1526307297.1-201537098349997 `" && echo ansible-tmp-1526307297.1-201537098349997="` echo ~admin/.ansible/tmp/ansible-tmp-1526307297.1-201537098349997 `" ) && sleep 0'
<n9kone.example.com> PUT /root/.ansible/tmp/ansible-local-28575RTf6Vw/tmpJT4HJZ TO ~admin/.ansible/tmp/ansible-tmp-1526307297.1-201537098349997/nxos_command.py
<n9kone.example.com> EXEC /bin/sh -c 'chmod u+x '"'"'~admin/.ansible/tmp/ansible-tmp-1526307297.1-201537098349997/'"'"' '"'"'~admin/.ansible/tmp/ansible-tmp-1526307297.1-201537098349997/nxos_command.py'"'"' && sleep 0'
<n9kone.example.com> EXEC /bin/sh -c '/usr/bin/python '"'"'~admin/.ansible/tmp/ansible-tmp-1526307297.1-201537098349997/nxos_command.py'"'"' && sleep 0'
<n9kone.example.com> EXEC /bin/sh -c 'rm -f -r '"'"'~admin/.ansible/tmp/ansible-tmp-1526307297.1-201537098349997/'"'"' > /dev/null 2>&1 && sleep 0'
ok: [n9kone.example.com] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "auth_pass": null, 
            "authorize": null, 
            "commands": [
                "show version | json"
            ], 
            "host": null, 
            "interval": 1, 
            "match": "all", 
            "password": null, 
            "port": null, 
            "provider": {
                "auth_pass": null, 
                "authorize": null, 
                "host": null, 
                "password": null, 
                "port": null, 
                "ssh_keyfile": null, 
                "timeout": null, 
                "transport": "cli", 
                "use_proxy": true, 
                "use_ssl": null, 
                "username": null, 
                "validate_certs": null
            }, 
            "retries": 10, 
            "ssh_keyfile": null, 
            "timeout": null, 
            "transport": "cli", 
            "use_proxy": true, 
            "use_ssl": null, 
            "username": null, 
            "validate_certs": null, 
            "wait_for": null
        }
    }, 
    "stdout": [
        {
            "TABLE_package_list": {
                "ROW_package_list": {
                    "package_id": null
                }
            }, 
            "bios_cmpl_time": "08/26/2016", 
            "bios_ver_str": "07.59", 
            "bootflash_size": "21693714", 
            "chassis_id": "Nexus9000 C9396PX Chassis", 
            "cpu_name": "Intel(R) Core(TM) i3- CPU @ 2.50GHz", 
            "header_str": "Cisco Nexus Operating System (NX-OS) Software\nTAC support: http://www.cisco.com/tac\nCopyright (C) 2002-2017, Cisco and/or its affiliates.\nAll rights reserved.\nThe copyrights to certain works contained in this software are\nowned by other third parties and used and distributed under their own\nlicenses, such as open source.  This software is provided \"as is,\" and unless\notherwise stated, there is no warranty, express or implied, including but not\nlimited to warranties of merchantability and fitness for a particular purpose.\nCertain components of this software are licensed under\nthe GNU General Public License (GPL) version 2.0 or \nGNU General Public License (GPL) version 3.0  or the GNU\nLesser General Public License (LGPL) Version 2.1 or \nLesser General Public License (LGPL) Version 2.0. \nA copy of each such license is available at\nhttp://www.opensource.org/licenses/gpl-2.0.php and\nhttp://opensource.org/licenses/gpl-3.0.html and\nhttp://www.opensource.org/licenses/lgpl-2.1.php and\nhttp://www.gnu.org/licenses/old-licenses/library.txt.", 
            "host_name": "dt-n9k5-1", 
            "kern_uptm_days": "0", 
            "kern_uptm_hrs": "0", 
            "kern_uptm_mins": "3", 
            "kern_uptm_secs": "13", 
            "kick_cmpl_time": "11/22/2017 13:00:00", 
            "kick_file_name": "bootflash:///nxos.7.0.3.I7.2.bin", 
            "kick_tmstmp": "11/22/2017 21:55:29", 
            "kickstart_ver_str": "7.0(3)I7(2)", 
            "manufacturer": "Cisco Systems, Inc.", 
            "mem_type": "kB", 
            "memory": "16400980", 
            "proc_board_id": "SAL1821T9EF", 
            "rr_ctime": "Mon May 14 14:10:38 2018", 
            "rr_reason": "Reset due to upgrade", 
            "rr_service": null, 
            "rr_sys_ver": "7.0(3)I7(4)", 
            "rr_usecs": "621100"
        }
    ], 
    "stdout_lines": [
        {
            "TABLE_package_list": {
                "ROW_package_list": {
                    "package_id": null
                }
            }, 
            "bios_cmpl_time": "08/26/2016", 
            "bios_ver_str": "07.59", 
            "bootflash_size": "21693714", 
            "chassis_id": "Nexus9000 C9396PX Chassis", 
            "cpu_name": "Intel(R) Core(TM) i3- CPU @ 2.50GHz", 
            "header_str": "Cisco Nexus Operating System (NX-OS) Software\nTAC support: http://www.cisco.com/tac\nCopyright (C) 2002-2017, Cisco and/or its affiliates.\nAll rights reserved.\nThe copyrights to certain works contained in this software are\nowned by other third parties and used and distributed under their own\nlicenses, such as open source.  This software is provided \"as is,\" and unless\notherwise stated, there is no warranty, express or implied, including but not\nlimited to warranties of merchantability and fitness for a particular purpose.\nCertain components of this software are licensed under\nthe GNU General Public License (GPL) version 2.0 or \nGNU General Public License (GPL) version 3.0  or the GNU\nLesser General Public License (LGPL) Version 2.1 or \nLesser General Public License (LGPL) Version 2.0. \nA copy of each such license is available at\nhttp://www.opensource.org/licenses/gpl-2.0.php and\nhttp://opensource.org/licenses/gpl-3.0.html and\nhttp://www.opensource.org/licenses/lgpl-2.1.php and\nhttp://www.gnu.org/licenses/old-licenses/library.txt.", 
            "host_name": "dt-n9k5-1", 
            "kern_uptm_days": "0", 
            "kern_uptm_hrs": "0", 
            "kern_uptm_mins": "3", 
            "kern_uptm_secs": "13", 
            "kick_cmpl_time": "11/22/2017 13:00:00", 
            "kick_file_name": "bootflash:///nxos.7.0.3.I7.2.bin", 
            "kick_tmstmp": "11/22/2017 21:55:29", 
            "kickstart_ver_str": "7.0(3)I7(2)", 
            "manufacturer": "Cisco Systems, Inc.", 
            "mem_type": "kB", 
            "memory": "16400980", 
            "proc_board_id": "SAL1821T9EF", 
            "rr_ctime": "Mon May 14 14:10:38 2018", 
            "rr_reason": "Reset due to upgrade", 
            "rr_service": null, 
            "rr_sys_ver": "7.0(3)I7(4)", 
            "rr_usecs": "621100"
        }
    ]
}

TASK [nxos_install_os : debug] ****************************************************************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:37
fatal: [n9ktwo.example.com]: FAILED! => {
    "msg": "The task includes an option with an undefined variable. The error was: 'list object' has no attribute 'kickstart_ver_str'\n\nThe error appears to have been in '/root/agents-ci/ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml': line 37, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n#- debug: msg=\"Version detected {{ output }}\"\n- debug: msg=\"Version detected {{ output['stdout_lines'][0]['kickstart_ver_str'] }}\"\n  ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes.  Always quote template expression brackets when they\nstart a value. For instance:\n\n    with_items:\n      - {{ foo }}\n\nShould be written as:\n\n    with_items:\n      - \"{{ foo }}\"\n"
}
<n9kone.example.com> attempting to start connection
<n9kone.example.com> using connection plugin network_cli
<n9kone.example.com> found existing local domain socket, using it!
<n9kone.example.com> updating play_context for connection
<n9kone.example.com> 
<n9kone.example.com> local domain socket path is /root/.ansible/pc/f76310c0b9
ok: [n9kone.example.com] => {
    "msg": "Version detected 7.0(3)I7(2)"
}

TASK [nxos_install_os : debug] ****************************************************************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_install_os/tests/common/upgrade.yaml:66
<n9kone.example.com> attempting to start connection
<n9kone.example.com> using connection plugin network_cli
<n9kone.example.com> found existing local domain socket, using it!
<n9kone.example.com> updating play_context for connection
<n9kone.example.com> 
<n9kone.example.com> local domain socket path is /root/.ansible/pc/f76310c0b9
ok: [n9kone.example.com] => {
    "msg": "END connection=network_cli nxos_os_install upgrade"
}
META: ran handlers
META: ran handlers
	to retry, use: --limit @/root/agents-ci/ansible/test/integration/nxos.retry

PLAY RECAP ************************************************************************************************************************************************************************************************************
n9kone.example.com        : ok=44   changed=1    unreachable=0    failed=0   
n9ktwo.example.com          : ok=42   changed=1    unreachable=0    failed=1   

agent-lab9-pm:integration:2024> 
```

As you can see, the meta_reset connection appears to have worked on n9kone, but not n9ktwo.

Resolving patch:
diff --git a/lib/ansible/plugins/strategy/linear.py b/lib/ansible/plugins/strategy/linear.py
--- a/lib/ansible/plugins/strategy/linear.py
+++ b/lib/ansible/plugins/strategy/linear.py
@@ -263,7 +263,7 @@ def run(self, iterator, play_context):
                         # for the linear strategy, we run meta tasks just once and for
                         # all hosts currently being iterated over rather than one host
                         results.extend(self._execute_meta(task, play_context, iterator, host))
-                        if task.args.get('_raw_params', None) != 'noop':
+                        if task.args.get('_raw_params', None) not in ('noop', 'reset_connection'):
                             run_once = True
                     else:
                         # handle step if needed, skip meta actions as they are used internally


Test patch: