Repository: ansible/ansible

Issue:
NXOS: ansible.module_utils.connection.ConnectionError: No JSON object could be decoded
##### SUMMARY
We have started seeing a high failure rate in our nightly Nexus integration tests using httpapi.

Many of the tests are failing with `ansible.module_utils.connection.ConnectionError: No JSON object could be decoded`.

NOTE: Test passes for connection `local` and `network_cli` and `nxapi` using provider property.

**Must fix for 2.6**

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
nxos_*

##### ANSIBLE VERSION
```
ansible 2.6.0a1 (detached HEAD c98043a4c0) last updated 2018/05/30 12:15:24 (GMT -400)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/usr/share/ansible']
  ansible python module location = /root/agents-ci/ansible/lib/ansible
  executable location = /root/agents-ci/ansible/bin/ansible
  python version = 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4]
```

```
ansible 2.7.0dev0 (devel c3d3b6cedc) last updated 2018/05/30 11:46:25 (GMT -400)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/usr/share/ansible']
  ansible python module location = /root/agents-ci/ansible/lib/ansible
  executable location = /root/agents-ci/ansible/bin/ansible
  python version = 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4]
```

##### CONFIGURATION
```
[DEPRECATION WARNING]: [defaults]hostfile option, The key is misleading as it can also be a list of hosts, a directory or a list of paths , use [defaults] inventory=/path/to/file|dir instead. This feature will be removed in version 2.8. 
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: DEFAULT_SUDO_EXE option, In favor of Ansible Become, which is a generic framework. See become_exe. , use become instead. This feature will be removed in version 2.8. Deprecation warnings can be disabled by setting 
deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: DEFAULT_SUDO_USER option, In favor of Ansible Become, which is a generic framework. See become_user. , use become instead. This feature will be removed in version 2.8. Deprecation warnings can be disabled by setting
 deprecation_warnings=False in ansible.cfg.
DEFAULT_ACTION_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible_plugins/action_plugins']
DEFAULT_CALLBACK_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible_plugins/callback_plugins']
DEFAULT_CONNECTION_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible_plugins/connection_plugins']
DEFAULT_FILTER_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible_plugins/filter_plugins']
DEFAULT_FORKS(/etc/ansible/ansible.cfg) = 5
DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = [u'/etc/ansible/hosts']
DEFAULT_LOOKUP_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible_plugins/lookup_plugins']
DEFAULT_MANAGED_STR(/etc/ansible/ansible.cfg) = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
DEFAULT_MODULE_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible']
DEFAULT_POLL_INTERVAL(/etc/ansible/ansible.cfg) = 15
DEFAULT_REMOTE_PORT(/etc/ansible/ansible.cfg) = 22
DEFAULT_ROLES_PATH(env: ANSIBLE_ROLES_PATH) = [u'/root/agents-ci/ansible/test/integration/targets']
DEFAULT_SUDO_EXE(/etc/ansible/ansible.cfg) = sudo
DEFAULT_SUDO_USER(/etc/ansible/ansible.cfg) = root
DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 10
DEFAULT_TRANSPORT(/etc/ansible/ansible.cfg) = smart
DEFAULT_VARS_PLUGIN_PATH(/etc/ansible/ansible.cfg) = [u'/usr/share/ansible_plugins/vars_plugins']
PARAMIKO_HOST_KEY_AUTO_ADD(env: ANSIBLE_PARAMIKO_HOST_KEY_AUTO_ADD) = True
PERSISTENT_COMMAND_TIMEOUT(env: ANSIBLE_PERSISTENT_COMMAND_TIMEOUT) = 600
PERSISTENT_CONNECT_TIMEOUT(env: ANSIBLE_PERSISTENT_CONNECT_TIMEOUT) = 600
```

##### OS / ENVIRONMENT
Ubuntu 14.04.3 LTS

##### STEPS TO REPRODUCE
Run the nxos_acl integration test and use `httpapi` as the transport.

##### ACTUAL RESULTS
```
2018-05-30T11:47:48: %LIB-INFO: TASK [nxos_acl : Configure ACE10] *************************************************************************************************************************************************************************************************************
2018-05-30T11:47:48: %LIB-INFO: task path: /root/agents-ci/ansible/test/integration/targets/nxos_acl/tests/common/sanity.yaml:16
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> loaded API plugin for network_os nxos
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> attempting to start connection
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> using connection plugin httpapi
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> found existing local domain socket, using it!
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> updating play_context for connection
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> 
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> local domain socket path is /root/.ansible/pc/fc3f314dd4
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-local-28139T6rZBd/ansible-tmp-1527695267.33-192093360815356 `" && echo ansible-tmp-1527695267.33-192093360815356="` echo /root/.ansible/tmp/ansible-local-28139T6rZBd/ansible-tmp-1527695267.33-192093360815356 `" ) && sleep 0'
2018-05-30T11:47:48: %LIB-INFO: Using module file /root/agents-ci/ansible/lib/ansible/modules/network/nxos/nxos_acl.py
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> PUT /root/.ansible/tmp/ansible-local-28139T6rZBd/tmppr47X3 TO /root/.ansible/tmp/ansible-local-28139T6rZBd/ansible-tmp-1527695267.33-192093360815356/nxos_acl.py
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-local-28139T6rZBd/ansible-tmp-1527695267.33-192093360815356/ /root/.ansible/tmp/ansible-local-28139T6rZBd/ansible-tmp-1527695267.33-192093360815356/nxos_acl.py && sleep 0'
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-local-28139T6rZBd/ansible-tmp-1527695267.33-192093360815356/nxos_acl.py && sleep 0'
2018-05-30T11:47:48: %LIB-INFO: <n9kv.example.com> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-local-28139T6rZBd/ansible-tmp-1527695267.33-192093360815356/ > /dev/null 2>&1 && sleep 0'
2018-05-30T11:47:48: %LIB-INFO: The full traceback is:
2018-05-30T11:47:48: %LIB-INFO: Traceback (most recent call last):
2018-05-30T11:47:48: %LIB-INFO:   File "/tmp/ansible_KEsUDL/ansible_module_nxos_acl.py", line 533, in <module>
2018-05-30T11:47:48: %LIB-INFO:     main()
2018-05-30T11:47:48: %LIB-INFO:   File "/tmp/ansible_KEsUDL/ansible_module_nxos_acl.py", line 452, in main
2018-05-30T11:47:48: %LIB-INFO:     existing_core, acl = get_acl(module, name, seq)
2018-05-30T11:47:48: %LIB-INFO:   File "/tmp/ansible_KEsUDL/ansible_module_nxos_acl.py", line 191, in get_acl
2018-05-30T11:47:48: %LIB-INFO:     body = execute_show_command(command, module)[0]
2018-05-30T11:47:48: %LIB-INFO:   File "/tmp/ansible_KEsUDL/ansible_module_nxos_acl.py", line 181, in execute_show_command
2018-05-30T11:47:48: %LIB-INFO:     body = run_commands(module, cmds)
2018-05-30T11:47:48: %LIB-INFO:   File "/tmp/ansible_KEsUDL/ansible_modlib.zip/ansible/module_utils/network/nxos/nxos.py", line 431, in run_commands
2018-05-30T11:47:48: %LIB-INFO:   File "/tmp/ansible_KEsUDL/ansible_modlib.zip/ansible/module_utils/network/nxos/nxos.py", line 150, in run_commands
2018-05-30T11:47:48: %LIB-INFO:   File "/tmp/ansible_KEsUDL/ansible_modlib.zip/ansible/module_utils/connection.py", line 149, in __rpc__
2018-05-30T11:47:48: %LIB-INFO: ansible.module_utils.connection.ConnectionError: No JSON object could be decoded
2018-05-30T11:47:48: %LIB-INFO: fatal: [n9kv.example.com]: FAILED! => {
2018-05-30T11:47:48: %LIB-INFO:     "changed": false, 
2018-05-30T11:47:48: %LIB-INFO:     "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_KEsUDL/ansible_module_nxos_acl.py\", line 533, in <module>\n    main()\n  File \"/tmp/ansible_KEsUDL/ansible_module_nxos_acl.py\", line 452, in main\n    existing_core, acl = get_acl(module, name, seq)\n  File \"/tmp/ansible_KEsUDL/ansible_module_nxos_acl.py\", line 191, in get_acl\n    body = execute_show_command(command, module)[0]\n  File \"/tmp/ansible_KEsUDL/ansible_module_nxos_acl.py\", line 181, in execute_show_command\n    body = run_commands(module, cmds)\n  File \"/tmp/ansible_KEsUDL/ansible_modlib.zip/ansible/module_utils/network/nxos/nxos.py\", line 431, in run_commands\n  File \"/tmp/ansible_KEsUDL/ansible_modlib.zip/ansible/module_utils/network/nxos/nxos.py\", line 150, in run_commands\n  File \"/tmp/ansible_KEsUDL/ansible_modlib.zip/ansible/module_utils/connection.py\", line 149, in __rpc__\nansible.module_utils.connection.ConnectionError: No JSON object could be decoded\n", 
2018-05-30T11:47:48: %LIB-INFO:     "module_stdout": "", 
2018-05-30T11:47:48: %LIB-INFO:     "msg": "MODULE FAILURE", 
2018-05-30T11:47:48: %LIB-INFO:     "rc": 1
2018-05-30T11:47:48: %LIB-INFO: }
2018-05-30T11:47:48: %LIB-INFO: 	to retry, use: --limit @/root/agents-ci/ansible/test/integration/nxos.retry
```



Discussion:
Files identified in the description:
None

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Here is an example of it passing on the same ansible server and devel code base using `nxapi` + provider property.

```
TASK [nxos_acl : Configure ACE10] *************************************************************************************************************************************************************************************************************
task path: /root/agents-ci/ansible/test/integration/targets/nxos_acl/tests/common/sanity.yaml:17
<n9kv.example.com> connection transport is nxapi
<n9kv.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
<n9kv.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1527697505.84-28376204456225 `" && echo ansible-tmp-1527697505.84-28376204456225="` echo $HOME/.ansible/tmp/ansible-tmp-1527697505.84-28376204456225 `" ) && sleep 0'
Using module file /root/agents-ci/ansible/lib/ansible/modules/network/nxos/nxos_acl.py
<n9kv.example.com> PUT /root/.ansible/tmp/ansible-local-860973rxW3/tmpDzaFXg TO /root/.ansible/tmp/ansible-tmp-1527697505.84-28376204456225/nxos_acl.py
<n9kv.example.com> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1527697505.84-28376204456225/ /root/.ansible/tmp/ansible-tmp-1527697505.84-28376204456225/nxos_acl.py && sleep 0'
<n9kv.example.com> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1527697505.84-28376204456225/nxos_acl.py && sleep 0'
<n9kv.example.com> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1527697505.84-28376204456225/ > /dev/null 2>&1 && sleep 0'
changed: [n9kv.example.com] => {
    "changed": true, 
    "commands": [
        "ip access-list TEST_ACL", 
        "10 permit tcp 1.1.1.1/24 range 1900 1910 any neq 1899 established psh urg log ack syn dscp af43 time-range ans-range rst fin"
    ], 
    "invocation": {
        "module_args": {
            "ack": "enable", 
            "action": "permit", 
            "auth_pass": null, 
            "authorize": null, 
            "dest": "any", 
            "dest_port1": "1899", 
            "dest_port2": null, 
            "dest_port_op": "neq", 
            "dscp": "af43", 
            "established": "enable", 
            "fin": "enable", 
            "fragments": null, 
            "host": "n9kv.example.com", 
            "log": "enable", 
            "name": "TEST_ACL", 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": 80, 
            "precedence": null, 
            "proto": "tcp", 
            "provider": {
                "auth_pass": null, 
                "authorize": null, 
                "host": "n9kv.example.com", 
                "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "port": 80, 
                "ssh_keyfile": null, 
                "timeout": 600, 
                "transport": "nxapi", 
                "use_proxy": true, 
                "use_ssl": false, 
                "username": "admin", 
                "validate_certs": true
            }, 
            "psh": "enable", 
            "remark": null, 
            "rst": "enable", 
            "seq": "10", 
            "src": "1.1.1.1/24", 
            "src_port1": "1900", 
            "src_port2": "1910", 
            "src_port_op": "range", 
            "ssh_keyfile": null, 
            "state": "present", 
            "syn": "enable", 
            "time_range": "ans-range", 
            "timeout": 600, 
            "transport": "nxapi", 
            "urg": "enable", 
            "url_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "url_username": "admin", 
            "use_proxy": true, 
            "use_ssl": false, 
            "username": "admin", 
            "validate_certs": true
        }
    }
}

```
@mikewiebe Can you share the output of `show ip access-list | json` of the device where you are getting the issue.
@Qalthos I think this might be the place https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/httpapi/nxos.py#L32
I can't reproduce this, even on nxos9kv, however I did add a little guarding around the HTTPAPI response, and if Trishna is right, `devel` should now give you a clearer indication of what is going on
@Qalthos I was able to capture the error with the additional debug info:

```
2018-05-31T20:16:25: %LIB-INFO: TASK [nxos_acl : Configure ACE10] ***************************************************************************************************************************************************************************************************
2018-05-31T20:16:25: %LIB-INFO: task path: /root/agents-ci/ansible/test/integration/targets/nxos_acl/tests/common/sanity.yaml:16
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> loaded API plugin for network_os nxos
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> attempting to start connection
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> using connection plugin httpapi
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> found existing local domain socket, using it!
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> updating play_context for connection
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> 
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> local domain socket path is /root/.ansible/pc/2c79a65aa5
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: root
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-local-957sfLVl0/ansible-tmp-1527812184.29-138485875509358 `" && echo ansible-tmp-1527812184.29-138485875509358="` echo /root/.ansible/tmp/ansible-local-957sfLVl0/ansible-tmp-1527812184.29-138485875509358 `" ) && sleep 0'
2018-05-31T20:16:25: %LIB-INFO: Using module file /root/agents-ci/ansible/lib/ansible/modules/network/nxos/nxos_acl.py
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> PUT /root/.ansible/tmp/ansible-local-957sfLVl0/tmpgoj0M6 TO /root/.ansible/tmp/ansible-local-957sfLVl0/ansible-tmp-1527812184.29-138485875509358/nxos_acl.py
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-local-957sfLVl0/ansible-tmp-1527812184.29-138485875509358/ /root/.ansible/tmp/ansible-local-957sfLVl0/ansible-tmp-1527812184.29-138485875509358/nxos_acl.py && sleep 0'
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-local-957sfLVl0/ansible-tmp-1527812184.29-138485875509358/nxos_acl.py && sleep 0'
2018-05-31T20:16:25: %LIB-INFO: <n9k.example.com> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-local-957sfLVl0/ansible-tmp-1527812184.29-138485875509358/ > /dev/null 2>&1 && sleep 0'
2018-05-31T20:16:25: %LIB-INFO: The full traceback is:
2018-05-31T20:16:25: %LIB-INFO: Traceback (most recent call last):
2018-05-31T20:16:25: %LIB-INFO:   File "/tmp/ansible_BQprE5/ansible_module_nxos_acl.py", line 533, in <module>
2018-05-31T20:16:25: %LIB-INFO:     main()
2018-05-31T20:16:25: %LIB-INFO:   File "/tmp/ansible_BQprE5/ansible_module_nxos_acl.py", line 452, in main
2018-05-31T20:16:25: %LIB-INFO:     existing_core, acl = get_acl(module, name, seq)
2018-05-31T20:16:25: %LIB-INFO:   File "/tmp/ansible_BQprE5/ansible_module_nxos_acl.py", line 191, in get_acl
2018-05-31T20:16:25: %LIB-INFO:     body = execute_show_command(command, module)[0]
2018-05-31T20:16:25: %LIB-INFO:   File "/tmp/ansible_BQprE5/ansible_module_nxos_acl.py", line 181, in execute_show_command
2018-05-31T20:16:25: %LIB-INFO:     body = run_commands(module, cmds)
2018-05-31T20:16:25: %LIB-INFO:   File "/tmp/ansible_BQprE5/ansible_modlib.zip/ansible/module_utils/network/nxos/nxos.py", line 431, in run_commands
2018-05-31T20:16:25: %LIB-INFO:   File "/tmp/ansible_BQprE5/ansible_modlib.zip/ansible/module_utils/network/nxos/nxos.py", line 150, in run_commands
2018-05-31T20:16:25: %LIB-INFO:   File "/tmp/ansible_BQprE5/ansible_modlib.zip/ansible/module_utils/connection.py", line 149, in __rpc__
2018-05-31T20:16:25: %LIB-INFO: ansible.module_utils.connection.ConnectionError: Response was not valid JSON, got SSH-2.0-OpenSSH_7.2 FIPS
2018-05-31T20:16:25: %LIB-INFO: 
2018-05-31T20:16:25: %LIB-INFO: Protocol mismatch.
2018-05-31T20:16:25: %LIB-INFO: 
2018-05-31T20:16:25: %LIB-INFO: fatal: [n9k.example.com]: FAILED! => {
2018-05-31T20:16:25: %LIB-INFO:     "changed": false, 
2018-05-31T20:16:25: %LIB-INFO:     "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_BQprE5/ansible_module_nxos_acl.py\", line 533, in <module>\n    main()\n  File \"/tmp/ansible_BQprE5/ansible_module_nxos_acl.py\", line 452, in main\n    existing_core, acl = get_acl(module, name, seq)\n  File \"/tmp/ansible_BQprE5/ansible_module_nxos_acl.py\", line 191, in get_acl\n    body = execute_show_command(command, module)[0]\n  File \"/tmp/ansible_BQprE5/ansible_module_nxos_acl.py\", line 181, in execute_show_command\n    body = run_commands(module, cmds)\n  File \"/tmp/ansible_BQprE5/ansible_modlib.zip/ansible/module_utils/network/nxos/nxos.py\", line 431, in run_commands\n  File \"/tmp/ansible_BQprE5/ansible_modlib.zip/ansible/module_utils/network/nxos/nxos.py\", line 150, in run_commands\n  File \"/tmp/ansible_BQprE5/ansible_modlib.zip/ansible/module_utils/connection.py\", line 149, in __rpc__\nansible.module_utils.connection.ConnectionError: Response was not valid JSON, got SSH-2.0-OpenSSH_7.2 FIPS\r\nProtocol mismatch.\n\n", 
2018-05-31T20:16:25: %LIB-INFO:     "module_stdout": "", 
2018-05-31T20:16:25: %LIB-INFO:     "msg": "MODULE FAILURE", 
2018-05-31T20:16:25: %LIB-INFO:     "rc": 1
2018-05-31T20:16:25: %LIB-INFO: }
2018-05-31T20:16:25: %LIB-INFO: 	to retry, use: --limit @/root/agents-ci/ansible/test/integration/nxos.retry
2018-05-31T20:16:25: %LIB-INFO: 
2018-05-31T20:16:25: %LIB-INFO: PLAY RECAP **************************************************************************************************************************************************************************************************************************
2018-05-31T20:16:25: %LIB-INFO: n9k.example.com  : ok=127  changed=20   unreachable=0    failed=1   
```
Right, so this is probably a vars issue. I'm assuming you have `ansible_port` set, which will be read by both the `network_cli` and `httpapi` connections. If you instead specify `ansible_ssh_port` and `ansible_httpapi_port` (if either is not the default value), you should be able to avoid the conflict
@Qalthos We are not explicitly setting `ansible_port` AFIK in our test runs.  Here is what confuses me about this issue.

We have 10+ Ansible servers that we are using in our nightly runs and we use the same scripts on all of them.  This problem only exhibits on some of them.

Here is my `group_vars/nxos.yaml` file content from one of the problem servers.

```yaml
---
cli:
  host: "{{ ansible_ssh_host }}"
  username: "{{ nxos_cli_user | default('admin') }}"
  password: "{{ nxos_cli_pass | default('password') }}"
  transport: cli

nxapi:
  host: "{{ ansible_ssh_host }}"
  username: "{{ nxos_nxapi_user | default('admin') }}"
  password: "{{ nxos_nxapi_pass | default('password') }}"
  transport: nxapi
  use_ssl: no

ansible_ssh_user: admin
ansible_ssh_pass: password
ansible_network_os: nxos
```

Here is my playbook for one of the tests that fail.

```yaml
---
- hosts: nxos
  gather_facts: no
  connection: local

  vars:
    debug: false

  roles:
    - { role: nxos_acl }
```

Where else should I look to see if `ansible_port` is set?
So the good news is that I added `ansible_httpapi_port: 80` to my groupvars file and the test now passes on the problem server but IMO this should not be needed.  It should default to `80` right?  
It will default to the appropriate value based on `ansible_httpapi_use_ssl` (80/443), unless otherwise overridden.
Understood which is why I am confused about the current behavior I am seeing.

I am not explicitly setting `ansible_httpapi_port`, `ansible_httpapi_use_ssl` or `ansible_port` so I should be using default ports and not getting errors right?
It could also be set in ansible.cfg as `remote_port` under `[defaults]` or as `ANSIBLE_REMOTE_PORT` env var
@Qalthos I can reproduce the issue. If `ansible_httpapi_port` is not set, the port is getting set to ssh port `22` in the connection plugin httpapi. This should be 80/443 by default.

This is apparently because of `port = self.get_option('port')`  on Line https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/connection/httpapi.py#L247
Did some debug. This is happening when httpapi port is not set in vars:
```
 0.1s _update_connection_state: self._play_context.port=22
 0.1s _connect: port=22
```
@Qalthos Thanks for your help on this.  I checked the `ansible.cfg` file on my problem server and you are correct that my `remote_port` is set under `[defaults]` to `22`.  The file exists on my server that works but that setting is commented out.

Not sure if this is all currently documented in one place but it would be good to:

1. Document the relationship between the different port settings and how they can be set at a global and non-global level.
2. If possible provide clear error messages with suggestions for checking all of the possible places that might cause errors like this.


Okay, I've taken this apart and back, and I've come to a few conclusions.

`ansible_ssh_port` seems to get special handling, and gets automatically turned to `ansible_port` before option processing happens. This is, I'm sure, for important historical reasons.

This means that I can't rely on `ansible_port` to be generic, if `ansible_ssh_port` is set, that gets pushed to `ansible_port`... presumably this will go away when plugins rely entirely on defined options, but that day does not appear to be close.

So as a result, I've removed `ansible_port` from possible sources for `httpapi`, the PR should be above this. In the meantime, setting `ansible_httpapi_port` should have higher precedence and continue to override the ssh value

Again, something somewhere is setting one of these values. When no value is set to any of the sources for httpapi port (including ansible_ssh_port), ConfigManager properly sets the option to None, which is then set to the appropriate value in `_connect` based on the value of `ansible_httpapi_use_ssl`. Nothing I have been able to find suggests that this is not happening properly.

(To see this for yourself, ConfigManager in [config/manager.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/config/manager.py) has a lovely method called `get_config_value_and_origin` which gets called when a new option is requested. It naturally gets called a lot, but if you set a break point while it's looking up httpapi values, it will return where it got the value from, in addition to what the value is)

If you do come up with some other place that 22 is getting sent to httpapi, please let me know

Resolving patch:
diff --git a/lib/ansible/plugins/connection/httpapi.py b/lib/ansible/plugins/connection/httpapi.py
--- a/lib/ansible/plugins/connection/httpapi.py
+++ b/lib/ansible/plugins/connection/httpapi.py
@@ -33,7 +33,6 @@
     env:
       - name: ANSIBLE_REMOTE_PORT
     vars:
-      - name: ansible_port
       - name: ansible_httpapi_port
   network_os:
     description:


Test patch: