Repository: ansible/ansible

Issue:
ios_config does not support the sub-level configuration mode related to "crypto pki trustpoint"
##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME

ios_config

##### ANSIBLE VERSION

Latest unstable version:
```
ansible 2.5.0 (devel c3199dca6c) last updated 2018/02/02 16:43:27 (GMT +200)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/actionmystique/Ansible/git-yang-networkop/ansible-101/library']
  ansible python module location = /media/actionmystique/SAMSUNG5-Shared/home/actionmystique/src/Ansible/git-ansible/lib/ansible
  executable location = /home/actionmystique/src/Ansible/git-ansible/bin/ansible
  python version = 2.7.14+ (default, Jan 23 2018, 14:47:36) [GCC 7.2.0]
```
##### CONFIGURATION
```
inventory   = ./hosts
library        = /home/actionmystique/Ansible/git-yang-networkop/ansible-101/library
forks = 1000
gathering = explicit
gather_timeout = 30
roles_path = /home/actionmystique/Ansible/Roles/roles
private_role_vars = yes
hash_behaviour = merge
log_path = /var/log/ansible.log
retry_files_enabled = False
show_custom_stats = True
timeout = 120
pipelining = True
connect_timeout = 240
connect_retry_timeout = 180
```
##### OS / ENVIRONMENT
- host: Ubuntu 17.10 4.13
- paramiko: 2.4.0

##### SUMMARY

ios_config requests the remote device to execute a command 'None' or empty which was not requested in the list of commands.
This situation seems to be triggered when subsequent commands follow by the specific command "crypto pki trustpoint".

##### STEPS TO REPRODUCE

**Role**: :

```
- name: Installing the TLS certifcate on the remote device
  ios_config:
        provider: "{{ connections.ssh }}"
        replace: line
        lines:
          - "crypto pki trustpoint actionmystique.net"
          - "enrollment selfsigned"
          - "subject-name cn=samsung-ubuntu.actionmystique.net-Certificate"
          - "revocation-check none"
          - "rsakeypair actionmystique.net"
          - "crypto ca certificate chain actionmystique.net"
...
        save_when: modified
  register: ios_config_return
```

##### EXPECTED RESULTS

The module should execute only the list of requested commands.

##### ACTUAL RESULTS: Playbook

```
 ___________________________________________________________
/ TASK [ios_tls_cert : Installing the TLS certifcate on the \
\ remote device]                                            /
 -----------------------------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

task path: /media/actionmystique/SAMSUNG5-Shared/home/actionmystique/Repositories/SDX-Live/Roles/roles/ios_tls_cert/tasks/push.yml:63
<172.21.85.132> using connection plugin network_cli (was local)
<172.21.85.132> starting connection from persistent connection plugin
<172.21.85.132> local domain socket does not exist, starting it
<172.21.85.132> control socket path is /root/.ansible/pc/a528aca52e
<172.21.85.132> connection to remote device started successfully
<172.21.85.132> local domain socket listeners started successfully
<172.21.85.132> 
<172.21.85.132> local domain socket path is /root/.ansible/pc/a528aca52e
<172.21.85.132> socket_path: /root/.ansible/pc/a528aca52e
Using module file /media/actionmystique/SAMSUNG5-Shared/home/actionmystique/src/Ansible/git-ansible/lib/ansible/modules/network/ios/ios_config.py
<172.21.85.132> ESTABLISH LOCAL CONNECTION FOR USER: root
<172.21.85.132> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_wptXPP/ansible_module_ios_config.py", line 551, in <module>
    main()
  File "/tmp/ansible_wptXPP/ansible_module_ios_config.py", line 480, in main
    load_config(module, commands)
  File "/tmp/ansible_wptXPP/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py", line 162, in load_config
  File "/tmp/ansible_wptXPP/ansible_modlib.zip/ansible/module_utils/connection.py", line 146, in __rpc__
ansible.module_utils.connection.ConnectionError:              ^
% Invalid input detected at '^' marker.

XEv_Spine_32(ca-trustpoint)#rsakeypair actionmystique.net
XEv_Spine_32(ca-trustpoint)#None
                              ^
% Invalid input detected at '^' marker.

XEv_Spine_32(ca-trustpoint)#

fatal: [XEv_Spine_32]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_wptXPP/ansible_module_ios_config.py\", line 551, in <module>\n    main()\n  File \"/tmp/ansible_wptXPP/ansible_module_ios_config.py\", line 480, in main\n    load_config(module, commands)\n  File \"/tmp/ansible_wptXPP/ansible_modlib.zip/ansible/module_utils/network/ios/ios.py\", line 162, in load_config\n  File \"/tmp/ansible_wptXPP/ansible_modlib.zip/ansible/module_utils/connection.py\", line 146, in __rpc__\nansible.module_utils.connection.ConnectionError:              ^\r\n% Invalid input detected at '^' marker.\r\n\r\nXEv_Spine_32(ca-trustpoint)#rsakeypair actionmystique.net\r\nXEv_Spine_32(ca-trustpoint)#None\r\n                              ^\r\n% Invalid input detected at '^' marker.\r\n\r\nXEv_Spine_32(ca-trustpoint)#\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}
```
The commands sent to the remote device by ios_config have been logged:
```
*Feb  2 15:45:24.228: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: admin] [Source: 172.21.0.1] [localport: 22] at 15:45:24 UTC Fri Feb 2 2018
*Feb  2 15:45:24.300: %HA_EM-6-LOG: CLIlog: terminal length 0
*Feb  2 15:45:24.367: %HA_EM-6-LOG: CLIlog: terminal width 512
*Feb  2 15:45:24.719: %HA_EM-6-LOG: CLIlog: show version 
*Feb  2 15:45:24.786: %HA_EM-6-LOG: CLIlog: show running-config all 
*Feb  2 15:45:28.394: %HA_EM-6-LOG: CLIlog: configure terminal 
*Feb  2 15:45:28.557: %HA_EM-6-LOG: CLIlog: crypto pki trustpoint actionmystique.net
*Feb  2 15:45:28.651: %HA_EM-6-LOG: CLIlog: enrollment selfsigned 
*Feb  2 15:45:28.843: %HA_EM-6-LOG: CLIlog: subject-name cn=samsung-ubuntu.actionmystique.net-Certificate
*Feb  2 15:45:28.908: %HA_EM-6-LOG: CLIlog: revocation-check none 
*Feb  2 15:45:29.004: %HA_EM-6-LOG: CLIlog: rsakeypair actionmystique.net
*Feb  2 15:45:29.070: %SYS-6-LOGOUT: User admin has exited tty session 3(172.21.0.1)
*Feb  2 15:45:29.070: %SYS-5-CONFIG_I: Configured from console by admin on vty2 (172.21.0.1)
```
Configuring manually the device on the CLI is correct:
```
...
XEv_Spine_32(ca-trustpoint)#revocation-check none
*Feb  2 15:52:22.840: %HA_EM-6-LOG: CLIlog: revocation-check none 
XEv_Spine_32(ca-trustpoint)#rsakeypair actionmystique.net
*Feb  2 15:52:32.370: %HA_EM-6-LOG: CLIlog: rsakeypair actionmystique.net
XEv_Spine_32(ca-trustpoint)#crypto ca certificate chain actionmystique.net
*Feb  2 15:53:05.612: %HA_EM-6-LOG: CLIlog: crypto ca certificate chain actionmystique.net
...
```


Discussion:
Files identified in the description:
* [lib/ansible/modules/network/ios/ios_config.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/ios/ios_config.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @kedarX @privateip @rcarrillocruz
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
On latest stable version:
```
ansible 2.4.3.0 (detached HEAD 8a7f9beab7) last updated 2018/02/02 17:17:09 (GMT +200)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/actionmystique/Ansible/git-yang-networkop/ansible-101/library']
  ansible python module location = /media/actionmystique/SAMSUNG5-Shared/home/actionmystique/src/Ansible/git-ansible/lib/ansible
  executable location = /home/actionmystique/src/Ansible/git-ansible/bin/ansible
  python version = 2.7.14+ (default, Jan 23 2018, 14:47:36) [GCC 7.2.0]
```
the issue is there but the symptom is different: this time the module breaks after the second command:
```
/ TASK [ios_tls_cert : Installing the TLS certifcate on the \
\ remote device]                                            /
 -----------------------------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

task path: /media/actionmystique/SAMSUNG5-Shared/home/actionmystique/Repositories/SDX-Live/Roles/roles/ios_tls_cert/tasks/push.yml:63
<172.21.85.132> using connection plugin network_cli
<172.21.85.132> socket_path: /root/.ansible/pc/1d677c45c4
Using module file /media/actionmystique/SAMSUNG5-Shared/home/actionmystique/src/Ansible/git-ansible/lib/ansible/modules/network/ios/ios_config.py
<172.21.85.132> ESTABLISH LOCAL CONNECTION FOR USER: root
<172.21.85.132> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
fatal: [XEv_Spine_32]: FAILED! => {
    "changed": false, 
    "command": "enrollment selfsigned", 
    "invocation": {
        "module_args": {
            "after": null, 
            "auth_pass": null, 
            "authorize": null, 
            "backup": false, 
            "before": null, 
            "defaults": false, 
            "diff_against": null, 
            "diff_ignore_lines": null, 
            "force": false, 
            "host": null, 
            "intended_config": null, 
            "lines": [
                "crypto pki trustpoint actionmystique.net", 
                "enrollment selfsigned", 
                "subject-name cn=samsung-ubuntu.actionmystique.net-Certificate", 
                "revocation-check none", 
                "rsakeypair actionmystique.net", 
...
            ], 
            "match": "line", 
            "multiline_delimiter": "@", 
            "parents": null, 
            "password": null, 
            "port": null, 
            "provider": {
                "auth_pass": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "authorize": true, 
                "host": "172.21.85.132", 
                "password": null, 
                "port": 22, 
                "ssh_keyfile": "/root/.ssh/id_rsa_4096", 
                "timeout": 60, 
                "username": "admin"
            }, 
            "replace": "line", 
            "running_config": null, 
            "save": false, 
            "save_when": "modified", 
            "src": null, 
            "ssh_keyfile": null, 
            "timeout": null, 
            "username": null
        }
    }, 
    "msg": "enrollment selfsigned\r\n                       ^\r\n% Invalid input detected at '^' marker.\r\n\r\nXEv_Spine_32(config)#", 
    "rc": 1
}
```

As a side-note, you may have noticed that this time, all the context is nicely printed (cf. https://github.com/ansible/ansible/issues/34554).
This issue happens even if I drop all commands below the second one, i.e with:
```
- name: Installing the TLS certifcate on the remote device
  ios_config:
        provider: "{{ connections.ssh }}"
        replace: line
        lines:
          - "crypto pki trustpoint {{ cert.private.ca_name }}"
          - "enrollment selfsigned"
        save_when: modified
```
which leads to:
```
 ___________________________________________________________
/ TASK [ios_tls_cert : Installing the TLS certifcate on the \
\ remote device]                                            /
 -----------------------------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

task path: /media/actionmystique/SAMSUNG5-Shared/home/actionmystique/Repositories/SDX-Live/Roles/roles/ios_tls_cert/tasks/push.yml:63
<172.21.85.131> using connection plugin network_cli
<172.21.85.131> socket_path: /root/.ansible/pc/9a98612d7e
Using module file /media/actionmystique/SAMSUNG5-Shared/home/actionmystique/src/Ansible/git-ansible/lib/ansible/modules/network/ios/ios_config.py
<172.21.85.131> ESTABLISH LOCAL CONNECTION FOR USER: root
<172.21.85.131> EXEC /bin/sh -c '/usr/bin/python && sleep 0'
fatal: [XEv_Spine_31]: FAILED! => {
    "changed": false, 
    "command": "enrollment selfsigned", 
    "invocation": {
        "module_args": {
            "after": null, 
            "auth_pass": null, 
            "authorize": null, 
            "backup": false, 
            "before": null, 
            "defaults": false, 
            "diff_against": null, 
            "diff_ignore_lines": null, 
            "force": false, 
            "host": null, 
            "intended_config": null, 
            "lines": [
                "crypto pki trustpoint actionmystique.net", 
                "enrollment selfsigned"
            ], 
            "match": "line", 
            "multiline_delimiter": "@", 
            "parents": null, 
            "password": null, 
            "port": null, 
            "provider": {
                "auth_pass": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "authorize": true, 
                "host": "172.21.85.131", 
                "password": null, 
                "port": 22, 
                "ssh_keyfile": "/root/.ssh/id_rsa_4096", 
                "timeout": 60, 
                "username": "admin"
            }, 
            "replace": "line", 
            "running_config": null, 
            "save": false, 
            "save_when": "modified", 
            "src": null, 
            "ssh_keyfile": null, 
            "timeout": null, 
            "username": null
        }
    }, 
    "msg": "enrollment selfsigned\r\n                       ^\r\n% Invalid input detected at '^' marker.\r\n\r\nXEv_Spine_31(config)#", 
    "rc": 1
}
```
If I change the second command by another one, the issue remains.
Since this is the first time I encounter it with a multiple commands call, we could conclude that it is due to the fact that the **second command is to be run in a sub-level configuration mode:**
Here, we have ```(ca-trustpoint)#``` instead of the usual 1st tier ```(config)#```
If I change the first 2 lines by the following commands, the issue is gone:
```
          - "interface GigabitEthernet 2" 
          - "ip address 1.1.1.1 255.255.0.0" 
```
So, it seems specific to the first command ```crypto pki trustpoint```.
Is this a multi prompt command?
`ios_config` doesn't support `prompt:` and `answer:` though `ios_command` does.
Currently `ios_command` will reject config command.
We may want to add `prompt:` & `answer:` into `ios_config`

Would be interesting to know where the `none` is coming from.

We will investigate some more
Alright, there seem to be a few things going on here, and I'd like your help confirming how many of them are actually the case.

One issue has to do with how we try to handle prompts in ios_config. There is a bug there, which should be fixed by #36183, which is why @gundalow thought there might be some issue with prompts.

The other issue seems to be the use of `replace: line` for sub config sections. This is why you're getting `Invalid input detected` for `enrollment selfsigned` in later tests- `crypto pki trustpoint actionmystique.net` is in the config, so it skips that and tries to run `enrollment selfsigned` without entering `(ca-trustpoint)#`, and fails. It seems like you would be better served by

```
 - ios_config:
      parents:
        - "crypto pki trustpoint example.com"
      lines:
        - "enrollment selfsigned"
        - "subject-name cn=samsung-ubuntu.actionmystique.net-Certificate"
        - "revocation-check none"
        - "rsakeypair example.com"
        - "crypto ca certificate chain actionmystique.net"
```
to get the right effect.

Resolving patch:
diff --git a/lib/ansible/plugins/cliconf/ios.py b/lib/ansible/plugins/cliconf/ios.py
--- a/lib/ansible/plugins/cliconf/ios.py
+++ b/lib/ansible/plugins/cliconf/ios.py
@@ -81,8 +81,7 @@ def edit_config(self, command):
                 answer = None
                 newline = True
 
-            self.send_command(to_bytes(command), to_bytes(prompt), to_bytes(answer),
-                              False, newline)
+            self.send_command(command, prompt, answer, False, newline)
 
     def get(self, command, prompt=None, answer=None, sendonly=False):
         return self.send_command(command, prompt=prompt, answer=answer, sendonly=sendonly)


Test patch: