Repository: ansible/ansible

Issue:
JunOs commit confirmed instant rollback
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->
`confirm` option of the `junos_config ` module does not work as expected.  Commits are immediately rolled back.
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
junos_config
##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.3
  config file = None
  configured module search path = [u'/path/to/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
ANSIBLE_NOCOWS(env: ANSIBLE_NOCOWS) = True
##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->

OS running ansible:
Ubuntu 16.04.3 LTS

OS being managed:
Model: qfx5100-48s-6q
Junos: 17.3R1.10

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->
Create and run a task to apply config changes to a juniper switch:
<!--- Paste example playbooks or commands between quotes below -->
```
  - name: Apply - Generated config
    junos_config:
      host: "{{ juniper_ssh_host }}"
      port: "{{ juniper_ssh_port }}"
      username: "{{ juniper_ssh_user }}"
      ssh_keyfile: "{{ juniper_ssh_key_file }}"
      src: "path/to/juniper-full-config.set"
      src_format: set
      confirm: 20
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
I expected the commit to have a timeout of 20 minutes before being rolled back.
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
The commit is immediately rolled back.

##### ADDITIONAL
This is an issue with the ncclient library being used by the ansible junos_* modules, which is detailed in the following relevant github links.
https://github.com/NetworkAutomation/jaide/issues/6
https://github.com/ncclient/ncclient/pull/37/files

TL;DR: Junos expects the xml tree for a commit RPC to have `commit-configuration` as the top level node, not `commit`, but ncclient continues to use the incorrect top level node name.

##### SUGGESTED FIX
Remove the dependency on `ncclient` or fork it, or shift to using vendor-provided libraries, such as https://github.com/Juniper/ansible-junos-stdlib


Discussion:
Files identified in the description:
* [lib/ansible/modules/network/junos/junos_config.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/junos/junos_config.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Qalthos @ganeshrn @privateip
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
We have experienced this issue as well, but only with ansible 2.5. We ran ansible 2.4 and upgraded to 2.5.3. In 2.5.3 we got the issue that even though we set 'confirm: 5' on the juniper device the commit was done with rollback in 0mins. We downgraded to ansible 2.4.4 and our junos_config task are working correctly again with rolling back in 5 minutes.

our ncclient version has stayed the same on version 0.5.3
Interesting.  I did confirm that the fix for me was to edit the ncclient source based on https://github.com/ncclient/ncclient/pull/37/files

Prior to our move to 2.5, we were using the vendor-provided module and so wouldn't have noticed the issue with the native ansible module prior to 2.5 had it even been present.
Juniper states that they do indeed expect the top level node to be commit-configuration and not commit which is what ncclient is sending.

https://www.juniper.net/documentation/en_US/junos/topics/reference/tag-summary/junos-xml-protocol-commit-configuration.html

If I understand the ansible code of 2.4.4 correctly, it didn't use ncclient to send the commit but set up the socket itself and did set the top level to be commit-configuration. My understanding of python is very basic and so might be wrong.
Would seem you're correct about how 2.4 was handling the configuration: https://github.com/ansible/ansible/blob/b3a1761365fdb7857a5581a955a9042e9e526d3f/lib/ansible/module_utils/connection.py#L63-L69

It seems another possible fix here is to toss out the usage of ncclient and simply revert to the previous method for loading configs.
I figured out a little bit more what is the issue. ncclient states that the confirm_timeout is in seconds and this is correct. By setting the confirm_timeout value in seconds I did see a confirm rollback in the correct minutes. 
This means that there are a few options:
* update Ansible documentation to state that the timeout of junos_config is in seconds instead of minutes
* multiply by 60 the value set by the confirm_timeout to transform it from minutes to seconds before passing the value to ncclient
* get ncclient to update their code to use the netconf top level commit-configuration instead of commit, which seems to be deprecated.

@ganeshrn what are your thoughts in this?
Interesting that you saw any difference.  I tried setting confirm_timeout to an enormous number and still rollback happened immediately.
I set confirm_timeout to 300 and I saw this when i did 'show | compare rollback ?'

2018-06-11 14:30:46 UTC by sa_ansible via netconf commit confirmed, rollback in 5mins

Did you happen to be on the switch at the moment you tested that?  IIRC, I also saw a rollback entry showing what appeared to be the correctly expected timeout value, but the rollback continued to happen immediately.  I only noticed when I was on the switch and saw the rollbacks actually happen.
I was connected to the SRX at the moment. I retried it again after your comment with a confirm timeout of 5 minutes and the rollback happened after 1 minute and than every time after that after a few seconds. I am unsure as to why the rollback is slower the first time around, but it in all cases it isn't working as it should.

I did a test to confirm that in ansible 2.4.4 it is working as it should and there it counted all the way down to 5 minutes. Also running ansible 2.5.3+ with the ncclient fix also causes the rollback to happen at the specified time.

So sadly the timeout seems to quite fast (usually within seconds) when top level of the netconf is 'commit'. Juniper won't fix that off course as their documentation states to use 'commit-configuration'.
@bcavns01 @rikkertk Thank you for figuring out the root cause of this issue.

As you rightly mentioned in your above comment in 2.4 Ansible generated the raw xml for all Netconf operation and used generic ncclient `rpc()` method to execute it on the remote host.

In 2.5 it was refactored to invoke individual ncclient api's with appropriate arguments. Hence the issue in ncclient `commit()`api for Juniper was exposed from 2.5 onwards.

As you suggested I have added a fix https://github.com/ansible/ansible/pull/41527 to generate raw `commit-configuration` xml with correct tags within Ansible itself. Please test this patch in your environment. 

If this PR fixes the issue I will backport it to `stable-2.5` and `stable-2.6` branches and fix will be available in next release of `2.5.X`

FYI I have raised an issue and fix for this on ncclient repo https://github.com/ncclient/ncclient/pull/239
@ganeshrn your change does a correct timeout when i push configuration against a srx. 

I saw that ncclient also had a look at your pull request, but they changed their code for timeout to be defined in seconds and than does a translation at the end to the minutes juniper expects. And at least for me their code had an issue with that conversion and didn't actually set a netconf connection.

@bcavns01 : do you have time to test fix #41527 aswell?
@rikkertk Since this fix creates the commit-configuration rpc and not uses ncclient commit api, the changes in ncclient won't affect this fix. However in future when ncclient version is released and we move to invoke ncclient commit api in that case need to handle minutes to seconds conversion within Ansible

Resolving patch:
diff --git a/lib/ansible/modules/network/junos/junos_config.py b/lib/ansible/modules/network/junos/junos_config.py
--- a/lib/ansible/modules/network/junos/junos_config.py
+++ b/lib/ansible/modules/network/junos/junos_config.py
@@ -73,8 +73,8 @@
     default: null
   confirm:
     description:
-      - The C(confirm) argument will configure a time out value for
-        the commit to be confirmed before it is automatically
+      - The C(confirm) argument will configure a time out value in minutes
+        for the commit to be confirmed before it is automatically
         rolled back.  If the C(confirm) argument is set to False, this
         argument is silently ignored.  If the value for this argument
         is set to 0, the commit is confirmed immediately.
diff --git a/lib/ansible/plugins/netconf/junos.py b/lib/ansible/plugins/netconf/junos.py
--- a/lib/ansible/plugins/netconf/junos.py
+++ b/lib/ansible/plugins/netconf/junos.py
@@ -32,7 +32,7 @@
     from ncclient import manager
     from ncclient.operations import RPCError
     from ncclient.transport.errors import SSHUnknownHostError
-    from ncclient.xml_ import to_ele, to_xml, new_ele
+    from ncclient.xml_ import to_ele, to_xml, new_ele, sub_ele
 except ImportError:
     raise AnsibleError("ncclient is not installed")
 
@@ -144,11 +144,6 @@ def reboot(self):
         """reboot the device"""
         return self.m.reboot().data_xml
 
-    @ensure_connected
-    def halt(self):
-        """reboot the device"""
-        return self.m.halt().data_xml
-
     @ensure_connected
     def get(self, *args, **kwargs):
         try:
@@ -184,3 +179,37 @@ def validate(self, *args, **kwargs):
     @ensure_connected
     def discard_changes(self, *args, **kwargs):
         return self.m.discard_changes(*args, **kwargs).data_xml
+
+    # Due to issue in ncclient commit() method for Juniper (https://github.com/ncclient/ncclient/issues/238)
+    # below commit() is a workaround which build's raw `commit-configuration` xml with required tags and uses
+    # ncclient generic rpc() method to execute rpc on remote host.
+    # Remove below method after the issue in ncclient is fixed.
+    @ensure_connected
+    def commit(self, confirmed=False, check=False, timeout=None, comment=None, synchronize=False, at_time=None):
+        """Commit the candidate configuration as the device's new current configuration.
+           Depends on the `:candidate` capability.
+           A confirmed commit (i.e. if *confirmed* is `True`) is reverted if there is no
+           followup commit within the *timeout* interval. If no timeout is specified the
+           confirm timeout defaults to 600 seconds (10 minutes).
+           A confirming commit may have the *confirmed* parameter but this is not required.
+           Depends on the `:confirmed-commit` capability.
+        :confirmed: whether this is a confirmed commit
+        :timeout: specifies the confirm timeout in seconds
+        """
+        obj = new_ele('commit-configuration')
+        if confirmed:
+            sub_ele(obj, 'confirmed')
+        if check:
+            sub_ele(obj, 'check')
+        if synchronize:
+            sub_ele(obj, 'synchronize')
+        if at_time:
+            subele = sub_ele(obj, 'at-time')
+            subele.text = str(at_time)
+        if comment:
+            subele = sub_ele(obj, 'log')
+            subele.text = str(comment)
+        if timeout:
+            subele = sub_ele(obj, 'confirm-timeout')
+            subele.text = str(timeout)
+        return self.rpc(obj)


Test patch: