Repository: napalm-automation/napalm

Issue:
Partial Idempotent Configuration Push
### Description of Issue/Question

Add the ability to push partial configurations idempotently. 

Napalm provides two greats configuration deployment abilities:
* Config Replace
* Config Merge

While it is clear, Config Replace is more powerful, it is also harder to get to the maturity of full configuration deployment. Often times in customer implementations, Config Merge is being used. 

In Netutils, there is the capability to understand what configurations are missing from a potential Merge. You can see an example of this in the tests there: https://github.com/networktocode/netutils/tree/develop/tests/unit/mock/config/compliance/diff_network_config/cisco_ios

My proposal is to add another method to idempotently push partial configurations. This is actually the norm in Ansible, and should provide better integrations to get to the preferred Config Replace strategy. 

I am still thinking of appropriate naming of the method, but hopefully that provides some context. 

### Did you follow the steps from https://github.com/napalm-automation/napalm#faq
(Place an ``x`` between the square brackets where applicable)

- [x] Yes
- [] No


### Setup

### napalm version
N/A (New Feature)

### Network operating system version
(Paste verbatim output from `show version` - or equivalent - between quotes below)

N/A (New Feature)

### Steps to Reproduce the Issue
N/A (New Feature)

### Error Traceback
(Paste the complete traceback of the exception between quotes below)

N/A (New Feature)

Partial Idempotent Configuration Push
### Description of Issue/Question

Add the ability to push partial configurations idempotently. 

Napalm provides two greats configuration deployment abilities:
* Config Replace
* Config Merge

While it is clear, Config Replace is more powerful, it is also harder to get to the maturity of full configuration deployment. Often times in customer implementations, Config Merge is being used. 

In Netutils, there is the capability to understand what configurations are missing from a potential Merge. You can see an example of this in the tests there: https://github.com/networktocode/netutils/tree/develop/tests/unit/mock/config/compliance/diff_network_config/cisco_ios

My proposal is to add another method to idempotently push partial configurations. This is actually the norm in Ansible, and should provide better integrations to get to the preferred Config Replace strategy. 

I am still thinking of appropriate naming of the method, but hopefully that provides some context. 

### Did you follow the steps from https://github.com/napalm-automation/napalm#faq
(Place an ``x`` between the square brackets where applicable)

- [x] Yes
- [] No


### Setup

### napalm version
N/A (New Feature)

### Network operating system version
(Paste verbatim output from `show version` - or equivalent - between quotes below)

N/A (New Feature)

### Steps to Reproduce the Issue
N/A (New Feature)

### Error Traceback
(Paste the complete traceback of the exception between quotes below)

N/A (New Feature)



Discussion:
This would fit well with the multi-vendor NAPALM driver based on gNMI I am planning for the Hackathon.

gNMI has a natural mapping for these "partial" updates - every gNMI transaction is a set of { DELETE, REPLACE, UPDATE } paths that either gets applied atomically, or not at all
Sounds interesting...I will try to ask a few follow-up questions next week.
I would like to work on this issue and had some thoughts on how to implement it. I would suggest two new functions in the base driver:
```
   def load_partial_candidate(self, filename=None, config=None):
        """
        Populates a partial candidate configuration. You can populate it from a file or from a string.
        If you send both a filename and a string containing the configuration, the file takes
        precedence.

        If you use this method the existing configuration will be updated with the partial
        configuration once you commit the changes. This method will not change the configuration
        by itself.
        
        The method uses the netutils library to create the partial candidate configuration.

        :param filename: Path to the file containing the desired configuration. By default is None.
        :param config: String containing the desired configuration.
        :raise ConfigException: If there is an error on the configuration sent.
        """
        raise NotImplementedError

    def commit_partial_config(self, message="", check_mode=False):
        """
        Commits the changes requested by the method load_partial_candidate.

        NAPALM drivers that support 'commit confirm' should cause self.has_pending_commit
        to return True when a 'commit confirm' is in progress.

        Implementations should raise an exception if commit_config is called multiple times while a
        'commit confirm' is pending.

        :param message: Optional - configuration session commit message
        :type message: str
        :param revert_in: Optional - number of seconds before the configuration will be reverted
        :type check_mode: boolean - True for check_mode run

        Returns:
        :type config_committed: boolean, True if not dry run
        :type config_text: changes that were pushed
        """
        raise NotImplementedError
```

That would be a really nice feature 👍 
 @mirceaulinic  could you please assign @cmsirbu as assignee to this ticket? He has put a lot of work in this and I do not want to take the credit =)
Since only the NXOS merge diff was not giving the correct output (from the base drivers), we have changed it to use `netutils` for a partial merge diff that respects command hierarchy.

Here's an example:

```
Intended configuration:

candidate_cfg = """
interface loopback0
  ip address 10.1.4.5/32
  ip router ospf 100 area 0.0.0.1
"""

Base - full configuration on device with the relevant section below:
...
interface loopback0
  ip address 10.1.4.4/32
  ip router ospf 100 area 0.0.0.1
...

nxos1.load_merge_candidate(config=candidate_cfg)

Before:

In [1]: nxos1.compare_config()
Out[1]: '  ip address 10.1.4.5/32'


With netutils, respecting required command hierarchy:

In [1]: print(nxos1.compare_config())
interface loopback0
  ip address 10.1.4.5/32
```
This would fit well with the multi-vendor NAPALM driver based on gNMI I am planning for the Hackathon.

gNMI has a natural mapping for these "partial" updates - every gNMI transaction is a set of { DELETE, REPLACE, UPDATE } paths that either gets applied atomically, or not at all
Sounds interesting...I will try to ask a few follow-up questions next week.
I would like to work on this issue and had some thoughts on how to implement it. I would suggest two new functions in the base driver:
```
   def load_partial_candidate(self, filename=None, config=None):
        """
        Populates a partial candidate configuration. You can populate it from a file or from a string.
        If you send both a filename and a string containing the configuration, the file takes
        precedence.

        If you use this method the existing configuration will be updated with the partial
        configuration once you commit the changes. This method will not change the configuration
        by itself.
        
        The method uses the netutils library to create the partial candidate configuration.

        :param filename: Path to the file containing the desired configuration. By default is None.
        :param config: String containing the desired configuration.
        :raise ConfigException: If there is an error on the configuration sent.
        """
        raise NotImplementedError

    def commit_partial_config(self, message="", check_mode=False):
        """
        Commits the changes requested by the method load_partial_candidate.

        NAPALM drivers that support 'commit confirm' should cause self.has_pending_commit
        to return True when a 'commit confirm' is in progress.

        Implementations should raise an exception if commit_config is called multiple times while a
        'commit confirm' is pending.

        :param message: Optional - configuration session commit message
        :type message: str
        :param revert_in: Optional - number of seconds before the configuration will be reverted
        :type check_mode: boolean - True for check_mode run

        Returns:
        :type config_committed: boolean, True if not dry run
        :type config_text: changes that were pushed
        """
        raise NotImplementedError
```

That would be a really nice feature 👍 
 @mirceaulinic  could you please assign @cmsirbu as assignee to this ticket? He has put a lot of work in this and I do not want to take the credit =)
Since only the NXOS merge diff was not giving the correct output (from the base drivers), we have changed it to use `netutils` for a partial merge diff that respects command hierarchy.

Here's an example:

```
Intended configuration:

candidate_cfg = """
interface loopback0
  ip address 10.1.4.5/32
  ip router ospf 100 area 0.0.0.1
"""

Base - full configuration on device with the relevant section below:
...
interface loopback0
  ip address 10.1.4.4/32
  ip router ospf 100 area 0.0.0.1
...

nxos1.load_merge_candidate(config=candidate_cfg)

Before:

In [1]: nxos1.compare_config()
Out[1]: '  ip address 10.1.4.5/32'


With netutils, respecting required command hierarchy:

In [1]: print(nxos1.compare_config())
interface loopback0
  ip address 10.1.4.5/32
```

Resolving patch:
diff --git a/napalm/nxos/nxos.py b/napalm/nxos/nxos.py
--- a/napalm/nxos/nxos.py
+++ b/napalm/nxos/nxos.py
@@ -46,6 +46,7 @@
 from netaddr.core import AddrFormatError
 from netmiko import file_transfer
 from requests.exceptions import ConnectionError
+from netutils.config.compliance import diff_network_config
 
 import napalm.base.constants as c
 
@@ -209,22 +210,31 @@ def _commit_merge(self) -> None:
 
     def _get_merge_diff(self) -> str:
         """
-        The merge diff is not necessarily what needs to be loaded
-        for example under NTP, even though the 'ntp commit' command might be
-        alread configured, it is mandatory to be sent
-        otherwise it won't take the new configuration - see:
-        https://github.com/napalm-automation/napalm-nxos/issues/59
-        therefore this method will return the real diff (but not necessarily what is
-        being sent by the merge_load_config()
+        Uses netutils diff_network_config to create a partial configuration
+        with the proper hierarchy.
+        Note: the netutils utility performs the diff offline.
+
+        Returns: diff with the proper hierarchy of commands
+        that are missing from the current config.
+        Examples:
+        Candidate configuration:
+        interface loopback0
+          ip address 10.1.4.5/32
+          ip router ospf 100 area 0.0.0.1
+
+        Base (on device) - relevant section:
+        ...
+        interface loopback0
+          ip address 10.1.4.4/32
+          ip router ospf 100 area 0.0.0.1
+        ...
+
+        Diff that respects the required command hierarchy:
+        interface loopback0
+          ip address 10.1.4.5/32
         """
-        diff = []
         running_config = self.get_config(retrieve="running")["running"]
-        running_lines = running_config.splitlines()
-        for line in self.merge_candidate.splitlines():
-            if line not in running_lines and line:
-                if line[0].strip() != "!":
-                    diff.append(line)
-        return "\n".join(diff)
+        return diff_network_config(self.merge_candidate, running_config, "cisco_nxos")
 
     def _get_diff(self) -> str:
         """Get a diff between running config and a proposed file."""


Test patch: