Repository: sonic-net/sonic-utilities

Issue:
[GCU] apply-patch failed to get token from 'using' group
<!--
    If you are reporting a new issue, make sure that we do not have any duplicates
    already open. You can ensure this by searching the issue list for this
    repository. If there is a duplicate, please close your issue and add a comment
    to the existing issue instead.

    If you suspect your issue is a bug, please edit your issue description to
    include the BUG REPORT INFORMATION shown below. If you fail to provide this
    information within 7 days, we cannot debug your issue and will close it. We
    will, however, reopen it if you later provide the information.

    For more information about reporting issues, see
    https://github.com/Azure/SONiC/wiki#report-issues

    ---------------------------------------------------
    GENERAL SUPPORT INFORMATION
    ---------------------------------------------------

    The GitHub issue tracker is for bug reports and feature requests.
    General support can be found at the following locations:

    - SONiC Support Forums - https://groups.google.com/forum/#!forum/sonicproject

    ---------------------------------------------------
    BUG REPORT INFORMATION
    ---------------------------------------------------
    Use the commands below to provide key information from your environment:
    You do NOT have to include this information if this is a FEATURE REQUEST
-->

#### Description
When `apply-patch` a json file which yang has grouping member, the method cannot grab token.
<!--
     Briefly describe the problem you are having in a few paragraphs.
-->

## Steps to reproduce the issue
1. bgp-monitor.yang has member that defined in sonic-bgp-common.yang
https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-bgp-monitor.yang#L43
2. When apply-patch, it failed to grab token.
```
admin@vlab-01:~$ sudo config apply-patch -i /FEATURE -i /QUEUE -i /SCHEDULER bgpl/add.json
Patch Applier: Patch application starting.
Patch Applier: Patch: [{"op": "add", "path": "/BGP_MONITORS", "value": {"11.0.0.1": {"admin_status": "up", "asn": "65100", "holdtime": "10", "keepalive": "3", "local_addr": "10.1.0.32", "name": "BGPMonitor", "nhopself": "0", "rrclient": "0"}}}]
...
Failed to apply patch
Usage: config apply-patch [OPTIONS] PATCH_FILE_PATH
Try "config apply-patch -h" for help.

Error: Token not found 
```
3. some debug info when error occur in https://github.com/Azure/sonic-utilities/blob/master/generic_config_updater/gu_common.py#L551
`raise ValueError("Token not found model {} token_index {} path_tokens {}".format(model, token_index, path_tokens))`
The token is only got from `leaf` not from `using`
```
model OrderedDict([('@name', 'BGP_MONITORS_LIST'), 
('key', OrderedDict([('@value', 'addr')])), 
('description', OrderedDict([('text', 'This list is to support BGP monitor configuration handled by bgpcfgd')])), 
('leaf', OrderedDict([('@name', 'addr'), ('type', OrderedDict([('@name', 'inet:ip-address')])), 
('description', OrderedDict([('text', 'BGP monitor address')])), ('__isleafList', False)])), 

('uses', OrderedDict([('@name', 'bgpcmn:sonic-bgp-cmn-neigh'),    <===================================================

('refine', OrderedDict([('@target-node', 'name'), 
('must', OrderedDict([('@condition', "current() = 'BGPMonitor'"), 
('error-message', OrderedDict([('value', 'Invalid BGP monitor name')]))]))]))]))]) 
token_index 2 path_tokens ['BGP_MONITORS', '11.0.0.1', 'asn']
```
#### Describe the results you received
See above.

#### Describe the results you expected
Fix in two ways:
1. We don't using grouping in Yang such that bgp-monitor has all member defined in itself.
2. Handle `using` keyword in yang model.

#### Additional information you deem important (e.g. issue happens only occasionally)


#### Output of `show version`

```
(paste your output here)
```

<!--
     Also attach debug file produced by `sudo generate_dump`
-->




Discussion:

Resolving patch:
diff --git a/generic_config_updater/gu_common.py b/generic_config_updater/gu_common.py
--- a/generic_config_updater/gu_common.py
+++ b/generic_config_updater/gu_common.py
@@ -366,6 +366,9 @@ def create_xpath(self, tokens):
 
         return f"{PathAddressing.XPATH_SEPARATOR}{PathAddressing.XPATH_SEPARATOR.join(str(t) for t in tokens)}"
 
+    def _create_sonic_yang_with_loaded_models(self):
+        return self.config_wrapper.create_sonic_yang_with_loaded_models()
+
     def find_ref_paths(self, path, config):
         """
         Finds the paths referencing any line under the given 'path' within the given 'config'.
@@ -407,7 +410,7 @@ def find_ref_paths(self, path, config):
         return self._find_leafref_paths(path, config)
 
     def _find_leafref_paths(self, path, config):
-        sy = self.config_wrapper.create_sonic_yang_with_loaded_models()
+        sy = self._create_sonic_yang_with_loaded_models()
 
         tmp_config = copy.deepcopy(config)
 
@@ -553,6 +556,13 @@ def _get_xpath_tokens_from_leaf(self, model, token_index, path_tokens, config):
             #   /module-name:container/leaf-list[.='val']
             # Source: Check examples in https://netopeer.liberouter.org/doc/libyang/master/html/howto_x_path.html
             return [f"{token}[.='{value}']"]
+        
+        # checking 'uses' statement
+        if not isinstance(config[token], list): # leaf-list under uses is not supported yet in sonic_yang
+            table = path_tokens[0]
+            uses_leaf_model = self._get_uses_leaf_model(model, table, token)
+            if uses_leaf_model:
+                return [token]
 
         raise ValueError(f"Path token not found.\n  model: {model}\n  token_index: {token_index}\n  " + \
                          f"path_tokens: {path_tokens}\n  config: {config}")
@@ -719,6 +729,13 @@ def _get_path_tokens_from_leaf(self, model, token_index, xpath_tokens, config):
             list_idx = list_config.index(leaf_list_value)
             return [leaf_list_name, list_idx]
 
+        # checking 'uses' statement
+        if not isinstance(config[leaf_list_name], list):  # leaf-list under uses is not supported yet in sonic_yang
+            table = xpath_tokens[1]
+            uses_leaf_model = self._get_uses_leaf_model(model, table, token)
+            if uses_leaf_model:
+                return [token]
+
         raise ValueError(f"Xpath token not found.\n  model: {model}\n  token_index: {token_index}\n  " + \
                          f"xpath_tokens: {xpath_tokens}\n  config: {config}")
 
@@ -754,6 +771,45 @@ def _get_model(self, model, name):
 
         return None
 
+    def _get_uses_leaf_model(self, model, table, token):
+        """
+          Getting leaf model in uses model matching the given token.
+        """
+        uses_s = model.get('uses')
+        if not uses_s:
+            return None
+
+        # a model can be a single dict or a list of dictionaries, unify to a list of dictionaries
+        if not isinstance(uses_s, list):
+            uses_s = [uses_s]
+
+        sy = self._create_sonic_yang_with_loaded_models()
+        # find yang module for current table
+        table_module = sy.confDbYangMap[table]['yangModule']
+        # uses Example: "@name": "bgpcmn:sonic-bgp-cmn"
+        for uses in uses_s:
+            if not isinstance(uses, dict):
+                raise GenericConfigUpdaterError(f"'uses' is expected to be a dictionary found '{type(uses)}'.\n" \
+                                                f"  uses: {uses}\n  model: {model}\n  table: {table}\n  token: {token}")
+
+            # Assume ':'  means reference to another module
+            if ':' in uses['@name']:
+                name_parts = uses['@name'].split(':')
+                prefix = name_parts[0].strip()
+                uses_module_name = sy._findYangModuleFromPrefix(prefix, table_module)
+                grouping = name_parts[-1].strip()
+            else:
+                uses_module_name = table_module['@name']
+                grouping = uses['@name']
+
+            leafs = sy.preProcessedYang['grouping'][uses_module_name][grouping]
+
+            leaf_model = self._get_model(leafs, token)
+            if leaf_model:
+                return leaf_model
+
+        return None
+
 class TitledLogger(logger.Logger):
     def __init__(self, syslog_identifier, title, verbose, print_all_to_console):
         super().__init__(syslog_identifier)


Test patch:
diff --git a/tests/generic_config_updater/files/config_db_with_bgp_neighbor.json b/tests/generic_config_updater/files/config_db_with_bgp_neighbor.json
new file mode 100644
--- /dev/null
+++ b/tests/generic_config_updater/files/config_db_with_bgp_neighbor.json
@@ -0,0 +1,32 @@
+{
+    "BGP_NEIGHBOR": {
+        "1.2.3.4": {
+            "admin_status": "up",
+            "asn": "65000",
+            "holdtime": "10",
+            "keepalive": "3",
+            "local_addr": "10.0.0.10",
+            "name": "ARISTA03T1",
+            "nhopself": "0",
+            "rrclient": "0"
+        },
+        "default|1.2.3.4": {
+            "local_asn": "65200",
+            "asn": "65100",
+            "name": "bgp peer 65100",
+            "ebgp_multihop_ttl": "3"
+        }
+    },
+    "BGP_MONITORS": {
+        "5.6.7.8": {
+            "admin_status": "up",
+            "asn": "65000",
+            "holdtime": "180",
+            "keepalive": "60",
+            "local_addr": "10.0.0.11",
+            "name": "BGPMonitor",
+            "nhopself": "0",
+            "rrclient": "0"
+        }
+    }
+}
\ No newline at end of file
diff --git a/tests/generic_config_updater/files/config_db_with_lldp.json b/tests/generic_config_updater/files/config_db_with_lldp.json
new file mode 100644
--- /dev/null
+++ b/tests/generic_config_updater/files/config_db_with_lldp.json
@@ -0,0 +1,14 @@
+{
+    "LLDP": {
+        "GLOBAL": {
+            "mode": "TRANSMIT",
+            "enabled": "true",
+            "hello_time": "12",
+            "multiplier": "5",
+            "supp_mgmt_address_tlv": "true",
+            "supp_system_capabilities_tlv": "false",
+            "system_name": "sonic",
+            "system_description": "sonic-system"
+        }
+    }
+}
\ No newline at end of file
diff --git a/tests/generic_config_updater/gu_common_test.py b/tests/generic_config_updater/gu_common_test.py
--- a/tests/generic_config_updater/gu_common_test.py
+++ b/tests/generic_config_updater/gu_common_test.py
@@ -749,6 +749,18 @@ def check(path, xpath, config=None):
         check(path="/PORTCHANNEL_INTERFACE/PortChannel0001|1.1.1.1~124",
               xpath="/sonic-portchannel:sonic-portchannel/PORTCHANNEL_INTERFACE/PORTCHANNEL_INTERFACE_IPPREFIX_LIST[name='PortChannel0001'][ip_prefix='1.1.1.1/24']",
               config=Files.CONFIG_DB_WITH_PORTCHANNEL_INTERFACE)
+        check(path="/BGP_NEIGHBOR/1.2.3.4/holdtime",
+              xpath="/sonic-bgp-neighbor:sonic-bgp-neighbor/BGP_NEIGHBOR/BGP_NEIGHBOR_TEMPLATE_LIST[neighbor='1.2.3.4']/holdtime",
+              config=Files.CONFIG_DB_WITH_BGP_NEIGHBOR)
+        check(path="/BGP_NEIGHBOR/default|1.2.3.4/asn",
+              xpath="/sonic-bgp-neighbor:sonic-bgp-neighbor/BGP_NEIGHBOR/BGP_NEIGHBOR_LIST[vrf_name='default'][neighbor='1.2.3.4']/asn",
+              config=Files.CONFIG_DB_WITH_BGP_NEIGHBOR)
+        check(path="/BGP_MONITORS/5.6.7.8/name",
+              xpath="/sonic-bgp-monitor:sonic-bgp-monitor/BGP_MONITORS/BGP_MONITORS_LIST[addr='5.6.7.8']/name",
+              config=Files.CONFIG_DB_WITH_BGP_NEIGHBOR)
+        check(path="/LLDP/GLOBAL/mode",
+              xpath="/sonic-lldp:sonic-lldp/LLDP/GLOBAL/mode",
+              config=Files.CONFIG_DB_WITH_LLDP)
 
     def test_convert_xpath_to_path(self):
         def check(xpath, path, config=None):
@@ -812,6 +824,18 @@ def check(xpath, path, config=None):
         check(xpath="/sonic-portchannel:sonic-portchannel/PORTCHANNEL_INTERFACE/PORTCHANNEL_INTERFACE_IPPREFIX_LIST[name='PortChannel0001'][ip_prefix='1.1.1.1/24']",
               path="/PORTCHANNEL_INTERFACE/PortChannel0001|1.1.1.1~124",
               config=Files.CONFIG_DB_WITH_PORTCHANNEL_INTERFACE)
+        check(xpath="/sonic-bgp-neighbor:sonic-bgp-neighbor/BGP_NEIGHBOR/BGP_NEIGHBOR_TEMPLATE_LIST[neighbor='1.2.3.4']/holdtime",
+              path="/BGP_NEIGHBOR/1.2.3.4/holdtime",
+              config=Files.CONFIG_DB_WITH_BGP_NEIGHBOR)
+        check(xpath="/sonic-bgp-neighbor:sonic-bgp-neighbor/BGP_NEIGHBOR/BGP_NEIGHBOR_LIST[vrf_name='default'][neighbor='1.2.3.4']/asn",
+              path="/BGP_NEIGHBOR/default|1.2.3.4/asn",
+              config=Files.CONFIG_DB_WITH_BGP_NEIGHBOR)
+        check(xpath="/sonic-bgp-monitor:sonic-bgp-monitor/BGP_MONITORS/BGP_MONITORS_LIST[addr='5.6.7.8']/name",
+              path="/BGP_MONITORS/5.6.7.8/name",
+              config=Files.CONFIG_DB_WITH_BGP_NEIGHBOR)
+        check(xpath="/sonic-lldp:sonic-lldp/LLDP/GLOBAL/mode",
+              path="/LLDP/GLOBAL/mode",
+              config=Files.CONFIG_DB_WITH_LLDP)
 
     def test_has_path(self):
         def check(config, path, expected):