Repository: sonic-net/sonic-mgmt

Issue:
Investigate RDMA nightly run failures on 202012
<!--
If you are reporting a new issue, make sure that we do not have any duplicates
already open. You can ensure this by searching the issue list for this
repository. If there is a duplicate, please close your issue and add a comment
to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to
include the BUG REPORT INFORMATION shown below. If you fail to provide this
information within 7 days, we cannot debug your issue and will close it. We
will, however, reopen it if you later provide the information.

For more information about reporting issues, see
https://github.com/Azure/SONiC/wiki#report-issues

---------------------------------------------------
GENERAL SUPPORT INFORMATION
---------------------------------------------------

The GitHub issue tracker is for bug reports and feature requests.
General support can be found at the following locations:

- SONiC Support Forums - https://groups.google.com/forum/#!forum/sonicproject

---------------------------------------------------
BUG REPORT INFORMATION
---------------------------------------------------
Use the commands below to provide key information from your environment:
You do NOT have to include this information if this is a FEATURE REQUEST
-->

**Description**
RDMA test runs on TD2 with 202012 are quite flaky. Different set of test failures are seen daily and sometimes test fails at pretest
09/09 run skipped all tgen tests with the following reason
SKIPPED [1] /azp/agent/_work/27/s/tests/common/helpers/assertions.py:13: Port is not mapped to the expected DUT




Discussion:
Ports connected to the ixia chassis are oper down.  Wei will look at the ixia chassis
Port issue resolved. failures seen currently for pfc_pause_lossy and pfcwd. pfcwd is known failure. Wei is looking at lossy failures
Wei mentioned that RX drops are seen on pc intf with 202012 vs 201811
Issue is because the lag member has cable len set to 0m and hence no buffer profile is attached. Need more modifications on minigraph generation for 'tgen' topology to ensure that non zero cable len is set

Resolving patch:
diff --git a/ansible/library/testbed_vm_info.py b/ansible/library/testbed_vm_info.py
--- a/ansible/library/testbed_vm_info.py
+++ b/ansible/library/testbed_vm_info.py
@@ -39,6 +39,7 @@
 ### Here are the assumption/expectation of files to gather VM informations, if the file location or name changes, please modify it here
 TOPO_PATH = 'vars/'
 VM_INV_FILE = 'veos'
+TGEN_MGMT_NETWORK = '10.65.32.0/24'
 
 
 class TestbedVMFacts():
@@ -51,7 +52,10 @@ def __init__(self, toponame, vmbase, vmfile):
         CLET_SUFFIX = "-clet"
         toponame = re.sub(CLET_SUFFIX + "$", "", toponame)
         self.topofile = TOPO_PATH+'topo_'+toponame +'.yml'
-        self.start_index = int(re.findall('VM(\d+)', vmbase)[0])
+        if vmbase != '':
+            self.start_index = int(re.findall('VM(\d+)', vmbase)[0])
+        else:
+            self.start_index = 0
         self.vmhosts = {}
         self.vmfile = vmfile
         self.inv_mgr = InventoryManager(loader=DataLoader(), sources=self.vmfile)
@@ -85,9 +89,12 @@ def main():
     try:
         vmsall = TestbedVMFacts(m_args['topo'], m_args['base_vm'], m_args['vm_file'])
         neighbor_eos = vmsall.get_neighbor_eos()
-        for eos in neighbor_eos:
+        tgen_mgmt_ips = list(ipaddress.IPNetwork(unicode(TGEN_MGMT_NETWORK)))
+        for index, eos in enumerate(neighbor_eos):
             vmname = 'VM'+format(neighbor_eos[eos], '04d')
-            if vmname in vmsall.inv_mgr.hosts:
+            if 'tgen' in topo_type:
+                vmsall.vmhosts[eos] = str(tgen_mgmt_ips[index])
+            elif vmname in vmsall.inv_mgr.hosts:
                 vmsall.vmhosts[eos] = vmsall.inv_mgr.get_host(vmname).get_vars()['ansible_host']
             else:
                 err_msg = "cannot find the vm " + vmname + " in VM inventory file, please make sure you have enough VMs for the topology you are using"


Test patch: