Repository: sonic-net/sonic-mgmt

Issue:
RTNETLINK answers: File exists\n'. cmd=ip link add inje-vms6-1-31 type veth peer name ddd8c3eth31_t
<!--
If you are reporting a new issue, make sure that we do not have any duplicates
already open. You can ensure this by searching the issue list for this
repository. If there is a duplicate, please close your issue and add a comment
to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to
include the BUG REPORT INFORMATION shown below. If you fail to provide this
information within 7 days, we cannot debug your issue and will close it. We
will, however, reopen it if you later provide the information.

For more information about reporting issues, see
https://github.com/Azure/SONiC/wiki#report-issues

---------------------------------------------------
GENERAL SUPPORT INFORMATION
---------------------------------------------------

The GitHub issue tracker is for bug reports and feature requests.
General support can be found at the following locations:

- SONiC Support Forums - https://groups.google.com/forum/#!forum/sonicproject

---------------------------------------------------
BUG REPORT INFORMATION
---------------------------------------------------
Use the commands below to provide key information from your environment:
You do NOT have to include this information if this is a FEATURE REQUEST
-->

**Description**

<!--
Briefly describe the problem you are having in a few paragraphs.
-->

**Steps to reproduce the issue:**
Not sure how to repro. but here is what I see.

once system in such bad states, all subsequent operations will fail. Need to improve vm_topology to handle such bad states gracefully.

```
lgh@jenkins-worker-19:/tmp$ cat vmtopology.exception.2020-11-05T12\:09\:10.471468.txt
Traceback (most recent call last):
  File "/tmp/ansible_vm_topology_payload_0e5muxs5/__main__.py", line 988, in main
    net.add_injected_fp_ports_to_docker()
  File "/tmp/ansible_vm_topology_payload_0e5muxs5/__main__.py", line 290, in add_injected_fp_ports_to_docker
    self.add_veth_if_to_docker(ext_if, int_if)
  File "/tmp/ansible_vm_topology_payload_0e5muxs5/__main__.py", line 383, in add_veth_if_to_docker
    VMTopology.cmd("ip link add %s type veth peer name %s" % (ext_if, t_int_if))
  File "/tmp/ansible_vm_topology_payload_0e5muxs5/__main__.py", line 694, in cmd
    raise Exception("ret_code=%d, error message=%s. cmd=%s" % (ret_code, stderr, cmdline))
Exception: ret_code=2, error message=b'RTNETLINK answers: File exists\n'. cmd=ip link add inje-vms6-1-31 type veth peer name ddd8c3eth31_t
lgh@jenkins-worker-19:/tmp$ ip link show | grep inj
2786824: ddd8c3eth31_t@inj-vms6-1-31: <BROADCAST,MULTICAST> mtu 9216 qdisc noop state DOWN mode DEFAULT group default qlen 1000
2786825: inj-vms6-1-31@ddd8c3eth31_t: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 9216 qdisc noqueue master ovs-system state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
2786826: ddd8c3eth30_t@inj-vms6-1-30: <BROADCAST,MULTICAST> mtu 9216 qdisc noop state DOWN mode DEFAULT group default qlen 1000
2786827: inj-vms6-1-30@ddd8c3eth30_t: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 9216 qdisc noqueue master ovs-system state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
2786828: ddd8c3eth29_t@inj-vms6-1-29: <BROADCAST,MULTICAST> mtu 9216 qdisc noop state DOWN mode DEFAULT group default qlen 1000
2786829: inj-vms6-1-29@ddd8c3eth29_t: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 9216 qdisc noqueue master ovs-system state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
2786830: ddd8c3eth28_t@inj-vms6-1-28: <BROADCAST,MULTICAST> mtu 9216 qdisc noop state DOWN mode DEFAULT group default qlen 1000
2786831: inj-vms6-1-28@ddd8c3eth28_t: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 9216 qdisc noqueue master ovs-system state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
600: inje-vms6-3-31@if599: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
602: inje-vms6-3-30@if601: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
604: inje-vms6-3-29@if603: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
606: inje-vms6-3-28@if605: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master ovs-system state UP mode DEFAULT group default qlen 1000
```

**Describe the results you received:**


**Describe the results you expected:**


**Additional information you deem important:**
<!--
software/ASIC/Hardware Flatform version and info
-->
    **Output of `show version`:**

    ```
    (paste your output here)
    ```

    **Attach debug file `sudo generate_dump`:**

    ```
    (paste your output here)
    ```



Discussion:
Is any duplicate add topology job running at the same time? maybe the ethxx_t interfaces conflict or the previous jobs not remove topology? 


no, i do not see that. ddd8c3eth31_t@inj-vms6-1-31 is an intermediate results during adding the injected ports, therefore I suspect something crashed and left the system in such state. I think the issue here is that vm topology script is not able to recover the system from that bad state. 
@gord1306 , is this something you can help?
see this issue again. https://sonic-jenkins.westus2.cloudapp.azure.com/job/common/job/sonic-mgmt-testing/job/sonic-mgmt-pr/2014/consoleFull
looks like I found the reason, it is because of this pr. https://github.com/Azure/sonic-mgmt/pull/2459

Resolving patch:
diff --git a/ansible/roles/vm_set/library/vm_topology.py b/ansible/roles/vm_set/library/vm_topology.py
--- a/ansible/roles/vm_set/library/vm_topology.py
+++ b/ansible/roles/vm_set/library/vm_topology.py
@@ -377,6 +377,11 @@ def add_veth_if_to_docker(self, ext_if, int_if):
 
         t_int_if = hashlib.md5((PTF_NAME_TEMPLATE % self.vm_set_name).encode("utf-8")).hexdigest()[0:6] + int_if + '_t'
 
+        if t_int_if in self.host_ifaces:
+            VMTopology.cmd("ip link del dev %s" % t_int_if)
+
+        self.update()
+
         if ext_if not in self.host_ifaces:
             VMTopology.cmd("ip link add %s type veth peer name %s" % (ext_if, t_int_if))
 


Test patch: