Repository: sonic-net/sonic-mgmt

Issue:
fragmented packets from DUT are not forwarded to VM
<!--
If you are reporting a new issue, make sure that we do not have any duplicates
already open. You can ensure this by searching the issue list for this
repository. If there is a duplicate, please close your issue and add a comment
to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to
include the BUG REPORT INFORMATION shown below. If you fail to provide this
information within 7 days, we cannot debug your issue and will close it. We
will, however, reopen it if you later provide the information.

For more information about reporting issues, see
https://github.com/Azure/SONiC/wiki#report-issues

---------------------------------------------------
GENERAL SUPPORT INFORMATION
---------------------------------------------------

The GitHub issue tracker is for bug reports and feature requests.
General support can be found at the following locations:

- SONiC Support Forums - https://groups.google.com/forum/#!forum/sonicproject

---------------------------------------------------
BUG REPORT INFORMATION
---------------------------------------------------
Use the commands below to provide key information from your environment:
You do NOT have to include this information if this is a FEATURE REQUEST
-->

**Description**
PR https://github.com/Azure/sonic-mgmt/pull/5320 introduced more flow control rules for packets being forwarded to the VM's on the testbed server.  However, some of the added rules cause fragmented packets from the DUT to not be forwarded over the ovs-vsctl bridge to the VM.



<!--
Briefly describe the problem you are having in a few paragraphs.
-->

**Steps to reproduce the issue:**
1. In a T1/T2 topology, ping the IPv6 loopback address of one of the VM's to the other VM. In the example, below, we have VM1104 with loopback address ```2064:100::7``` and we are pinging the loopback address ```2064:100::9```.
```
ARISTA01T1>en
ARISTA01T1#bash

Arista Networks EOS shell

bash-4.2# sudo ping -c1 2064:100::9 -W 3 -I 2064:100::7 
PING 2064:100::9(2064:100::9) from 2064:100::7 : 56 data bytes
64 bytes from 2064:100::9: icmp_seq=1 ttl=63 time=1.06 ms

--- 2064:100::9 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.069/1.069/1.069/0.000 ms

```
2. Now repeat the ping, but with a packet size of 1456 (which would cause the packet to be fragmented):
```
bash-4.2# sudo ping -c1 2064:100::9 -W 3 -I 2064:100::7 -s 1453 -t 255
PING 2064:100::9(2064:100::9) from 2064:100::7 : 1453 data bytes

--- 2064:100::9 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

bash-4.2# 
```

On the testbed server, we have the following bridge for VM1105:
```
ansible@sonic-tb11:~$ sudo ovs-vsctl list-ports br-VM1105-0
VM1105-t0
eno2.102
inje-vms_11-8

```
tcpdump on the testbed server on the fanout port connected to VM1105 (eno2.100) when the ping fails shows:
```
ansible@sonic-tb11:~$ sudo tcpdump -i eno2.102
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno2.102, link-type EN10MB (Ethernet), capture size 262144 bytes
.
.
17:58:59.440251 IP6 2064:100::7 > 2064:100::9: frag (0|1448) ICMP6, echo request, seq 1, length 1448
17:58:59.440378 IP6 2064:100::7 > 2064:100::9: frag (1448|13)
```
We can see the two fragments on the fanout port eno2.100.  
tcpdump on the testbed server on the injected port connected to VM1105 (eno2.100) when the ping fails shows:
```
ansible@sonic-tb11:~$ sudo tcpdump -i inje-vms_11-8
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on inje-vms_11-8, link-type EN10MB (Ethernet), capture size 262144 bytes

18:02:12.872360 IP6 2064:100::7 > 2064:100::9: frag (0|1448) ICMP6, echo request, seq 1, length 1448
18:02:12.872487 IP6 2064:100::7 > 2064:100::9: frag (1448|13)

```
Again we see both fragments forwarded to the injected port.

However, tcpdump on port that maps to the VM1105 (VM1105-t0) does not see both the fragments. It only sees the first fragment:
```
ansible@sonic-tb11:~$ sudo tcpdump -i VM1105-t0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on VM1105-t0, link-type EN10MB (Ethernet), capture size 262144 bytes

17:59:43.164243 IP6 2064:100::7 > 2064:100::9: frag (0|1448) ICMP6, echo request, seq 1, length 1448

```

The packet is not forwarded to the VM because of the following added rules:

```
ansible@sonic-tb11:~$ sudo ovs-ofctl dump-flows br-VM1105-0
 .
 .
 cookie=0x0, duration=4348.314s, table=0, n_packets=0, n_bytes=0, priority=5,ip,in_port="eno2.102" actions=output:"inje-vms_11-8"
 cookie=0x0, duration=4348.310s, table=0, n_packets=13, n_bytes=980, priority=5,ipv6,in_port="eno2.102" actions=output:"inje-vms_11-8"

```

Removing these flows makes the ping with large packet size pass.

**Describe the results you received:**
Ping with large packet size between the VMs does not work

**Describe the results you expected:**
Ping with large packet size between the VMs should work

**Additional information you deem important:**
<!--
software/ASIC/Hardware Flatform version and info
-->
    **Output of `show version`:**

    ```
    (paste your output here)
    ```

    **Attach debug file `sudo generate_dump`:**

    ```
    (paste your output here)
    ```



Discussion:
@sanmalho-git Could you please investigate why and where packets are fragmented?
According to current openflow rules, fragmented ICMP6 packets can be matched to the last ip rule and forward to injected interface not to neighbor, it's why you ping 1453 size packet failed.
If the packets are not fragmented, icmp6 will success even with 7000 size. Because MTU of DUT and EOS is 9100 and 9214 bytes. 
```
admin@str2-7050qx-32s-acs-01:~$ ping fc00::2 -s 7000
PING fc00::2(fc00::2) 7000 data bytes
7008 bytes from fc00::2: icmp_seq=1 ttl=64 time=2.48 ms
7008 bytes from fc00::2: icmp_seq=2 ttl=64 time=2.20 ms
```


```
Interface PortChannel102 is up, line protocol is up
  Link ups:       3    last: 2022/04/06 10:27:59.52
  Link downs:    13    last: 2022/04/06 10:27:59.52
  vrf: default
  index 9 metric 0 mtu 9100 speed 0 
  flags: <UP,BROADCAST,RUNNING,MULTICAST>
  Type: Unknown
  HWaddr: 00:1c:73:ce:6b:18
  inet 10.0.0.0/31
  inet6 fc00::1/126
  inet6 fe80::21c:73ff:fece:6b18/64
  Interface Type Other

ARISTA01T2#show interface
Ethernet1 is up, line protocol is up (connected)
  Hardware is Ethernet, address is 0ef8.b691.e058 (bia 0ef8.b691.e058)
  Member of Port-Channel1
  Ethernet MTU 9214 bytes
  Full-duplex, Unconfigured, auto negotiation: off, uni-link: n/a
  Up 21 hours, 30 minutes, 2 seconds
  Loopback Mode : None
  62 link status changes since last clear
  Last clearing of "show interface" counters never
  5 minutes input rate 0 bps (- with framing overhead), 0 packets/sec
  5 minutes output rate 0 bps (- with framing overhead), 0 packets/sec
     373736 packets input, 76184557 bytes
     Received 480 broadcasts, 112170 multicast
     0 runts, 0 giants
     0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards
     0 PAUSE input
     458883 packets output, 478036044 bytes
     Sent 1326 broadcasts, 3635 multicast
     0 output errors, 0 collisions
     0 late collision, 0 deferred, 0 output discards
     0 PAUSE output

```
@ZhaohuiS - the issue we are seeing is with ceos dockers(add-topo w/ -k ceos). Not sure if you are using voes KVMs. What I see in the ceos dockers is that the interfaces that have IP defined have mtu of 1500, but other interfaces are set to 9214. 

Below is the output of 'show interfaces' on one of the ceos dockers which has a PortChannel on Ethernet1 and Ethernet2

```
ARISTA01T3#show interfaces 
Ethernet1 is up, line protocol is up (connected)
  Hardware is Ethernet, address is fa9c.8af8.bf6c (bia fa9c.8af8.bf6c)
  Member of Port-Channel1
  Ethernet MTU 9214 bytes
  Full-duplex, Unconfigured, auto negotiation: off, uni-link: n/a
.
.
Ethernet2 is up, line protocol is up (connected)
  Hardware is Ethernet, address is 1610.8f20.a111 (bia 1610.8f20.a111)
  Member of Port-Channel1
  Ethernet MTU 9214 bytes
  Full-duplex, Unconfigured, auto negotiation: off, uni-link: n/a
.
.
Ethernet3 is up, line protocol is up (connected)
  Hardware is Ethernet, address is b269.881e.efd6 (bia b269.881e.efd6)
  Ethernet MTU 9214 bytes
  Full-duplex, Unconfigured, auto negotiation: off, uni-link: n/a
  Up 12 hours, 15 minutes, 17 seconds
  Loopback Mode : None
.
.
Port-Channel1 is up, line protocol is up (connected)
  Hardware is Port-Channel, address is 1610.8f2f.b124
  Internet address is 10.0.0.1/31
  Broadcast address is 255.255.255.255
  IPv6 link-local address is fe80::1410:8fff:fe2f:b124/64
  IPv6 global unicast address(es):
    fc00::2, subnet is fc00::/126
  IP MTU 1500 bytes (default)
  Full-duplex, Unconfigured 
```

Here is the output of show interfaces where we have no PortChannels, just IP on Ethernet1:
```
ARISTA03T3#show interfaces 
Ethernet1 is up, line protocol is up (connected)
  Hardware is Ethernet, address is 36da.b7f7.00fb (bia 36da.b7f7.00fb)
  Internet address is 10.0.0.5/31
  Broadcast address is 255.255.255.255
  IPv6 link-local address is fe80::34da:b7ff:fef7:fb/64
  IPv6 global unicast address(es):
    fc00::a, subnet is fc00::8/126
  IP MTU 1500 bytes (default)
  Full-duplex, Unconfigured, auto negotiation: off, uni-link: n/a
.
.
Ethernet2 is up, line protocol is up (connected)
  Hardware is Ethernet, address is 36e2.4fc4.c0f0 (bia 36e2.4fc4.c0f0)
  Ethernet MTU 9214 bytes
  Full-duplex, Unconfigured, auto negotiation: off, uni-link: n/a
```
We are testing with the latest version of ceos docker

```
ARISTA03T3#show version 
Arista cEOSLab
Hardware version: 
Serial number: 883E4A3C35E577353485FC720B185DEA
Hardware MAC address: 425c.2771.2898
System MAC address: 425c.2771.2898

Software image version: 4.27.3F-26391011.4273F (engineering build)
Architecture: x86_64
Internal build version: 4.27.3F-26391011.4273F
Internal build ID: 154beb0c-0b2a-45e6-9076-e1b53a8207f1
Image format version: 1.0
Image optimization: None

cEOS tools version: 1.1
Kernel version: 5.4.0-105-generic

Uptime: 2 minutes
Total memory: 196691280 kB
Free memory: 181301556 kB

```

Resolving patch:
diff --git a/ansible/roles/vm_set/library/vm_topology.py b/ansible/roles/vm_set/library/vm_topology.py
--- a/ansible/roles/vm_set/library/vm_topology.py
+++ b/ansible/roles/vm_set/library/vm_topology.py
@@ -790,7 +790,7 @@ def bind_ovs_ports(self, br_name, dut_iface, injected_iface, vm_iface, disconnec
             VMTopology.cmd("ovs-ofctl add-flow %s table=0,in_port=%s,action=output:%s" % (br_name, dut_iface_id, injected_iface_id))
         else:
             # Add flow from external iface to a VM and a ptf container
-            # Allow BGP, IPinIP, ICMP, SNMP packets and layer2 packets from DUT to neighbors
+            # Allow BGP, IPinIP, fragmented packets, ICMP, SNMP packets and layer2 packets from DUT to neighbors
             # Block other traffic from DUT to EOS for EOS's stability,
             # Allow all traffic from DUT to PTF.
             VMTopology.cmd("ovs-ofctl add-flow %s table=0,priority=10,tcp,in_port=%s,tp_src=179,action=output:%s,%s" % (br_name, dut_iface_id, vm_iface_id, injected_iface_id))
@@ -798,6 +798,8 @@ def bind_ovs_ports(self, br_name, dut_iface, injected_iface, vm_iface, disconnec
             VMTopology.cmd("ovs-ofctl add-flow %s table=0,priority=10,tcp6,in_port=%s,tp_src=179,action=output:%s,%s" % (br_name, dut_iface_id, vm_iface_id, injected_iface_id))
             VMTopology.cmd("ovs-ofctl add-flow %s table=0,priority=10,tcp6,in_port=%s,tp_dst=179,action=output:%s,%s" % (br_name, dut_iface_id, vm_iface_id, injected_iface_id))
             VMTopology.cmd("ovs-ofctl add-flow %s table=0,priority=10,ip,in_port=%s,nw_proto=4,action=output:%s,%s" % (br_name, dut_iface_id, vm_iface_id, injected_iface_id))
+            VMTopology.cmd("ovs-ofctl add-flow %s table=0,priority=8,ip,in_port=%s,nw_frag=yes,action=output:%s,%s" % (br_name, dut_iface_id, vm_iface_id, injected_iface_id))
+            VMTopology.cmd("ovs-ofctl add-flow %s table=0,priority=8,ipv6,in_port=%s,nw_frag=yes,action=output:%s,%s" % (br_name, dut_iface_id, vm_iface_id, injected_iface_id))
             VMTopology.cmd("ovs-ofctl add-flow %s table=0,priority=8,icmp,in_port=%s,action=output:%s,%s" % (br_name, dut_iface_id, vm_iface_id, injected_iface_id))
             VMTopology.cmd("ovs-ofctl add-flow %s table=0,priority=8,icmp6,in_port=%s,action=output:%s,%s" % (br_name, dut_iface_id, vm_iface_id, injected_iface_id))
             VMTopology.cmd("ovs-ofctl add-flow %s table=0,priority=8,udp,in_port=%s,udp_src=161,action=output:%s,%s" % (br_name, dut_iface_id, vm_iface_id, injected_iface_id))


Test patch: