Project: Inspektor Gadget (Observability and Analysis / Observability)
File: tcpdrop.md

title: 'Using trace tcpdrop' weight: 20 description: > Trace TCP kernel-dropped packets/segments. The trace tcpdrop gadget traces TCP packets dropped by the kernel. In terminal 1, start the trace tcpdrop gadget: ```bash $ kubectl gadget trace tcpdrop K8S.NODE K8S.NAMESPACE K8S.POD K8s.CONTAINER PID COMM IP SRC DST STATE TCPFLAGS REASON ``` In terminal 2, start a pod and configure the network emulator to drop 25% of the packets: ```bash $ kubectl create service nodeport nginx --tcp=80:80 $ kubectl create deployment nginx --image=nginx $ kubectl run --rm -ti --privileged --image ubuntu shell -- bash root@shell:/# apt-get update root@shell:/# apt install -y iproute2 curl root@shell:/# tc qdisc add dev eth0 root netem drop 25% root@shell:/# curl nginx ``` The results in terminal 1 will show that some packets are dropped by the network emulator qdisc: ``` K8S.NODE K8S.NAMESPACE K8S.POD K8s.CONTAINER PID COMM IP SRC DST STATE TCPFLAGS REASON minikube-docker default shell shell 0 4 p/default/shell:45979 s/kube-system/kube-dns:53 ESTABLISHED FIN QDISC_DROP minikube-docker default shell shell 406293 curl 4 p/default/shell:34482 s/default/nginx:80 ESTABLISHED ACK QDISC_DROP ``` The network emulator uses a random generator to drop 25% of the packets. The results may vary. The gadget tries its best to link the dropped packets to the process which generated it. In some cases, this information might be missing. The source and destination addresses are written in condensed form. It is possible to see more detailed information by reading specific columns or using the json or yaml ouput: ``` $ kubectl gadget trace tcpdrop \ -o columns=k8s.node,k8s.namespace,k8s.pod,k8s.container,pid,comm,ip,src.addr,src.port,src.kind,src.ns,src.name,dst.addr,dst.port,dst.kind,dst.ns,dst.name,state,tcpflags,reason ``` ``` $ kubectl gadget trace tcpdrop -o yaml comm: curl container: shell dst: addr: 10.101.116.61 kind: svc namespace: default podlabels: app: nginx podname: nginx port: 80 gid: 0 ipversion: 4 mountnsid: 4026533845 namespace: default netnsid: 4026533672 node: minikube-docker pid: 412491 pod: shell reason: QDISC_DROP src: addr: 10.244.0.91 kind: pod namespace: default podlabels: run: shell podname: shell port: 35802 state: ESTABLISHED tcpflags: ACK timestamp: 1681911565379499967 type: normal uid: 0 ``` In terminal 1, start the trace tcpdrop gadget: ```bash $ sudo ig trace tcpdrop -r docker CONTAINER PID COMM IP SRC DST STATE TCPFLAGS REASON ``` In terminal 2, start a container, configure the network emulator to drop 25% of the packets, and download a web page: ```bash $ docker run -ti --rm --cap-add NET_ADMIN --name=netem wbitt/network-multitool -- /bin/bash ``` The container needs NET_ADMIN capability to manage network interfaces The results in terminal 1 will show that some packets are dropped by the network emulator qdisc: ``` CONTAINER PID COMM IP SRC DST STATE TCPFLAGS REASON netem 456426 wget 4 172.17.0.2:35790

ESTABLISHED ACK QDISC_DROP ``` The following section tells us that QDISC_DROP means the packet was "dropped by qdisc when packet outputting (failed to enqueue to current qdisc)". The drop reason enum is not stable and may change between kernel versions. The tcpdrop gadget needs BTF information to decode the drop reason. The following table shows the list of drop reasons for Linux 6.2. <!-- markdown-link-check-disable --> | Name | Documentation | ||| | | skb is not dropped yet (used for no-drop case) | | | packet has been consumed | | | drop reason is not specified | | | socket not found | | | packet size is too small | | | TCP checksum error | | | dropped by socket filter | | | UDP checksum error | | | dropped by netfilter | | | packet don't belong to current host (interface is in promisc mode) | | | IP checksum error | | | there is something wrong with IP header (see IPSTATSMIBINHDRERRORS) | | | IP rpfilter validate failed. see the document for rp_filter in ip-sysctl.rst for more information | | | destination address of L2 is multicast, but L3 is unicast. | | | xfrm policy check failed | | | no support for IP protocol | | | socket receive buff is full | | | proto memory limition, such as udp packet drop out of

| | | no MD5 hash and one expected, corresponding to LINUXMIBTCPMD5NOTFOUND | | | MD5 hash and we're not expecting one, corresponding to LINUXMIBTCPMD5UNEXPECTED | | | MD5 hash and its wrong, corresponding to LINUXMIBTCPMD5FAILURE | | | failed to add skb to socket backlog ( see LINUXMIBTCPBACKLOGDROP) | | | TCP flags invalid | | | TCP receive window size is zero, see LINUXMIBTCPZEROWINDOWDROP | | | the TCP data reveived is already received before (spurious retrans may happened), see LINUXMIBDELAYEDACKLOST | | | the TCP data is out of window, the seq of the first byte exceed the right edges of receive window | | | the data of skb is already in the ofo queue, corresponding to LINUXMIBTCPOFOMERGE | | | PAWS check, corresponding to LINUXMIBPAWSESTABREJECTED | | | Not acceptable SEQ field | | | Invalid RST packet | | | Incoming packet has unexpected SYN flag | | | TCP socket in CLOSE state | | | dropped by FASTOPEN request socket | | | TCP ACK is old, but in window | | | TCP ACK is too old | | | TCP ACK for data we haven't sent yet | | | pruned from TCP OFO queue | | | data already in receive queue | | | route lookup failed | | | dropped by BPFPROGTYPECGROUPSKB eBPF program | | | IPv6 is disabled on the device | | | failed to create neigh entry | | | neigh entry in failed state | | | arp_queue for neigh entry is full | | | neigh entry is dead | | | dropped in TC egress HOOK | | | dropped by qdisc when packet outputting ( failed to enqueue to current qdisc) | | | failed to enqueue the skb to the per CPU backlog queue. This can be caused by backlog queue full (see netdevmaxbacklog in net.rst) or RPS flow limit | | | dropped by XDP in input path | | | dropped in TC ingress HOOK | | | protocol not implemented or not supported | | | sk_buff checksum computation error | | | gso segmentation error | | | failed to copy data from user space, e.g., via zerocopysgfromiter() or skborphanfragsrx() | | | device driver specific header/metadata is invalid | | | the device is not ready to xmit/recv due to any of its data structure that is not up/ready/initialized, e.g., the IFF_UP is not set, or driver specific tun->tfiles[txq] is not initialized | | | ring buffer is full | | | error due to OOM | | | failed to trunc/extract the header from networking data, e.g., failed to pull the protocol header from frags via pskbmaypull() | | | dropped by (ebpf) filter directly attached to tun/tap, e.g., via TUNSETFILTEREBPF | | | dropped by tx filter implemented at tun/tap, e.g., check_filter() | | | ICMP checksum error | | | the packet doesn't follow RFC 2211, such as a broadcasts ICMP_TIMESTAMP | | | host unreachable, corresponding to IPSTATSMIBINADDRERRORS | | | network unreachable, corresponding to IPSTATSMIBINADDRERRORS | | | packet size is too big (maybe exceed the MTU) | | | duplicate fragment | | | fragment reassembly timeout | | | ipv4 fragment too far. (/proc/sys/net/ipv4/ipfragmaxdist) | | | the maximum of drop reason, which shouldn't be used as a real 'reason' | <!-- markdown-link-check-enable --> This table can be generated with: ```bash $ go run ./pkg/gadgets/trace/tcpdrop/tracer/dropreasongen/... ``` The following tools can be used to show dropped packets but they are not focused on containers or Kubernetes: :