Project: Inspektor Gadget (Observability and Analysis / Observability)
File: tcprtt.md

title: 'Using profile tcprtt' weight: 20 description: > Analyze TCP connections through an Round-Trip Time (RTT) distribution The profile tcprtt gadget generates a histogram distribution of the TCP connections' Round-Trip Time (RTT). The RTT values used to create the histogram are collected from [the smoothed RTT](https://elixir.bootlin.com/linux/v5.11.22/source/include/linux/tcp.h#L258) information already provided by the Linux kernel for the TCP sockets. The histogram considers only the TCP connections that have been already established, so it does not take into account the connection phase (3-way TCP Handshake). If it is what you are looking for, please check the latency information the trace tcpconnect gadget provides. See further information . By default, the profile tcprtt gadget generates one single histogram per host, and one per node in case of Kubernetes. However, it also provides multiple ways to analyze specific connections. For instance, we can generate multiple histograms separated by local IP addresses to explore all the connections generated by each local address. And the same can be done for remote IP addresses. In addition, it is also possible to filter by a specific local and/or remote address to isolate the analysis. First of all, let's start the gadget on a terminal: ```bash kubectl gadget profile tcprtt ``` In another terminal, create a server using nginx: ```bash kubectl create service nodeport nginx --tcp=80:80 kubectl create deployment nginx --image=nginx ``` And then, create a pod to generate some traffic with the server: ```bash $ kubectl run -ti --privileged --image wbitt/network-multitool myclientpod -- bash ``` If we move back to the first terminal and stop the gadget, it will generate the histograms: ```bash All Addresses = [AVG 1211.066824] s : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 114 |* | 16 -> 31 : 397 || 32 -> 63 : 182 | | 64 -> 127 : 49 | | 128 -> 255 : 48 | | 256 -> 511 : 107 | | 512 -> 1023 : 108 | | 1024 -> 2047 : 86 | | 2048 -> 4095 : 31 |* | 4096 -> 8191 : 111 |* | 8192 -> 16383 : 28 | | 16384 -> 32767 : 11 |* | ``` Take into account that the generated histogram considers all TCP connections for each node, not only the ones we established. Notice that we used a cluster with a single node for this

So, let's repeat the test but this time filtering by remote address so that we can analyse the traffic we are generating toward our nginx service: ```bash $ kubectl get service nginx -o jsonpath={.spec.clusterIP} 10.0.38.234 $ kubectl gadget profile tcprtt --raddr 10.0.38.234 All Addresses = [AVG 2087.000000] s : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 3 || 2048 -> 4095 : 3 || ``` Now, let's use the [network emulator](https://wiki.linuxfoundation.org/networking/netem) to introduce some random delay to the packets and increase indirectly the RTT: ```bash ``` Now the average RTT value of the new histogram is clearly higher: ```bash $ kubectl gadget profile tcprtt --raddr 10.0.38.234 All Addresses = [AVG 68973.833333] s : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 0 | | 8192 -> 16383 : 0 | | 16384 -> 32767 : 0 | | 32768 -> 65535 : 3 || 65536 -> 131071 : 3 || ``` Start the profile tcprtt gadget on a first terminal: ```bash sudo ig profile tcprtt ``` Then, start a container and download a web page: ```bash $ docker run -ti --rm --cap-add NET_ADMIN --name=netem wbitt/network-multitool -- /bin/bash ``` Moving back to the first terminal and stopping the gadget, it will generate the histograms: ```bash All Addresses = [AVG

s : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 25 | | 8 -> 15 : 226 | | 16 -> 31 : 777 | | 32 -> 63 : 1532 | | 64 -> 127 : 2822 |* | 128 -> 255 : 2254 | | 256 -> 511 : 3305 | | 512 -> 1023 : 2863 |* | 1024 -> 2047 : 1284 | | 2048 -> 4095 : 1456 | | 4096 -> 8191 : 3612 || 8192 -> 16383 : 167 |* | 16384 -> 32767 : 0 | | 32768 -> 65535 : 14 | | 65536 -> 131071 : 75 | | 131072 -> 262143 : 0 | | 262144 -> 524287 : 0 | | 524288 -> 1048575 : 8 | | ``` This histogram represents the distribution of the RTT for all the TCP connections established in the host and not only the TCP connections we established in the test container. Let's repeat the test but this time filtering by remote address so that we can analyse the traffic we are generating: ```bash $ sudo ig profile tcprtt --raddr 1.1.1.1 All Addresses = [AVG 7359.357143] s : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 9 || 8192 -> 16383 : 5 | | ``` Now, let's introduce some random delay to the packets to increase indirectly the RTT using the [network emulator](https://wiki.linuxfoundation.org/networking/netem): ```bash ``` And, regenerate the histogram to see the change in the RTT: ```bash $ sudo ig profile tcprtt --raddr 1.1.1.1 All Addresses = [AVG 72278.307692] s : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 0 | | 32 -> 63 : 0 | | 64 -> 127 : 0 | | 128 -> 255 : 0 | | 256 -> 511 : 0 | | 512 -> 1023 : 0 | | 1024 -> 2047 : 0 | | 2048 -> 4095 : 0 | | 4096 -> 8191 : 0 | | 8192 -> 16383 : 0 | | 16384 -> 32767 : 0 | | 32768 -> 65535 : 0 | | 65536 -> 131071 : 13 || ``` We can see how the average RTT passed from 7359.357143 to 72278.307692.