Project: EaseAgent (Observability and Analysis / Observability)
File: benchmark.md

We use as the benchmark application because it is relatively close to the real user scenario. We use three AWS EC2 machines in a subnet for the benchmark test: Gateway EC2: deploy `api-gateway`, `config-service`, and `discovery-server` on this node. Service EC2: deploy `customers-service`, `vets-service`, and `visits-service` on this node. Tester EC2: we initiate testing requests and collect testing metric data on this node. All three AWS EC2 machines have the exact same configuration, and in a subnet: ``` EC2 Instance Type: t3-large, cn-northwest-1c CPU: 2 cores, Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz MEM: 7850092 kB SSD: XFS 100G OS: CentOS Linux release 8.5.2111 Docker version 20.10.14, build a224086 docker-compose version 1.27.4, build 40524192 ``` On Gateway EC2 node: ``` env COMPOSER=gateway-compose EASEAGENT="" ./benchmark.sh start env COMPOSER=gateway-compose EASEAGENT="true" ./benchmark.sh start ``` On Service EC2 node: ``` env COMPOSER=service-compose EASEAGENT="" ./benchmark.sh start env COMPOSER=service-compose EASEAGENT="true" ./benchmark.sh start ``` On Tester EC2 node, we start Grafana/Prometheus: ``` env COMPOSER=benchmark-tester-compose EASEAGENT="" PROMETHEUSCONFIGFILE=prometheusbenchmark2.yaml ./benchmark.sh start ``` We run `k6` on `Tester` node to start stress test, and collect the metric data. ``` k6 run vets.js ``` Environment details of the test, reference to the document in . We access the URI: `/api/vet/vets` of the application, it will generate 10-Spans in total, and distributed in two serivces: jmx-api-gateway: 1 span; jmx-vets: 9spans. By comparing the results of the two services, the impact of the number of SPANs can be observed. For each value pair in the table, the former is the baseline value and the latter is the agent value (baseline / agent / difference value). | URI-Label | CPU (baseline/agent/diff) | Heap Memory | P90 Latency | Throughput (15s) | |-|:--|:|:-|--| | jmx-api-gateway: 1spans | 33.2% / 44.4% / 21.2% | 208M / 253M / 45M | | | | jmx-vets: 9spans | 40.2% / 65.2% / 25.0% | 236M / 284M / 48M | 28.94 ms / 189.39 ms / 553% | 1164 / 1057 / -9.2% | | Label | jmx-api-gateway | jmx-vets | |--|:|:-| | Baseline | 33.2% | 40.2% | | Agent | 44.4% | 65.2% | | Diff | 21.2% | 25.0% | Baseline: Agent: Baseline: Agent: | Label | P90 Latency (ms) | Throughput (reqs/s) | |--|:|:-| | Baseline | 28.94 | 1164 | | Agent | 189.39 | 1057 | Baseline ``` running (11m40.4s), 000/320 VUs, 203885 complete and 0 interrupted iterations default [======================================] 000/320 VUs 11m40s data_received..................: 637 MB 910 kB/s data_sent......................: 78 MB 111 kB/s httpreqblocked...............: avg=3.98s min=687ns med=1.84s max=13.63ms p(90)=3.44s p(95)=4.86s httpreqconnecting............: avg=741ns min=0s med=0s max=13.58ms p(90)=0s p(95)=0s httpreqduration..............: avg=14.2ms min=1.1ms med=7.83ms max=475.17ms p(90)=28.94ms p(95)=45.38ms { expected_response:true }...: avg=14.2ms min=1.1ms med=7.83ms max=475.17ms p(90)=28.94ms p(95)=45.38ms httpreqfailed................: 0.00% 0 815540 httpreqreceiving.............: avg=285.63s min=8.6s med=73.14s max=89.64ms p(90)=462.23s p(95)=889.81s httpreqsending...............: avg=18.85s min=3.83s med=8.29s max=54.16ms p(90)=16.14s p(95)=22.15s httpreqtls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s httpreqwaiting...............: avg=13.9ms min=1.04ms med=7.57ms max=475.02ms p(90)=28.36ms p(95)=44.63ms http_reqs......................: 815540 1164.420944/s iteration_duration.............: avg=1.02s min=1s med=1.01s max=1.47s p(90)=1.04s p(95)=1.06s iterations.....................: 203885 291.105236/s vus............................: 5 min=5 max=320 vus_max........................: 320 min=320 max=320 ``` Agent ``` running

000/320 VUs, 185219 complete and 0 interrupted iterations default [======================================] 005/320 VUs 11m40s data_received..................: 579 MB 827 kB/s data_sent......................: 70 MB 100 kB/s httpreqblocked...............: avg=4.45s min=700ns med=1.9s max=71.05ms p(90)=3.47s p(95)=4.51s httpreqconnecting............: avg=1.39s min=0s med=0s max=70.99ms p(90)=0s p(95)=0s httpreqduration..............: avg=75.71ms min=1.52ms med=38.65ms max=1.18s p(90)=189.39ms p(95)=275.46ms { expected_response:true }...: avg=75.71ms min=1.52ms med=38.65ms max=1.18s p(90)=189.39ms p(95)=275.46ms httpreqfailed................: 0.00% 0 740876 httpreqreceiving.............: avg=1.37ms min=9.07s med=54.95s max=333.25ms p(90)=1.44ms p(95)=6.48ms httpreqsending...............: avg=49.77s min=3.85s med=8.36s max=228.08ms p(90)=16.97s p(95)=23.34s httpreqtls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s httpreqwaiting...............: avg=74.29ms min=1.47ms med=37.3ms max=1.18s p(90)=186.62ms p(95)=273.55ms http_reqs......................: 740876 1057.248498/s iteration_duration.............: avg=1.12s min=1s med=1.07s max=2.18s p(90)=1.3s p(95)=1.39s iterations.....................: 185219 264.312124/s vus............................: 5 min=5 max=320 vus_max........................: 320 min=320 max=320 ``` | URI-Label | CPU (baseline/agent/diff) | Heap Memory | P90 Latency | Throughput (15s) | |-|:--|:-|:-|--| | jmx-api-gateway: 1spans | 57.5% / 61.1% / 3.6% | 385M / 583M / 200M | | | | jmx-vets: 9spans | 69.0% / 81.2% / 12.2% | 303M / 464M / 151M | 564.69 ms / 737.55 ms / 30% | 1800 / 1526 / -15.2% | Process CPU Load: 3.6% - more than 10%, requiring optimisation, the number of span has a significant impact on CPU performance. Heap Memory: requiring optimisation. Latency & Throughput: Easeagent has an excessive impact on latency and throughput and is positively correlated with the number of Spans. CPU, Process CPU Load, collected through JMX:javalangOperatingSystem_ProcessCpuLoad. | Label | jmx-api-gateway | jmx-vets | |--|:|:-| | Baseline | 57.5% | 69.0% | | Agent | 61.1% | 81.2% | | Diff | 3.60% | 12.2% | Baseline: Agent: Baseline: Agent: | Label | P90 Latency (ms) | Throughput (reqs/s) | |--|:|:-| | Baseline | 564.69 | 1800 | | Agent | 737.55 | 1526 | Baseline: ``` running (13m20.6s), 000/700 VUs, 360354 complete and 0 interrupted iterations default [======================================] 009/700 VUs 13m20s data_received..................: 1.1 GB 1.4 MB/s data_sent......................: 137 MB 171 kB/s httpreqblocked...............: avg=6.6s min=704ns med=1.88s max=269.76ms p(90)=3.35s p(95)=5.16s httpreqconnecting............: avg=3.22s min=0s med=0s max=254.77ms p(90)=0s p(95)=0s httpreqduration..............: avg=262.62ms min=1.19ms med=219.42ms max=1.58s p(90)=564.69ms p(95)=669.43ms { expected_response:true }...: avg=262.62ms min=1.19ms med=219.42ms max=1.58s p(90)=564.69ms p(95)=669.43ms httpreqfailed................: 0.00% 0 1441416 httpreqreceiving.............: avg=7.48ms min=8.94s med=517.01s max=480.89ms p(90)=17.64ms p(95)=38.22ms httpreqsending...............: avg=105.42s min=3.72s med=8.55s max=236.91ms p(90)=20.64s p(95)=37.64s httpreqtls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s httpreqwaiting...............: avg=255.03ms min=1.12ms med=208.8ms max=1.58s p(90)=554.48ms p(95)=661.33ms http_reqs......................: 1441416 1800.434166/s iteration_duration.............: avg=1.37s min=1s med=1.36s max=2.69s p(90)=1.74s p(95)=1.86s iterations.....................: 360354 450.108541/s vus............................: 9 min=5 max=700 vus_max........................: 700 min=700 max=700 ``` Agent ``` running (13m20.9s), 000/700 VUs, 305670 complete and 0 interrupted iterations default [======================================] 000/700 VUs 13m20s data_received..................: 956 MB 1.2 MB/s data_sent......................: 116 MB 145 kB/s httpreqblocked...............: avg=11.59s min=688ns med=1.94s max=399.48ms p(90)=3.48s p(95)=4.62s httpreqconnecting............: avg=6.21s min=0s med=0s max=217.13ms p(90)=0s p(95)=0s httpreqduration..............: avg=360.83ms min=1.45ms med=252.27ms max=7.99s p(90)=737.55ms p(95)=1.04s { expected_response:true }...: avg=360.83ms min=1.45ms med=252.27ms max=7.99s p(90)=737.55ms p(95)=1.04s httpreqfailed................: 0.00% 0 1222680 httpreqreceiving.............: avg=4.88ms min=9.14s med=51.6s max=887.76ms p(90)=10.7ms p(95)=28.04ms httpreqsending...............: avg=193.63s min=3.72s med=8.54s max=756.82ms p(90)=19.57s p(95)=44.94s httpreqtls_handshaking.......: avg=0s min=0s med=0s max=0s p(90)=0s p(95)=0s httpreqwaiting...............: avg=355.75ms min=1.36ms med=248.41ms max=7.99s p(90)=727.94ms p(95)=1.03s http_reqs......................: 1222680 1526.574647/s iteration_duration.............: avg=1.62s min=1s med=1.41s max=8.99s p(90)=2.36s p(95)=2.76s iterations.....................: 305670 381.643662/s vus............................: 6 min=5 max=700 vus_max........................: 700 min=700 max=700 ``` Optimization is required in terms of CPU efficiency/memory usage and latency throughput.