Project: YugabyteDB (App Definition and Development / Database)
File: performance-troubleshooting.md

title: Troubleshooting performance issues in YugabyteDB headerTitle: Performance Troubleshooting linkTitle: Performance Troubleshooting description: Common steps to take when investigating performance issues in YugabyteDB headcontent: Performance Troubleshooting menu: v2.14: identifier: performance-troubleshooting parent: benchmark weight: 22 type: docs Use this page for general guidance and steps you can take when troubleshooting the performance of your Yugabyte cluster. Note that these locations refer to the default for clusters installed via YugabyteDB Anywhere. The software packages are symlinked at /home/yugabyte/{master|tserver}. Be aware that the master & tserver may be at different versions of the software (e.g. during rolling software upgrades). You should see something like this: ```output $ ls -lrt /home/yugabyte/master total 4 lrwxrwxrwx. 1 yugabyte yugabyte 27 Jan 15 19:27 logs -> /mnt/d0/yb-data/master/logs lrwxrwxrwx. 1 yugabyte yugabyte 66 Jan 15 19:28 bin -> /home/yugabyte/yb-software/yugabyte-2.5.1.0-b187-centos-x86_64/bin lrwxrwxrwx. 1 yugabyte yugabyte 66 Jan 15 19:28 lib -> /home/yugabyte/yb-software/yugabyte-2.5.1.0-b187-centos-x86_64/lib lrwxrwxrwx. 1 yugabyte yugabyte 72 Jan 15 19:28 linuxbrew -> /home/yugabyte/yb-software/yugabyte-2.5.1.0-b187-centos-x86_64/linuxbrew lrwxrwxrwx. 1 yugabyte yugabyte 85 Jan 15 19:28 linuxbrew-xxxxxxxxxxxx -> /home/yugabyte/yb-software/yugabyte-2.5.1.0-b187-centos-x86_64/linuxbrew-xxxxxxxxxxxx lrwxrwxrwx. 1 yugabyte yugabyte 71 Jan 15 19:28 postgres -> /home/yugabyte/yb-software/yugabyte-2.5.1.0-b187-centos-x86_64/postgres lrwxrwxrwx. 1 yugabyte yugabyte 68 Jan 15 19:28 pylib -> /home/yugabyte/yb-software/yugabyte-2.5.1.0-b187-centos-x86_64/pylib lrwxrwxrwx. 1 yugabyte yugabyte 68 Jan 15 19:28 share -> /home/yugabyte/yb-software/yugabyte-2.5.1.0-b187-centos-x86_64/share lrwxrwxrwx. 1 yugabyte yugabyte 68 Jan 15 19:28 tools -> /home/yugabyte/yb-software/yugabyte-2.5.1.0-b187-centos-x86_64/tools lrwxrwxrwx. 1 yugabyte yugabyte 65 Jan 15 19:28 ui -> /home/yugabyte/yb-software/yugabyte-2.5.1.0-b187-centos-x86_64/ui lrwxrwxrwx. 1 yugabyte yugabyte 84 Jan 15 19:28 versionmetadata.json -> /home/yugabyte/yb-software/yugabyte-2.5.1.0-b187-centos-x8664/version_metadata.json lrwxrwxrwx. 1 yugabyte yugabyte 66 Jan 15 19:28 www -> /home/yugabyte/yb-software/yugabyte-2.5.1.0-b187-centos-x86_64/www -rw-rw-r--. 1 yugabyte yugabyte 0 Jan 15 19:29 master.out -rw-rw-r--. 1 yugabyte yugabyte 2200 Jan 15 20:18 master.err drwxr-xr-x. 2 yugabyte yugabyte 25 Jan 15 20:18 conf ``` You can find config files for the master and tserver at `/home/yugabyte/{master|tserver}/conf/server.conf`. Assuming Yugabyte was run with `--fsdatadirs=/mnt/d0,/mnt/d1`, you can find WAL files at `/mnt/d*/yb-data/{master|tserver}/wals`. To pretty print the contents of the WAL files, use the `log-dump` utility: ```output $ pwd /home/yugabyte $ ./tserver/bin/log-dump /mnt/d0/yb-data/tserver/wals/table-e85a116bc557403e82f57037e7b13879/tablet-05bef5ed6fb74cabb420b648b6f850e3/ ``` You can find SSTable files at `/mnt/d*/yb-data/{master|tserver}/data`. To pretty print the contents of these SSTable files: ```output $ pwd /home/yugabyte $ ./tserver/bin/ldb dump --compression_type=snappy --db=/mnt/d0/yb-data/tserver/wals/table-e85a116bc557403e82f57037e7b13879/tablet-05bef5ed6fb74cabb420b648b6f850e3/ ``` Debug logs are output to `/home/yugabyte/{master|tserver}/logs`. These go to e.g. `/home/yugabyte/tserver/tserver.{err|out}. A cautionary note -- setting string flags dynamically is not a good idea, as it is not thread safe. That being said, the `yb-ts-cli` utility allows changing gflags dynamically. _Note: For this step you must identify the server using its RPC port (not the HTTP

For example, to increase the verbose logging level to 2: ```sh $ ./yb-ts-cli --serveraddress=localhost:9100 setflag v 2 ``` Slow responses (which take more than 75% of the configured RPC timeout) are already logged at WARNING level in this format, with a breakdown of time in various stages. ```log W0325 06:47:13.032176 116514816 inbound_call.cc:204] Call yb.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:61050 (request call id 22856) took 2644ms (client timeout 1000). W0325 06:47:13.033341 116514816 inbound_call.cc:208] Trace: 0325 06:47:10.388015 (+ 0us) service_pool.cc:109] Inserting onto call queue 0325 06:47:10.394859 (+ 6844us) service_pool.cc:170] Handling call 0325 06:47:10.450697 (+ 55838us) raft_consensus.cc:1026] Updating replica for 0 ops 0325 06:47:13.032064 (+2581367us) raft_consensus.cc:1242] Filling consensus response to leader. 0325 06:47:13.032106 (+ 42us) spinlock_profiling.cc:233] Waited 2.58 s on lock 0x108dc3d40. stack: 0000000103d63b0c 0000000103d639fc 00000001040ac908 0000000102f698ec 0000000102f698a4 0000000102f93039 0000000102f7e124 0000000102fdcf7b 0000000102fd90c9 00000001\ 02504396 00000001032959f5 0000000103470473 0000000103473491 00000001034733ef 000000010347338b 000000010347314c 0325 06:47:13.032168 (+ 62us) inbound_call.cc:125] Queueing success response ``` You can view metrics of various Yugabyte processes at a particular node (e.g. 127.0.0.1) at these ports: | Process | Address | | : | : | | Master | 127.0.0.1:7000 | | TServer | 127.0.0.1:9000 | | Yedis | 127.0.0.1:11000 | | YCQL | 127.0.0.1:12000 | | YSQL | 127.0.0.1:13000 | For each process, you can see the following types of metrics: | Description | Endpoint | | :- | :- | | Per-Tablet, JSON Metrics | /metrics | | Per-Table, Prometheus Metrics | /prometheus-metrics | To turn on tracing, you should set the `enable_tracing` flag: ```sh $ ./yb-ts-cli --serveraddress=localhost:9100 setflag enable_tracing 1 ``` To turn tracing on for all RPCs, not just the slow ones, including the `enabletracing` flag, you may also set the `rpcdumpalltraces` gflag: ```sh $ ./yb-ts-cli --serveraddress=localhost:9100 setflag rpcdumpall_traces 1 ``` To dump the contents of a file containing a proto (such as a file in the consensus-meta or tablet-meta directory), use 'yb-pbc-dump' utility: ```sh $ ./yb-pbc-dump /mnt/d0/yb-data/tserver/consensus-meta/dd57975ef2f2440497b5d96fc32146d3 $ ./yb-pbc-dump /mnt/d0/yb-data/tserver/tablet-meta/bfb3f18736514eeb841b0307a066e66c ``` Note: On macOS, the environment variable DYLDFALLBACKLIBRARYPATH needs to be set for pbc-dump to work. Add the following to `~/.bashprofile`: ```sh export DYLDFALLBACKLIBRARY_PATH=~/code/yugabyte/build/latest/rocksdb-build ``` You can run various tablet related commands with yb-ts-cli by pointing at the master: ```sh ./yb-ts-cli listtablets --serveraddress=localhost:9000 ./yb-ts-cli dumptablet --serveraddress=localhost:9000 e1bc59288ee849ab850ae0a40bd88649 ``` You can run various commands with `yb-admin`. Just specify the full set of master `{ip:ports}` with `-master_addresses`: ```sh ./yb-admin -masteraddresses 127.0.0.1:7000,127.0.0.2:7000,127.0.0.3:7000 listtables ./yb-admin -masteraddresses 127.0.0.1:7000,127.0.0.2:7000,127.0.0.3:7000 listtablets ybloadtest ./yb-admin -masteraddresses 127.0.0.1:7000,127.0.0.2:7000,127.0.0.3:7000 listtabletservers $(./yb-admin -masteraddresses 127.0.0.1:7000,127.0.0.2:7000,127.0.0.3:7000 listtablets ybload_test) ./yb-admin -masteraddresses 127.0.0.1:7000,127.0.0.2:7000,127.0.0.3:7000 listalltabletservers ./yb-admin -masteraddresses 127.0.0.1:7000,127.0.0.2:7000,127.0.0.3:7000 listall_masters ./yb-admin -masteraddresses 127.0.0.1:7000,127.0.0.2:7000,127.0.0.3:7000 dumpmasters_state ```