Project: Apache bRPC (Orchestration & Management / Remote Procedure Call)
File: bvar.md

is a set of counters to record and view miscellaneous statistics conveniently in multi-threaded applications. The implementation reduces cache bouncing by storing data in thread local storage(TLS), being much faster than UbMonitor(a legacy counting library inside Baidu) and even atomic operations in highly contended scenarios. brpc integrates bvar by default, namely all exposed bvars in a server are accessible through , and a single bvar is addressable by . Read to know how to query them in brpc servers. brpc extensively use bvar to expose internal status. If you are looking for an utility to collect and display metrics of your application, consider bvar in the first place. bvar definitely can't replace all counters, essentially it moves contentions occurred during write to read: which needs to combine all data written by all threads and becomes much slower than an ordinary read. If read and write on the counter are both frequent or decisions need to be made based on latest values, you should not use bvar. To understand how bvar works, read first, in which the mentioned counter example is just bvar. When many threads are modifying a counter, each thread writes into its own area without joining the global contention and all private data are combined at read, which is much slower than an ordinary one, but OK for low-frequency logging or display. The much faster and very-little-overhead write enables users to monitor systems from all aspects without worrying about hurting performance. This is the purpose that we designed bvar. Following graph compares overhead of bvar, atomics, static UbMonitor, dynamic UbMonitor when they're accessed by multiple threads simultaneously. We can see that overhead of bvar is not related to number of threads basically, and being constantly low (~20

As a contrast, dynamic UbMonitor costs 7 microseconds on each operation when there're 24 threads, which is the overhead of using the bvar for 300 times. Read to know how to add bvar in C++. bvar already provides stats of many process-level and system-level variables by default, which are prefixed with `process` and `system`, such as: ``` processcontextswitchesinvoluntarysecond : 14 processcontextswitchesvoluntarysecond : 15760 processcpuusage : 0.428 processcpuusage_system : 0.142 processcpuusage_user : 0.286 processdiskreadbytessecond : 0 processdiskwritebytessecond : 260902 processfaultsmajor : 256 processfaultsminor_second : 14 processmemoryresident : 392744960 systemcorecount : 12 systemloadavg15m : 0.040 systemloadavg1m : 0.000 systemloadavg5m : 0.020 ``` and miscellaneous bvars used by brpc itself: ``` bthreadswitchsecond : 20422 bthreadtimerscheduled_second : 4 bthreadtimertriggered_second : 4 bthreadtimerusage : 2.64987e-05 bthreadworkercount : 13 bthreadworkerusage : 1.33733 bvarcollectordump_second : 0 bvarcollectordumpthreadusage : 0.000307385 bvarcollectorgrab_second : 0 bvarcollectorgrabthreadusage : 1.9699e-05 bvarcollectorpending_samples : 0 bvardumpinterval : 10 bvar_revision : "34975" bvarsamplercollector_usage : 0.00106495 iobufblockcount : 89 iobufblockcounthittls_threshold : 0 iobufblockmemory : 729088 iobufnewbigviewsecond : 10 ``` New exported files overwrite previous files, which is different from regular logs which append new data. Turn on of bvar to export all exposed bvars to files, which are formatted just like above texts: each line is a pair of "name" and "value". Check if there're data under $PWD/monitor/ after enabling dump, for example: ``` $ ls monitor/ bvar.echoclient.data bvar.echoserver.data $ tail -5 monitor/bvar.echo_client.data process_swaps : 0 processtimereal : 2580.157680 processtimesystem : 0.380942 processtimeuser : 0.741887 process_username : "gejun" ``` The monitoring system should combine data on every single machine periodically and merge them together to provide on-demand queries. Take the "noah" system inside Baidu as an example, variables defined by bvar appear as metrics in noah, which can be checked by users to view historical curves. To export to , set the path in scraping target url to `/brpcmetrics`. For example, if brpc server is running on localhost:8080, the scraping target should be `127.0.0.1:8080/brpcmetrics`.