Project: Apache bRPC (Orchestration & Management / Remote Procedure Call)
File: bvar_c++.md

The basic usage of bvar is simple: ```c++ namespace foo { namespace bar { // bvar::Adder<T> used for running sum, we define a Adder for read_error as below bvar::Adder<int> greaderror; // put another bvar inside window so that we can get the value over this period of time bvar::Window<bvar::Adder<int> > greaderrorminute("foobar", "readerror", &gread_error, 60); // ^ ^ ^ // prefix1 monitor name time window, 10 by default // bvar::LatencyRecorder is a compound varibale, can be used for troughputqpsavg latency, latency percentile, max latency bvar::LatencyRecorder gwritelatency("foo_bar", "write"); // ^ ^ // prefix1 monitor entry, LatencyRecorder includes different bvar, and expose() will add the suffix for them by default, such as writeqps, writelatency etc // define a varible for the # of 'been-pushed task' bvar::Adder<int> gtaskpushed("foobar", "taskpushed"); // put nested bvar into PerSecond so that we can get the value per second within this time window. Over here what we get is the # of tasks pushed per second bvar::PerSecond<bvar::Adder<int> > gtaskpushedsecond("foobar", "taskpushedsecond", &gtaskpushed); // ^ ^ // different from Window, PerSecond will be divided by the time winodw. time window is the last param, we omit here, its 10 by default } // bar } // foo ``` how we use the bvar ```c++ // run into read errors foo::bar::greaderror << 1; // record down the latenct, which is 23ms foo::bar::gwritelatency << 23; // one task has been pushed foo::bar::gtaskpushed << 1; ``` Remember Window<> and PerSecond<> are derived variables, we don't have to push value to them and they will auto-update. Obviously, we can take bvar as local or member variables. There are essentially 7 commonly used bvar classes, they all extend from the base class bvar:Variable. `bvar::Adder<T>` : counter, 0 by default, varname << N equals to varname += N `bvar::Maxer<T>` : get the maximum value, std::numeric_limits::min() by default, varname << N equals to varname = max(varname, N) `bvar::Miner<T>` : get the minimum value, std::numeric_limits::max() by default, varname << N equals to varname = min(varname, N) `bvar::IntRecorder` : get the mean value since it was in use, notice here we don't use over a period of time, since this bvar always comes with Window<> to calculate the mean value within the predefined time window. `bvar::Window<VAR>` : get the running sum over a time window. Window derives from other existing bvar and will auto-update `bvar::PerSecond<VAR>` : get the value per second during a predefined amount of time. PerSecond will also auto-update and derives from other bvar `bvar::LatencyRecorder` : intended for recording latency and qps, when we push latency to it, for mean latency/max lantency/qps, we will get all in once. caveat: make sure the name of bvar is globally unique, otherwise, the expose() will fail. When the option -bvar_abort_on_same_name is true(false by default), program will abort. There are different bvar from different module, to avoid duplicating name, we'd better follow the rule as `moduleclassindicator a module usually refers to the program name, can be the acronym of product line, like infds, ecomretrbs etc. a class usually refers to the class name/ function name, like storagemanager, filetransfer, rank_stage1. an indicator usually refers to qps, count, latency etc. some legit naming examples ``` iobufblockcount : 29 # module=iobuf class=block indicator=count iobufblockmemory : 237568 # module=iobuf class=block indicator=memory processmemoryresident : 34709504 # module=process class=memory indicator=resident processmemoryshared : 6844416 # module=process class=memory indicator=shared rpcchannelconnectioncount : 0 # module=rpc class=channelconnection indicator=count rpccontrollercount : 1 # module=rpc class=controller indicator=count rpcsocketcount : 6 # module=rpc class=socket indicator=count ``` bvar will normalize the variable name, no matter whether we type foo::BarNum, foo.bar.num, foo bar num , foo-bar-num, they all go to foobarnum in the

Things about indicators: use `count` as suffix for number, like requestcount, error_count use `second` as suffix for number per second is clear enough, no need to use 'countsecond' or 'persecond', like requestsecond, processinblockssecond `minute` as suffix for number per minute like requestminute, processinblocksminute if we need to use a counter defined in another file, we have to declare that variable in header file ``` namespace foo { namespace bar { // notice greaderrorminute and gtaskpushedsecond are derived bvar, will auto update, no need to declare extern bvar::Adder<int> greaderror; extern bvar::LatencyRecorder gwritelatency; extern bvar::Adder<int> gtaskpushed; } // bar } // foo ``` Don't define golabel Window<> and PerSecond<> across files. The order for the initialization of global variables in different compile units is undefined. foo.cpp defines `Adder<int> foocount`, then defining `PerSecond<Adder<int> > fooqps(&foocount);` in fooqps.cpp is illegal Things about thread-safety: bvar is thread-compatible. We can manipulate a bvar in different threads, such as we can expose or hide different bvar in multiple threads simultaneously, they will safely do some job on global shared variables. Excpet read and write API, any other functions of bvar are not thread-safeu can not expose or hide a same bvar in different threads, this may cause the program crash. Generally speaking, we don't have to call any other API concurrently except read and write. we can use butil::Timer for timer, API is as below: ```C++ namespace butil { class Timer { public: enum TimerType { STARTED }; Timer(); // butil::Timer tm(butil::Timer::STARTED); // tm is already started after creation. explicit Timer(TimerType); // Start this timer void start(); // Stop this timer void stop(); // Get the elapse from start() to stop(). int64t nelapsed() const; // in nanoseconds int64t uelapsed() const; // in microseconds int64t melapsed() const; // in milliseconds int64t selapsed() const; // in seconds }; } // namespace butil ``` Varibale is the base class for all bvar, it provides registering, listing and searching functions. When user created a bvar with default params, it hasn't been registered into any global structure, it's merely a faster counter, which means we cannot use it elsewhere. The action of putting this bvar into the global registry is called `expose`, can be achieved by calling `expose()` The name for globally exposed bvar consists of 'name' or 'name+prefix', can be searched by function with suffix `exposed` , e.g. Variable::describeexposed("foo") will return the description of bvar with the name 'foo'. When there already exists the name, expose() will print FATAL log and return -1. When the option -bvar_abort_on_same_name is true(false by default), program will abort. Some examples for expose() as below ``` bvar::Adder<int> count1; // create a bvar with defalut params count1 << 10 << 20 << 30; // values add up to 60. count1.expose("count1"); // expose the variable globally CHECKEQ("60", bvar::Variable::describeexposed("count1")); count1.expose("anothernamefor_count1"); // expose the variable with another name CHECKEQ("", bvar::Variable::describeexposed("count1")); CHECKEQ("60", bvar::Variable::describeexposed("anothernamefor_count1")); bvar::Adder<int> count2("count2"); // exposed in constructor directly CHECKEQ("0", bvar::Variable::describeexposed("count2")); // default value of Adder<int> is 0 bvar::Status<std::string> status1("count2", "hello"); // the name conflicts. if -bvarabortonsamename is true, // program aborts, otherwise a fatal log is printed. ``` To avoid duplicate name, we should have prefix for bvar, we recommend the name as `<namespace><module><name>` For convenience, we provide expose_as() as it will accept a prefix. ```C++ // Expose this variable with a prefix. // Example: // namespace foo { // namespace bar { // class ApplePie { // ApplePie() { // // foobarapplepieerror //

"error"); // } // private: // bvar::Adder<int> _error; // }; // } // foo // } // bar int expose_as(const butil::StringPiece& prefix, const butil::StringPiece& name); ``` Common needs for exporting are querying by HTTP API and writing into local file, the former is provided by brpc service, the latter has been implemented in bvar, and it's turned off by default. A couple of methods can activate this function Using to parse the input params. We can add `-bvar_dump` during the starup of program or we can dynamically change the params thru brpc service after starup. gflags parsing is as below ```C++ ... int main(int argc, char* argv[]) { if (google::SetCommandLineOption("bvar_dump", "true").empty()) { LOG(FATAL) << "Fail to enable bvar dump"; } ... } ``` If u dont want to use gflags and expect them opened by default in program ```C++ ... int main(int argc, char* argv[]) { if (google::SetCommandLineOption("bvar_dump", "true").empty()) { LOG(FATAL) << "Fail to enable bvar dump"; } ... } ``` dump function is controlled by following gflags | Name | Default Value | Effect | | | -- | - | | bvardump | false | Create a background thread dumping all bvar periodically, all bvardump_* flags are not effective when this flag is off | | bvardumpexclude | "" | Dump bvar excluded from these wildcards(separated by comma), empty means no exclusion | | bvardumpfile | monitor/bvar.\<app\>.data | Dump bvar into this file | | bvardumpinclude | "" | Dump bvar matching these wildcards(separated by comma), empty means including all | | bvardumpinterval | 10 | Seconds between consecutive dump | | bvardumpprefix | \<app\> | Every dumped name starts with this prefix | | bvardumptabs | \<check the code\> | Dump bvar into different tabs according to the filters (separated by semicolon), format: *(tab_name=wildcards) | when the bvardumpfile is not empty, a background thread will be started to update `bvardumpfile` for the specified time interval called `bvardumpinterval` , including all the bvar which is matched by `bvardumpinclude` while not matched by `bvardumpexclude` such like we modify the gflags as below ](https://github.com/apache/brpc/blob/master/docs/images/bvardumpflags_2.png) exporting file will be like ``` $ cat bvar.echo_server.data rpcserver8002builtinservice_count : 20 rpcserver8002connectioncount : 1 rpcserver8002nsheadservice_adaptor : brpc::policy::NovaServiceAdaptor rpcserver8002servicecount : 1 rpcserver8002starttime : 2015/07/24-21:08:03 rpcserver8002uptimems : 14740954 ``` `iobufblockcount : 8` is filtered out by bvardumpinclude, `rpcserver8002error : 0` is ruled out by bvardump_exclude. if you didn't use brpc in your program, u also need to dynamically change gflagsnormally no need), we can call google::SetCommandLineOption(), as below ```c++ ... if (google::SetCommandLineOption("bvardumpinclude", "service").empty()) { LOG(ERROR) << "Fail to set bvardumpinclude"; return -1; } LOG(INFO) << "Successfully set bvardumpinclude to service"; ``` Do not directly set FLAGSbvardumpfile / FLAGSbvardumpinclude / FLAGSbvardump_exclude. On the one hand, these gflags are std::string, which are not thread-safe to be overriddenOn the other hand, the validator will not be triggered(call back to check the correctness), so the exporting thread will not be invoked users can also customize dump_exposed() to export all the exposed bvar ```c++ // Implement this class to write variables into different places. // If dump() returns false, Variable::dump_exposed() stops and returns -1. class Dumper { public: virtual bool dump(const std::string& name, const butil::StringPiece& description) = 0; }; // Options for Variable::dump_exposed(). struct DumpOptions { // Contructed with default options. DumpOptions(); // If this is true, string-type values will be quoted. bool quote_string; // The ? in wildcards. Wildcards in URL need to use another character // because ? is reserved. char question_mark; // Separator for whitewildcards and blackwildcards. char wildcard_separator; // Name matched by these wildcards (or exact names) are kept. std::string white_wildcards; // Name matched by these wildcards (or exact names) are skipped. std::string black_wildcards; }; class Variable { ... ... // Find all exposed variables matching `white_wildcards' but // `black_wildcards' and send them to `dumper'. // Use default options when `options' is

// Return number of dumped variables, -1 on error. static int dump_exposed(Dumper dumper, const DumpOptions options); }; ``` Reducer uses binary operators over several values to combine them into one final result, which are commutative, associative and without side effect. Only satisfying all of these thress, we can make sure the combined result is not affected by the distribution of the private variables of thread. Like substraciton does not satisfy associative nor commutative, so it cannot be taken as the operator here. ```C++ // Reduce multiple values into one with `Op': e1 Op e2 Op e3 ... // `Op' shall satisfy: // - associative: a Op (b Op c) == (a Op b) Op c // - commutative: a Op b == b Op a; // - no side effects: a Op b never changes if a and b are fixed. // otherwise the result is undefined. template <typename T, typename Op> class Reducer : public Variable; ``` reducer << e1 << e2 << e3 equals to reducer = e1 op e2 op e3 Common Redcuer subclass: bvar::Adder, bvar::Maxer, bvar::Miner we can tell from its name, it's intended for running sum. Opeator is `+` ```c++ bvar::Adder<int> value; value << 1 << 2 << 3 << -4; CHECKEQ(2, value.getvalue()); bvar::Adder<double> fp_value; // may have warning fp_value << 1.0 << 2.0 << 3.0 << -4.0; CHECKDOUBLEEQ(2.0, fpvalue.getvalue()); ``` Adder<> can be applied to the non-primitive type, which at least overrides `T operator+(T, T)`, an existing example is `std::string`, the code below will concatenate strings ```c++ // This is just proof-of-concept, don't use it for production code because it makes a // bunch of temporary strings which is not efficient, use std::ostringstream instead. bvar::Adder<std::string> concater; std::string str1 = "world"; concater << "hello " << str1; CHECKEQ("hello world", concater.getvalue()); ``` is producing the maximum value, operator is `std::max` ```c++ bvar::Maxer<int> value; value << 1 << 2 << 3 << -4; CHECKEQ(3, value.getvalue()); ``` Since Maxer<> use std::numericlimits::min() as the identity, it cannot be applied to generic types unless you specialized std::numericlimits<> (and overloaded operator<, yes, not operator>). producing minimum value, operator is std::min ```c++ bvar::Miner<int> value; value << 1 << 2 << 3 << -4; CHECKEQ(-4, value.getvalue()); ``` Since Miner<> use std::numericlimits::max() as the identity, it cannot be applied to generic types unless you specialized std::numericlimits<> (and overloaded operator<). used for mean value ```c++ // For calculating average of numbers. // Example: // IntRecorder latency; // latency << 1 << 3 << 5; // CHECK_EQ(3, latency.average()); class IntRecorder : public Variable; ``` A counter used for latency and qps. We can get latency / maxlatency / qps / count as long as the latency data filled in. Time window is the last param, omit by `bvardump_interval` LatencyRecoder is a compound variable, consisting of several bvar. ```c++ LatencyRecorder writelatency("table2mytablewrite"); // produces 4 variables: // table2mytablewritelatency // table2mytablewritemax_latency // table2mytablewriteqps // table2mytablewritecount // In your write function writelatency << thelatencyofwrite; ``` Get data within a time window. Window cannot exist alone, it relies on a counter. Window will auto-update, we don't have to send data to it. For the sake of performance, the data comes from every-second sampling over the original counter, in the worst case, Window has one-second latency ```c++ // Get data within a time window. // The time unit is 1 second fixed. // Window relies on other bvar which should be constructed before this window and destructs after this

// R must: // - have get_sampler() (not require thread-safe) // - defined valuetype and samplertype template <typename R> class Window : public Variable; ``` Get the mean value over the last amount of time. Its almost the same as Window, except for the value will be divided by the time window ```c++ bvar::Adder<int> sum; // sumpersecond.getvalue()is summing every-second value over the last 60 seconds, if we omit the time window, it's set to 'bvardump_interval' by default bvar::PerSecond<bvar::Adder<int> > sumpersecond(&sum, 60); ``` PerSecond does not always make sense There is no Maxer in the above code, since the max value over a period of time divided by the time window is meaningless. ```c++ bvar::Maxer<int> max_value; // WRONGmax value divided by time window is pointless bvar::PerSecond<bvar::Maxer<int> > maxvaluepersecondwrong(&max_value); // CORRECT. It's the right way to set the time window to 1s so that we can get the max value for every second bvar::Window<bvar::Maxer<int> > maxvaluepersecond(&maxvalue, 1); ``` Suppose we want the memory change since last minute, if we use Window<>, the meaning for the returning value is "the memory increase over the last minute is 18M" if we use PerSecond<>, the meaning for return value will be "the average memory increase per second over the last minute is 0.3M. Pros of Window is preciseness, it fits in some small-number cases, like the number of error produced over last minute, if we use PerSecond, we might get something like "the average error rate per second over the last minute is 0.0167", which is very unclear as opposed to "one error produced over last minute". Some other non-time-related variables also fit in Window<>, such like calculating the CPU ratio over the last minute is using a Adder by summing CPU time and real time, then we use Window<> on top of the Adder to get the last-mintue CPU time and real time, dividing these two value then we get the CPU ratio for the last minute, which is not time-related. It will get wrong when use PerSeond Record and display one value, has additional set_value() function ```c++ // Display a rarely or periodically updated value. // Usage: // bvar::Status<int> foo_count1(17); // foocount1.expose("myvalue"); // // bvar::Status<int> foo_count2; // foocount2.setvalue(17); // // bvar::Status<int> foocount3("myvalue", 17); // // Notice that Tp needs to be std::string or acceptable by boost::atomic<Tp>. template <typename Tp> class Status : public Variable; ``` Display the value when needed. In some cases, we are not able to actively setvalue nor setvalue in a certain time interval. We'd better print it out when needed, user can pass in the print-out callback function to achieve this. ```c++ // Display a updated-by-need value. This is done by passing in an user callback // which is called to produce the value. // Example: // int print_number(void* arg) { // ... // return 5; // } // // // number1 : 5 // bvar::PassiveStatus status1("number1", print_number, arg); // // // foo_number2 : 5 // bvar::PassiveStatus status2(typeid(Foo), "number2", print_number, arg); template <typename Tp> class PassiveStatus : public Variable; even though it looks simple, PassiveStatus is one of the most useful bvar, since most of the statistic values have already existed, we don't have to store it again, just fetch the data according to our need. Declare a bvar which can display user process name as below static void get_username(std::ostream& os, void*) { char buf[32]; if (getlogin_r(buf, sizeof(buf)) == 0) { buf[sizeof(buf)-1] = '\0'; os << buf; } else { os << "unknown"; } } PassiveStatus<std::string> gusername("processusername", get_username, NULL); ``` Expose important gflags as bvar so that they're monitored (in noah). ```c++ DEFINEint32(myflagthatmatters, 8, "..."); // Expose the gflag