Project: Tarantool (App Definition and Development / Database)
File: 1148-stacked-diagnostics.md

Status*: In progress Start date*: 30-07-2019 Authors*: Kirill Shcherbatov @kshcherbatov kshcherbatov@tarantool.org, Pettik Nikita @Korablev77 korablev@tarantool.org Issues*: The document describes a stacked diagnostics feature. It is needed for the cases, when there is a complex and huge call stack with variety of subsystems in it. It may turn out that the most low-level error is not descriptive enough. In turn user may want to be able to look at errors along the whole call stack from the place where a first error happened. In terms of implementation single fiber's error object is going to be replaced with a list of objects forming diagnostic stack. Its first element will always be created by the deepest and the most basic error. Both C and Lua APIs are extended to support adding errors in stack. Support stacked diagnostics for Tarantool allows to accumulate all errors occurred during request processing. It allows to better understand what happened, and handle errors appropriately. Consider following example: persistent Lua function referenced by functional index has a bug in it's definition, Lua handler sets an diag message. Then functional index extractor code setups an own, more specialized error. Currently Tarantool has `diag_set()` mechanism to set a diagnostic error. Object representing error featuring following properties: type (string) errors C++ class; code (number) errors number; message (string) errors message; file (string) Tarantool source file; line (number) line number in the Tarantool source file. The last error raised is exported with `box.error.last()` function. Type of error is represented by a few C++ classes (all are inherited from Exception class). For instance hierarchy for ClientError is following: ``` ClientError | LoggedError | AccessDeniedError | UnsupportedIndexFeature ``` All codes and names of ClientError class are available in box.error. User is able to create a new error instance of predefined type using box.error.new() function. For example: ``` tarantool> t = box.error.new(box.error.CREATE_SPACE, "myspace", "just cause") tarantool> t:unpack() type: ClientError code: 9 message: 'Failed to create space ''myspace'': just cause' trace: file: '[string "t = box.error.new(box.error.CREATE_SPACE, "my..."]' line: 1 ``` User is also capable of defining own errors with any code by means of: ``` box.error.new({code = usercode, reason = usererror_msg}) ``` For instance: ``` e = box.error.new({code = 500, reason = 'just cause'}) ``` Error cdata object has `:unpack()`, `:raise()`, `:match(...)`, `:serialize()` methods and `.type`, `.message` and `.trace`

In some cases a diagnostic area should be more complicated than one last raised error to provide decent information concerning incident (see motivating example above). Without stacked diagnostic area, only last error is delivered to user. One way to deal with this problem is to introduce stack accumulating all errors happened during request processing. Let's keep existent `diag_set()` method as is. It is supposed to replace the last error in diagnostic area with a new one. To add new error at the top of existing one, let's introduce new method `diag_add()`. It is assumed to keep an existent error message in diagnostic area (if any) and sets it as a previous error for a recently-constructed error object. Note that `diag_set()` is not going to preserve pointer to previous error which is held in error to be substituted. To illustrate last point consider example: ``` Errors: <NULL> diag_set(code = 1) Errors: <e1(code = 1) -> NULL> diag_add(code = 2) Errors: <e1(code = 1) -> e2(code = 2) -> NULL> diag_set(code = 3) Errors: <e3(code = 3) -> NULL> ``` Hence, developer takes responsibility of placing `diag_set()` where the most basic error should be raised. For instance, if during request processing `diagadd()` is called before `diagset()` then it will result in inheritance of all errors from previous error raise: ``` -- Processing of request #1 diag_set(code = 1) Errors: <e1(code = 1) -> NULL> diag_add(code = 2) Errors: <e1(code = 1) -> e2(code = 2) -> NULL> -- End of execution -- Processing of request #2 diag_add(code = 1) Errors: <e1(code = 1) -> e2(code = 2) -> e3(code = 1) -> NULL> -- End of execution ``` As a result, at the end of execution of second request, three errors in stack are reported instead of one. Another way to resolve this issue is to erase diagnostic area before request processing. However, it breaks current user-visible behaviour since box.error.last() will preserve last occurred error only until execution of the next request. The diagnostic area (now) contains (nothing but) pointer to the top error: ``` struct diag { struct error *last; }; ``` To organize errors in a list let's extend error structure with pointer to the previous element. Or alternatively, add member of any data structure providing list properties (struct rlist, struct stailq or whatever): ``` struct diag { struct stailq *errors; }; struct error { ... struct stailqentry *inerrors; }; ``` When error is set to diagnostics area, its reference counter is incremented; on the other hand if error is added

linked to the head of diagnostics area list), its reference counter remains unchanged. The same concerns linking two errors: only counter of referenced error is incremented. During error destruction (that is the moment when error's reference counter hits 0 value) the next error in the list (if any) is also affected: its reference counter is decremented as well. Tarantool returns a last-set (diag::last) error as `cdata` object from central diagnostic area to Lua in case of error. User should be unable to modify it (since it is considered to be a bad practice - in fact object doesn't belong to user). On the other hand, user needs an ability to inspect a collected diagnostic information. Hence, let's extend the error object API with a method which provides the way to get the previous error (reason): `:prev()` (and correspondingly `.prev` field). ``` -- Return a reason error object for given error object 'e' -- (when exists, nil otherwise). e:prev(error) == error.prev ``` Furthermore, let's extend signature of `box.error.new()` with new (optional) argument - 'prev' - previous error object: ``` e1 = box.error.new({code = 111, reason = "just cause"}) e2 = box.error.new({code = 222, reason = "just cause x2", prev = e1}) ``` User may want to link already existing errors. To achieve this let's add `set_prev` method to error object so that one can join two errors: ``` e1 = box.error.new({code = 111, reason = "just cause"}) e2 = box.error.new({code = 222, reason = "just cause x2"}) ... e2.set_prev(e1) -- e2.prev == e1 ``` Currently errors are sent as `(IPROTO_ERROR | errcode)` response with an string message containing error details as a payload. There are not so many options to extend current protocol wihtout breaking backward compatibility (which is obviously one of implementation requirements). One way is to extend existent binary protocol with a new key IPROTOERRORSTACK (or IPROTOERRORREASON or simply IPROTOERRORV2): ``` { // backward compatibility IPROTO_ERROR: "the most recent error message", // modern error message IPROTOERRORSTACK: { { // the most recent error object IPROTOERRORCODE: errorcodenumber, IPROTOERRORMESSAGE: errormessagestring, }, ... { // the oldest (reason) error object }, } } ``` IPROTO_ERROR is always sent (as in older versions) in case of error. IPROTOERRORSTACK is presented in response only if there's at least two elements in diagnostic list. Map which contains error stack can be optimized in terms of space, so that avoid including error which is already encoded in IPROTO_ERROR.