Mitigating failure in request handling

A network computer system operates to mitigate failures for a network service. The network computer system can generate a data path model for the network service, where the data path model identifies a probabilistic set of expectations with respect to the programs and program sequences which handle service requests for the network service. The data path models can be used to detect, analyze or mitigate service request failures of the network service.

TECHNICAL FIELD

This application pertains to mitigating failures in request handling by a network service.

BACKGROUND

The technology underlying many network services (e.g., ride-sharing, food-delivery, etc.) is complex and demanding. For example, on-demand network services generally utilize distributed network computing architectures to aggregate and process information from numerous sources, while at the same time, providing highly-responsive and relevant output to user requests and input. To fulfill service requests (e.g., requests generated from end users to receive service, service requests generated by predefined service events, etc.), a network service typically coordinates numerous workflows, some of which may depend on other workflows. Moreover, in some cases, the workflows may be implemented in-part using user devices and other third-party services.

In this regard, the complexities and demands of on-demand network services face a particular challenge with respect to detecting and mitigating failures by the various components. Given the dependencies of the components used for on-demand network services, an outage or failure by one component can have widespread impact on the service as a whole. For this reason, detecting and mitigating failures amongst individual components of an on-demand network computer system can be of paramount importance.

DETAILED DESCRIPTION

According to examples, a network computer system utilizes logging in a distributed environment to develop and deploy data path models from which failures of components or services deployed in the distributed environment can be detected and mitigated. Among other benefits, an example network computer system as described can reduce the time to mitigation (“TTM”) when an outage or an error occurs by a component that operates in the distributed environment.

In one aspect, the network computer system can provide an interactive tool to facilitate operators in quickly detecting failure by a component of the distributed environment, as well as to determine a cause of the detected failure. In variations, the network computer system can include logic to diagnose a detected failure, specifically to pinpoint a particular program (e.g., microservice, process, etc.) or candidate set of programs where a given failure likely occurred. Still further, in other variations, the network computer system can detect likely sources of failures, and further implement programmatic operations to mitigate or remediate such detected failures (e.g., trigger a restart of the program or process that is detected as having failed). By reducing the TTM for the distributed environment, examples as described can decrease the impact of an outage in the distributed environment. In regard to an on-demand network service that is provided through the distributed environment, for example, the reduced outage provides greater reliability in the availability of the on-demand network service for a given population of users.

In examples, a network service may implement one or multiple processes (e.g., performed by programs or services) to handle different types of service requests, where each process implements workflow logic and workflow(s) to handle particular types of service requests. A network service may utilize different processes to handle service-related events for different types of users (e.g., service provider and service requester). According to examples, a network service may utilize multiple processes in connection with service-related activities of an individual user, where each process is initiated by a request.

According to examples, a data path model includes a plurality of program nodes, with each program node representing one of multiple programs that receive and act on data associated with service requests of corresponding user devices. In a data path model, each program node is associated with corresponding sequence data that identifies an expectation that the program represented by the program node is called by or calls any of one or more entities (e.g., other programs, user device), to perform tasks in connection with handling of an incoming service request.

Additionally, a data path model can specify an arrangement for a corresponding plurality of program nodes, where the arrangement is in accordance with the sequence data associated with each of the plurality of program nodes. In this way, the arrangement of the data path model reflects at least one data path by which the corresponding programs execute to receive and act on data associated with a service request, so that the service request is successfully handled.

Still further, in examples, a network computer system operates to determine a data path model for a process of a network service, where the determined data path model identifies an expectation of how a workflow for the process will be implemented to successfully handle a service request. A determined data path model may further include one or more probabilistic determinations, relating to whether or not individual program events (e.g., a particular program being called or initiated) occur in a workflow of a process, and/or a sequence in which certain program events of the workflow occur. In examples, the network computer system can further utilize the determined data path models to detect, analyze and/or mitigate against service request failures. In examples, a service request that fails (or a failed service request) refers to a service request that was not successfully handled by the network service as a result of a failure by a program or process of the network service.

According to examples, a network computer system operates to determine, from analyzing one or more types of logging information, at least a first program of the multiple programs that acted on data associated with the failed service request. The network computer system further identifies, from the data path model, a second program of the multiple programs that was expected to receive and act on data associated with the service request that failed, where the second program is identified based at least in part on the sequence data associated with the second program. In examples, the network computer system can use logging information associated with the second program to detect a failed service request. The network computer system may also utilize the logging information associated with the second program to analyze the programs of a process which handled the failed service request. By way of example, the network system can analyze the handling of the failed service request to (i) pinpoint a specific program or set of programs where the failure is manifested; (ii) identify a variance or deviation as to how the network service handled the failed service request as compared to how the network service should have (or was expected to handle the service request); (iii) identify a candidate set of programs from which a cause of the failed service request may be inferred or detected; and/or (iv) determine a likely cause or source for the failure.

In some examples, the network service can respond to a failed service request by (i) generating a notification or alert for a network operator, and/or (ii) providing a user-interface for the network operator that organizes information that the network computer system determines is associated with the particular service request failure. The information provided with the network computer system can, for example, identify (i) program(s) of a workflow for handling the failed service request which performed as expected, (ii) program(s) of the workflow which reported an unexpected error in connection with the handling of the failed service request, and/or (iii) programs of the workflow which were not called as may have otherwise been expected for the service request to have been successfully handled.

Still further, in some examples, the network computer system can respond to a failed service request by determining a program that is a likely source of the failure. As an addition or variation, the network computer system can determine one or multiple candidate programs that are a source for a failed service request. For example, multiple programs may be identified, where each program is associated with a probability of being a source of the failed service request. A program that is the source of a failed service request can include a program that failed to properly execute when triggered in the workflow for handling the service request, where the failure of the program is not attributable to the failure of another program in the same workflow.

In examples, the network computer system can determine a program that is a source for a failed service request, and/or one or more programs which are candidate sources for the failed service request. The network computer system may identify the determined source program(s) on a user-interface that is generated for a network operator. As an addition or variation, the network computer system may programmatically implement one or more remedial actions to affect the operation of the one or more source program(s) to mitigate against future occurrence of service request failures.

Still further, in other examples, a network computer system can operate to mitigate failures on an on-demand network service. In examples, the network computer system may aggregate logging information for a collection of service requests to the on-demand network service, where the logging information including trace routes that identify (i) multiple programs which execute to fulfill each service request of the collection, and (ii) timing information that identifies a relative timing as between individual programs of the multiple programs executing in fulfilling service requests. The network computer system can determine, from the aggregated logging information, a data path model that identifies multiple possible data paths, where each data path represents a sequence by which at least some of the multiple programs execute to fulfill service requests. The network computer system may further analyze a trace route for a failed service request to the on-demand network service, and determine, from analyzing the trace route, at least a first program that executed to fulfill the failed service request. Additionally, the network computer system can determine, from the data path model, at least a second program of the multiple programs that was expected to act on the service request that failed, where the second program being identified based at least in part on the sequence data associated with at least one of the first program or the second program.

Some embodiments described herein can generally require the use of computing devices, including processing and memory resources. For example, one or more embodiments described herein may be implemented, in whole or in part, on computing devices such as servers, desktop computers, cellular or smartphones, tablets, wearable electronic devices, laptop computers, printers, digital picture frames, network equipment (e.g., routers) and tablet devices. Memory, processing, and network resources may all be used in connection with the establishment, use, or performance of any embodiment described herein (including with the performance of any method or with the implementation of any system).

Network Computer System

FIG. 1illustrates an example network computer system to generate and analyze data path models for service requests handled by a grouping of programs that implement an on-demand network service. With respect to examples as described, a network computer system100can be implemented on a server, on a combination of servers, and/or on a distributed set of computing devices which communicate over a network such as the Internet. Still further, some examples provide for the network computer system100to be distributed using one or more servers and/or mobile devices. As described in greater detail, network computer system100can develop and utilize data path models to reduce the TTM for failures that occur when the on-demand network service is deployed.

With reference toFIG. 1, network computer system100monitors and evaluates the handling of service requests by programs82of an on-demand network service80. In examples such as described withFIG. 1, the network computer system100implements the on-demand network service80in conjunction with performing evaluation, diagnosis, failure analysis and/or mitigation. In variations, the network computer system100and on-demand network service80can be logically separated, so as to be implemented on different computing environments. Still further, in other examples, the network computer system100can operate independently of the on-demand network service80.

In examples, the programs82can cooperate to implement the network service for a given group or population of users and/or devices. The on-demand network service80can be implemented by programs that operate on the network computer system100, in conjunction with device processes that operate on individual devices (e.g., user devices) that are external to the network system.

In some examples, the network computer system100provides the on-demand network service80using a microservice network architecture. In the micro-service architecture, each of the programs82implement a microservice for the network computer system100. In implementing a microservice, each program supports a specific set of tasks and a particular objective, with well-defined programmatic interfaces for providing functionality such as logging and communicating with other programs or entities. In this way, the on-demand network service80can be implemented in part by the programs82to provide a variety of outcomes for devices (e.g., end user devices) that generate service requests for the network service80. Accordingly, the on-demand network service80can use the programs82to implement a distributed architecture, where multiple workflows are possible to handle incoming requests, and where individual workflows differ from one another by the sequence in which the programs82are implemented. By way of example, the on-demand network service80can include transport arrangement services, delivery services, meetups, etc. While some examples ofFIG. 1are described in context of the programs82executing to handle incoming service requests101from user devices22, examples can be implemented in alternative context, such as handling requests to the on-demand network service80from other types of external entities (e.g., other network services).

With further reference toFIG. 1, the network computer system100can receive incoming service requests101from user devices22. The user devices22can, for example, execute respective service applications that communicate with a service interface102, in order to communicate and use the on-demand network service80. In this way, the user devices22can be used to generate service requests101of the on-demand network service80. The on-demand network service80can be implemented in part of a group of programs82, which execute cooperatively to perform a series of tasks with respect to the handling of individual service requests101.

According to examples, each of the program82executes using data associated with an incoming service requests, to perform tasks that support the progress of the service request101towards completion. The handling of a service request101to successful completion can involve the performance of numerous tasks by different programs, and when the tasks are complete, the service request is successfully handled. The on-demand network service80can implement workflow logic90, with the individual programs82and/or separately, to manage the initiation of individual programs82for receiving and acting on data associated with the incoming service requests101. The workflow logic90can execute as part of the individual programs82, in order to enable the programs to call and initiate other programs for successful completion of the handling of the individual service requests101. The workflow logic90can also execute to call individual programs82, such as from initial receipt and handling of the service request101by the on-demand network service80.

In examples, the determination of the sequence by which the individual programs82are called to perform tasks for the incoming service request101may be based on a variety of factors, such as (i) an existing load on the on-demand network service80; (ii) the availability of a resource which a particular program82requires when performing one or more of its tasks; (iii) network latency; (iv) information specified by a particular service request; (v) the objective or goal for a particular service request; (vi) the successful completion of task performed by other programs; and/or (vii) the particular network process for that handles the service request. As such, the particular sequence in which the programs82execute to handle service requests101can vary amongst different workflows. In some examples, upon receipt of an incoming service request101, workflow logic90can call an initial program82to perform a set of tasks that support the service request, and the initial program82may then call another one of the program82to initiate the next program in performing its set of tasks to support the incoming service request101.

Some program82can thus execute to call other programs82in accordance with a sequence that can be determined in part, by individual programs implementing aspects of the workflow logic90. Each program that is called for an incoming service request101can then execute, using data associated with the service request101, to perform its own respective set of tasks. Once a succession of programs82are called to complete their respective tasks, the handling of the service request101may be successfully completed.

The network computer system100can include a reporting module120which interfaces with the individual programs to retrieve log information. The reporting module120can interface with individual programs82(or their respective logging data) to collect, or otherwise generate various types of logging information for each successful service request that is received and handled by the on-demand network service. In examples, the logging information can include (i) logging events131, which can be generated independently by each program82, to identify the occurrence of predetermined events (including events that coincide with the program having a failure) when the particular program performs its respective tasks, and (ii) trace routes133, which can identify each program that previously performed a task or otherwise handled a service request. In some examples, the reporting module120can generate trace routes133through analysis of the logging events131, using information that identifies the reporting program of a logging event and the accompanying time stamp(s) of when the event occurred. In examples, the logging information can record multiple time stamps, including a time stamp for when a reporting program starts and/or a time stamp for when a reporting program completes.

Additionally, in examples, the reporting module120can generate partial trace routes for failed service requests. As described in greater detail, such partial trace routes can identify, for example, the program that last performed a task in connection with the failed service request, and/or the program that last recorded a logging event in connection with being called to perform a task for a given service request.

In examples, the reporting module120can store logging events131and trace routes133in the reporting data store130for both successful and failed service requests. As described in greater detail, the logging information can be aggregated and analyzed to determine data path models for individual processes of the network service. Additionally, the reporting module120can structure service request records135to associate service requests101with logging events131and trace routes133which are generated for the particular service request.

The network computer system100can include a log analyzer140to analyze the trace logs. In examples, the log analyzer140includes an aggregation component142to aggregate the logging information generated by the programs82of the on-demand network service80. The aggregation component142can aggregate trace logs for successful service requests (e.g., service requests which the on-demand network service receives and successfully completes), to determine aggregations145reflecting (i) a count of each instance that each program used by the on-demand network service received and acted on an incoming service request, and (ii) a sequence position of each program at each instance the program was called or started for a service request (e.g., as reflected by time stamp when program started), relative to at least one other program82. As an addition or variation, the sequence position of each program may reflect the time when the program completes its task(s) (e.g., as reflected by time stamp when program completes) in connection to other programs that execute to handle a particular service request. The respective aggregations145can be conducted over a sufficient period of time to reflect an accurate statistical sample. Additionally, in some variations, the respective aggregations145can be continually updated.

In examples, the log analyzer140can analyze the respective aggregations145to identify a probability or likelihood of a given program is used in a particular sequence position for a given process or the network service. Through additional analysis, sequences as between at least two programs82of the group, as well as a frequency by which the particular sequence occurred, can be determined. Still further, in some examples, multiple data paths can be determined in the workflow of a given process, where each data path represents a sequenced grouping of the programs82which can combine to fulfill a given service request. The sequenced groupings may be determined based on prior handling of service requests, where the sequenced groupings each include data that defines the relative timing as to how individual programs of the particular grouping perform a task in the fulfillment of a service request.

For a given process of the network service, the log analyzer140can analyze the aggregations145to identify a relative order when a program82reported a logging event. The order in time of logging events can be used to determine a sequence among programs that cooperated to successfully handle a service request101that was handled by the process. Each sequence of successive programs82that execute to successfully handle a service request can be identified as a data path for the service requests. Through aggregation, multiple data paths can be identified, and multiple possible data paths may exist for handling incoming service requests101. Still further, in aggregating the trace logs, the aggregation component142can identify, for each of the programs82, sequencing information that includes (i) a sequence of the program executing (or being called by another program) relative to one or more other programs (e.g., each program82that calls it), and/or (ii) a probability that the program is called by, or otherwise initiated after, one or more other programs perform (or initiate performance) of a task.

Based on the statistical determinations of the counts for each program, the relative position of each program when used, and the determined sequencing information, the log analyzer140can generate a data path model155for a corresponding process of the network service. The generated data path model may be in the form of a graph data structure, reflecting the possible data paths of the on-demand network service, with each data path reflecting a statistical probability of occurrence. In examples, the log analyzer140includes a data path modeling component146that uses the aggregations145that are determined from the logging events131and/or trace routes133(as recorded with the reporting data store130) to develop and/or update one or more data path models155, which can be stored in a data path models store148. In examples, each data path model155can represent each program of the on-demand network service80as a program node, and the data path model can identify sequenced groupings of programs in accordance with a logical arrangement that reflects possible or alternative sequences amongst programs. In this way, each data path model155can identify one or more data paths, each of which identify the relative order that a particular program is called or otherwise executed as compared to another program (e.g., the calling program).

The aggregation component142of the log analyzer140can continuously generate aggregations145of logging information for individual processes of the network service. The aggregations145of successful service requests can be used by the data path modeling component146to update the respective data path models155of corresponding processes of the network service80.

In examples, the arrangement of each data path model155can be in the form of a graph data structure, with alternative sequences being identified by links stemming from individual nodes to linked nodes. The possible sequences by which programs execute, beginning with an initial program that receives a service request, can be represented in the data path model155through use of sequence information. The sequence information can be associated with each program node to link the program node to one or more other nodes (e.g., linked nodes). For a given program node, the sequence information can represent other program(s)82that call the program represented by the linked node. In this way, the sequence information can be associated with a probability of occurrence (e.g., a probability that a linked node will be the next event following a given node performing a task in connection with a service request).

As an addition or variation, each program node can also be associated with probability information. The probability information can reflect a probability that a given program will be called by or calls any of the programs which the sequence information identifies as potentially calling the given program. As an addition or variation, the probability information can reflect a probability that the program82will be called at some point during the handling of an incoming service request.

As an addition or variation, each data path model155can also identify a collection of possible data paths, with each data path identifying a succession of programs that can be called to successfully handle an incoming service request in accordance with a given sequence. In examples, each data path model155can represent programs as program nodes, with program nodes being arranged through links to reflect sequence information and probability information (e.g., probability that a given sequence will occur, or that a particular program will be called after another program performs a task). By arranging program nodes in accordance with sequence information and probability information, the data path models155provide expectations as to what events may occur immediately after a given program being initiated or performing a task associated with a service request. Specifically, the data path models155can identify a next event or program that is initiated by or in response to a given program receiving a call. Thus, for example, when a given program82executes to receive and act on data associated with the service request, the data path model155provides an expectation of what other program(s) may be called next.

According to some examples, network computer system100includes a failure analysis component160. The failure analysis component160can access or otherwise receive logging events131from the reporting data store130, when the logging events131reflect a failure by the service request. The logging events for failed service requests can include partial trace routes, identifying programs that received and acted on data of the service, prior to the service request failing. In examples, the failure analysis component160uses the partial trace route of the failed service request to identify an appropriate data path model155. As described, many factors can determine the sequence by which programs82execute to handle individual service requests. Accordingly, individual data path model155can vary from other data path models155, by which program node(s) represent the beginning event, meaning which program82first receives the service request101, or which set of programs82receive an incoming service request. The failure analysis component160can thus parse or inspect the partial trace route to identify an initial set of programs that received and handled the service request. The failure analysis component160can then match the first identified program of the trace route, or the first set of programs of the trace route, to initial program nodes of each of the data path models155to identify the data path model that is a match to the partial trace route.

The failure analysis component160can use the partial trace of the failed service request (as determined from the reporting data store130) to identify one or more events that preceded the failure of the service request101. In examples, the failure analysis component160may parse or otherwise inspect the partial trace log of the failed service request101to identify an event corresponding to the last program that was called to act on a failed service request. In some examples, the failure analysis component160can then map the data path of failed service request to program nodes of the selected data path model155. Based on the last program82that is identified in the trace log, the failure analysis component160can determine an expectation of what event should have occurred next. For example, the data path model155can be used to identify which of the programs82were likely to have followed the last identified program which received and acted on the failed service request. In this way, the failure analysis component160can utilize one or more data path models to (i) identify, from the trace route of the failed service request, the last program to successfully perform a task for the failed service request; and (ii) identify, from a corresponding data path model155, a candidate set of programs that are the cause of the service request failure. For example, the candidate set of programs can include each program that is a possible next event after a particular logging event(s) (as identified by the data path model155), and/or the program that is the most likely program to be called after the particular logging event(s) (the “likely next” program, as determined from the probability information). As an alternative or variation, if the data path model155shows that the “likely next” program is a branch to a possible series of other applications being executed, the failure analysis component160may mark the “likely next” program as the starting point for another analysis to pinpoint the cause of the failed service request.

In some examples, the failure analysis component160can identify one or multiple programs of a process where the failure of the process is manifested. Additionally, the failure analysis component160can pinpoint one or more likely program sources of the failure by the process. The determination of the failure analysis component160may be based on a variance or deviation as to how the network service80handled the failed service request as compared to how the network service80should have (or was expected to handle the service request). In some examples, the determination may be made by the failure analysis component160comparing the implemented workflow for the failed service request with the data path model for the process. Still further, in examples, the failure analysis component160can utilize the comparisons to pinpoint one or more programs of the failed process which are likely program sources of the failure.

Additionally, in some examples, the failure analysis component160may correlate the programs which are identified as the source of the failed service request with one or more underlying causes. The failure analysis component160may use for example, historical information and models to correlate programs that are pinpointed as the source of the failure with one or more potential root causes for the failed service request.

In examples, the failure analysis component160can use additional logic and processes to pinpoint the source of a service request failure, once an initial candidate set of programs is identified as being the source of failure. For example, in the case where the the trace route for a failed service request identifies the last program to be a branch to multiple other downstream programs, the failure analysis component160can query the reporting data store130for event logs generated by one or more of the downstream programs to identify logging events which are indicative of a program failure (e.g., downstream programs reporting errors), either with the particular program that the generated the queried logging information, or with another program that called or was called by the program which generated the logging information.

In some examples, the failure analysis component160can also employ inference logic to infer a candidate program as being the cause of a service request failure. The failure analysis component160can identify the source of the failed service request by, for example, querying reporting data store130to determine real-time logging information for programs that are downstream, as identified by the respective data path model155, from the program that is identified from the partial trace route of the failed service request as being the last program to handle the failed service request. The failure analysis component160can process the logging information of the downstream programs to identify indicators, such as a reduction in the number of event logs (e.g., as compared to an immediately prior time period) that any one of the downstream programs may generate. Indicators such as a determination that a given program has reduced its logging events can be mapped, using the respective data path model, to, for example, an upstream program (e.g., a program that may have called the downstream program in accordance with the respective data path). In the case where the particular downstream programs logging information indicates normal operations but less frequent use of the program, the failure analysis component160can infer that the calling program is the source of the failed service request.

The network computer system100can include a presentation component170to generate content to facilitate comprehension and/or manual mitigation of network errors in the execution of the programs82. In examples, the presentation component170provides an interactive mitigation tool172for use by administrators and operators of the on-demand network service80. An example of an mitigation tool is shown and described with examples ofFIG. 3AthroughFIG. 3C, as well as with examples ofFIG. 4AthroughFIG. 4F. As described with such examples, the mitigation tool172can implement one or multiple interfaces from which an operator can view information for mitigating the cause of failures on the on-demand network service80.

The presentation component170may interface with the reporting data store130to retrieve logging information, from which the presentation component170may can generate one or more types of presentations for the mitigation tool172. In examples, the presentation component170can generate a record view interface174(e.g., seeFIG. 3A,FIG. 4CandFIG. 4D) for failed service requests. When logging information from the reporting data store130indicates a service request has failed, the presentation component170can retrieve and present information from the request record135from the reporting data store130. In examples, the record view interface174can include (or make available for rendering) the logging events131of the failed request, the trace route133for the failed service request, and/or other information maintained with the service request record135.

As an addition or variation, the presentation component170can also generate one or more data path view interfaces176(e.g., seeFIG. 3B) for a failed service request. The data path view interface(s)176can include a graphic representation of the data path of the failed service request when it was handled, up to the point where the service request was deemed to have failed. The graphic representation can be generated from the partial trace route for the failed service request. The data path view interface176also utilize, for example, an output of the failure analysis component160to identify one or more programs that are deemed to be the source (or candidates thereof) of the service request failure. In such examples, the presentation component170can generate the data path view interface176to visually indicate candidate programs which are the source of a detected service request failure. For example, a portion of a rendered data path can be delineated and/or visually marked to indicate a candidate program or source for the failed service request. In some variations, the presentation component170can also combine the candidate programs with a query to the reporting data store130, in order to, for example, enable the operator to query for logging information about programs that are identified as being a source, or a potential source of a service request failure.

In variations, the data path view interface176can identify which program last received and acted on the failed service request, based on the output of the failure analysis component160. The data path view interface176can also be mapped to the relevant portions of the selected data path model to enable the operator to see information such as (i) a rendering of a portion of possible or expected data path for the failed service request, following the point where the failure occurred; and/or (ii) a visual rendering of sequencing and/or probability for the next event that was to have followed the last program identified from the trace route performing a task for the failed service request. In this way, the data path view interface(s)176can present the portion of the relevant data path model155, while identifying the program (or programs) that were the most likely cause of the failed service request.

As an addition or variation, the presentation component170can also generate a service request feed178(e.g., seeFIG. 3C) that presents groupings of service requests in a given time interval, including a current time interval (e.g., service requests received by the on-demand network service80in a current one-minute interval). The service request feed178can be filtered to location, service type, time interval and/or other factors. To generate the service request feed178, the presentation component170can query or otherwise interface with the reporting data store130to identify service request records for a given time interval (e.g., current time interval). The presentation component170can utilize an analysis of the trace route133and/or other logging events131to determine a status of each service request displayed as part of the service request feed178. By way of example, the status of individual service requests can be designated to be one of in progress, completed successfully, in progress and potentially having failed, or failed. Each service request can be separately identified in the feed, and each service request may also include a visual marker to indicate the current determined status of the service request. In this way, the service request feed178can be used to enable an operator to visually view and understand the current state of the on-demand network service80, without requiring the operator to have to perform tasks to locate whether potential failures exist with the on-demand network service80.

In some variations, the network computer system100may also include one or more programmatic processes (mitigation component180) to implement a mitigation or remediation step. The mitigation component180can, for example, interface with individual programs82of the on-demand network service80. The mitigation component180can respond to an output of the failure analysis component160by selecting and performing a remedial step, such as signaling a program that is identified as having failed to restart, or isolating a set of programs which are deemed to be candidates for a service request failure.

FIG. 2AthroughFIG. 2Dillustrates example portions of data path models for programs that handle a service request to a network service. The portions of the data path models represented byFIG. 2AandFIG. 2Dcan be generated by, for example, network computer system100, such as described with examples ofFIG. 1. Accordingly, reference may be made to elements ofFIG. 1for purpose of illustration.

InFIG. 2A, a grouping of programs is shown, reflecting an expectation of events with respect to the handling of an incoming service request by the network service80(e.g., by a particular process of the network service80). In an example shown, individual program nodes210-216represent programs82of the on-demand network service80which execute in accordance with a particular sequence or timing, relative to other programs, in order to properly handle the service request. The sequencing of the individual programs82may be based on, for example, the relative start and/or completion times of the respective programs. The determination of expected events can be based on prior observation, such as aggregation of trace routes for successful service requests. As shown byFIG. 2A, each program node210-216is associated with sequencing information and probability information. The sequencing information (represented as a connector) can identify a relative position in time of each program being called, or otherwise executing to act on a service request, relative to one or more other programs.

The probability information can identify a likelihood that a particular sequence will take place. For example, inFIG. 2A, a sequence of program212(“Prog. K”) following program210(“Prog. A”) is certainty, while a sequence in which either “Prog. J2” or “Prog. J3” follows “Prog. K” is relatively small (e.g., 12% and 8%) as compared to the sequence where “Prog. J” following “Prog. K”. In variations, other probability information can also be maintained and used with given data path models, such as the probability that “Prog. J3” will act on the service request when the service request is first received and acted on by program210(Prog. A″).

For a given data path model, some programs can also be identified as being dependent on other programs. InFIG. 2A, for example, one of the programs214(“Prog.J”, “Prog.J2” and “Prog. J3”) may be called by program212(“Prog. K”), which may in turn be called by program210(“Prog. A”). In a given data path model155, the programs that may be called by other programs can be said to be dependent programs, meaning dependent on the calling program. Additionally, for a given data path model155, a succession of programs (e.g., including dependent programs) can act on an individual service request in accordance with a sequence, resulting in a final program that completes the tasks of the service request. The succession of programs that act on individual service requests in accordance with an expected sequence can define one of multiple possible data paths (e.g., data paths211,213) for a service request that is to be successfully handled under a given data path model155.

In examples, the sequencing information can be represented by the program's position with respect to an overall sequence of dependent programs that act on the service request until its completion. In an example ofFIG. 2A, a program sequence position can range from position0to position N, where position0can represent the program82that first receives the incoming service request, and position N representing the program that is the last to be called or otherwise executed before the service request is deemed complete.

FIG. 2Billustrates use of different data path models155to determine expectations for a common type of service request. InFIG. 2B, the initial program82having the position0(e.g., the first program82of the group to act on an incoming service request) is the program that occupies position1in the data path model155ofFIG. 2A. Thus, each data path model155can be characterized by one or more initial programs that receive and act on incoming service requests. The particular program82that is triggered to initiate handling of incoming service request can vary based on factors, as discussed, such as information specified by the load on the network service at the particular time, as well as the availability of individual programs82.

FIG. 2CandFIG. 2Dillustrate alternative scenarios by which a relevant portion of an applicable data path model155can be used to determine a cause of a network failure (e.g., service request fails). InFIG. 2C, for example, the failure analysis component160can use the partial trace route of a failed service request to identify the last program that acted on the service request as being “Prog. J”. The relevant portion of the applicable data path model155can indicate alternative possible sequences—to “Prog. K” (75%) and to “Prog. L” (25%). The indication can narrow the possibility of which program82in the grouping caused the failure by the service request to “Prog. K” and “Prog. L”. Any additional analysis needed to determine which program cause the service request failure can now be initiated on “Prog. K” as the most probable choice, followed by “Prog. L”.

InFIG. 2D, the failure analysis component160can perform additional analysis as described by retrieving logging information relating to a downstream program (“Prog. M”) of “Prog. L”. The failure analysis component160can retrieve, for example, logging information from the reporting data store130for “Prog. M” (e.g., query for logging events relating to “Prog M”), in order to determine whether there has been a recent drop in logging events from that program. As an addition or alternative, the failure analysis component160can query to determine whether Prog. M has is logging any information, or whether another program that is dependent on Prog. M is reporting a log error that indicative of that program not receiving calls.

As illustrated by examples ofFIG. 2AthroughFIG. 2D, the data path models155can be used to facilitate evaluation of the network service, in situations such as when service request are detected to fail. By generating multiple data path models155to represent alternative sequences for service request handling, a relevant data path model155can be identified for a failed service request. The relevant data path model155for a failed service request can be used to identify a set of expectations as to what programs were to execute for the successful handling of the service request, and such expectations may specifically identify the sequence by which at least some of the programs82were to execute. When the relevant data path model155is combined with, for example, information determined from trace routes, the logical juncture where the failure occurred can be identified and correlated to a specific program or set of programs82.

Example Operator Interfaces

FIG. 3AthroughFIG. 3Cillustrate an example of an operator user interface to facilitate mitigation of a network service failure.FIG. 4AthroughFIG. 4Dillustrate another example of an operator user interface to facilitate mitigation of a network service failure. Example interfaces as shown byFIG. 3AthroughFIG. 3C, andFIG. 4AthroughFIG. 4D, can be displayed on, for example, a terminal or workstation of an administrator or other operator who is delegated the task of monitoring performance and/or mitigating failures when the network computer system100is deployed to implement a network service. Accordingly, example interfaces as shown byFIG. 3AthroughFIG. 3CandFIG. 4AthroughFIG. 4Dcan be implemented through network computer system100and provided as part of, for example, an interactive mitigation tool172(seeFIG. 1) for enabling the operator to mitigate failures of an on-demand network service. Accordingly, in describing example operator interfaces withFIG. 3AthroughFIG. 3DandFIG. 4Athrough FIG.4D, reference to elements ofFIG. 1may be provided to illustrate relevant functionality and components for implementing interactive interfaces.

InFIG. 3A, an example interface310can be generated by the presentation component170to show a record view of a service request that was identified as failing. The interface310can identify a failed record by type information303. Additionally, the interface310can identify the priority level305of the detected failure. Depending on implementation, the service request can be a test case or the result of a live production. Information about the detected failure can be provided through, for example, analysis of the logging information associated with the corresponding service request record135.

The interface310can further provide active elements, such as links312and314, to enable different aspects of the failed service request to be viewed by an operator. In examples, the presentation component170can generate a data path view for a given request, and further enable an operator to view the data path visualization through selection of a corresponding active element (e.g., direct link312) that is displayed on the interface310. In this way, an operator can use the link312to access a data path view of the failed service request, as illustrated and described with an example ofFIG. 3B.

Additionally, the presentation component170can generate direct link314to provide an operator with access to different parts of a corresponding service request record135. For example, the direct link314can enable the operator to view a partial trace route for the failed service request. The partial trace route can, for example, display logging information for individual programs82which successfully handled the failed service request and/or the logging error generated by the program82where the failure occurred.

FIG. 3Billustrates an example interface that can be generated in response to the operator selecting to view a data path of a failed service request. The interface320can be generated by the presentation component170as a response to the operator's selection of the link312. As shown, the interface320displays a data path representation322of a failed service request. In an example ofFIG. 3B, the data path representation322is a partial view of the total data path for the particular request, with the partial view focusing on the portion of the failed request which is most-relevant to the failure analysis (e.g., the group of programs preceding the detected failure). The network computer system100can include functionality to generate the data path representation322by, for example, parsing logging events131and trace routes133of a failed service request to identify individual programs82which handled the service request. The data path representation322can represent individual programs82as nodes321, and the data path view322can display a series of nodes321to represent the sequence of programs which handled the failed service request. The data path representation322can also identify candidate program(s) which are a likely source of the service request failure. As shown by an example ofFIG. 3B, the identified candidate programs can be reflected on the data path representation322as corresponding problem nodes325,327. The problem nodes325,327can be displayed to be visually distinct from the other nodes of the data path representation322.

InFIG. 3Cillustrates an example interface that can be generated to provide status information about service requests received in a given time interval. In examples, the presentation component170can generate the service request feed330to display a service request feed330in which separate representations of multiple incoming service requests331are displayed concurrently. The request feed332can present individual requests331with corresponding status indicators, where each status indicator indicates whether (i) a service request is being handled without error or failure, (ii) a service request is being handled with potential error or failure (e.g., service request taking too long to handle by one or more programs of the respective data path), and/or (iii) a service request has encountered an error or failure. As shown, the status of the service indicators can be reflected by color coding (e.g., green, yellow and red). As described with other examples, other types of status indicators can also be determined and indicated with individual service requests of the service request feed330.

In examples, the successful service requests can be aggregated to determine and update one or more data path models155, while failed service requests can be analyzed to determine the cause of the failure, using an application data path model155. The visual indication of status on the request feed can provide another mechanism to facilitate an operator to readily identify the occurrence of errors or failures in individual service requests.

FIG. 4Aillustrates an example operator user interface400that provides a nodal representation of the individual workflows for multiple processes of a network service. With reference to an example ofFIG. 1, the operator user interface400may be generated by the presentation component170, in connection with the network computer system100implementing the network service80in any one of multiple possible environments (e.g., production environment where users are actively using the network service80, test environment, or simulation). In examples, the operator user interface400can be rendered on a terminal of a network operator to facilitate the network operator in viewing, analyzing and/or mitigating against errors that may occur with individual processes of the network service. In some examples, the network operator may also view remedial actions (e.g., as determined or implemented by the failure analysis component170) which are suggested and/or implemented by the network computer system100when service requests are detected as failing. The operator user interface400may provide information that reflects a current or real-time status of the network service80, such as with respect to service requests which were received by the network service80within a given duration of time (e.g., within a prior 10 second or 1 minute interval).

In an example ofFIG. 4A, the operator user interface400includes a process selection panel410and a nodal view420. The process selection panel410can include process selection features412, each of which are individually selectable by a network operator to generate a respective nodal view420of the process as currently implemented for the network service in a given environment. By way of example, an on-demand transport delivery service can include separate sets of processes for different classes of users (e.g., transport providers, deliverers, service requesters, etc.), and for different types of services (e.g., pooled transport, luxury transport, food delivery, etc.). In examples, the respective workflow implemented by each process can be triggered or otherwise initiated by a service request generated from an external device (e.g., user device), such as by a given user (e.g., service provider or driver, service requester or rider etc.) interacting with their respective user device to utilize the network service80(e.g., using a service application running on the respective user device). For example, a process selection feature412may be selectable by the network operator to generate the nodal view420for the process. As described with examples, the nodal view420of a selected process may represent one or more data path models that identify components (e.g., microservices) of the process, as well as sequencing amongst components in the workflow of the selected process.

In examples, each process, the corresponding nodal view420can provide a visualization that is based on the data path model for that process. Accordingly, the nodal view420for each process can represent each program of a corresponding workflow as a node, with the nodes being arranged to reflect an expected sequence (or sequences) by which the corresponding programs of the workflow will execute when a service request is received and successfully handled. In examples, the nodal view420can also reflect alternative sequences and variations amongst the programs of the workflow, with individual nodes of the respective workflows being associated with one or more probabilities. As described with other examples, a probability associated with a node of a nodal view420may represent a likelihood that the program will be called by another program of the workflow. In variations, a probability associated with a program may represent a likelihood that the program will call another program that is identified by a linked downstream node when a given service request is received and successfully handled.

In examples, the nodal view420can be provided with service request information to depict the handling of individual service requests by a given process of the network service. For example, the nodal view420can identify the programs that were used to handle individual service requests, including service requests which were successfully handled and service requests which experienced failures. For each service request, the nodal view420can identify the specific programs which executed to handle an aspect of the workflow, as well as sequence information (e.g., based on the program's respective start time or completion) for the program. In generating the nodal view420, the network computer system100can also compare the detected workflow (e.g., programs and sequence in which programs were executed) with the nodal representation of a data path model. For successful service requests, the comparison may identify, for example, programs and microservices which both executed and did not execute.

As an addition or alternative, the nodal view420can include service request information that visually represents information about the handling of a service request by a selected process. The nodal view420can, for example, include service request information which identifies the sequencing amongst programs and microservices which successfully handled one or more service requests for the selected process. Still further, the service request information can be provided with the nodal view420to provide information about service requests which failed or otherwise reported errors. In providing information about individual service requests, some examples further provide for the nodal view420to include service request information as visual indicators which identify components (or microservices) which (i) executed without reporting errors (or as expected), (ii) did not execute, and/or (iii) reported an error or failure in connection with failure of the service request. For example, the nodal view420can generate the nodal view420of a given process, with each node being color-coded or visually marked to reflect anyone of multiple possible outcomes for a corresponding program or microservice (e.g., executed without error, executed with error, did not execute, etc.) of the given process.

FIG. 4Billustrates the operator user interface400configured to identify information about service request failures which are detected in the implementation of the on-demand network service. In one implementation, the operator user interface400can display information determined from programmatic analysis of individual service request failures, with the operator user interface400being configured to depict information for individual service requests and/or aggregates of service requests. In variations, the operator user interface400can display information determined from analysis of multiple service requests, such that an instance of the operator user interface400can display information for multiple service request failures at one time (e.g., service request failures which occurred over a given time interval, such as preceding 5 minutes).

In an example ofFIG. 4B, the nodal view420includes service request information that visually marks individual nodes to reflect a determination as to whether the corresponding program executed, did not execute or partially executed, in connection with a specific service request. Still further, in examples the service request information may reflect whether a program reported an error or failure that is not otherwise expected when the process successfully handles the service request. In examples, the nodal view420may integrate service request information for individual requests in a variety of ways, such as through visual markers or text blocks. By way of example, the nodes422of nodal view420can, for a given service request, include (i) a first visual indicator (e.g., white or clear fill) to represent a determination that the corresponding program was determined to have successfully executed, (ii) a second visual indicator (e.g., solid fill) to represent a determination that the corresponding program was determined to not have executed, and (iii) a third visual indicator (e.g., nodal outline) to represent a determination that the corresponding program was determined to have failed and/or reported an error that is not otherwise expected when a service request is successfully handled by the process. In variants, the nodal view420can utilize additional or alternative markers to indicate other determinations of the network computer system100, such as a determination that one or more programs of the process executed in a sequence that is not expected based on the data path model for the process.

As an addition or variation, the nodal view420may include node-specific messages424that are probative of an underlying cause for a failed service request. With further reference toFIG. 1, the node-specific messages424may be generated by the presentation component160, based on an output of the failure analysis component160. In this way, the node-specific message424provides an example of how the network computer system100can pinpoint a specific program, or set of programs, as a source (or likely source) of a failure in the corresponding process. Accordingly, in an example ofFIG. 4B, the node-specific messages424can display, for example, information that reflects a determination about an outcome or state of a particular program of the process. In examples, the node-specific message424may be visually overlaid or linked to a particular node422to reflect a determination that the associated program did not behave in a manner that was expected, based on the data path model for the process. For example, as shown byFIG. 4B, the node-specific message424may render with a particular node422to reflect a determination that the program associated with the node did not execute but was expected to have executed (e.g., based on a statistical probability).

FIG. 4Cillustrates an example in which the operator user interface400is configured to provide a record view interface from which service request information can be viewed and analyzed for individual programs of a process. As shown by an example ofFIG. 4C, a node422(e.g., representing a first-in-time program of the workflow) is selected from a nodal view420to provide a record view interface430from which service request information for one or more service requests can be viewed. In examples, the record view interface430that is provided in connection with a selected node422displays service request information that is generated or otherwise related to the corresponding program of the selected node. The record view interface430can, for example, be generated by the presentation component170to display logging events and logging information from the from the reporting data store130for the given service request and the program associated with the selected node422. The record view interface430can, for example, render service request information from the corresponding service request record135, including the logging events, trace route, and/or other information maintained with the record.

FIG. 4Dillustrates an example in which the operator user interface400provides multiple record view interfaces430in connection with rendering the nodal view420for a given service request. In an example, a failed service request can be detected and indicated visually by, for example, marking the corresponding process and one or more programs of the selected process which generate logging events that are indicative of an error or failure having occurred in the respective workflow. Additionally, in examples, the operator user interface400can provide a record view interface430for multiple logs of the selected process. For example, the operator user interface400can provide record view interfaces the selected program or process. The record view interfaces430for the multiple logs can be rendered at one time. Among other benefits, examples as described enable a network operator to view and analyze logging events from multiple components or programs of a failed process at one time, to better correlate logging events from programs of the process which report failures in their respective logging information.

Additionally, as shown by an example ofFIG. 4D, the operator user interface400may be also configured to provide service request information that includes indicators reflecting a state of individual processes of the network service. In examples, the process selection feature412for a failed process may be marked to reflect the detection of an open or new service request failure. With reference toFIG. 1, the indication of the failed process may be generated by the presentation component170, based on an output of the failure analysis component160. In examples, a service request failure may be open when, for example, the service request failure has not been viewed, or alternatively, resolved. Similarly, a new service request may reflect a service request failure that occurred in a preceding time interval (e.g., prior minute). When a service request failure is detected for a given process, the process selection feature412for the process may be provided with a visual marker that reflects the state of the process, with the state value represents (i) whether any open or new service request failures are associated with the corresponding process, and/or (ii) a number of open or new service requests. In a variation, the process selection feature412of the corresponding process410may be provided with a visual marker (e.g., the process selection feature412may be color-coded) to reflect the state of the corresponding process. In another variation, process selection feature412may be provided with a numerical marker415which signifies the occurrence of one or more open or new service request failures.

Example Methods

FIG. 5Aillustrates an example method for determining a data path model that identifies one or more expected data paths for handling service requests to the network service.FIG. 5Billustrates an example method for diagnosing a failed service request using a data path model. Example methods such as described by examples ofFIG. 5AandFIG. 5Bmay be described with reference to elements ofFIG. 1, for purpose of illustrating suitable components or elements for performing a step or sub-step being described.

InFIG. 5A, the aggregation component142aggregates trace routes of successful service requests (510). The aggregation may be performed over a relevant time period, with sufficient number of trace routes being aggregated to determine a statistical sample that is representative of the various data paths used by the network computer system100to handle service requests.

The log analyzer140can analyze the aggregation to determine data path model information (520). The data path model information can identify sequence data, where the sequence data identifies two or more programs that are expected to be sequenced with respect to one another. The sequence information can also identify successive programs that act to receive and act on the service request until the handling of the service request is completed (522).

Additionally, the log analyzer140can use the aggregation of the trace routes to determine probability information, where the probability information identifies the likelihood that particular sequences may occur, or that a program may be called or otherwise execute to handle a particular service request at a particular time (524).

In examples, one or more data path models are generated to represent expected events, with respect to programs82handing requests to the on-demand network service80(530). Through monitoring and analysis of logging information, derived data path models can be updated repeatedly over time, to provide a more accurate depiction of expected events which may follow receipt of a service request. In some examples, the data path models can be rendered graphically (532), with program nodes representing individual programs of the on-demand network service80, with sequence data (along with probability data) being represented by links or connectors between program nodes. As an addition or variation, the data path models can be used to perform failure analysis (534), such as described with other examples (e.g., seeFIG. 2CandFIG. 2D).

With reference to an example ofFIG. 5B, a service request failure is detected by the network computer system100(550). In examples, the service request failure can be detected automatically, by, for example, the failure analysis component160analyzing logging information, or other programmatic resources which can scan and parse logging events to determine the request failures.

The failure analysis component160can analyze a trace route of the failed service request for analysis (560). The analysis may include, for example, a determination of the last program that received and acted on the service request (562). The network computer system100can use the data path model to identify a candidate set of programs, or possible next events following the last program that received and acted on the service request.

From the determination, the network computer system100can indicate a cause of the service request failure (570). In examples, the network computer system100can identify one or more candidate programs that are deemed to be the cause of the service request failure on an interface of an operator terminal. The network computer system100can, for example, generate an interface that illustrates the candidate programs which are the likely cause of the service request failure. In variations, the network computer system100can implement logic to identify which of a candidate set of programs that are the cause of the service request failure.

Still further, in some variations, the network computer system100can implement logic to mitigate an identified failure. For example, the network computer system100can implement logic to programmatically implement a remedial measure, where data paths that utilize the problematic program are minimized.

FIG. 6illustrates a computer system on which one or more embodiments can be implemented. A computer system600can be implemented on, for example, a server or combination of servers. For example, the computer system500may be implemented as part of network computer system100, as described with an example ofFIG. 1. Likewise, the computer system600can implement methods such as described with examples ofFIG. 5AandFIG. 5B.

In one implementation, the computer system600includes processing resources610, memory resources620(e.g., read-only memory (ROM) or random-access memory (RAM)), a storage device640, and a communication interface650. The computer system600includes at least one processor610for processing information stored in the memory resources620(e.g., main memory), such as provided by a random-access memory (RAM) or other dynamic storage device, for storing information and instructions which are executable by the processor610. The memory resources620may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by the processor610. The computer system600may also include the memory resources620or other static storage device for storing static information and instructions for the processor610. The storage device640, such as a magnetic disk or optical disk, is provided for storing information and instructions.

The communication interface660enables the computer system600to communicate with one or more networks (e.g., cellular network) through use of the network link680(wireless or a wire). Using the network link680, the computer system600can communicate with one or more computing devices, specialized devices and modules, and one or more servers. The executable instructions stored in the memory630can include instructions642, to implement a network computer system such as described with an example ofFIG. 1. The executable instructions stored in the memory620may also implement a method, such as described with one or more examples ofFIG. 6AandFIG. 6B.

As such, examples described herein are related to the use of the computer system600for implementing the techniques described herein. According to an aspect, techniques are performed by the computer system600in response to the processor610executing one or more sequences of one or more instructions contained in the memory620. Such instructions may be read into the memory620from another machine-readable medium, such as the storage device640. Execution of the sequences of instructions contained in the memory620causes the processor610to perform the process steps described herein. In alternative implementations, hard-wired circuitry may be used in place of or in combination with software instructions to implement examples described herein. Thus, the examples described are not limited to any specific combination of hardware circuitry and software.

Conclusion

Although examples are described in detail herein with reference to the accompanying drawings, it is to be understood that the concepts are not limited to those precise examples. Accordingly, it is intended that the scope of the concepts be defined by the following claims and their equivalents. Furthermore, it is contemplated that a particular feature described either individually or as part of an example can be combined with other individually described features, or parts of other examples, even if the other features and examples make no mentioned of the particular feature. Thus, the absence of describing combinations should not preclude having rights to such combinations.