Integrated remediation system for network-based services

This disclosure describes automatically collecting, analyzing, and remediating operational issues with respect to systems executing within a network. For example, a service provider network may include a monitoring service may generate notifications related to operational issues upon detection of operational issues within a system executing within the service provider network. The monitoring service may provide one or more notifications related to an aggregation service that may aggregate the one or more notifications into a standardized format. Contextual information related to the operational issues may be automatically gathered by an analytics service, which may analyze the contextual information to determine a potential cause of the operational issues. Based on the potential cause, a remediation service may automatically remediate the operational issues.

BACKGROUND

Service providers offer cloud-based services via service provider networks to fulfill user's computing-service needs without the users having to invest in and maintain computing infrastructure required to implement the services. These service providers may provide network-based computing resources and functionality to implement various types of cloud-based services, such as, for example, scalable-storage services, computer-processing services, and so forth. In some examples, developers may utilize services offered by the service provider to run the systems and/or applications using virtual services (or “instances”) provisioned on various configurations of hardware-based resources of a cloud-based service.

When users operate or execute systems and/or applications within service provider networks, various types of issues may arise. These issues may generally be referred to as operational issues or operational items. Operational issues may arise with respect to the executed systems and relationships among resources used within the service provider network, interaction with third parties external to the service provider network, flaws within the systems, hardware issues, etc. Currently, in order to solve such operational issues, engineers or technicians need to manually go through various consoles and/or manuals in order to gather information, data, metrics, etc., to determine potential causes for the operational issues and then to determine potential remedial actions.

DETAILED DESCRIPTION

This disclosure describes, at least in part, techniques and architectures for automatically collecting information related to operational issues with respect to systems executing within a network, automatically analyzing the information, and automatically remediating the operational issues. For example, in accordance with various configurations, a service provider network may include a monitoring service that may monitor and/or gather information related to systems and/or applications executing within the service provider network. As used herein, when reference is made to a system executing within a service provider network, the use of the term “system” may also refer to applications that may be executing within a service provider network. The monitoring service may generate notifications related to operational issues upon detection of operational issues within a system executing within the service provider network. The notification is thus a red flag or ticket that indicates an operational issue that needs to be addressed, e.g., remediated.

In particular, the techniques and architecture described herein provide for the generation of one or more notifications with respect to one or more operational issues or events discovered by the monitoring service with respect to a system executing within the service provider network. In configurations, the notifications may be forwarded from the monitoring service to an aggregation service provided by the service provider network. The aggregation service may aggregate received notifications into a standardized or structured format. The aggregation service may also aggregate multiple notifications into a single notification. For example, multiple related notifications may be aggregated into a single, “parent” notification. Thus, the related individual notifications may be referred to as “children” notifications. Upon receipt of the notification, the aggregation service may provide the notifications to an analytics service provided by the service provider network.

The analytics service may gather contextual information related to the operational issue(s). For example, the analytics service may gather contextual information related to one or more of metadata related to resource usage of the system during execution within the service provider network, operational metrics related to execution of the system within the service provider network, operational logs related to execution of the system within the service provider network, resource changes related to execution of the system within the service provider network, an overall health or status of the system, relationships between the system and resources of the service provider network, security details related to execution of the system within the service provider network, etc.

The analytics service may evaluate the contextual information based upon rules and/or parameters related to execution of the system within the service provider network. The rules and parameters may relate to, for example, usage thresholds related to resource usage within the service provider network, logical relationships with respect to the system and the resources of the service provider network, interactions (e.g., communications) between the system and third parties external to the service provider network, etc. The analysis of the contextual information may provide one or more potential causes of the operational issue(s).

Based upon the analysis of the contextual information, the results may be provided to a remediation service of the service provider network. The remediation service may evaluate the results from the analytics service. Based upon the evaluation, the remediation service may determine one or more potential actions that may be performed to remediate the operational issue(s). For example, the remediation service may inform a user executing the system within the service provider network regarding potential remediation actions that the user already possesses or of which the user is already aware. For example, the user may possess a “runbook” or “playbook,” e.g., that describes one or more potential remediation actions for remediating the operational issue, e.g., a remediation flow. For example, the runbook may be in the form of automation documents that may address the operational issue(s). In configurations, the runbook may specify that for operational issue A, execute operation X. However, the user may overrule the runbook and decide to execute remedial operation Y to address operational issue A.

In configurations, the remediation service may recommend to the user to follow the runbook. The recommendation may be based upon a threshold of certainty. For example, if the remediation service is ninety percent sure that the remediation flow will address the operational issue, then the remediation service may inform the user to apply the remediation flow. In configurations, the threshold of certainty may be configurable by the user and/or an operator of the service provider network. In configurations, the user may configure the remediation service, e.g., provide a user preference, such that for each instance of operational issue A, remedial operation Y is executed.

In configurations, the remediation service may inform or recommend to the user to execute a script as a remedial operation to address the operational issue. Multiple scripts may be generated by the operator of the service provider network to address various operational issues. In configurations, the user may generate one or more scripts and may share such scripts with the operator of the service provider network to share with other users who may encounter similar operational issue(s).

Since many of the same operational issues may occur among multiple systems executed by the service provider network by multiple users, the analytics service may utilize machine learning to recognize various situations and/or patterns of contextual information that occur and lead to various operational issues. Based upon the patterns recognized by analyzing the contextual information, the analytics service may quickly determine potential causes for various operational issues. Based upon the pattern recognition, the remediation service may inform and/or recommend various remedial actions that may be performed, either automatically or manually, by users executing systems within the service provider network.

FIG.1illustrates a system-architecture diagram of an example environment100that includes an example service provider network102. The service provider network102may comprise servers (not illustrated) that do not require end-user knowledge of the physical location and configuration of the system that delivers the services. Common expressions associated with the service provider network may include, for example, “on-demand computing,” “software as a service (SaaS),” “cloud services,” “data centers,” and so forth. Services provided by the service provider network102may be distributed across one or more physical or virtual devices.

As may be seen inFIG.1, the service provider network102includes a processing service104and a storage service106. As noted above, other services may be provided by the service provider network102. A user108, which may be an individual, a group of individuals, an entity, an organization, a business entity, etc., may obtain or purchase services via one or more user device(s)110, e.g., electronic devices, from an operator of the service provider network102. Thus, the user108may access the processing service104and/or the storage service106(as well as other services) using the one or more user devices110. For example, the user108may execute a system112within the service provider network102using the one or more user devices110. The system112may execute within the service provider network102utilizing various services, e.g., the processing service104and the storage service106, as well as other services not illustrated inFIG.1. For example, the system112executed by the user108within the service provider network102may involve instantiation of one or more virtual machine instances (not illustrated) via the processing service104. The user108may also, via execution of the system112, store and retrieve data via the storage service106.

In configurations, the service provider network102includes a monitoring service114that monitors the system112. When an operational issue or event arises with respect to execution of the system112that is detected by the monitoring service114, the monitoring service114may generate an operational issue notification116. For example, an operational issue may arise when a software patch failure occurs with respect to execution of the system112. Upon detection of the software patch failure, the monitoring service114may generate an operational issue notification116. In configurations, the monitoring service114may be a service external to the service provider network102, e.g., the monitoring service114may be a service provided by a third-party.

The operational issue notification116may be forwarded to an aggregation service118provided by the service provider network102. In configurations, when an operational issue arises, the user108may generate the operational issue notification116and provide it to the aggregation service118. In configurations, one or more operational issues may occur at one time and, thus, multiple operational issue notifications116or a single notification116may be provided to the aggregation service118depending on the configuration of the monitoring service114and/or the aggregation service118.

The aggregation service118may aggregate the operational issue notifications116received into a standardized or structured format or ticket120for a standardized console that may be based on the service provider network102. This may allow for operational issue notifications116to be received from various sources and still be processed. Thus, the operational issue notification116is an evolving item that evolves into the ticket120, e.g., the ticket120is the operational issue notification evolved into the ticket120.

Additionally, if multiple operational issue notifications116are provided to the aggregation service118at the same time, the aggregation service118may aggregate the multiple operational issue notifications116into a single operational issue notification116, e.g., a single ticket120. For example, multiple related operational issue notifications116may be aggregated into a single, “parent” notification. Thus, the related individual issue notifications may be referred to as “children” notifications. Furthermore, subsequent children operational issue notifications that are received may be associated with a corresponding aggregated parent operational issue notification. Thus, operational issue notifications116may be received from various sources, e.g., the monitoring service114, other monitoring services (e.g., external monitoring services), manually from the user108, etc. The received operational issue notifications116may then be aggregated into the standardized format to initiate the process of analyzing and remediating the operational issue(s) via the ticket120.

Additionally, in configurations, the aggregation service118may receive an operations issue notification116and may apply rules in order achieve deduplication and define parent operational issue notifications. This can help reduce noise caused by multiple alarms. Furthermore, machine learning, artificial intelligence and/or rules may be applied to help proactively determine other resources and/or services that may be affected by an issue that led to an operational issue notification116.

The aggregation service118may provide the ticket120to an analytics service122provided by the service provider network102. The analytics service122may gather contextual information124related to execution of the system112within the service provider network102. The contextual information124may be gathered for analysis in order to determine a potential cause of the operational issue. For example, contextual information may be gathered from the processing service104, the storage service106, the system112, other services (not illustrated) provided by the service provider network102, third parties external to the service provider network, the one or more user devices110, etc.

In configurations, the contextual information124may include, for example, information related to one or more of metadata related to resource usage of the system112during execution within the service provider network102, operational metrics related to execution of the system112within the service provider network102, operational logs related to execution of the system112within the service provider network102, instance logs related to execution of the system112within the service provider network102, configurations that occurred around the time of the operational issue, resource changes related to execution of the system112within the service provider network102, an overall health or status of the system112, relationships between the system112and resources of the service provider network102, security details related to execution of the system112within the service provider network102, timing of the operational issue, etc. In configurations, scripts may be executed to gather the contextual information124. Additionally, in configurations, the user108may indicate what contextual information124is important for analyzing operational issues. Contextual information124may also be obtained by checking with various services of the service provider network102.

The analytics service122may analyze the contextual information124based upon rules and/or parameters related to execution of the system112within the service provider network102. The rules and parameters may relate to, for example, usage thresholds related to resource usage within the service provider network102, logical relationships with respect to the system112and the resources of the service provider network102, interactions (e.g., communications) between the system112and third parties external to the service provider network102, etc.

Upon completion of analysis by the analytics service122, the analytics service122may provide results126to a remediation service128. In configurations, the results126may be provided to the remediation service128along with the ticket120. The remediation service128may evaluate the results126to determine one or more potential remedial actions130that may be performed to remediate the operational issue that resulted in the operational issue ticket116and corresponding ticket120. In configurations, the remediation service128may provide information136that includes one or more remedial actions130that may be one or more potential solutions to the operational issue. The one or more remedial actions130may represent a further evolution of the original issue notification116and corresponding ticket120.

In configurations, an account138of the user108may include a runbook134that includes remedial actions, where one or more of the remedial actions may be performed as a remedial flow to correct operational issue(s). In configurations, the runbook134may be included at one or more user devices110of the user108. Thus, in configurations, as a remedial action130, the remediation service128may provide information136to the user108regarding the runbook134and/or remedial actions that are included within the runbook134. For example, the runbook134may be in the form of automation documents that may address operational issue(s). Thus, the runbook134may specify that for operational issue A, execute remedial operation X. However, in configurations, the user108may overrule the runbook134and decide to execute remedial operation Y to address operational issue A. Such an overruling by the user108may be based on user preference, costs, timing, etc. In configurations, the user108may configure the remediation service128, e.g., provide a user preference, such that for operational issue A, always execute remedial operation Y.

In configurations, the remediation service128may provide a recommendation138to the user108to follow or execute the runbook134. The recommendation138may be based upon a threshold of certainty. For example, if the remediation service128is ninety percent sure that a remediation flow of the runbook134will address the operational issue, then the remediation service128may recommend that the user108apply the remediation flow provided by the runbook134. Other levels of certainty may be acceptable (e.g., different percentages) and thus, ninety percent is not meant to be limiting. In configurations, the threshold of certainty may be configurable by the user108and/or an operator of the service provider network102.

In configurations, the remediation service128may automatically remediate the operational issue. For example, the remediation service128may automatically execute a script140that addresses the operational issue with respect to execution of the system112within the service provider network102. In configurations, the user108may specify that certain scripts140may be automatically performed to address operational issues.

Multiple scripts140may be generated by an operator of the service provider network102to address various operational issues. In configurations, the user108may generate one or more scripts140and may share such scripts140with the operator of the service provider network102to share with other users who may encounter similar operational issue(s) when executing systems112within the service provider network102.

In configurations, a remedial action130may include further investigating the operational issue. For example, the analytics service122may not be able to determine a potential cause of the operational issue. Thus, additional contextual information124may need to be gathered. For example, some contextual information124may need to be gathered from third parties. Additionally, if a remedial action130is implemented and the operational issue is not resolved, then more contextual information124may be gathered and analyzed to determine additional possible cause(s) and potential remedial action(s)130.

In configurations, the remedial action130may be to inform the user108that there is nothing to be done at this time. For example, the remediation service128and/or the analytics service122may determine a root cause for the issue that resulted in the generation of the operational issue notification116. The root cause may be that a server is down and the issue will be resolved once the server is brought back online or a new server is provided by the service provider network102. Thus, the remedial action130may be to inform the user108that there is nothing wrong with the user's system112and that the root cause needs to be addressed by the operator of the service provider network102.

In configurations, the remediation service128may determine that the issue that resulted in the creation of the operational issue notification116has been remedied or fixed. The remediation service128may inform the originator of the operational issue notification116that the issue has been resolved. For example, the remediation service128may inform the monitoring service114that the issue has been resolved.

Since many of the same operational issues may occur among multiple systems112executed by the service provider network102by multiple users108, the analytics service122may utilize machine learning to recognize various situations and/or patterns of contextual information124that occur and lead to various operational issues. Based upon the patterns recognized by analyzing the contextual information124, the analytics service122may quickly determine potential causes for various operational issues. Based upon such pattern recognition, the remediation service128may inform and/or recommend various remedial actions130that may be performed, either automatically or manually, by users108executing systems112within the service provider network102.

In configurations, the machine learning may include the analytics service122training and utilizing a deep learning model. The deep learning model may comprise an architecture related to a recurrent neural network, for example, a long short-term memory (LSTM) neural network. LSTM is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. It can not only process single data points (such as images), but also entire sequences of data. Another example architecture for the deep learning model includes using random cut forest (RCF), which is an unsupervised algorithm. Other architectures and algorithms may be used for the deep learning model if desired.

In configurations, one or more of the aggregation service118, analytics service122and/or remediation service128may be integrated as part of one service provided by the service provider network102. Additionally, the monitoring service114may be part of such a single, integrated service that provides one or more of the aggregation service118, the analytics service122, and/or the remediation service128. Thus, in configurations, the monitoring service114, the aggregation service118, the analytics service122, and the remediation service128may be a single integrated service that automatically monitors and remediates systems that execute within the service provider network102.

FIG.2illustrates a system-architecture diagram of an example arrangement200of a network202within which a system204(which may be at least similar to system112) may be executing. The system204may be executing on behalf of a user206on one or more user devices208, e.g., electronic devices, that are coupled to the network202. The user206may be an individual, a group of individuals, an entity, an organization, a business entity, etc. In configurations, the network202may be within an entity, an organization, a business entity, etc. and/or among a group of individuals, entities, organizations, business entities, and/or etc.

A monitoring service210may monitor execution of the system204in order to monitor problems that arise, e.g., operational issues, with respect to execution of the system204. In some configurations, the monitoring service210may be a third-party monitoring service. In some configurations, the monitoring service210may be part of the network202. The monitoring service210may generate operational issue notifications212, e.g., tickets, when one or more operational issues arise. For example, an operational issue may arise when a software patch failure occurs with respect to execution of the system204. Upon detection of the software patch failure, the monitoring service210may generate an operational issue notification212.

The operational issue notifications212may be provided to a controller214of the network. In configurations, the controller214may aggregate the operational issue notifications212into a standardized format for a standardized console that may be based on the network202. In configurations, when an operational issue arises, the user206may generate the operational issue notification212and provide it to the controller214. In configurations, one or more operational issues may occur at one time and, thus, multiple operational issue notifications212or a single notification212may be provided to the controller214depending on the configuration of the monitoring service210and/or the controller214.

As previously noted, the controller214may aggregate the operational issue notifications212received into a standardized or structured format. Additionally, if multiple operational issue notifications212are provided to the controller214at the same time, the controller214may aggregate the multiple operational issue notifications212into a single operational issue notification212, e.g., a single ticket. For example, multiple related operational issue notifications212may be aggregated into a single, “parent” notification. Thus, the related individual issue notifications may be referred to as “children” notifications. Furthermore, subsequent children operational issue notifications that are received may be associated with a corresponding aggregated parent operational issue notification.

Additionally, in configurations, the controller214may receive an operations issue notification212and may apply rules in order achieve deduplication and define parent operational issue notifications. This can help reduce noise caused by multiple alarms. Furthermore, machine learning, artificial intelligence and/or rules may be applied to help proactively determine other resources and/or services that may be affected by an issue that led to an operational issue notification212.

In configurations, the controller214may gather information216with respect to operation aspects of execution of the system204and may be gathered from sources different than the monitoring service210. The information216may be gathered for analysis in order to determine a potential cause of the operational issue. In configurations, the information216may include, for example, information related to one or more of metadata related to resource usage of the system204during execution, operational metrics related to execution of the system204, operational logs related to execution of the system204, instance logs related to execution of the system204, configurations that occurred around the time of the operational issue, resource changes related to execution of the system204, an overall health or status of the system204, relationships between the system204and resources within the network202, security details related to execution of the system204, timing of the operational issue, nodes within the network202with which the system204has interacted when the operational issue arose, etc. In configurations, scripts may be executed by the controller214to gather the information216. Additionally, in configurations, the user206may indicate what information216is important for analyzing operational issues.

Upon completion of analysis by the controller214, the controller214may determine one or more potential remedial actions218that may be performed to remediate the operational issue. For example, an account220of the user206may include a runbook222that includes remedial actions, where one or more of the remedial actions may be performed as a remedial flow to correct operational issue(s). In configurations, the runbook222may be included at one or more user devices208of the user206. Thus, in configurations, as a remedial action218, the controller214may provide information224to the user206regarding the runbook222and/or remedial actions that are included within the runbook222. For example, the runbook222may be in the form of automation documents that may address operational issue(s). Thus, the runbook222may specify that for operational issue A, execute remedial operation X. However, in configurations, the user206may overrule the runbook222and decide to execute remedial operation Y to address operational issue A. Such an overruling by the user206may be based on user preference, costs, timing, etc. In configurations, the user206may configure the controller214, e.g., provide a user preference, such that for operational issue A, always execute remedial operation Y.

In configurations, the controller214may provide a recommendation226to the user206to follow or execute the runbook222. The recommendation226may be based upon a threshold of certainty. For example, if the controller is ninety percent sure that the remediation flow of the runbook222will address the operational issue, then the controller may recommend that the user206apply the remediation flow provided by the runbook222. Other levels of certainty may be acceptable (e.g., different percentages) and thus, ninety percent is not meant to be limiting. In configurations, the threshold of certainty may be configurable by the user206and/or an operator of the network202.

In configurations, the controller214may automatically remediate the operational issue. For example, the controller214may automatically execute a script228that addresses the operational issue with respect to the execution of the system204. In configurations, the user206may specify that certain scripts228may be automatically performed to address operational issues.

Multiple scripts228may be generated by an operator of the monitoring service210, and/or the controller214to address various operational issues. In configurations, the user206may generate one or more scripts228and may share such scripts with the operator of the monitoring service210and/or the214to share with other users who may encounter similar operational issue(s) when executing systems similar to system204.

In configurations, a remedial action218may include further investigating the operational issue. For example, the controller214may not be able to determine a potential cause of the operational issue. Thus, additional information216may need to be gathered. For example, some information216may need to be gathered from third parties. Additionally, if a remedial action218is implemented and the operational issue is not resolved, then more information216may be gathered and analyzed to determine additional possible cause(s) and potential remedial action(s)218.

In configurations, the remedial action218may be to inform the user206that there is nothing to be done at this time. For example, the controller214may determine a root cause for the issue that resulted in the generation of the operational issue notification212. The root cause may be that a server is down and the issue will be resolved once the server is brought back online or a new server is provided by the network202. Thus, the remedial action218may be to inform the user206that there is nothing wrong with the user's system204and that the root cause needs to be addressed by the operator of the network202.

In configurations, the controller214may determine that the issue that resulted in the creation of the operational issue notification212has been remedied or fixed. The controller214may inform the originator of the operational issue notification212that the issue has been resolved. For example, the controller214may inform the monitoring service210that the issue has been resolved.

Since many of the same operational issues may occur among multiple systems204executed by networks, e.g., network202, by multiple users206, the controller214may utilize machine learning to recognize various situations and/or patterns of information216that occur and lead to various operational issues. Based upon the patterns recognized by analyzing the information216, the controller214may quickly determine potential causes for various operational issues. Based upon such pattern recognition, the controller214may inform and/or recommend various remedial actions218that may be performed, either automatically or manually, by users206executing systems204within networks.

In configurations, the machine learning may include the analytics service218training and utilizing a deep learning model. The deep learning model may comprise an architecture related to a recurrent neural network, for example, a long short-term memory (LSTM) neural network. LSTM is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. It can not only process single data points (such as images), but also entire sequences of data. Another example architecture for the deep learning model includes using random cut forest (RCF), which is an unsupervised algorithm. Other architectures and algorithms may be used for the deep learning model if desired.

In configurations, the controller214may be made up of one or more of an aggregation service, similar to aggregation service118, an analytics service, similar to analytics service122and/or a remediation service similar to remediation service224that may be integrated as part of a single service. Additionally, the monitoring service210may be part of such a single, integrated service that provides one or more of the aggregation service, the analytics service, and/or the remediation service. Thus, in configurations, the monitoring service210and the controller214may be a single integrated entity that automatically monitors and remediates systems that execute within networks, e.g., the controller214may also perform the functions of the monitoring service210.

FIGS.3and4illustrate flow diagrams of example methods300and400that illustrate aspects of the functions performed at least partly by the services as described inFIGS.1and2. The logical operations described herein with respect toFIGS.3and4may be implemented (1) as a sequence of computer-implemented acts or program modules running on a computing system, and/or (2) as interconnected machine logic circuits or circuit modules within the computing system.

FIG.3illustrates a flow diagram of the example method300for automatically collecting, analyzing, and remediating operational issues with respect to a system, e.g., system112, executing within a network, e.g., service provider network102.

At302, one or more notifications, e.g., operational issue notifications116, relating to one or more operational issues are received. For example, the one or more operational issues may be received from a monitoring service, e.g., monitoring service114, of the network. The one or more operational issues may relate to a system, e.g., system112, executing within the network, e.g., the service provider network102, on behalf of a user, e.g., user108. At304, the one or more notifications may be aggregated into a standardized format. For example, the one or more notifications may be aggregated by the aggregation service118of the service provider network102into the standardized format for a standardized console that may be based on the service provider network102. Additionally, multiple notifications may be aggregated by the aggregation service118into a single notification.

At306, contextual information, e.g., contextual information124, may be gathered related to the one or more operational issues. For example, the analytics service122of the service provider network102may gather the contextual information. The contextual information may be gathered from sources different than the monitoring service and may comprise at least information related to execution of the system within the service provider network. At308, the contextual information may be analyzed to determine potential causes of the one or more operational issues. For example, the analytics service122may evaluate or analyze the contextual information124to determine potential causes of the operational issues. In configurations, the analytics service122may determine a level of certainty with respect to potential causes of the one or more operational issues.

At310, the method300determines if one of the potential causes is a root cause. For example, a remediation service, e.g., remediation service128, and/or the analytics service122may determine a root cause for the issue that resulted in the generation of the notification. The root cause may be that a server is down and the issue will be resolved once the server is brought back online or a new server is provided by the service provider network. Thus, if a root cause does exist, at312the user may be informed that the system is fine and that the root cause is responsible for the issue.

At314, if a root cause does not exist, based on the potential causes, the one or more operational issues may be remediated. For example, the remediation service, e.g., remediation service128, may inform the user regarding a playbook, e.g., playbook132, that includes one or more potential remediation actions or remediation flows that may be utilized to address the operational issue. In configurations, based on the determined level of certainty, the remediation service may recommend remedial actions. For example, the remediation service may recommend one or more remedial actions or remedial flows that may be performed to address the one or more operational issues.

FIG.4illustrates a flow diagram of the example method400for automatically collecting, analyzing, and remediating operational issues with respect to a system, e.g., system204, executing within a network, e.g., network202.

At402, a notification, e.g., notification212, of an operational issue may be received. The operational issue may relate to execution of the system204within the network202, where the execution of the system utilizes computing resources on behalf of a user, e.g., user206. In configurations, the notification may be received from a third-party monitoring service, e.g., monitoring service210. The third-party monitoring service may be external to the network that is executing the system. For example, the network may be a service provider network such as the service provider network102. In configurations, the monitoring service may be part of the network.

At404, information, e.g., information216, related to the operational issue may be automatically gathered. For example, a controller, e.g., controller214, within the network202may gather various pieces of information216related to the operational issue, where the information is gathered from sources different than the monitoring service. For example, the information216may include a time at which the operational issue arose, resources being utilized by the system when the operational issue arose, interaction of the system with third party entities, and/or nodes within the network with which the system interacted when the operational issue arose, etc.

At406, the information may be automatically analyzed to determine a potential cause of the operational issue. For example, the controller214of the network202may analyze the information216to determine a potential cause of the operational issue. At408, the method400determines if the potential cause is a root cause. For example, the controller214may determine a root cause for the issue that resulted in the generation of the notification. The root cause may be that a server is down and the issue will be resolved once the server is brought back online or a new server is provided by the service provider network. Thus, if a root cause does exist, at410the user may be informed that the system is fine and that the root cause is responsible for the issue.

At412, if a root cause does not exist, based on the potential cause, the operational issue may be automatically remediated. For example, the controller214may make recommendations to the user206, provide information to the user206, automatically remediate the operational issue, etc.

Accordingly, the techniques and architecture described herein allow for automatically collecting information related to operational issues with respect to systems executing within a network analyzing the information, and based on the analyzed information, remediating the operational issues. A monitoring service may generate notifications related to operational issues upon detection of operational issues within a system executing within the network. The notifications may be aggregated into a standardized format thereby allowing for notifications to be received in various formats. Contextual information related to the operational issues may be automatically gathered by an analytics service, which may analyze the contextual information to determine a potential cause of the operational issues. Based on the potential cause, a remediation service may automatically remediate the operational issues. This allows for operational issues to be addressed more quickly and efficiently without manual actions on the part of engineers and technicians. In particular, engineers or technicians do not need to manually go through various consoles and/or manuals in order to gather information, data, metrics, etc., to determine potential causes for the operational issues and then to determine potential remedial actions to address the operational issues.

FIG.5is a system and network diagram that shows one illustrative operating environment502for the configurations disclosed herein that includes a service provider network102that can be configured to perform the techniques disclosed herein. The service provider network102can provide computing resources, like VM instances and storage, on a permanent or an as-needed basis. Among other types of functionality, the computing resources provided by the service provider network102may be utilized to implement the various services described above. As also discussed above, the computing resources provided by the service provider network102can include various types of computing resources, such as data processing resources like VM instances, data storage resources, networking resources, data communication resources, network services, and the like.

The computing resources provided by the service provider network102may be enabled in one embodiment by one or more data centers504A-504N (which might be referred to herein singularly as “a data center504” or in the plural as “the data centers504”). The data centers504are facilities utilized to house and operate computer systems and associated components. The data centers504typically include redundant and backup power, communications, cooling, and security systems. The data centers504can also be located in geographically disparate locations. One illustrative embodiment for a data center504that can be utilized to implement the technologies disclosed herein will be described below with regard toFIG.6.

The data centers504may be configured in different arrangements depending on the service provider network102. For example, one or more data centers504may be included in or otherwise make-up an availability zone. Further, one or more availability zones may make-up or be included in a region. Thus, the service provider network102may comprise one or more availability zones, one or more regions, and so forth. The regions may be based on geographic areas, such as being located within a predetermined geographic perimeter.

The customers and other users108of the service provider network102may access the computing resources provided by the service provider network102over any wired and/or wireless network(s)522, which can be a wide area communication network (“WAN”), such as the Internet, an intranet or an Internet service provider (“ISP”) network or a combination of such networks. For example, and without limitation, a user device110operated by a customer or other user108of the cloud-based service provider network102may be utilized to access the service provider network102by way of the network(s)522. It should be appreciated that a local-area network (“LAN”), the Internet, or any other networking topology known in the art that connects the data centers504to remote customers and other users can be utilized. It should also be appreciated that combinations of such networks can also be utilized.

Each of the data centers504may include computing devices that include software, such as applications that receive and transmit data424. For instance, the computing devices included in the data centers504may include software components which transmit, retrieve, receive, or otherwise provide or obtain the data424from a data store426that may be provided by the storage service106. For example, the data centers504may include or store the data store426, which may include the data424that may include, for example collective information124.

FIG.6is a computing system diagram that illustrates one configuration for a data center504that implements aspects of the technologies disclosed herein. The example data center504shown inFIG.6includes several server computers602A-602F (which might be referred to herein singularly as “a server computer602” or in the plural as “the server computers602”) for providing computing resources604A-604E.

The server computers602can be standard tower, rack-mount, or blade server computers configured appropriately for providing the computing resources described herein (illustrated inFIG.6as the computing resources604A-604E). As mentioned above, the computing resources provided by the service provider network102can be data processing resources such as VM instances or hardware computing systems, database clusters, computing clusters, storage clusters, data storage resources, database resources, networking resources, and others. Some of the servers602can also be configured to execute a resource manager606capable of instantiating and/or managing the computing resources. In the case of VM instances, for example, the resource manager606can be a hypervisor or another type of program configured to enable the execution of multiple VM instances on a single server computer602. Server computers602in the data center504can also be configured to provide network services and other types of services, some of which are described in detail below with regard toFIG.6.

The data center504shown inFIG.6also includes a server computer602F that can execute some or all of the software components described above. For example, and without limitation, the server computer602F can be configured to execute components of the service provider network102, including the processing service104, the storage service106, the monitoring service114, and/or the other software components described above. The server computer602F can also be configured to execute other components and/or to store data for providing some or all of the functionality described herein. In this regard, it should be appreciated that the services illustrated inFIG.7as executing on the server computer602F can execute on many other physical or virtual servers in the data centers504in various embodiments.

In the example data center504shown inFIG.6, an appropriate LAN608is also utilized to interconnect the server computers602A-602F. It should be appreciated that the configuration and network topology described herein has been greatly simplified and that many more computing systems, software components, networks, and networking devices can be utilized to interconnect the various computing systems disclosed herein and to provide the functionality described above. Appropriate load balancing devices or other types of network infrastructure components can also be utilized for balancing a load between each of the data centers504A-504N, between each of the server computers602A-602F in each data center504, and, potentially, between computing resources in each of the server computers602. It should be appreciated that the configuration of the data center504described with reference toFIG.6is merely illustrative and that other implementations can be utilized.

FIG.7is a system and network diagram that shows aspects of several network services that can be provided by and utilized within a service provider network102in one embodiment disclosed herein. In particular, and as discussed above, the service provider network102can provide a variety of network services to customers and other users of the service provider network102including, but not limited to, the monitoring service114, the aggregation service118, the analytics service122, and the remediation service128. The service provider network102can also provide other types of services including, but not limited to, an on-demand computing service702A (e.g., processing service104), a deployment service702B, a cryptography service702C, and an authentication service702D, and/or a storage service702D (e.g., storage service106), some of which are described in greater detail below. Additionally, the service-provider network102can also provide other services, some of which are also described in greater detail below.

It should be appreciated that customers of the service provider network102can include organizations or individuals that utilize some or all of the services provided by the service provider network102. As described herein, a customer or other user can communicate with the service provider network102through a network, such as the network522shown inFIG.5. Communications from a customer computing device, such as the user device110shown inFIG.5, to the service provider network102can cause the services provided by the service provider network102to operate in accordance with the described configurations or variations thereof.

It is noted that not all embodiments described include the services described with reference toFIG.7and that additional services can be provided in addition to or as an alternative to services explicitly described. Each of the services shown inFIG.7can also expose network services interfaces that enable a caller to submit appropriately configured API calls to the various services through web service requests. In addition, each of the services can include service interfaces that enable the services to access each other (e.g., to enable a virtual computer system provided by the on-demand computing service702A to store data in or retrieve data from a storage service). Additional details regarding some of the services shown inFIG.7will now be provided.

As discussed above, the on-demand computing service702A (e.g., the processing service104) can be a collection of computing resources configured to instantiate VM instances and to provide other types of computing resources on demand. For example, a customer or other user of the service provider network102can interact with the on-demand computing service702A (via appropriately configured and authenticated network services API calls) to provision and operate VM instances that are instantiated on physical computing devices hosted and operated by the service provider network102.

The VM instances can be used for various purposes, such as to operate as servers supporting a web site, to operate business applications or, generally, to serve as computing resources for the customer. Other applications for the VM instances can be to support database applications such as those described herein, electronic commerce applications, business applications and/or other applications. Although the on-demand computing service702A is shown inFIG.7, any other computer system or computer system service can be utilized in the service provider network102, such as a computer system or computer system service that does not employ virtualization and instead provisions computing resources on dedicated or shared computers/servers and/or other physical devices.

A storage service702D (e.g., storage service106) can include software and computing resources that collectively operate to store data using block or file-level storage devices (and/or virtualizations thereof) into data store426, which may be part of the storage service702D. The storage devices of the storage service702D, e.g., storage service106, can, for instance, be operationally attached to virtual computer systems provided by the on-demand computing service702A to serve as logical units (e.g., virtual drives) for the computer systems. A storage device can also enable the persistent storage of data used/generated by a corresponding virtual computer system where the virtual computer system service might only provide ephemeral data storage.

The service provider network102can also include a cryptography service702C. The cryptography service702C can utilize storage services of the service provider network102, such as the storage service702F, to store encryption keys in encrypted form, whereby the keys are usable to decrypt customer keys accessible only to particular devices of the cryptography service702C. The cryptography service702C can also provide other types of functionality not specifically mentioned herein.

The service provider network102can additionally maintain other services702based, at least in part, on the needs of its customers. For instance, the service provider network102can maintain a deployment service702B for deploying program code and/or a data warehouse service in some embodiments. Other services include object-level archival data storage services, database services, and services that manage, monitor, interact with, or support other services. The service provider network102can also be configured with other services not specifically mentioned herein in other embodiments.

FIG.8shows an example computer architecture for a computer800capable of executing program components for implementing the functionality described above. The computer architecture shown inFIG.8illustrates a server computer, workstation, desktop computer, laptop, tablet, network appliance, e-reader, smartphone, or other computing device, and can be utilized to execute any of the software components presented herein.

The computer800includes a baseboard802, or “motherboard,” which is a printed circuit board to which a multitude of components or devices can be connected by way of a system bus or other electrical communication paths. In one illustrative configuration, one or more central processing units (“CPUs”)804operate in conjunction with a chipset806. The CPUs804can be standard programmable processors that perform arithmetic and logical operations necessary for the operation of the computer800.

The chipset806provides an interface between the CPUs804and the remainder of the components and devices on the baseboard802. The chipset806can provide an interface to a RAM808, used as the main memory in the computer800. The chipset806can further provide an interface to a computer-readable storage medium such as a read-only memory (“ROM”)810or non-volatile RAM (“NVRAM”) for storing basic routines that help to startup the computer800and to transfer information between the various components and devices. The ROM810or NVRAM can also store other software components necessary for the operation of the computer800in accordance with the configurations described herein.

The computer800can operate in a networked environment using logical connections to remote computing devices and computer systems through a network, such as the network608. The chipset806can include functionality for providing network connectivity through a Network Interface Controller (NIC)812, such as a gigabit Ethernet adapter. The NIC812is capable of connecting the computer800to other computing devices over the network608(or522). It should be appreciated that multiple NICs812can be present in the computer800, connecting the computer to other types of networks and remote computer systems.

The computer800can be connected to a mass storage device818that provides non-volatile storage for the computer. The mass storage device818can store an operating system820, programs822(e.g., agents, etc.), data, and/or applications(s)824, which have been described in greater detail herein. The mass storage device818can be connected to the computer800through a storage controller814connected to the chipset806. The mass storage device818can consist of one or more physical storage units. The storage controller814can interface with the physical storage units through a serial attached SCSI (“SAS”) interface, a serial advanced technology attachment (“SATA”) interface, a fiber channel (“FC”) interface, or other type of interface for physically connecting and transferring data between computers and physical storage units.

The computer800can store data on the mass storage device818by transforming the physical state of the physical storage units to reflect the information being stored. The specific transformation of physical states can depend on various factors, in different embodiments of this description. Examples of such factors can include, but are not limited to, the technology used to implement the physical storage units, whether the mass storage device818is characterized as primary or secondary storage, and the like.

In addition to the mass storage device818described above, the computer800can have access to other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. It should be appreciated by those skilled in the art that computer-readable storage media is any available media that provides for the non-transitory storage of data and that can be accessed by the computer800. In some examples, the operations performed by the service provider network102, and or any components included therein, may be supported by one or more devices similar to computer800. Stated otherwise, some or all of the operations performed by the service provider network102, and or any components included therein, may be performed by one or more computer devices800operating in a cloud-based arrangement.

As mentioned briefly above, the mass storage device818can store an operating system820utilized to control the operation of the computer800. According to one embodiment, the operating system comprises the LINUX operating system. According to another embodiment, the operating system comprises the WINDOWS® SERVER operating system from MICROSOFT Corporation of Redmond, Wash. According to further embodiments, the operating system can comprise the UNIX operating system or one of its variants. It should be appreciated that other operating systems can also be utilized. The mass storage device818can store other system or application programs and data utilized by the computer800.

In one embodiment, the mass storage device818or other computer-readable storage media is encoded with computer-executable instructions which, when loaded into the computer800, transform the computer from a general-purpose computing system into a special-purpose computer capable of implementing the embodiments described herein. These computer-executable instructions transform the computer800by specifying how the CPUs804transition between states, as described above. According to one embodiment, the computer800has access to computer-readable storage media storing computer-executable instructions which, when executed by the computer800, perform the various processes described above with regard toFIGS.1-4. The computer800can also include computer-readable storage media having instructions stored thereupon for performing any of the other computer-implemented operations described herein.

The computer800may transmit, receive, retrieve, or otherwise provide and/or obtain data and/or results to and/or from the service provider network102. The computer800may store the data on the operating system820, and/or the programs22that are stored in the mass storage device818to update or otherwise modify the operating system820and/or the programs822.

Although the application describes embodiments having specific structural features and/or methodological acts, it is to be understood that the claims are not necessarily limited to the specific features or acts described. Rather, the specific features and acts are merely illustrative of some embodiments that fall within the scope of the claims of the application.