Patent Description:
Methods for one click monitors in impact time detection for noise reduction in at-scale monitoring are performed by systems and devices. The methods automatically configure time window sizes and numbers of consecutive time windows for optimally detecting system alerts in at-scale systems and per dimension combinations, including updating settings over time to adapt to changing system behaviors. The past behavior of system performance metrics are analyzed against a provided threshold value per metric to match configuration options and determine a best fitting or optimal combination of a highest detection accuracy in lowest time to detect for alerting. Noisy metrics are handled by varying numbers of consecutive time windows for metric monitoring, and times to detect are optimized by analyzing changes in threshold metric conditions being met across the different consecutive time windows. Optimal monitoring configurations are determined for each of up to hundreds of thousands of the metric dimensions across the system, and an end user is enabled to apply the determined, optimal configurations for system monitoring with a single selection.

Further features and advantages, as well as the structure and operation of various examples, are described in detail below with reference to the accompanying drawings. It is noted that the ideas and techniques are not limited to the specific examples described herein. Such examples are presented herein for illustrative purposes only. Additional examples will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein.

The features and advantages of embodiments will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout.

In the discussion, unless otherwise stated, adjectives such as "substantially," "approximately," and "about" modifying a condition or relationship characteristic of a feature or features of an embodiment of the disclosure, are understood to mean that the condition or characteristic is defined to be within tolerances that are acceptable for operation of the embodiment for an application for which it is intended.

Furthermore, it should be understood that spatial descriptions (e.g., "above," "below," "up," "left," "right," "down," "top," "bottom," "vertical," "horizontal," etc.) used herein are for purposes of illustration only, and that practical implementations of the structures and drawings described herein can be spatially arranged in any orientation or manner. Additionally, the drawings may not be provided to scale, and orientations or organization of elements of the drawings may vary in embodiments.

Section II below describes example embodiments for one click monitors in impact time detection for noise reduction in at-scale monitoring. Section III below describes example computing device embodiments that may be used to implement features of the embodiments described herein. Section IV below describes additional examples and advantages, and Section V provides some concluding remarks. Example Embodiments for One Click Monitors in Impact Time Detection for.

Times to detect alerts for computing system issues are important to reduce impacts on system performance and availability, while the ability to filter noisy system metrics and improves accuracy in triggering alerts is also important. In at-scale implementations, e.g., in cloud-based hosting of services and applications combinations of metrics required to be monitored for system alerts range in numbers up to hundreds of thousands or more, and end users are not capable of configuring so many individual settings. Embodiments for one click monitors in impact time detection for noise reduction in at-scale monitoring are provided herein.

For instance, a user is provided with a user interface (UI) having a selectable object or control by which the user is enabled to accept automatically generated configuration settings having parameters for managing system alert monitoring, in embodiments. The automatically generated configuration settings may include default options when operation data for monitored metrics is not yet available and/or may be options generated based on past system metric data that is analyzed according to models that optimize the accuracy of alert detection against reductions in times to detect conditions that trigger alerts. The described monitoring systems and operations therefor balance settings for lengths of time windows utilized for monitoring metrics and the numbers of consecutive time windows required to declare an alert-in other words, embodiments enable configurations to monitoring systems that are set to a desired meaningful impact time of a number of failures in a given, overall time period. In this way, information technology (IT) personnel are not burdened with false-positive and false-negative alerts while the time to detect system issues for any given metric or combination of metrics is maintained or reduced.

One example described herein is for consecutive time window settings, or a number of time windows in sequence, each having a given duration, in which a threshold metric condition for metric data must be met to trigger an alert. This parameter is applied in embodiments against different considerations. One such consideration is that the higher the value of consecutive threshold metric conditions being met, the more confidence is associated with a received alert for an incident that the problem is real and needs handling. For instance, while monitoring, if a threshold metric condition is met for central processing unit (CPU) utilization in each of four consecutive time windows, this provides more confidence that a real issue exists and that an alert should be triggered than if the same threshold metric condition is met for only one or two consecutive time windows. An opposing consideration, however, that when there is an actual problem, there is a need to notify IT personnel and/or the end user of the problem as soon as possible so that the impact of the issue on the system is minimal before the problem is resolved. This creates a tradeoff as on one hand early issues detection is valuable, while on the other hand, false alerts being triggered is also problematic. Additionally, monitoring configuration settings that allow for false alerts creates may lead to "alert storms," or many alerts firing for large/scaled-out services, which causes real problems to be lost in the noise as so many false alerts are triggered that the real problems cannot be seen and addressed.

Embodiments herein are model agnostic, and may be applied to various types of detections, including but not limited to, system alerts for resource utilization, network utilization, etc. In some examples contemplated herein, a system/device one click monitors in impact time detection for noise reduction in at-scale monitoring receives as inputs a threshold metric condition, also referred to as a border, and metric data. The threshold metric condition is a value or other condition against which metric data is compared to detect issues or operations outside of desired boundaries for system resources, etc., and the metric data may include actual testing or in-production data for a metric that is collected and stored, e.g., by the system alert monitor. In embodiments, threshold metric conditions, or boundaries, and metric data, are received as the inputs to an alert manager according to the described embodiments, and the alert manager outputs, for the system alert monitor, recommended time window durations and corresponding numbers of consecutive time windows that are determined as optimal based on the received inputs for various metrics to be monitored.

Embodiments are applicable to various types of models and are not envisioned as being limited to specific implementations, and machine learning (ML) models are contemplated herein for determinations of monitoring window duration and numbers of consecutive monitoring windows. ML models, various threshold metric conditions or borders for different metrics that are monitored may be generated by the system alert monitor and may be either static, simple values and/or may be dynamic thresholds/borders.

In embodiments, the alert manager receives data associated with the history of metric behavior. This metric data may be split into validation/testing data and training data. As a non-limiting example, metric data of a given metric is received from over a prior ten day period for determining monitoring window duration and number of consecutive monitoring windows. As another, for metric data of a given metric that is received from over a prior eleven day period, the last or most recent day may be used as a validation data portion, and the previous data as a training data portion, for determining monitoring window duration and number of consecutive monitoring windows. For one or more window durations/sizes (e.g., <NUM> minutes, <NUM> minutes, <NUM> minutes, <NUM> minutes, <NUM> minutes, etc.), an analysis is performed. This analysis includes calculating/determining an amount of noise per consecutive window in graphical or data set form on the training data with the given threshold metric condition. In embodiments, the noise may be characterized as the number of alerts that would occur for a given consecutive window setting in the training data, or as a ratio of alerting time windows to the total number of utilized consecutive monitoring time windows, using the threshold metric condition (e.g., a static border of CPU utilization being greater than, or greater than or equal to, <NUM>%). As noted above, a dynamic threshold metric condition or border, such as via the Dynamic Threshold Service in Microsoft® Azure® from Microsoft Corporation of Redmond, WA. An example of such a graphical/data set representation, and utilization thereof, is described below with respect to <FIG>.

Next, the noise per consecutive window, in graphical or data set form, is calculated/determined on the validation data, in embodiments. Here, the noise is the number of alerts that would occur for a given consecutive window setting in the validation data using the same, or a different, threshold metric condition or border. In some embodiments, this validation data determination is performed only for ML-based models. The point of maximum curvature (or change in slope, or change in value) for each graph or data set over the consecutive window values is then determined as the optimal consecutive window value. In embodiments where noise per consecutive window graphs, or data sets, are determined for both training and validation data, the maximum among the two values is selected as the number of consecutive window instances or values. Some implementations also provide for increasing the number of consecutive window instances or values if the sixtieth percentile of the daily alert counts is greater than zero in order to insure that noise reduction is adequate for a given metric.

The above-described process may be performed for any number of metrics and metric combinations across a system, including very large numbers of monitored metrics in scaled-out cloud platforms.

Accordingly, methods for one click monitors in impact time detection for noise reduction in at-scale monitoring are performed by systems and devices. The embodiments herein provide solutions that improve times to detect and trigger alerts for system issues while also reducing and/or eliminating false positive and false negative alerts. These and other embodiments for one click monitors in impact time detection for noise reduction in at-scale monitoring will be described in further detail below in association with the Figures, and in the Sections/Subsections that follow.

Systems, devices, and apparatuses may be configured in various ways for one click monitors in impact time detection for noise reduction in at-scale monitoring. For instance, <FIG> will now be described. <FIG> shows a block diagram of a system 100A, and <FIG> shows a block diagram of a cloud-based system 100B, each configured for one click monitors in impact time detection for noise reduction in at-scale monitoring.

As shown in <FIG>, system 100A includes user device(s) <NUM> (also user device <NUM> herein), a first domain host <NUM>, and a second domain host <NUM>. In embodiments, user) device <NUM>, first domain host <NUM>, and second domain host <NUM> communicate with each other over a network <NUM>. It should be noted that in various embodiments different numbers of user devices, first domain hosts, and/or second domain hosts are present. Additionally, according to embodiments, any combination of the systems and/or components illustrated in <FIG> are present in system 100A.

Network <NUM> comprises different numbers and/or types of communication links that connect computing devices and hosts/servers such as, but not limited to, the Internet, wired or wireless networks and portions thereof, point-to-point connections, local area networks, enterprise networks, cloud networks, and/or the like, in embodiments. In an example, network <NUM> may be a cloud-based platform network and/or enterprise network through which at least one user device <NUM> connects to a domain or server thereof.

User device <NUM> in different embodiments is any number, type, or combination of computing devices or computing systems, including a terminal, a personal computer, a laptop computer, a tablet device, a smart phone, a personal digital assistant, a server(s), a gaming console, and/or the like, including internal/external storage devices, that are utilized to execute functions/operations described herein for one click monitors in impact time detection for noise reduction in at-scale monitoring, as well as for performing client-side functions/operations of client-server scenarios associated with embodiments such as receiving a UI for selection of a plurality of configurations for monitory system metrics via activation of a single UI control, element, object, etc., for end users, and/or receiving alert notifications from a system alert monitor for IT personnel. User device <NUM> also includes additional components (not shown for brevity and illustrative clarity) including, but not limited to, components and subcomponents of other devices and/or systems herein, in various embodiments.

A domain, as used herein, generally refers to a physical and/or logical system boundary under the control of an entity within which applications and/or services are hosted, offered, managed, and/or otherwise implemented, and also encompasses subdomains and/or the like in embodiments. Exemplary, non-limiting domains include, without limitation, web domains, tenancies of hosted cloud platforms, cloud service providers, enterprise systems, and/or any other type of network or system. A tenant is particular type of domain that is a representation of an organization in a cloud platform. The domain of the tenant in the cloud platform is its tenancy in which the tenant registers and manages applications, stores datalfiles, accesses services, etc..

First domain host <NUM> comprises one or more server computers or computing devices, such as an on-premises server(s) in addition to, or in lieu of, cloud-based servers, associated with a first domain. First domain host <NUM>, as shown, includes an alert manager <NUM>, which may be an instantiation of an alert manager described herein. Alert manager <NUM> is configured to determine numbers of consecutive instances of time windows that provide an optimal balance between generating accurate system alerts for respective system metrics and times to detect the system alerts, in embodiments. Alert manager <NUM> may also be configured to provide a UI in which a user is enabled to activate a single UI control, element, object, etc., that applies each of the determined numbers of consecutive instances of time windows to a system alert monitor, while in some embodiments, the UI may be provided by a system alert monitor of first domain host <NUM> (not shown).

Second domain host <NUM> comprises one or more server computers or computing devices, such as an on-premises server(s) in addition to, or in lieu of, cloud-based servers, associated with a second domain. Second domain host <NUM>, as shown, includes an alert manager <NUM>, which may be an instantiation of an alert manager described herein. Alert manager <NUM> is configured to determine numbers of consecutive instances of time windows that provide an optimal balance between generating accurate system alerts for respective system metrics and times to detect the system alerts, in embodiments. Alert manager <NUM> may also be configured to provide a UI in which a user is enabled to activate a single UI control, element, object, etc., that applies each of the determined consecutive instances of time windows to a system alert monitor, while in some embodiments, the UI may be provided by a system alert monitor of second domain host <NUM> (not shown).

Turning now to <FIG>, system 100B is a cloud-based embodiment of system 100A of <FIG>. As shown, system 100B includes a cloud platform <NUM>. In embodiments, cloud platform <NUM> is a cloud-based platform such as Microsoft® Azure® from Microsoft Corporation of Redmond, WA, that is accessible by a user(s) of user device(s) <NUM> (also user device <NUM> herein)over a network (not shown for illustrative clarity and brevity).

User device <NUM> may be any type and/or number of user device, such as devices similar to those described for user device <NUM> in <FIG>, and may correspond to end users and/or IT personnel with credentials for different domains, such as different tenancies within cloud platform <NUM>.

A tenant is a representation of an organization in a cloud platform. The domain of the tenant in the cloud platform is its tenancy in which the tenant registers and manages applications, stores datalfiles, accesses services, etc., hosted by cloud platform <NUM>. That is, tenants are enabled to provide applications/services, hosted by cloud platform <NUM>, to users such as end users. In doing so, a tenant may lease or purchase the use of system resources within cloud platform <NUM> for such hosting and may monitor the system resources and/or operations.

For instance, cloud platform <NUM> includes a tenant A <NUM> and a tenant B/partner <NUM> (e.g., as a partner or service provider of the cloud platform owner), although different numbers of tenants are contemplated in embodiments. In embodiments, tenant A <NUM> corresponds to the first domain of system 100A, and tenant B/partner <NUM> corresponds to the second domain of system 100A. Users of user device(s) <NUM> having credentials for tenant A <NUM> are allowed to authenticate for this tenancy and access data, information, services, applications, etc., e.g., services/applications <NUM> (also "services/apps" <NUM> herein) of cloud platform <NUM>, allowed or instantiated for tenant A <NUM>. Likewise, users of user device(s) <NUM> having credentials for tenant B/partner <NUM> are allowed to authenticate for this tenancy and access data, information, services, applications, etc., e.g., services/apps <NUM> of cloud platform <NUM>, allowed or instantiated for tenant B/partner <NUM>.

Services/applications <NUM> is shown as including an alert manager <NUM>, which may be an instantiation of an alert manager described herein. Alert manager <NUM> is configured to determine numbers of consecutive instances of time windows that provide an optimal balance between generating accurate system alerts for respective system metrics and times to detect the system alerts, in embodiments. Alert manager <NUM> may also be configured to provide a UI in which a user, e.g., a member of a tenancy and user of user device <NUM>, is enabled to activate a single UI control, element, object, etc., that applies each of the determined consecutive instances of time windows to a system alert monitor, while in some embodiments, the UI may be provided by a system alert monitor of cloud platform <NUM>, e.g., as a part of services/applications <NUM> (not shown).

Cloud platform <NUM> includes one or more distributed or "cloud-based" servers, in embodiments. That is, cloud platform <NUM> is a network, or "cloud," implementation for applications and/or services in a network architecture/cloud platform. A cloud platform includes a networked set of computing resources, including servers, routers, etc., that are configurable, shareable, provide data security, and are accessible over a network such as the Internet, according to embodiments. Cloud applications/services are configured to run on these computing resources, often atop operating systems that run on the resources, for entities that access the applications/services, locally and/or over the network. A cloud platform such as cloud platform <NUM> is configured to support multi-tenancy as noted above, where cloud platform-based software services multiple tenants, with each tenant including one or more users who share common access to certain software services and applications of cloud platform <NUM>, as noted herein. Furthermore, a cloud platform is configured to support hypervisors implemented as hardware, software, and/or firmware that run virtual machines (emulated computer systems, including operating systems) for tenants. A hypervisor presents a virtual operating platform for tenants.

Portions of <FIG>, and system 100A and system 100B respectively, such as first domain host <NUM>, second domain host <NUM>, and/or cloud platform <NUM> also include additional components (not shown for brevity and illustrative clarity) including, but not limited to, components and subcomponents of other devices and/or systems herein, e.g., an operating system, as shown in <FIG> described below, in embodiments.

Additionally, as would be understood by persons of skill in the relevant art(s) having the benefit of this disclosure, system 100A and system 100B illustrate embodiments in which system resources may be scaled out on demand or as needed, and the embodiments herein provide for determining and feasibly enabling configuration settings for system alert monitors in such scaled-out systems, even to the extent of many thousands of system metrics requiring alert monitoring configurations.

Systems, devices, and apparatuses are configured in various ways for one click monitors in impact time detection for noise reduction in at-scale monitoring, in embodiments. For instance, <FIG> and <FIG> will now be described in this context.

Referring first to <FIG>, a block diagram of a system <NUM> is shown for one click monitors in impact time detection for noise reduction in at-scale monitoring, according to an example embodiment. System <NUM> as exemplarily illustrated and described is configured to be an embodiment of system 100A of <FIG> and/or system 100B of <FIG>. <FIG> shows a flowchart <NUM> for one click monitors in impact time detection for noise reduction in at-scale monitoring, according to an example embodiment. System <NUM> may be configured to operate in accordance with flowchart <NUM>. System <NUM> is described as follows.

System <NUM> includes a computing system <NUM> which is any type of server or computing system, as mentioned elsewhere herein, or as otherwise known, including without limitation cloud-based systems, on-premises servers, distributed network architectures, and/or the like. As shown in <FIG>, computing system <NUM> includes one or more processors ("processor") <NUM>, one or more of a memory and/or other physical storage device ("memory") <NUM>, as well as one or more network interfaces ("network interface") <NUM>. Computing system <NUM> also includes an alert manager <NUM> that is an embodiment of alert manager <NUM> and alert manager <NUM>, and/or of alert manager <NUM>, of <FIG>, respectively, an alert monitor <NUM>, models <NUM>, and UI logic <NUM>. It is contemplated herein that any components of system <NUM> may be grouped, combined, separated, etc., from any other components in various embodiments, and that the illustrated example of system <NUM> in <FIG> is non-limiting in its configuration and/or numbers of components, as well the exemplary arrangement thereof.

Processor <NUM> and memory <NUM> may respectively be any type of processor circuit(s)/system(s) and memory that is described herein, and/or as would be understood by a person of skill in the relevant art(s) having the benefit of this disclosure. Processor <NUM> and memory <NUM> may each respectively comprise one or more processors or memories, different types of processors or memories (e.g., at least one cache for query processing), remote processors or memories, and/or distributed processors or memories. Processor <NUM> may be multi-core processors configured to execute more than one processing thread concurrently. Processor <NUM> may comprise circuitry that is configured to execute and/or process computer program instructions such as, but not limited to, embodiments of alert manager <NUM>, alert monitor <NUM>, and/or UI logic <NUM>, including one or more of the components thereof as described herein, which may be implemented as computer program instructions, as described herein. For example, in performance of/operation for flowchart <NUM> of <FIG>, processor <NUM> may execute program instructions as described.

Memory <NUM> includes volatile storage portions such as a random access memory (RAM) and/or persistent storage portions such as hard drives, non-volatile RAM, and/or the like, to store or be configured to store computer program instructions/code for one click monitors in impact time detection for noise reduction in at-scale monitoring as described herein, as well as to store other information and data described in this disclosure including, without limitation, embodiments of alert manager <NUM>, alert monitor <NUM>, and/or UI logic <NUM>, including one or more of the components thereof as described herein, and/or the like, in different embodiments. Memory <NUM> also includes storage of models <NUM>, in embodiments, as well as metric data, data sets, determined and/or default configuration settings for monitoring windows, and/or the like, as described herein.

Network interface <NUM> may be any type or number of wired and/or wireless network adapter, modem, etc., configured to enable system <NUM>, including computing system <NUM>, to communicate intra-system with components thereof, as well as with other devices and/or systems over a network, such as communications between computing system <NUM> and other devices, systems, hosts, of system 100A in <FIG> and/or system 100B in <FIG>, over a network/cloud platform such as network <NUM> and/or cloud platform <NUM>.

System <NUM> also includes additional components (not shown for brevity and illustrative clarity) including, but not limited to, components and subcomponents of other devices and/or systems herein, as well as those described below with respect to <FIG>, e.g., an operating system, etc., according to embodiments.

Alert manager <NUM> of computing system <NUM> includes a plurality of components for performing the functions and operations described herein for one click monitors in impact time detection for noise reduction in at-scale monitoring, in embodiments. As illustrated, alert manager <NUM> includes metric data logic <NUM>, a threshold determiner <NUM>, consecutive logic <NUM>, data set logic <NUM>, a consecutive instance determiner <NUM>, and a model engine <NUM>, although additional components, as described herein or otherwise, are also included and some components may be excluded, in various embodiments. For example, model engine <NUM> may be a portion of alert manager <NUM> in some embodiments, or may be a separate component in other embodiments. Additionally, alert manager <NUM> may comprise a portion of alert monitor <NUM> in embodiments.

Metric data logic <NUM> is configured to receive metric data associated with performance of a computing system, e.g., a server or group of servers, a cloud platform, and/or the like. Metric data may be received from a system alert monitor such as alert monitor <NUM>, or from any other type of system monitor. Metric data may be received in real time by metric data logic <NUM>, or may be received in batches from stored metric data. Metric data logic <NUM> is configured to identify and/or group portions of received metric data for training models (i.e., training data) and for validating models (i.e., testing/validation data). In embodiments, metric data logic <NUM> may be configured to receive threshold metric conditions, e.g., as set by a user or dynamically generated, for one or more metrics.

Threshold determiner <NUM> is configured to determine if a threshold metric condition is met during the training of a model, in embodiments. For example, threshold determiner <NUM> determines when received metric data meets the threshold metric condition. Additional details regarding the determination of threshold metric conditions being met are described below. In embodiments, threshold determiner <NUM> may be configured to receive threshold metric conditions, e.g., as set by a user or dynamically generated, for one or more metrics.

Consecutive logic <NUM> is configured to provide different combinations of durations of monitoring windows and numbers of consecutive monitoring windows in the determination of configuration settings. Consecutive logic <NUM> may generate such combinations based on preferred or predetermined duration options and/or numbers of consecutives. As an example, a given type of metric may typically be monitored with a <NUM> minute time window, and consecutive logic <NUM> may use this duration to generate a plurality of consecutive monitoring window combinations with each window having a <NUM> minute duration. In some cases, noisy metrics may have combinations of durations and numbers of consecutives generated by consecutive logic <NUM> with longer durations, and may provide for more stable (i.e., less noisy) metrics combinations with shorter durations. However, embodiments are not so limited, and consecutive logic <NUM> is configured to provide any duration and number of consecutives combinations for any metric. Consecutive logic <NUM> generates combinations for training data and/or for validation data, according to embodiments.

Data set logic <NUM> is configured to generate data sets associated with threshold metric conditions met over the different combinations of durations of monitoring windows and numbers of consecutive monitoring windows provided by consecutive logic <NUM>. Data sets may be represented as monotonically decreasing sequences of data points, and/or may be represented graphically as shown in <FIG>. Embodiments herein are not limited by the representations generated by data set logic <NUM>, and any equivalent representation of a given set of data are contemplated in this disclosure.

Consecutive instance determiner <NUM> is configured to select an optimal number of consecutive instances of a time window for a combination of a duration for a monitoring window and a number of consecutive monitoring windows for metrics to be monitored. For example, consecutive instance determiner <NUM> selects a combination based on an amount of change between adjacent data values that across a data set for a metric, in embodiments. An optimal configuration setting for to the determined/selected consecutive instance may correspond to an amount of change between two of the adjacent data values where the reduction in threshold metric conditions met for a number of consecutive windows (i.e., a reduction in the alert to consecutive ratio) decreases by a relative or predetermined amount. In a graphical context, this optimal configuration setting can be illustrated as the point where the slope of the monotonically decreasing graph begins to flatten, becomes less negative, or becomes closer to zero. In other embodiments, consecutive instance determiner <NUM> selects a combination derived from the initially determined optimal setting based on a maximum number of determined alerts allowed or desired in a given time while minimizing a time to detect (e.g., a number of consecutive monitoring windows determined). Additional considerations for the maximum number of determined alerts may include, e.g., a portion of days, from a number of days, in which no alerts would be triggered (e.g., in cases of noisy metrics), while other measures of time, in addition to or in lieu of days, are also contemplated.

In embodiments, as noted above, the automatically generated configuration settings provided by alert manager <NUM> may include default options when operation data for monitored metrics is not yet available. In an example, a given type of metric may typically be monitored with a time window of a certain duration and with a certain number of consecutive monitoring windows (e.g., a time window duration of <NUM> minutes and <NUM> consecutive monitoring windows), which may be initially selected and applied as default configuration settings for a system alert monitor. Additionally, in some embodiments, default configuration settings may be applied based on other determined configuration settings for the same or similar system metrics in similar contexts. As a non-limiting example, an application or service hosted at a cloud platform for a tenant may be determined as requiring a specific configuration setting for monitoring CPU utilization, and another tenant may also have the same application hosted and expect similar usage; in such a case, the same configuration setting for monitoring CPU utilization may be set as a default configuration setting for the other tenant. These default configuration settings may be subsequently updated based on operation metric data either with, or without, user input or acceptance.

Model engine <NUM> is configured to generate and apply models via the components of alert manager <NUM> described above in order to determine configuration settings for monitoring system metrics. In embodiments, model engine <NUM> comprises a ML engine that generates and applies ML models. Models are generated and/or validated using system metric data, as described herein, such as past metric data that is not used for testing and/or validation of models.

Models <NUM> includes one or more models, in embodiments, utilized by alert manager <NUM> to determine configuration settings such as durations of monitoring windows and numbers of consecutive monitoring windows for monitoring system metrics. Models <NUM> may be generated by model engine <NUM>, and may comprise ML models as well as other types of models, and may be stored as part of memories <NUM>. Various ones of models <NUM> are directed to different types of system metrics, in embodiments, and within types of metrics that are monitored, different models may be applied. As an example, a first model may be utilized for determining configuration settings for CPU utilization of a first service for a tenant of a cloud platform, while a different second model may be utilized for determining configuration settings for CPU utilization of a second service for the tenant. In other words, it is contemplated herein any specific contexts/implementations for the same metric may require different configuration settings for alert monitoring based on expected utilization, user preferences, scale of implementations, etc..

Turning also now to <FIG>, flowchart <NUM> begins with step <NUM>. In step <NUM>, numbers of consecutive instances of time windows that provide an optimal balance between generating accurate system alerts for respective system metrics and times to detect the system alerts are determined. For example, alert manager <NUM> as described above is configured to perform step <NUM> of flowchart <NUM>. That is, alert manager <NUM> is configured to determine respective configuration settings, for monitoring any number of system metrics, which provide for an optimal selection of both alert triggering accuracy and low times to detect for individual system metrics, regardless of their number. In other words, alert manager <NUM> provides for determinations of specific numbers of consecutive instances of a time window duration with a number of consecutive monitoring windows for individual system metric contexts, including for a same system metric that is monitored different scenarios, e.g., different applications, services, expected utilizations, and/or the like.

Referring again to system <NUM> in <FIG>, UI logic <NUM> is configured to generate a UI for presentation to a user by which default and/or recommended configuration settings such as durations of monitoring windows and numbers of consecutive monitoring windows may be accepted/selected by a user of a user device via a single control, object, element, etc., of the UI. UI logic <NUM> may be configured to provide the UI for presentation to the user via network interface <NUM>. UI logic <NUM> is also configured to generate similar UIs for dynamic updating of configuration settings, after initial configuration settings are applied, via a single user selectable option, in embodiments. In some implementations, UI logic <NUM> comprises a portion of alert manager <NUM> and/or alert monitor <NUM>.

Flowchart <NUM> of <FIG> continues with step <NUM>. In step <NUM>, a user interface is provided in which a user is enabled to activate a single user interface control that applies each of the determined numbers of consecutive instances of time windows to a system alert monitor. For example, UI logic <NUM> is configured to generate and provide a UI to a user that enables a one-click selection of determined configuration settings for any number of system metrics to be monitored.

Accordingly, the embodiments herein provide for a user to be enabled to quickly, and feasibly, select any number of optimized configuration settings for system metrics via a single selection.

Again with respect to system <NUM> of <FIG>, alert monitor <NUM> is configured to monitor system operations and functions for various system metrics, including combinations thereof, and to provide alerts for monitored system metrics when alerting conditions described herein are reached. For example, and without limitation, alert monitor <NUM> may be configured to monitor system metrics such as CPU utilization for physical processors and/or virtual machines, memory load, virtual memory, network traffic load, types of network traffic, temperature, persistent storage accesses, persistent storage volumes, other input/output (I/O) accesses, and/or the like. Alert monitor <NUM> is configured to receive configuration settings for monitoring metrics and combinations of metrics from alert manager <NUM>, where the configuration settings include durations of monitoring windows and numbers of consecutive monitoring windows to be used for triggering alerts. Alerts for met altering conditions may be provided as messages, etc., to IT personnel of user devices as described herein over a network.

As noted above for <FIG>, embodiments herein provide for one click monitors in impact time detection for noise reduction in at-scale monitoring. System 100A of <FIG>, system 100B of <FIG>, and/or system <NUM> of <FIG> may be configured to perform functions and operations for such embodiments. It is further contemplated that the systems and components described above are configurable to be combined in any way. <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG> will now be described.

<FIG> shows a flowchart <NUM> for one click monitors in impact time detection for noise reduction in at-scale monitoring, according to example embodiments. System 100A in <FIG>, System 100B in <FIG>, and/or system <NUM> in <FIG> are configured to operate according to flowchart <NUM>, which is an embodiment of flowchart <NUM> of <FIG>. Further structural and operational examples will be apparent to persons skilled in the relevant art(s) based on the following descriptions. Flowchart <NUM> is described below with respect to system <NUM> of <FIG>, and with respect to <FIG>, <FIG>, <FIG>.

<FIG>, <FIG>, each show a graphical representation of a system metric data (plot 500A, plot 500B, plot 500C, and plot 500D, respectively, of a metric data value over time), <FIG> shows a graphical representation of a monotonically decreasing data set corresponding to a system metric (data set <NUM> of alerts per consecutive monitoring windows), and <FIG> shows a graphical representation of monotonically decreasing data sets corresponding to a system metric for training data and validation data (data sets <NUM> of alerts per consecutive monitoring windows), according to example embodiments for one click monitors in impact time detection for noise reduction in at-scale monitoring.

Regarding <FIG>, flowchart <NUM> begins with step <NUM>. In step <NUM>, a threshold metric condition and operational metric data, associated with a past time period, of a metric are received. For example, metric data logic <NUM> and/or threshold determiner <NUM> of alert manager <NUM> in system <NUM> of <FIG> are configured to receive a threshold metric condition and operational metric data associated with a past time period. This information may be received from a system alert monitor such as alert monitor <NUM> of system <NUM> in <FIG>, from an internal or external data storage, from a dynamic threshold service, from a user, and/or the like, in different embodiments.

Referring also to <FIG>, in each of plot 500A, plot 500B, plot 500C, and plot 500D, operational metric data <NUM> of an example metric, and a threshold metric condition <NUM> (or a boundary <NUM>) as a gray box, are shown. In embodiments, such operational metric data and a corresponding threshold metric condition are provided to alert manager <NUM> of system <NUM>.

In step <NUM> of flowchart <NUM> in <FIG>, a threshold determination is performed, based on a first time window defined by a first length, that includes, for each of a different number of consecutive instances of the first time window over a first portion of the past time period, determining a first data value that corresponds to a first number of times the threshold metric condition is met by the operational metric data in the number of consecutive instances of the first time window. For example, threshold determinations are performed by threshold determiner <NUM> of alert manager <NUM> in <FIG>. As described herein, for a given time window having a duration, determinations are made as to how many times, or an average number of times as the data values in embodiments, a threshold metric condition is met by the operational metric data using different numbers of consecutive time windows (i.e., numbers of consecutive time windows) where the condition is met in each of the number of consecutive instances of the time window.

Referring again to <FIG>, plot 500A, plot 500B, plot 500C, and plot 500D, are illustrated as performing threshold determinations, with a time window duration of <NUM> minutes, over number of consecutive instances of the time window of <NUM> window, <NUM> windows, <NUM> windows, and <NUM> windows, respectively, for operational metric data <NUM> against threshold metric condition <NUM>. Plot 500A shows total values <NUM> (as gray circles) for threshold metric condition <NUM> being met, which are many or noisy, given the number of consecutive instances of the time window of <NUM> window. Plot 500B shows total values <NUM> (as gray circles) for threshold metric condition <NUM> being met, which are still many or noisy but are lower in number than in plot 500A, given the number of consecutive instances of the time window of <NUM> windows. Plot 500C shows a total value <NUM> (as a gray circle) for threshold metric condition <NUM> being met, which is a single meeting of threshold metric condition <NUM>, due to the number of consecutive instances of the time window of <NUM> windows being used. Likewise, plot 500D shows a total value <NUM> (as a gray circle) for threshold metric condition <NUM> being met, which is a single meeting of threshold metric condition <NUM>, due to the number of consecutive instances of the time window of <NUM> windows being used. While not shown for brevity, additional iterations with increasing numbers of consecutive instances of the time window may also be performed, according to embodiments.

Referring back to <FIG>, in step <NUM>, a first data set is generated that is monotonically decreasing and that includes a portion of the first data values. For example, data set logic <NUM> of alert manager <NUM> in <FIG> is configured to generate monotonically decreasing data sets and/or graphical representations thereof, based on a portion or all of the data values determined in step <NUM> described above. As an example, each data value determined in step <NUM> may represent an average number of times per number of consecutive instances of the time windows that the threshold metric condition is met. Data set logic <NUM> generates a data set with data values corresponding to the number of consecutives.

Referring now to <FIG> and data set <NUM>, graphically showing numbers alerts per each number of consecutive monitoring time windows, a monotonically decreasing data set representation <NUM> is illustrated as a curve fitting data values as would be generated for each number of consecutive instances of time windows in step <NUM> of flowchart <NUM>.

As would be understood by persons of skill in the relevant art(s) having the benefit of this disclosure, embodiments also provide for performing step <NUM> and/or step <NUM> for one or more other time window durations such as, but not limited to, <NUM> minutes, <NUM> minutes, <NUM> minutes, <NUM> minutes, etc., to generate additional data sets from which number of consecutive instances of the first time window is selected in step <NUM> below.

Referring back again to <FIG>, in step <NUM>, a first number of consecutive instances of the first time window is determined based at least on an amount of change between first data values that are adjacent across the first data set. For example, consecutive instance determiner <NUM> is configured to determine number of consecutive instances of the time window of the different consecutive monitoring time windows (e.g., from <NUM> window, <NUM> windows, <NUM> windows, <NUM> windows, etc., as described above and as shown in <FIG>), in embodiments. Consecutive instance determiner <NUM> determines the number of consecutive instances of the time window based on amounts of change between adjacent data values. In the illustrated example for data set <NUM> in <FIG>, a number of consecutive instances of the time window of <NUM> consecutive time windows is determined.

As shown in <FIG> and data set <NUM>, the data value for a single consecutive time window drops significantly when two consecutive time windows are used (the value drops from <NUM> to <NUM>). Again the data value drops going from two consecutive time windows to three consecutive time windows (the value drops from <NUM> to <NUM>). However, when increasing the consecutive from three to four or more consecutive time windows per data set <NUM>, the decrease in the data value for alerts only decreases by a relatively small amount. In other words, after the data value illustrated as a data value <NUM>, the amount of change between adjacent data values is minimal, while the amount of change between prior adjacent data values is significant. Put another way, at data value <NUM>, the slope of the curve that fits data set representation <NUM> begins to flatten and has a maximum amount of change in the first data set, meaning that there are not significant alert reductions after this point regardless of increasing the number of consecutive time windows, whereas before data value <NUM> the decrease in alerts was relatively large in number. Accordingly, as described above for step <NUM> of flowchart <NUM>, consecutive instance determiner <NUM> determines that a number of consecutive instances of the time window of <NUM> consecutive time windows <NUM> is the initially optimal number of consecutive instances of the time window.

This idea is also illustrated in <FIG> which show plot 500A with many total values <NUM> for threshold metric condition <NUM> being met at a number of consecutive instances of the time window of <NUM> window, plot 500B with many (but significantly fewer than plot 500B) total values <NUM> for threshold metric condition <NUM> being met at a number of consecutive instances of the time window of <NUM> windows, plot 500C with only one total value <NUM> for threshold metric condition <NUM> being met at a number of consecutive instances of the time window of <NUM> windows, and plot 500D also with only one total value <NUM> for threshold metric condition <NUM> being met at a number of consecutive instances of the time window of <NUM> windows. That is, there is not a significant decrease in alerts that would be generated when the consecutive time windows increase from <NUM> to <NUM> windows.

In some embodiments, step <NUM> of flowchart <NUM> may also include adding another time window to the first number of consecutive instances of the first time window based at least on an absolute value of the amount of change between the first data values that are adjacent across the first data set being greater than, or greater than or equal to, a change threshold. Referring also to data set <NUM> in <FIG>, the number of alerts for a number of consecutive instances of the time window of <NUM> windows may be initially determined because the rate of decrease in alerts with <NUM> consecutive time windows is less than that for <NUM> consecutive time windows. However, consecutive instance determiner <NUM> may be configured to finally determine the first number of consecutive instances of the first time window in step <NUM> as <NUM> consecutive time windows as the decrease in the amount of alerts from <NUM> consecutive time windows to <NUM> consecutive time windows has an absolute value that meets or exceeds a change threshold despite the decrease in the slope of the curve between the data values of <NUM> consecutive time windows and <NUM> consecutive time windows.

In <FIG>, and step <NUM> of flowchart <NUM>, the first number of consecutive instances of the first time window is selected as a monitoring interval for a metric corresponding to the operational metric data. For example, consecutive instance determiner <NUM> is configured to select the instance determined in step <NUM> of flowchart <NUM> over any other number of consecutive instances of time windows determined/generated in flowchart <NUM>, for the same or for different time window durations. In some embodiments, the selection of the first number of consecutive instances of the first time window may be made responsive to an indication from a user, via a UI as described herein, of acceptance of the determined and/or default configuration settings. Additionally, validations may be performed for determined numbers of consecutive instances of the time window (e.g., as in step <NUM>), prior to selections thereof, according to embodiments. As an example, and with reference to <FIG>, <FIG> expressly illustrates validation data <NUM> (i.e., test data) that is utilized to verify the determined number of consecutive instances of the time window (the preceding metric data may be referred to as training data and may be used in the steps of flowchart <NUM> described above). In embodiments where ML models are used to perform one or more steps of flowchart <NUM>, validation is performed to confirm that the selection is optimal for all metric data and that the utilized model does not over-fit the metric data used for training. In some examples, validation may be performed using the same, or a different, threshold metric condition.

In a non-limiting, illustrative example implementation of flowchart <NUM>, where the operational metric data used to determine the number of consecutive instances of the time window includes the <NUM> days of past data, validation operations may utilize <NUM> days of past data, where the oldest <NUM> days are used to determine a number of consecutive instances of the time window (a first portion of data), and the most recent day is used to validate the selection (a second portion of data), although any division into separate portions of past data may be used. The validation process may select a different number of consecutive instances of the time window (e.g., in the embodiments shown in <FIG>, validation may select the number of consecutive instances of the time window of <NUM> consecutive time windows). In such cases, the higher number of consecutives, either based on training data or on validation data, may be selected as the number of consecutive instances of the time window.

Referring now to <FIG> and data sets <NUM>, a representation of a training data set <NUM> and a representation of a validation data set <NUM> are shown, according to an example embodiment. As noted above, ML models trained on training data may be validated based on a validation portion of the past metric data. In the illustrated example, a selection of a number of consecutive instances of the time window <NUM> of <NUM> consecutive time windows is determined for training data set <NUM>, while a number of consecutive instances of the time window <NUM> of <NUM> consecutive time windows is determined for validation data set <NUM>. According to embodiments, consecutive instance determiner <NUM> is configured to select the number of consecutive windows as the maximum between number of consecutive instances of the time window <NUM> and number of consecutive instances of the time window <NUM>-here, number of consecutive instances of the time window <NUM> is selected by consecutive instance determiner <NUM>. Additionally, it should be noted that the similarity between number of consecutive instances of the time window <NUM> and number of consecutive instances of the time window <NUM> serves as indicia of validation of the model used.

Again referring to <FIG>, in step <NUM>, the first number of consecutive instances of the first time window is applied, as a monitoring interval, to a system alert monitor that monitors the metric and that triggers system alerts based on the monitoring interval. For example, the instance determined and/or selected by consecutive instance determiner <NUM> in flowchart <NUM> is applied to a system alert monitor such as alert monitor <NUM> for monitoring of the metric during production or operation according to the monitoring interval, i.e., the number of consecutive instances of the time window. As an example, if the number of consecutive instances of the time window is determined to be <NUM>, alert monitor <NUM> will trigger an alert if the threshold metric condition for the metric that is monitored is met in <NUM> consecutive time windows. In some embodiments, the application of the first number of consecutive instances of the time window to the system alert monitor may be made responsive to an indication from a user, via a UI as described herein, of acceptance of the determined and/or default configuration settings.

In <FIG>, a block diagram of a user interface (UI) <NUM> for one click monitors in impact time detection for noise reduction in at-scale monitoring, according to an example embodiment. In embodiments, UI <NUM> is configured for user acceptance of configuration settings for a system alerts monitor, and includes one or more UI controls, elements, and/or objects that may be activated by a user, e.g., via a touch input, a mouse input, etc..

As exemplarily illustrated, UI <NUM> includes a first UI control <NUM>, a second UI control <NUM>, and a third UI control <NUM>. First UI control <NUM>, when activated by a user, indicates acceptance by the user to apply determined and/or default configurations settings, as described herein, to the system alerts monitor. As noted herein, scaled-out systems may include thousands, hundreds of thousands, or more, different metrics and combinations thereof for which system alert monitoring is desired. Activation of first UI control <NUM> enables a user, with a single "click," to configure these settings for monitoring such metrics, and without such an option, it would not be feasible, or even possible, for a user to configure such settings for monitoring. Second UI control <NUM> may allow a user to decline the determined and/or default settings, e.g., a user may manually configure a much reduced number of configuration settings, and third UI control <NUM> may allow a user to view a list recommended configuration settings.

In <FIG>, a flowchart <NUM> for one click monitors in impact time detection for noise reduction in at-scale monitoring is shown, according to an example embodiment. System 100A in <FIG>, System 100B in <FIG>, and/or system <NUM> in <FIG> are configured to operate according to flowchart <NUM>, which is an embodiment of flowchart <NUM> of <FIG>. Further structural and operational examples will be apparent to persons skilled in the relevant art(s) based on the following descriptions. Flowchart <NUM> is described below with respect to system <NUM> of <FIG> and flowchart <NUM> of <FIG>.

Flowchart <NUM> begins at step <NUM>. In step <NUM>, additional operational metric data that is associated with the metric in a time period subsequent to said applying is received from the system alert monitor. For instance, models and configuration settings for metrics monitored by a system alerts monitor may be updated, e.g., dynamically, after initial application such as in step <NUM> of flowchart <NUM> in <FIG> or after prior updating for the system alerts monitor. When additional operational metric data is acquired by a system alert monitor, such as alert monitor <NUM> of system <NUM> in <FIG>, and/or when alerts for one or more metrics are too frequent or have times to detect that are too long, the additional operational metric data is provided to and received by alert manager <NUM> of system <NUM>, as similarly described with respect to step <NUM> of flowchart <NUM>.

In step <NUM>, a second number of consecutive instances of a second time window from a different number consecutive instances of the second time window over a second portion of the time period is determined subsequent to said applying the first number of consecutive instances of the first time window and based at least on a second data set that is monotonically decreasing and that is associated with a second number of times the threshold metric condition is met by the additional operational metric data. For example, consecutive instance determiner <NUM> of system <NUM> in <FIG> is configured to determine an updated number of different consecutive monitoring time windows (e.g., as described above and as shown in step <NUM> of flowchart in <FIG> and in <FIG>), in embodiments. Consecutive instance determiner <NUM> determines the new number consecutive instances of the time windows based on amounts of change between adjacent data values in the additional operational metric data received in step <NUM>.

In step <NUM>, the second number of consecutive instances of the second time window is applied to the system alert monitor, as the monitoring interval, replacing the first number of consecutive instances of the first time window. For example, the number of consecutive instances of the second time window determined and/or selected by consecutive instance determiner <NUM> in step <NUM> is applied to a system alert monitor such as alert monitor <NUM> for monitoring of the metric during production or operation, which replaces the first number of consecutive instances of the first time window that was applied in step <NUM> of flowchart <NUM> in <FIG>. In some embodiments, the application of the updated second number of consecutive instances of the second time window to the system alert monitor may be made responsive to an indication from a user, via a UI as described herein, of acceptance of the updated configuration settings.

Embodiments herein also provide for reversion to other, or prior, models when a generated or updated model determines a number of consecutive instances of the time window with high number of consecutive time windows such that the time to detect for alert triggering is longer than desired. In embodiments, acceptable times to detect may be determined based on IT personnel preference, end user preference, monitoring context, operational situations, time window duration, and/or the like. As a non-limiting example, a number of consecutive instances of the time window with <NUM> or more windows of <NUM> minute duration, may exceed the desired time to detect, while a number of consecutive instances of the time window with <NUM> or more windows of <NUM> minute duration, may exceed the desired time to detect.

In <FIG>, a flowchart <NUM> for one click monitors in impact time detection for noise reduction in at-scale monitoring is shown, according to an example embodiment. System 100A in <FIG>, System 100B in <FIG>, and/or system <NUM> in <FIG> are configured to operate according to flowchart <NUM>, which is an embodiment of flowchart <NUM> of <FIG>. In embodiments, flowchart <NUM> is performed subsequent to determining the first number of consecutive instances of the first time window in step <NUM> of flowchart <NUM> in <FIG> and prior to applying the first number of consecutive instances of the first time window in step <NUM> of flowchart <NUM>. Further structural and operational examples will be apparent to persons skilled in the relevant art(s) based on the following descriptions. Flowchart <NUM> is described below with respect to system <NUM> of <FIG> and flowchart <NUM> of <FIG>.

Flowchart <NUM> begins at step <NUM>. In step <NUM>, it is determined that a number of time windows having met the threshold metric condition in the first number of consecutive instances of the first time window meets a noise condition. For example, one or more of data set logic <NUM> or consecutive instance determiner <NUM> may be configured to detect and determine that the noise for a the first number of consecutive instances of the first time window of consecutive time windows is equally distributed over the training metric data across the time period which it spans. One example of this is when an alert is triggered each day based on application of the model. In other words, the monotonically decreasing data set is initially flattened, and a very high consecutive time window setting is thus needed to reduce the number alerts. Different scenarios may cause this equal distribution of noise, such as an ineffectively trained model, or a threshold metric condition or boundary that is not practical (e.g., a <NUM>% CPU utilization threshold).

In step <NUM>, one or more time windows are added to the first number of consecutive instances of the first time window such that the noise condition is not met. For example, consecutive instance determiner <NUM> may be configured to change the determined number of consecutive instances from step <NUM> of flowchart <NUM> by adding one or more time windows to the number of time windows previously determined. This effectively determines a different number of consecutive instances of the first time window with a longer time to detect, according to embodiments. As a non-limiting example, an initially determined number of consecutive instances of the first time window may use a consecutive time window setting of <NUM> windows, while the different instance in step <NUM> is determined to use a consecutive time window setting of <NUM>, <NUM>, etc., windows As another non-limiting example, consecutive instance determiner <NUM> forces the model to use number of consecutive instances of the first time window such that the sixtieth percentile of alerts each day is zero (e.g., <NUM> days out of the <NUM> day span of data should not have alerts triggered).

<FIG> will now be described. In scenarios for which time windows with longer durations are utilized, there may be a small number, e.g., one or two data points for metric data, which are anomalous such that an alert would be triggered. These data points can be considered justified and should be considered as an alert. As an example, this scenario may present itself during the use of longer durations for monitoring time windows, e.g., a <NUM>-minute duration for a monitoring time window, which is a more stable, or less noisy, duration, and in many cases, the alerts detected in model training are thus justified.

In the context of the embodiments herein, in an application of a single consecutive time window with a <NUM>-minute duration, these justified alerts can be removed by increasing the number of consecutive instances of the first time window to <NUM> consecutive time windows to reduce the alerts triggered. However, this is not always a desired outcome as when alerts are justified, as in these examples here, false negatives are produced. Thus, additional consecutive time windows may be needed to prevent such undesired effects, however, this significantly increases the time to detect alerts for long window durations, e.g., <NUM> minutes or <NUM> minutes, instead of <NUM> minutes. To account for this issue, embodiments also contemplate the "weakening" of the model training to ignore two alerts, by way of example, and determine the number of consecutive instances of the time window for monitoring after this removal. Doing this allows for model generation that results in few consecutive time windows, and thus optimizes the monitoring interval for a time to detect the system alerts that are monitored by the system alert monitor, while still recognizing and allowing for justified alerts to be triggered. In some embodiments, ignoring a number of alerts in this manner may be performed when the window duration meets, or exceeds, a length of a predetermined duration, e.g., <NUM> minutes, <NUM> minutes, <NUM> minutes, and/or the like.

<FIG> each show a graphical representation of system metric data for one click monitors in impact time detection for noise reduction in at-scale monitoring. In <FIG>, a plot 1100A is shown for a single consecutive time window with a <NUM>-minute duration of the example above. Plot 1100A includes training metric data <NUM>, having an anomalous data point <NUM>, and testing metric data <NUM>, having an anomalous data point <NUM>. As noted above for such examples, anomalies in a stable, single consecutive time window with a <NUM>-minute duration are often justified and should not be ignored.

In <FIG>, a plot 1100B is shown for a number of consecutive instances of the first time window having <NUM> windows and a <NUM>-minute time window duration, continuing the example above. Plot 1100B includes training metric data <NUM> and testing metric data <NUM>, which are illustrated as seasonal data, as well as a threshold metric condition <NUM>. There are no data points illustrated in plot 1100B as triggering an alert as models may be trained according to embodiments to increase the consecutive time windows used to reduce alerts. However, for stable time window durations like <NUM> minutes in which anomalous data points would justifiably trigger alerts, models may be trained against data sets that ignore, e.g., <NUM> alert conditions. In this way, the model may determine a single consecutive window instance for a <NUM>-minute time window duration, while still being enabled to detect the anomalous data points as justified alerts.

For example, in <FIG>, a plot 1100C is shown for a consecutive time window instance having <NUM> window and a <NUM>-minute time window duration, continuing the example above. Plot 1100C also includes training metric data <NUM> and testing metric data <NUM>, as well as threshold metric condition <NUM>, as in plot 1100B. However, with a single consecutive time window applied for the model used in plot 1100C detects that the threshold metric condition is met (illustrated as gray circles) in training metric data <NUM> and in testing metric data <NUM>: a value <NUM> and a value <NUM>, respectively. Accordingly, a number of consecutive instances of the time window may be determined, e.g., by consecutive instance determiner <NUM> in <FIG>, as a single window instance for longer duration time windows, in some embodiments.

Embodiments described herein may be implemented in hardware, or hardware combined with software and/or firmware. For example, embodiments described herein may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer readable storage medium. Alternatively, embodiments described herein may be implemented as hardware logic/electrical circuitry.

As noted herein, the embodiments described, including but not limited to, system 100A in <FIG>, system 100B in <FIG>, system <NUM> in <FIG>, UI <NUM> of <FIG>, along with any components and/or subcomponents thereof, as well any operations and portions of flowcharts/flow diagrams described herein and/or further examples described herein, may be implemented in hardware, or hardware with any combination of software and/or firmware, including being implemented as computer program code configured to be executed in one or more processors and stored in a computer readable storage medium, or being implemented as hardware logic/electrical circuitry, such as being implemented together in a system-on-chip (SoC), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a trusted platform module (TPM), and/or the like. A SoC may include an integrated circuit chip that includes one or more of a processor (e.g., a microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and/or further circuits and/or embedded firmware to perform its functions.

Embodiments described herein may be implemented in one or more computing devices similar to a mobile system and/or a computing device in stationary or mobile computer embodiments, including one or more features of mobile systems and/or computing devices described herein, as well as alternative features. The descriptions of computing devices provided herein are provided for purposes of illustration, and are not intended to be limiting. Embodiments may be implemented in further types of computer systems, as would be known to persons skilled in the relevant art(s).

<FIG> depicts an exemplary implementation of a computing device <NUM> in which embodiments may be implemented. For example, embodiments described herein may be implemented in one or more computing devices or systems similar to computing device <NUM>, or multiple instances of computing device <NUM>, in stationary or mobile computer embodiments, including one or more features of computing device <NUM> and/or alternative features. The description of computing device <NUM> provided herein is provided for purposes of illustration, and is not intended to be limiting. Embodiments may be implemented in further types of computer systems, servers, and/or clusters, etc., as would be known to persons skilled in the relevant art(s).

A number of program modules may be stored on the hard disk, magnetic disk, optical disk, ROM, or RAM. These programs include operating system <NUM>, one or more application programs <NUM>, other programs <NUM>, and program data <NUM>. Application programs <NUM> or other programs <NUM> may include, for example, computer program logic (e.g., computer program code or instructions) for implementing embodiments described herein, such as but not limited to system 100A in <FIG>, system 100B in <FIG>, system <NUM> in <FIG>, UI <NUM> of <FIG>, along with any components and/or subcomponents thereof, as well as the flowcharts/flow diagrams described herein, including portions thereof, and/or further examples described herein.

TPM <NUM> may be connected to bus <NUM>, and may be an embodiment of any TPM, as would be understood by one of skill in the relevant art(s) having the benefit of this disclosure. For example, TPM <NUM> may be configured to perform one or more functions or operations of TPMs for various embodiments herein.

As used herein, the terms "computer program medium," "computer-readable medium," "computer-readable storage medium," and "computer-readable storage device," etc., are used to refer to physical hardware media. Examples of such physical hardware media include the hard disk associated with hard disk drive <NUM>, removable magnetic disk <NUM>, removable optical disk <NUM>, other physical hardware media such as RAMs, ROMs, flash memory cards, digital video disks, zip disks, MEMs, nanotechnology-based storage devices, and further types of physical/tangible hardware storage media (including memory <NUM> of <FIG>). Such computer-readable media and/or storage media (e.g., computer-readable storage medium) are distinguished from and non-overlapping with communication media and propagating signals (do not include communication media and propagating signals).

Embodiments are also directed to computer program products comprising computer code or instructions stored on any computer-readable medium or computer-readable storage medium.

As described, systems and devices embodying the techniques herein may be configured and enabled in various ways to perform their respective functions for one click monitors in impact time detection for noise reduction in at-scale monitoring. In embodiments, one or more of the steps or operations of any flowchart and/or flow diagram described herein may not be performed. Moreover, steps or operations in addition to or in lieu of those in any flowchart and/or flow diagram described herein may be performed. Further, in examples, one or more operations of any flowchart and/or flow diagram described herein may be performed out of order, in an alternate sequence, or partially (or completely) concurrently with each other or with other operations.

As described herein, systems, devices, components, etc., of the embodiments that are configured to perform functions and/or operations are also contemplated as performing such functions and/or operations.

According to the described embodiments for one click monitors in impact time detection for noise reduction in at-scale monitoring, an alert manager associated with a system alert monitor is enabled to determine and recommend configuration settings for monitoring any number of metrics, or combinations thereof, during system operation, and enables an end user to accept the determined configuration settings via single UI control, even in at-scale systems with otherwise unmanageable numbers of configuration settings for metrics to be monitored.

The embodiments herein provide for increased accuracy of alert detection with reduced times to detect network issues associated with the alerts. Implementation and deployment efficiencies are also increased by the one-click application of configuration settings to a system alert monitor. This in turn provides improved overall system operations and efficiencies, including less down time and faster diagnoses of issues.

The embodiments herein utilize uniquely trained models and scalable management of configuration settings, and provide for dynamically modifying/updating models for configuration settings based on operational/production metric data to improve issue detection accuracy, while also providing optimal times to detect such issues for the increased accuracy, that were previously not available for software-based services, much less for the specific embodiments described herein. The described embodiments are also adaptable to implementations of scaled-out microservices which provides system-wide flexibility for monitoring operational metrics.

The additional examples and embodiments described in this Section may be applicable to examples disclosed in any other Section or subsection of this disclosure.

Embodiments in this description provide for systems, devices, and methods for one click monitors in impact time detection for noise reduction in at-scale monitoring. For example, a method performed by computing device is described herein for performing such embodiments. The method includes receiving a threshold metric condition, and operational metric data, associated with a past time period, of a metric, and performing a threshold determination, based on a first time window defined by a first length, that includes, for each of a different number of consecutive instances of the first time window over a first portion of the past time period, determining a first data value that corresponds to a first number of times the threshold metric condition is met by the operational metric data in the number of consecutive instances of the first time window. The method also includes generating a first data set that is monotonically decreasing and that includes a portion of the first data values, determining, based at least on an amount of change between first data values that are adjacent across the first data set, a first number of consecutive instances of the first time window, and applying the first number of consecutive instances of the first time window, as a monitoring interval, to a system alert monitor that monitors the metric and that triggers system alerts based on the monitoring interval.

In an embodiment, the method includes, prior to applying, performing a second threshold determination, based on a second time window that is defined by a second length, that includes, for each of a different number of consecutive instances of the second time window over the first portion of the past time period, determining a second data value that corresponds to a second number of times the threshold metric condition is met by the operational metric data in the number of consecutive instances of the second time window, generating a second data set that is monotonically decreasing and that includes a portion of the second data values, determining, based at least on an amount of change between second data values that are adjacent across the second data set, a second number of consecutive instances of the second time window, and selecting the first number of consecutive instances of the first time window as the monitoring interval based at least on a first total amount of time of the first number of consecutive instances of the first time window being less than, or less than or equal to, a second total amount of time of the second number of consecutive instances of the second time window.

In an embodiment, the method includes receiving, prior to said applying, a user input accepting the first number of consecutive instances of the first time window that was determined and a plurality of other numbers of consecutive instances of other time windows, respectively corresponding to other metrics, that are determined based at least on data sets therefor that are monotonically decreasing and that are determined via corresponding threshold determinations, where applying includes applying each of the plurality of other numbers of consecutive instances of other time windows, as other monitoring intervals, to the system alert monitor that monitors the other metrics and that triggers other system alerts based on the other monitoring intervals.

In an embodiment of the method, the first data values are associated with alerts, and the amount of change between first data values that are adjacent across the first data set corresponds to a maximum amount of change in the first data set.

In an embodiment of the method, determining, based at least on the amount of change between the first data values that are adjacent across the first data set, the first number of consecutive instances of the first time window includes adding another time window to the first number of consecutive instances of the first time window based at least on an absolute value of the amount of change between the first data values that are adjacent across the first data set being greater than, or greater than or equal to, a change threshold.

In an embodiment, the method includes, subsequent to determining the first number of consecutive instances of the first time window and prior to applying, determining that a number of time windows having met the threshold metric condition in the first number of consecutive instances of the first time window meets a noise condition, and adding one or more time windows to the first number of consecutive instances of the first time window such that the noise condition is not met.

In an embodiment of the method, the first length meets or exceeds a predetermined duration, and generating the first data set that is monotonically decreasing and that includes the portion of the first data values comprises excluding at least one of the first data values from the first data set that optimizes the monitoring interval for a time to detect the system alerts that are monitored by the system alert monitor.

In an embodiment, the method includes, subsequent to determining the first number of consecutive instances of the first time window and prior to applying, validating the first number of consecutive instances of the first time window by performing the threshold determination, based on the first time window, that includes, for each of a different number of consecutive instances of the first time window over a second portion of the past time period, determining a validation data value that corresponds to a second number of times the threshold metric condition is met by the operational metric data in the number of consecutive instances of the first time window, generating a validation data set that is monotonically decreasing and that includes a portion of the validation data values, determining, based at least on an amount of change between validation data values that are adjacent across the validation data set, a validation number of the consecutive instances of the first time window and validating the first number of consecutive instances of the first time window against the validation number of the consecutive instances of the first time window.

A system is also described herein. The system may be configured and enabled in various ways for one click monitors in impact time detection for noise reduction in at-scale monitoring, as described herein. In an embodiment, the system includes a memory that stores program instructions, and a processing system configured to execute the program instructions. The program instructions include performing a threshold determination, based on a first time window defined by a first length, that includes, for each of a different number of consecutive instances of the first time window over a first portion of the past time period, determining a first data value that corresponds to a first number of times a threshold metric condition, that is received by the system, is met by operational metric data for a metric that associated with a past time period and that is also received by the system, in the number of consecutive instance of the first time window. The program instructions also include generating a first data set that is monotonically decreasing and that includes a portion of the first data values, determining, based at least on an amount of change between first data values that are adjacent across the first data set, a first number of consecutive instances of the first time window, and applying the first number of consecutive instances of the first time window, as a monitoring interval, to a system alert monitor that monitors the metric and that triggers system alerts based on the monitoring interval.

In an embodiment of the system, the program instructions include, prior to applying, performing a second threshold determination, based on a second time window that is defined by a second length, that includes, for each of a different number of consecutive instances of the second time window over the first portion of the past time period, determining a second data value that corresponds to a second number of times the threshold metric condition is met by the operational metric data in the number of consecutive instances of the second time window, generating a second data set that is monotonically decreasing and that includes a portion of the second data values, determining, based at least on an amount of change between second data values that are adjacent across the second data set, a second number of consecutive instances of the second time window, and selecting the first number of consecutive instances of the first time window as the monitoring interval based at least on a first total amount of time of the first number of consecutive instances of the first time window being less than, or less than or equal to, a second total amount of time of the second number of consecutive instances of the second time window.

In an embodiment of the system, the first data values are associated with alerts, and the amount of change between first data values that are adjacent across the first data set corresponds to a maximum amount of change in the first data set.

In an embodiment of the system, determining, based at least on the amount of change between the first data values that are adjacent across the first data set, the first number of consecutive instances of the first time window includes adding another time window to the first number of consecutive instances of the first time window based at least on an absolute value of the amount of change between the first data values that are adjacent across the first data set being greater than, or greater than or equal to, a change threshold.

In an embodiment of the system, the program instructions further include, subsequent to determining the first instance and prior to applying the first instance, determining that a number of time windows having met the threshold metric condition in the first number of consecutive instances of the first time window meets a noise condition, and adding one or more time windows to the first number of consecutive instances of the first time window such that the noise condition is not met.

In an embodiment of the system, the first length meets or exceeds a predetermined duration, and generating the first data set that is monotonically decreasing and that includes the portion of the first data values comprises excluding at least one of the first data values from the first data set that optimizes the monitoring interval for a time to detect the system alerts that are monitored by the system alert monitor.

In an embodiment of the system, the program instructions further include, subsequent to determining the first number of consecutive instances of the first time window and prior to applying, validating the first number of consecutive instances of the first time window by performing the threshold determination, based on the first time window, that includes, for each of a different number of consecutive instances of the first time window over a second portion of the past time period, determining a validation data value that corresponds to a second number of times the threshold metric condition is met by the operational metric data in the number of consecutive instances of the first time window, generating a validation data set that is monotonically decreasing and that includes a portion of the validation data values, determining, based at least on an amount of change between validation data values that are adjacent across the validation data set, a validation number of the consecutive instances of the first time window, and validating the first number of consecutive instances of the first time window against the validation number of the consecutive instances of the first time window.

In an embodiment of the system, validating the first number of consecutive instances of the first time window against the validation number of consecutive instances of the first time window includes at least one of selecting the first number of consecutive instances of the first time window as the monitoring interval based at least on a correlation between the first number of consecutive instances of the first time window and the validation number of consecutive instances of the first time window, wherein the first number of consecutive instances of the first time window includes a greater number of first time windows than the validation number of consecutive instances of the first time window, or selecting the validation number of consecutive instances of the first time window as the first number of consecutive instances of the first time window for the monitoring interval based at least on a correlation between the first number of consecutive instances of the first time window and the validation number of consecutive instances of the first time window, wherein the first number of consecutive instances of the first time window includes a lesser number of first time windows than the validation number of consecutive instances of the first time window.

A computer-readable storage medium having program instructions recorded thereon that, when executed by a processing system, perform a method, is also described. The method and program instructions are for one click monitors in impact time detection for noise reduction in at-scale monitoring, as described herein. The program instructions include receiving a threshold metric condition, and operational metric data, associated with a past time period, of a metric, performing a threshold determination, based on a first time window defined by a first length, that includes, for each of a different number of consecutive instances of the first time window over a first portion of the past time period, determining a first data value that corresponds to a first number of times the threshold metric condition is met by the operational metric data in the number of consecutive instances of the first time window. The program instructions also include generating a first data set that is monotonically decreasing and that includes a portion of the first data values, determining, based at least on an amount of change between first data values that are adjacent across the first data set, a first number of consecutive instances of the first time window, and applying the first number of consecutive instances of the first time window, as a monitoring interval, to a system alert monitor that monitors the metric and that triggers system alerts based on the monitoring interval.

In an embodiment of the computer-readable storage medium, the program instructions include receiving, prior to applying, a user input accepting the first number of consecutive instances of the first time window that was determined and a plurality of other numbers of consecutive instances of other time windows, respectively corresponding to other metrics, that are determined based at least on data sets therefor that are monotonically decreasing and that are determined via corresponding threshold determinations, and applying includes applying each of the plurality of other numbers of consecutive instances of other time windows, as other monitoring intervals, to the system alert monitor that monitors the other metrics and that triggers other system alerts based on the other monitoring intervals.

In an embodiment of the computer-readable storage medium, the program instructions include receiving, from the system alert monitor, additional operational metric data that is associated with the metric in a time period subsequent to said applying the first number of consecutive instances of the first time window, determining, based at least on a second data set that is monotonically decreasing and that is associated with a second number of times the threshold metric condition is met by the additional operational metric data, a second number of consecutive instances of a second time window from a different number consecutive instances of the second time window over a second portion of the time period subsequent to said applying the first number of consecutive instances of the first time window, and applying the second number of consecutive instances of the second time window to the system alert monitor, as the monitoring interval, replacing the first number of consecutive instances of the first time window. In a further embodiment of the computer-readable storage medium, the second time window comprises a same duration as the first time window and the second number of consecutive instances of the second time window comprises a total amount of time greater than the first number of consecutive instances of the first time window.

Claim 1:
A method performed by a computing system, the method comprising:
receiving (<NUM>) a threshold metric condition, and operational metric data, associated with a past time period, of a metric;
performing (<NUM>) a threshold determination, based on a first time window defined by a first length, that includes, for each of a different number of consecutive instances of the first time window over a first portion of the past time period, determining a first data value that corresponds to a first number of times the threshold metric condition is met by the operational metric data in the number of consecutive instances of the first time window;
generating (<NUM>) a first data set that is monotonically decreasing and that includes the first data values;
determining (<NUM>), based at least on an amount of change between first data values that are adjacent across the first data set, a first number of consecutive instances of the first time window; and
applying (<NUM>) the first number of consecutive instances of the first time window, as a monitoring interval, to a system alert monitor that monitors the metric and that triggers system alerts based on the monitoring interval.