Patent Publication Number: US-11641310-B1

Title: Entity lifecycle management in service monitoring system

Description:
TECHNICAL FIELD 
     The present disclosure relates to service monitoring systems and, more particularly, to implementing entity lifecycle management in service monitoring systems. 
     BACKGROUND 
     Modern data centers can have thousands of hosts that operate collectively to service requests from even larger numbers of remote clients. During operation, components of these data centers can produce significant volumes of machine-generated data. The unstructured nature of much of this data has made it challenging to perform indexing and searching operations because of the difficulty of associating semantic meanings with the unstructured data. As the number of data center hosts and clients continues to grow, processing large volumes of machine-generated data in an intelligent manner and effectively presenting the results of such processing continues to be a priority. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present disclosure will be understood more fully from the detailed description given below and from the accompanying drawings of various implementations of the disclosure. 
         FIG.  1    illustrates a block diagram of an example of entities providing a service, in accordance with one or more implementations of the present disclosure. 
         FIG.  2    is a block diagram of one implementation of a service monitoring system, in accordance with one or more implementations of the present disclosure. 
         FIG.  3    is a block diagram illustrating an entity definition for an entity, in accordance with one or more implementations of the present disclosure. 
         FIG.  4    is a block diagram illustrating a service definition that relates one or more entities with a service, in accordance with one or more implementations of the present disclosure. 
         FIG.  5    is a flow diagram of an implementation of a method for creating one or more key performance indicators for a service, in accordance with one or more implementations of the present disclosure. 
         FIG.  6    is a flow diagram of an implementation of a method for creating an entity definition for an entity, in accordance with one or more implementations of the present disclosure. 
         FIG.  7    illustrates an example of a graphical user interface (GUI) for creating and/or editing entity definition(s) and/or service definition(s), in accordance with one or more implementations of the present disclosure. 
         FIG.  8    illustrates an example of a GUI for creating and/or editing entity definitions, in accordance with one or more implementations of the present disclosure. 
         FIG.  9 A  illustrates an example of a GUI for creating an entity definition, in accordance with one or more implementations of the present disclosure. 
         FIG.  9 B  illustrates an example of input received via GUI for creating an entity definition, in accordance with one or more implementations of the present disclosure. 
         FIG.  10    illustrates an example of a GUI of a service monitoring system for creating an entity definition, in accordance with one or more implementations of the present disclosure. 
         FIG.  11    is a flow diagram of an implementation of a method for creating a service definition for a service, in accordance with one or more implementations of the present disclosure. 
         FIG.  12    illustrates an example of a GUI for creating and/or editing service definitions, in accordance with one or more implementations of the present disclosure. 
         FIG.  13    illustrates an example of a GUI for identifying a service for a service definition, in accordance with one or more implementations of the present disclosure. 
         FIG.  14    illustrates an example of a GUI for creating a service definition, in accordance with one or more implementations of the present disclosure. 
         FIG.  15    illustrates an example of a GUI for associating one or more entities with a service by associating one or more entity definitions with a service definition, in accordance with one or more implementations of the present disclosure. 
         FIG.  16    illustrates an example of a GUI facilitating user input for creating an entity definition, in accordance with one or more implementations of the present disclosure. 
         FIG.  17 A  illustrates an example of a GUI indicating one or more entities associated with a service based on input, in accordance with one or more implementations of the present disclosure. 
         FIG.  17 B  illustrates an example of the structure for storing a service definition, in accordance with one or more implementations of the present disclosure. 
         FIG.  17 C  is a block diagram of an example of using filter criteria to dynamically identify one or more entities and to associate the entities with a service, in accordance with one or more implementations of the present disclosure. 
         FIG.  17 D  is a flow diagram of an implementation of a method for using filter criteria to associate entity definition(s) with a service definition, in accordance with one or more implementations of the present disclosure. 
         FIG.  17 E  illustrates an example of a GUI of a service monitoring system for using filter criteria to identify one or more entity definitions to associate with a service definition, in accordance with one or more implementations of the present disclosure. 
         FIG.  17 F  illustrates an example of a GUI of a service monitoring system for specifying filter criteria for a rule, in accordance with one or more implementations of the present disclosure. 
         FIG.  17 G  illustrates an example of a GUI of a service monitoring system for specifying one or more values for a rule, in accordance with one or more implementations of the present disclosure. 
         FIG.  17 H  illustrates an example of a GUI of a service monitoring system for specifying multiple rules for associating one or more entity definitions with a service definition, in accordance with one or more implementations of the present disclosure. 
         FIG.  17 I  illustrates an example of a GUI of a service monitoring system for displaying entity definitions that satisfy filter criteria, in accordance with one or more implementations of the present disclosure. 
         FIG.  18    illustrates an example of a GUI for specifying dependencies for the service, in accordance with one or more implementations of the present disclosure. 
         FIG.  19    is a flow diagram of an implementation of a method for creating one or more key performance indicators (KPIs) for a service, in accordance with one or more implementations of the present disclosure. 
         FIG.  20    is a flow diagram of an implementation of a method for creating a search query, in accordance with one or more implementations of the present disclosure. 
         FIG.  21    illustrates an example of a GUI for creating a KPI for a service, in accordance with one or more implementations of the present disclosure. 
         FIG.  22    illustrates an example of a GUI for creating a KPI for a service, in accordance with one or more implementations of the present disclosure. 
         FIG.  23    illustrates an example of a GUI for receiving input of search processing language for defining a search query for a KPI for a service, in accordance with one or more implementations of the present disclosure. 
         FIG.  24    illustrates an example of a GUI for defining a search query for a KPI using a data model, in accordance with one or more implementations of the present disclosure. 
         FIG.  25    illustrates an example of a GUI for facilitating user input for selecting a data model and an object of the data model to use for the search query, in accordance with one or more implementations of the present disclosure. 
         FIG.  26    illustrates an example of a GUI for displaying a selected statistic, in accordance with one or more implementations of the present disclosure. 
         FIG.  27    illustrates an example of a GUI for editing which entity definitions to use for the KPI, in accordance with one or more implementations of the present disclosure. 
         FIG.  28    is a flow diagram of an implementation of a method for defining one or more thresholds for a KPI, in accordance with one or more implementations of the present disclosure. 
         FIGS.  29 A-B , illustrate examples of a graphical interface enabling a user to set a threshold for the KPI, in accordance with one or more implementations of the present disclosure. 
         FIG.  29 C  illustrates an example GUI  2960  for configuring KPI monitoring in accordance with one or more implementations of the present disclosure. 
         FIG.  30    illustrates an example GUI for enabling a user to set one or more thresholds for the KPI, in accordance with one or more implementations of the present disclosure. 
         FIG.  31 A-C  illustrate example GUIs for defining thresholds for a KPI, in accordance with one or more implementations of the present disclosure. 
         FIGS.  31 D- 31 F  illustrate example GUIs for defining threshold settings for a KPI, in accordance with alternative implementations of the present disclosure. 
         FIG.  31 G  is a flow diagram of an implementation of a method for defining one or more thresholds for a KPI on a per entity basis, in accordance with one or more implementations of the present disclosure. 
         FIG.  32    is a flow diagram of an implementation of a method for calculating an aggregate KPI score for a service based on the KPIs for the service, in accordance with one or more implementations of the present disclosure. 
         FIG.  33 A  illustrates an example GUI  3300  for assigning a frequency of monitoring to a KPI based on user input, in accordance with one or more implementations of the present disclosure. 
         FIG.  33 B  illustrates an example GUI for defining threshold settings, including state ratings, for a KPI, in accordance with one or more implementations of the present disclosure. 
         FIG.  34 A  is a flow diagram of an implementation of a method for calculating a value for an aggregate KPI for the service, in accordance with one or more implementations of the present disclosure. 
         FIG.  34 B  is a flow diagram of an implementation of a method of causing display of a GUI presenting information pertaining to notable events produced as a result of correlation searches, in accordance with one or more implementations of the present disclosure. 
         FIG.  35    is a flow diagram of an implementation of a method for creating a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. 
         FIG.  36 A  illustrates an example GUI for creating and/or editing a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. 
         FIG.  36 B  illustrates an example GUI for a dashboard-creation graphical interface for creating a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. 
         FIG.  37    illustrates an example GUI for a dashboard-creation graphical interface including a user selected background image, in accordance with one or more implementations of the present disclosure. 
         FIG.  38 A  illustrates an example GUI for displaying of a set of KPIs associated with a selected service, in accordance with one or more implementations of the present disclosure. 
         FIG.  38 B  illustrates an example GUI for displaying a set of KPIs associated with a selected service for which a user can select for a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. 
         FIG.  39 A  illustrates an example GUI facilitating user input for selecting a location in the dashboard template and style settings for a KPI widget, and displaying the KPI widget in the dashboard template, in accordance with one or more implementations of the present disclosure. 
         FIG.  39 B  illustrates example KPI widgets, in accordance with one or more implementations of the present disclosure. 
         FIG.  40 A  illustrates an example GUI for creating and/or editing a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. 
         FIG.  40 B  illustrates an example GUI for specifying information for a new service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. 
         FIG.  40 C  illustrates an example GUI for editing a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. 
         FIG.  40 D  illustrates an example interface for using a data model to define an adhoc KPI, in accordance with one or more implementations of the present disclosure. 
         FIG.  40 E  illustrates an example interface for setting one or more thresholds for the adhoc KPI, in accordance with one or more implementations of the present disclosure. 
         FIG.  40 F  illustrates an example interface for a service-related KPI, in accordance with one or more implementations of the present disclosure. 
         FIG.  41 A  is a flow diagram of an implementation of a method for creating and causing for display a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. 
         FIG.  41 B  describes an example service-monitoring dashboard GUI, in accordance with one or more implementations of the present disclosure. 
         FIG.  41 C  illustrates an example service-monitoring dashboard GUI that is displayed in view mode based on the dashboard template, in accordance with one or more implementations of the present disclosure. 
         FIG.  42    describes an example home page GUI for service-level monitoring, in accordance with one or more implementations of the present disclosure. 
         FIG.  43 A  describes an example home page GUI for service-level monitoring, in accordance with one or more implementations of the present disclosure. 
         FIG.  43 B  is a flow diagram of an implementation of a method for creating a home page GUI for service-level and KPI-level monitoring, in accordance with one or more implementations of the present disclosure. 
         FIG.  43 C  illustrates an example of a service-monitoring page, in accordance with one or more implementations of the present disclosure. 
         FIG.  43 D  illustrates an example of a service-monitoring page including a notable events region, in accordance with one or more implementations of the present disclosure. 
         FIGS.  43 E-F  illustrate an example of a service-monitoring page, in accordance with one or more implementations of the present disclosure. 
         FIG.  44    schematically illustrates an example workflow  4400  of entity lifecycle management in a service monitoring system, in accordance with aspects of the present disclosure. 
         FIGS.  45 A- 45 B  schematically illustrate example GUIs for creating and/or editing entity lifecycle management policy definitions, in accordance with aspects of the present disclosure. 
         FIG.  46    schematically illustrates an example GUI for displaying entity lifecycle management policies and performing bulk actions upon user-selected policies, in accordance with aspects of the present disclosure. 
         FIG.  47    schematically illustrates an example GUI for selecting a bulk action to be applied to chosen applied to the entity lifecycle management policies, in accordance with aspects of the present disclosure. 
         FIG.  48    schematically illustrates an example GUI for displaying identified retirement candidate entities and allowing the user to perform individual or bulk actions upon selected entities, in accordance with aspects of the present disclosure. 
         FIG.  49    schematically illustrates an example GUI for displaying identified retired entities and allowing the user to perform individual or bulk actions upon selected entities, in accordance with aspects of the present disclosure. 
         FIG.  50    is a flow diagram of an example method of implementing entity lifecycle management in a service monitoring system operating in accordance with aspects of the present disclosure. 
         FIG.  51    is a flow diagram of an example method of managing retired entities by a service monitoring system operating in accordance with aspects of the present disclosure. 
         FIG.  52    is a flow diagram of an example method of managing retired entities by a service monitoring system operating in accordance with aspects of the present disclosure. 
         FIG.  53    is a flow diagram of an example method  5400  of translating an entity lifecycle management policy by a service monitoring system operating in accordance with aspects of the present disclosure. 
         FIG.  54    presents a block diagram of an event processing system in accordance with one or more implementations of the present disclosure. 
         FIG.  55    presents a flowchart illustrating how indexers process, index, and store data received from forwarders in accordance with one or more implementations of the present disclosure. 
         FIG.  56    presents a flowchart illustrating how a search head and indexers perform a search query in accordance with one or more implementations of the present disclosure. 
         FIG.  57 A  presents a block diagram of a system for processing search requests that uses extraction rules for field values in accordance with one or more implementations of the present disclosure. 
         FIG.  57 B  illustrates an example data model structure, in accordance with some implementations of the present disclosure. 
         FIG.  57 C  illustrates an example definition of a root object of a data model, in accordance with some implementations. 
         FIG.  57 D  illustrates example definitions and of child objects, in accordance with some implementations. 
         FIG.  58    illustrates an exemplary search query received from a client and executed by search peers in accordance with one or more implementations of the present disclosure. 
         FIG.  59 A  illustrates a search screen in accordance with one or more implementations of the present disclosure. 
         FIG.  59 B  illustrates a data summary dialog that enables a user to select various data sources in accordance with one or more implementations of the present disclosure. 
         FIG.  60 A  illustrates a key indicators view in accordance with one or more implementations of the present disclosure. 
         FIG.  60 B  illustrates an incident review dashboard in accordance with one or more implementations of the present disclosure. 
         FIG.  60 C  illustrates a proactive monitoring tree in accordance with one or more implementations of the present disclosure. 
         FIG.  60 D  illustrates a screen displaying both log data and performance data in accordance with one or more implementations of the present disclosure. 
         FIG.  61    depicts a block diagram of an example computing device operating in accordance with one or more implementations of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     The present disclosure is directed to entity lifecycle management in service monitoring systems. A service monitoring system can provide users with insight to the performance of monitored services, such as, services pertaining to an information technology (IT) environment. Each service can be provided by one or more entities. An entity that provides a service can be associated with various structured and unstructured machine data. 
     In some implementations, a service monitoring system can monitor one or more aspects of a service using one or more key performance indicators (KPIs). For example, users can wish to monitor the CPU (central processing unit) usage of a web hosting service, the memory usage of the web hosting service, and the request response time for the web hosting service. Each KPI can be defined by a search query that produces a value derived from the machine data identified in the entity definitions specified in the service definition. In some implementations, a service monitoring system can perform correlation searches to generate notable events and/or alarms based on monitoring a set of KPIs over a defined period. A correlation search can be implemented by a search query associated with a triggering condition and one or more actions corresponding to the trigger condition. The service monitoring system can provide a graphical user interface (GUI), such as a service-monitoring dashboard that can display one or more KPI widgets providing numerical or graphical representations of corresponding KPI values. 
     During the lifetime of a service being monitored, each of the entities providing the service can transition through the stages of its own lifecycle: an entity can be commissioned, associated with a service being monitored, disassociated from the service being monitored, decommissioned, undergo maintenance, returned to service, etc. Failure of the service monitoring system to adequately track and reflect the lifecycle of each entity can result in various undesirable effects, such as multiple false alarms triggered due to the lack of data feeds from inactive entities, user interfaces and reports becoming visually clogged due to a large number of inactive entities, etc. 
     Implementations of the present disclosure address the above-references and other issues by applying user-configurable policies for identifying non-responsive or orphan entities of a service monitoring system and providing users with the ability to identify, retire, and eventually delete these entities. Retired entities are excluded from interacting with any other components of the service monitoring system. A retired entity can stay in the retired state until it is deleted from the system or restored to the active state. 
     The entity lifecycle management is driven by one or more user-defined entity lifecycle management policies. Each policy can specify one or more entity lifecycle management action (e.g., entity retirement) criteria. In an illustrative example, an entity retirement criterion can target non-responsive entities by specifying the threshold period of time in which an entity has not been sending any data. Accordingly, an entity that has not been sending any data for a period of time exceeding the threshold period specified by the entity retirement criterion would be identified as a candidate entity for retirement. In another illustrative example, an entity retirement criterion can target orphan entities by specifying the threshold period of time in which the entity has not been associated with any service. Accordingly, an entity that has not been associated with any service for a period exceeding the threshold period of time specified by the entity retirement criterion would be identified as a candidate entity for retirement. In some implementations, in addition to entity lifecycle management action criteria, an entity lifecycle management policy can specify the entity evaluation frequency, which specifies the frequency of running the search commands produced by translating the entity lifecycle management policy. 
     The search commands derived from the entity lifecycle policy definitions can be executed with the desired frequency, thus identifying certain entities as retirement candidates. These entities can be presented to a user via a GUI that lists all identified candidate entities and allows the user to perform individual or bulk actions upon selected entities. In an illustrative example, the user can retire one or more selected entities, which would become effectively excluded from interacting with any other components of the service monitoring system. In another illustrative example, the user can delete one or more selected entities from the system. In yet another illustrative example, the user can put one or more selected entities into the maintenance mode. In yet another illustrative example, the user can edit the definitions of one or more selected entities. 
     In some implementations, the retired entities can be presented to a user via a GUI that lists all retired entities and allows the user to perform individual or bulk actions upon selected retired entities. In an illustrative example, the user can restore to the active status (un-retire) one or more selected retired entities. In another illustrative example, the user can delete one or more selected retired entities from the system. In yet another illustrative example, the user can edit the definitions of one or more selected retired entities. 
     In some implementations, the service monitoring system can further implement restoring, to the active status, the retired entities that have become active and/or associated with at least one service. In an illustrative example, a saved search can identify retirement candidate entities that started sending data since the retirement event. In another illustrative example, a saved search can identify retirement candidate entities that have become associated with at least one service since the retirement event. The identified entities will then be automatically transitioned into an active status. 
     Thus, implementations of the present disclosure provide an efficient technical solution for efficient entity lifecycle management in service monitoring systems, thus enhancing the functionality and improving the efficiency of these systems, as described in more detail herein below. 
     While the illustrative examples described herein are related to service monitoring systems, in various other implementations, the systems and methods of the present disclosure can be applied to other software systems, platforms, and/or or applications, including, e.g., application performance monitoring systems, data intake and query systems, event processing systems, etc. Likewise, the systems and methods described herein can be applied to cloud-based and on-premises installations. 
       FIG.  1    illustrates a block diagram of an example service provided by entities, in accordance with one or more implementations of the present disclosure. One or more entities  104 A,  104 B provide service  102 . An entity  104 A,  104 B can be a component in an IT environment. Examples of an entity can include, and are not limited to a host machine, a virtual machine, a switch, a firewall, a router, a sensor, etc. For example, the service  102  can be a web hosting service, and the entities  104 A,  104 B can be web servers running on one or more host machines to provide the web hosting service. In another example, an entity could represent a single process on different (physical or virtual) machines. In another example, an entity could represent communication between two different machines. 
     The service  102  can be monitored using one or more KPIs  106  for the service. A KPI is a type of performance measurement. One or more KPIs can be defined for a service. In the illustrated example, three KPIs  106 A-C are defined for service  102 . KPI  106 A can be a measurement of CPU (central processing unit) usage for the service  102 . KPI  106 B can be a measurement of memory usage for the service  102 . KPI  106 C can be a measurement of request response time for the service  102 . 
     In one implementation, KPI  106 A-C is derived based on machine data pertaining to entities  104 A and  104 B that provide the service  102  that is associated with the KPI  106 A-C. In another implementation, KPI  106 A-C is derived based on machine data pertaining to entities other than and/or in addition to entities  104 A and  104 B. In another implementation, input (e.g., user input) can be received that defines a custom query, which does not use entity filtering, and is treated as a KPI. Machine data pertaining to a specific entity can be machine data produced by that entity or machine data about that entity, which is produced by another entity. For example, machine data pertaining to entity  104 A can be derived from different sources that can be hosted by entity  104 A and/or some other entity or entities. 
     A source of machine data can include, for example, a software application, a module, an operating system, a script, an application programming interface, etc. For example, machine data  110 B can be log data that is produced by the operating system of entity  104 A. In another example, machine data  110 C can be produced by a script that is executing on entity  104 A. In yet another example, machine data  110 A can be about an entity  104 A and produced by a software application  120 A that is hosted by another entity to monitor the performance of the entity  104 A through an application programming interface (API). 
     For example, entity  104 A can be a virtual machine and software application  120 A can be executing outside of the virtual machine (e.g., on a hypervisor or a host operating system) to monitor the performance of the virtual machine via an API. The API can generate network packet data including performance measurements for the virtual machine, such as, memory utilization, CPU usage, etc. 
     Similarly, machine data pertaining to entity  104 B can include, for example, machine data  110 D, such as log data produced by the operating system of entity  104 B, and machine data  110 E, such as network packets including http responses generated by a web server hosted by entity  104 B. 
     An association between an entity (e.g., a physical machine) and machine data pertaining to that entity (e.g., machine data produced by different sources hosted by the entity or machine data about the entity that can be produced by sources hosted by some other entity or entities) can be provided via an entity definition that identifies machine data from different sources and links the identified machine data with the actual entity to which the machine data pertains, as will be discussed in more detail below. Entities that are part of a particular service can be further grouped via a service definition that specifies entity definitions of the entities providing the service, as will be discussed in more detail below. 
     In the illustrated example, an entity definition for entity  104 A can associate machine data  110 A,  110 B and  110 C with entity  104 A, an entity definition for entity  104 B can associate machine data  110 D and  110 E with entity  104 B, and a service definition for service  102  can group entities  104 A and  104 B together, thereby defining a pool of machine data that can be operated on to produce KPIs  106 A,  106 B and  106 C for the service  102 . In particular, each KPI  106 A,  106 B,  106 C of the service  102  can be defined by a search query that produces a value  108 A,  108 B,  108 C derived from the machine data  110 A-E. As will be discussed in more detail below, according to one implementation, the machine data  110 A-E is identified in entity definitions of entities  104 A and  104 B, and the entity definitions are specified in a service definition of service  102  for which values  108 A-C are produced to indicate how the service  102  is performing at a point in time or during a period of time. For example, KPI  106 A can be defined by a search query that produces value  108 A indicating how the service  102  is performing with respect to CPU usage. KPI  106 B can be defined by a different search query that produces value  108 B indicating how the service  102  is performing with respect to memory usage. KPI  106 C can be defined by yet another search query that produces value  108 C indicating how the service  102  is performing with respect to request response time. 
     The values  108 A-C for the KPIs can be produced by executing the search query of the respective KPI. In one example, the search query defining a KPI  106 A-C can be executed upon receiving a request (e.g., user request). For example, a service-monitoring dashboard, which is described in greater detail below, can display KPI widgets providing a numerical or graphical representation of the value  108  for a respective KPI  106 . A user can request the service-monitoring dashboard to be displayed at a point in time, and the search queries for the KPIs  106  can be executed in response to the request to produce the value  108  for the respective KPI  106 . The produced values  108  can be displayed in the service-monitoring dashboard. 
     In another example, the search query defining a KPI  106 A-C can be executed in real-time (continuous execution until interrupted). For example, a user can request the service-monitoring dashboard to be displayed, and the search queries for the KPIs  106  can be executed in response to the request to produce the value  108  for the respective KPI  106 . The produced values  108  can be displayed in the service-monitoring dashboard. The search queries for the KPIs  106  can be continuously executed until interrupted and the values for the search queries can be refreshed in the service-monitoring dashboard with each execution. Examples of interruption can include changing graphical interfaces, stopping execution of a program, etc. 
     In another example, the search query defining a KPI  106  can be executed based on a schedule. For example, the search query for a KPI (e.g., KPI  106 A) can be executed at one or more particular times (e.g., 6:00 am, 12:00 μm, 6:00 pm, etc.) and/or based on a period of time (e.g., every 5 minutes). In one example, the values (e.g., values  108 A) produced by a search query for a KPI (e.g., KPI  106 A) by executing the search query on a schedule are stored in a data store, and are used to calculate an aggregate KPI score for a service (e.g., service  102 ), as described in greater detail below. An aggregate KPI score for the service  102  is indicative of an overall performance of the KPIs  106  of the service. 
     In one implementation, the machine data (e.g., machine data  110 A-E) used by a search query defining a KPI (e.g., KPI  106 A) to produce a value can be based on a time range. The time range can be a user-defined time range or a default time range. For example, in the service-monitoring dashboard example above, a user can select, via the service-monitoring dashboard, a time range to use to further specify, for example, based on time-stamps, which machine data should be used by a search query defining a KPI. For example, the time range can be defined as “Last 15 minutes,” which would represent an aggregation period for producing the value. In other words, if the query is executed periodically (e.g., every 5 minutes), the value resulting from each execution can be based on the last 15 minutes on a rolling basis, and the value resulting from each execution can be, for example, the maximum value during a corresponding 15-minute time range, the minimum value during the corresponding 15-minute time range, an average value for the corresponding 15-minute time range, etc. 
     In another implementation, the time range is a selected (e.g., user-selected) point in time and the definition of an individual KPI can specify the aggregation period for the respective KPI. By including the aggregation period for an individual KPI as part of the definition of the respective KPI, multiple KPIs can run on different aggregation periods, which can more accurately represent certain types of aggregations, such as, distinct counts and sums, improving the utility of defined thresholds. In this manner, the value of each KPI can be displayed at a given point in time. In one example, a user can also select “real time” as the point in time to produce the most up to date value for each KPI using its respective individually defined aggregation period. 
     An event processing system can process a search query that defines a KPI of a service. An event processing system can aggregate heterogeneous machine-generated data (machine data) received from various sources (e.g., servers, databases, applications, networks, etc.) and optionally provide filtering such that data is only represented where it pertains to the entities providing the service. In one example, a KPI can be defined by a user-defined custom query that does not use entity filtering. The aggregated machine data can be processed and represented as events. An event can be represented by a data structure that is associated with a certain point in time and comprises a portion of raw machine data (i.e., machine data). Events are described in greater detail below. The event processing system can be conFig.d to perform real-time indexing of the machine data and to execute real-time, scheduled, or historic searches on the source data. An exemplary event processing system is described in greater detail below. 
       FIG.  2    is a block diagram  200  of one implementation of a service monitoring system  210  for monitoring performance of one or more services using key performance indicators derived from machine data, in accordance with one or more implementations of the present disclosure. The service monitoring system  210  can be hosted by one or more computing machines and can include components for monitoring performance of one or more services. The components can include, for example, an entity module  220 , a service module  230 , a key performance indicator module  240 , a user interface (UI) module  250 , a dashboard module  260 , a deep dive module  270 , and a home page module  280 . The components can be combined together or separated in further components, according to a particular embodiment. The components and/or combinations of components can be hosted on a single computing machine and/or multiple computing machines. The components and/or combinations of components can be hosted on one or more client computing machines and/or server computing machines. 
     The entity module  220  can create entity definitions. “Create” hereinafter includes “edit” throughout this document. An entity definition is a data structure that associates an entity (e.g., entity  104 A in  FIG.  1   ) with machine data (e.g., machine data  110 A-C in  FIG.  1   ). The entity module  220  can determine associations between machine data and entities, and can create an entity definition that associates an individual entity with machine data produced by different sources hosted by that entity and/or other entity(ies). In one implementation, the entity module  220  automatically identifies the entities in an environment (e.g., IT environment), automatically determines, for each entity, which machine data is associated with that particular entity, and automatically generates an entity definition for each entity. In another implementation, the entity module  220  receives input (e.g., user input) for creating an entity definition for an entity, as will be discussed in greater detail below. 
       FIG.  3    is a block diagram  300  illustrating an entity definition for an entity, in accordance with one or more implementations of the present disclosure. The entity module  220  can create entity definition  350  that associates an entity  304  with machine data (e.g., machine data  310 A, machine data  310 B, machine data  310 C) pertaining to that entity  304 . Machine data that pertains to a particular entity can be produced by different sources  315  and can be produced in different data formats  330 . For example, the entity  304  can be a host machine that is executing a server application  334  that produces machine data  310 B (e.g., log data). The entity  304  can also host a script  336 , which when executed, produces machine data  310 C. A software application  330 , which is hosted by a different entity (not shown), can monitor the entity  304  and use an API  333  to produce machine data  310 A about the entity  304 . 
     Each of the machine data  310 A-C can include an alias that references the entity  304 . At least some of the aliases for the particular entity  304  can be different from each other. For example, the alias for entity  304  in machine data  310 A can be an identifier (ID) number  315 , the alias for entity  304  in machine data  310 B can be a hostname  317 , and the alias for entity  304  in machine data  310 C can be an IP (internet protocol) address  319 . 
     The entity module  220  can receive input for an identifying name  360  for the entity  304  and can include the identifying name  360  in the entity definition  350 . The identifying name  360  can be defined from input (e.g., user input). For example, the entity  304  can be a web server and the entity module  220  can receive input specifying webserver01.splunk.com as the identifying name  360 . The identifying name  360  can be used to normalize the different aliases of the entity  304  from the machine data  310 A-C to a single identifier. 
     A KPI, for example, for monitoring CPU usage for a service provided by the entity  304 , can be defined by a search query directed to search machine data  310 A-C based a service definition, which is described in greater detail below, associating the entity definition  350  with the KPI, the entity definition  350  associating the entity  304  with the identifying name  360 , and associating the identifying name  360  (e.g., webserver01.splunk.com) with the various aliases (e.g., ID number  315 , hostname  317 , and IP address  319 ). 
     Referring to  FIG.  2   , the service module  230  can create service definitions for services. A service definition is a data structure that associates one or more entities with a service. The service module  230  can receive input (e.g., user input) of a title and/or description for a service definition.  FIG.  4    is a block diagram illustrating a service definition that associates one or more entities with a service, in accordance with one or more implementations of the present disclosure. In another implementation, a service definition specifies one or more other services which a service depends upon and does not associate any entities with the service, as described in greater detail below. In another implementation, a service definition specifies a service as a collection of one or more other services and one or more entities. 
     In one example, a service  402  is provided by one or more entities  404 A-N. For example, entities  404 A-N can be web servers that provide the service  402  (e.g., web hosting service). In another example, a service  402  can be a database service that provides database data to other services (e.g., analytical services). The entities  404 A-N, which provides the database service, can be database servers. 
     The service module  230  can include an entity definition  450 A- 450 N, for a corresponding entity  404 A-N that provides the service  402 , in the service definition  460  for the service  402 . The service module  230  can receive input (e.g., user input) identifying one or more entity definitions to include in a service definition. 
     The service module  230  can include dependencies  470  in the service definition  460 . The dependencies  470  indicate one or more other services for which the service  402  is dependent upon. For example, another set of entities (e.g., host machines) can define a testing environment that provides a sandbox service for isolating and testing untested programming code changes. In another example, a specific set of entities (e.g., host machines) can define a revision control system that provides a revision control service to a development organization. In yet another example, a set of entities (e.g., switches, firewall systems, and routers) can define a network that provides a networking service. The sandbox service can depend on the revision control service and the networking service. The revision control service can depend on the networking service. If the service  402  is the sandbox service and the service definition  460  is for the sandbox service  402 , the dependencies  470  can include the revision control service and the networking service. The service module  230  can receive input specifying the other service(s) for which the service  402  is dependent on and can include the dependencies  470  between the services in the service definition  460 . In one implementation, the service associated defined by the service definition  460  can be designated as a dependency for another service, and the service definition  460  can include information indicating the other services which depend on the service described by the service definition  460 . 
     Referring to  FIG.  2   , the KPI module  240  can create one or more KPIs for a service and include the KPIs in the service definition. For example, in  FIG.  4   , various aspects (e.g., CPU usage, memory usage, response time, etc.) of the service  402  can be monitored using respective KPIs. The KPI module  240  can receive input (e.g., user input) defining a KPI for each aspect of the service  402  to be monitored and include the KPIs (e.g., KPIs  406 A- 406 N) in the service definition  460  for the service  402 . Each KPI can be defined by a search query that can produce a value. For example, the KPI  406 A can be defined by a search query that produces value  408 A, and the KPI  406 N can be defined by a search query that produces value  408 N. 
     The KPI module  240  can receive input specifying the search processing language for the search query defining the KPI. The input can include a search string defining the search query and/or selection of a data model to define the search query. Data models are described in greater detail below. The search query can produce, for a corresponding KPI, value  408 A-N derived from machine data that is identified in the entity definitions  450 A-N that are identified in the service definition  460 . 
     The KPI module  240  can receive input to define one or more thresholds for one or more KPIs. For example, the KPI module  240  can receive input defining one or more thresholds  410 A for KPI  406 A and input defining one or more thresholds  410 N for KPI  406 N. Each threshold defines an end of a range of values representing a certain state for the KPI. Multiple states can be defined for the KPI (e.g., unknown state, trivial state, informational state, normal state, warning state, error state, and critical state), and the current state of the KPI depends on which range the value, which is produced by the search query defining the KPI, falls into. The KPI module  240  can include the threshold definition(s) in the KPI definitions. The service module  230  can include the defined KPIs in the service definition for the service. 
     The KPI module  240  can calculate an aggregate KPI score  480  for the service for continuous monitoring of the service. The score  480  can be a calculated value  482  for the aggregate of the KPIs for the service to indicate an overall performance of the service. For example, if the service has 10 KPIs and if the values produced by the search queries for 9 of the 10 KPIs indicate that the corresponding KPI is in a normal state, then the value  482  for an aggregate KPI can indicate that the overall performance of the service is satisfactory. Some implementations of calculating a value for an aggregate KPI for the service are discussed in greater detail below. 
     Referring to  FIG.  2   , the service monitoring system  210  can be coupled to one or more data stores  290 . The entity definitions, the service definitions, and the KPI definitions can be stored in the data store(s)  290  that are coupled to the service monitoring system  210 . The entity definitions, the service definitions, and the KPI definitions can be stored in a data store  290  in a key-value store, a configuration file, a lookup file, a database, or in metadata fields associated with events representing the machine data. A data store  290  can be a persistent storage that is capable of storing data. A persistent storage can be a local storage unit or a remote storage unit. Persistent storage can be a magnetic storage unit, optical storage unit, solid state storage unit, electronic storage units (main memory), or similar storage unit. Persistent storage can be a monolithic device or a distributed set of devices. A ‘set’, as used herein, refers to any positive whole number of items. 
     The user interface (UI) module  250  can generate graphical interfaces for creating and/or editing entity definitions for entities, creating and/or editing service definitions for services, defining key performance indicators (KPIs) for services, setting thresholds for the KPIs, and defining aggregate KPI scores for services. The graphical interfaces can be user interfaces and/or graphical user interfaces (GUIs). 
     The UI module  250  can cause the display of the graphical interfaces and can receive input via the graphical interfaces. The entity module  220 , service module  230 , KPI module  240 , dashboard module  260 , deep dive module  270 , and home page module  280  can receive input via the graphical interfaces generated by the UI module  250 . The entity module  220 , service module  230 , KPI module  240 , dashboard module  260 , deep dive module  270 , and home page module  280  can provide data to be displayed in the graphical interfaces to the UI module  250 , and the UI module  250  can cause the display of the data in the graphical interfaces. 
     The dashboard module  260  can create a service-monitoring dashboard. In one implementation, dashboard module  260  works in connection with UI module  250  to present a dashboard-creation graphical interface that includes a modifiable dashboard template, an interface containing drawing tools to customize a service-monitoring dashboard to define flow charts, text and connections between different elements on the service-monitoring dashboard, a KPI-selection interface and/or service selection interface, and a configuration interface for creating service-monitoring dashboard. The service-monitoring dashboard displays one or more KPI widgets. Each KPI widget can provide a numerical or graphical representation of one or more values for a corresponding KPI indicating how an aspect of a service is performing at one or more points in time. Dashboard module  260  can work in connection with UI module  250  to define the service-monitoring dashboard in response to user input, and to cause display of the service-monitoring dashboard including the one or more KPI widgets. The input can be used to customize the service-monitoring dashboard. The input can include for example, selection of one or more images for the service-monitoring dashboard (e.g., a background image for the service-monitoring dashboard, an image to represent an entity and/or service), creation and representation of adhoc search in the form of KPI widgets, selection of one or more KPIs to represent in the service-monitoring dashboard, selection of a KPI widget for each selected KPI. The input can be stored in the one or more data stores  290  that are coupled to the dashboard module  260 . In other implementations, some other software or hardware module can perform the actions associated with generating and displaying the service-monitoring dashboard, although the general functionality and features of the service-monitoring dashboard should remain as described herein. Some implementations of creating the service-monitoring dashboard and causing display of the service-monitoring dashboard are discussed in greater detail below. 
     In one implementation, deep dive module  270  works in connection with UI module  250  to present a wizard for creation and editing of the deep dive visual interface, to generate the deep dive visual interface in response to user input, and to cause display of the deep dive visual interface including the one or more graphical visualizations. The input can be stored in the one or more data stores  290  that are coupled to the deep dive module  270 . In other implementations, some other software or hardware module can perform the actions associated with generating and displaying the deep dive visual interface, although the general functionality and features of deep dive should remain as described herein. Some implementations of creating the deep dive visual interface and causing display of the deep dive visual interface are discussed in greater detail below. 
     The home page module  280  can create a home page graphical interface. The home page graphical interface can include one or more tiles, where each tile represents a service-related alarm, service-monitoring dashboard, a deep dive visual interface, or the value of a particular KPI. In one implementation home page module  280  works in connection with UI module  250 . The UI module  250  can cause the display of the home page graphical interface. The home page module  280  can receive input (e.g., user input) to request a service-monitoring dashboard or a deep dive to be displayed. The input can include for example, selection of a tile representing a service-monitoring dashboard or a deep dive. In other implementations, some other software or hardware module can perform the actions associated with generating and displaying the home page graphical interface, although the general functionality and features of the home page graphical interface should remain as described herein. An example home page graphical interface is discussed in greater detail below. 
     Referring to  FIG.  2   , the service monitoring system  210  can be coupled to an event processing system  205  via one or more networks. The event processing system  205  can receive a request from the service monitoring system  210  to process a search query. For example, the dashboard module  260  can receive input request to display a service-monitoring dashboard with one or more KPI widgets. The dashboard module  260  can request the event processing system  205  to process a search query for each KPI represented by a KPI widget in the service-monitoring dashboard. Some implementations of an event processing system  205  are discussed in greater detail below. 
     The one or more networks can include one or more public networks (e.g., the Internet), one or more private networks (e.g., a local area network (LAN) or one or more wide area networks (WAN)), one or more wired networks (e.g., Ethernet network), one or more wireless networks (e.g., an 802.11 network or a Wi-Fi network), one or more cellular networks (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and/or a combination thereof. 
       FIG.  5    is a flow diagram of an implementation of a method  500  for creating one or more key performance indicators for a service, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, at least a portion of method is performed by a client computing machine. In another implementation, at least a portion of method is performed by a server computing machine. 
     At block  502 , the computing machine creates one or more entity definitions, each for a corresponding entity. Each entity definition associates an entity with machine data that pertains to that entity. As described above, various machine data can be associated with a particular entity, but can use different aliases for identifying the same entity. The entity definition for an entity normalizes the different aliases of that entity. In one implementation, the computing machine receives input for creating the entity definition. The input can be user input. Some implementations of creating an entity definition for an entity from input received via a graphical user interface are discussed in greater detail below. 
     In another implementation, the computing machine imports a data file (e.g., CSV (comma-separated values) data file) that includes information identifying entities in an environment and uses the data file to automatically create entity definitions for the entities described in the data file. The data file can be stored in a data store (e.g., data store  290  in  FIG.  2   ) that is coupled to the computing machine. 
     In another implementation, the computing machine automatically (without any user input) identifies one or more aliases for an entity in machine data, and automatically creates an entity definition in response to automatically identifying the aliases of the entity in the machine data. For example, the computing machine can execute a search query from a saved search to extract data to identify an alias for an entity in machine data from one or more sources, and automatically create an entity definition for the entity based on the identified aliases. Some implementations of creating an entity definition from importing a data file and/or from a saved search are discussed in greater detail below. 
     At block  504 , the computing machine creates a service definition for a service using the entity definitions of the one or more entities that provide the service, according to one implementation. A service definition can relate one or more entities to a service. For example, the service definition can include an entity definition for each of the entities that provide the service. In one implementation, the computing machine receives input (e.g., user input) for creating the service definition. Some implementations of creating a service definition from input received via a graphical interface are discussed in more detail below. In one implementation, the computing machine automatically creates a service definition for a service. In another example, a service can not directly be provided by one or more entities, and the service definition for the service can not directly relate one or more entities to the service. For example, a service definition for a service can not contain any entity definitions and can contain information indicating that the service is dependent on one or more other services. A service that is dependent on one or more other services is described in greater detail below. For example, a business service can not be directly provided by one or more entities and can be dependent on one or more other services. For example, an online store service can depend on an e-commerce service provided by an e-commerce system, a database service, and a network service. The online store service can be monitored via the entities of the other services (e.g., e-commerce service, database service, and network service) upon which the service depends on. 
     At block  506 , the computing machine creates one or more key performance indicators (KPIs) corresponding to one or more aspects of the service. An aspect of a service can refer to a certain characteristic of the service that can be measured at various points in time during the operation of the service. For example, aspects of a web hosting service can include request response time, CPU usage, and memory usage. Each KPI for the service can be defined by a search query that produces a value derived from the machine data that is identified in the entity definitions included in the service definition for the service. Each value is indicative of how an aspect of the service is performing at a point in time or during a period of time. In one implementation, the computing machine receives input (e.g., user input) for creating the KPI(s) for the service. Some implementations of creating KPI(s) for a service from input received via a graphical interface will be discussed in greater detail below. In one implementation, the computing machine automatically creates one or more key performance indicators (KPIs) corresponding to one or more aspects of the service. 
       FIG.  6    is a flow diagram of an implementation of a method  600  for creating an entity definition for an entity, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, at least a portion of method is performed by a client computing machine. In another implementation, at least a portion of method is performed by a server computing machine. 
     At block  602 , the computing machine receives input of an identifying name for referencing the entity definition for an entity. The input can be user input. The user input can be received via a graphical interface. Some implementations of creating an entity definition via input received from a graphical interface are discussed in greater detail below. The identifying name can be a unique name. 
     At block  604 , the computing machine receives input (e.g., user input) specifying one or more search fields (“fields”) representing the entity in machine data from different sources, to be used to normalize different aliases of the entity. Machine data can be represented as events. As described above, the computing machine can be coupled to an event processing system (e.g., event processing system  205  in  FIG.  2   ). The event processing system can process machine data to represent the machine data as events. Each of the events is raw data, and when a late-binding schema is applied to the events, values for fields defined by the schema are extracted from the events. A number of “default fields” that specify metadata about the events rather than data in the events themselves can be created automatically. For example, such default fields can specify: a timestamp for the event data; a host from which the event data originated; a source of the event data; and a source type for the event data. These default fields can be determined automatically when the events are created, indexed or stored. Each event has metadata associated with the respective event. Implementations of the event processing system processing the machine data to be represented as events are discussed in greater detail below. 
     At block  606 , the computing machine receives input (e.g., user input) specifying one or more search values (“values”) for the fields to establish associations between the entity and machine data. The values can be used to search for the events that have matching values for the above fields. The entity can be associated with the machine data that is represented by the events that have fields that store values that match the received input. 
     The computing machine can optionally also receive input (e.g., user input) specifying a type of entity to which the entity definition applies. The computing machine can optionally also receive input (e.g., user input) associating the entity of the entity definition with one or more services. Some implementations of receiving input for an entity type for an entity definition and associating the entity with one or more services are discussed in greater detail below. 
       FIG.  7    illustrates an example of a GUI  700  of a service monitoring system for creating and/or editing entity definition(s) and/or service definition(s), in accordance with one or more implementations of the present disclosure. One or more GUIs of the service monitoring system can include GUI elements to receive input and to display data. The GUI elements can include, for example, and are not limited to, a text box, a button, a link, a selection button, a drop down menu, a sliding bar, a selection button, an input field, etc. In one implementation, GUI  700  includes a menu item, such as Configure  702 , to facilitate the creation of entity definitions and service definitions. 
     Upon the selection of the Configure  702  menu item, a drop-down menu  704  listing configuration options can be displayed. If the user selects the entities option  706  from the drop-down menu  704 , a GUI for creating an entity definition can be displayed, as discussed in more detail below. If the user selects the services option  708  from the drop-down menu  704 , a GUI for creating a service definition can be displayed, as discussed in more detail below. 
       FIG.  8    illustrates an example of a GUI  800  of a service monitoring system for creating and/or editing entity definitions, in accordance with one or more implementations of the present disclosure. GUI  800  can display a list  802  of entity definitions that have already been created. Each entity definition in the list  802  can include a button  804  for requesting a drop-down menu  810  listing editing options to edit the corresponding entity definition. Editing can include editing the entity definition and/or deleting the entity definition. When an editing option is selected from the drop-down menu  810 , one or more additional GUIs can be displayed for editing the entity definition. GUI  800  can include an import button  806  for importing a data file (e.g., CSV file) for auto-discovery of entities and automatic generation of entity definitions for the discovered entities. The data file can include a list of entities that exist in an environment (e.g., IT environment). The service monitoring system can use the data file to automatically create an entity definition for an entity in the list. In one implementation, the service monitoring system uses the data file to automatically create an entity definition for each entity in the list. GUI  800  can include a button  808  that a user can activate to proceed to the creation of an entity definition, which leads to GUI  900  of  FIG.  9 A . The automatic generation of entity definitions for entities is described in greater detail below. 
       FIG.  9 A  illustrates an example of a GUI  900  of a service monitoring system for creating an entity definition, in accordance with one or more implementations of the present disclosure. GUI  900  can facilitate user input specifying an identifying name  904  for the entity, an entity type  906  for the entity, field(s)  908  and value(s)  910  for the fields  908  to use during the search to find events pertaining to the entity, and any services  912  that the entity provides. The entity type  906  can describe the particular entity. For example, the entity can be a host machine that is executing a webserver application that produces machine data.  FIG.  9 B  illustrates an example of input received via GUI  900  for creating an entity definition, in accordance with one or more implementations of the present disclosure. 
     For example, the identifying name  904  is webserver01.splunk.com and the entity type  906  is web server. Examples of entity type can include, and are not limited to, host machine, virtual machine, type of server (e.g., web server, email server, database server, etc.) switch, firewall, router, sensor, etc. The fields  908  that are part of the entity definition can be used to normalize the various aliases for the entity. For example, the entity definition specifies three fields  920 , 922 , 924  and four values  910  (e.g., values  930 , 932 , 934 , 936 ) to associate the entity with the events that include any of the four values in any of the three fields. 
     For example, the event processing system (e.g., event processing system  205  in  FIG.  2   ) can apply a late-binding schema to the events to extract values for fields (e.g., host field, ip field, and dest field) defined by the schema and determine which events have values that are extracted for a host field that includes 10.11.12.13, webserver01.splunk.com, webserver01, or vm-0123, determine which events have values that are extracted for an ip field that includes 10.11.12.13, webserver01.splunk.com, webserver01, or vm-0123, or a dest field that includes 10.11.12.13, webserver01.splunk.com, webserver01, or vm-0123. The machine data that relates to the events that are produced from the search is the machine data that is associated with the entity webserver01.splunk.com. 
     In another implementation, the entity definition can specify one or more values  910  to use for a specific field  908 . For example, the value  930  (10.11.12.13) can be used for extracting values for the ip field and determine which values match the value  930 , and the value  932  (webserver01.splunk.com) and the value  936  (vm-0123) can be used for extracting values for the host  920  field and determining which values match the value  932  or value  936 . 
     In another implementation, GUI  900  includes a list of identifying field/value pairs. A search term that is modeled after these entities can constructed, such that, when a late-binding schema is applied to events, values that match the identifiers associated with the fields defined by the schema will be extracted. For example, if identifier.fields=“X,Y” then the entity definition should include input specifying fields labeled “X” and “Y”. The entity definition should also include input mapping the fields. For example, the entity definition can include the mapping of the fields as “X”:“1”,“Y”:[“2”,“3”]. The event processing system (e.g., event processing system  205  in  FIG.  2   ) can apply a late-binding schema to the events to extract values for fields (e.g., X and Y) defined by the schema and determine which events have values extracted for an X field that include “1”, or which events have values extracted for a Y field that include “2”, or which events have values extracted for a Y field that include “3”. 
     GUI  900  can facilitate user input specifying any services  912  that the entity provides. The input can specify one or more services that have corresponding service definitions. For example, if there is a service definition for a service named web hosting service that is provided by the entity corresponding to the entity definition, then a user can specify the web hosting service as a service  912  in the entity definition. 
     The save button  916  can be selected to save the entity definition in a data store (e.g., data store  290  in  FIG.  2   ). The saved entity definition can be edited. 
       FIG.  10    illustrates an example of a GUI  950  of a service monitoring system for creating an entity definition, in accordance with one or more implementations of the present disclosure. GUI  950  can include text boxes  952 A-B that enables a user to specify a field name—field value pair  951  to use during the search to find events pertaining to the entity. User input can be received via GUI  950  for specify one or more field name—field value pairs  951 . In one implementation, the text boxes  952 A-B are automatically populated with field name—field value pair  951  information that was previous specified for the entity definition. GUI  950  can include a button  955 , which when selected, display additional text boxes  952 A-B for specifying a field name—field value pair  951 . 
     GUI  950  can include text boxes  953 A-B that enables a user to specify a name—value pair for informational fields. Informational fields are described in greater detail below. GUI  950  can include a button, which when selected, display additional text boxes  953 A-B for specifying a name—value pair for an informational field. 
     GUI  950  can include a text box  954  that enables a user to associate the entity being represented by the entity definition with one or more services. In one implementation, user input of one or more strings that identify the one or more service is received via text box  954 . In one implementation, when text box  954  is selected (e.g., clicked) a list of service definition is displayed which a user can select from. The list can be populated using service definitions that are stored in a service monitoring data store, as described in greater detail below. 
       FIG.  11    is a flow diagram of an implementation of a method  1100  for creating a service definition for a service, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, at least a portion of method is performed by a client computing machine. In another implementation, at least a portion of method is performed by a server computing machine. 
     At block  1102 , the computing machine receives input of a title for referencing a service definition for a service. At block  1104 , the computing machine receives input identifying one or more entities providing the service and associates the identified entities with the service definition of the service at block  1106 . 
     At block  1108 , the computing machine creates one or more key performance indicators for the service and associates the key performance indicators with the service definition of the service at block  1110 . 
     At block  1112 , the computing machine receives input identifying one or more other services which the service is dependent upon and associates the identified other services with the service definition of the service at block  1114 . The computing machine can include an indication in the service definition that the service is dependent on another service for which a service definition has been created. 
     At block  1116 , the computing machine can optionally define an aggregate KPI score to be calculated for the service to indicate an overall performance of the service. The score can be a value for an aggregate of the KPIs for the service. The aggregate KPI score can be periodically calculated for continuous monitoring of the service. For example, the aggregate KPI score for a service can be updated in real-time (continuously updated until interrupted). In one implementation, the aggregate KPI score for a service is updated periodically (e.g., every second). 
       FIG.  12    illustrates an example of a GUI  1200  of a service monitoring system for creating and/or editing service definitions, in accordance with one or more implementations of the present disclosure. GUI  1200  can display a list  1202  of service definitions that have already been created. Each service definition in the list  1202  can include a button  1204  to proceed to a drop-down menu  1208  listing editing options related to the corresponding service definition. Editing options can include editing the service definition, editing one or more KPIs for the service, editing a title and/or description of the service description, and/or deleting the service definition. When an editing option is selected from the drop-down menu  1208 , one or more other GUIs can be displayed for editing the service definition. GUI  1200  can include a button  1210  to proceed to the creation of a new service definition. 
       FIG.  13    illustrates an example of a GUI  1300  of a service monitoring system for creating a service definition, in accordance with one or more implementations of the present disclosure. GUI  1300  can facilitate user input specifying a title  1302  and optionally a description  1304  for the service definition for a service. GUI  1300  can include a button  1306  to proceed to GUI  1400  of  FIG.  14   , for associating entities with the service, creating KPIs for the service, and indicating dependencies for the service. 
       FIG.  14    illustrates an example of a GUI  1400  of a service monitoring system for defining elements of a service definition, in accordance with one or more implementations of the present disclosure. GUI  1400  can include an accordion pane (accordion section)  1402 , which when selected, displays fields for facilitating input for creating and/or editing a title  1404  of a service definition, and input for a description  1406  of the service that corresponds to the service definition. If input for the title  1404  and/or description  1406  was previously received, for example, from GUI  1300  in  FIG.  13   , GUI  1400  can display the title  1404  and description  1406 . 
     GUI  1400  can include a drop-down  1410  for receiving input for creating one or more KPIs for the service. If the drop-down  1410  is selected, GUI  1900  in  FIG.  19    is displayed as described in greater detail below. 
     GUI  1400  can include a drop-down  1412  for receiving input for specifying dependencies for the service. If the drop-down  1412  is selected, GUI  1800  in  FIG.  18    is displayed as described in greater detail below. 
     GUI  1400  can include one or more buttons  1408  to specify whether entities are associated with the service. A selection of “No”  1416  indicates that the service is not associated with any entities and the service definition is not associated with any entity definitions. For example, a service can not be associated with any entities if an end user intends to use the service and corresponding service definition for testing purposes and/or experimental purposes. In another example, a service can not be associated with any entities if the service is dependent one or more other services, and the service is being monitored via the entities of the one or more other services upon which the service depends upon. For example, an end user can wish to use a service without entities as a way to track a business service based on the services which the business service depends upon. If “Yes”  1414  is selected, GUI  1500  in  FIG.  15    is displayed as described in greater detail below. 
       FIG.  15    illustrates an example of a GUI  1500  of a service monitoring system for associating one or more entities with a service by associating one or more entity definitions with a service definition, in accordance with one or more implementations of the present disclosure. GUI  1500  can include a button  1510  for creating a new entity definition. If button  1510  is selected, GUI  1600  in  FIG.  16    is displayed facilitating user input for creating an entity definition. 
       FIG.  16    illustrates an example of a GUI  1600  facilitating user input for creating an entity definition, in accordance with one or more implementations of the present disclosure. For example, GUI  1600  can include multiple fields  1601  for creating an entity definition, as discussed above. GUI  1600  can include a button  1603 , which when selected can display one or more UIs (e.g., GUIs or command line interface) for importing a data file for creating an entity definition. The data file can be a CSV (comma-separated values) data file that includes information identifying entities in an environment. The data file can be used to automatically create entity definitions for the entities described in the data file. GUI  1600  can include a button  1605 , which when selected can display one or more UIs (e.g., GUIs or command line interface) for using a saved search for creating an entity definition. For example, the computing machine can execute a search query from a saved search to extract data to identify an alias for an entity in machine data from one or more sources, and automatically create an entity definition for the entity based on the identified aliases. 
     Referring to  FIG.  15   , GUI  1500  can include an availability list  1504  of entity definitions for entities, which can be selected to be associated with the service definition. The availability list  1504  can include one or more entity definitions. For example, the availability list  1504  can include thousands of entity definitions. GUI  1500  can include a filter box  1502  to receive input for filtering the availability list  1504  of entity definitions to display a portion of the entity definitions. Each entity definition in the availability list  1502  can include the entity definition name  1506  and the entity type  1508 . GUI  1500  can facilitate user input for selecting an entity definition from the availability list  1504  and dragging the selected entity definition to a selected list  1512  to indicate that the entity for the selected entity definition is associated with service of the service definition. For example, entity definition  1514  (e.g., webserver01.splunk.com) can be selected and dragged to the selected list  1512 . 
       FIG.  17 A  illustrates an example of a GUI  1700  indicating one or more entities associated with a service based on input, in accordance with one or more implementations of the present disclosure. The selected list  1712  can include the entity definition (e.g., webserver01.splunk.com) that was dragged from the availability list  1704 . The availability list  1704  can remove any selected entity definitions (e.g., webserver01.splunk.com). The selected list  1712  indicates which entities are members of a service via the entity definitions of the entities and service definition for the service. 
       FIG.  17 B  illustrates an example of the structure  1720  for storing a service definition, in accordance with one or more implementations of the present disclosure. A service definition can be stored in a service monitoring data store as a record that contains information about one or more characteristics of a service. Various characteristics of a service include, for example, a name of the service, the entities that are associated with the service, the key performance indicators (KPIs) for the service, one or more other services that depend upon the service, one or more other services which the service depends upon, and other information pertaining to the service. 
     The service definition structure  1720  includes one or more components. Each service definition component relates to a characteristic of the service. For example, there is a service name component  1721 , one or more entity filter criteria components  1723 A-B, one or more entity association indicator components  1725 , one or more KPI components  1727 , one or more service dependencies components  1729 , and one or more components for other information  1731 . The characteristic of the service being represented by a particular component is the particular service definition component&#39;s type. In one implementation, the entity filter criteria components  1723 A are stored in a service definition. In another implementation, the entity filter criteria components  1723 B are stored in association with a service definition (e.g., separately from the service definition but linked to the service definition using, for example, identifiers of the entity filter criteria components  1723 B and/or an identifier of the service definition). 
     The entity definitions that are associated with a service definition can change. In one implementation, as described above, users can manually and explicitly select entity definitions from a list (e.g., list  1504  in GUI  1500  in  FIG.  15   ) of pre-defined entities to include in a service definition to reflect the environment changes. In another implementation, the entity filter criteria component(s)  1723 A-B can include filter criteria that can be used for automatically identifying one or more entity definitions to be associated with the service definition without user interaction. The filter criteria in the entity filter criteria components  1723 A-B can be processed to search the entity definitions that are stored in a service monitoring data store for any entity definitions that satisfy the filter criteria. The entity definitions that satisfy the filter criteria can be associated with the service definition. The entity association indicator component(s)  1725  can include information that identifies the one or more entity definitions that satisfy the filter criteria and associates those entity definitions with the service definition, thereby creating an association between a service and one or more entities. One implementation for using filter criteria and entity association indicators to identify entity definition(s) and to associate the identified entity definition(s) with a service definition is described in greater detail below. 
     The KPI component(s)  1727  can include information that describes one or more KPIs for monitoring the service. As described above, a KPI is a type of performance measurement. For example, various aspects (e.g., CPU usage, memory usage, response time, etc.) of the service can be monitored using respective KPIs. 
     The service dependencies component(s)  1729  can include information describing one or more other services for which the service is dependent upon, and/or one or more other services which depend on the service being represented by the service definition. In one implementation, a service definition specifies one or more other services which a service depends upon and does not associate any entities with the service, as described in greater detail below. In another implementation, a service definition specifies a service as a collection of one or more other services and one or more entities. Each service definition component stores information for an element. The information can include an element name and one or more element values for the element. 
     In one implementation, the element name—element value pair(s) within a service definition component serves as a field name-field value pair for a search query. In one implementation, the search query is directed to search a service monitoring data store storing service monitoring data pertaining to the service monitoring system. The service monitoring data can include, and is not limited to, entity definition, service definitions, and key performance indicator (KPI) specifications. 
     In one example, an element name—element value pair in the entity filter criteria component  1723 A-B in the service definition can be used to search the entity definitions in the service monitoring data store for the entity definitions that have matching values for the elements that are named in the entity filter criteria component  1723 A-B. 
     Each entity filter criteria component  1723 A-B corresponds to a rule for applying one or more filter criteria defined by the element name-element value pair to the entity definitions. A rule for applying filter criteria can include an execution type and an execution parameter. User input can be received specifying filter criteria, execution types, and execution parameters via a graphical user interface (GUI), as described in greater detail below. The execution type specifies whether the rule for applying the filter criteria to the entity definitions should be executed dynamically or statically. For example, the execution type can be static execution or dynamic execution. A rule having a static execution type can be executed to create associations between the service definition and the entity definitions on a single occurrence based on the content of the entity definitions in a service monitoring data store at the time the static rule is executed. A rule having a dynamic execution type can be initially executed to create current associations between the service definition and the entity definitions, and can then be re-executed to possibly modify those associations based on the then-current content of the entity definitions in a service monitoring data store at the time of re-execution. For example, if the execution type is static execution, the filter criteria can be applied to the entity definitions in the service monitoring data store only once. If the execution type is dynamic execution, the filter criteria can automatically be applied to the entity definitions in the service monitoring data store repeatedly. 
     The execution parameter specifies when the filter criteria should be applied to the entity definitions in the service monitoring data store. For example, for a static execution type, the execution parameter can specify that the filter criteria should be applied when the service definition is created or when a corresponding filter criteria component is added to (or modified in) the service definition. In another example, for a static execution type, the execution parameter can specify that the filter criteria should be applied when a corresponding KPI is first calculated for the service. 
     For a dynamic execution type, the execution parameter can specify that the filter criteria should be applied each time a change to the entity definitions in the service monitoring data store is detected. The change can include, for example, adding a new entity definition to the service monitoring data store, editing an existing entity definition, deleting an entity definition, etc. In another example, the execution parameter can specify that the filter criteria should be applied each time a corresponding KPI is calculated for the service. 
     In one implementation, for each entity definition that has been identified as satisfying any of the filter criteria in the entity filter criteria components  1723 A-B for a service, an entity association indicator component  1725  is added to the service definition  1720 . 
       FIG.  17 C  is a block diagram  1750  of an example of using filter criteria to dynamically identify one or more entities and to associate the entities with a service, in accordance with one or more implementations of the present disclosure. 
     A service monitoring data store can store any number of entity definitions  1751 A-B. As described above, an entity definition  1751 A-B can include an entity name component  1753 A-B, one or more alias components  1755 A-D, one or more informational field components, one or more service association components  1759 A-B, and one or more other components for other information. A service definition  1760  can include one or more entity filter criteria components  1763 A-B that can be used to associate one or more entity definitions  1751 A-B with the service definition. 
     A service definition can include a single service name component that contains all of the identifying information (e.g., name, title, key, and/or identifier) for the service. The value for the name component type in a service definition can be used as the service identifier for the service being represented by the service definition. For example, the service definition  1760  includes a single entity name  1761  component that has an element name of “name” and an element value of “TestService”. The value “TestService” becomes the service identifier for the service that is being represented by service definition  1760 . 
     There can be one or multiple components having the same service definition component type. For example, the service definition  1760  has two entity filter criteria component types (e.g., entity filter criteria components  1763 A-B). In one implementation, some combination of a single and multiple components of the same type are used to store information pertaining to a service in a service definition. 
     Each entity filter criteria component  1763 A-B can store a single filter criterion or multiple filter criteria for identifying one or more of the entity definitions (e.g., entity definitions  1751 A-B). For example, the entity filter criteria component  1763 A stores a single filter criterion that includes an element name “dest” and a single element value “192.*” A value can include one or more wildcard characters as described in greater detail below. The entity filter criterion in component  1763 A can be applied to the entity definitions  1753 A-B to identify the entity definitions that satisfy the filter criterion “dest=192.*” Specifically, the element name-element value pair can be used for a search query. For example, a search query can search for fields named “dest” and containing a value that begins with the pattern “192.”. 
     An entity filter criteria component that stores multiple filter criteria can include an element name and multiple values. In one implementation, the multiple values are treated disjunctively. For example, the entity filter criteria  1763 B include an element name “name” and multiple values “192.168.1.100” and “hope.mbp14.local”. The entity filter criteria in component  1763 B can be applied to the entity definition records  1753 A-B to identify the entity definitions that satisfy the filter criteria “name=192.168.1.100” or “name=hope.mbp14.local”. Specifically, the element name and element values can be used for a search query that uses the values disjunctively. For example, a search query can search for fields in the service monitoring data store named “name” and having either a “192.168.1.100” or a “hope.mbp14.local” value. 
     An element name in the filter criteria in an entity filter criteria component  1763 A-B can correspond to an element name in an entity name component (e.g., entity name component  1753 A-B), an element name in an alias component (e.g., alias component  1755 A-D), or an element name in an informational field component (not shown) in at least one entity definition  1753 A-B in a service monitoring data store. The filter criteria can be applied to the entity definitions in the service monitoring data store based on the execution type and execution parameter in the entity filter criteria component  1763 A-B. 
     In one implementation, an entity association indicator component  1765 A-B is added to the service definition  1760  for each entity definition that satisfies any of the filter criteria in the entity filter criteria component  1763 A-B for the service. The entity association indicator component  1765 A-B can include an element name-element value pair to associate the particular entity definition with the service definition. For example, the entity definition record  1751 A satisfies the rule “dest=192.*” and the entity association indicator component  1765 A is added to the service definition record  1760  to associate the entity definition record  1751 A with the TestService specified in the service definition record  1760 . 
     In one implementation, for each entity definition that has been identified as satisfying any of the filter criteria in the entity filter criteria components  1763 A-B for a service, a service association component  1758 A-B is added to the entity definition  1751 A-B. The service association component  1758 A-B can include an element name-element value pair to associate the particular service definition  1760  with the entity definition  1751 A. For example, the entity definition  1751 A satisfies the filter criterion “dest=192.*” associated with the service definition  1760 , and the service association component  1758 A is added to the entity definition  1751 A to associate the TestService with the entity definition  1753 A. 
     In one implementation, the entity definitions  1751 A-B that satisfy any of the filter criteria in the service definition  1760  are associated with the service definition automatically. For example, an entity association indicator component  1765 A-B can be automatically added to the service definition  1760 . In one example, an entity association indicator component  1765 A-B can be added to the service definition  1760  when the respective entity definition has been identified. 
     As described above, the entity definitions  1751 A-B can include alias components  1755 A-D for associating machine data (e.g., machine data 1-4) with a particular entity being represented by a respective entity definition  1751 A-B. For example, entity definition  1753 A includes alias component  1755 A-B to associate machine data 1 and machine data 2 with the entity named “foobar”. When any of the entity definition components of an entity definition satisfy filter criteria in a service definition  1760 , all of the machine data that is associated with the entity named “foobar” can be used for the service being represented by the service definition  1760 . For example, the alias component  1755 A in the entity definition  1751 A satisfies the filter criteria in entity filter criteria  1763 A. If a KPI is being determined for the service “TestService” that is represented by service definition  1760 , the KPI can be determined using machine data 1 and machine data 2 that are associated with the entity represented by the entity definition  1751 A, even though only machine data 1 (and not machine data 2) is associated with the entity represented by definition record  1751 A via alias  1755 A (the alias used to associate entity definition record  1751 A with the service represented by definition record  1760  via filter criteria  1763 A). 
     When filter criteria in the entity filter criteria components  1763 A-B are applied to the entity definitions dynamically, changes that are made to the entity definitions  1753 A-B in the service monitoring data store can be automatically captured by the entity filter criteria components  1763 A-B and reflected, for example, in KPI determinations for the service, even after the filter criteria have been defined. The entity definitions that satisfy filter criteria for a service can be associated with the respective service definition even if a new entity is created significantly after a rule has already been defined. 
     For example, a new machine can be added to an IT environment and a new entity definition for the new machine can be added to the service monitoring data store. The new machine has an IP address containing “192.” and can be associated with machine data X and machine data Y. The filter criteria in the entity filter criteria component  1763  can be applied to the service monitoring data store and the new machine can be identified as satisfying the filter criteria. The association of the new machine with the service definition  1760  for TestService is made without user interaction. An entity association indicator for the new machine can be added to the service definition  1760  and/or a service association can be added to the entity definition of the new machine. A KPI for the TestService can be calculated that also takes into account machine data X and machine data Y for the new machine. 
     As described above, in one implementation, a service definition  1760  stores no more than one component having a name component type. The service definition  1760  can store zero or more components having an entity filter criteria component type, and can store zero or more components having an informational field component type. In one implementation, user input is received via a GUI (e.g., service definition GUI) to add one or more other service definition components to a service definition record. 
     Various implementations can use a variety of data representation and/or organization for the component information in a service definition record based on such factors as performance, data density, site conventions, and available application infrastructure, for example. The structure (e.g., structure  1720  in  FIG.  17 B ) of a service definition can include rows, entries, or tuples to depict components of an entity definition. A service definition component can be a normalized, tabular representation for the component, as can be used in an implementation, such as an implementation storing the entity definition within an RDBMS. Different implementations can use different representations for component information; for example, representations that are not normalized and/or not tabular. Different implementations can use various data storage and retrieval frameworks, a JSON-based database as one example, to facilitate storing entity definitions (entity definition records). Further, within an implementation, some information can be implied by, for example, the position within a defined data structure or schema where a value, such as “192.*” in  FIG.  17 C , is stored—rather than being stored explicitly. For example, in an implementation having a defined data structure for a service definition where the first data item is defined to be the value of the name element for the name component of the service, only the value need be explicitly stored as the service component and the element name (name) are known from the data structure definition. 
       FIG.  17 D  is a flow diagram of an implementation of a method  1740  for using filter criteria to associate entity definition(s) with a service definition, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, at least a portion of method is performed by a client computing machine. In another implementation, at least a portion of method is performed by a server computing machine. 
     At block  1741 , the computing machine causes display of a graphical user interface (GUI) that enables a user to specify filter criteria for identifying one or more entity definitions. An example GUI that enables a user to specify filter criteria is described in greater detail below. 
     At block  1743 , the computing machine receives user input specifying one or more filter criteria corresponding to a rule. A rule with a single filter criterion can include an element name—element value pair where there is a single value. For example, the single filter criterion can be “name=192.168.1.100”. A rule with multiple filter criteria can include an element name and multiple values. The multiple values can be treated disjunctively. For example, the multiple criteria can be “name=192.168.1.100 or hope.mbp14.local”. In one example, an element name in the filter criteria corresponds to an element name of an alias component in at least one entity definition in a data store. In another example, an element name in the filter criteria corresponds to an element name of an informational field component in at least one entity definition in the data store. 
     At block  1744 , the computing machine receives user input specifying an execution type and execution parameter for each rule. The execution type specifies how the filter criteria should be applied to the entity definitions. The execution type can be static execution or dynamic execution. The execution parameter specifies when the filter criteria should be applied to the entity definitions. User input can be received designating the execution type and execution parameter for a particular rule via a GUI, as described below. 
     Referring to  FIG.  17 D , at block  1745 , the computing machine stores the filter criteria in association with a service definition. The filter criteria can be stored in one or more entity filter criteria components. In one implementation, the entity filter criteria components (e.g., entity filter criteria components  1723 B in  FIG.  17 B ) are stored in association with a service definition. In another implementation, the entity filter criteria components (e.g., entity filter criteria components  1723 A in  FIG.  17 B ) are stored within a service definition. 
     At block  1746 , the computing machine stores the execution type for each rule in association with the service definition. As described above, the execution type for each rule can be stored in a respective entity filter criteria component. 
     At block  1747 , the computing machine applies the filter criteria to identify one or more entity definitions satisfying the filter criteria. The filter criteria can be applied to the entity definitions in the service monitoring data store based on the execution type and the execution parameter that has been specified for a rule to which the filter criteria pertains. For example, if the execution type is static execution, the computing machine can apply the filter criteria a single time. For a static execution type, the computing machine can apply the filter criteria a single time when user input, which accepts the filter criteria that are specified via the GUI, is received. In another example, the computing machine can apply the filter criteria a single time the first KPI is being calculated for the service. 
     If the execution type is dynamic execution, the computing machine can apply the filter criteria multiple times. For example, for a dynamic execution type, the computing machine can apply the filter criteria each time a change to the entity definitions in the service monitoring data store is detected. The computing machine can monitor the entity definitions in the service monitoring data store to detect any change that is made to the entity definitions. The change can include, for example, adding a new entity definition to the service monitoring data store, editing an existing entity definition, deleting an entity definition, etc. In another example, the computing machine can apply the filter criteria each time a KPI is calculated for the service. 
     At block  1749 , the computing machine associates the identified entity definitions with the service definition. The computing machine stores an association indicator in a stored service definition or a stored entity definition. 
     A static filter criterion can be executed once (or on demand). Static execution of the filter criteria for a particular rule can produce one or more entity associations with the service definition. For example, a rule can have the static filter criterion “name=192.168.1.100”. The filter criterion “name=192.168.1.100” can be applied to the entity definitions in the service monitoring data store once, and a search query is performed to identify the entity definition records that satisfy “name=192.168.1.100”. The result can be a single entity definition, and the single entity definition is associated with the service definition. The association will not the static filter criterion “name=192.168.1.100” is applied another time (e.g., on demand). 
     Dynamic filter criterion can be run multiple times automatically, i.e., manual vs. automatic. Dynamic execution of the filter criteria for a particular rule can produce a dynamic entity association with the service definition. The filter criteria for the rule can be executed at multiple times, and the entity associations can be different from execution to execution. For example, a rule can have the dynamic filter criterion “name=192.*”. When the filter criterion “name=192.*” is applied to the entity definitions in the service monitoring data store at time X, a search query is performed to identify the entity definitions that satisfy “name=192.*”. The result can be one hundred entity definitions, and the one hundred entity definitions are associated with the service definition. One week later, a new data center can be added to the IT environment, and the filter criterion “name=192.*” can be again applied to the entity definitions in the service monitoring data store at time Y. A search query is performed to identify the entity definitions that satisfy “name=192.*”. The result can be four hundred entity definitions, and the four hundred entity definitions are associated with the service definition. The filter criterion “name=192.168.1.100” can be applied multiple times and the entity definitions that satisfy the filter criterion can differ from time to time. 
       FIG.  17 E  illustrates an example of a GUI  1770  of a service monitoring system for using filter criteria to identify one or more entity definitions to associate with a service definition, in accordance with one or more implementations of the present disclosure. In one implementation, GUI  1770  is displayed when button  1306  in  FIG.  13    is activated. 
     GUI  1770  can include a service definition status bar  1771  that displays the various stages for creating a service definition using the GUIs of the service monitoring system. The stages can include, for example, and are not limited to, a service information stage, a key performance indicator (KPI) stage, and a service dependencies stage. The status bar  1771  can be updated to display an indicator (e.g., shaded circle) corresponding to a current stage. 
     GUI  1770  can include a save button  1789  and a save-and-next button  1773 . For each stage, if the save button  1789  is activated, the settings that have been specified via the GUI  1770  for a particular stage (e.g., service information stage) can be stored in a data store, without having to progress to a next stage. For example, if user input for the service name, description, and entity filter criteria has been received, and the save button  1789  is selected, the specified service name, description, and entity filter criteria can be stored in a service definition record (e.g., service definition record  1760  in  FIG.  17 C ) and stored in the service monitoring data store, without navigating to a subsequent GUI to specify any KPI or dependencies for the service. If the save and next button  1773  is activated, the settings that have been specified via the GUI  1770  for a particular stage can be stored in a data store, and a GUI for the next stage can be displayed. In one implementation, user interaction with the save button  1789  or the save-and-next button  1773  produces the same save operation that stores service definition information in the service monitoring data store. Unlike the save button  1789 , save-and-next button  1773  has a further operation of navigating to a subsequent GUI. GUI  1770  includes a previous button  1772 , which when selected, displays the previous GUI for creating the service definition. 
     GUI  1770  can facilitate user input specifying a name  1775  and optionally a description  1777  for the service definition for a service. For example, user input of the name “TestService” and the description “Service that contains entities” is received. 
     GUI  1770  can include one or more buttons (e.g., “Yes” button  1779 ,“No” button  1781 ) that can be selected to specify whether entities are associated with the service. A selection of the “No” button  1781  indicates that the service being defined will not be associated with any entities, and the resulting service definition has no associations with any entity definitions. For example, a service can not be associated with any entities if an end user intends to use the service and corresponding service definition for testing purposes and/or experimental purposes. In another example, a service can not be associated with any entities if the service is dependent on one or more other services, and the service is being monitored via the entities of the one or more other services upon which the service depends upon. For example, an end user can wish to use a service without entities as a way to track a business service based on the services which the business service depends upon. 
     If the “Yes” button  1779  is selected, an entity portion  1783  enabling a user to specify filter criteria for identifying one or more entity definitions to associate with the service definition is displayed. The filter criteria can correspond to a rule. The entity portion  1783  can include a button  1785 , which when selected, displays a button and text box to receive user input specifying an element name and one or more corresponding element values for filter criteria corresponding to a rule, as described below. 
     Referring to  FIG.  17 E , the entity portion  1783  can include preview information  1787  that displays information pertaining to any entity definitions in the service monitoring data store that satisfy the particular filter criteria for the rule. The preview information  1787  can be updated as the filter criteria are being specified, as described in greater detail below. GUI  1770  can include a link  1791 , which when activated, can display a GUI that presents a list of the matching entity definitions, as described in greater detail below. 
       FIG.  17 F  illustrates an example of a GUI  17100  of a service monitoring system for specifying filter criteria for a rule, in accordance with one or more implementations of the present disclosure. GUI  17100  can display a button  17107  for selecting an element name for filter criteria of a rule, and a text box  17109  for specifying one or more values that correspond to the selected element name. If button  17107  is activated, a list  17105  of element names can be displayed, and a user can select an element name for the filter criteria from the list  17105 . 
     In one implementation, the list  17105  is populated using the element names that are in the alias components that are in the entity definition records that are stored in the service monitoring data store. In one implementation, the list  17105  is populated using the element names from the informational field components in the entity definitions. In one implementation, the list  17105  is populated using field names that are specified by a late-binding schema that is applied to events. In one implementation, the list  17105  is populated using any combination of alias component element names, informational field component element names, and/or field names. 
     User input can be received that specifies one or more values for the specified element name. For example, a user can provide a string for specifying one or more values via text box  17109 . In another example, a user can select text box  17109 , and a list of values that correspond to the specified element name can be displayed as described below. 
       FIG.  17 G  illustrates an example of a GUI  17200  of a service monitoring system for specifying one or more values for filter criteria of a rule, in accordance with one or more implementations of the present disclosure. In this example, filter criteria for rule  17203  is being specified via GUI  17200 . GUI  17200  displays a selection of an element name “name”  17201  for the filter criteria of rule  17203 . When text box  17205  is activated (e.g., when a user selects text box  17205  by, for example, clicking or tapping on text box  17205 , or moving the cursor to text box  17205 ), a list  17207  of values that correspond to the element name “name”  17201  is displayed. For example, various entity definitions can include a name component having the element name “name”, and the list  17207  can be populated with the values from the name components from those various entity definition records. 
     One or more values from the list  17207  can be specified for the filter criteria of a rule. For example, the filter criteria for rule  17203  can include the value “192.168.1.100”  17209  and the value “hope.mbp14.local”  17211 . In one implementation, when multiple values are part of the filter criteria for a rule, the rule treats the values disjunctively. For example, when the rule  17203  is to be executed, the rule triggers a search query to be performed to search for entity definition records that have either an element name “name” and a corresponding “192.168.1.100” value, or have an element name “name” and a corresponding “hope.mbp14.local” value. 
     A service definition can include multiple sets of filter criteria corresponding to different rules. In one implementation, the different rules are treated disjunctively, as described below. 
       FIG.  17 H  illustrates an example of a GUI  17300  of a service monitoring system for specifying multiple sets of filter criteria for associating one or more entity definitions with a service definition, in accordance with one or more implementations of the present disclosure. As described above, a service definition can include multiple sets of filter criteria corresponding to different rules. For example, two sets of filter criteria for two rules  17303  and  17305  can be specified via GUI  17300 . 
     Rule  17303  has multiple filter criteria that include an element name “name”  17301  and multiple element values (e.g., the value “192.168.100”  17309  and the value “hope.mbp14.local”  17391 ). In one implementation, the multiple filter criteria are processed disjunctively. For example, rule  17303  can be processed to search for entity definitions that satisfy “name=192.168.1.100” or “name=hope.mbp14.local”. Rule  17305  has a single filter criterion that includes element name “dest”  17307  and a single element value “192.*”  17313  for a single filter criterion of “dest=192.*”. 
     In one example, an element value for filter criteria of a rule can be expressed as an exact string (e.g., “192.168.1.100” and “hope.mbp14.local”) and the rule can be executed to perform a search query for an exact string match. In another example, an element value for filter criteria of a rule can be expressed as a combination of characters and one or more wildcard characters. For example, the value “192.*” for rule  17305  contains an asterisk as a wildcard character. A wildcard character in a value can denote that when the rule is executed, a wildcard search query is to be performed to identify entity definitions using pattern matching. In another example, an element value for a filter criteria rule can be expressed as a regular expression (regex) as another possible option to identify entity definitions using pattern matching. 
     In one implementation, when multiple sets of filter criteria for different rules are specified for a service definition, the multiple rules are processed disjunctively. The entity definitions that satisfy any of the rules are the entity definitions that are to be associated with the service definition. For example, any entity definitions that satisfy “name=192.168.1.100 or hope.mbp14.local” or “dest=192.*” are the entity definitions that are to be associated with the service definition. 
     GUI  17300  can display, for each rule being specified, a button  17327 A-B for selecting the execution parameter for the particular rule. GUI  17300  can display, for each rule being specified, a button  17325 A-B for selecting the execution type (e.g., static execution type, dynamic execution type) for the particular rule. For example, rule  17303  has a static execution type, and rule  17305  has a dynamic execution type. 
     A user can wish to select a static execution type for a rule, for example, if the user anticipates that one or more entity definitions can not satisfy a rule that has a wildcard-based filter criterion. For example, a service can already have the rule with filter criterion “dest=192.*”, but the user can wish to also associate a particular entity, which does not have “192” in its address, with the service. A static rule that searches for the particular entity by entity name, such as rule with filter criterion “name=hope.mbp14.local” can be added to the service definition. 
     In another example, a user can wish to select a static execution type for a rule, for example, if the user anticipates that only certain entities will ever be associated with the service. The user can not want any changes to be made inadvertently to the entities that are associated with the service by the dynamic execution of a rule. 
     GUI  17300  can display preview information for the entity definitions that satisfy the filter criteria for the rule(s). The preview information can include a number of the entity definitions that satisfy the filter criteria and/or the execution type of the rule that pertains to the particular entity definition. For example, preview information  17319  includes the type “static” and the number “2”. In one implementation, when the execution type is not displayed, the preview information represents a dynamic execution type. For example, preview information  17315  and preview information  17318  pertain to rules that have a dynamic execution type. 
     The preview information can represent execution of a particular rule. For example, preview information  17315  is for rule  17305 . A combination of the preview information can represent execution of all of the rules for the service. For example, the combination of preview information  17318  and preview information  17319  is a summary of the execution of rule  17303  and rule  17305 . 
     GUI  17300  can include one or more buttons  17317 ,  17321 , which when selected, can re-apply the corresponding rule(s) to update the corresponding preview information. For example, the filter criteria for rule  17305  can be edited to “dest=192.168.*” and button  17317  can be selected to apply the edited filter criteria for rule  17305  to the entity definitions in the service monitoring data store. The corresponding preview information  17315  and the preview information  17318  in the summary can or can not change depending on the search results. 
     In one implementation, the preview information includes a link, which when selected, can display a list of the entity definitions that are being represented by the preview information. For example, preview information  17315  for rule  17307  indicates that there are 4 entity definitions that satisfy the rule “dest=192.*”. The preview information  17315  can include a link, which when activated can display a list of the  4  entity definition, as described in greater detail below. Referring to  FIG.  17 H , GUI  17300  can include a link  17323 , which when selected can display a list of all of the entity definitions that satisfy all of the rules (having both static and dynamic execution types such as rule  17303  and rule  17305 ) for the service definition. 
       FIG.  17 I  illustrates an example of a GUI  17400  of a service monitoring system for displaying entity definitions that satisfy filter criteria, in accordance with one or more implementations of the present disclosure. GUI  17400  can display list  17401  of the entity definitions that satisfy a particular rule “dest=192.*” (e.g., rule  17305  in  FIG.  17 H ). The list  17401  can include, for each entity definition, the value (e.g., value 192.168.1.100  17403 A, value 192.168.0.1  17403 B, value 192.168.0.2  17403 B, and value 192.168.0.3  17403 B) that satisfies the filter criteria for the rule. 
       FIG.  18    illustrates an example of a GUI  1800  of a service monitoring system for specifying dependencies for the service, in accordance with one or more implementations of the present disclosure. GUI  1800  can include an availability list  1804  of services that each has a corresponding service definition. The availability list  1804  can include one or more services. For example, the availability list  1804  can include dozens of services. GUI  1800  can include a filter box  1802  to receive input for filtering the availability list  1804  of services to display a portion of the services. GUI  1800  can facilitate user input for selecting a service from the availability list  1804  and dragging the selected service to a dependent services list  1812  to indicate that the service is dependent on the services in the dependent services list  1812 . For example, the service definition can be for a Sandbox service. For example, the drop-down  1801  can be selected to display a title “Sandbox” in the service information for the service definition. The availability list  1804  can initially include four other services: (1) Revision Control service, (2) Networking service, (3) Web Hosting service, and (4) Database service. The Sandbox service can depend on the Revision Control service and the Networking service. A user can select the Revision Control service and Networking service from the availability list  1804  and drag the Revision Control service and Networking service to the dependent services list  1812  to indicate that the Sandbox service is dependent on the Revision Control service and Networking service. In one implementation, GUI  1800  further displays a list of other services which depend on the service described by the service definition that is being created and/or edited. 
       FIG.  19    is a flow diagram of an implementation of a method  1900  for creating one or more key performance indicators for a service, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method is performed by the client computing machine. In another implementation, the method is performed by a server computing machine coupled to the client computing machine over one or more networks. 
     At block  1902 , the computing machine receives input (e.g., user input) of a name for a KPI to monitor a service or an aspect of the service. For example, a user can wish to monitor the service&#39;s response time for requests, and the name of the KPI can be “Request Response Time.” In another example, a user can wish to monitor the load of CPU(s) for the service, and the name of the KPI can be “CPU Usage.” 
     At block  1904 , the computing machine creates a search query to produce a value indicative of how the service or the aspect of the service is performing. For example, the value can indicate how the aspect (e.g., CPU usage, memory usage, request response time) is performing at point in time or during a period of time. Some implementations for creating a search query are discussed in greater detail below. In one implementation, the computing machine receives input (e.g., user input), via a graphical interface, of search processing language defining the search query. Some implementations for creating a search query from input of search processing language are discussed in greater detail below. In one implementation, the computing machine receives input (e.g., user input) for defining the search query using a data model. Some implementations for creating a search query using a data model are discussed in greater detail below. 
     At block  1906 , the computing machine sets one or more thresholds for the KPI. Each threshold defines an end of a range of values. Each range of values represents a state for the KPI. The KPI can be in one of the states (e.g., normal state, warning state, critical state) depending on which range the value falls into. Some implementations for setting one or more thresholds for the KPI are discussed in greater detail below. 
       FIG.  20    is a flow diagram of an implementation of a method  2000  for creating a search query, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method is performed by the client computing machine. In another implementation, the method is performed by a server computing machine coupled to the client computing machine over one or more networks. 
     At block  2002 , the computing machine receives input (e.g., user input) specifying a field to use to derive a value indicative of the performance of a service or an aspect of the service to be monitored. As described above, machine data can be represented as events. Each of the events is raw data. A late-binding schema can be applied to each of the events to extract values for fields defined by the schema. The received input can include the name of the field from which to extract a value when executing the search query. For example, the received user input can be the field name “spent” that can be used to produce a value indicating the time spent to respond to a request. 
     At block  2004 , the computing machine optionally receives input specifying a statistical function to calculate a statistic using the value in the field. In one implementation, a statistic is calculated using the value(s) from the field, and the calculated statistic is indicative of how the service or the aspect of the service is performing. As discussed above, the machine data used by a search query for a KPI to produce a value can be based on a time range. For example, the time range can be defined as “Last 15 minutes,” which would represent an aggregation period for producing the value. In other works, if the query is executed periodically (e.g., every 5 minutes), the value resulting from each execution can be based on the last 15 minutes on a rolling basis, and the value resulting from each execution can be based on the statistical function. Examples of statistical functions include, and are not limited to, average, count, count of distinct values, maximum, mean, minimum, sum, etc. For example, the value can be from the field “spent” the time range can be “Last 15 minutes,” and the input can specify a statistical function of average to define the search query that should produce the average of the values of field “spent” for the corresponding 15 minute time range as a statistic. In another example, the value can be a count of events satisfying the search criteria that include a constraint for the field (e.g., if the field is “response time,” and the KPI is focused on measuring the number of slow responses (e.g., “response time” below x) issued by the service). 
     At block  2006 , the computing machine defines the search query based on the specified field and the statistical function. The computing machine can also optionally receive input of an alias to use for a result of the search query. The alias can be used to have the result of the search query to be compared to one or more thresholds assigned to the KPI. 
       FIG.  21    illustrates an example of a GUI  2100  of a service monitoring system for creating a KPI for a service, in accordance with one or more implementations of the present disclosure. GUI  2100  can display a list  2104  of KPIs that have already been created for the service and associated with the service via the service definition. For example, the service definition “Web Hosting” includes a KPI “Storage Capacity” and a KPI “Memory Usage”. GUI  2100  can include a button  2106  for editing a KPI. A KPI in the list  2104  can be selected and the button  2106  can be activated to edit the selected KPI. GUI  2100  can include a button  2102  for creating a new KPI. If button  2102  is activated, GUI  2200  in  FIG.  22    is displayed facilitating user input for creating a KPI. 
       FIG.  22    illustrates an example of a GUI  2200  of a service monitoring system for creating a KPI for a service, in accordance with one or more implementations of the present disclosure. GUI  2200  can facilitate user input specifying a name  2202  and optionally a description  2204  for a KPI for a service. The name  2202  can indicate an aspect of the service that is to be monitored using the KPI. As described above, the KPI is defined by a search query that produces a value derived from machine data pertaining to one or more entities identified in a service definition for the service. The produced value is indicative of how an aspect of the service is performing. In one example, the produced value is the value extracted from a field when the search query is executed. In another example, the produced value is a result from calculating a statistic based on the value in the field. 
     In one implementation, the search query is defined from input (e.g., user input), received via a graphical interface, of search processing language defining the search query. GUI  2200  can include a button  2206  for facilitating user input of search processing language defining the search query. If button  2206  is selected, a GUI for facilitating user input of search processing language defining the search query can be displayed, as discussed in greater detail below. 
     Referring to  FIG.  22   , in another implementation, the search query is defined using a data model. GUI  2200  can include a button  2208  for facilitating user input of a data model for defining the search query. If button  2208  is selected, a GUI for facilitating user input for defining the search query using a data model can be displayed, as discussed in greater detail below. 
       FIG.  23    illustrates an example of a GUI  2300  of a service monitoring system for receiving input of search processing language for defining a search query for a KPI for a service, in accordance with one or more implementations of the present disclosure. GUI  2300  can facilitate user input specifying a KPI name  2301 , which can optionally indicate an aspect of the service to monitor with the KPI, and optionally a description  2302  for a KPI for a service. For example, the aspect of the service to monitor can be response time for received requests, and the KPI name  2301  can be Request Response Time. GUI  2300  can facilitate user input specifying search processing language  2303  that defines the search query for the Request Response Time KPI. The input for the search processing language  2303  can specify a name of a field (e.g., spent  2313 ) to use to extract a value indicative of the performance of an aspect (e.g., response time) to be monitored for a service. The input of the field (e.g., spent  2313 ) designates which data to extract from an event when the search query is executed. 
     The input can optionally specify a statistical function (e.g., avg  2311 ) that should be used to calculate a statistic based on the value corresponding to a late-binding schema being applied to an event. The late-binding schema will extract a portion of event data corresponding to the field (e.g., spent  2313 ). For example, the value associated with the field “spent” can be extracted from an event by applying a late-binding schema to the event. The input can specify that the average of the values corresponding to the field “spent” should be produced by the search query. The input can optionally specify an alias (e.g., rsp_time  2315 ) to use (e.g., as a virtual field name) for a result of the search query (e.g., avg(spent)  2314 ). The alias  2315  can be used to have the result of the search query to be compared with one or more thresholds assigned to the KPI. 
     GUI  2300  can display a link  2304  to facilitate user input to request that the search criteria be tested by running the search query for the KPI. In one implementation, when input is received requesting to test the search criteria for the search query, a search GUI is displayed. 
     In some implementations, GUI  2300  can facilitate user input for creating one or more thresholds for the KPI. The KPI can be in one of multiple states (e.g., normal, warning, critical). Each state can be represented by a range of values. During a certain time, the KPI can be in one of the states depending on which range the value, which is produced at that time by the search query for the KPI, falls into. GUI  2300  can include a button  2307  for creating the threshold for the KPI. Each threshold for a KPI defines an end of a range of values, which represents one of the states. Some implementations for creating one or more thresholds for the KPI are discussed in greater detail below. 
     GUI  2300  can include a button  2309  for editing which entity definitions to use for the KPI. Some implementations for editing which entity definitions to use for the KPI are discussed in greater detail below. 
     In some implementations, GUI  2300  can include a button  2320  to receive input assigning a weight to the KPI to indicate an importance of the KPI for the service relative to other KPIs defined for the service. The weight can be used for calculating an aggregate KPI score for the service to indicate an overall performance for the service, as discussed in greater detail below. GUI  2300  can include a button  2323  to receive input to define how often the KPI should be measured (e.g., how often the search query defining the KPI should be executed) for calculating an aggregate KPI score for the service to indicate an overall performance for the service, as discussed in greater detail below. The importance (e.g., weight) of the KPI and the frequency of monitoring (e.g., a schedule for executing the search query) of the KPI can be used to determine an aggregate KPI score for the service. The score can be a value of an aggregate of the KPIs of the service. Some implementations for using the importance and frequency of monitoring for each KPI to determine an aggregate KPI score for the service are discussed in greater detail below. 
     GUI  2300  can display an input box  2305  for a field to which the threshold(s) can be applied. In particular, a threshold can be applied to the value produced by the search query defining the KPI. Applying a threshold to the value produced by the search query is described in greater detail below. 
       FIG.  24    illustrates an example of a GUI  2400  of a service monitoring system for defining a search query for a KPI using a data model, in accordance with one or more implementations of the present disclosure. GUI  2400  can facilitate user input specifying a name  2403  and optionally a description  2404  for a KPI for a service. For example, the aspect of the service to monitor can be CPU utilization, and the KPI name  2403  can be CPU Usage. If button  2402  is selected, GUI  2400  displays button  2406  and button  2408  for defining the search query for the KPI using a data model. A data model refers to one or more objects grouped in a hierarchical manner and can include a root object and, optionally, one or more child objects that can be linked to the root object. A root object can be defined by search criteria for a query to produce a certain set of events, and a set of fields that can be exposed to operate on those events. Each child object can inherit the search criteria of its parent object and can have additional search criteria to further filter out events represented by its parent object. Each child object can also include at least some of the fields of its parent object and optionally additional fields specific to the child object, as will be discussed in greater detail below. 
     If button  2402  is selected, GUI  2500  in  FIG.  25    is displayed for facilitating user input for selecting a data model to assist with defining the search query.  FIG.  25    illustrates an example of a GUI  2500  of a service monitoring system for facilitating user input for selecting a data model and an object of the data model to use for defining the search query, in accordance with one or more implementations of the present disclosure. GUI  2500  can include a drop-down menu  2503 , which when expanded, displays a list of available data models. When a data model is selected, GUI  2500  can display a list  2505  of objects pertaining to the selected data model. For example, the data model Performance is selected and the objects pertaining to the Performance data model are included in the list  2505 . Objects of a data model are described in greater detail below. When an object in the list  2505  is selected, GUI  2500  can display a list  2511  of fields pertaining to the selected object. For example, the CPU object  2509  is selected and the fields pertaining to the CPU object  2509  are included in the list  2511 . GUI  2500  can facilitate user input of a selection of a field in the list  2511 . The selected field (e.g., cpu_load_percent  2513 ) is the field to use for the search query to derive a value indicative of the performance of an aspect (e.g., CPU usage) of the service. The derived value can be, for example, the field&#39;s value extracted from an event when the search query is executed, a statistic calculated based on one or more values of the field in one or more events located when the search query is executed, a count of events satisfying the search criteria that include a constraint for the field (e.g., if the field is “response time” and the KPI is focused on measuring the number of slow responses (e.g., “response time” below x) issued by the service). 
     Referring to  FIG.  24   , GUI  2400  can display a button  2408  for optionally selecting a statistical function to calculate a statistic using the value(s) from the field (e.g., cpu_load_percent  2513 ). If a statistic is calculated, the result from calculating the statistic becomes the produced value from the search query, which indicates how an aspect of the service is performing. When button  2408  is selected, GUI  2400  can display a drop-down list of statistics. The list of statistics can include, and are not limited to, average, count, count of distinct values, maximum, mean, minimum, sum, etc. For example, a user can select “average” and the value produced by the search query can be the average of the values of field cpu_load_percent  2513  for a specified time range (e.g., “Last 15 minutes”).  FIG.  26    illustrates an example of a GUI  2600  of a service monitoring system for displaying a selected statistic  2601  (e.g., average), in accordance with one or more implementations of the present disclosure. 
     Referring to  FIG.  24   , GUI  2400  can facilitate user input for creating one or more thresholds for the KPI. GUI  2400  can include a button  2410  for creating the threshold(s) for the KPI. Some implementations for creating one or more thresholds for the KPI are discussed in greater detail below. 
     GUI  2400  can include a button  2412  for editing which entity definitions to use for the KPI. Some implementations for editing which entity definitions to use for the KPI are discussed in greater detail below. 
     GUI  2400  can include a button  2418  for saving a definition of a KPI and an association of the defined KPI with a service. The KPI definition and association with a service can be stored in a data store. 
     The value for the KPI can be produced by executing the search query of the KPI. In one example, the search query defining the KPI can be executed upon receiving a request (e.g., user request). For example, a service-monitoring dashboard, which is described in greater detail below, can display a KPI widget providing a numerical or graphical representation of the value for the KPI. A user can request the service-monitoring dashboard to be displayed, and the computing machine can cause the search query for the KPI to execute in response to the request to produce the value for the KPI. The produced value can be displayed in the service-monitoring dashboard 
     In another example, the search query defining the KPI can be executed based on a schedule. For example, the search query for a KPI can be executed at one or more particular times (e.g., 6:00 am, 12:00 pm, 6:00 pm, etc.) and/or based on a period of time (e.g., every 5 minutes). In one example, the values produced by a search query for a KPI by executing the search query on a schedule are stored in a data store, and are used to calculate an aggregate KPI score for a service, as described in greater detail below. An aggregate KPI score for the service is indicative of an overall performance of the KPIs of the service. 
     Referring to  FIG.  24   , GUI  2400  can include a button  2416  to receive input specifying a frequency of monitoring (schedule) for determining the value produced by the search query of the KPI. The frequency of monitoring (e.g., schedule) of the KPI can be used to determine a resolution for an aggregate KPI score for the service. The aggregate KPI score for the service is indicative of an overall performance of the KPIs of the service. The accuracy of the aggregate KPI score for the service for a given point in time can be based on the frequency of monitoring of the KPI. For example, a higher frequency can provide higher resolution which can help produce a more accurate aggregate KPI score. 
     The machine data used by a search query defining a KPI to produce a value can be based on a time range. The time range can be a user-defined time range or a default time range. For example, in the service-monitoring dashboard example above, a user can select, via the service-monitoring dashboard, a time range to use (e.g., Last 15 minutes) to further specify, for example, based on time-stamps, which machine data should be used by a search query defining a KPI. In another example, the time range can be to use the machine data since the last time the value was produced by the search query. For example, if the KPI is assigned a frequency of monitoring of 5 minutes, then the search query can execute every 5 minutes, and for each execution use the machine data for the last 5 minutes relative to the execution time. In another implementation, the time range is a selected (e.g., user-selected) point in time and the definition of an individual KPI can specify the aggregation period for the respective KPI. By including the aggregation period for an individual KPI as part of the definition of the respective KPI, multiple KPIs can run on different aggregation periods, which can more accurately represent certain types of aggregations, such as, distinct counts and sums, improving the utility of defined thresholds. In this manner, the value of each KPI can be displayed at a given point in time. In one example, a user can also select “real time” as the point in time to produce the most up to date value for each KPI using its respective individually defined aggregation period. 
     GUI  2400  can include a button  2414  to receive input assigning a weight to the KPI to indicate an importance of the KPI for the service relative to other KPIs defined for the service. The importance (e.g., weight) of the KPI can be used to determine an aggregate KPI score for the service, which is indicative of an overall performance of the KPIs of the service. Some implementations for using the importance and frequency of monitoring for each KPI to determine an aggregate KPI score for the service are discussed in greater detail below.  FIG.  27    illustrates an example of a GUI  2700  of a service monitoring system for editing which entity definitions to use for a KPI, in accordance with one or more implementations of the present disclosure. GUI  2700  can be displayed in response to the user activation of button  2412  in GUI  2400  of  FIG.  24   . GUI  2700  can include a button  2710  for creating a new entity definition. If button  2710  is selected, GUI  1600  in  FIG.  16    can be displayed and an entity definition can be created as described above. 
     Referring to  FIG.  27   , GUI  2700  can display buttons  2701 ,  2703  for receiving a selection of whether to include all of the entity definitions, which are associated with the service via the service definition, for the KPI. If the Yes button  2701  is selected, the search query for the KPI can produce a value derived from the machine data pertaining to all of the entities represented by the entity definitions that are included in the service definition for the service. If the No button  2703  is selected, a member list  2704  is displayed. The member list  2704  includes the entity definitions that are included in the service definition for the service. GUI  2700  can include a filter box  2702  to receive input for filtering the member list  2704  of entity definitions to display a subset of the entity definitions. 
     GUI  2700  can facilitate user input for selecting one or more entity definitions from the member list  2704  and dragging the selected entity definition(s) to an exclusion list  2712  to indicate that the entities identified in each selected entity definition should not be considered for the current KPI. This exclusion means that the search criteria of the search query defining the KPI is changed to no longer search for machine data pertaining to the entities identified in the entity definitions from the exclusion list  2712 . For example, entity definition  2705  (e.g., webserver07.splunk.com) can be selected and dragged to the exclusion list  2712 . When the search query for the KPI produces a value, the value will be derived from machine data, which does not include machine data pertaining to webserver07.splunk.com. 
       FIG.  28    is a flow diagram of an implementation of a method  2800  for defining one or more thresholds for a KPI, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method is performed by the client computing machine. In another implementation, the method is performed by a server computing machine coupled to the client computing machine over one or more networks. 
     At block  2802 , the computing machine identifies a service definition for a service. In one implementation, the computing machine receives input (e.g., user input) selecting a service definition. The computing machine accesses the service definition for a service from memory. 
     At block  2804 , the computing machine identifies a KPI for the service. In one implementation, the computing machine receives input (e.g., user input) selecting a KPI of the service. The computing machine accesses data representing the KPI from memory. 
     At block  2806 , the computing machine causes display of one or more graphical interfaces enabling a user to set a threshold for the KPI. The KPI can be in one of multiple states. Example states can include, and are not limited to, unknown, trivial state, informational state, normal state, warning state, error state, and critical state. Each state can be represented by a range of values. At a certain time, the KPI can be in one of the states depending on which range the value, which is produced by the search query for the KPI, falls into. Each threshold defines an end of a range of values, which represents one of the states. Some examples of graphical interfaces for enabling a user to set a threshold for the KPI are discussed in greater detail below. 
     At block  2808 , the computing machine receives, through the graphical interfaces, an indication of how to set the threshold for the KPI. The computing machine can receive input (e.g., user input), via the graphical interfaces, specifying the field or alias that should be used for the threshold(s) for the KPI. The computing machine can also receive input (e.g., user input), via the graphical interfaces, of the parameters for each state. The parameters for each state can include, for example, and not limited to, a threshold that defines an end of a range of values for the state, a unique name, and one or more visual indicators to represent the state. 
     In one implementation, the computing machine receives input (e.g., user input), via the graphical interfaces, to set a threshold and to apply the threshold to the KPI as determined using the machine data from the aggregate of the entities associated with the KPI. 
     In another implementation, the computing machine receives input (e.g., user input), via the graphical interfaces, to set a threshold and to apply the threshold to a KPI as the KPI is determine using machine data on a per entity basis for the entities associated with the KPI. For example, the computing machine can receive a selection (e.g., user selection) to apply thresholds on a per entity basis, and the computing machine can apply the thresholds to the value of the KPI as the value is calculated per entity. 
     For example, the computing machine can receive input (e.g., user input), via the graphical interfaces, to set a threshold of being equal or greater than 80% for the KPI for Avg CPU Load, and the KPI is associated with three entities (e.g., Entity-1, Entity-2, and Entity-3). When the KPI is determined using data for Entity-1, the value for the KPI for Avg CPU Load can be at 50%. When the KPI is determined using data for Entity-2, the value for the KPI for Avg CPU Load can be at 50%. When the KPI is determined using data for Entity-3, the value for the KPI for Avg CPU Load can be at 80%. If the threshold is applied to the values of the aggregate of the entities (two at 50% and one at 80%), the aggregate value of the entities is 60%, and the KPI would not exceed the 80% threshold. If the threshold is applied using an entity basis for the thresholds (applied to the individual KPI values as calculated pertaining to each entity), the computing machine can determine that the KPI pertaining to one of the entities (e.g., Entity-3) satisfies the threshold by being equal to 80%. 
     At block  2810 , the computing machine determines whether to set another threshold for the KPI. The computing machine can receive input, via the graphical interface, indicating there is another threshold to set for the KPI. If there is another threshold to set for the KPI, the computing machine returns to block  2808  to set the other threshold. 
     If there is not another threshold to set for the KPI (block  2810 ), the computing machine determines whether to set a threshold for another KPI for the service at block  2812 . The computing machine can receive input, via the graphical interface, indicating there is a threshold to set for another KPI for the service. In one implementation, there are a maximum number of thresholds that can be set for a KPI. In one implementation, a same number of states are to be set for the KPIs of a service. In one implementation, a same number of states are to be set for the KPIs of all services. The service monitoring system can be coupled to a data store that stores configuration data that specifies whether there is a maximum number of thresholds for a KPI and the value for the maximum number, whether a same number of states is to be set for the KPIs of a service and the value for the number of states, and whether a same number of states is to be set for the KPIs of all of the service and the value for the number of states. If there is a threshold to set for another KPI, the computing machine returns to block  2804  to identity the other KPI. 
     At block  2814 , the computing machine stores the one or more threshold settings for the one or more KPIs for the service. The computing machine associates the parameters for a state defined by a corresponding threshold in a data store that is coupled to the computing machine. 
     As will be discussed in more detail below, implementations of the present disclosure provide a service-monitoring dashboard that includes KPI widgets (“widgets”) to visually represent KPIs of the service. A widget can be a Noel gauge, a spark line, a single value, or a trend indicator. A Noel gauge is indicator of measurement as described in greater detail below. A widget of a KPI can present one or more values indicating how a respective service or an aspect of a service is performing at one or more points in time. The widget can also illustrate (e.g., using visual indicators such as color, shading, shape, pattern, trend compared to a different time range, etc.) the KPI&#39;s current state defined by one or more thresholds of the KPI. 
       FIGS.  29 A-B  illustrate examples of a graphical interface enabling a user to set one or more thresholds for the KPI, in accordance with one or more implementations of the present disclosure. 
       FIG.  29 A  illustrates an example GUI  2900  for receiving input for search processing language  2902  for defining a search query, in accordance with one or more implementations of the present disclosure. The KPI can be in one of multiple states (e.g., normal, warning, critical). Each state can be represented by a range of values. At a certain time, the KPI can be in one of the states depending on which range the value, which is produced by the search query for the KPI, falls into. GUI  2900  can display an input box  2904  for a field to which the threshold(s) can be applied. In particular, a threshold can be applied to the value produced by the search query defining the KPI. The value can be, for example, the field&#39;s value extracted from an event when the search query is executed, a statistic calculated based on one or more values of the field in one or more events located when the search query is executed, a count of events satisfying the search criteria that include a constraint for the field, etc. GUI  2900  can include the name  2904  of the actual field used in the search query or the name of an alias that defines a desired statistic or count to be produced by the search query. For example, the threshold can be applied to an average response time produced by the search query, and the average response time can be defined by the alias “rsp_time” in the input box  2904 . 
       FIG.  29 B  illustrates an example GUI  2950  for receiving input for selecting a data model for defining a search query, in accordance with one or more implementations of the present disclosure. GUI  2950  can be displayed if a KPI is defined using a data model. 
     GUI  2950  in  FIG.  29 B  can include a statistical function  2954  to be used for producing a value when executing the search query of the KPI. As shown, the statistical function  2954  is a count, and the resulting statistic (the count value) should be compared with one or more thresholds of the KPI. The GUI  2950  also includes a button  2956  for creating the threshold(s) for the KPI. When either button  2906  is selected from GUI  2900  or button  2956  is selected from GUI  2950 , GUI  3000  of  FIG.  30    is displayed. 
       FIG.  29 C  illustrates an example GUI  2960  for configuring KPI monitoring in accordance with one or more implementations of the present disclosure. GUI  2960  can present information specifying a service definition corresponding to a service provided by a plurality of entities, and a specification for determining a KPI for the service. The service definition refers to a data structure, organization, or representation that can include information that associates one or more entities with a service. The service definition can include information for identifying the service definition, such as, for example, a name or other identifier for the service or service definition as can be indicated using GUI element  2961 . The specification for determining a KPI for the service refers to the KPI definitional information that can include source-related definitional information of a group of GUI elements  2963  and monitoring-related parameter information of a group of GUI elements  2965 . The source-related definitional information of a group of GUI elements  2963  can include, as illustrated by  FIG.  29 C , a search defining the KPI as presented in a GUI element  2902 , one or more entity identifiers for entities providing the service as presented in a GUI element  2906 , one or more threshold field names for fields derived from the entities&#39; machine data as presented in a GUI element  2904 . (The named fields derived from the entities&#39; machine data can be used to derive a value produced by the search of  2902 .) The monitoring-related parameter information of a group of GUI elements  2963  can include, as illustrated in  FIG.  29 C , an importance indicator presented by GUI element  2962 , a calculation frequency indicator presented by GUI element  2964 , and a calculation period indicator presented by GUI element  2966 . Once KPI definitional information ( 2963  and  2965 ) is adequately indicated using GUI  2960 , a specification for determining a KPI can be stored as part of the service definition (e.g., in the same database or file, for example), or in association with the service definition (e.g., in a separate database or file, for example, where the service definition, the KPI specification, or both, include information for associating the other). The adequacy of KPI definitional information can be determined in response to a specific user interaction with the GUI, by an automatic analysis of one or more user interactions with the GUI, or by some combination, for example. 
     The search of  2902  is represented by search processing language for defining a search query that produces a value derived from machine data pertaining to the entities that provide the service and which are identified in the service definition. The value can indicate a current state of the KPI (e.g., normal, warning, critical). An entity identifier of  2906  specifies one or more fields (e.g., dest, ip_address) that can be used to identify one or more entities whose machine data should be used in the search of  2902 . The threshold field GUI element  2904  enables specification of one or more fields from the entities&#39; machine data that should be used to derive a value produced by the search of  2902 . One or more thresholds can be applied to the value associated with the specified field(s) of  2904 . In particular, the value can be produced by a search query using the search of  2902  and can be, for example, the value of threshold field  2904  associated with an event satisfying search criteria of the search query when the search query is executed, a statistic calculated based on values for the specified threshold field of  2904  associated with the one or more events satisfying the search criteria of the search query when the search query is executed, or a count of events satisfying the search criteria of the search query that include a constraint for the threshold field of  2904 , etc. In the example illustrated in GUI  2960 , the designated threshold field of  2904  is “cpu_load_percent,” which can represent the percentage of the maximum processor load currently being utilized on a particular machine. In other examples, the threshold(s) can be applied a field specified in  2904  which can represent other metrics such as total memory usage, remaining storage capacity, server response time, or network traffic, for example. 
     In one implementation, the search query includes a machine data selection component and a determination component. The machine data selection component is used to arrive at a set of machine data from which to calculate a KPI. The determination component is used to derive a representative value for an aggregate of the set of machine data. In one implementation, the machine data selection component is applied once to the machine data to gather the totality of the machine data for the KPI, and returns the machine data sorted by entity, to allow for repeated application of the determination component to the machine data pertaining to each entity on an individual basis. In one implementation, portions of the machine data selection component and the determination component can be intermixed within search language of the search query (the search language depicted in  2902 , as an example of search language of a search query). 
     KPI monitoring parameters  2965  refer to parameters that indicate how to monitor the state of the KPI defined by the search of  2902 . In one embodiment, KPI monitoring parameters  2965  include the importance indicator of  2962 , the calculation frequency indicator of  2964 , and the calculation period indicator of element  2966 . 
     GUI element  2964  can include a drop-down menu with various interval options for the calculation frequency indicator. The interval options indicate how often the KPI search should run to calculate the KPI value. These options can include, for example, every minute, every 15 minutes, every hour, every 5 hours, every day, every week, etc. Each time the chosen interval is reached, the KPI is recalculated and the KPI value is populated into a summary index, allowing the system to maintain a record indicating the state of the KPI over time. 
     GUI element  2966  can include individual GUI elements for multiple calculation parameters, such as drop-down menus for various statistic options  2966   a , periods of time options  2966   b , and bucketing options  2966   c . The statistic options drop-down  2966   a  indicates a selected one (i.e., “Average”) of the available methods in the drop-down (not shown) that can be applied to the value(s) associated with the threshold field of  2904 . The expanded drop-down can display available methods such as average, maximum, minimum, median, etc. The periods of time options drop-down  2966   b  indicates a selected one (i.e., “Last Hour”) of the available options (not shown). The selected period of time option is used to identify events, by executing the search query, associated with a specific time range (i.e., the period of time) and each available option represents the period over which the KPI value is calculated, such as the last minute, last 15 minutes, last hour, last 4 hours, last day, last week, etc. Each time the KPI is recalculated (e.g., at the interval specified using  2964 ), the values are determined according to the statistic option specified using  2966   a , over the period of time specified using  2966   b . The bucketing options of drop-down  2966   c  each indicate a period of time from which the calculated values should be grouped together for purposes of determining the state of the KPI. The bucketing options can include by minute, by 15 minutes, by hour, by four hours, by day, by week, etc. For example, when looking at data over the last hour and when a bucketing option of 15 minutes is selected, the calculated values can be grouped every 15 minutes, and if the calculated values (e.g., the maximum or average) for the 15 minute bucket cross a threshold into a particular state, the state of the KPI for the whole hour can be set to that particular state. 
     Importance indicator of  2962  can include a drop-down menu with various weighting options. As discussed in more detail with respect to  FIGS.  32  and  33   , the weighting options indicate the importance of the associated KPI value to the overall health of the service. These weighting options can include, for example, values from 1 to 10, where the higher values indicate higher importance of the KPI relative to the other KPIs for the service. When determining the overall health of the service, the weighting values of each KPI can be used as a multiplier to normalize the KPIs, so that the values of KPIs having different weights can be combined together. In one implementation, a weighting option of 11 can be available as an overriding weight. The overriding weight is a weight that overrides the weights of all other KPIs of the service. For example, if the state of the KPI, which has the overriding weight, is “warning” but all other KPIs of the service have a “normal” state, then the service can only be considered in a warning state, and the normal state(s) for the other KPIs can be disregarded. 
       FIG.  30    illustrates an example GUI  3000  for enabling a user to set one or more thresholds for the KPI, in accordance with one or more implementations of the present disclosure. Each threshold for a KPI defines an end of a range of values, which represents one of the states. GUI  3000  can display a button  3002  for adding a threshold to the KPI. If button  3002  is selected, a GUI for facilitating user input for the parameters for the state associated with the threshold can be displayed, as discussed in greater detail below. 
     Referring to  FIG.  30   , if button  3002  is selected three times, there will be three thresholds for the KPI. Each threshold defines an end of a range of values, which represents one of the states. GUI  3000  can display a UI element (e.g., column  3006 ) that includes sections representing the defined states for the KPI, as described in greater detail below. GUI  3000  can facilitate user input to specify a maximum value  3004  and a minimum value  3008  for defining a scale for a widget that can be used to represent the KPI on the service-monitoring dashboard. Some implementations of widgets for representing KPIs are discussed in greater detail below. 
     Referring to  FIG.  30   , GUI  3000  can optionally include a button  3010  for receiving input indicating whether to apply the threshold(s) to the aggregate of the KPIs of the service or to the particular KPI. Some implementations for applying the threshold(s) to the aggregate of the KPIs of the service or to a particular KPI are discussed in greater detail below. 
       FIG.  31 A  illustrates an example GUI  3100  for defining threshold settings for a KPI, in accordance with one or more implementations of the present disclosure. GUI  3100  is a modified view of GUI  3000 , which is provided once the user has requested to add several thresholds for a KPI via button  3002  of GUI  3000 . In particular, in response to the user request to add a threshold, GUI  3100  dynamically adds a GUI element in a designated area of GUI  3100 . A GUI element can be in the form of an input box divided into several portions to receive various user input and visually illustrate the received input. The GUI element can represent a specific state of the KPI. When multiple states are defined for the KPI, several GUI elements can be presented in the GUI  3100 . For example, the GUI elements can be presented as input boxes of the same size and with the same input fields, and those input boxes can be positioned horizontally, parallel to each other, and resemble individual records from the same table. Alternatively, other types of GUI elements can be provided to represent the states of the KPI. 
     Each state of the KPI can have a name, and can be represented by a range of values, and a visual indicator. The range of values is defined by one or more thresholds that can provide the minimum end and/or the maximum end of the range of values for the state. The characteristics of the state (e.g., the name, the range of values, and a visual indicator) can be edited via input fields of the respective GUI element. 
     In the example shown in  FIG.  31 A , GUI  3100  includes three GUI elements representing three different states of the KPI based on three added thresholds. These states include states  3102 ,  3104 , and  3106 . 
     For each state, GUI  3100  can include a GUI element that displays a name (e.g., a unique name for that KPI)  3109 , a threshold  3110 , and a visual indicator  3112  (e.g., an icon having a distinct color for each state). The unique name  3109 , a threshold  3110 , and a visual indicator  3112  can be displayed based on user input received via the input fields of the respective GUI element. For example, the name “Normal” can be specified for state  3106 , the name “Warning” can be specified for state  3104 , the name “Critical” can be specified for state  3102 . 
     The visual indicator  3112  can be, for example, an icon having a distinct visual characteristic such as a color, a pattern, a shade, a shape, or any combination of color, pattern, shade and shape, as well as any other visual characteristics. For each state, the GUI element can display a drop-down menu  3114 , which when selected, displays a list of available visual characteristics. A user selection of a specific visual characteristic (e.g., a distinct color) can be received for each state. 
     For each state, input of a threshold value representing the minimum end of the range of values for the corresponding state of the KPI can be received via the threshold portion  3110  of the GUI element. The maximum end of the range of values for the corresponding state can be either a preset value or can be defined by (or based on) the threshold associated with the succeeding state of the KPI, where the threshold associated with the succeeding state is higher than the threshold associated with the state before it. 
     For example, for Normal state  3106 , the threshold value 0 can be received to represent the minimum end of the range of KPI values for that state. The maximum end of the range of KPI values for the Normal state  3106  can be defined based on the threshold associated with the succeeding state (e.g., Warning state  3104 ) of the KPI. For example, the threshold value 50 can be received for the Warning state  3104  of the KPI. Accordingly, the maximum end of the range of KPI values for the Normal state  3106  can be set to a number immediately preceding the threshold value of 50 (e.g., it can be set to 49 if the values used to indicate the KPI state are integers). 
     The maximum end of the range of KPI values for the Warning state  3104  is defined based on the threshold associated with the succeeding state (e.g., Critical state  3102 ) of the KPI. For example, the threshold value 75 can be received for the Critical state  3102  of the KPI, which can cause the maximum end of the range of values for the Warning state  3104  to be set to 74. The maximum end of the range of values for the highest state (e.g., Critical state  3102 ) can be a preset value or an indefinite value. 
     When input is received for a threshold value for a corresponding state of the KPI and/or a visual characteristic for an icon of the corresponding state of the KPI, GUI  3100  reflects this input by dynamically modifying a visual appearance of a vertical UI element (e.g., column  3118 ) that includes sections that represent the defined states for the KPI. Specifically, the sizes (e.g., heights) of the sections can be adjusted to visually illustrate ranges of KPI values for the states of the KPI, and the threshold values can be visually represented as marks on the column  3118 . In addition, the appearance of each section is modified based on the visual characteristic (e.g., color, pattern) selected by the user for each state via a drop-down menu  3114 . In some implementations, once the visual characteristic is selected for a specific state, it is also illustrated by modified appearance (e.g., modified color or pattern) of icon  3112  positioned next to a threshold value associated with that state. 
     For example, if the color green is selected for the Normal state  3106 , a respective section of column  3118  can be displayed with the color green to represent the Normal state  3106 . In another example, if the value 50 is received as input for the minimum end of a range of values for the Warning state  3104 , a mark  3117  is placed on column  3118  to represent the value 50 in proportion to other marks and the overall height of the column  3118 . As discussed above, the size (e.g., height) of each section of the UI element (e.g., column)  3118  is defined by the minimum end and the maximum end of the range of KPI values of the corresponding state. 
     In one implementation, GUI  3100  displays one or more pre-defined states for the KPI. Each predefined state is associated with at least one of a pre-defined unique name, a pre-defined value representing a minimum end of a range of values, or a predefined visual indicator. Each pre-defined state can be represented in GUI  3100  with corresponding GUI elements as described above. 
     GUI  3100  can facilitate user input to specify a maximum value  3116  and a minimum value  3120  for the combination of the KPI states to define a scale for a widget that represents the KPI. Some implementations of widgets for representing KPIs are discussed in greater detail below. GUI  3100  can display a button  3122  for receiving input indicating whether to apply the threshold(s) to the aggregate KPI of the service or to the particular KPI or both. The application of threshold(s) to the aggregate KPI of the service or to a particular KPI is discussed in more detail below. 
       FIGS.  31 B- 31 C  illustrate GUIs for defining threshold settings for a KPI, in accordance with an alternative implementation of the present disclosure. In GUI  3150  of  FIG.  31 B , adjacent to column  3118 , a line chart  3152  is displayed. The line chart  3152  represents the KPI values for the current KPI over a period of time selected from drop down menu  3154 . The KPI values are plotted over the period of time on a first horizontal axis and against a range of values set by the maximum value  3116  and minimum value  3120  on a second vertical axis. In one implementation when a mark  3156  is added to column  3118  indicating the end of a range of values for the a particular state a horizontal line  3158  is displayed along the length of line chart  3152 . The horizontal line  3158  makes it easy to visually correlate the KPI values represented by line chart  3152  with the end of the range of values. For example, in  FIG.  31 B , with the “Critical” state having a range below 15 GB, the horizontal line  3158  indicates that the KPI values drop below the end of the range four different times. This can provide information to a user that the end of the range of values indicated by mark  3156  can be adjusted. 
     In GUI  3160  of  FIG.  31 C , the user has adjusted the position of mark  3156 , thereby decreasing the end of the range of values for the “Critical” state to 10 GB. Horizontal line  3158  is also lowered to reflect the change. In one implementation, the user can click and drag mark  3156  down to the desired value. In another implementation, the user can type in the desired value. The user can tell that the KPI values now drop below the end of the only once, thereby limiting the number of alerts associated with the defined threshold. 
       FIGS.  31 D- 31 F  illustrate example GUIs for defining threshold settings for a KPI, in accordance with alternative implementations of the present disclosure. In one implementation, for services that have multiple entities, the method for determining the KPI value from data across the multiple entities is applied on a per entity basis. For example, if machine data pertaining to a first entity searched to produce a value relevant to the KPI (e.g., CPU load) every minute while machine data pertaining to a second entity is searched to produce the value relevant to the KPI every hour, simply averaging all the values together would give a skewed result, as the sheer number of values produced from the machine data pertaining to the first entity would mask any values produced from the machine data pertaining to the second entity in the average. Accordingly, in one implementation, the average value (e.g., cpu_load_percent) per entity is calculated over the selected time period and that average value for each entity is aggregated together to determine the KPI for the service. A per-entity average value that is calculated over the selected time period can represent a contribution of a respective KPI entity to the KPI of the service. Since the values are calculated on a per entity basis, thresholds can not only be applied to the KPI of the service (calculated based on contributions of all KPI entities of the service) but also to a KPI contribution of an individual entity. Different threshold types can be defined depending on threshold usage. 
     In GUI  3159  of  FIG.  31 D , different threshold types  3161  are presented. Threshold types  3161  include an aggregate threshold type, a per-entity threshold type and a combined threshold type. An aggregate threshold type represents thresholds applied to a KPI, which represents contributions of all KPI entities in the service. With an aggregate threshold type, a current KPI state can be determined by applying the determination component of the search query to an aggregate of machine data pertaining to all individual KPI entities to produce a KPI value and applying at least one aggregate threshold to the KPI value. 
     A per-entity threshold type represents thresholds applied separately to KPI contributions of individual KPI entities of the service. With a per-entity threshold type, a current KPI state can be determined by applying the determination component to an aggregate of machine data pertaining to an individual KPI entity to determine a KPI contribution of the individual KPI entity, comparing at least one per-entity threshold with a KPI contribution separately for each individual KPI entity, and selecting the KPI state based on a threshold comparison with a KPI contribution of a single entity. In other words, a contribution of an individual KPI entity can define the current state of the KPI of the service. For example, if the KPI of the service is below a critical threshold corresponding to the start of a critical state but a contribution of one of the KPI entities is above the critical threshold, the state of the KPI can be determined as critical. 
     A combined threshold type represents discrete thresholds applied separately to the KPI values for the service and to the KPI contributions of individual entities in the service. With a combined threshold type, a current KPI state can be determined twice—first by comparing at least one aggregate threshold with the KPI of the service, and second by comparing at least one per-entity threshold with a KPI contribution separately for each individual KPI entity. 
     In the example of  FIG.  31 D , the aggregate threshold type has been selected using a respective GUI element (e.g., one of buttons  3161 ), and thresholds have been provided to define different states for the KPI of the service. In response to the selection of the aggregate threshold type, GUI  3159  presents an interface component including line chart  3163  that visualizes predefined KPI states and how a current state of the KPI changes over a period of time selected from the monitoring GUI  2960 . In one implementation, the interface component includes a horizontal axis representing the selected period of time (e.g., last 60 minutes) and a vertical axis representing the range of possible KPI values. The various states of the KPI are represented by horizontal bands, such as  3164 ,  3165 ,  3166 , displayed along the horizontal length of the interface component. In one implementation, when a mark is added to column  3162  indicating the start or end of a range of values for a particular state, a corresponding horizontal band is also displayed. The marks in column  3162  can be dragged up and down to vary the KPI thresholds, and correspondingly, the ranges of values that correspond to each different state. Line chart  3163  represents KPI values for the current KPI over a period of time selected from the monitoring GUI  2960  and determined by the determination component of the search query, as described above. The KPI values are plotted over the period of time on a horizontal axis and against a range of values set by the maximum value and minimum value on a vertical axis. The horizontal bands  3164 - 3166  make it easy to visually correlate the KPI values represented by line chart  3163  with the start and end of the range of values of a particular state. For example, in  FIG.  31 D , with the “Critical” state having a range above 69.34%, the horizontal band  3164  indicates that the KPI value exceeds the start of the range one time. Since line chart  3163  represents the KPI of the service, the values plotted by line chart  3163  can include the average of the average cpu_load_percent of all KPI entities in the service, calculated over the selected period of time. Accordingly, the state of the KPI can only change when the aggregate contribution of all KPI entities crosses the threshold from one band  3164  to another  3165 . 
     In GUI  3170  of  FIG.  31 E , adjacent to column  3162 , an interface component with two line charts  3173  and  3177  is displayed. In this implementation, the per entity threshold type has been selected using a respective GUI element (e.g., one of buttons  3161 ). Accordingly, the line charts  3173  and  3177  represent the KPI contributions of individual entities in the service over the period of time selected from the monitoring GUI  2960 . The per-entity contributions are plotted over the period of time on a first horizontal axis and against a range of values set by the maximum value and minimum value on a second vertical axis. Since line charts  3173  and  3177  represent per entity KPI contributions, the values plotted by line chart  3173  can include the average cpu_load_percent of a first entity over the selected period of time, while the values plotted by line chart  3177  can include the average cpu_load_percent of a second entity over the same period of time. In one implementation, the determination component of the search query determines a contribution of an individual KPI entity from an aggregate of machine data corresponding to the individual KPI entity, applies at least one entity threshold to the contribution of the individual KPI entity, and selects a KPI state based at least in part on the determined contribution of the individual KPI entity in view of the applied threshold. Accordingly, the state of the KPI can change when any of the per entity contributions cross the threshold from one band  3166  to another  3165 . 
     In GUI  3180  of  FIG.  31 F , the combined threshold type has been selected using a respective GUI element (e.g., one of buttons  3161 ). Accordingly GUI  3180  includes two separate interface components with one line chart  3183  on a first set of axes that represents the KPI of the service in the first interface component, and two additional line charts  3187  and  3188  on a second set of axes that represent the per entity KPI contributions in the second interface component. Both sets of axes represent the same period of time on the horizontal axes, however, the range of values on the vertical axes can differ. Similarly, separate thresholds can be applied to the service KPI represented by line chart  3183  and to the per entity KPI contributions represented by line charts  3187  and  3188 . Since line chart  3183  represents the service KPI, the values plotted by line chart  3183  can include the average of the average cpu_load_percent of all entities in the service, calculated over the selected period of time. Accordingly, the state of the KPI can only change when the aggregate value crosses the thresholds that separate any of bands  3184 ,  3185 ,  3186  or  3189 . Since line charts  3187  and  3188  represent per entity contributions for the KPI, the values plotted by line chart  3187  can include the average cpu_load_percent of a first entity over the selected period of time, while the values plotted by line chart  3188  can include the average cpu_load_percent of a second entity over the same period of time. Accordingly, the state of the KPI can change when any of the per entity values cross the thresholds that separate any of bands  3164 ,  3165  or  3166 . In cases where the aggregate thresholds and per entity thresholds result in different states for the KPI, in one implementation, the more severe state can take precedence and be set as the state of the KPI. For example, if the aggregate threshold indicates a state of “Medium” but one of the per entity thresholds indicates a state of “High,” the more severe “High” state can be used as the overall state of the KPI. 
     In one implementation, a visual indicator, also referred to herein as a “lane inspector,” can be present in any of the GUIs  3150 - 3180 . The lane inspector includes, for example, a line or other indicator that spans vertically across the bands at a given point in time along the horizontal time axis. The lane inspector can be user manipulable such that it can be moved along the time axis to different points. In one implementation, the lane inspector includes a display of the point in time at which it is currently located. In one implementation, the lane inspector further includes a display of a KPI value reflected in each of the line charts at the current point in time illustrated by the lane inspector. Additional details of the lane inspector are described below, but are equally applicable to this implementation. 
       FIG.  31 G  is a flow diagram of an implementation of a method for defining one or more thresholds for a KPI on a per entity basis, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method  3422  is performed by the client computing machine. In another implementation, the method  3422  is performed by a server computing machine coupled to the client computing machine over one or more networks. 
     At block  3191 , the computing machine causes display of a GUI that presents information specifying a service definition for a service and a specification for determining a KPI for the service. In one implementation, the service definition identifies a service provided by a plurality of entities each having corresponding machine data. The specification for determining the KPI refers to the KPI definitional information (e.g., which entities, which records/fields from machine data, what time frame, etc.) that is being defined and is stored as part of the service definition or in association with the service definition. In one implementation, the KPI is defined by a search query that produces a value derived from the machine data pertaining to one or more KPI entities selected from among the plurality of entities. The KPI entities can include a set of entities of the service (i.e., service entities) whose relevant machine data is used in the calculation of the KPI. Thus, the KPI entities can include either whole set or a subset of the service entities. The value produced by the search query can be indicative of a performance assessment for the service at a point in time or during a period of time. In one implementation, the search query includes a machine data selection component that is used to arrive at a set of data from which to calculate a KPI and a determination component to derive a representative value for an aggregate of machine data. The determination component is applied to the identified set of data to produce a value on a per-entity basis (a KPI contribution of an individual entity). In one alternative, the machine data selection component is applied once to the machine data to gather the totality of the machine data for the KPI, and returns the machine data sorted by entity, to allow for repeated application of the determination component to the machine data pertaining to each entity on an individual basis. 
     At block  3192 , the computing machine receives user input specifying one or more entity thresholds for each of the KPI entities. The entity thresholds each represent an end of a range of values corresponding to a particular KPI state from among a set of KPI states, as described above. 
     At block  3193 , the computing machine stores the entity thresholds in association with the specification for determining the KPI for the service. In one implementation, the entity thresholds are added to the service definition. 
     At block  3194 , the computing machine makes the stored entity thresholds available for determining a state of the KPI. In one implementation, determining the state of the KPI includes determining a contribution of an individual KPI entity by applying the determination component to an aggregate of machine data corresponding to the individual KPI entity, and then applying at least one entity threshold to a KPI contribution of the individual KPI entity. Further, the computing machine selects a KPI state based at least in part on the determined contribution of the individual KPI entity in view of the applied entity threshold. In one implementation, the entity thresholds are made available by exposing them through an API. In one implementation, the entity thresholds are made available by storing information for referencing them in an index of definitional components. In one implementation, the entity thresholds are made available as an integral part of storing them in a particular logical or physical location, such as logically storing them as part of a KPI definitional information collection associated with a particular service definition. In such an implementation, a single action or process, then, can accomplish both the storing of the entity thresholds, and the making available of the entity thresholds. 
       FIG.  32    is a flow diagram of an implementation of a method  3200  for calculating an aggregate KPI score for a service based on the KPIs for the service, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method is performed by the client computing machine. In another implementation, the method is performed by a server computing machine coupled to the client computing machine over one or more networks. 
     At block  3201 , the computing machine identifies a service to evaluate. The service is provided by one or more entities. The computing system can receive user input, via one or more graphical interfaces, selecting a service to evaluate. The service can be represented by a service definition that associates the service with the entities as discussed in more detail above. 
     At block  3203 , the computing machine identifies key performance indicators (KPIs) for the service. The service definition representing the service can specify KPIs available for the service, and the computing machine can determine the KPIs for the service from the service definition of the service. Each KPI can pertain to a different aspect of the service. Each KPI can be defined by a search query that derives a value for that KPI from machine data pertaining to entities providing the service. As discussed above, the entities providing the service are identified in the service definition of the service. According to a search query, a KPI value can be derived from machine data of all or some entities providing the service. 
     In some implementations, not all of the KPIs for a service are used to calculate the aggregate KPI score for the service. For example, a KPI can solely be used for troubleshooting and/or experimental purposes and can not necessarily contribute to providing the service or impacting the performance of the service. The troubleshooting/experimental KPI can be excluded from the calculation of the aggregate KPI score for the service. 
     In one implementation, the computing machine uses a frequency of monitoring that is assigned to a KPI to determine whether to include a KPI in the calculation of the aggregate KPI score. The frequency of monitoring is a schedule for executing the search query that defines a respective KPI. As discussed above, the individual KPIs can represent saved searches. These saved searches can be scheduled for execution based on the frequency of monitoring of the respective KPIs. In one example, the frequency of monitoring specifies a time period (e.g., 1 second, 2 minutes, 10 minutes, 30 minutes, etc.) for executing the search query that defines a respective KPI, which then produces a value for the respective KPI with each execution of the search query. In another example, the frequency of monitoring specifies particular times (e.g., 6:00 am, 12:00 μm, 6:00 pm, etc.) for executing the search query. The values produced for the KPIs of the service, based on the frequency of monitoring for the KPIs, can be considered when calculating a score for an aggregate KPI of the service, as discussed in greater detail below. 
     Alternatively, the frequency of monitoring can specify that the KPI is not to be measured (that the search query for a KPI is not to be executed). For example, a troubleshooting KPI can be assigned a frequency of monitoring of zero. 
     In one implementation, if a frequency of monitoring is unassigned for a KPI, the KPI is automatically excluded in the calculation for the aggregate KPI score. In one implementation, if a frequency of monitoring is unassigned for a KPI, the KPI is automatically included in the calculation for the aggregate KPI score. 
     The frequency of monitoring can be assigned to a KPI automatically (without any user input) based on default settings or based on specific characteristics of the KPI such as a service aspect associated with the KPI, a statistical function used to derive a KPI value (e.g., maximum versus average), etc. For example, different aspects of the service can be associated with different frequencies of monitoring, and KPIs can inherit frequencies of monitoring of corresponding aspects of the service. 
     Values for KPIs can be derived from machine data that is produced by different sources. The sources can produce the machine data at various frequencies (e.g., every minute, every 10 minutes, every 30 minutes, etc.) and/or the machine data can be collected at various frequencies (e.g., every minute, every 10 minutes, every 30 minutes, etc.). In another example, the frequency of monitoring can be assigned to a KPI automatically (without any user input) based on the accessibility of machine data associated with the KPI (associated through entities providing the service). For example, an entity can be associated with machine data that is generated at a medium frequency (e.g., every 10 minutes), and the KPI for which a value is being produced using this particular machine data can be automatically assigned a medium frequency for its frequency of monitoring. 
     Alternatively, frequency of monitoring can be assigned to KPIs based on user input.  FIG.  33 A  illustrates an example GUI  3300  for creating and/or editing a KPI, including assigning a frequency of monitoring to a KPI, based on user input, in accordance with one or more implementations of the present disclosure. GUI  3300  for can include a button  3311  to receive a user request to assign a frequency of monitoring to the KPI being created or modified. Upon activating button  3311 , a user can enter (e.g., via another GUI or a command line interface) a frequency (e.g., a user defined value) for the KPI, or select a frequency from a list presented to the user. In one example, the list can include various frequency types, where each frequency type is mapped to a pre-defined and/or user-defined time period. For example, the frequency types can include Real Time (e.g., 1 second), High Frequency (e.g., 2 minutes), Medium Frequency (e.g., 10 minutes), Low Frequency (e.g., 30 minutes), Do Not Measure (e.g., no frequency). 
     The assigned frequency of monitoring of KPIs can be included in the service definition specifying the KPIs, or in a separate data structure together with other settings of a KPI. 
     Referring to  FIG.  32   , at block  3205 , the computing machine derives one or more values for each of the identified KPIs. The computing machine can cause the search query for each KPI to execute to produce a corresponding value. In one implementation, as discussed above, the search query for a particular KPI is executed based on a frequency of monitoring assigned to the particular KPI. When the frequency of monitoring for a KPI is set to a time period, for example, High Frequency (e.g., 2 minutes), a value for the KPI is derived each time the search query defining the KPI is executed every 2 minutes. The derived value(s) for each KPI can be stored in an index. In one implementation, when a KPI is assigned a frequency of monitoring of Do Not Measure or is assigned a zero frequency (no frequency), no value is produced (the search query for the KPI is not executed) for the respective KPI and no values for the respective KPI are stored in the data store. 
     At block  3207 , the computing machine calculates a value for an aggregate KPI score for the service using the value(s) from each of the KPIs of the service. The value for the aggregate KPI score indicates an overall performance of the service. For example, a Web Hosting service can have 10 KPIs and one of the 10 KPIs can have a frequency of monitoring set to Do Not Monitor. The other nine KPIs can be assigned various frequencies of monitoring. The computing machine can access the values produced for the nine KPIs in the data store to calculate the value for the aggregate KPI score for the service, as discussed in greater detail below. Based on the values obtained from the data store, if the values produced by the search queries for 8 of the 9 KPIs indicate that the corresponding KPI is in a normal state, then the value for an aggregate KPI score can indicate that the overall performance of the service is normal. 
     An aggregate KPI score can be calculated by adding the values of all KPIs of the same service together. Alternatively, an importance of each individual KPI relative to other KPIs of the service is considered when calculating the aggregate KPI score for the service. For example, a KPI can be considered more important than other KPIs of the service if it has a higher importance weight than the other KPIs of the service. 
     In some implementations, importance weights can be assigned to KPIs automatically (without any user input) based on characteristics of individual KPIs. For example, different aspects of the service can be associated with different weights, and KPIs can inherit weights of corresponding aspects of the service. In another example, a KPI deriving its value from machine data pertaining to a single entity can be automatically assigned a lower weight than a KPI deriving its value from machine data pertaining to multiple entities, etc. 
     Alternatively, importance weights can be assigned to KPIs based on user input. Referring again to  FIG.  33 A , GUI  3300  can include a button  3309  to receive a user request to assign a weight to the KPI being created or modified. Upon selecting button  3309 , a user can enter (e.g., via another GUI or a command line interface) a weight (e.g., a user defined value) for the KPI, or select a weight from a list presented to the user. In one implementation, a greater value indicates that a greater importance is placed on a KPI. For example, the set of values can be 1-10, where the value 10 indicates high importance of the KPI relative to the other KPIs for the service. For example, a Web Hosting service can have three KPIs: (1) CPU Usage, (2) Memory Usage, and (3) Request Response Time. A user can provide input indicating that the Request Response Time KPI is the most important KPI and can assign a weight of 10 to the Request Response Time KPI. The user can provide input indicating that the CPU Usage KPI is the next most important KPI and can assign a weight of 5 to the CPU Usage KPI. The user can provide input indicating that the Memory Usage KPI is the least important KPI and can assign a weight of 1 to the Memory Usage KPI. 
     In one implementation, a KPI is assigned an overriding weight. The overriding weight is a weight that overrides the importance weights of the other KPIs of the service. Input (e.g., user input) can be received for assigning an overriding weight to a KPI. The overriding weight indicates that the status (state) of KPI should be used a minimum overall state of the service. For example, if the state of the KPI, which has the overriding weight, is warning, and one or more other KPIs of the service have a normal state, then the service can only be considered in either a warning or critical state, and the normal state(s) for the other KPIs can be disregarded. 
     In another example, a user can provide input that ranks the KPIs of a service from least important to most important, and the ranking of a KPI specifies the user selected weight for the respective KPI. For example, a user can assign a weight of 1 to the Memory Usage KPI, assign a weight of 2 to the CPU Usage KPI, and assign a weight of 3 to the Request Response Time KPI. The assigned weight of each KPI can be included in the service definition specifying the KPIs, or in a separate data structure together with other settings of a KPI. 
     Alternatively or in addition, a KPI can be considered more important than other KPIs of the service if it is measured more frequently than the other KPIs of the service. In other words, search queries of different KPIs of the service can be executed with different frequency (as specified by a respective frequency of monitoring) and queries of more important KPIs can be executed more frequently than queries of less important KPIs. 
     As will be discussed in more detail below, the calculation of a score for an aggregate KPI can be based on ratings assigned to different states of an individual KPI. Referring again to  FIG.  33 A , a user can select button  3313  for defining threshold settings, including state ratings, for a KPI to display GUI  3350  in  FIG.  33 B .  FIG.  33 B  illustrates an example GUI  3350  for defining threshold settings, including state ratings, for a KPI, in accordance with one or more implementations of the present disclosure. Similarly to GUI  3100  of  FIG.  31 A , GUI  3350  includes horizontal GUI elements (e.g., in the form of input boxes)  3352 ,  3354  and  3356  that represent specific states of the KPI. For each state, a corresponding GUI element can display a name  3359 , a threshold  3360 , and a visual indicator  3362  (e.g., an icon having a distinct color for each state). The name  3359 , a threshold  3360 , and a visual indicator  3362  can be displayed based on user input received via the input fields of the respective GUI element. GUI  3350  can include a vertical GUI element (e.g., a column)  3368  that changes appearance (e.g., the size and color of its sectors) based on input received for a threshold value for a corresponding state of the KPI and/or a visual characteristic for an icon of the corresponding state of the KPI. In some implementations, once the visual characteristic is selected for a specific state via the menu  3364 , it is also illustrated by the modified appearance (e.g., modified color or pattern) of icon  3362  positioned next to a threshold value associated with that state. 
     In addition, GUI  3350  provides for configuring a rating for each state of the KPI. The ratings indicate which KPIs should be given more or less consideration in view of their current states. When calculating an aggregate KPI, a score of each individual KPI reflects the rating of that KPI&#39;s current state, as will be discussed in more detail below. Ratings for different KPI states can be assigned automatically (e.g., based on a range of KPI values for a state) or specified by a user. GUI  3350  can include a field  3380  that displays an automatically generated rating or a rating entered or selected by a user. Field  3380  can be located next to (or in the same row as) a horizontal GUI element representing a corresponding state. Alternatively, field  3380  can be part of the horizontal GUI element. In one example, a user can provide input assigning a rating of 1 to the Normal State, a rating of 2 to the Warning State, and a rating of 3 to the Critical State. 
     In one implementation, GUI  3350  displays a button  3372  for receiving input indicating whether to apply the threshold(s) to the aggregate KPI of the service or to the particular KPI or both. If a threshold is conFig.d to be applied to a certain individual KPI, then a specified action (e.g., generate alert, add to report) will be triggered when a value of that KPI reaches (or exceeds) the individual KPI threshold. If a threshold is conFig.d to be applied to the aggregate KPI of the service, then a specified action (e.g., create notable event, generate alert, add to incident report) will be triggered when a value (e.g., a score) of the aggregate KPI reaches (or exceeds) the aggregate KPI threshold. In some implementations, a threshold can be applied to both or either the individual or aggregate KPI, and different actions or the same action can be triggered depending on the KPI to which the threshold is applied. The actions to be triggered can be pre-defined or specified by the user via a user interface (e.g., a GUI or a command line interface) while the user is defining thresholds or after the thresholds have been defined. The action to be triggered in view of thresholds can be included in the service definition identifying the respective KPI(s) or can be stored in a data structure dedicated to store various KPI settings of a relevant KPI. 
       FIG.  34 A  is a flow diagram of an implementation of a method  3400  for calculating a score for an aggregate KPI for the service, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method is performed by the client computing machine. In another implementation, the method is performed by a server computing machine coupled to the client computing machine over one or more networks. 
     At block  3402 , the computing machine identifies a service to be evaluated. The service is provided by one or more entities. The computing system can receive user input, via one or more graphical interfaces, selecting a service to evaluate. 
     At block  3404 , the computing machine identifies key performance indicators (KPIs) for the service. The computing machine can determine the KPIs for the service from the service definition of the service. Each KPI indicates how a specific aspect of the service is performing at a point in time. 
     As discussed above, in some implementations, a KPI pertaining to a specific aspect of the service (also referred to herein as an aspect KPI) can be defined by a search query that derives a value for that KPI from machine data pertaining to entities providing the service. Alternatively, an aspect KPI can be a sub-service aggregate KPI. Such a KPI is sub-service in the sense that it characterizes something less than the service as a whole. Such a KPI is an aspect KPI in the almost definitional sense that something less than the service as a whole is an aspect of the service. Such a KPI is an aggregate KPI in the sense that the search which defines it produces its value using a selection of accumulated KPI values in the data store (or of contemporaneously produced KPI values, or a combination), rather than producing its value using a selection of event data directly. The selection of accumulated KPI values for such a sub-service aggregate KPI includes values for as few as two different KPI&#39;s defined for a service, which stands in varying degrees of contrast to a selection including values for all, or substantially all, of the active KPI&#39;s defined for service as is the case with a service-level KPI. (A KPI is an active KPI when its definitional search query is enabled to execute on a scheduled basis in the service monitoring system. See the related discussion in regards to  FIG.  32   . Unless otherwise indicated, discussion herein related to KPI&#39;s associated with a service, or the like, can presume the reference is to active KPI definitions, particularly where the context relates to available KPI values, such that the notion of “all” can reasonably be understood to represent something corresponding to technically less than “all” of the relevant, extant KPI definitions.) A method for determining (e.g., by calculating) a service-level aggregate KPI is discussed in relation to the flow diagram of  FIG.  32   . A person of ordinary skill in the art now will understand how the teachings surrounding  FIG.  32    can be adapted to determine or produce an aggregate KPI that is a sub-service aggregate KPI. Similarly, a person of skill in the art now will understand how teachings herein regarding GUIs for creating, establishing, modifying, viewing, or otherwise processing KPI definitions (such as GUIs discussed in relation to  FIGS.  22 - 27   ) can be adapted to accommodate a KPI having a defining search query that produces its value using a selection of accumulated KPI values in the data store (or of contemporaneously produced KPI values, or a combination), rather than producing its value using a selection of event data directly. 
     At block  3406 , the computing machine optionally identifies a weighting (e.g., user selected weighting or automatically assigned weighting) for each of the KPIs of the service. As discussed above, the weighting of each KPI can be determined from the service definition of the service or a KPI definition storing various setting of the KPI. 
     At block  3408 , the computing machine derives one or more values for each KPI for the service by executing a search query associated with the KPI. As discussed above, each KPI is defined by a search query that derives the value for a corresponding KPI from the machine data that is associated with the one or more entities that provide the service. 
     As discussed above, the machine data associated with the one or more entities that provide the same service is identified using a user-created service definition that identifies the one or more entities that provide the service. The user-created service definition also identifies, for each entity, identifying information for locating the machine data pertaining to that entity. In another example, the user-created service definition also identifies, for each entity, identifying information for a user-created entity definition that indicates how to locate the machine data pertaining to that entity. The machine data can include for example, and is not limited to, unstructured data, log data, and wire data. The machine data associated with an entity can be produced by that entity. In addition or alternatively, the machine data associated with an entity can include data about the entity, which can be collected through an API for software that monitors that entity. 
     The computing machine can cause the search query for each KPI to execute to produce a corresponding value for a respective KPI. The search query defining a KPI can derive the value for that KPI in part by applying a late-binding schema to machine data or, more specifically, to events containing raw portions of the machine data. The search query can derive the value for the KPI by using a late-binding schema to extract an initial value and then performing a calculation on (e.g., applying a statistical function to) the initial value. 
     The values of each of the KPIs can differ at different points in time. As discussed above, the search query for a KPI can be executed based on a frequency of monitoring assigned to the particular KPI. When the frequency of monitoring for a KPI is set to a time period, for example, Medium Frequency (e.g., 10 minutes), a value for the KPI is derived each time the search query defining the KPI is executed every 10 minutes. The derived value(s) for each KPI can be stored in a data store. When a KPI is assigned a zero frequency (no frequency), no value is produced (the search query for the KPI is not executed) for the respective KPI. 
     The derived value(s) of a KPI is indicative of how an aspect of the service is performing. In one example, the search query can derive the value for the KPI by applying a late-binding schema to machine data pertaining to events to extract values for a specific fields defined by the schema. In another example, the search query can derive the value for that KPI by applying a late-binding schema to machine data pertaining to events to extract an initial value for a specific field defined by the schema and then performing a calculation on (e.g., applying a statistical function to) the initial value to produce the calculation result as the KPI value. In yet another example, the search query can derive the value for the KPI by applying a late-binding schema to machine data pertaining to events to extract an initial value for specific fields defined by the late-binding schema to find events that have certain values corresponding to the specific fields, and counting the number of found events to produce the resulting number as the KPI value. 
     At block  3410 , the computing machine optionally maps the value produced by a search query for each KPI to a state. As discussed above, each KPI can have one or more states defined by one or more thresholds. In particular, each threshold can define an end of a range of values. Each range of values represents a state for the KPI. At a certain point in time or a period of time, the KPI can be in one of the states (e.g., normal state, warning state, critical state) depending on which range the value, which is produced by the search query of the KPI, falls into. For example, the value produced by the Memory Usage KPI can be in the range representing a Warning State. The value produced by the CPU Usage KPI can be in the range representing a Warning State. The value produced by the Request Response Time KPI can be in the range representing a Critical State. 
     At block  3412 , the computing machine optionally maps the state for each KPI to a rating assigned to that particular state for a respective KPI (e.g., automatically or based on user input). For example, for a particular KPI, a user can provide input assigning a rating of 1 to the Normal State, a rating of 2 to the Warning State, and a rating of 3 to the Critical State. In some implementations, the same ratings are assigned to the same states across the KPIs for a service. For example, the Memory Usage KPI, CPU Usage KPI, and Request Response Time KPI for a Web Hosting service can each have Normal State with a rating of 1, a Warning State with a rating of 2, and a Critical State with a rating of 3. The computing machine can map the current state for each KPI, as defined by the KPI value produced by the search query, to the appropriate rating. For example, the Memory Usage KPI in the Warning State can be mapped to 2. The CPU Usage KPI in the Warning State can be mapped to 2. The Request Response Time KPI in the Critical State can be mapped to 3. In some implementations, different ratings are assigned to the same states across the KPIs for a service. For example, the Memory Usage KPI can each have Critical State with a rating of 3, and the Request Response Time KPI can have Critical State with a rating of 5. 
     At block  3414 , the computing machine calculates an impact score for each KPI. In some implementations, the impact score of each KPI can be based on the importance weight of a corresponding KPI (e.g., weight×KPI value). In other implementations, the impact score of each KPI can be based on the rating associated with a current state of a corresponding KPI (e.g., rating×KPI value). In yet other implementations, the impact score of each KPI can be based on both the importance weight of a corresponding KPI and the rating associated with a current state of the corresponding KPI. For example, the computing machine can apply the weight of the KPI to the rating for the state of the KPI. The impact of a particular KPI at a particular point in time on the aggregate KPI can be the product of the rating of the state of the KPI and the importance (weight) assigned to the KPI. In one implementation, the impact score of a KPI can be calculated as follows:
 
Impact Score of KPI=(weight)×(rating of state)
 
     For example, when the weight assigned to the Memory Usage KPI is 1 and the Memory Usage KPI is in a Warning State, the impact score of the Memory Usage KPI=1×2. When the weight assigned to the CPU Usage KPI is 2 and the CPU Usage KPI is in a Warning State, the impact score of the CPU Usage KPI=2×2. When the weight assigned to the Request Response Time KPI is 3 and the Request Response Time KPI is in a Critical State, the impact score of the Request Response Time KPI=3×3. 
     In another implementation, the impact score of a KPI can be calculated as follows:
 
Impact Score of KPI=(weight)×(rating of state)×(value)
 
     In yet some implementations, the impact score of a KPI can be calculated as follows:
 
Impact Score of KPI=(weight)×(value)
 
     At block  3416 , the computing machine calculates an aggregate KPI score (“score”) for the service based on the impact scores of individual KPIs of the service. The score for the aggregate KPI indicates an overall performance of the service. The score of the aggregate KPI can be calculated periodically (as conFig.d by a user or based on a default time interval) and can change over time based on the performance of different aspects of the service at different points in time. For example, the aggregate KPI score can be calculated in real time (continuously calculated until interrupted). The aggregate KPI score can be calculated can be calculated periodically (e.g., every second). 
     In some implementations, the score for the aggregate KPI can be determined as the sum of the individual impact scores for the KPIs of the service. In one example, the aggregate KPI score for the Web Hosting service can be as follows:
 
Aggregate KPI Web Hosting =(weight×rating of state) Memory Usage KPI +(weight×rating of state) CPU Usage KPI +(weight×rating of state) Request Response Time KPI =(1×2)+(2×2)+(3×3)=15.
 
     In another example, the aggregate KPI score for the Web Hosting service can be as follows:
 
Aggregate KPI Web Hosting =(weight×rating of state×value) Memory Usage KPI +(weight×rating of state×value) CPU Usage KPI +(weight×rating of state×value) Request Response Time KPI =(1×2×60)+(2×2×55)+(3×3×80)=1060.
 
     In yet some other implementations, the impact score of an aggregate KPI can be calculated as a weighted average as follows:
 
Aggregate KPI Web Hosting =[(weight×rating of state) Memory Usage KPI +(weight×rating of state) CPU Usage KPI +(weight×rating of State) Request Response Time KPI )]/(weight Memory Usage KPI +weight CPU Usage KPI +weight Request Response Time KPI )
 
     A KPI can have multiple values produced for the particular KPI for different points in time, for example, as specified by a frequency of monitoring for the particular KPI. The multiple values for a KPI can be that in a data store. In one implementation, the latest value that is produced for the KPI is used for calculating the aggregate KPI score for the service, and the individual impact scores used in the calculation of the aggregate KPI score can be the most recent impact scores of the individual KPIs based on the most recent values for the particular KPI stored in a data store. Alternatively, a statistical function (e.g., average, maximum, minimum, etc.) is performed on the set of the values that is produced for the KPI is used for calculating the aggregate KPI score for the service. The set of values can include the values over a time period between the last calculation of the aggregate KPI score and the present calculation of the aggregate KPI score. The individual impact scores used in the calculation of the aggregate KPI score can be average impact scores, maximum impact score, minimum impact scores, etc. over a time period between the last calculation of the aggregate KPI score and the present calculation of the aggregate KPI score. 
     The individual impact scores for the KPIs can be calculated over a time range (since the last time the KPI was calculated for the aggregate KPI score). For example, for a Web Hosting service, the Request Response Time KPI can have a high frequency (e.g., every 2 minutes), the CPU Usage KPI can have a medium frequency (e.g., every 10 minutes), and the Memory Usage KPI can have a low frequency (e.g., every 30 minutes). That is, the value for the Memory Usage KPI can be produced every 30 minutes using machine data received by the system over the last 30 minutes, the value for the CPU Usage KPI can be produced every 10 minutes using machine data received by the system over the last 10 minutes, and the value for the Request Response Time KPI can be produced every 2 minutes using machine data received by the system over the last 2 minutes. Depending on the point in time for when the aggregate KPI score is being calculated, the value (e.g., and thus state) of the Memory Usage KPI can not have been refreshed (the value is stale) because the Memory Usage KPI has a low frequency (e.g., every 30 minutes). Whereas, the value (e.g., and thus state) of the Request Response Time KPI used to calculate the aggregate KPI score is more likely to be refreshed (reflect a more current state) because the Request Response Time KPI has a high frequency (e.g., every 2 minutes). Accordingly, some KPIs can have more impact on how the score of the aggregate KPI changes overtime than other KPIs, depending on the frequency of monitoring of each KPI. 
     In one implementation, the computing machine causes the display of the calculated aggregate KPI score in one or more graphical interfaces and the aggregate KPI score is updated in the one or more graphical interfaces each time the aggregate KPI score is calculated. In one implementation, the configuration for displaying the calculated aggregate KPI in one or more graphical interfaces is received as input (e.g., user input), stored in a data store coupled to the computing machine, and accessed by the computing machine. 
     At block  3418 , the computing machine compares the score for the aggregate KPI to one or more thresholds. As discussed above with respect to  FIG.  33 B , one or more thresholds can be defined and can be conFig.d to apply to a specific individual KPI and/or an aggregate KPI including the specific individual KPI. The thresholds can be stored in a data store that is coupled to the computing machine. If the thresholds are conFig.d to be applied to the aggregate KPI, the computing machine compares the score of the aggregate KPI to the thresholds. If the computing machine determines that the aggregate KPI score exceeds or reaches any of the thresholds, the computing machine determines what action should be triggered in response to this comparison. 
     Referring to  FIG.  34 A , at block  3420 , the computing machine causes an action be performed based on the comparison of the aggregate KPI score with the one or more thresholds. For example, the computing machine can generate an alert if the aggregate KPI score exceeds or reaches a particular threshold (e.g., the highest threshold). In another example, the computing machine can generate a notable event if the aggregate KPI score exceeds or reaches a particular threshold (e.g., the second highest threshold). In one implementation, the KPIs of multiple services is aggregated and used to create a notable event. In one implementation, the configuration for which of one or more actions to be performed is received as input (e.g., user input), stored in a data store coupled to the computing machine, and accessed by the computing machine. 
       FIG.  34 B  is a flow diagram of an implementation of a method of causing display of a GUI presenting information pertaining to notable events produced as a result of correlation searches, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method  34500  is performed by a client computing machine. In another implementation, the method  34500  is performed by a server computing machine coupled to the client computing machine over one or more networks. 
     At block  34501 , the computing machine performs a correlation search associated with a service provided by one or more entities that each have corresponding machine data. The service can include one or more key performance indicators (KPIs) that each indicate a state of a particular aspect of the service or a state of the service as a whole at a point in time or during a period of time. Each KPI can be derived from the machine data pertaining to the corresponding entities. Depending on the implementation, the KPIs can include an aggregate KPI and/or one or more aspect KPIs. A value of an aggregate KPI indicates how the service as a whole is performing at a point in time or during a period of time. A value of each aspect KPI indicates how the service in part (i.e., with respect to a certain aspect of the service) is performing at a point in time or during a period of time. As discussed above, the correlation search associated with the service can include search criteria pertaining to the one or more KPIs (i.e., an aggregate KPI and/or one or more aspect KPIs), and a triggering condition to be applied to data produced by a search query using the search criteria. 
     At block  34503 , the computing machine stores a notable event in response to the data produced by the search query satisfying the triggering condition. A notable event can represent a system occurrence that is likely to indicate a security threat or operational problem. Notable events can be detected in a number of ways: (1) an analyst can notice a correlation in the data and can manually identify a corresponding group of one or more events as “notable;” or (2) an analyst can define a “correlation search” specifying criteria for a notable event, and every time one or more events satisfy the criteria, the system can indicate that the one or more events are notable. An analyst can alternatively select a pre-defined correlation search provided by the application. Note that correlation searches can be run continuously or at regular intervals (e.g., every hour) to search for notable events. Upon detection, notable events can be stored in a dedicated “notable events index,” which can be subsequently accessed to generate various visualizations containing security-related information. As discussed above, the creation of a notable event can be the resulting action taken in response to the KPI correlation search producing data that satisfies the defined triggering condition. In addition, a notable event can also be created as a result of a correlation search (also referred to as a trigger-based search), that does not rely on a KPI, or the state of the KPI or of the corresponding service, but rather operates on any values produced in the system being monitored, and has a triggering condition and one or more actions that correspond to the triggering condition. 
     At block  34505 , the computing machine causes display of a graphical user interface presenting information pertaining to a stored notable event. The presented information can include an identifier of the correlation search that triggered the storing of the notable event and an identifier of the service associated with the correlation search. In other implementations, the graphical user interface can present additional information pertaining to the stored notable event, and can receive user input to modify or take action with respect to the notable event, as will be described further below. 
       FIG.  35    is a flow diagram of an implementation of a method  3500  for creating a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method is performed by the client computing machine. In another implementation, the method is performed by a server computing machine coupled to the client computing machine over one or more networks. 
     At block  3501 , the computing machine causes display of a dashboard-creation graphical interface that includes a modifiable dashboard template, and a KPI-selection interface. A modifiable dashboard template is part of a graphical interface to receive input for editing/creating a custom service-monitoring dashboard. A modifiable dashboard template is described in greater detail below. The display of the dashboard-creation graphical interface can be caused, for example, by a user selecting to create a service-monitoring dashboard from a GUI.  FIG.  36 A  illustrates an example GUI  3650  for creating and/or editing a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. In one implementation, GUI  3650  includes a menu item, such as Service-Monitoring Dashboards  3652 , which when selected can present a list  3656  of existing service-monitoring dashboards that have already been created. The list  3656  can represent service-monitoring dashboards that have data that is stored in a data store for displaying the service-monitoring dashboards. Each service-monitoring dashboard in the list  3656  can include a button  3658  for requesting a drop-down menu listing editing options to edit the corresponding service-monitoring dashboard. Editing can include editing the service-monitoring dashboard and/or deleting the service-monitoring dashboard. When an editing option is selected from the drop-down menu, one or more additional GUIs can be displayed for editing the service-monitoring dashboard. 
     The dashboard creation graphical interface can be a wizard or any other type of tool for creating a service-monitoring dashboard that presents a visual overview of how one or more services and/or one or more aspects of the services are performing. The services can be part of an IT environment and can include, for example, a web hosting service, an email service, a database service, a revision control service, a sandbox service, a networking service, etc. A service can be provided by one or more entities such as host machines, virtual machines, switches, firewalls, routers, sensors, etc. Each entity can be associated with machine data that can have different formats and/or use different aliases for the entity. As discussed above, each service can be associated with one or more KPIs indicating how aspects of the service are performing. The KPI-selection interface of the dashboard creation GUI allows a user to select KPIs for monitoring the performance of one or more services, and the modifiable dashboard template of the dashboard creation GUI allows the user to specify how these KPIs should be presented on a service-monitoring dashboard that will be created based on the dashboard template. The dashboard template can also define the overall look of the service-monitoring dashboard. The dashboard template for the particular service-monitoring dashboard can be saved, and subsequently, the service-monitoring dashboard can be generated for display based on the customized dashboard template and KPI values derived from machine data, as will be discussed in more details below. 
     GUI  3650  can include a button  3654  that a user can activate to proceed to the creation of a service-monitoring dashboard, which can lead to GUI  3600  of  FIG.  36 B .  FIG.  36 B  illustrates an example dashboard-creation GUI  3600  for creating a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. GUI  3600  includes a modifiable dashboard template  3608  and a KPI-selection interface  3606  for selecting a key performance indicator (KPI) of a service. GUI  3600  can facilitate input (e.g., user input) of a name  3602  of the particular service-monitoring dashboard that is being created and/or edited. GUI  3600  can include a button  3612  for storing the dashboard template  3608  for creating the service-monitoring dashboard. GUI  3600  can display a set of identifiers  3604 , each corresponding to a service. The set of identifies  3604  is described in greater detail below. GUI  3600  can also include a configuration interface  3610  for configuring style settings pertaining to the service-monitoring dashboard. The configuration interface  3610  is described in greater detail below. GUI  3600  can also include a customization toolbar  3601  for customizing the service-monitoring dashboard as described in greater detail below. The configuration interface  3610  can also include entity identifiers and facilitate input (e.g., user input) for selecting entity identifier of entities to be included in the service-monitoring dashboard. 
       FIG.  38 B  illustrates an example GUI  3810  for displaying a set of KPIs associated with a selected service for which a user can select for a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. When button  3812  is activated a list  3814  of a set of KPIs that are associated with the service can be displayed. The list  3814  can include an item  3816  for selecting all of the KPIs that are associated with the service into a modifiable dashboard template (e.g., modifiable dashboard template  3710  in  FIG.  37   ). The list  3814  can include a health score  3818  for the service. In one implementation, the health score is an aggregate KPI that is calculated for the service. An aggregate KPI can be calculated for a service as described above. 
     Returning to  FIG.  35   , at block  3503 , the computing machine optionally receives, via the dashboard-creation graphical interface, input for customizing an image for the service-monitoring dashboard and causes the customized image to be displayed in the dashboard-creation graphical interface at block  3505 . In one example, the computing machine optionally receives, via the dashboard-creation graphical interface, a selection of a background image for the service-monitoring dashboard and causes the selected background image to be displayed in the dashboard-creation graphical interface. The computing machine can display the selected background image in the modifiable dashboard template.  FIG.  37    illustrates an example GUI  3700  for a dashboard-creation graphical interface including a user selected background image, in accordance with one or more implementations of the present disclosure. GUI  3700  displays the user selected image  3708  in the modifiable dashboard template  3710 . 
     Referring again to  FIG.  35   , in another example, at block  3503 , the computing machine optionally receives input (e.g., user input) via a customization toolbar (e.g., customization toolbar  3601  in  FIG.  36 B ) for customizing an image for the service-monitoring dashboard. The customization toolbar can be a graphical interface containing drawing tools to customize a service-monitoring dashboard to define, for example, flow charts, text and connections between different elements on the service-monitoring dashboard. For example, the computing machine can receive input of a user drawing a flow chart or a representation of an environment (e.g., IT environment). In another example, the computing machine can receive input of a user drawing a representation of an entity and/or service. In another example, the computing machine can receive input of a user selection of an image to represent of an entity and/or service. 
     At block  3507 , the computing machine receives, through the KPI-selection interface, a selection of a particular KPI for a service. As discussed above, each KPI indicates how an aspect of the service is performing at one or more points in time. A KPI is defined by a search query that derives one or more values for the KPI from the machine data associated with the one or more entities that provide the service whose performance is reflected by the KPI. 
     In one example, prior to receiving the selection of the particular KPI, the computing machine causes display of a context panel graphical interface in the dashboard-creation graphical interface that contains service identifiers for the services (e.g., all of the services) within an environment (e.g., IT environment). The computing machine can receive input, for example, of a user selecting one or more of the service identifiers, and dragging and placing one or more of the service identifiers on the dashboard template. In another example, the computing machine causes display of a search box to receive input for filtering the service identifiers for the services. 
     In another example, prior to receiving the selection of the particular KPI, the computing machine causes display of a drop-down menu of selectable services in the KPI selection interface, and receives a selection of one of the services from the drop-down menu. In some implementations, selectable services can be displayed as identifiers corresponding to individual services, where each identifier can be, for example, the name of a particular service or the name of a service definition representing the particular service. As discussed in more detail above, a service definition can associate the service with one or more entities (and thereby with heterogeneous machine data pertaining to the entities) providing the service, and can specify one or more KPIs created for the service to monitor the performance of different aspects of the service. 
     In response to the user selection of a particular service, the computing machine can cause display of a list of KPIs associated with the selected service in the KPI selection interface, and can receive the user selection of the particular KPI from this list. 
     Referring again to  FIG.  37   , a user can select Web Hosting service  3701  in  FIG.  37    from the set of KPI identifiers  3702 , and in response to the selection of the Web Hosting service  3701 , the computing machine can cause display of a set of KPIs available for the Web Hosting service  3701 .  FIG.  38 A  illustrates an example GUI  3800  for displaying a set of KPIs associated with a selected service, in accordance with one or more implementations of the present disclosure. GUI  3800  can be a pop-up window that includes a drop-down menu  3801 , which when selected, displays a set of KPIs (e.g., Request Response Time and CPU Usage) associated with the service (e.g., Web Hosting service) corresponding to the selected service identifier. The user can then select a particular KPI from the menu. In another implementation, GUI  3800  also displays an aggregate KPI associated with the selected service, which can be selected to be represented by a KPI widget in the dashboard template for display in the service-monitoring dashboard. 
     Returning to  FIG.  35   , at block  3509 , the computing machine receives a selection of a location for placing the selected KPI in the dashboard template for displaying a KPI widget in a dashboard. Each KPI widget can provide a numerical or graphical representation of one or more values for a corresponding KPI or service health score (aggregate KPI for a service) indicating how a service or an aspect of a service is performing at one or more points in time. For example, a user can select the desired location for a KPI widget by clicking (or otherwise indicating) a desired area in the dashboard template. Alternatively, a user can select the desired location by dragging the selected KPI (e.g., its identifier in the form of a KPI name), and dropping the selected KPI at the desired location in the dashboard template. For example, when the user selects the KPI, a default KPI widget is automatically displayed at a default location in the dashboard template. The user can then select the location by dragging and dropping the default KPI widget at the desired location. As will be discussed in greater detail below, a KPI widget is a KPI identifier that provides a numerical and/or visual representation of one or more values for the selected KPI. A KPI widget can be, for example, a Noel gauge, a spark line, a single value, a trend indicator, etc. 
     At block  3511 , the computing machine receives a selection of one or more style settings for a KPI identifier (a KPI widget) to be displayed in the service-monitoring dashboard. For example, after the user selects the KPI, the user can provide input for creating and/or editing a title for the KPI. In one implementation, the computing machine causes the title that is already assigned to the selected KPI, for example via GUI  2200  in  FIG.  22   , to be displayed at the selected location in the dashboard template. In another example, after the user selects the KPI, the user is presented with available style settings, and the user can then select one or more of the style settings for the KPI widget to be displayed in the dashboard. In another example, in which a default KPI widget is displayed in response to the user selection of the KPI, the user can choose one or more of the available style setting(s) to replace or modify the default KPI widget. Style settings define how the KPI widget should be presented and can specify, for example, the shape of the widget, the size of the widget, the name of the widget, the metric unit of a KPI value, and/or other visual characteristics of the widget. Some implementations for receiving a selection of style setting(s) for a KPI widget to be displayed in the dashboard are discussed in greater detail below. At block  3513 , the computing machine causes display of a KPI identifier, such as a KPI widget, for the selected KPI at the selected location in the dashboard template. The KPI widget that is displayed in the dashboard template can be displayed using the selected style settings. The computing machine can receive further input (e.g., user input) for resizing a KPI widget via an input device (e.g., mouse, touch screen, etc.) For example, the computing device can receive user input via mouse device resizing (e.g., stretching, shrinking) the KPI widget. 
       FIG.  39 A  illustrates an example GUI  3900  facilitating user input for selecting a location in the dashboard template and style settings for a KPI widget, editing the service-monitoring dashboard by editing the dashboard template for the service-monitoring dashboard, and displaying the KPI widget in the dashboard template, in accordance with one or more implementations of the present disclosure. GUI  3900  includes a configuration interface  3906  to display a set of selectable thumbnail images (or icons or buttons)  3911  representing different types or styles of KPI widgets. The KPI widget styles can include, for example, and not limited to, a single value widget, a spark line widget, a Noel gauge widget, and a trend indicator widget.  FIG.  39 B  illustrates example KPI widgets, in accordance with one or more implementations of the present disclosure. Widget  3931  is an example of one implementation of a Noel gauge widget. Widget  3932  is an example of one implementation of a spark line widget. Widget  3933  is an example of one implementation of a trend indicator widget. 
     Referring to  FIG.  39 A , configuration interface  3905  can display a single value widget thumbnail image  3907 , a spark line widget thumbnail image  3908 , a Noel gauge widget thumbnail image  3909 , and a trend indicator widget thumbnail image  3910 . For example, a user can have selected the Web Hosting service  3901 , dragged the Web Hosting service  3901 , and dropped the Web Hosting service  3901  on location  3905 . The user can also have selected the CPU Usage KPI for the Web Hosting service  3901  and the Noel gauge widget thumbnail image  3909  to display the KPI widget for the CPU Usage KPI at the location  3905 . In response, the computing machine can cause display of the Noel Gauge widget for the selected KPI (e.g., CPU Usage KPI) at the selected location (e.g., location  3905 ) in the dashboard template  3903 . Some implementations of widgets for representing KPIs are discussed in greater detail below. In response to a user selection of a style setting for the KPI widget, one or more GUIs can be presented for customizing the selected KPI widget for the KPI. Input can be received via the GUIs to select a label for a KPI widget and the metric unit to be used for the KPI value with the KPI widget. 
     In one implementation, GUI  3900  includes an icon  3914  in the customization toolbar, which can be selected by a user, for defining one or more search queries. The search queries can produce results pertaining to one or more entities. For example, icon  3914  can be selected and an identifier  3918  for a search widget can be displayed in the dashboard template  3903 . The identifier  3918  for the search widget can be the search widget itself, as illustrated in  FIG.  39 A . The search widget can be a shape (e.g., box) and can display results (e.g., value produced by a corresponding search query) in the shape in the service-monitoring dashboard when the search query is executed for displaying the service—monitoring dashboard to a user. 
     The identifier  3918  can be displayed in a default location in the dashboard template  3903  and a user can optionally select a new location for the identifier  3918 . The location of the identifier  3918  in the dashboard template specifies the location of the search widget in the service-monitoring dashboard when the service-monitoring dashboard is displayed to a user. GUI  3900  can display a search definition box (e.g., box  3915 ) that corresponds to the search query. A user can provide input for the criteria for the search query via the search definition box (e.g., box  3915 ). For example, the search query can produce a stats count for a particular entity. The input pertaining to the search query is stored as part of the dashboard template. The search query can be executed when the service-monitoring dashboard is displayed to a user and the search widget can display the results from executing the search query. 
     Referring to  FIG.  35   , in one implementation, the computing machine receives input (e.g., user input), via the dashboard-creation graphical interface, of a time range to use for the KPI widget, editing the service-monitoring dashboard, and clearing data in the dashboard template. 
     At block  3515 , the computing machine stores the resulting dashboard template in a data store. The dashboard template can be saved in response to a user request. For example, a request to save the dashboard template can be received upon selection of a save button (e.g., save button  3612  in GUI  3600  of  FIG.  36   ). In one implementation, an image source byte for the resulting dashboard template is stored in a data store. In one implementation, an image source location for the resulting dashboard template is stored in a data store. The resulting dashboard template can be stored in a structure where each item (e.g., widget, line, text, image, shape, connector, etc.) has properties specified by the service-monitoring dashboard creation GUI. 
     Referring to  FIG.  35   , at block  3517 , the computing machine can receive a user request for a service-monitoring dashboard, and can then generate and cause display of the service-monitoring dashboard based on the dashboard template at block  3519 . Some implementations for causing display of a service-monitoring dashboard based on the dashboard template are discussed in greater detail below. 
       FIG.  40 A  illustrates an example GUI  4610  for creating and/or editing a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. GUI  4610  can present a list  4612  of existing service-monitoring dashboards that have already been created. The list  4612  can represent service-monitoring dashboards that have data that is stored in a data store for displaying the service-monitoring dashboards. In one implementation, the list  4612  includes one or more default service-monitoring dashboards that can be edited. 
     Each service-monitoring dashboard in the list  4612  can include a title  4611 . In one implementation, the title  4611  is a link, which when selected, can display the particular service-monitoring dashboard in a GUI in view mode, as described in greater detail below. 
     Each service-monitoring dashboard in the list  4612  can include a button  4613 , which when selected, can present a list of actions, which can be taken on a particular service-monitoring dashboard, from which a user can select from The actions can include, and are not limited to, editing a service-monitoring dashboard, editing a title and/or description for a service-monitoring dashboard, editing permissions for a service-monitoring dashboard, cloning a service-monitoring dashboard, and deleting a service-monitoring dashboard. When an action is selected, one or more additional GUIs can be displayed for facilitating user input pertaining to the action, as described in greater detail below. For example, button  4613  can be selected, and an editing action can be selected to display a GUI (e.g., GUI  4620  in  FIG.  40 C  described below) for editing the “Web Arch” service-monitoring dashboard. 
     GUI  4610  can display application information  4615  for each service-monitoring dashboard in the list  4612 . The application information  4615  can indicate an application that is used for creating and/or editing the particular service-monitoring dashboard. GUI  4610  can display owner information  4614  for each service-monitoring dashboard in the list  4612 . The owner information  4614  can indicate a role that is assigned to the owner of the particular service-monitoring dashboard. 
     GUI  4610  can display permission information  4616  for each service-monitoring dashboard in the list  4612 . The permission information can indicate a permission level (e.g., application level, private level). An application level permission level allows any user that is authorized to access to the service-monitoring dashboard creation and/or editing GUIs permission to access and edit the particular service-monitoring dashboard. A private level permission level allows a single user (e.g., owner, creator) permission to access and edit the particular service-monitoring dashboard. In one implementation, a permission level include permissions by role. In one implementation, one or more specific users can be specified for one or more particular levels. 
     GUI  4610  can include a button  4617 , which when selected can display GUI  4618  in  FIG.  40 B  for specifying information for a new service-monitoring dashboard. 
       FIG.  40 B  illustrates an example GUI  4618  for specifying information for a new service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. GUI  4618  can include a text box  4619 A enabling a user to specify a title for the service-monitoring dashboard, a text box  4619 B enabling a user to specify a description for the service-monitoring dashboard, and buttons  4916 C enabling a user to specify permissions for the service-monitoring dashboard. 
       FIG.  40 C  illustrates an example GUI  4620  for editing a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. GUI  4620  is displaying the service-monitoring dashboard in an edit mode that enables a user to edit the service-monitoring dashboard via a KPI-selection interface  4632 , a modifiable dashboard template  4360 , a configuration interface  4631 , and a customization toolbar  4633 . 
     The current configuration for the “Web Arch” service-monitoring dashboard that is stored in a data store can be used to populate the modifiable dashboard template  4630 . One or more widgets that have been selected for one or more KPIs can be displayed in the modifiable dashboard template  4630 . 
     A KPI that is being represented by a widget in the modifiable dashboard template  4630  can be a service-related KPI or an adhoc KPI. A service-related KPI is a KPI that is related to one or more services and/or one or more entities. A service-related KPI can be defined using service monitoring GUIs, as described in above. An ad-hoc KPI is a key performance indicator that is not related to any service or entity. For example, service-related KPI named “Web performance” is represented by Noel gauge widget  4634 . The Web performance can be a KPI that is related to “Splunk Service”  4635 . 
     The configuration interface  4631  can display data that pertains to a KPI (e.g., service-related KPI, adhoc KPI) that is selected in the modifiable dashboard template  4630 . For example, an adhoc KPI can be defined via GUI  4620 . For example, an adhoc search button  4621  can be activated and a location (e.g., location  4629 ) can be selected in the modifiable dashboard template  4630 . A widget  4628  for the adhoc KPI can be displayed at the selected location  4629 . In one implementation, a default widget (e.g., single value widget) is displayed for the adhoc KPI. 
     The configuration interface  4631  can display data that pertains to the adhoc KPI. For example, configuration interface  4631  can display source information for the adhoc KPI. The source information can indicate whether the adhoc KPI is derived from an adhoc search or data model. An adhoc KPI can be defined by a search query. The search query can be derived from a data model or an adhoc search query. An adhoc search query is a user-defined search query. 
     In one implementation, when the adhoc search button  4621  is activated for creating an adhoc KPI, the adhoc KPI is derived from an adhoc search query by default, and the adhoc type button  4624  is displayed as enabled. The adhoc type button  4624  can also be user-selected to indicate that the adhoc KPI is to be derived from an adhoc search query. 
     When the adhoc type button  4624  is enabled, a text box  4626  can be displayed for the search language defining the adhoc search query. In one implementation, the text box  4626  is populated with the search language for a default adhoc search query. In one implementation, the default adhoc search query is a count of events, and the search language “index=_internal|timechart count is displayed in the text box  4626 . A user can edit the search language via the text box  4626  to change the adhoc search query. 
     When the data model type button  4625  is selected, the configuration interface  4631  can display an interface for using a data model to define the adhoc KPI is displayed.  FIG.  40 D  illustrates an example interface  4640  for using a data model to define an adhoc KPI, in accordance with one or more implementations of the present disclosure. If button  4641  is selected, a GUI is displayed that enables a user to specify a data model, an object of the data model, and a field of the object for defining the adhoc KPI. If button  4643  is selected, a GUI is displayed that enables a user to select a statistical function (e.g., count, distinct count) to calculate a statistic using the value(s) from the field. 
     Referring to  FIG.  40 C , one or more types of KPI widgets can support the configuration of thresholds for the adhoc KPI. For example, a Noel gauge widget, a spark line widget, and a trend indicator widget (also referred to as a” single value delta widget” throughout this document) can support setting one or more thresholds for the adhoc KPI. For example, if the Noel gauge button  4627  is activated, the configuration interface  4631  can display an interface for setting one or more thresholds for the adhoc KPI. 
       FIG.  40 E  illustrates an example interface  4645  for setting one or more thresholds for the adhoc KPI, in accordance with one or more implementations of the present disclosure. The configuration interface  4645  can include a button  4647 , which when selected, displays a GUI (e.g., GUI  3100  in  FIG.  31 A , GUI  3150  in  FIG.  31 B ) for setting one or more thresholds for the adhoc KPI. If the update button  4648  is activate, the widget for the adhoc KPI can be updated, as described below. 
     Referring to  FIG.  40 C , if the update button (e.g., update button  4648  in  FIG.  40 E ) is activated, the widget  4628  can be updated to display a Noel gauge widget. If the adhoc KPI is being defined using a data model, the configuration interface  4631  can display the user selected settings for the adhoc KPI that have been specified, for example, using GUI  4640  in  FIG.  40 D . 
     Referring to  FIG.  40 C , if a service-related KPI widget is selected in the modifiable dashboard template  4630 , the configuration interface  4631  can display information pertaining to the service-related KPI. For example, the Noel gauge widget  4634  can be selected, and the configuration interface  4631  can display information pertaining to the “Web performance” KPI that is related to the Splunk Service  4635 . 
       FIG.  40 F  illustrates an example interface  4650  for a service-related KPI, in accordance with one or more implementations of the present disclosure. The text box  4651  can display the search language for the search query used to define the service-related KPI. The text box  4651  can be disabled to indicate that the service-related KPI cannot be edited from the glass table. 
     Referring to  FIG.  40 C , if the run search link  4636  is activated, a search GUI that displays information (e.g., search language, search result set) for a KPI (e.g., service KPI, adhoc KPI) that is selected in the modifiable dashboard template  4630 . 
       FIG.  41 A  is a flow diagram of an implementation of a method  4750  for creating and causing for display a service-monitoring dashboard, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method is performed by the client computing machine. In another implementation, the method is performed by a server computing machine coupled to the client computing machine over one or more networks. 
     At block  4751 , the computing machine identifies one or more key performance indicators (KPIs) for one or more services to be monitored via a service-monitoring dashboard. A service can be provided by one or more entities. Each entity can be associated with machine data. The machine data can include unstructured data, log data, and/or wire data. The machine data associated with an entity can include data collected from an API for software that monitors that entity. 
     A KPI can be defined by a search query that derives one or more values from machine data associated with the one or more entities that provide the service. Each KPI can be defined by a search query that is either entered by a user or generated through a graphical interface. In one implementation, the computing machine accesses a dashboard template that is stored in a data store that includes information identifying the KPIs to be displayed in the service-monitoring dashboard. In one implementation, the dashboard template specifies a service definition that associates the service with the entities providing the service, specifies the KPIs of the service, and also specifies the search queries for the KPIs. As discussed above, the search query defining a KPI can derive one or more values for the KPI using a late-binding schema that it applies to machine data. In some implementations, the service definition identified by the dashboard template can also include information on predefined states for a KPI and various visual indicators that should be used to illustrate states of the KPI in the dashboard. 
     The computing machine can optionally receive input (e.g., user input) identifying one or more ad hoc searches to be monitored via the service-monitoring dashboard without selecting services or KPIs. 
     At block  4753 , the computing machine identifies a time range. The time range can be a default time range or a time range specified in the dashboard template. The machine data can be represented as events. The time range can be used to indicate which events to use for the search queries for the identified KPIs. 
     At block  4755 , for each KPI, the computing machine identifies a KPI widget style to represent the respective KPI. In one implementation, the computing machine accesses the dashboard template that includes information identifying the KPI widget style to use for each KPI. As discussed above, examples of KPI widget styles can include a Noel gauge widget style, a single value widget style, a spark line widget style, and a trend indicator widget style. The computing machine can also obtain, from the dashboard template, additional visual characteristics for each KPI widget, such as, the name of the widget, the metric unit of the KPI value, settings for using nominal colors or colors to represent states and/or trends, the type of value to be represented in KPI widget (e.g., the type of value to be represented by value  4407  in a spark line widget), etc. 
     The KPIs widget styles can display data differently for representing a respective KPI. The computing machine can produce a set of values and/or a value, depending on the KPI widget style for a respective KPI. If the KPI widget style represents the respective KPI using values for multiple points in time in the time range, method  4750  proceeds to block  4757 . Alternatively, if the KPI widget style represents the respective KPI using a single value during the time range, method  4750  proceeds to block  4759 . 
     At block  4759 , if the KPI widget style represents the respective KPI using a single value, the computing machine causes a value to be produced from a set of machine data or events whose timestamps are within the time range. The value can be a statistic calculated based on one or more values extracted from a specific field in the set of machine data or events when the search query is executed. The statistic can be an average of the extracted values, a mean of the extracted values, a maximum of the extracted values, a last value of the extracted values, etc. A single value widget style, a Noel gauge widget style, and trend indicator widget style can represent a KPI using a single value. 
     The search query that defines a respective KPI can produce a single value which a KPI widget style can use. The computing machine can cause the search query to be executed to produce the value. The computing machine can send the search query to an event processing system. As discussed above, machine data can be represented as events. The machine data used to derive the one or more KPI values can be identifiable on a per entity basis by referencing entity definitions that are aggregated into a service definition corresponding to the service whose performance is reflected by the KPI. 
     The event processing system can access events with time stamps falling within the time period specified by the time range, identify which of those events should be used (e.g., from the one or more entity definitions in the service definition for the service whose performance is reflected by the KPI), produce the result (e.g., single value) using the identified events, and send the result to the computing machine. The computing machine can receive the result and store the result in a data store. 
     At block  4757 , if the KPI widget style represents the respective KPI using a set of values, the computing machine causes a set of values for multiple points in time in the time range to be produced. A spark line widget style can represent a KPI using a set of values. Each value in the set of values can represent an aggregate of data in a unit of time in the time range. For example, if the time range is “Last 15 minutes”, and the unit of time is one minute, then each value in the set of values is an aggregate of the data in one minute in the last 15 minutes. 
     If the search query that defines a respective KPI produces a single value instead of a set of multiple values as required by the KPI widget style (e.g., for the graph of the spark line widget), the computing machine can modify the search query to produce the set of values (e.g., for the graph of the spark line widget). The computing machine can cause the search query or modified search query to be executed to produce the set of values. The computing machine can send the search query or modified search query to an event processing system. The event processing system can access events with time stamps falling within the time period specified by the time range, identify which of those events should be used, produce the results (e.g., set of values) using the identified events, and send the results to the computing machine. The computing machine can store the results in a data store. 
     At block  4761 , for each KPI, the computing machine generates the KPI widget using the KPI widget style and the value or set of values produced for the respective KPI. For example, if a KPI is being represented by a spark line widget style, the computing machine generates the spark line widget using a set of values produced for the KPI to populate the graph in the spark line widget. The computing machine also generates the value for the spark line widget based on the dashboard template. The dashboard template can store the selection of the type of value that is to be represented in a spark line widget. For example, the value can represent the first data point in the graph, the last data point the graph, an average of all of the points in the graph, the maximum value from all of the points in the graph, or the mean of all of the points in the graph. 
     In addition, in some implementations, the computing machine can obtain KPI state information (e.g., from the service definition) specifying KPI states, a range of values for each state, and a visual characteristic (e.g., color) associated with each state. The computing machine can then determine the current state of each KPI using the value or set of values produced for the respective KPI, and the state information of the respective KPI. Based on the current state of the KPI, a specific visual characteristic (e.g., color) can be used for displaying the KPI widget of the KPI, as discussed in more detail above. 
     At block  4763 , the computing machine generates a service-monitoring dashboard with the KPI widgets for the KPIs using the dashboard template and the KPI values produced by the respective search queries. In one implementation, the computing machine accesses a dashboard template that is stored in a data store that includes information identifying the KPIs to be displayed in the service-monitoring dashboard. As discussed above, the dashboard template defines locations for placing the KPI widgets, and can also specify a background image over which the KPI widgets can be placed. 
     At block  4765 , the computing machine causes display of the service-monitoring dashboard with the KPI widgets for the KPIs. Each KPI widget provides a numerical and/or graphical representation of one or more values for a corresponding KPI. Each KPI widget indicates how an aspect of the service is performing at one or more points in time. For example, each KPI widget can display a current KPI value, and indicate the current state of the KPI using a visual characteristic associated with the current state. In one implementation, the service-monitoring dashboard is displayed in a viewing/investigation mode based on a user selection to view the service-monitoring dashboard is displayed in the viewing/investigation mode. 
     At block  4767 , the computing machine optionally receives a request for detailed information for one or more KPIs in the service-monitoring dashboard. The request can be received, for example, from a selection (e.g., user selection) of one or more KPI widgets in the service-monitoring dashboard. 
     At block  4759 , the computing machine causes display of the detailed information for the one or more KPIs. In one implementation, the computing machine causes the display of a deep dive visual interface, which includes detailed information for the one or more KPIs. A deep dive visual interface is described in greater detail below. 
     The service-monitoring dashboard can allow a user to change a time range. In response, the computing machine can send the search query and the new time range to an event processing system. The event processing system can access events with time stamps falling within the time period specified by the new time range, identify which of those events should be used, produce the result (e.g., one or more values) using the identified events, and send the result to the computing machine. The computing machine can then cause the service-monitoring dashboard to be updated with new values and modified visual representations of the KPI widgets. 
       FIG.  41 B  illustrates an example service-monitoring dashboard GUI  4700  that is displayed based on the dashboard template, in accordance with one or more implementations of the present disclosure. GUI  4700  includes a user selected background image  4702  and one or more KPI widgets for one or more services that are displayed over the background image  4702 . GUI  4700  can include other elements, such as, and not limited to text, boxes, connections, and widgets for ad hoc searches. Each KPI widget provides a numerical or graphical representation of one or more values for a corresponding key performance indicator (KPI) indicating how an aspect of a respective service is performing at one or more points in time. For example, GUI  4700  includes a spark line widget  4718  which can be for an aspect of Service-B, and a Noel gauge widget  4708  which can be for another aspect of Service-B. In some implementations, the appearance of the widgets  4718  and  4708  (as well as other widgets in the GUI  4700 ) can reflect the current state of the respective KPI (e.g., based on color or other visual characteristic). 
     Each service is provided by one or more entities. Each entity is associated with machine data. The machine data can include for example, and is not limited to, unstructured data, log data, and wire data. The machine data that is associated with an entity can include data collected from an API for software that monitors that entity. The machine data used to derive the one or more values represented by a KPI is identifiable on a per entity basis by referencing entity definitions that are aggregated into a service definition corresponding to the service whose performance is reflected by the KPI. 
     Each KPI is defined by a search query that derives the one or more values represented by the corresponding KPI widget from the machine data associated with the one or more entities that provide the service whose performance is reflected by the KPI. The search query for a KPI can derive the one or more values for the KPI it defines using a late-binding schema that it applies to machine data. 
     In one implementation, the GUI  4700  includes one or more search result widgets (e.g., widget  4712 ) displaying a value produced by a respective search query, as specified by the dashboard template. For example, widget  4712  can represent the results of a search query producing a stats count for a particular entity. 
     In one implementation, GUI  4700  facilitates user input for displaying detailed information for one or more KPIs. A user can select one or more KPI widgets to request detailed information for the KPIs represented by the selected KPI widgets. The detailed information for each selected KPI can include values for points in time during the period of time. The detailed information can be displayed via one or more deep dive visual interfaces. A deep dive visual interface is described in greater detail below. 
     Referring to  FIG.  41 B , GUI  4700  facilitates user input for changing a time range. The machine data used by a search query to produce a value for a KPI is based on a time range. As described above, the time range can be a user-defined time range. For example, if the time range “Last 15 minutes” is selected, the last 15 minutes would be an aggregation period for producing the value. GUI  4700  can be updated with one or more KPI widgets that each represent one or more values for a corresponding KPI indicating how a service provided is performing at one or more points in time based on the change to the time range. 
       FIG.  41 C  illustrates an example service-monitoring dashboard GUI  4750  that is displayed in view mode based on the dashboard template, in accordance with one or more implementations of the present disclosure. In one implementation, when a service-monitoring dashboard is in view mode, the service-monitoring dashboard cannot be edited. GUI  4750  can include a button  4755 , which when selected, can display a dashboard creation GUI (e.g., GUI  4620  in  FIG.  40 C ) for editing a service-monitoring dashboard. 
     GUI  4750  can display the items  4751  (e.g., service-related KPI widgets, adhoc KPI widgets, images, connectors, text, shapes, line etc.) as specified using the KPI-selection interface, modifiable dashboard template, configuration interface, and customization tool bar. 
     In one implementation, one or more widgets (e.g., service-related KPI widgets, adhoc KPI widgets) that are presented in view mode can be selected by a user to display one or more GUIs presenting more detailed information, for example, in a deep dive visualization, as described in greater detail below. 
     For example, a service-related KPI widget for a particular KPI can be displayed in view mode. When the service-related KPI widget is selected, a deep dive visualization can be displayed that presents data pertaining to the service-related KPI. The service-related KPI is related to a particular service and one or more other services based on dependency. The data that is presented in the deep dive visualization can include data for all of the KPIs that are related to the particular service and/or all of the KPIs from one or more dependent services. 
     When an adhoc KPI widget is displayed in view mode, and is selected, a deep dive visualization can be displayed that presents data pertaining to the adhoc search for the adhoc KPI. 
     GUI  4750  can include a button  4753  for displaying an interface (e.g., interface  4312  in  FIG.  43 B ) for specifying an end date and time for a time range to use when executing a search query defining a KPI displayed in GUI  4750 . 
       FIG.  42    describes an example home page GUI  4800  for service-level monitoring, in accordance with one or more implementations of the present disclosure. GUI  4800  can include one or more tiles each representing a service-monitoring dashboard. The GUI  4800  can also include one or more tiles representing a service-related alarm, or the value of a particular KPI. In one implementation, a tile is a thumbnail image of a service-monitoring dashboard. When a service-monitoring dashboard is created, a tile representing the service-monitoring dashboard can be displayed in the GUI  4800 . GUI  4800  can facilitate user input for selecting to view a service-monitoring dashboard. GUI  4800  can include tiles representing the most recently accessed dashboards, and user selected favorites of dashboards. 
       FIG.  43 A  describes an example home page GUI  4900  for service-level monitoring, in accordance with one or more implementations of the present disclosure. GUI  4900  can include one or more tiles representing a deep dive. In one implementation, a tile is a thumbnail image of a deep dive. When a deep dive is created, a tile representing the deep dive can be displayed in the GUI  4900 . GUI  4900  can facilitate user input for selecting to view a deep dive. GUI  4900  can include tiles representing the most recently accessed deep dives, and user selected favorites of deep dives. 
       FIG.  43 B  is a flow diagram of an implementation of a method for creating a home page GUI for service-level and KPI-level monitoring, in accordance with one or more implementations of the present disclosure. The method can be performed by processing logic that can comprise hardware (circuitry, dedicated logic, etc.), software (such as is run on a general purpose computer system or a dedicated machine), or a combination of both. In one implementation, the method  4910  is performed by a client computing machine. In another implementation, the method  4910  is performed by a server computing machine coupled to the client computing machine over one or more networks. 
     At block  4911 , the computing machine receives a request to display a service-monitoring page (also referred to herein as a “service-monitoring home page” or simply as a “home page”). In one implementation, the service monitoring page includes visual representations of the health of a system that can be easily viewed by a user (e.g., a system administrator) with a quick glance. The system can include one or more services. The performance of each service can be monitored using an aggregate KPI characterizing the respective service as a whole. In addition, various aspects (e.g., CPU usage, memory usage, response time, etc.) of a particular service can be monitored using respective aspect KPIs typifying performance of individual aspects of the service. For example, a service can have 10 separate aspect KPIs, each monitoring a various aspect of the service. 
     As discussed above, each KPI is associated with a service provided by one or more entities, and each KPI is defined by a search query that produces a value derived from machine data pertaining to the one or more entities. A value of each aggregate KPI indicates how the service in whole is performing at a point in time or during a period of time. A value of each aspect KPI indicates how the service in part (with respect to a certain aspect of the service) is performing at a point in time or during a period of time. 
     At block  4912 , the computing machine can determine data associated with one or more aggregate KPI definitions and one or more aspect KPI definitions, useful for creating the home page GUI. In an implementation, determining the data can include referencing service definitions in a data store, and/or referencing KPI definitions is a data store, and/or referencing stored KPI values, and/or executing search queries to produce KPI values. In an implementation, determining the data can include determining KPI-related information for each of a set of aggregate KPI definitions and for each of a set of aspect KPI definitions. The KPI-related information for each aggregate or aspect KPI definition can include a KPI state. At block  4912 , the computing machine can determine an order for both the set of aggregate KPI definitions and the set of aspect KPI definitions. (Information related to the KPI definition can vicariously represent the KPI definition in the ordering process such that if the information related to the KPI definition is ordered with respect to the information related to other KPI definitions, the KPI definition is considered equivalently ordered by implication.) Many criteria are possible on which to base the ordering of a set of KPI definitions including, for example, the most recently produced KPI value or the most recently indicated KPI state. 
     At block  4913 , the computing machine causes display of the requested service-monitoring page having a services summary region and a services aspects region. The service summary region contains an ordered plurality of interactive summary tiles. In one implementation, each summary tile corresponds to a respective service and provides a character or graphical representation of at least one value for an aggregate KPI characterizing the respective service as a whole. The services aspects region contains an ordered plurality of interactive aspect tiles. In one implementation, each aspect tile corresponds to a respective aspect KPI and provides a character or graphical representation of one or more values for the respective aspect KPI. Each aspect KPI can have an associated service and can typify performance for an aspect of the associated service. 
     The requested service-monitoring page can also include a notable events region presenting an indication of one or more correlation searches that generate the highest number of notable events in a given period of time. In one implementation, the notable events region includes the indication of each correlation search, a value representing the number of notable events generated in response to execution of each correlation search, and a graphical representation of the number of notable events generated over the given period of time. 
     In one implementation, the computing machine is a client device that causes display of the requested service-monitoring page by receiving a service monitoring web page or a service monitoring UI document from a server and rendering the service monitoring web page using a web browser on the client device or rendering the service monitoring UI document using a mobile application (app) on the client device. Alternatively, the computing machine is a server computer that causes display of the requested service-monitoring page by creating a service monitoring web page or a service monitoring UI document, and providing it to a client device for display via a web browser or a mobile application (app) on the client device. 
     In one implementation, creating a service monitoring web page or a service monitoring UI document includes determining the current and past values of the aggregate and aspect KPIs, determining the states of the aggregate and aspect KPIs, and identifying the most critical aggregate and aspect KPIs. In one implementation, various aspects (e.g., CPU usage, memory usage, response time, etc.) of a particular service can be monitored using a search query defined for an aspect KPI which is executed against raw machine data from entities that make up the service. The values from the raw machine data that are returned as a result of the defined search query represent the values of the aspect KPI. An aggregate KPI can be conFig.d and calculated for a service to represent an overall summary of a service. (The overall summary of a service, in an embodiment, can convey the health of the service, i.e., its sufficiency for meeting, or satisfaction of, operational objectives.) In one example, a service can have multiple separate aspect KPIs. The separate aspect KPIs for a service can be combined (e.g., averaged, weighted averaged, etc.) to create an aggregate KPI whose value is representative of the overall performance of the service. In one implementation, various thresholds can be defined for either aggregate KPIs or aspect KPIs. The defined thresholds correspond to ranges of values that represent the various states of the service. The values of the aggregate KPIs and/or aspect KPIs can be compared to the corresponding thresholds to determine the state of the aggregate or aspect KPI. The various states have an ordered severity that can be used to determine which KPIs should be displayed in service-monitoring page. In one implementation, the states include “critical,” “high,” “medium,” “normal,” and “low,” in order from most severe to least severe. In one implementation, some number of aggregate and aspect KPIs that have the highest severity level according to their determined state can be displayed in the corresponding region of the service-monitoring page. Additional details of thresholding, state determination and severity are described above with respect to  FIGS.  31 A-G . 
     At block  4914 , the computing machine performs monitoring related to the homepage. Such monitoring can include receiving notification of an operating system event such as a timer pop, or receiving notification of a GUI event such as a user input. Blocks  4915  through  4917  each signify a determination as to whether a particular monitored event has occurred and the processing that should result if it has. In one embodiment, each of blocks  4915 - 4917  can be associated with the execution of an event handler. At block  4915 , a determination is made whether notification has been received indicating that dynamic update or refresh of the homepage should occur. The notification can ensue from a user clicking a refresh button of the GUI, or from the expiration of a refresh interval timer established for the homepage, for example. If so, processing returns to block  4912  in one embodiment. At block  4916 , a determination is made whether notification has been received indicating that a display mode for the homepage should be changed. The notification can ensue from a user clicking a display mode button of the GUI, such as one selecting a network operations center display mode over a standard display mode, for example. If so, processing returns to block  4913  where the homepage is caused to be displayed in accordance, presumably, with the user input. At block  4917 , a determination is made whether notification has been received indicating some other user interaction or input. If so, processing proceeds to block  4918  where an appropriate response to the user input is executed. 
       FIG.  43 C  illustrates an example of a service-monitoring page  4920 , in accordance with one or more implementations of the present disclosure. In one implementation, service-monitoring page  4920  includes services summary region  4921  and services aspects region  4924 . Each of services summary region  4921  and services aspects region  4924  present dynamic visual representations including character and/or graphical indications of the states of various components in the system, including respective services in the system, as shown in services summary region  4921 , and individual aspect KPIs associated with one or more of the services, as shown in services aspects region  4924 . The information provided on service-monitoring page  4920  can be dynamically updated over time, so as to provide the user with the most recent available information. In one implementation, the visual representations on service-monitoring page  4920  are updated each time the underlying aggregate KPIs and aspect KPIs are recalculated according to the defined schedule in the corresponding KPI definition. In another implementation, the visual representations can be automatically updated in response to a specific user request, when the aggregate KPIs and aspect KPIs can be recalculated outside of their normal schedules specifically for the purpose of updating service-monitoring page  4920 . In yet another implementation, the visual representations can be static such that they do not change once initially displayed. The aggregate KPIs and aspect KPIs can be determined in response to the initial user request to view the service-monitoring page  4920 , and then displayed and refreshed at predefined time intervals or in real time once new values are calculated based on KPI monitoring parameters discussed above. Alternatively, the aggregate KPIs and aspect KPIs can be displayed, but not updated until a subsequent request to view the service-monitoring page  4920  is received. 
     In one implementation, the visual representations in services summary region  4921  contain an ordered plurality of interactive summary tiles  4922 . Each of interactive summary tiles  4922  corresponds to a respective service in the system (e.g., Activesync, Outlook, Outlook RPC) and provides a character or graphical representation of at least one value for an aggregate KPI characterizing the respective service as a whole. In one implementation, each of interactive summary tiles  4922  includes an indication of the corresponding service (i.e., the name or other identifier of the service), a numerical value indicating the aggregate KPI, and a sparkline indicating how the value of the aggregate KPI has changed over time. In one implementation, each of interactive summary tiles  4922  has a background color indicative of the state of the service. The state of the service can be determined by comparing the aggregate KPI of the service to one or more defined thresholds, as described above. In addition, each of interactive summary tiles  4922  can include a numerical value representing the state of the aggregate KPI characterizing the service and/or a textual indication of the state of the aggregate KPI (e.g., the name of the current state). In one implementation, only a certain number of interactive summary tiles  4922  can be displayed in services summary region  4921  at one time. For example, some number (e.g., 15, 20, etc.) of the most critical services, as measured by the severity of the states of their aggregate KPIs, can be displayed. In another implementation, tiles for user selected services can be displayed (i.e., the most important services to the user). In one implementation, which services are displayed, as well as the number of services displayed can be conFig.d by the user through menu element  4927 . 
     The interactive summary tiles  4922  of service monitoring page  4920  are depicted as rectangular tiles arranged in an orthogonal array within a region, without appreciable interstices. Another implementation can include tiles that are not rectangular, or arranged in a pattern that is not an orthogonal array, or that has interstitial spaces (grout) between tiles, or some combination. Another implementation can include tiles having no background color such that a tile has no clearly visible delineated shape or boundary. Another implementation can include tiles of more than one size. These and other implementations are possible. 
     In one implementation, services summary region  4921  further includes a health bar gage  4923 . The health bar gage  4923  can indicate distribution of aggregate KPIs of all services across each of the various states, rather than just the most critical services. The length of a portion of the health bar gage  4923 , which is colored according to a specific KPI state, depends on the number of services with aggregate KPIs in that state. In addition, the health bar gage  4923  can have numeric indications of the number of services with KPIs in each state, as well as the total number of services in the system being monitored. 
     In one implementation, the visual representations in services aspects region  4924  contain an ordered plurality of interactive aspect tiles  4925 . Each of interactive aspect tiles  4925  corresponds to a respective aspect KPI and provides a character or graphical representation of one or more values for the respective aspect KPI. Each aspect KPI can have an associated service and can typify performance for an aspect of the associated service. In one implementation, each of interactive aspect tiles  4925  includes an indication of the corresponding aspect KPI (i.e., the name or other identifier of the aspect KPI), an indication of the service with which the aspect KPI is associated, a numerical value indicating the current value of the aspect KPI, and a sparkline indicating how the value of the aspect KPI has changed over time. In one implementation, each of interactive aspect tiles  4925  has a background color indicative of the state of the aspect KPI. The state of the aspect KPI can be determined by comparing the aspect KPI to one or more defined thresholds, as described above. In addition, each of interactive aspect tiles  4925  can include a numerical value representing the state of the aspect KPI and/or a textual indication of the state of the aspect KPI (e.g., the name of the current state). In one implementation, only a certain number of interactive aspect tiles  4925  can be displayed in services aspects region  4924  at one time. For example, some number (e.g., 15, 20, etc.) of the most critical aspect KPIs, as measured by the severity of the states of the KPIs, can be displayed. In another implementation, tiles for user selected aspect KPIs can be displayed (i.e., the most important KPIs to the user). In one implementation, which aspect KPIs are displayed, as well as the number of aspect KPIs displayed can be conFig.d by the user through menu element  4928 . 
     In one implementation, services aspects region  4924  further includes an aspects bar gage  4926 . The aspects bar gage  4926  can indicate the distribution of all aspect KPIs across each of the various states, rather than just the most critical KPIs. The length of a portion of the aspects bar gage  4926  that is colored according to a specific state depends on the number of aspect KPIs in that state. In addition, the aspects bar gage  4926  can have numeric indications of the number of aspect KPIs in each state, as well as the total number of aspect KPIs in the system being monitored. 
     The tiles of a region (e.g.,  4922  of  4921 ,  4925  of  4924 ) each occupy an ordered position within the region. In one embodiment, the order of region tiles proceeds from left-to-right then top-to-bottom, with the first tile located in the leftmost, topmost position. In one embodiment, the order of region tiles proceeds from top-to-bottom then left-to-right. In one embodiment, the order of region tiles proceeds from right-to-left then top-to-bottom. In one embodiment, different regions can have different ordering arrangements. Other ordering is possible. A direct use of the ordered positions of tiles within a region is for making the association between a particular KPI definition and the particular tile for displaying information related to it. For example, a set of aspect KPI definitions with a determined order such as discussed in relation to block  4912  of  FIG.  43 B  can be mapped in order to the successively ordered tiles ( 4925 ) of an aspects region ( 4924 ). 
     In one embodiment service-monitoring page  4920  includes a display mode selection GUI element  4929  enabling a user to indicate a selection of a display mode. In one embodiment, display mode selection element  4929  enables the user to select between a network operations center (NOC) display mode and a home display mode. In one embodiment, tiles displaying KPI-related information while in NOC mode are larger (occupy more relative display area) than corresponding tiles displayed while in home mode. In an embodiment, display area is acquired to accommodate the larger tiles by a combination of one or more of reducing the total tile count, reducing or eliminating interstitial space between tiles or between displayed elements of the GUI, generally, reducing or eliminating GUI elements (such as any auxiliary regions area), or other methods. The transformation of the GUI display from home to NOC mode changes the size of tiles relative to one or more other GUI elements and, so, is not a simple zoom function applied to the service-monitoring page  4920 . In one embodiment, an indicator within a tile displaying KPI-related information while in NOC mode is larger (occupies more relative display area) than the corresponding indicator displayed while in home mode. For example, a character-type indicator within a tile can display using a larger or bolder font while in NOC mode than while in home mode. In one embodiment, display area is acquired to accommodate the larger indicator by a combination of reducing or eliminating other indicators appearing within the tile. Embodiments with more than two display mode selection options, such as associated with GUI element  4929 , are possible. 
       FIG.  43 D  illustrates an example of a service-monitoring page  4920  including a notable events region  4930 , in accordance with one or more implementations of the present disclosure. Depending on the implementation, notable events region  4930  can be displayed adjacent to, beneath, above or between services summary region  4921  and services aspects region  4924 . In another implementation, notable events region  4930  can be displayed on a different page or in a different interface than services summary region  4921  and services aspects region  4924 . In one implementation, notable events region  4930  contains an indication (such as a list) of one or more correlation searches (also referred to herein as “rules”) that generate the highest number of notable events in a given period of time. A notable event can be triggered by a correlation search associated with a service. As discussed above, a correlation search can include search criteria pertaining to one or more KPIs (e.g., an aggregate KPI or one or more aspect KPIs) of the service, and a triggering condition to be applied to data produced by a search query using the search criteria. A notable event is generated when the data produced by the search query satisfies the triggering condition. A correlation search can be pre-defined and provided by the system or can be newly created by an analyst or other user of the system. In one implementation, the correlation searches can be run continuously or at regular intervals (e.g., every hour) to generate notable events. Generated notable events can be stored in a dedicated “notable events index,” which can be subsequently accessed to create various visualizations, including notable events region  4930  of service-monitoring page  4920 . 
     In one implementation, the notable events region  4930  includes the indication (e.g., the name) of each correlation search  4931 , a value representing the number of notable events generated in response to execution of each correlation search  4932 , and a graphical representation (e.g., a sparkline) of the number of notable events generated over the given period of time  4933 . In one implementation, the correlation searches shown in notable events region  4930  can be sorted according to the data in each of columns  4931 ,  4932 , and  4933 . 
     In one implementation, only a certain number of correlation searches can be displayed in notable events region  4930  at one time. For example, some number (e.g., 5, 10, etc.) of the correlation searches that generate the most notable events in a given period of time can be displayed. In another implementation, all correlation searches that generated a minimum number of notable events in a given period of time can be displayed. In one implementation, which correlation searches are displayed, as well as the number of correlation searches displayed can be conFig.d by the user. 
     In an embodiment, notable events region  4930  can be replaced by, or supplemented with, one or more other information regions. For example, one embodiment of an other-information region can display most-recently-used items, such as most-recently-viewed service-monitoring dashboards, or most-recently-used deep dive displays. Each most-recently-used item can contain the item name or some other identifier for the item. Any notable event regions and other information regions in a GUI display can be collectively referred to as auxiliary regions. In one embodiment, items displayed in auxiliary regions support user interaction. User interaction may, for example, provide an indication to the computing machine of a user&#39;s desire to navigate to a GUI component related to the item with which the user interacted. For example, a user can click on a notable event name in the notable event region to navigate to a GUI displaying detailed information about the event. For example, a user can click on the name of a most-recently-viewed service-monitoring dashboard in an other-information region to navigate to the dashboard GUI. In one embodiment, auxiliary regions are displayed together in an auxiliary regions area. An auxiliary regions area can be located in a GUI display as described above for the notable events region  4930 . 
       FIGS.  43 E-F  illustrate an example of a service-monitoring page  4920 , in accordance with one or more implementations of the present disclosure. As shown in  FIG.  43 E , a particular tile  4940  of the plurality of interactive aspect tiles  4925  in services aspects region  4924  has been activated. The user can activate tile  4940 , for example, by hovering a cursor over the tile  4940  or tapping the tile  4940  on a touchscreen. Once the tile  4940  is activated, a selectable graphical element  4941 , such as a check box, radio button, etc., can be displayed for the chosen tile  4940 . Further user interaction with the selectable graphical element  4941 , such as a mouse click or additional tap, can activate the selectable graphical element  4941  and cause the corresponding tile  4940  to be selected for further viewing. Upon selection of tile  4940 , a similar selectable graphical element can be displayed for each of interactive aspects tiles  4925  in services aspects region  4924 , as shown in  FIG.  43 F . In one implementation, additional white space can be displayed between each of interactive aspect tiles  4925 . If the user desires, they can select one or more additional tiles by similarly interacting with the corresponding selectable graphical element of any of the other interactive aspect tiles  4925 . In one implementation, the selected tiles can have the selectable graphical element highlighted, or otherwise emphasized, to indicate that the corresponding tile has been selected. In addition, the appearance (e.g., color, shading, etc.) of the selected titles can change to further emphasize that they have been selected. 
     In response to one or more of interactive aspect tiles  4925  being selected, menu elements  4942  and  4943  can be displayed in service-monitoring page  4920 . Menu element  4942  can be used to cancel the selection of any interactive aspects tiles  4925  in services aspects region  4924 . Activation of menu element  4942  can cause the selected tiles to be unselected and revert to the non-selected state as shown in  FIG.  43 C . Menu element  4943  can be used to view the selected aspect KPIs in a deep dive visual interface, which includes detailed information for the one or more selected aspect KPIs. The deep dive visual interface displays time-based graphical visualizations corresponding to the selected aspect KPIs to allow a user to visually correlate the aspect KPIs over a defined period of time. 
     As noted herein above, implementations of the present disclosure allow applying user-configurable policies for identifying non-responsive or orphan entities of a service monitoring system and provide users with the ability to identify, retire, and eventually delete these entities. Retired entities are excluded from interacting with any other components of the service monitoring system. A retired entity can stay in the retired state until it is deleted from the system or restored to the active state. 
       FIG.  44    schematically illustrates an example workflow  4400  of entity lifecycle management in a service monitoring system, in accordance with aspects of the present disclosure. As schematically illustrated by  FIG.  44   , the service monitoring system can process events and metrics data  4410  stored in the event repository  4402  and the metrics repository  4404 . In an illustrative example, the service monitoring system can perform various saved searches  4420  on the events and metrics data  4410 . The saved searches  4420  can produce service intelligence summary  4430  and service intelligence metric summary  4440 . The saved searches  4420  include scheduled search stage  4412 , entity filtering stage  4414 , data aggregation stage  4416 , set severity stage  4417 , and alert stage  4418 . The entity filtering stage  4414  filters the KPI data to the specific entities according to the rules associated with the service. The data aggregation stage  4416  can be implemented by a subsearch appended to the scheduled KPI search as a result of entity filtering. Set severity stage  4417  can be implemented by a custom search command to add the severity attributes to the KPI data on a scheduled execution basis. The alert stage  4418  can trigger an alert action to specify the injection of KPI data into the event and metric index. The entity filter rule collection  4470  maps the association of an entity filter search to the specific entities. 
     The entity lifecycle management is driven by one or more user-defined entity lifecycle management policies  4450 , which can be created via a GUI based on predefined templates, as described in more detail herein below. The user-defined entity lifecycle management policies  4450  can be translated into respective custom search commands  4460 , which can be applied to the IT Service Intelligence (ITSI) summary (event data)  4430  and to the metrics data  4440  produced by the service monitoring system. 
     Each of the entity lifecycle management policies  4450  can be associated with a user-specified symbolic name and can specify one or more entity lifecycle management action (e.g., entity retirement) rules. In an illustrative example, an entity retirement rule can target non-responsive entities by specifying the threshold period of time in which an entity has not been sending any data. Accordingly, an entity that has not been sending any data for a period of time exceeding the threshold period specified by the entity retirement rule would be identified as a candidate entity for retirement. In another illustrative example, an entity retirement rule can target orphan entities by specifying the threshold period of time in which the entity has not been associated with any service. Accordingly, an entity that has not been associated with any service for a period exceeding the threshold period of time specified by the entity retirement rule would be identified as a candidate entity for retirement. 
     In some implementations, in addition to entity lifecycle management action rules, an entity lifecycle management policy can specify the entity evaluation frequency, which specifies the frequency of running the search commands produced by translating the entity lifecycle management policy. 
     In some implementations, the service monitoring system can provide a GUI for creating and/or editing entity lifecycle management policy definitions, as schematically illustrated by  FIGS.  45 A- 45 B . In the illustrative example of  FIG.  45 A , the GUI  4502  can allow the user to specify the symbolic name  4510  for the entity lifecycle management policy. The GUI can further allow the user to specify the entity lifecycle management rule  4520 . The rule type can be selected from the drop-down list  4530 . In the illustrative example of  FIG.  45 A , the entity lifecycle management action rule targets non-responsive entities, i.e., entities that have not been sending any data for at least the user-specified time period  4540 . In the illustrative example of  FIG.  45 B , the entity lifecycle management action rule targets orphan entities, i.e., entities that have not been associated with any service for at least the specified time period  4540 . The GUI can further allow the user to select the entity evaluation frequency, which specifies the frequency of running the search commands produced by translating the entity lifecycle management policy. 
     In some implementations, the service monitoring system can provide a GUI for displaying the defined entity lifecycle management policies and performing bulk actions upon user-selected policies, as schematically illustrated by  FIG.  46   . In the illustrative example of  FIG.  46 A , the GUI  4602  can include a list of defined entity lifecycle management policies  4610 A- 4610 N. 
     In some implementations, the GUI  4602  can provide, for each entity lifecycle management policy  4670 A- 4670 N, a corresponding toggle switch  4680  for enabling/disabling the entity lifecycle management policy  4670 A- 4670 N. Disabling the entity lifecycle management policy can remove retirement candidate designations from the one or more candidate retirement entities that had been identified by the policy before the policy was disabled. Similarly, permanently deleting the entity lifecycle management policy can remove retirement candidate designations from the one or more candidate retirement entities that had been identified by the policy before the policy was deleted. 
     In some implementations, the GUI  4602  can further include, for each entity lifecycle management policy  4670 A- 4670 N, a corresponding last evaluation date  4672 A- 4672 N and/or a corresponding policy evaluation frequency  4672 A- 4674 N. 
     In some implementations, the GUI  4602  can further provide, for each entity lifecycle management policy  4670 A- 4670 N, a corresponding checkbox  4676 A- 4676 N for selecting the policies for performing a bulk action that is selected by the drop-down list  4678 . The chosen bulk action is then applied to the entity lifecycle management policies  4670  that are selected by their corresponding checkboxes  4676 . As schematically illustrated by  FIG.  47   , the bulk actions  4710  can include enabling the selected entity lifecycle management policies, disabling the selected entity lifecycle management policies, evaluating the selected entity lifecycle management policies, editing the selected entity lifecycle management policies, and deleting the selected entity lifecycle management policies. 
     In some implementations, the service monitoring system allows a user to specify multiple non-conflicting entity lifecycle management rules, each of which would be translated into corresponding search commands, which are then applied to the entity definitions and to the events/metrics data produced by the service monitoring system. Translating an entity lifecycle management rules into search commands can involve selecting a template corresponding to the entity lifecycle management rule type, and substituting the template fields with the values supplied by the rule (e.g., the threshold period of time for identifying non-responsive or orphan entities). 
     In some implementations, the service monitoring system allows a user to associate entity lifecycle management policies with specific entity types, such that a set of entity lifecycle management rules to be applied to a given entity is defined based on the entity type. By default, a newly defined entity lifecycle management rule can be applied to all entity types. The entity lifecycle management policy definition GUI can allow the user to select specific entity types to be associated with that policy. 
     As noted herein above, the search commands derived from the entity lifecycle policy definitions can be executed with the desired frequency, thus identifying certain entities as retirement candidates. These entities can be presented to a user via a GUI that lists all identified retirement candidate entities and allows the user to perform individual or bulk actions upon selected entities, as schematically illustrated by  FIG.  48   . In the illustrative example of  FIG.  48   , the GUI  4802  can include a list of defined entities  4810 . In some implementations, the GUI  4802  can further display, for each entity  4810 , its alias  4812 , associated services  4814 , and entity type  4816 . 
     In some implementations, the list of entities displayed by the GUI  4802  can display one or more entities that have been selected, by applicable entity lifecycle management policy, as candidate entities for performing entity lifecycle actions (e.g., retirement). In the list of entities, such entity can have an attention icon displayed  4820  in a visual association to the entity name. If a cursor is hovered over a displayed attention icon, the entity lifecycle management policy that updated the entity is shown in a popup bubble  4830 . 
     As schematically illustrated by  FIG.  48   , the GUI  4802  can further provide, for each entity  4810 , a list of available actions  4840  that can be performed with respect to the entity. The actions  4840  can include retiring or deleting the entity. In some implementations, the actions  4840  can further include editing the entity and/or putting the entity into the maintenance mode. 
     In some implementations, the GUI  4802  can further provide, for each entity  4810 , a corresponding checkbox  4850  for selecting the entity for performing a bulk action that is selected by the drop-down list  4840 . The chosen bulk action is then applied to the entities  4810  that are selected by their corresponding checkboxes  4850 . The bulk actions  4840  can include retiring or deleting the selected entities. In some implementations, the bulk actions  4840  can further include editing the selected entities and/or putting the selected entities into the maintenance mode. 
     In some implementations, the service monitoring system allows a user to restore a retired entity to the active state or delete a retired entity from the system, as schematically illustrated by  FIG.  49   . The retired entities can be presented to a user via a GUI  4902  that lists all retired entities and allows the user to perform individual or bulk actions upon selected retired entities. In the illustrative example of  FIG.  49   , the GUI  4902  can display a list of retired entities  4970 . In some implementations, the GUI  4902  can further display, for each entity  4970 , its alias  4972 , associated services  4974 , and entity type  4976 . 
     For each entity  4970 , the GUI  4902  can provide a list of available actions  4978  that can be performed with respect to the entity. The actions  4978  can include restoring the entity to the active status (un-retiring), editing, or deleting the entity. 
     In some implementations, the GUI  4902  can further provide, for each entity  4970 , a corresponding checkbox  4980  for selecting the entity for performing a bulk action that is selected by the drop-down list that is triggered by the button  4982 . The chosen bulk action is then applied to the entities  4970  that are selected by their corresponding checkboxes  4980 . The bulk actions can include restoring the entity to the active status (un-retiring), editing, or deleting the entity. 
     In some implementations, the service monitoring system can further implement restoring, to the active status, the retired entities that have become active and/or associated with at least one service. In an illustrative example, a saved search can identify retirement candidate entities that started sending data since the retirement event. In another illustrative example, a saved search can identify retirement candidate entities that have become associated with at least one service since the retirement event. The identified entities can then be transitioned into an active status automatically or be presented to a user via a GUI for review and selection of entities to be restored to the active status. 
       FIG.  50    is a flow diagram of an example method  5100  of entity lifecycle management by a service monitoring system operating in accordance with aspects of the present disclosure. Method  5100  and/or each of its individual functions, routines, subroutines, or operations can be performed by one or more processors of the service monitoring system  210  of  FIG.  2    and/or other components of the service monitoring system  210  of  FIG.  2   . In certain implementations, method  5100  can be performed by a single processing thread. Alternatively, method  5100  can be performed by two or more processing threads, each thread executing one or more individual functions, routines, subroutines, or operations of the method. In an illustrative example, the processing threads implementing method  5100  can be synchronized (e.g., using semaphores, critical sections, and/or other thread synchronization mechanisms). Alternatively, the processing threads implementing method  5100  can be executed asynchronously with respect to each other. Therefore, while  FIG.  50    and the associated description lists the operations of method  5100  in certain order, various implementations of the method can perform at least some of the described operations in parallel and/or in arbitrary selected orders. While example method  5100  is described as being implemented by the service monitoring system  210  of  FIG.  2   , method  5100  can also be implemented by other software systems, platforms, or applications (e.g., by a software application for monitoring a network environment, by an application performance monitoring system, by a data intake and query system, by an event processing systems, etc.). 
     At block  5110 , the processing device implementing the method receives one or more policy definitions specifying respective one or more entity lifecycle management policies. In an illustrative examples, the policy definitions can be created by a policy creation/editing GUI based on a set of predefined templates. Each policy lifecycle management policy can include one or more entity lifecycle management rules. In an illustrative example, an entity retirement rule can target non-responsive entities by specifying the threshold period of time in which an entity has not been sending any data. In another illustrative example, an entity retirement rule can target orphan entities by specifying the threshold period of time in which the entity has not been associated with any service, as described in more detail herein above. 
     At block  5120 , the processing device applies one or more active entity lifecycle management policies to the active entities defined in the service monitoring system in order to identify one or more candidate entities for retirement. In various illustrative examples, entities can include devices, applications, services, and/or user accounts. In some implementations, applying an entity lifecycle management policy involves translating the policy into a set of search commands, which are then executed based on the schedule associated with the entity lifecycle management policy, as described in more detail herein above. 
     At block  5130 , the processing device identifies, as retired entities, at least a subset of the one or more candidate entities. In an illustrative example, the processing device can cause a GUI to be displayed, such that the GUI lists the identified retirement candidate entities. The processing device can then receive the user selection of one or more candidate entities for retirement, as described in more detail herein above. 
     At block  5140 , the processing device excludes the retired entities from the active entities defined in the system. In an illustrative example, each entity definition includes a status flag, which can be set to active or retired status. In some implementations, the status flag can support three states: active, retirement candidate, or retired. Setting the status flag to the retired status would effectively exclude the affected entity from interacting with any other entities or components of the service monitoring system. In some implementations, setting the status flag to the retired status would also suppress generation of alerts associated with the affected entity, as described in more detail herein above. In some implementations, two separate flags can be supported for each entity definition, such that one flag would correspond to the candidate retirement state, while the other flag would correspond to the retired state. 
     At block  5150 , the processing device executes one or more search queries that derive, from machine data associated with one or more active entities, values of one or more key performance indicators (KPIs) reflecting respective aspects of performance of one or more services, as described in more detail herein above. 
       FIG.  51    is a flow diagram of an example method  5200  of managing retired entities by a service monitoring system operating in accordance with aspects of the present disclosure. Method  5200  and/or each of its individual functions, routines, subroutines, or operations can be performed by one or more processors of the service monitoring system  210  of  FIG.  2    and/or other components of the service monitoring system  210  of  FIG.  2   . In certain implementations, method  5200  can be performed by a single processing thread. Alternatively, method  5200  can be performed by two or more processing threads, each thread executing one or more individual functions, routines, subroutines, or operations of the method. In an illustrative example, the processing threads implementing method  5200  can be synchronized (e.g., using semaphores, critical sections, and/or other thread synchronization mechanisms). Alternatively, the processing threads implementing method  5200  can be executed asynchronously with respect to each other. Therefore, while  FIG.  51    and the associated description lists the operations of method  5200  in certain order, various implementations of the method can perform at least some of the described operations in parallel and/or in arbitrary selected orders. While example method  5200  is described as being implemented by the service monitoring system  210  of  FIG.  2   , method  5200  can also be implemented by other software systems, platforms, or applications (e.g., by a software application for monitoring a network environment, by an application performance monitoring system, by a data intake and query system, by an event processing systems, etc.). 
     At block  5210 , the processing device implementing the method identifies a plurality of retired entities. In an illustrative example, the processing device can cause a GUI to be displayed, such that the GUI lists the identified retirement candidate entities. The processing device can then receive the user selection of one or more candidate entities for retirement, as described in more detail herein above. 
     At block  5220 , the processing device restores one or more retired entities to the active status. In an illustrative example, the processing device can cause a GUI to be displayed, such that the GUI lists the retired entities. The processing device can then receive the user selection of one or more retired entities to be restored to the active status, as described in more detail herein above. 
     At block  5230 , the processing device permanently deletes one or more retired entities. In an illustrative example, the processing device can cause a GUI to be displayed, such that the GUI lists the retired entities. The processing device can then receive the user selection of one or more retired entities to be permanently deleted from the system, as described in more detail herein above. 
       FIG.  52    is a flow diagram of an example method  5300  of managing retired entities by a service monitoring system operating in accordance with aspects of the present disclosure. Method  5300  and/or each of its individual functions, routines, subroutines, or operations can be performed by one or more processors of the service monitoring system  210  of  FIG.  2    and/or other components of the service monitoring system  210  of  FIG.  2   . In certain implementations, method  5300  can be performed by a single processing thread. Alternatively, method  5300  can be performed by two or more processing threads, each thread executing one or more individual functions, routines, subroutines, or operations of the method. In an illustrative example, the processing threads implementing method  5300  can be synchronized (e.g., using semaphores, critical sections, and/or other thread synchronization mechanisms). Alternatively, the processing threads implementing method  5300  can be executed asynchronously with respect to each other. Therefore, while  FIG.  52    and the associated description lists the operations of method  5300  in certain order, various implementations of the method can perform at least some of the described operations in parallel and/or in arbitrary selected orders. While example method  5300  is described as being implemented by the service monitoring system  210  of  FIG.  2   , method  5300  can also be implemented by other software systems, platforms, or applications (e.g., by a software application for monitoring a network environment, by an application performance monitoring system, by a data intake and query system, by an event processing systems, etc.). 
     At block  5310 , the processing device implementing the method receives one or more policy definitions specifying respective one or more entity lifecycle management policies, as described in more detail herein above. 
     At block  5320 , the processing device applies one or more active entity lifecycle management policies to the active entities defined in the service monitoring system in order to identify one or more candidate entities for retirement, as described in more detail herein above. 
     At block  5330 , the processing device disables at least one of the active entity lifecycle management policies. In an illustrative example, the processing device can cause a GUI to be displayed, such that the GUI lists the active entity lifecycle management policies. The processing device can then receive the user selection of one or more active entity lifecycle management policies to be disabled, as described in more detail herein above. 
     At block  5340 , the processing device removes retirement candidate designations from the one or more candidate retirement entities that had been previously identified by the disabled entity lifecycle management policies. 
       FIG.  53    is a flow diagram of an example method  5400  of translating an entity lifecycle management policy by a service monitoring system operating in accordance with aspects of the present disclosure. Method  5400  and/or each of its individual functions, routines, subroutines, or operations can be performed by one or more processors of the service monitoring system  210  of  FIG.  2    and/or other components of the service monitoring system  210  of  FIG.  2   . In certain implementations, method  5400  can be performed by a single processing thread. Alternatively, method  5400  can be performed by two or more processing threads, each thread executing one or more individual functions, routines, subroutines, or operations of the method. In an illustrative example, the processing threads implementing method  5400  can be synchronized (e.g., using semaphores, critical sections, and/or other thread synchronization mechanisms). Alternatively, the processing threads implementing method  5400  can be executed asynchronously with respect to each other. Therefore, while  FIG.  53    and the associated description lists the operations of method  5400  in certain order, various implementations of the method can perform at least some of the described operations in parallel and/or in arbitrary selected orders. While example method  5400  is described as being implemented by the service monitoring system  210  of  FIG.  2   , method  5400  can also be implemented by other software systems, platforms, or applications (e.g., application performance monitoring systems, data intake and query systems, event processing systems, etc.). 
     At block  5410 , the processing device implementing the method receives a policy definition specifying an entity lifecycle management policy. The policy may include one or more entity lifecycle management rules. Each rule may specify a logical condition to be satisfied by an active entity definition in order to qualify the active entity to the candidate retirement status. In an illustrative example, an entity retirement rule can target non-responsive entities by specifying the threshold period of time in which an entity has not been sending any data. In another illustrative example, an entity retirement rule can target orphan entities by specifying the threshold period of time in which the entity has not been associated with any service, as described in more detail herein above. 
     At block  5420 , the processing device identifies, for each entity lifecycle management rule, a template corresponding to the entity lifecycle management rule type. A template can include a sequence of statements in a chosen query language, in which certain intermediate values are replaced by fillable template fields. 
     At block  5430 , the processing device translates each entity lifecycle management rule, based on the identified corresponding template, to a set of search commands. In an illustrative example, the translation may involve substituting the template fields with the values supplied by the rule (e.g., the threshold period of time for identifying non-responsive or orphan entities). 
     Modern data centers often comprise thousands of host computer systems that operate collectively to service requests from even larger numbers of remote clients. During operation, these data centers generate significant volumes of performance data and diagnostic information that can be analyzed to quickly diagnose performance problems. In order to reduce the size of this performance data, the data is typically pre-processed prior to being stored based on anticipated data-analysis needs. For example, pre-specified data items can be extracted from the performance data and stored in a database to facilitate efficient retrieval and analysis at search time. However, the rest of the performance data is not saved and is essentially discarded during pre-processing. As storage capacity becomes progressively cheaper and more plentiful, there are fewer incentives to discard this performance data and many reasons to keep it. 
     This plentiful storage capacity is presently making it feasible to store massive quantities of minimally processed performance data at “ingestion time” for later retrieval and analysis at “search time.” Note that performing the analysis operations at search time provides greater flexibility because it enables an analyst to search all of the performance data, instead of searching pre-specified data items that were stored at ingestion time. This enables the analyst to investigate different implementations of the performance data instead of being confined to the pre-specified set of data items that were selected at ingestion time. 
     However, analyzing massive quantities of heterogeneous performance data at search time can be a challenging task. A data center can generate heterogeneous performance data from thousands of different components, which can collectively generate tremendous volumes of performance data that can be time-consuming to analyze. For example, this performance data can include data from system logs, network packet data, sensor data, and data generated by various applications. Also, the unstructured nature of much of this performance data can pose additional challenges because of the difficulty of applying semantic meaning to unstructured data, and the difficulty of indexing and querying unstructured data using traditional database systems. 
     These challenges can be addressed by using an event-based system, such as the SPLUNK® ENTERPRISE system produced by Splunk Inc. of San Francisco, Calif., to store and process performance data. The SPLUNK® ENTERPRISE system is the leading platform for providing real-time operational intelligence that enables organizations to collect, index, and harness machine-generated data from various websites, applications, servers, networks, and mobile devices that power their businesses. The SPLUNK® ENTERPRISE system is particularly useful for analyzing unstructured performance data, which is commonly found in system log files. Although many of the techniques described herein are explained with reference to the SPLUNK® ENTERPRISE system, the techniques are also applicable to other types of data server systems. 
     In the SPLUNK® ENTERPRISE system, performance data is stored as “events,” wherein each event comprises a collection of performance data and/or diagnostic information that is generated by a computer system and is correlated with a specific point in time. Events can be derived from “time series data,” wherein time series data comprises a sequence of data points (e.g., performance measurements from a computer system) that are associated with successive points in time and are typically spaced at uniform time intervals. Events can also be derived from “structured” or “unstructured” data. Structured data has a predefined format, wherein specific data items with specific data formats reside at predefined locations in the data. For example, structured data can include data items stored in fields in a database table. In contrast, unstructured data does not have a predefined format. This means that unstructured data can comprise various data items having different data types that can reside at different locations. For example, when the data source is an operating system log, an event can include one or more lines from the operating system log containing raw data that includes different types of performance and diagnostic information associated with a specific point in time. Examples of data sources from which an event can be derived include, but are not limited to: web servers; application servers; databases; firewalls; routers; operating systems; and software applications that execute on computer systems, mobile devices, and sensors. The data generated by such data sources can be produced in various forms including, for example and without limitation, server log files, activity log files, configuration files, messages, network packet data, performance measurements and sensor measurements. An event typically includes a timestamp that can be derived from the raw data in the event, or can be determined through interpolation between temporally proximate events having known timestamps. 
     The SPLUNK® ENTERPRISE system also facilitates using a flexible schema to specify how to extract information from the event data, wherein the flexible schema can be developed and redefined as needed. Note that a flexible schema can be applied to event data “on the fly,” when it is needed (e.g., at search time), rather than at ingestion time of the data as in traditional database systems. Because the schema is not applied to event data until it is needed (e.g., at search time), it is referred to as a “late-binding schema.” 
     During operation, the SPLUNK® ENTERPRISE system starts with raw data, which can include unstructured data, machine data, performance measurements or other time-series data, such as data obtained from weblogs, syslogs, or sensor readings. It divides this raw data into “portions,” and optionally transforms the data to produce timestamped events. The system stores the timestamped events in a data store, and enables a user to run queries against the data store to retrieve events that meet specified criteria, such as containing certain keywords or having specific values in defined fields. Note that the term “field” refers to a location in the event data containing a value for a specific data item. 
     As noted above, the SPLUNK® ENTERPRISE system facilitates using a late-binding schema while performing queries on events. A late-binding schema specifies “extraction rules” that are applied to data in the events to extract values for specific fields. More specifically, the extraction rules for a field can include one or more instructions that specify how to extract a value for the field from the event data. An extraction rule can generally include any type of instruction for extracting values from data in events. In some cases, an extraction rule comprises a regular expression, in which case the rule is referred to as a “regex rule.” 
     In contrast to a conventional schema for a database system, a late-binding schema is not defined at data ingestion time. Instead, the late-binding schema can be developed on an ongoing basis until the time a query is actually executed. This means that extraction rules for the fields in a query can be provided in the query itself, or can be located during execution of the query. Hence, as an analyst learns more about the data in the events, the analyst can continue to refine the late-binding schema by adding new fields, deleting fields, or changing the field extraction rules until the next time the schema is used by a query. Because the SPLUNK® ENTERPRISE system maintains the underlying raw data and provides a late-binding schema for searching the raw data, it enables an analyst to investigate questions that arise as the analyst learns more about the events. 
     In the SPLUNK® ENTERPRISE system, a field extractor can be conFig.d to automatically generate extraction rules for certain fields in the events when the events are being created, indexed, or stored, or possibly at a later time. Alternatively, a user can manually define extraction rules for fields using a variety of techniques. 
     Also, a number of “default fields” that specify metadata about the events rather than data in the events themselves can be created automatically. For example, such default fields can specify: a timestamp for the event data; a host from which the event data originated; a source of the event data; and a source type for the event data. These default fields can be determined automatically when the events are created, indexed or stored. 
     In some embodiments, a common field name can be used to reference two or more fields containing equivalent data items, even though the fields can be associated with different types of events that possibly have different data formats and different extraction rules. By enabling a common field name to be used to identify equivalent fields from different types of events generated by different data sources, the system facilitates use of a “common information model” (CIM) across the different data sources. 
       FIG.  54    presents a block diagram of an exemplary event processing system  7100 , similar to the SPLUNK® ENTERPRISE system. System  7100  includes one or more forwarders  7101  that collect data obtained from a variety of different data sources  7105 , and one or more indexers  7102  that store, process, and/or perform operations on this data, wherein each indexer operates on data contained in a specific data store  7103 . These forwarders and indexers can comprise separate computer systems in a data center, or can alternatively comprise separate processes executing on various computer systems in a data center. 
     During operation, the forwarders  7101  identify which indexers  7102  will receive the collected data and then forward the data to the identified indexers. Forwarders  7101  can also perform operations to strip out extraneous data and detect timestamps in the data. The forwarders next determine which indexers  7102  will receive each data item and then forward the data items to the determined indexers  7102 . 
     Note that distributing data across different indexers facilitates parallel processing. This parallel processing can take place at data ingestion time, because multiple indexers can process the incoming data in parallel. The parallel processing can also take place at search time, because multiple indexers can search through the data in parallel. 
     System  7100  and the processes described below with respect to  FIGS.  55 - 56    are further described in “Exploring Splunk Search Processing Language (SPL) Primer and Cookbook” by David Carasso, CITO Research, 2012, and in “Optimizing Data Analysis With a Semi-Structured Time Series Database” by Ledion Bitincka, Archana Ganapathi, Stephen Sorkin, and Steve Zhang, SLAML, 2010, each of which is hereby incorporated herein by reference in its entirety for all purposes. 
       FIG.  55    presents a flowchart illustrating how an indexer processes, indexes, and stores data received from forwarders in accordance with the disclosed embodiments. At block  7201 , the indexer receives the data from the forwarder. Next, at block  7202 , the indexer apportions the data into events. Note that the data can include lines of text that are separated by carriage returns or line breaks and an event can include one or more of these lines. During the apportioning process, the indexer can use heuristic rules to automatically determine the boundaries of the events, which for example coincide with line boundaries. These heuristic rules can be determined based on the source of the data, wherein the indexer can be explicitly informed about the source of the data or can infer the source of the data by examining the data. These heuristic rules can include regular expression-based rules or delimiter-based rules for determining event boundaries, wherein the event boundaries can be indicated by predefined characters or character strings. These predefined characters can include punctuation marks or other special characters including, for example, carriage returns, tabs, spaces or line breaks. In some cases, a user can fine-tune or conFig. the rules that the indexers use to determine event boundaries in order to adapt the rules to the user&#39;s specific requirements. 
     Next, the indexer determines a timestamp for each event at block  7203 . As mentioned above, these timestamps can be determined by extracting the time directly from data in the event, or by interpolating the time based on timestamps from temporally proximate events. In some cases, a timestamp can be determined based on the time the data was received or generated. The indexer subsequently associates the determined timestamp with each event at block  7204 , for example by storing the timestamp as metadata for each event. 
     Then, the system can apply transformations to data to be included in events at block  7205 . For log data, such transformations can include removing a portion of an event (e.g., a portion used to define event boundaries, extraneous text, characters, etc.) or removing redundant portions of an event. Note that a user can specify portions to be removed using a regular expression or any other possible technique. 
     Next, a keyword index can optionally be generated to facilitate fast keyword searching for events. To build a keyword index, the indexer first identifies a set of keywords in block  7206 . Then, at block  7207  the indexer includes the identified keywords in an index, which associates each stored keyword with references to events containing that keyword (or to locations within events where that keyword is located). When an indexer subsequently receives a keyword-based query, the indexer can access the keyword index to quickly identify events containing the keyword. 
     In some embodiments, the keyword index can include entries for name-value pairs found in events, wherein a name-value pair can include a pair of keywords connected by a symbol, such as an equals sign or colon. In this way, events containing these name-value pairs can be quickly located. In some embodiments, fields can automatically be generated for some or all of the name-value pairs at the time of indexing. For example, if the string “dest=10.0.1.2” is found in an event, a field named “dest” can be created for the event, and assigned a value of “10.0.1.2.” 
     Finally, the indexer stores the events in a data store at block  7208 , wherein a timestamp can be stored with each event to facilitate searching for events based on a time range. In some cases, the stored events are organized into a plurality of buckets, wherein each bucket stores events associated with a specific time range. This not only improves time-based searches, but it also allows events with recent timestamps that can have a higher likelihood of being accessed to be stored in faster memory to facilitate faster retrieval. For example, a bucket containing the most recent events can be stored as flash memory instead of on hard disk. 
     Each indexer  7102  is responsible for storing and searching a subset of the events contained in a corresponding data store  7103 . By distributing events among the indexers and data stores, the indexers can analyze events for a query in parallel, for example using map-reduce techniques, wherein each indexer returns partial responses for a subset of events to a search head that combines the results to produce an answer for the query. By storing events in buckets for specific time ranges, an indexer can further optimize searching by looking only in buckets for time ranges that are relevant to a query. 
       FIG.  56    presents a flowchart illustrating how a search head and indexers perform a search query in accordance with the disclosed embodiments. At the start of this process, a search head receives a search query from a client at block  7301 . Next, at block  7302 , the search head analyzes the search query to determine what portions can be delegated to indexers and what portions need to be executed locally by the search head. At block  7303 , the search head distributes the determined portions of the query to the indexers. Note that commands that operate on single events can be trivially delegated to the indexers, while commands that involve events from multiple indexers are harder to delegate. 
     Then, at block  7304 , the indexers to which the query was distributed search their data stores for events that are responsive to the query. To determine which events are responsive to the query, the indexer searches for events that match the criteria specified in the query. This criteria can include matching keywords or specific values for certain fields. In a query that uses a late-binding schema, the searching operations in block  7304  can involve using the late-binding scheme to extract values for specified fields from events at the time the query is processed. Next, the indexers can either send the relevant events back to the search head, or use the events to calculate a partial result, and send the partial result back to the search head. 
     Finally, at block  7305 , the search head combines the partial results and/or events received from the indexers to produce a final result for the query. This final result can comprise different types of data depending upon what the query is asking for. For example, the final results can include a listing of matching events returned by the query, or some type of visualization of data from the returned events. In another example, the final result can include one or more calculated values derived from the matching events. 
     Moreover, the results generated by system  7100  can be returned to a client using different techniques. For example, one technique streams results back to a client in real-time as they are identified. Another technique waits to report results to the client until a complete set of results is ready to return to the client. Yet another technique streams interim results back to the client in real-time until a complete set of results is ready, and then returns the complete set of results to the client. In another technique, certain results are stored as “search jobs,” and the client can subsequently retrieve the results by referencing the search jobs. 
     The search head can also perform various operations to make the search more efficient. For example, before the search head starts executing a query, the search head can determine a time range for the query and a set of common keywords that all matching events must include. Next, the search head can use these parameters to query the indexers to obtain a superset of the eventual results. Then, during a filtering stage, the search head can perform field-extraction operations on the superset to produce a reduced set of search results. 
       FIG.  57 A  presents a block diagram illustrating how fields can be extracted during query processing in accordance with the disclosed embodiments. At the start of this process, a search query  7402  is received at a query processor  7404 . Query processor  7404  includes various mechanisms for processing a query, wherein these mechanisms can reside in a search head  7104  and/or an indexer  7102 . Note that the exemplary search query  7402  illustrated in  FIG.  57 A  is expressed in Search Processing Language (SPL), which is used in conjunction with the SPLUNK® ENTERPRISE system. SPL is a pipelined search language in which a set of inputs is operated on by a first command in a command line, and then a subsequent command following the pipe symbol “1” operates on the results produced by the first command, and so on for additional commands. Search query  7402  can also be expressed in other query languages, such as the Structured Query Language (“SQL”) or any suitable query language. 
     Upon receiving search query  7402 , query processor  7404  sees that search query  7402  includes two fields “IP” and “target.” Query processor  7404  also determines that the values for the “IP” and “target” fields have not already been extracted from events in data store  7414 , and consequently determines that query processor  7404  needs to use extraction rules to extract values for the fields. Hence, query processor  7404  performs a lookup for the extraction rules in a rule base  7406 , wherein rule base  7406  maps field names to corresponding extraction rules and obtains extraction rules  7408 - 7409 , wherein extraction rule  7408  specifies how to extract a value for the “IP” field from an event, and extraction rule  7409  specifies how to extract a value for the “target” field from an event. As is illustrated in  FIG.  57 A , extraction rules  7408 - 7409  can comprise regular expressions that specify how to extract values for the relevant fields. Such regular-expression-based extraction rules are also referred to as “regex rules.” In addition to specifying how to extract field values, the extraction rules can also include instructions for deriving a field value by performing a function on a character string or value retrieved by the extraction rule. For example, a transformation rule can truncate a character string, or convert the character string into a different data format. In some cases, the query itself can specify one or more extraction rules. 
     Next, query processor  7404  sends extraction rules  7408 - 7409  to a field extractor  7412 , which applies extraction rules  7408 - 7409  to events  7416 - 7418  in a data store  7414 . Note that data store  7414  can include one or more data stores, and extraction rules  7408 - 7409  can be applied to large numbers of events in data store  7414 , and are not meant to be limited to the three events  7416 - 7418  illustrated in  FIG.  57 A . Moreover, the query processor  7404  can instruct field extractor  7412  to apply the extraction rules to all the events in a data store  7414 , or to a subset of the events that have been filtered based on some criteria. 
     Next, field extractor  7412  applies extraction rule  7408  for the first command “Search IP=“10*” to events in data store  7414  including events  7416 - 7418 . Extraction rule  7408  is used to extract values for the IP address field from events in data store  7414  by looking for a pattern of one or more digits, followed by a period, followed again by one or more digits, followed by another period, followed again by one or more digits, followed by another period, and followed again by one or more digits. Next, field extractor  7412  returns field values  7420  to query processor  7404 , which uses the criterion IP=“10*” to look for IP addresses that start with “10”. Note that events  7416  and  7417  match this criterion, but event  7418  does not, so the result set for the first command is events  7416 - 7417 . 
     Query processor  7404  then sends events  7416 - 717  to the next command “stats count target.” To process this command, query processor  7404  causes field extractor  7412  to apply extraction rule  7409  to events  7416 - 7417 . Extraction rule  7409  is used to extract values for the target field for events  7416 - 7417  by skipping the first four commas in events  7416 - 7417 , and then extracting all of the following characters until a comma or period is reached. Next, field extractor  7412  returns field values  7421  to query processor  7404 , which executes the command “stats count target” to count the number of unique values contained in the target fields, which in this example produces the value “2” that is returned as a final result  7422  for the query. 
     Note that query results can be returned to a client, a search head, or any other system component for further processing. In general, query results can include: a set of one or more events; a set of one or more values obtained from the events; a subset of the values; statistics calculated based on the values; a report containing the values; or a visualization, such as a graph or chart, generated from the values. 
     Creating queries requires knowledge of the fields that are included in the events being searched, as well as knowledge of the query processing language used for the queries. While a data analyst can possess domain understanding of underlying data and knowledge of the query processing language, an end user responsible for creating reports at a company (e.g., a marketing specialist) can not have such expertise. In order to assist end users, implementations of the event processing system described herein provide data models that simplify the creation of reports and other visualizations. 
     A data model encapsulates semantic knowledge about certain events. A data model can be composed of one or more objects grouped in a hierarchical manner. In general, the objects included in a data model can be related to each other in some way. In particular, a data model can include a root object and, optionally, one or more child objects that can be linked (either directly or indirectly) to the root object. A root object can be defined by search criteria for a query to produce a certain set of events, and a set of fields that can be exposed to operate on those events. A root object can be a parent of one or more child objects, and any of those child objects can optionally be a parent of one or more additional child objects. Each child object can inherit the search criteria of its parent object and have additional search criteria to further filter out events represented by its parent object. Each child object can also include at least some of the fields of its parent object and optionally additional fields specific to the child object. 
       FIG.  57 B  illustrates an example data model structure  7428 , in accordance with some implementations. As shown, example data model “Buttercup Games”  7430  includes root object “Purchase Requests”  7432 , and child objects “Successful Purchases”  7434  and “Unsuccessful Purchases”  7436 . 
       FIG.  57 C  illustrates an example definition  7440  of root object  7432  of data model  7430 , in accordance with some implementations. As shown, definition  7440  of root object  7432  includes search criteria  7442  and a set of fields  7444 . Search criteria  7442  require that a search query produce web access requests that qualify as purchase events. Fields  7444  include inherited fields  7446  which are default fields that specify metadata about the events of the root object  7432 . In addition, fields  7444  include extracted fields  7448 , whose values can be automatically extracted from the events during search using extraction rules of the late binding schema, and calculated fields  7450 , whose values can be automatically determined based on values of other fields extracted from the events. For example, the value of the productName field can be determined based on the value in the productID field (e.g., by searching a lookup table for a product name matching the value of the productID field). In another example, the value of the price field can be calculated based on values of other fields (e.g., by multiplying the price per unit by the number of units). 
       FIG.  57 D  illustrates example definitions  7458  and  7460  of child objects  7434  and  7436  respectively, in accordance with some implementations. Definition 7458 of child object  7434  includes search criteria  7462  and a set of fields  7464 . Search criteria  7462  inherits search criteria  7442  of the parent object  7432  and includes an additional criterion of “status=200,” which indicates that the search query should produce web access requests that qualify as successful purchase events. Fields  7464  consist of the fields inherited from the parent object  7432 . 
     Definition 7460 of child object  7436  includes search criteria  7470  and a set of fields  7474 . Search criteria  7470  inherits search criteria  7442  of the parent object  7432  and includes an additional criterion of “status!=200,” which indicates that the search query should produce web access requests that qualify as unsuccessful purchase events. Fields  7474  consist of the fields inherited from the parent object  7432 . As shown, child objects  7434  and  7436  include all the fields inherited from the parent object  7432 . In other implementations, child objects can only include some of the fields of the parent object and/or can include additional fields that are not exposed by the parent object. 
     When creating a report, a user can select an object of a data model to focus on the events represented by the selected object. The user can then view the fields of the data object and request the event processing system to structure the report based on those fields. For example, the user can request the event processing system to add some fields to the report, to add calculations based on some fields to the report, to group data in the report based on some fields, etc. The user can also input additional constraints (e.g., specific values and/or mathematical expressions) for some of the fields to further filter out events on which the report should be focused. 
       FIG.  59 A  illustrates an exemplary search screen  7600  in accordance with the disclosed embodiments. Search screen  7600  includes a search bar  7602  that accepts user input in the form of a search string. It also includes a time range picker  7612  that enables the user to specify a time range for the search. For “historical searches” the user can select a specific time range, or alternatively a relative time range, such as “today,” “yesterday” or “last week.” For “real-time searches,” the user can select the size of a preceding time window to search for real-time events. Search screen  7600  also initially displays a “data summary” dialog as is illustrated in  FIG.  59 B  that enables the user to select different sources for the event data, for example by selecting specific hosts and log files. 
     After the search is executed, the search screen  7600  can display the results through search results tabs  7604 , wherein search results tabs  7604  includes: an “events tab” that displays various information about events returned by the search; a “statistics tab” that displays statistics about the search results; and a “visualization tab” that displays various visualizations of the search results. The events tab illustrated in  FIG.  59 A  displays a timeline graph  7605  that graphically illustrates the number of events that occurred in one-hour intervals over the selected time range. It also displays an events list  7608  that enables a user to view the raw data in each of the returned events. It additionally displays a fields sidebar  7606  that includes statistics about occurrences of specific fields in the returned events, including “selected fields” that are pre-selected by the user, and “interesting fields” that are automatically selected by the system based on pre-specified criteria. 
     The above-described system provides significant flexibility by enabling a user to analyze massive quantities of minimally processed performance data “on the fly” at search time instead of storing pre-specified portions of the performance data in a database at ingestion time. This flexibility enables a user to see correlations in the performance data and perform subsequent queries to examine interesting implementations of the performance data that can not have been apparent at ingestion time. 
     However, performing extraction and analysis operations at search time can involve a large amount of data and require a large number of computational operations, which can cause considerable delays while processing the queries. Fortunately, a number of acceleration techniques have been developed to speed up analysis operations performed at search time. These techniques include: (1) performing search operations in parallel by formulating a search as a map-reduce computation; (2) using a keyword index; (3) using a high performance analytics store; and (4) accelerating the process of generating reports. These techniques are described in more detail below. 
     To facilitate faster query processing, a query can be structured as a map-reduce computation, wherein the “map” operations are delegated to the indexers, while the corresponding “reduce” operations are performed locally at the search head. For example,  FIG.  58    illustrates how a search query  7501  received from a client at search head  7104  can split into two phases, including: (1) a “map phase” comprising subtasks  7502  (e.g., data retrieval or simple filtering) that can be performed in parallel and are “mapped” to indexers  7102  for execution, and (2) a “reduce phase” comprising a merging operation  7503  to be executed by the search head when the results are ultimately collected from the indexers. 
     During operation, upon receiving search query  7501 , search head  7104  modifies search query  7501  by substituting “stats” with “prestats” to produce search query  7502 , and then distributes search query  7502  to one or more distributed indexers, which are also referred to as “search peers.” Note that search queries can generally specify search criteria or operations to be performed on events that meet the search criteria. Search queries can also specify field names, as well as search criteria for the values in the fields or operations to be performed on the values in the fields. Moreover, the search head can distribute the full search query to the search peers as is illustrated in  FIG.  56   , or can alternatively distribute a modified version (e.g., a more restricted version) of the search query to the search peers. In this example, the indexers are responsible for producing the results and sending them to the search head. After the indexers return the results to the search head, the search head performs the merging operations  7503  on the results. Note that by executing the computation in this way, the system effectively distributes the computational operations while minimizing data transfers. 
     As described above with reference to the flow charts in  FIGS.  55  and  56   , event processing system  7100  can construct and maintain one or more keyword indices to facilitate rapidly identifying events containing specific keywords. This can greatly speed up the processing of queries involving specific keywords. As mentioned above, to build a keyword index, an indexer first identifies a set of keywords. Then, the indexer includes the identified keywords in an index, which associates each stored keyword with references to events containing that keyword, or to locations within events where that keyword is located. When an indexer subsequently receives a keyword-based query, the indexer can access the keyword index to quickly identify events containing the keyword. 
     To speed up certain types of queries, some embodiments of system  7100  make use of a high performance analytics store, which is referred to as a “summarization table,” that contains entries for specific field-value pairs. Each of these entries keeps track of instances of a specific value in a specific field in the event data and includes references to events containing the specific value in the specific field. For example, an exemplary entry in a summarization table can keep track of occurrences of the value “94107” in a “ZIP code” field of a set of events, wherein the entry includes references to all of the events that contain the value “94107” in the ZIP code field. This enables the system to quickly process queries that seek to determine how many events have a particular value for a particular field, because the system can examine the entry in the summarization table to count instances of the specific value in the field without having to go through the individual events or do extractions at search time. Also, if the system needs to process all events that have a specific field-value combination, the system can use the references in the summarization table entry to directly access the events to extract further information without having to search all of the events to find the specific field-value combination at search time. 
     In some embodiments, the system maintains a separate summarization table for each of the above-described time-specific buckets that stores events for a specific time range, wherein a bucket-specific summarization table includes entries for specific field-value combinations that occur in events in the specific bucket. Alternatively, the system can maintain a separate summarization table for each indexer, wherein the indexer-specific summarization table only includes entries for the events in a data store that is managed by the specific indexer. 
     The summarization table can be populated by running a “collection query” that scans a set of events to find instances of a specific field-value combination, or alternatively instances of all field-value combinations for a specific field. A collection query can be initiated by a user, or can be scheduled to occur automatically at specific time intervals. A collection query can also be automatically launched in response to a query that asks for a specific field-value combination. 
     In some cases, the summarization tables can not cover all of the events that are relevant to a query. In this case, the system can use the summarization tables to obtain partial results for the events that are covered by summarization tables, but can also have to search through other events that are not covered by the summarization tables to produce additional results. These additional results can then be combined with the partial results to produce a final set of results for the query. This summarization table and associated techniques are described in more detail in U.S. Pat. No. 8,682,925, issued on Mar. 25, 2014. 
     In some embodiments, a data server system such as the SPLUNK® ENTERPRISE system can accelerate the process of periodically generating updated reports based on query results. To accelerate this process, a summarization engine automatically examines the query to determine whether generation of updated reports can be accelerated by creating intermediate summaries. (This is possible if results from preceding time periods can be computed separately and combined to generate an updated report. In some cases, it is not possible to combine such incremental results, for example where a value in the report depends on relationships between events from different time periods.) If reports can be accelerated, the summarization engine periodically generates a summary covering data obtained during a latest non-overlapping time period. For example, where the query seeks events meeting a specified criteria, a summary for the time period includes only events within the time period that meet the specified criteria. Similarly, if the query seeks statistics calculated from the events, such as the number of events that match the specified criteria, then the summary for the time period includes the number of events in the period that match the specified criteria. 
     In parallel with the creation of the summaries, the summarization engine schedules the periodic updating of the report associated with the query. During each scheduled report update, the query engine determines whether intermediate summaries have been generated covering portions of the time period covered by the report update. If so, then the report is generated based on the information contained in the summaries. Also, if additional event data has been received and has not yet been summarized, and is required to generate the complete report, the query can be run on this additional event data. Then, the results returned by this query on the additional event data, along with the partial results obtained from the intermediate summaries, can be combined to generate the updated report. This process is repeated each time the report is updated. Alternatively, if the system stores events in buckets covering specific time ranges, then the summaries can be generated on a bucket-by-bucket basis. Note that producing intermediate summaries can save the work involved in re-running the query for previous time periods, so only the newer event data needs to be processed while generating an updated report. These report acceleration techniques are described in more detail in U.S. Pat. No. 8,589,403, issued on Nov. 19, 2013, and U.S. Pat. No. 8,412,696, issued on Apr. 2, 2011. 
     The SPLUNK® ENTERPRISE platform provides various schemas, dashboards and visualizations that make it easy for developers to create applications to provide additional capabilities. One such application is the SPLUNK® APP FOR ENTERPRISE SECURITY, which performs monitoring and alerting operations and includes analytics to facilitate identifying both known and unknown security threats based on large volumes of data stored by the SPLUNK® ENTERPRISE system. This differs significantly from conventional Security Information and Event Management (SIEM) systems that lack the infrastructure to effectively store and analyze large volumes of security-related event data. Traditional SIEM systems typically use fixed schemas to extract data from pre-defined security-related fields at data ingestion time, wherein the extracted data is typically stored in a relational database. This data extraction process (and associated reduction in data size) that occurs at data ingestion time inevitably hampers future incident investigations, when all of the original data can be needed to determine the root cause of a security issue, or to detect the tiny fingerprints of an impending security threat. 
     In contrast, the SPLUNK® APP FOR ENTERPRISE SECURITY system stores large volumes of minimally processed security-related data at ingestion time for later retrieval and analysis at search time when a live security threat is being investigated. To facilitate this data retrieval process, the SPLUNK® APP FOR ENTERPRISE SECURITY provides pre-specified schemas for extracting relevant values from the different types of security-related event data, and also enables a user to define such schemas. 
     The SPLUNK® APP FOR ENTERPRISE SECURITY can process many types of security-related information. In general, this security-related information can include any information that can be used to identify security threats. For example, the security-related information can include network-related information, such as IP addresses, domain names, asset identifiers, network traffic volume, uniform resource locator strings, and source addresses. Security-related information can also include endpoint information, such as malware infection data and system configuration information, as well as access control information, such as login/logout information and access failure notifications. The security-related information can originate from various sources within a data center, such as hosts, virtual machines, storage devices and sensors. The security-related information can also originate from various sources in a network, such as routers, switches, email servers, proxy servers, gateways, firewalls and intrusion-detection systems. 
     During operation, the SPLUNK® APP FOR ENTERPRISE SECURITY facilitates detecting so-called “notable events” that are likely to indicate a security threat. These notable events can be detected in a number of ways: (1) an analyst can notice a correlation in the data and can manually identify a corresponding group of one or more events as “notable;” or (2) an analyst can define a “correlation search” specifying criteria for a notable event, and every time one or more events satisfy the criteria, the application can indicate that the one or more events are notable. An analyst can alternatively select a pre-defined correlation search provided by the application. Note that correlation searches can be run continuously or at regular intervals (e.g., every hour) to search for notable events. Upon detection, notable events can be stored in a dedicated “notable events index,” which can be subsequently accessed to generate various visualizations containing security-related information. Also, alerts can be generated to notify system operators when important notable events are discovered. 
     The SPLUNK® APP FOR ENTERPRISE SECURITY provides various visualizations to aid in discovering security threats, such as a “key indicators view” that enables a user to view security metrics of interest, such as counts of different types of notable events. For example,  FIG.  60 A  illustrates an exemplary key indicators view  7700  that comprises a dashboard, which can display a value  7701 , for various security-related metrics, such as malware infections  7702 . It can also display a change in a metric value  7703 , which indicates that the number of malware infections increased by 63 during the preceding interval. Key indicators view  7700  additionally displays a histogram panel  7704  that displays a histogram of notable events organized by urgency values, and a histogram of notable events organized by time intervals. 
     These visualizations can also include an “incident review dashboard” that enables a user to view and act on “notable events.” These notable events can include: (1) a single event of high importance, such as any activity from a known web attacker; or (2) multiple events that collectively warrant review, such as a large number of authentication failures on a host followed by a successful authentication. For example,  FIG.  60 B  illustrates an exemplary incident review dashboard  7710  that includes a set of incident attribute fields  7711  that, for example, enables a user to specify a time range field  7712  for the displayed events. It also includes a timeline  7713  that graphically illustrates the number of incidents that occurred in one-hour time intervals over the selected time range. It additionally displays an events list  7714  that enables a user to view a list of all of the notable events that match the criteria in the incident attributes fields  7711 . To facilitate identifying patterns among the notable events, each notable event can be associated with an urgency value (e.g., low, medium, high, critical), which is indicated in the incident review dashboard. The urgency value for a detected event can be determined based on the severity of the event and the priority of the system component associated with the event. 
     As mentioned above, the SPLUNK® ENTERPRISE platform provides various features that make it easy for developers to create various applications. One such application is the SPLUNK® APP FOR VMWARE®, which performs monitoring operations and includes analytics to facilitate diagnosing the root cause of performance problems in a data center based on large volumes of data stored by the SPLUNK® ENTERPRISE system. 
     This differs from conventional data-center-monitoring systems that lack the infrastructure to effectively store and analyze large volumes of performance information and log data obtained from the data center. In conventional data-center-monitoring systems, this performance data is typically pre-processed prior to being stored, for example by extracting pre-specified data items from the performance data and storing them in a database to facilitate subsequent retrieval and analysis at search time. However, the rest of the performance data is not saved and is essentially discarded during pre-processing. In contrast, the SPLUNK® APP FOR VMWARE® stores large volumes of minimally processed performance information and log data at ingestion time for later retrieval and analysis at search time when a live performance issue is being investigated. 
     The SPLUNK® APP FOR VMWARE® can process many types of performance-related information. In general, this performance-related information can include any type of performance-related data and log data produced by virtual machines and host computer systems in a data center. In addition to data obtained from various log files, this performance-related information can include values for performance metrics obtained through an application programming interface (API) provided as part of the vSphere Hypervisor™ system distributed by VMware, Inc. of Palo Alto, Calif. For example, these performance metrics can include: (1) CPU-related performance metrics; (2) disk-related performance metrics; (3) memory-related performance metrics; (4) network-related performance metrics; (5) energy-usage statistics; (6) data-traffic-related performance metrics; (7) overall system availability performance metrics; (8) cluster-related performance metrics; and (9) virtual machine performance statistics. For more details about such performance metrics, please see U.S. patent Ser. No. 14/167,316 filed 29 Jan. 2014, which is hereby incorporated herein by reference. Also, see “vSphere Monitoring and Performance,” Update 1, vSphere 5.5, EN-001357-00, http://pubs.vmware.com/vsphere-55/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-551-monitoring-performance-guide.pdf. 
     To facilitate retrieving information of interest from performance data and log files, the SPLUNK® APP FOR VMWARE® provides pre-specified schemas for extracting relevant values from different types of performance-related event data, and also enables a user to define such schemas. 
     The SPLUNK® APP FOR VMWARE® additionally provides various visualizations to facilitate detecting and diagnosing the root cause of performance problems. For example, one such visualization is a “proactive monitoring tree” that enables a user to easily view and understand relationships among various factors that affect the performance of a hierarchically structured computing system. This proactive monitoring tree enables a user to easily navigate the hierarchy by selectively expanding nodes representing various entities (e.g., virtual centers or computing clusters) to view performance information for lower-level nodes associated with lower-level entities (e.g., virtual machines or host systems). Exemplary node-expansion operations are illustrated in  FIG.  60 C , wherein nodes  7733  and  7734  are selectively expanded. Note that nodes  7731 - 7739  can be displayed using different patterns or colors to represent different performance states, such as a critical state, a warning state, a normal state or an unknown/offline state. The ease of navigation provided by selective expansion in combination with the associated performance-state information enables a user to quickly diagnose the root cause of a performance problem. 
     The SPLUNK® APP FOR VMWARE® also provides a user interface that enables a user to select a specific time range and then view heterogeneous data, comprising events, log data and associated performance metrics, for the selected time range. For example, the screen illustrated in  FIG.  60 D  displays a listing of recent “tasks and events” and a listing of recent “log entries” for a selected time range above a performance-metric graph for “average CPU core utilization” for the selected time range. Note that a user is able to operate pull-down menus  7742  to selectively display different performance metric graphs for the selected time range. This enables the user to correlate trends in the performance-metric graph with corresponding event and log data to quickly determine the root cause of a performance problem. 
       FIG.  61    illustrates a diagrammatic representation of a machine in the exemplary form of a computer system  7800  within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, can be executed. The system  7800  can be in the form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, can be executed. In alternative embodiments, the machine can be connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, or the Internet. The machine can operate in the capacity of a server machine in client-server network environment. The machine can be a personal computer (PC), a set-top box (STB), a server, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. In one embodiment, computer system  7800  can represent system  210  of  FIG.  2   . 
     The exemplary computer system  7800  includes a processing device (processor)  7802 , a main memory  7804  (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM)), a static memory  7806  (e.g., flash memory, static random access memory (SRAM)), and a data storage device  7818 , which communicate with each other via a bus  7830 . 
     Processing device  7802  represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More particularly, the processing device  7802  can be a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. The processing device  7802  can also be one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device  7802  is conFig.d to execute the notification manager  210  for performing the operations and steps discussed herein. 
     The computer system  7800  can further include a network interface device  7808 . The computer system  7800  also can include a video display unit  7810  (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device  7812  (e.g., a keyboard), a cursor control device  7814  (e.g., a mouse), and a signal generation device  7816  (e.g., a speaker). 
     The data storage device  7818  can include a computer-readable medium  7828  on which is stored one or more sets of instructions  7822  (e.g., instructions for search term generation) embodying any one or more of the methodologies or functions described herein. The instructions  7822  can also reside, completely or at least partially, within the main memory  7804  and/or within processing logic  7826  of the processing device  7802  during execution thereof by the computer system  7800 , the main memory  7804  and the processing device  7802  also constituting computer-readable media. The instructions can further be transmitted or received over a network  7820  via the network interface device  7808 . 
     While the computer-readable storage medium  7828  is shown in an exemplary embodiment to be a single medium, the term “computer-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “computer-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present invention. The term “computer-readable storage medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical media, and magnetic media. 
     The preceding description sets forth numerous specific details such as examples of specific systems, components, methods, and so forth, in order to provide a good understanding of several embodiments of the present invention. It will be apparent to one skilled in the art, however, that at least some embodiments of the present invention can be practiced without these specific details. In other instances, well-known components or methods are not described in detail or are presented in simple block diagram format in order to avoid unnecessarily obscuring the present invention. Thus, the specific details set forth are merely exemplary. Particular implementations can vary from these exemplary details and still be contemplated to be within the scope of the present invention. 
     In the above description, numerous details are set forth. It will be apparent, however, to one of ordinary skill in the art having the benefit of this disclosure, that embodiments of the invention can be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the description. 
     Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. 
     It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilizing terms such as “determining”, “identifying”, “adding”, “selecting” or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (e.g., electronic) quantities within the computer system&#39;s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. 
     Embodiments of the invention also relate to an apparatus for performing the operations herein. This apparatus can be specially constructed for the required purposes, or it can comprise a general purpose computer selectively activated or reconFig.d by a computer program stored in the computer. Such a computer program can be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions. 
     The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct a more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear from the description below. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the invention as described herein. 
     Implementations that are described can include graphical user interfaces (GUIs). Frequently, an element that appears in a GUI display is associated or bound to particular data in the underlying computer system. The GUI element can be used to indicate the particular data by displaying the data in some fashion, and can possibly enable the user to interact to indicate the data in a desired, changed form or value. In such cases, where a GUI element is associated or bound to particular data, it is a common shorthand to refer to the data indications of the GUI element as the GUI element, itself, and vice versa. The reader is reminded of such shorthand and that the context renders the intended meaning clear to one of skill in the art where a distinction between a GUI element and the data to which it is bound is meaningful. 
     It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reading and understanding the above description. The scope of the invention should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.