Patent Publication Number: US-9413685-B1

Title: Method and apparatus for cross domain and cross-layer event correlation

Description:
BACKGROUND 
     As is known in the art, cloud computing systems, even in pre-architected and pre-qualified environments, contain a relatively large number of hardware devices and components and software applications, modules, and components. In the presence of a fault, alert, or other condition needing attention, it can be difficult to identify the source of the fault or alert since there are many complex components that may be provided by multiple vendors which may make it difficult to correlate information in an efficient manner. 
     For example, in a cloud computing environment, alerts and events from various event sources in platforms normally contain limited information that may not be meaningful and may seem unrelated to the environment from which they originate. It is challenging for IT personnel to extract executable data from the alerts and take appropriate action. 
     With large volumes of alerts/events constantly coming from various sources, it is time consuming to troubleshoot all of them all any prioritization. It is challenging to prioritize the alerts/events and take appropriate actions without correlating them and knowing which of the alerts or events are root causes and which are just symptoms. In addition, many of the IT resources are managed in silos by IT personnel specialized in certain technology domains. For example, when a blade in the Cisco Unified Computing System (UCS) fails or has performance issues its impact propagates to the ESX server deployed on the blade, to the virtual machines deployed on the ESX server, to the applications or critical services running on those virtual machines, to the critical business that relies on those services. It may take hours or even days to sort through those alerts or events, which may result in significant detrimental impact on an enterprise. 
     SUMMARY 
     In one aspect of the invention, a method comprises: in a cloud computing system having system domains including a storage domain, a network domain, a compute domain, and a virtual layer: receiving a first alert in a first domain of the system domains and a second alert in a second domain of the system domains, the first and second alerts caused by an event, determining a topology of the system including physical and logical connections for components in the storage, network, compute domains, and the virtual layer, performing, using a computer processor, connectivity matching to identify connections between a port in the first domain and a port in the second domain, and performing identify matching, using unique identifiers for domain components, to determine that a first component in the first domain associated with the first alert is the same component as a second component associated with the second alert in the second domain for cross domain event correlation. 
     The method can further including one or more of the following features: the first component comprises a storage volume in the storage domain, the first component comprises a port in the network domain for a switch component, the first component comprises a cable connected to a fabric in the compute domain, the first comprises a server in the virtual layer, querying the first alert, the unique identifiers include a WWPN or a UUID, and/or the first alert includes one of a server (blade or ESX) down or connection lost, a storage volume error, heartbeat timeout between a server and a datastore, a zoneset being deactivated, a port failure, a link down, a host adaptor down, a virtual circuit down, a SAN port down, a Fibre Channel link down, and/or a virtual interface down. 
     In another aspect of the invention, an article comprises: a computer readable medium containing non-transitory stored instructions that enable a machine to perform, in a cloud computing system having system domains including a storage domain, a network domain, a compute domain, and a virtual layer, receiving a first alert in a first domain of the system domains and a second alert in a second domain of the system domains, the first and second alerts caused by an event, determining a topology of the system including physical and logical connections for components in the storage, network, compute domains, and the virtual layer, performing, using a computer processor, connectivity matching to identify connections between a port in the first domain and a port in the second domain, and performing identify matching, using unique identifiers for domain components, to determine that a first component in the first domain associated with the first alert is the same component as a second component associated with the second alert in the second domain for cross domain event correlation. 
     The article can further include one or more of the following features: the first component comprises a storage volume in the storage domain, the first component comprises a port in the network domain for a switch component, the first component comprises a cable connected to a fabric in the compute domain, the first component comprises a server in the virtual layer, querying the first alert, the unique identifiers include a WWPN or a UUID, and/or the first alert includes one of a server (blade or ESX) down or connection lost, a storage volume error, heartbeat timeout between a server and a datastore, a zoneset being deactivated, a port failure, a link down, a host adaptor down, a virtual circuit down, a SAN port down, a Fibre Channel link down, and/or a virtual interface down. 
     In a further aspect of the invention, a cloud computing system having system domains including a storage domain, a network domain, a compute domain, and a virtual layer, the system comprising: a processor, and a memory containing stored instructions to enable the processor to: receive a first alert in a first domain of the system domains and a second alert in a second domain of the system domains, the first and second alerts caused by an event, determine a topology of the system including physical and logical connections for components in the storage, network, compute domains, and the virtual layer, perform connectivity matching to identify connections between a port in the first domain and a port in the second domain, and perform identify matching, using unique identifiers for domain components, to determine that a first component in the first domain associated with the first alert is the same component as a second component associated with the second alert in the second domain for cross domain event correlation. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The foregoing features of this invention, as well as the invention itself, may be more fully understood from the following description of the drawings in which: 
         FIG. 1  is a high level schematic representation of a cloud computing system having cross domain event correlation in accordance with exemplary embodiments of the invention; 
         FIG. 2  is a schematic representation showing further detail of a pre-architected and pre-qualified cloud computing system of  FIG. 1  including interconnections; 
         FIG. 3  is a schematic representation showing further detail of a pre-architected and pre-qualified cloud computing system of  FIG. 2  including system components; 
         FIG. 4  is a schematic representation showing further detail of a storage domain of the cloud computing system of  FIG. 3  using NAS for the storage domain; 
         FIG. 5  is a schematic representation of a unified infrastructure manager (UIM) module showing component layering; 
         FIG. 6  is a tabular representation of alerts caused by a single failure in a pre-architected and pre-qualified cloud computing system; 
         FIG. 7  is a schematic representation of cross-domain event correlation; 
         FIG. 8  is a schematic representation of an exemplary system having domains and layers; 
         FIG. 9  is a schematic representation of components in the event correlation engine in the UIM/Operations product; 
         FIG. 10  is a schematic representation of a topology stitcher receiving information from components in the system; 
         FIG. 11  is a schematic representation of output information from the topology stitcher; 
         FIG. 11A  is a schematic representation of the flow of event correlation processing implemented in the UIM/Operations product; 
         FIG. 12  is an exemplary display of cross-domain alert information; 
         FIG. 13  is an exemplary display of alert causality and root cause; 
         FIG. 14  is a schematic representation of an exemplary computer that can perform at least some of the processing described herein. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  shows an exemplary cloud computing environment  100  having cross domain event correlation in accordance with exemplary embodiments of the invention. The environment includes an infrastructure layer  110  and a virtual layer  108 . The infrastructure layer  110  is further divided into compute domain  102 , a network domain  104 , and a storage domain  106 . The environment may be referred to as a platform. It is understood that any practical number of platforms can be combined into a cloud computing environment. 
     The compute domain  102  comprises components, such as blade servers, chassis and fabric interconnects that provide the computing power for the platform. The storage domain  106  comprises the storage components for the platform. The network domain  104  comprises the components that provide switching and routing between the compute and storage domains  102 ,  106  within and between platforms, and to the client or customer network. 
       FIG. 2  shows further detail for the environment  100  of  FIG. 1 . The storage domain  106  can include storage components  150 , such as CLARIION storage components from EMC Corporation of Hopkinton Mass. The network domain  104  can include a pair of switches  152 , such as MDS 9000 Series Multilayer SAN Switches from Cisco of San Jose, Calif., coupled to the storage components and to a LAN. The compute domain  102  can include a pair of fabric interconnects  154 , such as CISCO 6100 series devices. The compute domain can further include a number of blade servers  156 , such as CISCO 5100 blade chassis. 
       FIG. 3  shows further detail of an exemplary cloud environment having a compute domain  302 , a network domain  304  and a storage domain  306 . The network domain  302  is coupled to a customer network  308  in a manner known in the art. The network domain  302  includes switches  310  coupled to the customer network  308 . The network domain  302  also includes multilayer fabric switches  312  coupled to fabric interconnects  314  in the compute domain  302  and to storage processors  316  in the storage domain  306 . The fabric interconnects  314  are coupled to blade server chassis  318  containing blades. Data movers  320  in the storage domain  306  are coupled between the storage processors  316  and the switches  310  in the network domain. Disk array enclosures  322  are coupled to the storage processors  316 .  FIG. 4  shows interconnections for a system similar to that shown in  FIG. 3  with physical disks  326 . In the illustrated embodiment, the storage domain includes 8 to 16 front end fibre channel ports and 2-4 GB iSCSI front end ports. It is understood that a variety of other configurations having different interconnections and storage configuration can be provided to meet the needs of a particular application. The virtual layer can include a number of applications to perform various functions for overall control, configuration, etc of the various platform components. For example, virtual layer applications can include a virtualization function, such as VSPHERE/VCENTER, by VMware of Palto Alto, Calif. 
     Each of the infrastructure domains and the virtual layer is managed by a separate element management system. One of such element management systems is the Unified Computing System (UCS) Manager by Cisco. It is understood that the blade chassis and fabric interconnection can be considered part of the UCS. Another management application can include a management interface, such as EMC UNISPHERE, to provide a flexible, integrated experience for managing existing storage systems, such as CLARIION and CELERRA storage devices from EMC. A further management application includes a platform element manager, such as Unified Infrastructure Manager (UIM) by EMC, for managing the configuration, provisioning, and compliance of the platform. 
       FIG. 5  shows an exemplary unified infrastructure manager  500 . In one embodiment, the unified infrastructure manager  500  includes a configuration center module  502 , a provisioning center module  504 , and an operations center module  506  having cross domain event correlation in accordance with exemplary embodiments of the invention. Below these modules are a platform infrastructure service catalog  506  and a cross domain context and visibility module  508 . 
     The unified infrastructure manager  500  further includes a change and configuration management module  510 , a policy-based compliance and analysis module  512 , a unified infrastructure provisioning module  514 , a consolidation topology and event service module  516 , and an operational awareness module  518 . The various modules interact with platform elements, such as devices in compute, network and storage domains, and other management applications. 
     The unified infrastructure manager  500  performs platform deployment by abstracting the overall provisioning aspect of the platform(s) and offering granular access to platform components for trouble shooting and fault management. 
     In one aspect of the invention, a uniform infrastructure management module includes a cross domain event correlation module to process alerts from physical infrastructure, logical services, virtual applications, and tenant/organizations. It is understood that cloud environments can provide storage for multiple tenants/organizations. 
     For example, a Vblock leverages VMware, Cisco, and EMC packages with integrated computing, network, storage, and management capabilities. Each of the domains has its own management system. When a component, for example, fails, multiple domain management systems may report an alert indication representing the same failure. For example, when a UCS port fails, a fault is triggered in the UCS. At the same time, vCenter also reports an alarm on the corresponding VMNIC. These two events indicate the same failure. To correlate duplicate events across different domains, the affected objects are matched using unique identifiers. In one embodiment, the following is used to perform matching and event correlation: UCS/vHBA and vSphere/vmhba using WWPN; match UCS/FCPort and MDS/FCPort using WWPN; match UCS/Blade and vCenter/ESX using UUID; match Storage/Volume and vCenter/Datastore using UUID; and matching StoragePort and vCenter/ScsiTarget using WWPN. 
     It is understood that various vendor specific terminology, product name, jargon, etc., may be used herein. It is further understood that such vendor specific information is used to facilitate an understanding of embodiments of the invention and should not limit the invention in any way. Any specific vendor information should be construed mean a generic product, function, or module. 
     Some exemplary items are set forth below. It is understood that one of ordinary skill in the art is familiar with the generic architecture and functionality of a vendor specific terms. 
     UIM/Provisioning or UIM/P: EMC Unified Infrastructure Management/Provisioning that provides simplified management for VCE VBLOCK by managing the components of VBLOCK platforms as a single entity and easily define and create infrastructure service profiles to match business requirements. 
     Cisco UCS: Cisco Unified Computing System. 
     VMWARE VSPHERE: A virtualization platform for building cloud infrastructures 
     ESX/ESXi: An enterprise-level computer virtualization product offered by VMware. 
     VM: Virtual Machine 
     VBLOCK: A pre-architected and pre-qualified environment for virtualization at scale: storage, fabric, compute, hypervisor, management and security. 
     WWPN: World Wide Port Name 
     UUID: Universally Unique Identifier 
     HBA: Host Bus Adapter 
     NIC: Network Interface Card 
     FC: Fiber Channel 
     The cloud computing environment is broken up into layers: infrastructure (or physical) layer and the virtual layer. In exemplary embodiments, the infrastructure layer is divided into three domains (silos): compute, network, storage, because traditionally they are managed separately with respective element management systems. 
       FIG. 6  shows an exemplary representation  600  of a port failure in a SAN component listing the domain  602 , raw alert  604 , alert count  606 , and correlation  608 , which are described below in detail. In the SAN switch domain  610 , there is a link down  612 , which is the root cause  614  of the alert as shown in the correlation column. The alert count  616  is one. 
     The UCS domain  620  lists a series of raw alerts  622  generated by the SAN port failure in SAN switch domain. As can be seen, the UCS alerts  622  include a FC link down, a host HBA down, a virtual interface down, and a virtual circuit down for a count of 13 in the UCS domain. The vCenter domain  624  lists two raw alerts  626  with a total count of 23 alerts (10+13). A total of 37 alerts are generated in three domains due the root cause SAN port failure. 
       FIG. 7  shows an exemplary representation of cross domain event correlation for root cause events generated in a network domain NL, compute domain CL, storage domain SL, and virtual layer VL. The root causes have respective symptoms. For example, a host HBA down in the compute domain CL has as a symptom a virtual circuit down. While a domain management system may correlate events within that domain, cross-domain event correlation correlates events across the various domains, as described more fully below. 
       FIG. 8  shows cross domain matching with connectivity and identity of components. The exemplary system  800  includes a storage domain  802 , a network domain  804 , a compute domain  806  and a virtual  808 . The system is shown having a virtual layer  810 , a logical layer  812 , and a physical infrastructure layer  814 . 
     Each of the storage  802 , network  804 , compute  806  domains and the virtual layer is managed by respective element management systems. In an exemplary embodiment, cross-domain correlation establishes relationships between objects in different domains or layers, as shown in the dash-lined boxes. 
     The storage domain  802  comprises a LUN  810  and a storage system  812  having first and second controllers  814   a,b  and disk arrays  816 . The network domain  804  comprises first and second switches  818 ,  820  coupled between the storage domain  802  and the compute domain  806 , which includes first and second fabric switches  822   a,b  and a UCS blade  824 . The virtual layer  808  comprises an ESX server  826 , a virtualization application  828 , such as VM, and a datastore application  830 . As can be seen, the components, domains and layers are connected by physical, logical and transaction connections. For example, physical connections between ports are shown between the first UCS fabric switch B  822   a  and the first network switch  818 . A cross-domain logical connection is shown between the ESX server  826  and the storage LUN  810 . A virtual connection is shown between VM  828  and the datastore  830 . 
     In one particular embodiment, first and second categories of alert matching include connectivity matching and identity matching. For connectivity matching, a port  840  in the UCS fabric  822   a  is connected to a port  842  in the MDS switch via a cable. If the MDS reports that port  842  is down, then it is very likely the UCSM  806  will also report the connected UCS port  840  down. The system stores port-to-port connectivity for the components in the domains. 
     For identity matching, the same storage volume is represented in both the storage system and the vCenter/vSphere environment, for example. If there is a failure on the storage volume  810 , the management systems for the storage domain  802  and the virtual layer  808  will report an alert. The two alerts in the different domains are reconciled to determine the root cause, as described more fully below. 
       FIG. 9  shows an exemplary implementation of the cross-domain event correlation in accordance with exemplary embodiments of the invention. An event correlation engine/module  900  includes a topology repository module  902  to store the infrastructural topology objects of all domains and their connections, the virtual layer objects, and their relationships with the infrastructural objects. The schema of the topology objects are driven by a predefined model  904 . A codebook module  906  takes the repository data (topology, relationship, status) as the input and calculates the causality relationships between the events or alerts. 
     A topology importer module  910  obtains topology information for the system, such as the UCS, storage, network domains and the virtual layer, for transmission to the topology repository module  902 . The topology importer creates and maintains topology objects and intra-domain relationships by periodically querying data sources and instantiating objects. A topology stitcher module  912  creates cross-domain relationships, as described more fully below. In general, the topology stitcher  912  builds relationships among cross-domain objects, which enables the correlation engine  900  to correlate alerts from different domains. 
     State updates are performed based upon the received alerts by mapping an alert to a state of a topology object. A so-called “event” is triggered by a certain condition of the state, and the correlation engine  900  determines the relationships among the events based on the model  904  and the topology relationships instantiated by the topology importer  910  and stitcher  912 . 
     The causality output  920  from the topology repository  902  enables the system to fetch events, and then for each event, to query the related events. For example, if event A causes event B, then B will be returned in the list of getExplainedBy(A). 
       FIG. 10  shows further detail for an exemplary topology stitcher  1000 . The topology stitcher  1000  examines objects and their unique identifications from different domains. The input from vCenter  1002  to the topology stitcher, for example, is the UUID (Universally Unique IDentifier) of ESX and Datastore, as well as the WWPN (World Wide Port Name) of ScsiTarget and vmhba. The UUID and the WWPN are used as unique identifiers for the topology stitcher to match related objects from other domains. 
     Similarly, input from the UCS  1004  includes the WWPN for the FC ports, the WWPN for the vHBAs, and UUID for the blade. Input from the UCS  1006  includes the peer WWPN for FC ports and the WWPN for the zones. Input from the storage domain  1008  includes the UUID for the storage volumes and the WWPN for the storage ports. 
     In one embodiment, the input and the matching criteria are defined in an XML file, so that newly identified matching criteria can be easily added without code changes for the topology stitcher to perform its function. Below is an example: 
     &lt;ScsiTarget&gt; 
     . . . 
     &lt;matching targetClass=“StoragePort” relationship=“LayeredOver” 
     isRelationshipset=“true”&gt; 
     &lt;property selfProperty=“WWPN” propertyInTarget=“WWPN”/&gt; 
     &lt;/matching&gt; 
     &lt;/ScsiTarget&gt; 
     This section of data instructs the topology stitcher to do the following: For objects in class ScsiTarget, find objects in StoragePort that have the same WWPN, and create a relationship called “LayeredOver” between them. In other words, a matching criteria is characterized by the following elements:
         classA   classB   A list of matching property pairs {propertyA1, propertyB1; propertyA2, propertyB2; . . . }   relationshipName   isRelationshipSet—indicates whether the relationship is a one-to-one or one-to-many relationship       

       FIG. 11  shows exemplary topology stitcher outputs. In one embodiment, the topology stitcher  1100  performs 7 matches, resulting in 5 identity groups of classes shown in the diagram. The 7 pairs of matches, counter-clockwise starting from the left, are as follows:
         Matching UCS/Blade and vCenter/ESX using UUID  1102     Matching UCS/FCPort and MDS/FCPort using Peer WWPN  1104     Matching MDS/FCPort and Storage/Port using Peer WWPN  1106     Matching Storage/Port and vCenter/ScsiTarget using WWPN  1108     Matching Storage/Volume and vCenter/Datastore using UUID  1110     Matching UCS/vHBA and vCenter/vmhba using WWPN  1112     Matching MDS/Zone and UCS/vHBA using WWPN  1114         

     Exemplary pseudo code for generating matches is set forth below: 
     for each matching criteria {
         for each object objB in classB {
           generate a string key that concatenate the values of all matching properties: key=objB.propertyB1.value+delim+objB.propertyB2.value+ . . . ;   save the object in a hash map, using the generated key: bMap.put(key, objB);   
           }   for each object objA in classA {
           generate a string key that concatenate the values of all matching properties: key=objA.propertyA1.value+delim+objA.propertyA2.value+ . . . ;   lookup bMap using the key to see if matches are found: matchedObjBSet=bMap.get(key);   for each objB in matchedObjBSet {
               if (isRelationshipSet) {objA.relationshipName.insert(objB);}   else {objA.relationshipName=objB;}   
               
           }       

     } 
       FIG. 11A  shows an exemplary sequence of steps for providing cross-domain event correlation. In step  1150 , topology data is retrieved from various infrastructure domains and the virtual layer. In step  1152 , connectivity data is examined. For example, the port-to-port connectivity for the various components in various domains is examined. In step  1154 , connectivity matching of corresponding components from various domains is performed. 
     In parallel, in step  1156 , the objects are examined, regardless of whether there is alert, along with the unique identifiers from the domains. In step  1158 , identity matching is performed based on the unique identifiers. In  1160 , corresponding relationships are created based on the matches for cross-domain event correlation, as shown in  FIG. 11 . 
     In step  1170 , an event in the system, such as a link failure, generates alerts in multiple domains that are received by the management domain. In step  1172 , the alerts are mapped to object states in the topology store. In step  1180 , relationships of alerts are made consumable by users. 
       FIG. 12  shows an exemplary screen display  1200  of an event correlation result between a StorageVolume alert  1202  and a Datastore alert:  1204 . The Storage Volume alert  1202 , called StorageVolume-Error, indicates the Storage Volume operational state is Error. The Datastore alert  1204 , called EsxProblemVmfsHeartbeatTimeout, indicates the datastore is no longer reachable by the ESX. It is understood that the storage volume alert  1202  corresponds to the root cause of the alert. 
       FIG. 13  shows an exemplary screen display  1250  of a further event correlation result including the first level of the causality tree under the link down alert of an MDS/FCPort. This event correlates the following alerts:
         The link down alert  1252  of two UCS/vHBA interfaces since they share the same zone with the MDS/FCPort, shown by the first two lines under the Causes node;   The link down alert  1254  of the matched UCS/FCPort, shown by the third line under the Causes node.       

     The following table illustrates the cross domain identity matching between the storage volume or LUN on storage arrays and the datastore in VMware vSphere 
     
       
         
           
               
               
               
             
               
                   
               
               
                 Domain 
                 Storage 
                 vSphere 
               
               
                   
               
             
            
               
                 Resource 
                 Storage Volume [LUN] 
                 Datestore 
               
               
                 type 
                   
                   
               
               
                 Identifiers 
                 UUID 
                 UUID 
               
               
                 Examples 
                 60:06:01:60:3B:40:2D:00:42:2E:06:59:4F:62:E1:11 
                 naa.600601603b402d00422e06594f62e111 
               
               
                   
               
            
           
         
       
     
       FIG. 14  shows an exemplary computer that can perform at least a part of the processing described herein. A computer includes a processor  1302 , a volatile memory  1304 , an output device  1305 , a non-volatile memory  1306  (e.g., hard disk), and a graphical user interface (GUI)  1308  (e.g., a mouse, a keyboard, a display, for example). The non-volatile memory  1306  stores computer instructions  1312 , an operating system  1316  and data  1318 , for example. In one example, the computer instructions  1312  are executed by the processor  1302  out of volatile memory  1304  to perform all or part of the processing described above. An article  1319  can comprise a machine-readable medium that stores executable instructions causing a machine to perform any portion of the processing described herein. 
     Processing is not limited to use with the hardware and software described herein and may find applicability in any computing or processing environment and with any type of machine or set of machines that is capable of running a computer program. Processing may be implemented in hardware, software, or a combination of the two. Processing may be implemented in computer programs executed on programmable computers/machines that each includes a processor, a storage medium or other article of manufacture that is readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and one or more output devices. Programs may be implemented in a high level procedural or object-oriented programming language to communicate with a computer system. However, the programs may be implemented in assembly or machine language. The language may be a compiled or an interpreted language and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network. A computer program may be stored on a storage medium or device (e.g., CD-ROM, hard disk, or magnetic diskette) that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform processing. 
     One skilled in the art will appreciate further features and advantages of the invention based on the above-described embodiments. Accordingly, the invention is not to be limited by what has been particularly shown and described, except as indicated by the appended claims. All publications and references cited herein are expressly incorporated herein by reference in their entirety.