Automatic recognition of entities related to cloud incidents

Systems and methods for automatic recognition of entities related to cloud incidents are described. A method, implemented by at least one processor, for processing cloud incidents related information, including entity names and entity values associated with incidents having a potential to adversely impact products or services offered by a cloud service provider is provided. The method may include using at least one processor, processing the cloud incidents related information to convert at least words and symbols corresponding to a cloud incident into machine learning formatted data. The method may further include using a machine learning pipeline, processing at least a subset of the machine learning formatted data to recognize entity names and entity values associated with the cloud incident.

BACKGROUND

The public cloud includes a global network of servers that perform a variety of functions, including storing and managing data, running applications, and delivering content or services, such as streaming videos, provisioning electronic mail, providing office productivity software, or handling social media. The servers and other components may be located in data centers across the world. While the public cloud offers services to the public over the Internet, businesses may use private clouds or hybrid clouds. Both private and hybrid clouds also include a network of servers housed in data centers.

Managing cloud incidents is difficult because of the large size of the unstructured information related to cloud incidents.

SUMMARY

In one example, the present disclosure relates to a method, implemented by at least one processor, for processing cloud incidents related information, including entity names and entity values associated with incidents having a potential to adversely impact products or services offered by a cloud service provider. The method may include using the at least one processor, processing the cloud incidents related information to convert at least words and symbols corresponding to a cloud incident into machine learning formatted data. The method may further include using a machine learning pipeline, processing at least a subset of the machine learning formatted data to recognize entity names and entity values associated with the cloud incident.

In another example, the present disclosure relates to a system, including at least one processor, for processing cloud incidents related information, including entity names and entity values associated with incidents having a potential to adversely impact products or services offered by a cloud service provider. The system may be configured to using the at least one processor, process the cloud incidents related information to convert at least words and symbols corresponding to a cloud incident into machine learning formatted data. The system may further be configured to using a machine learning pipeline, process at least a subset of the machine learning formatted data to recognize entity names and entity values associated with the cloud incident.

In yet another example, the present disclosure relates to a method, implemented by at least one processor, for processing cloud incidents related information, including entity names, entity values, and data types associated with incidents having a potential to adversely impact products or services offered by a cloud service provider. The method may include using the at least one processor, processing the cloud incidents related information to convert at least words and symbols corresponding to a cloud incident into machine learning formatted data. The method may further include using a first machine learning pipeline, as part of a first prediction task, processing at least a subset of the machine learning formatted data to recognize entity names and entity values associated with the cloud incident. The method may further include using a second machine learning pipeline, as part of a second prediction task, processing at least a subset of the machine learning formatted data to recognize data types associated with the cloud incident.

DETAILED DESCRIPTION

Examples described in this disclosure relate to automatic recognition of entities related to cloud incidents. Certain examples relate to automatically recognizing entity names and data types related to cloud incidents using a machine learning pipeline. The public cloud includes a global network of servers that perform a variety of functions, including storing and managing data, running applications, and delivering content or services, such as streaming videos, electronic mail, office productivity software, or social media. The servers and other components may be located in data centers across the world. While the public cloud offers services to the public over the Internet, businesses may use private clouds or hybrid clouds. Both private and hybrid clouds also include a network of servers housed in data centers. Regardless of the arrangement of the cloud infrastructure, incidents requiring attention by the cloud service provider occur frequently.

Incident management includes activities such as automated triaging of incidents and incident diagnosis/detection. Structured knowledge extraction from incidents may require the use of machine learning. Machine learning may be used to extract information from sources, such as sources accessible via uniform resource links (e.g., web pages). In software artifacts like incidents, the vocabulary is not limited to the English language or other human languages. As an example, incidents' related information contains not just textual information concerning the incidents, but also information concerning entities such as GUIDs, Exceptions, IP Addresses, etc. Certain examples described in the present disclosure leverage a multi-task deep learning model for unsupervised knowledge extraction from information concerning incidents, such as cloud incidents. Advantageously, the unsupervised learning may eliminate the inefficiency of annotating a large amount of training data.

In certain examples, a framework for unsupervised knowledge extraction from service incidents is described. As part of certain examples, the knowledge extraction problem is framed as a named-entity recognition task for extracting factual information related to the cloud incidents. Certain examples related to the present disclosure leverage structural patterns like key, value pairs and tables for bootstrapping the training data. Other examples relate to using a multi-task learning based Bi-LSTM-CRF model, which leverages not only the semantic context associated with the incident descriptions, but also the data-types associated with the extracted named entities. Experiments with this unsupervised machine learning based approach show good results with a high precision of 0.96. In addition, because the described systems and methods in the present disclosure are domain agnostic, they can be applied to other types of services and teams. Moreover, these systems and methods can be extended to other artifacts, including support tickets and logs. Using the knowledge extracted by the example approaches described herein, significantly more accurate models for downstream tasks like incident triaging can also be built.

FIG.1is a block diagram of an incident lifecycle100in accordance with one example. In this example, incident lifecycle100may broadly be classified into four phases: alerting phase110, investigation phase120, triaging phase140, and resolution phase150. In alerting phase110, during an incident alert stage112, an incident may be triggered when the service monitoring metrics fall below a predefined level in terms of the performance (e.g., slow response to a query), a slow transfer rate, a customer complaint or escalation, a system hang or crash, or the like. In general, telemetry systems deployed for monitoring services being offered via the cloud platform may collect telemetry data via various sensors. The monitoring of such sensor data may trigger an incident as part of incident alert stage112. Once an incident alert is generated, as part of investigation phase120, the information related to the incident alert may be stored in an incident database as part of incident creation stage122. Investigation phase120may further include an escalation to team stage124, during which the incident may then be escalated to a relevant team. In one example, the identification of the relevant team may be automatic (e.g., based on heuristics or component ownership). Investigation phase120may further include an investigation by the team stage126. As part of this stage, the relevant team may investigate the incident(s), and as part of engagement or reassignment stage128, may engage with the relevant stakeholders or re-route the incident(s) to the appropriate team. As part of investigation phase120, in problem identification stage132, the cause(s) of the problem(s) that resulted in the incident alert(s) may be identified.

With continued reference toFIG.1, once the appropriate team identifies the cause(s) of the problem(s) that resulted in the incident alert(s), the processing may move to triaging phase140. In this phase, the incident(s) may be triaged according to any prioritization scheme. Next, in reporting error(s)/bug(s) stage144, the appropriate error(s)/bug(s) related to the incident may be reported to the engineering teams. Next, in resolution phase150, the incident may be resolved as part of incident resolution stage152. Finally, as part of resolution phase150, during fixing error(s)/bug(s) stage154, any error(s) and/or bug(s) may be fixed such that the incidents caused by such error(s) and/or bug(s) do not recur. Other activities including root cause analysis may be pursued in parallel to ensure that incidents do not repeat in the future. AlthoughFIG.1shows a certain number of phases as part of lifecycle100that are arranged in a certain manner, lifecycle100may include additional or fewer phases. In addition, althoughFIG.1shows a certain arrangement of stages within each phase, the phases may include additional or fewer stages, which may be arranged differently.

FIG.2shows a block diagram of a machine learning pipeline200for automatically extracting entity names and data types related to cloud incidents. Machine learning pipeline200may include a storage210, which may store incident descriptions. As explained earlier, the incident descriptions may include various unstructured pieces of information that may be generated as a result of incident alerts. Storage210may also be used to store incident logs, telemetry data, and support tickets.

With continued reference toFIG.2, in this example, machine learning pipeline200may include several components, including preprocessing220, unsupervised data labeling230, label propagation240, and multi-task learning250. Machine learning pipeline200may be implemented using both offline training components and online prediction components. Offline training components may be responsible for training of the various machine language models, validating the models, and publishing the validated models.

Still referring toFIG.2, preprocessing220may be configured to process the incident descriptions and incident summaries, including applying a data cleaning process. Service incident descriptions and summaries may be created by various sources such as external customers, feature engineers, or automated monitoring systems. The incidents related information could be in various forms, such as textual statements, conversations, stack traces, shell scripts, images, etc. While each of these types of unstructured information may be difficult to process, these descriptions contain useful information. In this example, preprocessing220may include several steps. As an example, first, the tables in the incident descriptions that have more than two columns may be pruned. In addition, the HTML tags may be removed using regexes and HTML parsers. As part of preprocessing220, the incident descriptions and incidence summaries may be segmented into sentences using newline characters. Next, the individual sentences may be processed by cleaning up extra spaces and then they may be tokenized into words. The tokenization technique may be selected to handle even camel-case tokens (e.g., iPhone) and URLs as well.

Still referring toFIG.2, unsupervised data labeling230may include identifying a set of entity names and then using the identified entity names as labels for tagging individual tokens in every incident description from a selected dataset. Identification of the set of entity names may include identifying patterns232. Patterns232may include key value pairs (e.g., separated by a colon or a hyphen, such as key:value or key-value), tables, or any other data structure that can be used to represent relationships among keys, values, other such types of information. Patterns232may be extracted by identifying relationships in the incident descriptions. As an example, a key value pair in an incident description may be “Status code: 401.” In this example, the text preceding the colon may be extracted as the entity name—Status code—and the text following the colon may be extracted as the entity value—401. In another example, another key value pair in an incident description may be “Problem type: VM not found.” In this example, the text preceding the colon may be extracted as an entity name—Problem type—and the text following the colon may be extracted as the entity value—VM not found. Tables also occur quite frequently in the incident descriptions, especially the ones that are created by bots or by monitoring services. The text in the header tags ‘<th>’ may be extracted as the entity name and the values in the corresponding rows may be extracted as entity values.

Entity names may correspond to various cloud services. Table 1 below shows an example of cloud services and related entity names.

The initial candidate set of entity names and values may be noisy since pattern extraction232includes extracting almost all of the text that matches certain patterns. In certain examples, entity names may correspond to the category names (e.g., instance, people, location, etc.). To reduce noise in the initial candidate set, any entity names that contain symbols or numbers may be filtered out. To generate a more robust set of named-entities, n-grams (n: 1 to 3) may be extracted from the entity names of the candidates by selecting the top 100, or another number depending on the size of the data and other factors, most frequently occurring n-grams. In this process, less frequently used entity names (likely noisy candidate entity names) such as “token acquisition starts,” may be pruned. Also with the n-gram analysis, a candidate entity such as [“My Subscription ID is”, “6572”] may be transformed to [“Subscription ID”, “6572”] since “Subscription ID” is a commonly occurring bi-gram in the candidate set.

Next, as part of data type tagging236, for the refined entity name candidate set, the data type of the entity values may be determined. As an example, along with regexes, certain Python functions such as “isnumeric” may be used. The use of the data types may help improve the accuracy for the individual prediction tasks. An example set of data types may include the following data types: (1) basic types (e.g., numeric, Boolean, alphabetical, alphanumeric, non-alphanumeric); (2) complex types (e.g., GUID, URI, IP address, URL); and (3) other types (e.g., any data types that do not fit neatly into the basic or the complex types of data types). In one example, to arrive at the most likely data type, the data type may be determined for each instance of a named entity. Then, conflicts may be resolved by taking the most frequent type. For instance, if “VM IP” entity is most commonly specified as an IP Address but sometimes is specified as a Boolean, due to noise or dummy values, the data type may be resolved to be an IP Address. Table 2 below shows additional examples of entity names, the corresponding data types, and an example of each entity name.

Once the set of entity names is finalized, the incident descriptions may be parsed and each token in the incident descriptions may be tagged. As part of entity name tagging234, unsupervised machine learning algorithms may be used to tag the incident descriptions with entity names. An example of a tagged sentence, which may be part of an incident description, is shown in Table 3 below.

In Table 3, <O>, which may be viewed as <Other> or <Outside> refers to tokens that are not entities. The tagged sentences, such as the one shown in Table 3, may be used to create a labeled dataset that can be used to train the machine learning models used as part of multi-task learning250.

Referring back toFIG.2, machine learning pipeline200may further include label propagation240. Unsupervised data labeling230allows bootstrapping of the training data using the pattern extraction. While this allows the generation of a seed dataset, the recall may suffer since the entities could occur inline within the incident descriptions without the key-value pair patterns or tabular patterns. The absence of any ground truth or any labeled data poses a problem. In one example, label propagation240may be used to solve this challenge. Label propagation240may use unsupervised machine learning techniques to label the incident descriptions, which may then be used to train a deep learning based model. In this example, to avoid over-fitting the model on the specific patterns, the labels may be diversified as part of this process.

In this example, the entity names and values extracted in the bootstrapping process and their types may be propagated to an entire corpus of incident descriptions. As an example, if the IP Address “127.0.0.1” was extracted as a “Source IP” entity, then all un-tagged occurrences of “127.0.0.1” in the corpus may be tagged as “Source IP.” Certain corner cases may need to be handled differently. For instance, the aforementioned technique may not be usable for entities with the Boolean data type. As an example, an entity name may be “Is Customer Impacted” and the value may be “true” or “false.” In this case, all occurrences of the word true or false cannot be labeled as corresponding to the entity “Is Customer Impacted.” Label propagation240may also not work for all multi token entities, particularly the ones which are descriptive.

To the extent different occurrences of a particular value were tagged as different entities during bootstrapping, conflicts may be resolved using various techniques. As an example, an IP address (e.g., “127.0.0.1”) can be “Source IP” in one incident while it may be “Destination IP” in another incident. In this example, during label propagation240, such conflicts may be resolved based on popularity, (e.g., the value may be tagged with the entity name which occurs more frequently across the corpus). The frequency of occurrences may be tracked using histograms or other similar data structures.

Still referring toFIG.2, machine learning pipeline200may further include multi-task learning250. Multi-task learning250may automate the task of creating labeled data for deep learning models which can further generalize knowledge extraction. Multi-task learning may include an embedding layer252. Incident descriptions may be converted to word level vectors using an embedding layer252. As an example, an incident description may include words W1, W2, W3, and WN, which may be converted into vectors for further processing. Multi-task learning250may further include shared neural network layers254and task-specific layers. Multi-task learning250may solve two entity recognition tasks simultaneously—entity name recognition task (l1) and data type recognition task (l7). The entity name prediction is treated as the main task and data type prediction is treated as the auxiliary task. In this example, entity name recognition may include the use of shared neural network layers254and layers labeled as262,264, and266. In addition, in this example, data type recognition may include the use of shared neural network layers254and layers labeled as272,274, and276, In this example, layers262and272may comprise a time distributed dense layer460ofFIG.4B; layers264and274may comprise an attention layers470ofFIG.4B; and layers266and276may comprise a conditional random fields (CRF) layer480ofFIG.4B.

The losses may initially be calculated individually for both tasks, l1and l2, and then combined into losscusing a weighted sum. The parameter lossweights=(∝, β) may be used to control the importance between the main task and the auxiliary task as follows: lossc=∝×l1+β×l2. During the training, multi-task learning250may aim to minimize the losscbut the individual losses are back-propagated to only those layers that produced the output. With such an approach, the lower level common layers are trained by both tasks, whereas the task specific layers are trained by individual losses. Additional details concerning various components of machine learning pipeline200are provided later with respect toFIGS.4A and4B, AlthoughFIG.2shows certain components of machine learning pipeline200that are arranged in a certain manner, machine learning pipeline200may include additional or fewer components arranged differently. In addition, certain components of machine learning pipeline200may be used for training of the machine learning models and other components may be used for prediction tasks. Thus, machine learning pipeline200may include only one of these types of components or both of these types of components depending upon the functions being performed using such a pipeline.

FIG.3is a block diagram of a system300for performing methods associated with the present disclosure in accordance with one example. As an example, system300may be used to implement the various parts of machine learning pipeline200ofFIG.2. System300may include a processor(s)302, I/O component(s)304, memory306, presentation component(s)308, sensors310, database(s)312, networking interfaces314, and I/O port(s)316, which may be interconnected via bus320. Processor(s)302may execute instructions stored in memory306. Processor(s)302may include CPUs, GPUs, ASICs, FPGAs, or other types of logic configured to execute instructions. I/O component(s)304may include components such as a keyboard, a mouse, a voice recognition processor, or touch screens. Memory306may be any combination of non-volatile storage or volatile storage (e.g., flash memory, DRAM, SRAM, or other types of memories). Presentation component(s)308may include displays, holographic devices, or other presentation devices. Displays may be any type of display, such as LCD, LED, or other types of display. Sensor(s)310may include telemetry or other types of sensors configured to detect, and/or receive, information (e.g., conditions associated with the various devices in a data center). Sensor(s)310may include sensors configured to sense conditions associated with CPUs, memory or other storage components, FPGAs, motherboards, baseboard management controllers, or the like. Sensor(s)310may also include sensors configured to sense conditions associated with racks, chassis, fans, power supply units (PSUs), or the like. Sensor(s)310may also include sensors configured to sense conditions associated with Network Interface Controllers (NICs), Top-of-Rack (TOR) switches, Middle-of-Rack (MOR) switches, routers, power distribution units (PDUs), rack level uninterrupted power supply (UPS) systems, or the like.

Still referring toFIG.3, database(s)312may be used to store any of the data or files (e.g., incident descriptions or the like) as needed for the performance of methods described herein. Database(s)312may be implemented as a collection of distributed databases or as a single database. Network interface(s)314may include communication interfaces, such as Ethernet, cellular radio, Bluetooth radio, UWB radio, or other types of wireless or wired communication interfaces. I/O port(s)316may include Ethernet ports, Fiber-optic ports, wireless ports, or other communication ports.

Instructions corresponding to preprocessing220, unsupervised data labeling230, label propagation240, and multi-task learning250and their respective constituent parts may be stored in memory306or another memory. These instructions when executed by processor(s)302, or other processors, may provide the functionality associated with machine learning pipeline200. The instructions corresponding to machine learning pipeline200, and related components, could be encoded as hardware corresponding to an A/I processor. In this case, some or all of the functionality associated with the learning-based analyzer may be hard-coded or otherwise provided as part of an Ail processor. As an example, A/I processor may be implemented using a field programmable gate array (FPGA) with the requisite functionality. Other types of hardware such as ASICs and GPUs may also be used. The functionality associated with machine learning pipeline200may be implemented using any appropriate combination of hardware, software, or firmware. AlthoughFIG.3shows system300as including a certain number of components arranged and coupled in a certain way, it may include fewer or additional components arranged and coupled differently. In addition, the functionality associated with system300may be distributed or combined, as needed.

FIGS.4A and4Bshow a deep learning model400with a multi-head architecture in accordance with one example. In this example, deep learning model400may be used to implement certain aspects of multi-task learning250ofFIG.2. In this example, words and symbols extracted from an incident description may be first converted into a sequence of vectors. The sequence of vectors may be interpreted, both in a forward direction and in a reverse direction, by a Bi-directional Long Short-term Memory (LSTM) layer430. The two prediction tasks may include entity name prediction and data type prediction. The two tasks may be handled in a way that some common parameters and layers (e.g., layers within the box420ofFIG.4) may be shared for both tasks, but there may also be task specific layers (e.g., separate time distributed dense layers462and464, separate attention layers472and474, and separate conditional random fields (CRF) layers482and484). Although separate such layers are used, for ease of explanation, these layers are addressed using common reference numerals as shown inFIG.4B: time distributed dense layer460, attention layer470, and CRF layer480. Time distributed dense layer460may transpose the Bi-directional LSTM hidden vectors to the shape of the output labels. An attention mechanism (e.g., attention layer470) may help the model bias the learning towards the more relevant sections of the sentences. In addition, in this example, a conditional random fields (CRF) layer480may produce a valid sequence of output labels. As shown inFIG.4B, the output may include entity name prediction492and data type prediction494. Back propagation using a combination of loss functions may be performed during training and the individual tag precision may be evaluated using recall and F1 metrics.

In certain examples, by using the underlying common information contained among related tasks multi-task learning may be used to improve generalization. In the context of classification or sequence labelling, the multi-task learning may improve the performance of individual tasks by learning them jointly. In certain examples described herein, named-entity recognition is the primary task. In this task, the machine learning models may primarily learn from context words that support occurrences of entities. Incorporating a complimentary task of predicting the data type of a token may reinforce intuitive constraints, resulting in better training of the machine learning models. For example, in an input like “The SourceIPAddress is 127.0.0.1,” the token 127.0.0.1 is identified more accurately by the machine learning models described herein, as the entity name “Source IP Address” because it is also identified as the data-type “IP Address”, in parallel. In sum, the machine learning models supplement the intuition that all Source IP Addresses are of the data type IP addresses; thus, improving the model performance. Accordingly, in these examples data type prediction is used as the auxiliary task for the deep learning models. Various types of architectures may allow multi-task learning, including but not limited to, multi-head architectures, cross-snitch networks, and sluice networks. Certain examples described herein use a multi-head architecture, where the lower level features generated by the two neural network layers are shared, whereas the other layers are task specific.

As noted previously, the entity name prediction is treated as the main task and data type prediction is treated as the auxiliary task. The losses are initially calculated individually for both tasks, l1and l2, and then combined into losscusing a weighted sum. The parameter lossweights=(∝, β) may be used to control the importance between the main and the auxiliary task as follows: lossc=∝×l1+β×l2. During the training, deep learning model400aims to minimize the toss, but the individual losses are back-propagated to only those layers that produced the output. With such an approach, the lower level common layers are trained by both tasks, whereas the task specific layers are trained by individual losses.

With continued reference toFIG.4A, in this example, incident descriptions402may be converted to word level vectors using a pre-trained embedding layer410. As an example, an incident description may include words W1, W2, W3, and WN, which may be converted into vectors for further processing. Pre-trained embedding layer410may be implemented as a GloVe embedding layer or a word2vec embedding layer. GloVe relates to a model that captures linear substructure relations in a global corpus of words, revealing regularities in syntax as well as semantics. The GloVe model, trained on five different corpora, covers a vast range of topics and tokens. In this example, in a preferred embodiment, the 100 dimension version of GloVe may be used to create pre-trained embedding layer410with the pre-trained GloVe weights.

Vector size may be a 768-dimension vector or a 1024-dimension vector. Additional operations, including position embedding, sentence embedding, and token masking may also be performed as part of pre-trained embedding layer410. Position embedding may be used to identify token positions within a sequence. Sentence embedding may be used to map sentences to vectors. Token masking may include replacing a certain percentage of the words in each sequence with a mask token. These vectors may improve the performance of the prediction tasks being performed using deep learning model400. In this example, these vectors may act as characteristic features in named entity recognition being performed using deep learning model400.

Still referring toFIG.4A, Bi-directional LSTM network430may be implemented as one or more Recurrent Neural Networks (RNNs). An RNN maintains historic information extracted from a sequence or a series like data. This feature may enable an RNN-based model to make predictions at a certain time step, conditional to viewed history. Thus, an RNN may take a sequence of vectors (x1, x2, . . . , xn) as input and return as sequence of vectors (h1, h2, . . . , h3) that encodes information at every time step. Although RNNs are capable of encoding and learning dependencies that are spread over long time steps, at times they may fail to do so; this is because RNNs tend to be biased towards more recent updates in a long sequence of situations.

In one example, Long Short-term Memory (LSTM) networks may be used to capture long range dependencies using several gates. These gates may control a portion of the input and pass to the memory cell, and the portion from the previous hidden state to forget. An example LSTM network may comprise a sequence of repeating RNN layers or other types of layers. Each layer of the LSTM network may consume an input at a given time step, e.g., a layer's state from a previous time step, and may produce a new set of outputs or states. In the case of using the LSTM, a single chunk of content may be encoded into a single vector or multiple vectors. As an example, a word or a combination of words (e.g., a phrase, a sentence, or a paragraph) may be encoded as a single vector. Each chunk may be encoded into an individual layer (e.g., a particular time step) of an LSTM network. In this example, Bi-directional LSTM network430may include a first LSTM network440and a second LSTM network450. LSTM network440may be configured to process a sequence of words from left to right and LSTM network450may be configured to process a sequence of words from right to left. LSTM network440may include LSTM cell442, LSTM cell444, LSTM cell446, and LSTM cell448, which may be coupled to receive inputs and to provide outputs, as shown inFIG.4A. LSTM network450may include LSTM cell452, LSTM cell454, LSTM cell456, and LSTM cell458, which may be coupled to receive inputs and to provide outputs, as shown inFIG.4A. In addition, as shown inFIG.4A, both LSTM cell442and LSTM cell452may provide their output to hidden layer H1453. Likewise, both LSTM cell444and LSTM cell454may provide their output to hidden layer H2455. Similarly, both LSTM cell446and LSTM cell456may provide their output to hidden layer H3457. Finally, both LSTM cell448and LSTM cell458may provide their output to hidden layer HN459.

An example LSTM layer may be described using a set of equations, such as the ones below:
ft=σ(Wf·[ht-1xt]+bc)
it=σ(Wf·[ht-1xt]+bi)
{tilde over (c)}t=tanh(Wc·[ht-1xt]+bc)
ct=ft∘ct-1+it∘{tilde over (c)}t
ot=σ(Wo·[ht-1xt]+bo)
ht=ot∘ tanh(ct)
In this example, in the above equations a is the element wise sigmoid function and ∘ represents Hadamard product (element-wise). In this example, ft, it, and otare forget, input, and output gate vectors respectively, and ctis the cell state vector. Using the above equations, given a sentence as a sequence of real valued vectors (x1, x2, . . . , xn), the LTSM (e.g., LTSM network440ofFIG.4A) computes {right arrow over (h)}tthat represents the leftward context of the word at the current time step t. In this example, a word at the current time step t, receives context from other words that occur on either sides. Thus, a second LSTM (e.g., LSTM network450ofFIG.4A) interprets the same sequence in reverse, returningtat each time step. In this example, this combination of forward and backward LSTMs corresponds to Bi-directional LSTM network430. The final representation of the word may be produced by concatenating the left and right context, ht=[{right arrow over (h)}t;t]. In this example, inside each LSTM layer, the inputs and hidden states may be processed using a combination of vector operations (e.g., dot-product, inner product, or vector addition) or non-linear operations, if needed.

The instructions corresponding to the machine learning system could be encoded as hardware corresponding to an A/I processor. In this case, some or all of the functionality associated with the learning-based analyzer may be hard-coded or otherwise provided as part of an A/I processor. As an example, A/I processor may be implemented using an FPGA with the requisite functionality.

Any of the learning and inference techniques such as Linear Regression, Support Vector Machine (SVM) set up for regression, Random Forest set up for regression, Gradient-boosting trees set up for regression and neural networks may be used. Linear regression may include modeling the past relationship between independent variables and dependent output variables. Neural networks may include artificial neurons used to create an input layer, one or more hidden layers, and an output layer. Each layer may be encoded as matrices or vectors of weights expressed in the form of coefficients or constants that might have been obtained via off-line training of the neural network. Neural networks may be implemented as Recurrent Neural Networks (RNNs), Long Short Term Memory (LSTM) neural networks, or Gated Recurrent Unit (GRUs). All of the information required by a supervised learning-based model may be translated into vector representations corresponding to any of these techniques.

With reference toFIG.4B, deep learning model400for entity name and data type prediction from the incident descriptions or other sources of incidents' related information may include additional layers, including a time distributed dense layer460, an attention layer470, and a conditional random fields (CRF) layer480. Time distributed dense layer460may transpose the Bi-directional LSTM hidden vectors to the shape of the output labels. Attention layer470may help the model bias it is learning towards the more relevant sections of the sentences. In addition, CRF layer480may produce a valid sequence of output labels. As shown inFIG.4B, each of these layers may process outputs received from bi-directional LSTM network430.

Still referring toFIG.4B, time distributed dense layer460may be trained to reshape the vectors received from bi-directional LSTM network430. In this example, attention layer470may be implemented by using the Bidirectional Encoder Representations from Transformers (BERT) model. Attention layer470may take as input the hidden states from Bi-directional LSTM network430, after these inputs have been transposed to output dimensions using time distributed dense layer460. In this example, attention layer460may be implemented at the words level as a neural layer, with a weight parameter Wα. In one example, let h=(h1, h2, . . . , hT) be the input to the attention layer470, the attention weights and final representation h* of the sentence is formed as follows:
scores=WαTh
α=softmax(scores)
r=hαT
h*=tanh(r)

In the example equations shown above, the softmax and tan h functions are applied element-wise on the input vectors. The values corresponding to h and h* may be concatenated and passed to the next layer. In one example, attention layer460may include transformers corresponding to the BERT model. Transformers may convert input sequences into output sequences using self-attention. Transformers may be configured to have either 12 or 24 hidden (h) layers. Transformers may include fully-connected network (FCN) layers, including the EON (Query), EON (Key), and EON (Value) layers.

Referring now toFIG.5, a visual representation500of the degree of attention paid to various parts of an incident description is shown in accordance with one example. The attention vector a for a test sentence, shown inFIG.5, illustrates that the attention layer learns to give more emphasis to tokens that have a higher likelihood of being entities. The degree of attention varies from lower to higher. In this example, the different degrees of attention, from a lower degree of attention to a higher degree of attention, are shown as510,520,530,540,550,560, and570. In case of long sequences, the different degrees of attention to certain sections of the sequence, which are more likely to contain entities, helps improve the sensitivity of deep learning model400.

Referring back toFIG.4B, the use of the hidden state representations (ht) as word features to make independent tagging decisions at the word level may still leave the issue of inherent dependencies across the output labels unaddressed. For example, the entity names and corresponding values may have contextual or other types of constraints. Similarly, data types may be constrained in terms of the data types that are usable with certain entity names. In one example, by learning these dependencies and generalizing them to sentences without such constraints, the tagging decisions may be jointly modeled using conditional random fields as part of CRF layer480.

To explain one example implementation of CRF layer480, consider an input sequence X=(x1, x2, . . . , x3) and an output sequence y=(y1, y2, . . . , yn), where n is the number of words in the sentence. Assuming, for this example, P is the matrix of the probability scores of shape n×k, where k is the number of distinct tags in the output of bi-directional LSTM network430, including the dense and attention layers. In other words, in this example Pi,jis a score that the ithword corresponds to the jthtag. In this example, as part of CRF layer480, first a score is computed for the output sequence, y, using the example equation below:

s⁡(X,y)=∑i=0n⁢Ayi,yi+1+∑i=0n⁢P⁢i,yi
where A represents the matrix of transition scores. Thus, in this example, Ai,jis the score for the transition from tagito tagj. Then the score is converted to a probability for the sequence y to be the right output using a softmax over Y (all possible output sequences) using the example equation below:

In this example, the model corresponding to CRF layer480learns by maximizing the log-probability of the correct y. While extracting the tags for the input, the output sequence with the highest score is predicted using the following example equation:
y*=argmaxp(y′|X)
y′∈Y

Thus, in this example implementation of CRF layer480, CRF layer480and attention layer470push the model towards learning a valid sequence of tags. As an example, for a sentence that includes the entity name subscription ID and the entity value 12345 (separated by a colon), attention layer470may tag the colon as a tenant ID.

In one example, the hyper-parameters for the deep learning models may be set as follows: word embedding size is set to 100, the hidden LSTM layer size is set to 200 cells, and the maximum length of a sequence is limited to 300. These example hyper-parameters may be used with all models. The machine learning models may be trained using any set of computing resources, including using system300ofFIG.3. Each computing resource may be implemented using any number of graphics processing units (GPUs), computer processing units (CPUs), memory (e.g., SRAM or other types of memory), or field programmable gate arrays (FPGAs). Application Specific Integrated Circuits (ASICs), Erasable and/or Complex programmable logic devices (PLDs), Programmable Array Logic (PAL) devices, and Generic Array Logic (GAL) devices may also be used to implement the computing resources. In addition, althoughFIG.4Bdescribes the use of the BERT model for attention layer460, any serializable neural network model may be partitioned and used.

FIG.6shows a system environment for implementing a machine learning pipeline200for automatically extracting entity names and data types related to cloud incidents in accordance with one example. In this example, system environment600may correspond to a portion of a data center. As an example, the data center may include several clusters of racks including platform hardware, such as server nodes, storage nodes, networking nodes, or other types of nodes. Server nodes may be connected to switches to form a network. The network may enable connections between each possible combination of switches. As used in this disclosure, the term data center may include, but is not limited to, some or all of the data centers owned by a cloud service provider, some or all of the data centers owned and operated by a cloud service provider, some or all of the data centers owned by a cloud service provider that are operated by a customer of the service provider, any other combination of the data centers, a single data center, or even some clusters in a particular data center, System environment600may include server1610and serverN630. System environment600may further include data center related functionality660, including deployment/monitoring670, directory/identity services672, load balancing674, data center controllers676(e.g., software defined networking (SDN) controllers and other controllers), and routers/switches678. Server1610may include host processor(s)611, host hypervisor612, memory613, storage interface controller(s) (SIC(s))614, cooling615, network interface controller(s) (NIC(s))616, and storage disks617and618. ServerN630may include host processor(s)631, host hypervisor632, memory633, storage interface controller(s) (SIC(s))634, cooling635, network interface controller(s) (MC(s))636, and storage disks637and638.

With continued reference toFIG.6, server1610may be configured to support virtual machines, including VM1619, VM2620, and VMN621. The virtual machines may further be configured to support applications, such as APP1622, APP2623, and APPN624. ServerN630may be configured to support virtual machines, including VM1639, VM2640, and VMN641. The virtual machines may further be configured to support applications, such as APP1642, APP2643, and APPN644. Each of server1610and serverN630may also support various types of services, including file storage, application storage, and block storage for the various tenants of the cloud service provider responsible for managing system environment. In this example, system environment600may be enabled for multiple tenants using the Virtual eXtensible Local Area Network (VXLAN) framework. Each virtual machine (VM) may be allowed to communicate with VMs in the same VXLAN segment. Each VXLAN segment may be identified by a VXLAN Network Identifier (VNI).

Deployment/monitoring670may interface with a sensor API that may allow sensors to receive and provide information via the sensor API. Software configured to detect or listen to certain conditions or events may communicate via the sensor API any conditions associated with devices that are being monitored by deployment/monitoring670. Remote sensors or other telemetry devices may be incorporated within the data centers to sense conditions associated with the components installed therein. Remote sensors or other telemetry may also be used to monitor other adverse signals in the data center and feed the information to deployment/monitoring670. As an example, if fans that are cooling a rack stop working then that may be sensed by the sensors and reported to the deployment/monitoring670. AlthoughFIG.6shows system environment600as including a certain number of components arranged and coupled in a certain way, it may include fewer or additional components arranged and coupled differently. In addition, the functionality associated with system environment600may be distributed or combined, as needed. Moreover, althoughFIG.6shows VMs, other types of compute entities, such as containers, micro-VMs, microservices, unikernels for serverless functions, may be supported by the host servers in a like manner.

FIG.7shows a layout700for an incident description in accordance with one example. Layout700may correspond to an incident description being displayed, or otherwise being communicated, to a person/team assigned to address the incident at Issue. Layout700may include user interface elements to allow interaction. As an example, the following menu options may be associated with layout700of the example incident description: Details702, Diagnostics704, Notifications706, Postmortem708, Activity Log (History)710, and Similar Incidents712. When a user selects Details702menu option, the information displayed in box720may be displayed. The example incident description shown in layout700relates to an issue with a virtual machine (VM) in a failed state. Additional details associated with the incident description are shown in box720. Although example layout700shows certain aspects associated with an incident description, other incident descriptions may have a different layout and may include information other than shown in layout700. Table 4, below, shows entity names and entity values for layout700.

FIG.8shows another layout800for an incident description in accordance with one example. Layout800may correspond to another incident description being displayed, or otherwise being communicated, to a person/team assigned to address the incident at issue. Layout800may also include user interface elements to allow interaction. As an example, similar to layout700, the following menu options may be associated with layout800of the example incident description: Details802, Diagnostics804, Notifications806, Postmortem808, Activity Log (History)810, and Similar Incidents812. When a user selects Details802menu option, the information displayed in box820may be displayed. The example incident description shown in layout800relates to an issue with an error associated with a virtual network (Vnet). Additional details associated with the incident description are shown in box820. Although example layout800shows certain aspects associated with an incident description, other incident descriptions may have a different layout and may include information other than shown in layout800. Table 5, below, shows entity names and entity values for layout800.

FIG.9shows a flow chart900of a method, implemented by at least one processor, for processing cloud incidents related information, including entity names and entity values associated with incidents having a potential to adversely impact products or services offered by a cloud service provider. Step910may include using the at least one processor (e.g., processor(s)302ofFIG.3), processing the cloud incidents related information to convert at least words and symbols corresponding to a cloud incident description into machine learning formatted data. As explained earlier, with respect toFIGS.2-4B, using a pre-trained embedding layer (e.g., pre-trained embedding layer410) words and symbols corresponding to the cloud incident may be converted into machine learning formatted data. As an example, the words and symbols may be converted into vector data for processing by neural networks.

Step920may include using a machine learning pipeline, processing at least a subset of the machine learning formatted data to recognize entity names and entity values associated with the cloud incident. As explained earlier, with respect toFIGS.2-4B, the machine learning formatted data (e.g., vector data) may be processed to recognize entity names and entity values.

FIG.10shows a flow chart1000of a method, implemented by at least one processor, for processing cloud incidents related information, including entity names, entity values, and data types associated with incidents having a potential to adversely impact products or services offered by a cloud service provider. Step1010may include using the at least one processor (e.g., processor(s)302ofFIG.3), processing the cloud incidents related information to convert at least words and symbols corresponding to a cloud incident description into machine learning formatted data. As explained earlier, with respect toFIGS.2-4B, using a pre-trained embedding layer (e.g., pre-trained embedding layer410) words and symbols corresponding to the cloud incident may be converted into machine learning formatted data. As an example, the words and symbols may be converted into vector data for processing by neural networks.

Step1020may include using a first machine learning pipeline, as part of a first prediction task, processing at least a subset of the machine learning formatted data to recognize entity names and entity values associated with the cloud incident. As explained earlier, with respect toFIGS.2-4B, at least a subset of the machine learning formatted data (e.g., vector data) may be processed to recognize entity names and entity values.

Step1030may include using a second machine learning pipeline, as part of a second prediction task, processing at least a subset of the machine learning formatted data to recognize data types associated with the cloud incident. As explained earlier, with respect toFIGS.2-4B, the machine learning formatted data (e.g., vector data) may be processed to recognize data types.

In one example, machine learning pipeline200and the corresponding deep learning model for entity name recognition and data type recognition may be deployed as part of system environment600. As an example, machine learning pipeline200and the corresponding deep learning model may be deployed as a REST API (e.g., a REST API developed using the Python Flask web app framework). The REST API may offer a POST endpoint which takes the incident description as input and returns the recognized entities in JSON format. The deployment of the REST API in system environment600advantageously allows automatically scaling up of the service in response to demand variation. This enables the service to be cost efficient since the majority of the incidents are created during the day. In addition, deployment and monitoring tools in conjunction with machine learning pipeline200may enable application monitoring, as part of which service latency or failure issues may be communicated via alerts.

By efficiently recognizing entity names, entity values, and data types, systems and methods described in the present disclosure may enable other applications, as well. As an example, these systems and methods may be used for incident triaging. Advantageously, the recognized entity names and the recognized data types may reduce the feature space because a significant amount of unstructured information in the incident descriptions is not helpful. This may further help in creating incident summaries that are concise and yet informative for a service team. As a result, instead of parsing the verbose incident descriptions, the service team member may quickly analyze the concise summary and act on it, as required, per service agreements and protocols.

In addition, automated health checks may also be performed, alleviating the need for the service team member to review detailed telemetry data and logs. As an example, oversubscription (or undersubscription) of resources may be automatically identified using the automated health checks.

In conclusion, the present disclosure relates to a method, implemented by at least one processor, for processing cloud incidents related information, including entity names and entity values associated with incidents having a potential to adversely impact products or services offered by a cloud service provider. The method may include using the at least one processor, processing the cloud incidents related information to convert at least words and symbols corresponding to a cloud incident into machine learning formatted data. The method may further include using a machine learning pipeline, processing at least a subset of the machine learning formatted data to recognize entity names and entity values associated with the cloud incident.

The method may further include using the machine learning pipeline, jointly processing at least a second subset of the machine learning formatted data with the at least the subset of the machine learning formatted data to recognize data types associated with the cloud incident. The method may further include using a multi-task learning layer, processing both the subset of the machine learning formatted data and the second subset of the machine learning formatted data to generate output data.

The method may further include: (1) using a first time distributed dense layer, reshaping a first subset of the output data, wherein the first subset of the output data corresponds to entity names and entity values, to generate a first set of reshaped data and (2) using a second time distributed dense layer reshaping a second subset of the output data, wherein the second subset of the output data corresponds to data types, to generate a second set of reshaped data. The method may further include: (1) using a first attention layer, processing the first set of reshaped data, emphasizing a first set of tokens more likely to be entity names or entity types and (2) using a second attention layer, processing the second set of reshaped data, emphasizing a second set of tokens more likely to be data types.

The method may further include (1) using learned constraints associated with entity names and entity values, helping recognize the entity names and the entity values associated with the cloud incident, and (2) using learned constraints associated with data types, helping recognize the data types associated with the cloud incident. The method may further include generating a seed database of tagged entity names and tagged entity values by unsupervised tagging of entity names and entity values based on patterns extracted from cloud incidents related information. The method may further include using unsupervised label propagation of the tagged entity names and the tagged entity values, to generate training data for training the machine learning pipeline.

In another example, the present disclosure relates to a system, including at least one processor, for processing cloud incidents related information, including entity names and entity values associated with incidents having a potential to adversely impact products or services offered by a cloud service provider. The system may be configured to using the at least one processor, process the cloud incidents related information to convert at least words and symbols corresponding to a cloud incident into machine learning formatted data. The system may further be configured to using a machine learning pipeline, process at least a subset of the machine learning formatted data to recognize entity names and entity values associated with the cloud incident.

The system may further be configured to jointly process at least a second subset of the machine learning formatted data with the at least the subset of the machine learning formatted data to recognize data types associated with the cloud incident. The system may further be configured to using a multi-task learning layer, process both the subset of the machine learning formatted data and the second subset of the machine learning formatted data to generate output data.

The system may further be configured to: (1) using a first time distributed dense layer, reshape a first subset of the output data, wherein the first subset of the output data corresponds to entity names and entity values, to generate a first set of reshaped data and (2) using a second time distributed dense layer reshape a second subset of the output data, wherein the second subset of the output data corresponds to data types, to generate a second set of reshaped data. The system may further be configured to: (1) using a first attention layer, process the first set of reshaped data, emphasizing a first set of tokens more likely to be entity names or entity types and (2) using a second attention layer, process the second set of reshaped data, emphasizing a second set of tokens more likely to be data types. The system may further be configured to: (1) using learned constraints associated with entity names and entity values, help recognize the entity names and the entity values associated with the cloud incident, and (2) using learned constraints associated with data types, help recognize the data types associated with the cloud incident.

In yet another example, the present disclosure relates to a method, implemented by at least one processor, for processing cloud incidents related information, including entity names, entity values, and data types associated with incidents having a potential to adversely impact products or services offered by a cloud service provider. The method may include using the at least one processor, processing the cloud incidents related information to convert at least words and symbols corresponding to a cloud incident into machine learning formatted data. The method may further include using a first machine learning pipeline, as part of a first prediction task, processing at least a subset of the machine learning formatted data to recognize entity names and entity values associated with the cloud incident. The method may further include using a second machine learning pipeline, as part of a second prediction task, processing at least a subset of the machine learning formatted data to recognize data types associated with the cloud incident.

The method may further include using a multi-task learning layer, processing both the first subset of the machine learning formatted data and the second subset of the machine learning formatted data to generate output data. The method may further include: (1) using a first time distributed dense layer, reshaping a first subset of the output data, wherein the first subset of the output data corresponds to entity names and entity values, to generate a first set of reshaped data and (2) using a second time distributed dense layer reshaping a second subset of the output data, wherein the second subset of the output data corresponds to data types, to generate a second set of reshaped data.

The method may further include: (1) using a first attention layer, processing the first set of reshaped data, emphasizing a first set of tokens more likely to be entity names or entity types and (2) using a second attention layer, processing the second set of reshaped data, emphasizing a second set of tokens more likely to be data types. The method may further include: (1) using learned constraints associated with entity names and entity values, helping recognize the entity names and the entity values associated with the cloud incident, and (2) using learned constraints associated with data types, helping recognize the data types associated with the cloud incident. The method may further include: (1) generating a seed database of tagged entity names and tagged entity values by unsupervised tagging of entity names and entity values based on patterns extracted from cloud incidents related information, and (2) using unsupervised label propagation of the tagged entity names and the tagged entity values to generate training data for training the machine learning pipeline.

It is to be understood that the methods, modules, and components depicted herein are merely exemplary. Alternatively, or in addition, the functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-a-Chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc. In an abstract, but still definite sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or inter-medial components. Likewise, any two components so associated can also be viewed as being “operably connected,” or “coupled,” to each other to achieve the desired functionality.

The functionality associated with some examples described in this disclosure can also include instructions stored in a non-transitory media. The term “non-transitory media” as used herein refers to any media storing data and/or instructions that cause a machine to operate in a specific manner. Exemplary non-transitory media include non-volatile media and/or volatile media. Non-volatile media include, for example, a hard disk, a solid-state drive, a magnetic disk or tape, an optical disk or tape, a flash memory, an EPROM, NVRAM, PRAM, or other such media, or networked versions of such media. Volatile media include, for example, dynamic memory such as DRAM, SRAM, a cache, or other such media. Non-transitory media is distinct from, but can be used in conjunction with transmission media. Transmission media is used for transferring data and/or instruction to or from a machine. Exemplary transmission media include coaxial cables, fiber-optic cables, copper wires, and wireless media, such as radio waves.

Although the disclosure provides specific examples, various modifications and changes can be made without departing from the scope of the disclosure as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure. Any benefits, advantages, or solutions to problems that are described herein with regard to a specific example are not intended to be construed as a critical, required, or essential feature or element of any or all the claims.