Classifying errors in a failure log

Techniques are disclosed relating to a method that includes accessing, by a failure management program, a failure log that includes a plurality of character strings corresponding to errors that are associated with execution of one or more batch processes. The failure management program may compare a particular character string of the plurality of character strings to a set of character strings that are associated with respective ones of a plurality of failure categories. This comparing may include determining whether particular keywords that are included in respective ones of the set of character strings are included in the particular character string. In response to the comparing, the failure management program may assign a particular error corresponding to the particular character string to a particular failure category, or may determine a new failure category if the particular character string does not match an existing failure category.

RELATED APPLICATIONS

The present application claims priority under 35 U.S.C. § 119 to Indian patent application IN 202041016158, filed Apr. 14, 2020, the disclosure of which is incorporated by reference herein in its entirety.

BACKGROUND

Technical Field

This disclosure relates generally to computer system operation, and more particularly to classifying and resolving batch execution failures within a computer system.

Description of the Related Art

A business, government agency, school system, or any other form of large, multi-user entity may rely on an enterprise computing system to provide a computing infrastructure for a plurality of users. An enterprise computing system typically includes one or more server computers to provide computing power and one or more databases to provide network accessible information storage for these users. One or more system administrators, or admins for short, may be responsible for maintaining the enterprise computer system. As part of maintenance, one or more batch processes may be scheduled to perform a series of jobs as resources are available, with each job including one or more tasks. For example, a batch process may comb through databases in the computer system to identify particular types of data, such as sensitive data that should be stored in compliance with a set of security rules. In another example, a batch process may analyze files stored on one or more storage devices to identify redundancies that may be eliminated to increase storage capacity. An example job may include accessing files in one particular database or on one particular storage device. Tasks associated with such jobs may include, for example, performing a login operation to a particular database, reading a particular directory in the particular database, and the like.

During an execution of a batch file, an error may occur. For example, as part of a login task, the login credentials may be wrong due to another user or admin changing a username or a password expiring. An access error may occur if a path to directory to be analyzed has changed. Each occurrence of an error may be captured in a failure log. In some cases, a single issue may cause a plurality of jobs to generate errors, thereby creating a plurality of errors to be captured in the failure log. In a large enterprise computer system, a large number of errors may be captured in a failure log within a short period of time. In some cases, many thousands of errors may be captured within a day or even an hour.

DETAILED DESCRIPTION

In an enterprise computer system, execution of batch processes may generate errors that are then captured in one or more failure logs, each logged error corresponding to an event that causes the error. As used herein, an “event” refers to an occurrence that results in one or more errors and/or other notifications being generated. An “error” refers, herein, to a failure of a job or task to complete successfully. A “notification” refers to a message received as a result of performing a job or task, but that does not prevent the job or task from completing successfully. Errors and notifications may be captured in a respective entry in a failure log. Each event has a cause, e.g., an expired password may cause one or more login errors, a change in the path to the directory where the file is stored may cause a “file not found” error, etc. In some cases, one particular cause may be responsible for a plurality of errors. If a name of a high-level directory is changed, then the path to all files within that directory and any subdirectories will be changed. If multiple tasks are used for each subdirectory, then one entry may be generated in the failure log for each subdirectory beneath the high-level directory. In an enterprise computer system, a number of errors that may occur from execution of a single batch file may be quite large. Multiple batch files executing across multiple domains and databases may generate a number of logged errors and notifications for an admin or even a team of admins to handle.

The inventors have recognized a benefit to having an automated process for reviewing, classifying, and resolving errors logged into one or more failure logs. By automating management of failure logs, admins may have spare bandwidth for other activities, such as system maintenance, preventative care of the computer system, and system enhancements.

Techniques are disclosed herein for accessing a failure log that includes a plurality of logged entries that are associated with execution of one or more batch processes. A process such as a failure classification process may compare a character string associated with a particular logged entry to a set of character strings that are associated with respective failure categories. Based on the comparing, the process may assign the particular logged entry to a particular one of the failure categories. A process such as a failure remediation process may then determine, based on analyzing one or more errors associated with the particular failure category, a remediation action corresponding to the particular logged entry, and then apply the remediation action to the particular entry. If the particular logged entry cannot be associated with an existing failure category, then the failure classification process may perform a similarity analysis of the character string of the particular logged entry to one or more ungrouped character strings that do not correspond to any of the set of character strings. Based on the similarity analysis, the failure classification process may add a new failure category to the plurality of failure categories, where the new failure category includes the particular logged entry.

A block diagram of an embodiment of a computing system is illustrated inFIG.1. As shown, computer system100may correspond to a server computer system, personal computer system, desktop computer, laptop or notebook computer, mainframe computer system, server computer system operating in a datacenter facility, tablet computer, handheld computer, workstation, network computer, etc. Computer system100includes batch process150that may generate entries120a-120c(collectively120) with respective character strings122a-122c(collectively122). Computer system100also includes failure management program102that further includes failure classification process105and failure remediation process125. Failure classification process105accesses failure log110and failure categories130a-130c(collectively130), while remediation process125may activate batch process150, or a portion thereof.

As illustrated, failure management program102, executing on computer system100, calls failure classification process105which accesses failure log110. Failure log110includes a plurality of character strings122corresponding to entries120that are associated with execution of one or more batch processes, such as batch process150. Batch process150includes a plurality of jobs to be performed by computer system100. The jobs may be performed within computer system100, or may be performed by computer system100on one or more other computer systems that are communicatively coupled to computer system100. Each job corresponds to any type of computer-performed task or set of tasks that may executed on a computer system. For example, some jobs may include scanning a particular database for computer viruses or for security rule compliance. A particular job may be configured to scan an entire database, files associated with one user's account on the database, a particular type of data, or any other suitable level of detail. When an error occurs, batch process150receives an associated error message and, in transaction111, creates a new entry in failure log110. The new entry is assigned an identification as an entry120and stores the error message as an associated character string122. Entry120may include any suitable information to link the entry to a particular job and/or task within batch process150.

It is noted that entries120that are generated in failure log110may not always correspond to errors. In some cases, performance of a job may result in a notification even when the job is performed successfully. For example, a login job may result in a notification that a passcode used in the login job will expire in the next two weeks. Such a notification may, in some embodiments, be treated as an error by generating an entry120in failure log110. In such embodiments, use of the term error can include references to notifications.

Failure classification process105, as shown, analyzes entries120in order to classify each of entries120a-120cinto one of failure categories130. Failure categories130may correspond to any suitable classification of a logged failure associated with batch process150. For example, a batch process that is configured to scan a database for security compliance of data objects associated with a particular user account may generate errors such as invalid login credentials for accessing the data objects, particular data objects not found, invalid command, disk read errors, disk write errors, and the like. Failure classifications may be established by a system administrator or may be generated by failure classification process105using keywords included in character strings122.

To select a particular one of failure categories130, failure classification process105compares a particular character string (e.g., character string122b) of the plurality of character strings122to a set of character strings133a-133c(collectively133) that are associated with respective ones of failure categories130. In some embodiments, failure classification process105compares each of character strings122, one at a time, to the set of character strings133to identify similarities that are indicative of the corresponding entry120being associated with a respective one of failure categories130. Failure classification process105, in various embodiments, may analyze failure log110in response to an indication that a new entry120has been added, or in response to an indication that batch process150has completed, or in response to other similar actions.

The comparing includes determining whether particular keywords that are included in respective ones of character strings133are included in character string122b. In addition, the comparing may further include determining a frequency of usage of the particular keywords in character string122b. For example, character string122bmay include one use of the term “database,” while one of character strings133(e.g., character string133a) may include three uses of the term. In some embodiments, failure classification process105may not associate character string122bwith character string133adue to the difference in the frequency of usage of the term “database.”

Based on the comparing, failure classification process105assigns entry120b, corresponding to character string122b, to failure category130cof the plurality of failure categories130. In response to identifying similar keyword usage between character string122band character string133c, failure classification process105determines that entry120bbelongs to failure category130c. In addition to keyword usage, failure classification process may identify other similarities between character string122band character string133c, such as a common database being accessed, and/or common types of jobs being performed. After an entry120is associated with a particular one of failure categories130, the entry120may be considered an error. Although referred to as errors after an assignment, in some embodiments, one or more failure categories130may be utilized to classify entries that are not associated with an actual error, such as a notification of a soon to expire password. Additional details regarding the comparison of character strings in a failure log and character strings133associated with failure categories is disclosed below in regards toFIG.3.

In response to assigning entry120bto failure category130c, failure classification process105sends entry120b(now identified as categorized error121b) to failure remediation process125. In some embodiments, failure classification process105generates categorized error121bas an output to a different process in failure management program102, and this different process, e.g., a main process or a scheduling process, calls failure remediation process125, including categorized error121bin the call. As illustrated, failure remediation process125determines that remediation action126corresponds to categorized error121bby analyzing one or more errors associated with failure category130c. In some embodiments, determining a remediation action may include determining, at a first point in time, that no remediation action has been defined for failure category130c. Failure remediation process125waits for a remediation action to be defined, and, at a second point in time, determines that remediation action126has been defined for failure category130c. For example, at the first time, remediation action126has not been created and/or configured for use by failure remediation process125due to a lack of a sufficient number of categorized errors121having been assigned to failure category130c. The addition of categorized error121bmay provide a sufficient number of errors assigned to failure category130cto allow remediation process125, a different process running on computer system100, or a system administrator, or a combination thereof, to identify one or more actions that may overcome, mitigate, work around, or otherwise address a failure mechanism that resulted in a plurality of categorized errors121assigned to failure category130cfrom occurring.

Once remediation action126is available for use, failure remediation process125applies remediation action126to categorized error121b. As shown, applying remediation action126to categorized error121bincludes creating, by failure remediation process125, one or more child processes to perform tasks associated with remediation action126. For example, if errors in failure category130care related to a disk full error in a particular database, then remediation action126may be to perform a compaction operation on the database to remove obsolete and/or redundant data objects, thereby freeing space in the database to allow one or more tasks included in batch process150corresponding to categorized error121bto be re-executed. In the case of a particular failure category130that is associated with entries120that are not actual errors, the remediation action may be to ignore the entries assigned to the particular failure category130.

In response to determining that remediation action126has completed, failure remediation process updating may update a respective state of categorized error121b. Each entry120may include a state value that indicates where in the remediation process that a particular entry120currently resides. For example, possible states for an entry may include “open,” “in progress,” “ignore,” “resolved,” and “invalid.” “Open” may be a first state that a particular entry is assigned. The particular entry remains in the open state until it is assigned to a failure category130. “In progress” is assigned to entries that have been assigned, but for which no remediation action has been performed. An entry may be placed in the “ignore” state if a failure mechanism associated with the failure category130cannot be remedied within a particular amount of time and no other remediation action is available. For example, a particular failure category may be associated with a storage device that has been taken offline due to a malfunction. Entries120associated with jobs that are directed to this storage device may be ignored until the storage device is replaced or repaired. Once a remediation action has been successfully performed for a given entry, the given entry is updated to the “resolved” state. In some cases, an entry may be created in response to a message that is not related to a failure. For example, a login task may receive a notification that a password used will expire at a future point in time (e.g., “Password will expire in 7 days.”). For a current performance of batch process150, such a notification is not related to an actual error, although a new entry may be logged into failure log110due to receiving the message. Accordingly, such an entry is given a state of “invalid” and may be ignored.

Implementation of failure management program102in a computer system may help to reduce workloads for a system administrator or team of system administrators. By classifying errors and then implementing a respective remediation action to address the error, consumption of a system administrator's bandwidth may be avoided, allowing the system administrator to work on other activities. Additionally, performing the classifying and remediation processes by the computer system may result in a reduction in time to achieve a successful execution of batch process150, which, in turn, may increase an available bandwidth of computer system100.

It is noted that the embodiment ofFIG.1is merely an example for demonstration of disclosed concepts. In other embodiments, the illustrated computing system may include a different combination of elements, and/or a different distribution of tasks between elements. For example, failure classification process105and failure remediation process125are each described as performing particular tasks. In other embodiments, the described tasks assigned to these processes may be different, such as a single process performing all of the described tasks. Other embodiments may include additional processes. For example, failure remediation process125is described as assigning states to entries120. In other embodiments, a different process may be included that maintains state information for each entry120in failure log110.

The enterprise computing system ofFIG.1illustrates how a computing system may classify a log entry and determine a remediation action for an error identified by the entry. In some cases, a failure category may not exist for a given entry. InFIG.2, an example of a technique for identifying uncategorized entries and determining a new failure category for a subset of the uncategorized entries is shown.

Moving toFIG.2, a block diagram of another embodiment of computer system100is shown, in which a log entry is not matched to an existing failure category. Computer system100includes failure log110, batch process150, failure categories130, and failure management program102as previously shown inFIG.1. Failure management program102includes failure classification process105, as before, and further includes similarity analysis process235. As illustrated, similarity analysis process235is used to add new failure category230din response to identifying an unmatched log entry.

As illustrated, failure management program102, executing on computer system100, accesses failure log110that includes a plurality of character strings122corresponding to entries120that are associated with execution of batch process150. In various embodiments, batch process may be performed by computer system100or by a different computer system. As batch process150is performed, errors or other issues may occur that are then captured as entries120in failure log110. A given entry120includes a respective character string122that is indicative of the nature of the given entry120, including, for example, an error code, an error message, a time indicative of when the error occurred, a job and/or task that was being performed when the error occurred, and the like. As shown, failure management program102uses failure classification process105to access failure log110. In other embodiments, however, other processes included in failure management program102may be used for accessing failure log110.

Failure management program102uses failure classification process105to compare a particular character string (e.g., character string122c) of the plurality of character strings122to a set of character strings133that are associated with respective ones of a plurality of failure categories130. As shown, failure classification process105attempts to match entry120cto one of failure categories130based on character strings133. The comparing includes determining whether particular keywords that are included in respective ones of character strings133are included in character string122c. The comparison may also include determining a frequency of usage of the particular keywords in character string122c, and comparing the frequency to the frequency of usage of similar keywords in character strings133.

In response to character string122cnot corresponding to any of character strings133, failure management program102performs a similarity analysis of character string122cto one or more ungrouped character strings222that do not correspond to any of character strings133. The similarity analysis includes determining frequencies of occurrence for a plurality of keywords. After failing to match character string122cto any of character strings133, failure classification process105sends character string122c, as ungrouped character string123c, to similarity analysis process235. Similarity analysis process235tracks one or more ungrouped character strings222that failure classification process105has also failed to match to any of character strings133. Similarity analysis process235compares keywords identified in ungrouped character string123cto keywords associated with ones of ungrouped character strings222. Keywords may be identified using any suitable method, such as identifying words in ungrouped character string123cand ungrouped character strings222, that are not common across all strings. For example, the word “error” may appear in many of the character strings, while the word “access” may be found in less than half of the character strings. In this example, “access” may be identified as a keyword while “error” is excluded. In addition, some simpler words, for example, articles “a,” “an,” and “the,” may be excluded from use as keywords.

In some embodiments, keyword frequencies of usage may be used to generate string vectors for each of ungrouped character string123cand ungrouped character strings222that are indicative of keyword usage in a corresponding character string. A technique, such as cosine similarity, may be used to compare string vectors from various character strings and identify two or more of the compared character strings that are similar. If a subset of the compared character strings have adequate similarities, then the subset may be grouped to form new failure category230d.

As illustrated, failure management program102adds, in transaction237, new failure category230dto the plurality of failure categories130. The addition may be performed by similarity analysis process235or a different process included in failure management program102. New failure category230dmay include entry120ccorresponding to ungrouped character string123c. In addition, relevant keywords that are included in the subset of character strings are identified as character strings233dand are associated with new failure category230d. When a next character string122from failure log110is compared to character strings133, character string233dis included in the comparison.

In some embodiments, failure management program102generates a notification that new failure category230dhas been added. This notification may be sent to one or more system administrators who may then analyze the errors assigned to new failure category230dand identify a remediation action for these errors and any errors that are assigned to new failure category230din the future.

As previously described, implementation of failure management program102in a computer system may help to reduce workloads for one or more system administrators. By identifying new failure categories for errors that are not associated with existing failure categories, computer system100may manage execution of one or more batch processes while avoiding consumption of a system administrator's bandwidth, while additionally resulting in a reduction in time to achieve a successful execution of batch process150.

It is noted that the system ofFIG.2is merely an example. The number of entries in failure log110and the number of failure categories130are selected for clarity. In other embodiments, any suitable number of entries and failure categories may be included. Two processes are illustrated as being included in failure management program102, failure classification process105and similarity analysis process235. In other embodiments, additional process may be included, such as failure remediation process125inFIG.1. Alternately, these various processes could be implemented as a single process.

It is also noted that the techniques described inFIGS.1and2may be implemented, in part or in whole, as program instructions included a software program, such as failure management program102. Such a software program may be stored in a non-transitory, computer-readable medium having program instructions stored thereon that are executable by the computing device to cause the operations described with reference to the processes shown inFIGS.1and2.

In the descriptions ofFIGS.1and2, a failure log is described for use in capturing errors (including non-error notifications) associated with performance of one or more batch processes. The failure log is described as including a plurality of entries with associated character strings. Furthermore, a character analysis is described in regards to comparing an ungrouped character string to a plurality of other ungrouped character strings. Examples of a failure log and a character analysis are presented below in regards toFIG.3.

Turning toFIG.3, two tables are shown, one depicting a failure log, and the other depicting a character analysis. Failure log110includes eight entries, entry320ato entry320h, that associated with errors that occur during execution of one or more batch processes, such as batch process150. Five columns are shown for each entry320. The column for entry320includes an identifying value for each entry. Character string322includes a string of characters associated with each entry320. Domain340includes an identifier for a particular domain into which a batch process was logged when the associated error occurred. Database342identifies a particular database that was being accessed during an associated error. It is noted, that not all errors may be associated with a particular database, resulting in this field not having a valid database identifier for such cases. As shown, only entries320band320ginclude values for database342. The last illustrated column is job type344that identifies a type of job being performed at a time of the associated error.

As described above, failure classification process105compares character string122bto the set of character strings133to determine that entry120bshould be assigned to failure category130c. As part of this comparison, failure classification process105may compare more than just character strings. Failure classification process105may include additional information associated with entry120b, such as respective values for domain340, database342, and job type344, and compare this additional information to similar information associated with errors included in each of the failure categories130. For example, failure classification process105may use the character string comparison to narrow a selection down to two or more of failure categories130, and then use the additional information associated with errors in the two or more failure categories130to narrow the selection down to a single failure category130.

As an example, failure classification process105, based on the comparing, may determine that entry120bis associated with a same database342as at least some other errors that correspond to failure category130c. For example, entry120bmay correspond to entry320bwhich corresponds to an access of database342a. One or more of the errors associated with failure category130cmay also have occurred while accessing database342a. Failure classification process105may determine that entry120bhas a same job type344as at least some other errors that correspond to failure category130c. In a similar manner, failure classification process105may determine, based on the comparing, that entry120bis associated with a same domain340as at least some other errors that correspond to failure category130c. Accordingly, data other than character strings may be captured in failure log110and used to select a particular one of failure categories130to assign to a given one of entries320.

As described above in regards toFIG.2, if a particular one of failure categories130is not matched to a given entry320, then the uncategorized entry320is passed to similarity analysis process235. As illustrated, character analysis370corresponds to at least a portion of the analysis that is performed by similarity analysis process235. Character analysis370includes a table with three columns that includes data associated with each of entries320in failure log110. The columns include identifiers for entries320, a respective set of string vectors360, and a corresponding correlation value365.

Similarity analysis process235generates string vectors360from ungrouped ones of character strings322by removing words of character strings322that occur at a frequency that is less than a threshold frequency. As disclosed above, string vectors360may correspond to keywords that are found in the character strings322with a frequency that is high enough to provide matches to other character strings. For example, if entries320a-320hall correspond to uncategorized entries, then similarity analysis process235may identify keywords that occur in more than one character string322. If a word appears in only one character string322, then that word may not be useful for matching to other character strings322. In addition, simple words, such as articles “a,” “an,” and “the,” may be removed to simplify the analysis since such words may not provide suitable distinction between character strings. For example, the words “35,” “insufficient,” and “space” in the character string322for entry320bonly appear in failure log110for entry320b, and are therefore removed from string vector360for entry320b, leaving the words “disk” and “error” as keywords.

As illustrated, similarity analysis process235determines correlation values365using string vectors360. A given correlation value365indicates a degree of correlation between a corresponding group of string vectors360. For example, correlation value365amay include a set of seven values, each value indicating a correlation of the string vector360for entry320ato a respective one of string vectors360for the other entries320b-320h. In such an embodiment, correlation value365amay indicate a high correlation to entry320d, a moderate correlation to entries320c,320e, and320h, and a low correlation to entries320b,320f, and320g. Similarly, correlation value365cmay indicate a high correlation to entries320eand320h, a moderate correlation to entries320aand320d, and a low correlation to entries320b,320f, and320g. Each correlation value may be represented with any suitable set of values. For example, in one embodiment, each correlation value may be a real number between 0 and 1.

Similarity analysis process235, as shown, creates keyword cluster380based on the set of correlation values365. Using correlation values365for entries320that indicated high or moderate correlation, (e.g.,365a,365c,365d,365e, and365h), similarity analysis process235generates keyword cluster380, including the keywords “123,” “not,” “valid,” “username,” “Login,” “invalid,” “user,” and “ID.” It is noted that the word “error” that appears in all eight string vectors360is not included in keyword cluster380. If a word appears in all or most of character string vectors360, then that word may be too commonly used, and therefore, not useful for narrowing down a group to a subset of entries320.

As illustrated, similarity analysis process235identifies a new failure category (e.g., new failure category230dinFIG.2) by associating a subset of character strings322to keyword cluster380. Once keyword cluster380has been generated, similarity analysis process235compares each character string322to keyword cluster380to determine a degree of correlation between keywords in a given character string322and the keywords of keyword cluster380. If a suitable number of character strings322have a high correlation to keyword cluster380, then new failure category230dis created.

In some embodiments, a numeric vector may be generated for each character string322. For example, each position in the numeric vector corresponds to a particular one of the keywords in keyword cluster380. If a given character string322includes that keyword, then the corresponding position for that keyword is set to a first value, and otherwise to a second value. In some embodiments, the first value may be the number of times that keyword occurs in the given character string322. In other embodiments, the first value may simply be ‘1’ regardless of a number of occurrences. These numeric vectors may be compared, for example two at a time, using a cosine similarity algorithm to generate a correlation value. If the correlation value satisfies a particular threshold (e.g., greater than or equal to an 85% correlation) then the two corresponding entries320may be linked together as a subset. After all ungrouped entries320have been evaluated, any resulting subsets including a sufficient number of entries320may be assigned to a respective new failure category.

In some cases, the comparison of a particular character string322that is ungrouped to other ungrouped ones of character strings322may include comparing additional information included in failure log110. For example, performing similarity analysis process235may include determining that a particular entry320is associated with a same data source (e.g., a same value for database342) as at least some entries320corresponding to a subset of ungrouped character strings322. In addition, performing similarity analysis process235may include determining that a particular entry320has a same job type344as at least some entries in a subset of ungrouped character strings322. Furthermore, performing similarity analysis process235may include determining that a particular entry320is associated with a same domain340as at least some entries corresponding to a subset of ungrouped character strings322.

It is noted that the failure log and character analysis depicted inFIG.3are merely examples. The depicted logs and analysis have been simplified for clarity. In other embodiments, the failure log and/or the character analysis may include additional or different sets of information for performing the comparisons.

The tables ofFIG.3depict a case in which a character string for an entry is not matched to an existing failure category, resulting in identifying a new failure category. InFIG.4, a case is depicted in which a remediation action is generated and utilized for a new failure category.

Turning toFIG.4, two tables are shown, one depicting a failure log, and the other depicting a failure category with an associated character string and remediation action. As illustrated, failure log110is the same as shown inFIG.3. Failure category130may correspond, for example, to any one of failure categories130a-130cinFIG.1or to new failure category230dinFIG.2. Failure category430includes associated character string433and remediation action426.

Failure log110and failure category430may be used to demonstrate how a remediation action is determined after a new failure category has been added. Failure category430, as illustrated, is added using the techniques described in regards toFIG.3above. Entries320a,320c,320d,320e, and320hare added to failure category430based on the previously described techniques. Using keyword cluster380and character strings322, relevant keywords are determined for inclusion in character string433. For example, keywords from the character strings322a,322c,322d,322e, and322hthat were matched to keyword cluster380may be selected for use in character string433. When a character string associated with a new entry is captured and compared to existing failure categories, character string433is used to determine whether the new entry should be assigned to failure category430.

When failure category430is added, failure management program102may generate a notification indicating that new failure category430has been added. This notification may be sent to one or more system administrators using any suitable method, such as email, text message, pop-up notice, and the like. A notified system administrator may review the entries320assigned to failure category430, to determine a suitable remediation action426.

In some embodiments, failure remediation process125may, in response to the notification, analyze errors included in new failure category430to determine particular keywords that may be indicative of a potential remediation and/or to determine a remediation action. For example, failure remediation process125may track a plurality of predetermined remediation actions associated with particular types of failure events. Based on character string433, failure remediation process125may select a particular one of the predetermined remediation actions and assign this action to failure category430as remediation action426. As shown, character string433includes the keywords “login,” “username,” “user,” “ID,” “valid,” and “invalid,” all associated with login errors due to an invalid username or user ID. Failure remediation process125compares these keywords to character strings associated with remediation actions associated with login errors, and may determine that a predetermined remediation action exists for addressing username and user ID errors. Failure remediation process125assigns failure category430to the determined remediation action as remediation action426.

After remediation action426is assigned to failure category430(either by failure remediation process125or by a system administrator), failure remediation process125applies remediation action426to entries320included in new failure category430. For example, to address an invalid username error in a login type job, a first step may be to select new username. A username may be invalidated if a user associated with the username cancels the account or changes their login credentials. In some cases, a system administrator or team of system administrators may utilize specific login credentials for use with batch processes. These credentials may be changed, for example, periodically for security reasons, or if there is a change to membership of the administration team. Selecting a new username may include prompting a system administrator to select the new username, or may be accomplished by failure remediation process using a predetermined list of valid login credentials. After a new username has been selected, failure remediation process125causes batch process150(inFIGS.1and2) to repeat, thereby causing batch process150to login with the new username.

Remediation action426may generate new errors, which are captured as entries320in failure log110, or in a different failure log associated with the repeat of batch process150. In some cases, remediation action426may cause the same entries320to repeat. In such cases, failure remediation process125may determine that new entries in the new failure log correspond to entries320associated with failure category430. In various embodiments, failure remediation process125may repeat the determining of a remediation action by selecting a different action from remediation action426. In other embodiments, failure remediation process125may send a notification indicating a repeated failure.

It is noted that the tables illustrated inFIG.4are examples for demonstrating disclosed concepts. The character string and remediation action shown for the failure category have been simplified for clarity and may be different in other embodiments. Additional or different sets of information may be included in these tables in some embodiments.

Failure management programs, as described above, may be operable to perform a variety of methods.FIGS.5-8, described below, provide examples of such methods.

Proceeding toFIG.5, a flow diagram illustrating an example method500for assigning an entry to a failure category is depicted, according to some embodiments. In various embodiments, method500may be performed by computer system100ofFIGS.1and2to assign entry120binFIG.1to failure category130c. Computer system100may, for example, include (or have access to) a non-transitory, computer-readable medium having program instructions stored thereon that are executable by computer system100to cause the operations described with reference toFIG.5. Referring collectively toFIGS.1and5, method500begins in block501.

At block510, in the illustrated embodiment, method500includes accessing, by failure management program102executing on computer system100, failure log110that includes a plurality of character strings122corresponding to errors that are associated with execution of batch process150. Batch process150may be performed for any suitable reason. For example, batch process150may include virus scan software, processes for detecting compliance with security rules, data storage clean-up of old or orphaned files, and the like. While batch process150is being performed on computer system100, one or more errors may occur. A respective character string122is captured for each entry120and an identifier for an entry120and a corresponding character string122is stored in failure log110. Failure management program102may use failure classification process105to determine whether a given one of entries120(e.g., entry120b) corresponds to one of failure categories130.

Method500, at block520, includes comparing, by failure management program102, character string122bof the plurality of character strings122to a set of character strings133that are associated with respective ones of a plurality of failure categories130. The comparing includes determining whether particular keywords that are included in respective ones of the set of character strings133are included in character string122b. Failure management program102calls failure classification process105to read character string122bfrom failure log110and compare character string122bto character strings133. The comparison may include identifying particular keywords within character string122bby eliminating commonly used words that do not provide much distinction. For example, simple words such as articles, conjunctions, an pronouns may be excluded from the comparison. In some embodiments, correlation values may be generated that indicates how well character string122bcorresponds to each of character strings133. Such a correlation value may include any suitable range of values, for example, real numbers between zero and one, integers from one to ten, a percentage from 0% to 100%, and the like.

Method500further includes, at block530, assigning, by failure management program102based on the comparing, entry120bcorresponding to character string122bto failure category130cof the plurality of failure categories130. In response to determining that character string122bmeets a threshold value for similarity to character string133c, failure classification process105assigns entry120bto failure category130c. The assigning may occur by adding a field to the entry for entry120bin failure log110that assigns entry120bto failure category130c, or the assigning may occur by adding entry120bto a different log of assigned errors, a corresponding entry in this different log providing the indication that entry120bis assigned to failure category130c.

At block540, method500further includes determining, by failure management program102, that remediation action126corresponds to the error associated with entry120b. After entry120bhas been assigned to failure category130, failure management program102calls failure remediation process125to determine a particular remediation action for failure category130c, including entry120b. Failure remediation process125determines remediation action126corresponds to entry120bbased on analyzing one or more errors associated with failure category130c. In some cases, a remediation action may have been previously associated with failure category130c, and failure remediation process125uses this remediation action to resolve an error indicated by entry120b. In other cases, a remediation action may not have been determined for failure category130c. In such cases, failure remediation process125analyses character strings133cand character strings associated with errors assigned to failure category130c, including character string122b, to identify an event that caused the errors to occur and to select, for example, remediation action126from a plurality of predetermined remediation actions associated with such events.

Furthermore, method500includes, at block550, applying, by failure management program102, remediation action126to the error associated with entry120b. After remediation action126has been selected for use with entry120b, failure management program102may cause batch process150to be repeated, in whole or in part, with actions associated with remediation action126replacing corresponding actions associated the previous performance of batch process150that resulted in entry120boccurring. In some cases, remediation action126may be to send a notification to one or more system administrators that entry120boccurred, along with any relevant information that may help the system administrator to determine a cause and/or a remedy for the occurrence of entry120b. The method ends in block590. In some embodiments, method500is repeated continuously or periodically.

It is noted that the illustrated example of method500includes elements501-590. In other embodiments, additional and/or different operations may be utilized to assign an entry to a failure category. For example, operations of block550are described as occurring after a remediation action has been selected. In some embodiments, the remediation action may not be applied until batch process150has completed and all entries in the failure log have been reviewed by the failure management program.

The method ofFIG.5describes selection and application of a remediation action for a failure category. In some cases, a remediation action may not currently be available for a failure category. A method is illustrated inFIG.6for handling cases in which a remediation action is not currently available.

Moving now toFIG.6, a flow diagram is shown for an embodiment of a method for determining whether a remediation action is defined for a particular failure category. Method600, like method500above, may be performed by computer system100ofFIGS.1and2to determine a remediation action for a particular failure category. In some embodiments, a non-transitory, computer-readable medium has program instructions stored thereon that may be executable by computer system100to cause the operations described in regards toFIG.6. Referring collectively toFIGS.1and6, method600begins in block601.

At block610, method600includes determining, at a first point in time, that no remediation action has been defined for the particular failure category. As illustrated, failure remediation process125, included in failure management program102, accesses information associated with failure category130cto determine if a particular remediation action has been assigned. At the first point in time, no remediation action has been assigned to failure category130c.

Method600also includes, at block620, failure management program102sending, by the failure management program, a notification that a remediation action has not been assigned to the particular failure category. In response to determining that no remediation action has been assigned to failure category130c, failure remediation process125generates a message indicating that a remediation action has not been assigned failure category130c. This message is sent to one or more system administrators and/or other entities associated with performance of batch process150. In some embodiments, failure remediation process125may process other entries120that are assigned to other failure categories130to identify remediation actions for the other categories. In other embodiments, failure remediation process125may be active in determining a remediation action for failure category130cby analyzing character strings associated with entries120assigned to failure category130cto determine a particular cause for the assigned entries120.

At block630, method600further includes determining, at a second point in time, that the remediation action has been defined for the particular failure category. After remediation action126has been defined and assigned to failure category130c, failure remediation process125determines that the assignment has been made. In some embodiments, the determination may include a status flag being set when the assignment of remediation action126is made. In other embodiments, the determination may include receiving an indication from a system administrator that remediation action126has been assigned to failure category130c. In some embodiments, failure remediation process125may be active in defining remediation action126, and therefore, the determination is included as part of assigning remediation action126to failure category130c. Failure management program102may, in some embodiments, initiate a repeat performance of batch process150using remediation action126. Method600ends in block690. Method600may be repeated in response to other failure categories130not having remediation actions currently defined.

FIGS.5and6describe operations associated with assigning an entry to an existing failure category. In some cases, an entry in a failure log will not correspond to one of an existing set of failure categories. A method is described in regards toFIG.7that describes actions associated with identifying a new failure category.

Turning now toFIG.7, a flow diagram is shown for an embodiment of a method for determining whether an entry included in a failure log corresponds to one of a set of failure categories. Method700, like methods500and600above, may be performed by computer system100ofFIGS.1and2. In such embodiments, a non-transitory, computer-readable medium has program instructions stored thereon that may be executable by computer system100to cause the operations described in regards toFIG.7. Referring collectively toFIGS.2and7, method700begins in block701.

At block710, method700includes accessing, by failure management program102executing on computer system100, failure log110that includes a plurality of character strings122corresponding to errors that are associated with execution of batch process150. Batch process150, while being performed on computer system100, causes one or more errors to occur. A respective character string122is captured for each error and an identifier for an entry120and a corresponding character string122is stored in failure log110. Failure classification process105determines whether a given one of entries120(e.g., entry120c) corresponds to one of failure categories130.

Method700, at block720, includes comparing, by failure management program102, character string122cof the plurality of character strings122to a set of character strings133that are associated with respective ones of a plurality of failure categories130. The comparing includes determining whether particular keywords that are included in respective ones of the set of character strings133are included in character string122c. Failure classification process105retrieves character string122cfrom failure log110and compares it to character strings133. As described previously, the comparison may include identifying particular keywords within character string122cby eliminating commonly used words that do not provide much distinction. In some embodiments, correlation values, as described above, may be generated that indicate a degree of correlation between character string122cand each of character strings133.

At block730, method700further includes, in response to character string122cnot corresponding to any of the set of character strings133, performing, by failure management program102, a similarity analysis of character string122cto one or more ungrouped character strings222that do not correspond to any of the set of character strings133. As illustrated, failure management program102calls similarity analysis process235in response to failing to identify a failure category130for entry120c. Similarity analysis process235includes determining frequencies of occurrence for a plurality of keywords. For example, similarity analysis process235extracts keywords from character string122cand from ungrouped character strings222by removing simple words (e.g., articles, conjunctions, etc.). Each of the keywords in character string122care compared to the keywords from ungrouped character strings222and a respective frequency of usage is determined for each keyword in character string122c. An upper and a lower threshold may be utilized to identify a subset of ungrouped character strings222that are similar to character string122c. The upper threshold may eliminate keywords that occur too frequently to provide distinction between the character strings222. The lower threshold determines a minimum number of character strings that may form a subset. It is noted that techniques described above in regards toFIGS.2and3may also be utilized in block730to perform the similarity analysis.

Method700, at block740, also includes adding, by failure management program102, new failure category230dto the plurality of failure categories130. In response to identifying a subset of ungrouped character strings222that include similar keywords as character string122c, similarity analysis process235creates new failure category230dand adds it to the set of failure categories130. New failure category230dincludes an error corresponding to character string122c, as well as other errors corresponding to the subset of ungrouped character strings222. In addition, similarity analysis process235adds the similar keywords to character string233dfor use in identifying other entries120that may be assigned to new failure category230d. Method700ends in block790. In some embodiments, method700may be repeated for other entries included in failure log110.

It is noted that method700is one example for identifying and adding a new failure category to an existing set of failure categories. Method700includes performing a similarity analysis between a particular character string and a group of ungrouped character strings. Method800inFIG.8depicts a method for performing a similarity analysis.

Proceeding now toFIG.8, a flow diagram is shown for an embodiment of a method for performing a similarity analysis between a particular character string and a group of ungrouped character strings. Method800, like the methods described above, may be performed by computer system100ofFIGS.1and2by accessing a non-transitory, computer-readable medium that has program instructions stored thereon that may be executable by computer system100to cause the operations described in regards toFIG.8. Referring collectively toFIGS.2,3, and8, method800begins in block801.

Method800, at block810, includes generating string vectors360from a particular character string (e.g., associated with entry320a) and a group of ungrouped character strings (e.g., associated with entries320b-320h) by removing words of the character strings322that occur at a frequency that is less than a threshold frequency. For example, failure management program102removes any words from the particular and the ungrouped character strings322that occur once or twice across all eight character strings322. In addition, failure management program102may also remove simple words, such as articles or pronouns. In character analysis370, a corresponding string vector360is generated for each entry320.

At block820, method800includes determining a set of correlation values365using string vectors360, wherein a given correlation value365indicates a degree of correlation between a corresponding group of the string vectors360. As illustrated, correlation values365may include a plurality of values for each entry320. For example, failure management program102compares a string vector360for entry320ato a string vector360for each of entries320b-320h. Correlation values365amay include seven values, one for each comparison to each of entries320b-320h. In some embodiments, each correlation value is a numeric value, for example between ‘0’ and ‘1’ or between ‘1’ and ‘100’ or any suitable range of values that may be used to indicate a degree of correlation between two or more string vectors360.

Method800, at block830, also includes creating keyword cluster380based on the set of correlation values365. Failure management program102uses the set of correlation values365to identify a subset of entries320that satisfy a threshold degree of correlation with one another. For example, a subset that includes entries320a,320c,320d,320e, and320hmay have correlation values that indicate a medium to high level of correlation to one another. From this subset of five entries320, keyword cluster380is generated by including keywords from the corresponding string vectors360that are found in, e.g., string vectors for two or more of the subset of entries320. It is noted that the word “Error” is not included in keyword cluster380. Since “Error” appears in all eight string vectors360, and is not limited to the subset of entries320, “Error” is removed due to not being distinctive to the subset.

At block840, method800further includes identifying the new failure category by associating a subset of the ungrouped character strings to the keyword cluster. Failure management program102may compare character strings322for all ungrouped entries320to keyword cluster380to identify entries320that should be assigned to a new failure group. In other embodiments, failure management program102may limit the comparison to the entries included in the subset. A determination is made for each entry320whether the entry320belongs to a new failure category associated with the keyword cluster380. Although the subset of entries is used to generate keyword cluster380, repeating a comparison of the character strings322to keyword cluster380may improve an accuracy for classifying related entries320.

Additionally, method800includes, at block850, assigning, to the new failure category, a set of errors corresponding to the subset of ungrouped character strings322. For each comparison to keyword cluster380, failure management program102may determine another correlation value between a given entry320and keyword cluster380. If this correlation value satisfies a threshold value, then an error associated with the given entry320is assigned to the new failure category. The method ends in block890. In some embodiments, method800may be repeated as new entries320are captured in failure log110.

It is noted that the methods ofFIGS.5-8include multiple operations that are presented as occurring in a particular order. In various embodiments, the order may differ. In addition, various methods may be performed concurrently or as part of a different method. For example, method800may be performed as part of block730of method700.

Referring now toFIG.9, a block diagram of an example computer system900is depicted, which may implement one or more computing devices in a computer system, such as computer system100ofFIGS.1and2. Computer system900includes a processor subsystem920that is coupled to a system memory940and I/O interfaces(s)960via an interconnect980(e.g., a system bus). I/O interface(s)960is coupled to one or more I/O devices970. Computer system900may be any of various types of devices, including, but not limited to, a server computer system, personal computer system, desktop computer, laptop or notebook computer, mainframe computer system, server computer system operating in a datacenter facility, tablet computer, handheld computer, workstation, network computer, etc. Although a single computer system900is shown inFIG.9for convenience, computer system900may also be implemented as two or more computer systems operating together.

Processor subsystem920may include one or more processors or processing units. In various embodiments of computer system900, multiple instances of processor subsystem920may be coupled to interconnect980. In various embodiments, processor subsystem920(or each processor unit within920) may contain a cache or other form of on-board memory.

System memory940is usable to store program instructions executable by processor subsystem920to cause computer system900to perform various operations described herein. System memory940may be implemented using different physical, non-transitory memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM-SRAM, EDO RAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read only memory (PROM, EEPROM, etc.), and so on. Memory in computer system900is not limited to primary storage such as system memory940. Rather, computer system900may also include other forms of storage such as cache memory in processor subsystem920and secondary storage on I/O devices970(e.g., a hard drive, storage array, etc.). In some embodiments, these other forms of storage may also store program instructions executable by processor subsystem920.

I/O interfaces960may be any of various types of interfaces configured to couple to and communicate with other devices, according to various embodiments. In one embodiment, I/O interface960is a bridge chip (e.g., Southbridge) from a front-side to one or more back-side buses. I/O interfaces960may be coupled to one or more I/O devices970via one or more corresponding buses or other interfaces. Examples of I/O devices970include storage devices (hard drive, optical drive, removable flash drive, storage array, SAN, or their associated controller), network interface devices (e.g., to a local or wide-area network), or other devices (e.g., graphics, user interface devices, etc.). In one embodiment, I/O devices970includes a network interface device (e.g., configured to communicate over WiFi, Bluetooth, Ethernet, etc.), and computer system900is coupled to a network via the network interface device.

Although the embodiments disclosed herein are susceptible to various modifications and alternative forms, specific embodiments are shown by way of example in the figures and are described herein in detail. It should be understood, however, that figures and detailed description thereto are not intended to limit the scope of the claims to the particular forms disclosed. Instead, this application is intended to cover all modifications, equivalents and alternatives falling within the spirit and scope of the disclosure of the present application as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description.

This disclosure includes references to “one embodiment,” “a particular embodiment,” “some embodiments,” “various embodiments,” “an embodiment,” etc. The appearances of these or similar phrases do not necessarily refer to the same embodiment. Particular features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

As used herein, the terms “first,” “second,” etc. are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.), unless stated otherwise. As used herein, the term “or” is used as an inclusive or and not as an exclusive or. For example, the phrase “at least one of x, y, or z” means any one of x, y, and z, as well as any combination thereof (e.g., x and y, but not z).

Reciting in the appended claims that a structure is “configured to” perform one or more tasks is expressly intended not to invoke 35 U.S.C. § 112(f) for that claim element. Should Applicant wish to invoke Section 112(f) during prosecution, it will recite claim elements using the “means for” [performing a function] construct.

In this disclosure, various “processes” operable to perform designated functions are shown in the figures and described in detail above (e.g., failure classification process105, similarity analysis process235, etc.). As used herein, the term “process” refers to circuitry configured to perform specified operations or to physical, non-transitory computer-readable media that stores information (e.g., program instructions) that instructs other circuitry (e.g., a processor) to perform specified operations. Such circuitry may be implemented in multiple ways, including as a hardwired circuit or as a memory having program instructions stored therein that are executable by one or more processors to perform the operations. The hardware circuit may include, for example, custom very-large-scale integration (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A process may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, or the like. A process may also be any suitable form of non-transitory computer readable media storing program instructions executable to perform specified operations.