System and method for automating formation and execution of a backup strategy using machine learning

Disclosed herein are systems and method for forming and executing a backup strategy. In one aspect, an exemplary method comprises, for each file of files from one or more data sources that is being evaluated to form the backup strategy for the file, updating a frequency database, evaluating a uniqueness for the file stored at a data source of the one or more data sources by comparing at least a portion of data of the file to the frequency database, categorizing the file into a hierarchy of logical types according to properties of the file, and forming the backup strategy for the file according to the uniqueness and categorization of the file.

FIELD OF TECHNOLOGY

The present disclosure relates generally to the field of data archiving, and more specifically, to systems and methods for automating formation and execution of a backup strategy using machine learning.

BACKGROUND

Performing regular backup of large amounts of data is a time and power intensive operation. Additionally, a significant amount of storage space, whether locally or in cloud storage, is normally required to complete such large backup operations. This has led to the implementation of backup plans which optimize the performance of the backups. Often, such backup plans take into account backup parameters such as the periodicity of creation of backup copies, whether full or incremental, the timing of the beginning of the backup operation, the location of the source data, the backup location, whether local storage, cloud storage or instant copy, and finally, whether the data requires encryption based on the confidentiality of data contained therein.

However, the backup operation may be for large volumes of data from various sources with varying file types. Moreover, the data can change very quickly and often. Modifying the parameters described above becomes unwieldy and ineffective without considering alternative strategies.

Therefore, there is a need for a an efficient and flexible method and system for forming and executing backup strategies, e.g., for backing up large volumes of dynamic data from various sources and file types.

SUMMARY

Aspects of the disclosure relate to forming and executing backup strategies for files using machine learning.

In one exemplary aspect, a method for forming and executing a backup strategy is implemented in a system that comprises a data analysis engine and a learning engine, the system comprising one or more processors, the method comprising: for each file of files from one or more data sources that is being evaluated to form the backup strategy for the file, updating a frequency database, evaluating a uniqueness for the file stored at a data source of the one or more data sources by comparing at least a portion of data of the file to the frequency database, categorizing the file into a hierarchy of logical types according to properties of the file, and forming the backup strategy for the file according to the uniqueness and categorization of the file.

According to one aspect of the disclosure, a system is provided for forming and executing a backup strategy, the system comprising at least one processor configured to: for each file of files from one or more data sources that is being evaluated to form the backup strategy for the file, update a frequency database, evaluate a uniqueness for the file stored at a data source of the one or more data sources by comparing at least a portion of data of the file to the frequency database, categorize the file into a hierarchy of logical types according to properties of the file, and form the backup strategy for the file according to the uniqueness and categorization of the file.

In one exemplary aspect, a non-transitory computer-readable medium is provided storing a set of executable instructions thereon for forming and executing a backup strategy, including instructions for: for each file of files from one or more data sources that is being evaluated to form the backup strategy for the file, updating a frequency database, evaluating a uniqueness for the file stored at a data source of the one or more data sources by comparing at least a portion of data of the file to the frequency database, categorizing the file into a hierarchy of logical types according to properties of the file, and forming the backup strategy for the file according to the uniqueness and categorization of the file.

In one aspect, the categorizing of the file further comprises: analyzing metadata associated with the file, analyzing contents of the file, and classifying the file based on the metadata associated with the file and the contents of the file.

In one aspect, the method further comprises: adding a secure backup strategy to the backup strategy for the file, when the uniqueness or importance of the portion of data is determined as being greater than a respective predetermined uniqueness or importance threshold, adding an instant copying to the backup strategy for the file, when the file is determined as having a desired recovery time less than a predetermined recovery time threshold, adding a distributed backup to the backup strategy for the file, when a criticality of data loss associated with the portion of data is higher than a predetermined criticality threshold, adding a local backup to the backup strategy for the file, when the criticality of data loss associated with the portion of data is lower than the predetermined criticality threshold, copying the file to a cloud storage, when the instant copying is added to the backup strategy, and executing the backup strategy for the file.

In one aspect, the method further comprises: evaluating a confidentiality of a file prior to forming the backup strategy, the evaluation being to determine whether the file is confidential and, when the file is confidential, to determine a degree of confidentiality, and selecting a strength of one or more encryption algorithms according to the determined degree of confidentiality.

In one aspect, the backup strategy comprises one or more of: an instant copy, a local copy, and a cloud storage.

In one aspect, the backup strategy is further formed according to one or more of: an importance, a recovery time, and a recovery point objective of the file.

In one aspect, the uniqueness is evaluated based on a deep learning analysis of one of: a local data set and an external data set. In one aspect, the local data set comprises an archive and the external data set comprises the Internet.

In one aspect, the evaluation of the uniqueness based on the deep learning comprises one or more of: considering an authorship of the file and availability of the file in the local and external data sets, considering metadata regarding a location of the file in combination with the authorship of the file, and considering an availability of the file in the local and external data sets.

DETAILED DESCRIPTION

Exemplary aspects are described herein in the context of a system, method and computer program product for automating formation and execution of a backup strategy using machine learning. Those of ordinary skill in the art will realize that the following description is illustrative only and is not intended to be in any way limiting. Other aspects will readily suggest themselves to those skilled in the art having the benefit of this disclosure. Reference will now be made in detail to implementations of the example aspects as illustrated in the accompanying drawings. The same reference indicators will be used to the extent possible throughout the drawings and the following description to refer to the same or like items.

Exemplary aspects of the present disclosure use deep learning algorithms to optimize and automate formation and execution of backup strategies for data endpoints.

FIG. 1is a block diagram illustrating a system100for automating a formation and execution of a backup plan according to an exemplary aspect of the disclosure. The system100comprises a data analysis engine101, a learning engine110, one or more backup sources1to N (or alternatively referred to as “endpoints”), one or more analysis sources1to M and one or more storage destinations1to P. The data analysis engine101comprises a document evaluation module102, a categorization processor104, a content analyzer105and a strategy processor106. Aspects of this disclosure do not limit the network location of the data analysis engine101relative to the storage destinations1to P, the backup sources1to N or the analysis sources1to M.

The data analysis engine101performs backup operations on data121from one or more of the backup sources1to N. The backup sources1to N may be one or more storage devices local to computing workstations, server storage devices, or any other device storage contemplated by one of ordinary skill in the art. The data contained in the backup sources1to N comprises, in some aspects, database files, personal files, files associated with software applications or the like. The data stored in the backup sources1to N is not limited to those types described in this disclosure and may contain any types of files which a person or enterprise may desire to have backed up. The data analysis engine101may be directly coupled to the backup sources1to N, e.g. on the same local area network (LAN), may communicatively couple to the backup sources1to N over network140, or a combination of both.

The data analysis engine101performs stores backups of the data121in one or more of the storage destinations1to P. The storage destinations1to P are distinct storage types. In one aspect, the storage destination1is an “Instant Copy” destination, the storage destination2is a local copy destination and the storage destination3is a remote cloud storage destination. In the example of a local copy destination, the storage destination2is located local to the backup source. In aspects of the disclosure, local destinations can be external drives, network shares, tape drives, though not limited thereto. In this aspect, remote destinations can be any cloud destination. Finally, an instant copy is a way of saving files quickly for short periods of time, generally using RAM or snapshotting techniques. In some aspects of the disclosure, the data analysis engine101selects a combination of storage destinations, for example, instant copy and local copy, local copy and cloud copy, and any other combination thereof deemed appropriate by the data analysis engine101given properties of the data.

In one aspect, the data analysis engine101is also communicatively coupled to the learning engine110via network140or via some other communicative configuration. The learning engine110comprises a frequency database112and a categorizer114. The learning engine110is further configured to communicate with analysis sources1,2. . . M. The analysis sources are used by the learning engine110as deep learning sources used to classify and categorize any data that passes through system100. In one aspect, the analysis sources comprise the Internet, data archives, or previously backed up data. The learning engine110performs deep learning on the analysis sources1to M to recognize and classify data efficiently and accurately as the data changes over time. For example, the frequency database112inspects the Internet as an analysis source and creates hashes of all documents (e.g., including but not limited to: ACROBAT® PDF, WORD®, POWERPOINT®) that are globally available. The frequency database112associates these hashes with the commonality and availability of the document for future use by the data analysis engine101. The learning engine110also performs machine learning on these documents in analysis sources1to M and classifies the types of documents according to, at least, importance of data, recovery time, recovery point objective and data confidentiality. Importance of data relates to the availability of the document. Recovery Point Objective (RPO) refers to a maximum targeted period of time in which data might be lost from a service due to a major incident, while recovery time refers to how soon a document may be needed from backup. Finally, confidentiality refers to the level of secrecy or public availability of the document.

The data analysis engine101reviews and analyzes data to be backed up from the backup sources1to N. The data121is categorized using the categorization processor104to determine the hierarchical nature of the data. In one aspect, the hierarchical nature of the data comprises the file types such as databases, documents, media files and the like. The data is also classified into types, e.g., curriculum vitae, presentations, reports, user manuals and the like, and subtypes: such as certain products (e.g., “Software1”, “Software2”), language, region and version. In one aspect, older versions of a document are rated as less important compared with newer versions of the same document. This classification is obtained by the categorization processor104passing the data along to the learning engine110. The categorizer114of the learning engine110is initially seeded with a few hierarchical rules, and subsequently performs categorization/learning on files from the analysis sources1to M and thus can compare the data from the backup sources1to N with already categorized data to match the hierarchical structure of the file or the type of the file, and enhance the rules by the new data. Once the data matches data in the categorizer114, the categorizer114returns the hierarchical category of the matched data to the categorization processor104. In other embodiments, the data stored on backup sources1to N is automatically and periodically categorized into hierarchies by the categorizer114of the learning engine110reflecting any changes that have occurred in the data121. According to one aspect of the present disclosure, classification of data is based on the analysis of document metadata in the data121, as well as analysis of titles and contents of the data extracted by the content analyzer105(e.g., a semantic analysis, or keyword indexing). Using a keyword index allows categorization to be more efficient.

The document evaluation module102is also invoked by the data analysis engine101to evaluate each file and determine a uniqueness (e.g., frequency of occurrence) and importance of individual files in the data121by passing comparing the data121to data stored in the frequency database112. The learning engine112scrapes data from the Internet, data archives, or even backup data to populate the frequency database112with a representation of the scraped data. In some embodiments, the representation is a hash of the scraped data, while in other embodiments the data itself is compressed and stored. Commonly available files (those whose count is significantly high in the frequency database112) are therefore considered less unique by the document evaluation module102than files which are not found as often in the frequency database112.

With the information regarding uniqueness of data from the document evaluation module102and the categorization of the data from the categorization processor104, the data analysis engine101invokes the strategy processor106. The strategy processor106forms a backup strategy to use for the particular data based on machine learning, the information received from the processors102and104(i.e., document evaluation module102and categorization processor104), and a set of data criteria108. A portion of the data criteria108is generated by the strategy processor106by considering several aspects of the data121such as RPO, confidentiality, document hierarchy and recovery time, discussed in more detail with respect toFIG. 2, based on machine learning performed by the strategy processor106to compare the data121with data form the learning engine110. Additionally, the strategy processor106considers the confidentiality of the file and may include encryption as a part of the backup strategy if the degree of confidentiality is greater than a particular threshold. In one aspect, the strategy processor106forms backup strategies based on similarity of the data being backed up with previously backed-up data and performs machine learning in order to automate selection of the strategy. In other words, the above described rules are known by the strategy processor106, but these rules are made more precise as the data being backed-up increases. Once a strategy is formed by the strategy processor106, the strategy processor106executes the backup strategy to back up the data to one or more of the storage destinations1to P which may be in a local store, in the cloud or an instant copy can be created (e.g., in RAM, as a file snapshot, or both). According to one aspect of the disclosure, the backup strategy formed by the strategy processor106ensures optimal data integrity, optimizes the time and resources needed to create backups during a backup window (e.g., the time during which backup operations are performed), and optimize storage space by storing items in different storage destinations according to the their importance, uniqueness, categorization, criticality, and the like and reducing clutter in cloud storage, local storage (e.g., internal or external hard drives) and instant copy storage.

FIG. 2is a block diagram illustrating example operations of the data analysis engine101according to an exemplary aspect. The data analysis engine101invokes the strategy processor106taking into consideration the data criteria108, as illustrated inFIG. 2. The document evaluation module102evaluates each document and generates a uniqueness score200and an importance score210. The categorization processor104generates a document hierarchy202. The strategy processor106forms data criteria108that includes the uniqueness score200and the document hierarchy202, in addition to importance score210, recovery time212and RPO214. The categorization processor104allows for tuning and adjustment of backup plans dynamically. For example, a document that was unique one week ago may no longer be unique because the categorization processor104finds the document in several other locations. In another example, a secret finance report with very high importance was updated, so a previous version became less important. Thus the backup strategy changes for this particular document and it can be stored with less redundancy. The RPO214, confidentiality216, document hierarchy202and recovery time212are generated by the strategy processor106using machine learning to compare the data121being backed up with data form the learning engine110. In other aspects, over time, based on historical values of the document hierarchy202, the uniqueness score200and the importance score210, the strategy processor106calculates the values of RPO214, recovery time212and confidentiality216and becomes more accurate over time in these calculations given more data to process.

The uniqueness score200and importance score210are related. For example, an electronic book from a well-known publishing house or a user manual that can easily be found on the web is not considered unique and the risk of losing such a document is not relevant. Thus the uniqueness score200and the importance score210will be comparatively low. While uniqueness and importance are related and complementary, these two scores are theoretically not necessary equal. For example, an author of a document is the same person as a user of a given computer, it is very likely that the document is unique and exists as a single copy. However, this does not mean that the document is extremely important. Additional parameters or characteristics of this document need to be assessed before being considered important, such as the document being marked as “highly confidential”, or the like. The unique document becomes more important based on confidentiality (for example), as compared with a simple textual reminder, for example. Another criterion that makes a unique document more or less important is the location of the document. For example, a user stores the document in a folder named “Important” or a folder that can be identified as important in some manner—the importance score210will increase. In another example, the importance score210will increase when the document is stored in a secure and safe location such as special external drive, for example. Machine Learning is employed by the strategy processor106to analyze criteria and forming the uniqueness score200and the importance score210accordingly. However, the document evaluation module102always scores a unique document as more important by default than a non-unique document. If the owner of the computer upon which backup is being performed and the author of the document being inspected are the same person, then such a document is likely unique, relatively increasing the uniqueness score200and the importance score200. Thus importance score210and uniqueness score200are interrelated, but not necessarily equal. A higher uniqueness score200and/or a higher importance score200influences the strategy processor106to form a secure backup strategy such as, in one aspect, cloud backup, or local backup and cloud backup.

The recovery time212represents a desired recovery time for a file. For example, if the recovery time (i.e., the time the file needs to be restored by)212is lesser than a predetermined threshold, the strategy processor212uses instant copying as a backup strategy. In some aspects, instant copying comprises, but is not limited to, backing the file up in RAM as a file snapshot, or the like. Instant copy is generally used when protecting data from malware, when an unmodified version of a file requires quick saving and quick restoration or the like. Instant copy is generally considered the least reliable way of backing up data and therefore if the importance score210or uniqueness score200is higher than corresponding threshold values, the backup strategy will include cloud storage in addition to instant copy.

The RPO214represents a criticality of data loss for a particular piece of data. If the RPO214is lower than a predefined threshold, the strategy processor106forms the backup strategy220that includes only local storage, such as on an external hard disk. If the RPO214is higher than another predefined threshold, the backup strategy220comprises cloud storage or another form of distributed storage where redundancy can be chosen, giving an added degree of safety and reliability.

Additionally, the data criteria108comprises confidentiality216of the data. Depending on the importance of confidentiality of the file, the file may need to be stored in an encrypted form. Accordingly, the confidentiality216(e.g., “yes/no”) and the degree of this confidentiality is additionally considered by the strategy processor106, or in other aspects may be combined with the importance score210. Accordingly, the backup strategy220comprises a decision to encrypt the data in the backup process along with choosing the strength of encryption algorithms (e.g. synchronous, asynchronous, etc.).

Accordingly the strategy processor106creates a dynamic backup strategy220which is configured each time for each new piece of data, based on the deep learning of the learning engine110and the data criteria108. The strategy processor106executes the backup strategy220on the data, storing the backup in one or more storage destinations1to P.

FIG. 3is a block diagram illustrating example operations of a data classification and categorization stage according to an exemplary aspect of the present disclosure.

In one aspect of the disclosure, the document evaluation module102also parses the data121ofFIG. 1and generates a plurality of file identifiers300which are stored in the frequency database112. In this way, the frequency database112keeps a record of all files (in addition to other properties associated with those files) that pass through the system100so that future data can be evaluated for uniqueness and importance. For a certain file, a file identifier is generated and checked against the frequency database112. If the database match results in more than a predefined threshold number of results, the uniqueness score200is proportionally decreased. If the database match results in fewer than another predefined threshold number of results, the uniqueness200score is proportionally increased. In some aspects of the present invention file identifiers300comprise hashes of file contents, checksums, or some other identifier calculated using the contents of the file.

In a similar manner, the categorization processor104receives data121from the backup sources1to N. Also, the data121passes through a content analyzer105which extracts the actual binary or textual data stored in each of the files in data121. The categorization processor104operates on the data121itself, and also via the categorizer114to categorize the contents of the data into hierarchical structures. Once all of the documents are categorized by the categorizer114, the categorization processor104generates a document hierarchy202for use by the strategy processor106. The document hierarchy202includes the type of file, e.g., whether the file is a database file, a media file, a textual document, or the like, in addition to the logical category or subcategory of the file, e.g., whether the file is a presentation, a user manual, a menu, a report, a memo, a curriculum vitae, or the like. With operation of every backup, the learning engine110performs deep learning via the frequency database112being updated and the categorizer114learning about new types of files or similarly between other types of files. This results in future categorizations and determinations of uniqueness to be significantly more accurate.

FIG. 4is a flowchart illustrating a method400for automating formation and execution of a backup strategy using machine learning according to an exemplary aspect of the disclosure. The method ofFIG. 4may be carried out by the data analysis engine101and the learning engine110. The data analysis engine101and the learning engine110may be implemented via a general-purpose computer system20, as shown inFIG. 5, for executing the method400by a processor, such as a central processing unit21of the computer system20.

The method400begins at step402and proceeds to step404. At step404, the learning engine110updates the frequency database112for each file evaluated from the backup sources1to N. The method400then proceeds to step406where the data analysis engine101invokes the document evaluation module102which evaluates uniqueness for a file from the backup sources1to N by comparing the file to frequency database112. In some instances, the comparison comprises a comparison of hash checks or checksums of the file and respective hash checks or checksums stored in the frequency database112.

At step408, the data analysis engine101invokes the categorization processor104which categorizes the file into a hierarchy of types according to the properties of the file in addition to similarities with previously encountered and analyzed files. The categorization processor104, as previously discussed, performs deep learning on one or more analysis sources1to M so that individual files from a backup source can be categorized and classified quickly and accurately. Then, the method proceeds to step409to form the backup strategy for the file in accordance with the uniqueness and the categorization of the file. In other words, the results of steps406and408are used to form the backup strategy for the file.

The method proceeds to step410where the data analysis engine101determines a confidentiality of a document. Whether the document should be stored confidentially and whether the degree of confidentiality affects the ultimate backup strategy. If the file does not have a confidentiality greater than a predetermined threshold, the method proceeds to step414. If the file has a confidentiality greater than the predetermined threshold, the backup strategy is formed with an encryption parameter at step412that indicates the file is to be encrypted, along with a degree of confidentiality which is ultimately used to determine which encryption algorithm is used during backup. The method then proceeds to step414.

At step414, the data analysis engine101determines whether the importance score and/or the uniqueness score, or both in aggregate, of the file are greater than a predetermined threshold. If the scores are greater than the predetermined threshold, the method proceeds to step416. Otherwise, the method proceeds to step422. In step416, the strategy processor106adds distributed backup to the backup strategy. In some instances, a local backup is also added to the backup strategy at step418, for example, when further security is desired.

The method400then proceeds to step422where the data analysis engine422determines whether the file has a criticality score (e.g., RPO214ofFIG. 2) greater than a predetermined criticality threshold. If so, the method400proceeds to step424where a distributed backup is added to the backup strategy where a degree of redundancy can be chosen. If the criticality score is lower than the threshold, then the method proceeds step423where a local backup is added to the backup strategy.

The method400then proceeds to step426where the data analysis engine422determines whether the file has a recovery time less than a recovery time threshold. If so, the method400proceeds to step428where an instant backup is added to the backup strategy.

The method then proceeds to step430where the strategy processor430performs the backup operation using the added backup strategies. This process is repeated for each file, thus a fully formed backup strategy contains strategies for all file/file types in the backup sources1to N. The method400terminates at step440.

FIG. 5is a block diagram illustrating a general-purpose computer system20on which aspects of systems and methods for automating formation and execution of a backup strategy may be implemented in accordance with an exemplary aspect of the disclosure. It should be noted that the computer system20can correspond to the system100, data analysis engine101, or learning engine110, for example, described earlier.

As shown, the computer system20(which may be a personal computer or a server) includes a central processing unit21, a system memory22, and a system bus23connecting the various system components, including the memory associated with the central processing unit21. As will be appreciated by those of ordinary skill in the art, the system bus23may comprise a bus memory or bus memory controller, a peripheral bus, and a local bus that is able to interact with any other bus architecture. The system memory may include permanent memory (ROM)24and random-access memory (RAM)25. The basic input/output system (BIOS)26may store the basic procedures for transfer of information between elements of the computer system20, such as those at the time of loading the operating system with the use of the ROM24.

The computer system20, may also comprise a hard disk27for reading and writing data, a magnetic disk drive28for reading and writing on removable magnetic disks29, and an optical drive30for reading and writing removable optical disks31, such as CD-ROM, DVD-ROM and other optical media. The hard disk27, the magnetic disk drive28, and the optical drive30are connected to the system bus23across the hard disk interface32, the magnetic disk interface33and the optical drive interface34, respectively. The drives and the corresponding computer information media are power-independent modules for storage of computer instructions, data structures, program modules and other data of the computer system20.

An exemplary aspect comprises a system that uses a hard disk27, a removable magnetic disk29and a removable optical disk31connected to the system bus23via the controller55. It will be understood by those of ordinary skill in the art that any type of media56that is able to store data in a form readable by a computer (solid state drives, flash memory cards, digital disks, random-access memory (RAM) and so on) may also be utilized.

The computer system20has a file system36, in which the operating system35, may be stored, as well as additional program applications37, other program modules38, and program data39. A user of the computer system20may enter commands and information using keyboard40, mouse42, or any other input device known to those of ordinary skill in the art, such as, but not limited to, a microphone, joystick, game controller, scanner, etc. . . . . Such input devices typically plug into the computer system20through a serial port46, which in turn is connected to the system bus, but those of ordinary skill in the art will appreciate that input devices may be also be connected in other ways, such as, without limitation, via a parallel port, a game port, or a universal serial bus (USB). A monitor47or other type of display device may also be connected to the system bus23across an interface, such as a video adapter48. In addition to the monitor47, the personal computer may be equipped with other peripheral output devices (not shown), such as loudspeakers, a printer, etc.

Computer system20may operate in a network environment, using a network connection to one or more remote computers49. The remote computer (or computers)49may be local computer workstations or servers comprising most or all of the aforementioned elements in describing the nature of a computer system20. Other devices may also be present in the computer network, such as, but not limited to, routers, network stations, peer devices or other network nodes.

Network connections can form a local-area computer network (LAN)50and a wide-area computer network (WAN). Such networks are used in corporate computer networks and internal company networks, and they generally have access to the Internet. In LAN or WAN networks, the personal computer20is connected to the local-area network50across a network adapter or network interface51. When networks are used, the computer system20may employ a modem54or other modules well known to those of ordinary skill in the art that enable communications with a wide-area computer network such as the Internet. The modem54, which may be an internal or external device, may be connected to the system bus23by a serial port46. It will be appreciated by those of ordinary skill in the art that said network connections are non-limiting examples of numerous well-understood ways of establishing a connection by one computer to another using communication modules.