Automatic dispatch of multiple tables into consistency groups within an active-active solution

In computer methods and systems for operating a database management system using a catalog table collection module each table in the database management system is cataloged. Multiple tables are automatically dispatched into multiple consistency groups by collecting real-time statistics using a real time statistics module. Workload profile data of the database management system is generated using a workload profile collection module, and an optimized multiple consistency group definition for the database management system is generated using a transaction splitting analysis module.

BACKGROUND

The present invention generally relates to software data replication tools, and more specifically, to a system that automatically dispatches multiple tables into constancy groups within an active-active solution.

Multi-user computer systems, such as mainframes with active-active user bases, frequently have multiple users simultaneous performing computational operations that required data transfer and replication between databases. In order to maintain consistency between all the simultaneous users, the operations are broken into consistency groups by the database management systems.

SUMMARY

Embodiments of the present invention are directed to a computer-implemented method for A non-limiting example of the computer-implemented method includes using a catalog table collection module each table in the database management system is cataloged. Multiple tables are automatically dispatched into multiple consistency groups by collecting real-time statistics using a real time statistics module. Workload profile data of the database management system is generated using a workload profile collection module, and an optimized multiple consistency group definition for the database management system is generated using a transaction splitting analysis module. Embodiments of the present invention are directed to a system and to a computer program product for performing the same, the computer program product comprising a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a processor to cause the processor to perform a method.

DETAILED DESCRIPTION

One or more embodiments described herein can utilize machine learning techniques to perform prediction and or classification tasks, for example. In one or more embodiments, machine learning functionality can be implemented using an artificial neural network (ANN) having the capability to be trained to perform a function. In machine learning and cognitive science, ANNs are a family of statistical learning models inspired by the biological neural networks of animals, and in particular the brain. ANNs can be used to estimate or approximate systems and functions that depend on a large number of inputs. Convolutional neural networks (CNN) are a class of deep, feed-forward ANNs that are particularly useful at tasks such as, but not limited to analyzing visual imagery and natural language processing (NLP). Recurrent neural networks (RNN) are another class of deep, feed-forward ANNs and are particularly useful at tasks such as, but not limited to, unsegmented connected handwriting recognition and speech recognition. Other types of neural networks are also known and can be used in accordance with one or more embodiments described herein.

ANNs can be embodied as so-called “neuromorphic” systems of interconnected processor elements that act as simulated “neurons” and exchange “messages” between each other in the form of electronic signals. Similar to the so-called “plasticity” of synaptic neurotransmitter connections that carry messages between biological neurons, the connections in ANNs that carry electronic messages between simulated neurons are provided with numeric weights that correspond to the strength or weakness of a given connection. The weights can be adjusted and tuned based on experience, making ANNs adaptive to inputs and capable of learning. For example, an ANN for handwriting recognition is defined by a set of input neurons that can be activated by the pixels of an input image. After being weighted and transformed by a function determined by the network's designer, the activation of these input neurons are then passed to other downstream neurons, which are often referred to as “hidden” neurons. This process is repeated until an output neuron is activated. The activated output neuron determines which character was input.

A container is a VCE that uses operating-system-level virtualization. This refers to an operating system feature in which the kernel allows the existence of multiple isolated user-space instances, called containers. These isolated user-space instances typically behave as real computers from the point of view of programs running in them. A computer program running on an ordinary operating system can utilize all resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and devices assigned to the container, a feature which is known as containerization.

Turning now to an overview of technologies that are more specifically relevant to aspects of the invention, mainframe Active-Active (A-A) users require a substantial number of user application tables. All of the application tables run different workloads with various amount of insert/update/delete operations (or other similar data replication or manipulation operations). Users using data replication operations typically utilize a software data replication tool to ensure the data consistency which has multiple consistency groups to provide scalability and parallel replication ability.

In the example ofFIG.2, one CG210,212,214includes all the tables220that are replicated on a single receive queue202and each table220is assigned to one and only one CG210,212,214. Assuming a customer has a large number of tables, the system200is faced with the task of how to optimally split the tables into different CG's210,212,214. When multiple CG's210,212,214replicate excessive amounts of data and exceed their capacity, the total consistency of the data can be negatively impacted.

To prevent the negative impact, the system200is required to ensure all the tables maintain a balanced status, so all the consistency groups can replicate data with a similar pace. The system ofFIGS.1and2provides a method to automatically, and optimally, dispatch many tables into different consistency groups in active-active (A-A) solutions, thereby ensuring data consistency across multiple consistency groups.

Turning now to an overview of the aspects of the invention, one or more embodiments of the invention address the above-described shortcomings of the prior art using the architecture illustrated inFIG.3. The architecture300includes a database management system302, with multiple tables304. A catalog table collect module306catalogs the tables304using any known cataloguing process and generates a table schema308. A real-time statistic collection module (RTS module310) generates real time statistic data312including an average workload rate and peak workload rate for each table304and monitors and collects real-time statistics about the tables304. A workload profiling collection module (WPC module320) collects the workload running snapshot list including the transaction pattern after decoding DBMS302logs and generates a workload profile data322output. In some examples, the workload profile data322output may be snapshots of transaction patters for every time interval.

The real-time statistics data312, table schema308, and workload profile data314are provided to a transaction splitting analysis module330to calculate a distribution relationship between the tables304and groups and to minimize splitting the tables of a single transaction into different consistency groups using a Girvan-Newman (GN) algorithm. The graph node algorithm generates an optimized multiple consistency group definition that is used to organize and maintain consistency across all the consistency groups, and the algorithm illustrated inFIGS.5-8.

The real-time statistic data is a portion of an input for, and is provided to, a transaction splitting analysis module. Workload profile data is going to be a part of input of a transaction splitting analysis module.

real-time statistics collect module310includes software configured to collect the real-time statistic from database management systems (DBMS302) running time, and includes specifying a concrete sampling start and end timestamp, combining the table schema308from catalog tables306and collecting the real-time statistics of each table304from DBMS302, and generating an average workload rate and peak workload rate for each table304. This information is collected as the real time statistic data312and is provided to the transaction splitting analysis module330. The optimized constancy groups are output from the transaction splitting analysis module as an optimized multiple consistency group definition332and the consistency groups are utilized in the standard operations of the computer system(s).

The workload profile collect module320is responsible to collect the workload running snapshot list and includes functions to read a DBMS302active/archive log for every requested time window and decode the log records. Identified transactions are clustered based on their accessed/updated tables and each cluster is a pattern. For each transaction pattern, the total transaction counts is recorded, the total insert/update/delete count for each table is recorded and the total insert/update/delete replication-based message size in bytes is recorded for each table. Then a snapshot is generated, with the snapshot including all the pattern statistics per sampling interval. The sampling interval is determined in one example based on database log timestamps.

The above-described aspects of the invention address the shortcomings of the prior art by using the transaction splitting analysis module330to apply a graph node algorithm to optimize the placement of tables within consistency groups.

Turning now to a more detailed description of aspects of the present invention, and with continued reference toFIGS.1-3,FIG.4depicts a graph node chart400as utilized by the transaction splitting analysis module330ofFIG.3. The graph node chart400includes multiple nodes410, with each node410representing one of the tables304. The weight of any given node410is the time series insert/update/delete (IUD) statistics generated by the RTS module310. Each edge420connecting two nodes indicates that there exists one or more transaction patterns that correlates both tables (nodes410). The weight of the total transaction instance counts from all related patterns.

With continued reference toFIGS.1-4,FIGS.5-8illustrate a process of operations of the transaction splitting analysis module330on an exemplary graph. It is appreciated that the exemplary graph is simplified for explanatory purposes, and the quantities and relative positions of nodes510are non-limiting and are purely for explanation.

The specific operations of the transaction splitting analysis module330include splitting the graph (e.g., graph400) based on a graph node algorithm. The graph node algorithm initially sums a total graph weight (e.g. 32300) with their nodes weight for each graph (See Graph500,FIG.5).

Then the algorithm computes each edge's betweenness centrality “g” according to:

Where σst(v) is a number of the shortest paths from vertex “s” to vertex “t” through edge υ, and σstis a total number of the shortest paths from vertex s to vertex t. Then the algorithm sorts edges according to their betweenness centrality value/edges weight and cuts some edges with higher betweenness centrality and/or edges weight to split the graph into multi sub-graphs510as shown inFIG.6. The algorithm repeats cutting edges with a higher betweenness centrality/edges weight to split the graph into multi sub-graphs, until the number of sub-graphs520is from 2 to 3 times of the total number of consistency groups.

The algorithm then sums the weight for every sub-graph520and computes the average weight for the sub-graphs520. The algorithm evaluates whether the weight of each sub-graph520is greater than one half the average weight of all the sub-graphs520and less then 3 half's of the average weight of the sub-graphs520. When the condition is not met (e.g. when at least one of the sub-graphs520has a weight that is less than half to average weight or more than 3 half's the average weight, the sub-graphs520are re-sorted by either one or both of splitting overweight sub-graphs520(sub-graphs520with a weight greater than 3 half's the average weight) to ensure that the weight of each sub-graph is less than 3 half's the average weight and combine under sized sub-graphs520(e.g. sub-graphs with a weight less than half the average sub-graph weight) according betweenness centrality value/edges weight among sub-graphs520to ensure that the formerly undersized sub-graphs520each have a weight greater than one half the average weight. The newly formed sub-graphs520are rechecked according to the same weight criteria, as shown inFIG.7.

The above process is re-iterated until all the sub-graphs520meet the criteria, as inFIG.8. Once the criteria is met, the nodes within each sub-graph520are grouped as a corresponding consistency group, and each sub-graph520constitutes a consistency group. The optimized consistency group are then output to the computing system as shown inFIG.3.

In some examples, the above system can be further enhanced by running the entire operation multiple times for different solutions in order to identify the minimum number of consistency groups that can meet the criteria.

By implementing the above processes within a computer system, the computer system can automatically dispatch many tables into different consistency groups in an active-active solution. This enhances the replication workload balance in the whole replication procedure from the log read, message queue transmission to transaction replay, and can minimize the impact of the transaction atomicity during replication.