Patent Description:
In the current information era, an "information overload" or even "information explosion" problem often accompanies abundant information resources shared by people. When massive data is processed, if a data transmission speed fluctuates, for example, when the data transmission speed increases sharply, a data processor cannot bear sharply increased data in a short time. Consequently, data congestion is caused in the data processor, and performance of a data processing system deteriorates, which reduces the data processing speed (e.g. transactions per second, TPS) of the system. To improve the data processing speed of the system, a performance monitor may be disposed in the system. When detecting that running of the system reaches an alarm condition, the performance monitor sends a system alarm signal, so that the system performs a data traffic control operation to reduce the data transmission speed, and reduce the data congestion. However, the data traffic control operation is performed on data of the entire system, and affects the data from different partitions. As a result, processing performance of the data processing system fluctuates.

<CIT> describes a stream processing system comprising a meta-scheduler node and a processing element execution node. The meta-scheduler node is to control CPU resources of the processing element execution node. Thereto, the meta-scheduler node communicates resource allocation targets to the processing element execution node. The processing element execution node is to execute a plurality of processing elements. A processing element is an application that performs a particular computation on data. The processing execution node includes a processing element monitor, which monitors the processing rate, input rate, and quantity of buffered data for each processing element in the processing element execution node.

<CIT> describes that a data flow in a communications network is controlled by a downstream node specifying the maximum average speeds of a plurality of data streams sent from an upstream node on a plurality of data paths.

In "<NPL>, techniques for parallelizing sliding window processing over data streams on a shared-nothing cluster of commodity hardware are describes. Data streams are first partitioned on the fly via a continuous split stage that takes the query semantics into account in a way that respects the natural chunking (windowing) of the stream by the query.

Advantageous embodiments of the invention are given by the dependent claims.

The following describes accompanying drawings used in embodiments of this application.

To help understanding of embodiments of this application, some related concepts or terms in the embodiments of this application are explained.

Data partition is allocating data in a table with a large amount of data to different system partitions, hard disks, or different server devices according to a preset partitioning policy, to achieve balanced data allocation. The data partition can balance the large amount of data to different storage media, so that each partition evenly stores a part of data. When the data in the data table is operated, a partition in which the data is located may be located. In addition, the data partition may also facilitate data management. A partition may be located for processing the data, without affecting another partition. This improves data processing efficiency. For example, when partitioning is performed based on a data storage date, and when data stored in a time period needs to be deleted, a partition corresponding to the time period may be directly located for deleting the data, without affecting a partition in which data in another time period is located. A control node in a data processing system may construct different data transmission pipes according to a partitioning policy to store the data from different partitions. In the embodiments of this application, the data partition may be a partition from which the data comes, or a partition into which the data is to be stored.

A Partitioning policy may be obtained by a control node in a data processing system based on a configuration file or through an operation node interface. The partitioning policy indicates a storage node on which data received by the data processing system needs to be stored. The control node constructs different data transmission pipes according to the partitioning policy. The different data transmission pipes store, in partitions according to the partitioning policy, data received from the outside or data actively obtained. Specifically, for example, one data transmission pipe is constructed for data from each partition. In other words, a quantity of partitions indicated by the partitioning policy is the same as a quantity of constructed data transmission pipes. For another example, one data transmission pipe may be constructed for data from several partitions indicated by the partitioning policy. A specific policy of constructing the data transmission pipe according to the partitioning policy is not limited in the embodiments of this application.

For example, the data partitioning policy may be partitioning based on a type of data, partitioning based on a value of a field of data, partitioning based on an initiation place, an initiator, or the like of data, or may be another policy. The specific partitioning policy is not limited in the embodiments of this application. Each data transmission pipe may be used to store data of one partition, or data of a plurality of partitions. Data storage and traffic control in the different data transmission pipes are independent, without affecting each other.

Traffic control is controlling data traffic in a data processing system when data processing performance of the data processing system deteriorates. The traffic control may include two types: One is a conventional traffic control manner. To be specific, a quality of service (QoS) module may be used to perform the traffic control based on a source address, a destination address, a source port, a destination port, and a protocol type. The other is an intelligent hardware or system software traffic control manner. To be specific, a traffic control device or traffic control software may be used to control a transmission rate and traffic control time of a data packet. A traffic control manner used in the embodiments of this application may be either of the two manners, or may be another traffic control manner, for example, a new traffic control manner in the future. When the traffic control is performed on data in a data transmission pipe, traffic control between different data transmission pipes is independent, without affecting each other. In other words, the traffic control is performed on data in a data transmission pipe, without affecting traffic of data in another data transmission pipe.

An operation node is configured to process data in a data processing system. Specifically, the data processing may be data operation, data importing, data query, or the like, or may be detecting whether a transaction is fraudulent. The operation node may extract data from a data transmission pipe, and perform processing processes such as operation, importing, or query. One operation node may extract data from a plurality of data transmission pipes and process the data, or may obtain data from only one data transmission pipe and process the data. This is not limited in the embodiments of this application. The operation node may be, for example, a database, or a processing node having a specific operation function. In the embodiments of this application, a function of the operation node may be implemented by a device, or may be implemented by several devices. This is not limited in the embodiments of this application. When the data processing system is a big data processing system, the operation node may be used to perform processing such as operation, data importing, and data query on big data. Refer to <FIG> is a schematic diagram of an architecture of a data processing system <NUM> according to an embodiment of this application. The data processing system <NUM> includes a control node <NUM>, a data transmission pipe <NUM>, a monitor <NUM>, and at least one operation node <NUM>. The monitor <NUM> may be disposed in the operation node <NUM>. The control node <NUM> is configured to construct the data transmission pipe <NUM>. The control node <NUM> is further configured to receive and combine data, and temporarily store the received data in the constructed data transmission pipe <NUM>. The data received by the control node <NUM> may be data from different partitions. As shown in <FIG>, the data received by the control node <NUM> may include data from a partition A, data from a partition B, and data from a partition C. The control node <NUM> may be specifically a start node that receives the data and initiates an operation in any framework of Storm, Flink, and Spark. The control node <NUM> may also be a node in another framework. This is not limited in this application. In this embodiment of this application, a function of the control node <NUM> may be implemented by a device, or may be implemented by several devices. This is not limited in the embodiment of this application. In the data processing system <NUM>, there may be one or more control nodes <NUM>. As shown in <FIG>, there may be one control node in the data processing system <NUM>. It may be understood that there may be a plurality of control nodes <NUM> in the data processing system <NUM>, and each control node <NUM> of the plurality of control nodes <NUM> is connected to the operation node <NUM> through the data transmission pipe <NUM>.

The data transmission pipe <NUM> is configured to transmit data from a user or another node to the operation node <NUM>. The data transmission pipe <NUM> is a logical concept and corresponds to a physical storage area. Different data transmission pipes <NUM> may be distributed on physical entities such as different system partitions, different hard disks, or different server devices. The data transmission pipe <NUM> may be a global data transmission pipe, and is configured to store data from all partitions. Alternatively, the data transmission pipe <NUM> may be a separated data transmission pipe, and is configured to separately store the data from different partitions. For example, as shown in <FIG>, for the data from the different partitions: the data from the partition A, the data from the partition B, and the data from the partition C, the control node <NUM> may establish a data transmission pipe. The data transmission pipe stores the data from the partition A, the data from the partition B, and the data from the partition C. In addition, the control node <NUM> may also establish three data transmission pipes. The three data transmission pipes respectively store the data from the partition A, the data from the partition B, and the data from the partition C.

The control node <NUM> is further configured to create the monitor <NUM> when the data processing system <NUM> is deployed or initialized. The monitor <NUM> is configured to monitor data processing performance. The monitor <NUM> may detect a response speed in a data processing process. For example, the monitor <NUM> detects a data request delay in a data processing process. The monitor <NUM> may further detect a data processing speed of an operation node. For example, the monitor <NUM> detects a data processing speed of a database.

The monitor <NUM> may be located in the operation node <NUM>, and may be created by the control node <NUM> when the data processing system <NUM> is deployed or initialized. The monitor <NUM> may be configured to perform performance monitoring in the data processing process in the operation node <NUM>. When detecting that the processing performance in the data processing process is excessively low, the monitor <NUM> sends an alarm signal indicating that the data processing performance is excessively low to the control node <NUM>. Specifically, when detecting that the data request delay in the data processing process exceeds a preset delay requirement threshold, or detecting that a data processing speed of an operation node is lower than a preset processing speed threshold, the monitor <NUM> sends the alarm signal to the control node <NUM>. The alarm signal is used to indicate that the data processing performance is excessively low.

The operation node <NUM> is configured to perform processing processes such as an operation, importing, or query on data. The operation node <NUM> may be specifically, for example, a database, or a processing node having a specific operation function. The database is configured to store data, and may further be configured to query data, delete data, or the like. The processing node is configured to extract data from the data transmission pipe <NUM> for processing the data, for example, perform an addition operation on the data, perform a subtraction operation on the data, query the data, or detect whether transaction data is fraudulent. After processing the data, the processing node may perform a cleanup work as required, for example, delete a message. When processing the data, the operation node <NUM> may distinguish between data transmission pipes from which the data comes, or may not distinguish between data transmission pipes from which the data comes. In the data processing system <NUM>, there may be one or more operation nodes <NUM>. As shown in <FIG>, there may be one operation node in the data processing system <NUM>. It may be understood that there may be a plurality of operation nodes <NUM> in the data processing system <NUM>, and each operation node <NUM> of the plurality of operation nodes is connected to the control node <NUM> through the data transmission pipe <NUM>.

In this embodiment of this application, a function of the operation node <NUM> may be implemented by a device, or may be implemented by several devices. This is not limited in this embodiment of this application. The operation node <NUM> is not limited to the foregoing example, or may be another node that needs to be detected. This is not limited in this embodiment of this application.

The control node <NUM> is further configured to, when receiving the alarm signal, perform traffic control on the data from the data transmission pipe <NUM>. Specifically, that the control node <NUM> performs the traffic control may be reducing a speed at which data is put into the data transmission pipe <NUM>, or may be reducing a speed at which the operation node <NUM> extracts data from the data transmission pipe <NUM>. When performing the traffic control on the data from the data transmission pipe <NUM>, the control node <NUM> may separately perform the traffic control in the different data transmission pipes <NUM>. To be specific, performing the traffic control in a data transmission pipe does not affect traffic in another data transmission pipe.

It should be noted that the data processing system shown in <FIG> is only intended to more clearly describe technical solutions in this application, but is not intended to limit this application. A person of ordinary skill in the art may know that as a system architecture evolves and a new service scenario emerges, the technical solutions provided in this application are applied to a similar technical problem.

When a data traffic control operation is performed, global traffic control is performed on data of the entire system without distinguishing partitions. The data from the different partitions is affected by the traffic control operation, and processing performance of the data processing system fluctuates.

Based on the schematic diagram of the architecture of the data processing system in <FIG>, the embodiments of this application provide a data processing method and system, to improve the data processing performance in the data processing system.

In this application, for data from a data transmission pipe in a data processing system, a monitor is used to monitor data processing performance. When the monitor detects that the processing performance of the data from a first data transmission pipe is lower than a first threshold, and after a control node receives an alarm signal, the control node reduces a data amount that is from the first data transmission pipe and that is allocated to the operation node. The first data transmission pipe may be any data transmission pipe in a plurality of data transmission pipes. The foregoing procedure is performed to monitor data from different data transmission pipes at a finer granularity level. When processing performance of data from a data transmission pipe is detected to be comparatively low, traffic control is performed only on the data from the data transmission pipe, without affecting traffic of data from another data transmission pipe. Therefore, this can reduce a case in which traffic of data from different partitions is reduced during the traffic control, and reduce fluctuation in processing performance of the data processing system during the traffic control.

The embodiments of this application provide a data processing system. Refer to <FIG> is a schematic diagram of a structure of a data processing system <NUM> according to an embodiment of this application. As shown in <FIG>, in the data processing system <NUM>, a control node <NUM> establishes a plurality of data transmission pipes <NUM> for data from different partitions. The control node <NUM> transmits received data to an operation node <NUM> through the plurality of data transmission pipes <NUM>. A plurality of monitors <NUM> are disposed in the data processing system <NUM> for the data from the data transmission pipe. The monitor is configured to monitor data processing performance for data from each data transmission pipe.

The control node <NUM> is configured to, when receiving an alarm signal, reduce a data amount that is from a first data transmission pipe and that is allocated to the operation node <NUM>. The first data transmission pipe is any data transmission pipe in the plurality of data transmission pipes.

Specifically, as shown in <FIG>, the control node <NUM> may separately establish three independent data transmission pipes for data from a partition A, data from a partition B, and data from a partition C according to a partitioning policy: a data transmission pipe <NUM>, a data transmission pipe <NUM>, and a data transmission pipe <NUM>. Data transmission and traffic control between the three data transmission pipes do not affect each other.

The data transmission pipe <NUM> is configured to transmit received data from the partition A to the operation node <NUM>. The data transmission pipe <NUM> is configured to transmit received data from the partition B to the operation node <NUM>. The data transmission pipe <NUM> is configured to transmit received data from the partition C to the operation node <NUM>. In the data processing system <NUM>, the monitor is disposed for the data from each data transmission pipe to detect the data processing performance. As shown in <FIG>, a monitor <NUM> is disposed for data from the data transmission pipe <NUM>. A monitor <NUM> is disposed for data from the data transmission pipe <NUM>. A monitor <NUM> is disposed for data from the data transmission pipe <NUM>. For the data from the data transmission pipe <NUM>, when the monitor <NUM> detects that processing performance of the data from the data transmission pipe <NUM> is lower than a first threshold, the monitor <NUM> may send the alarm signal to the control node <NUM>. The alarm signal is used to indicate that the processing performance of the data from the data transmission pipe <NUM> is excessively low. After receiving the alarm signal, the control node <NUM> reduces a data amount that is from the data transmission pipe <NUM> and that is allocated to the operation node <NUM>. For the data from the data transmission pipe <NUM>, when the monitor <NUM> detects that processing performance of the data from the data transmission pipe <NUM> is lower than the first threshold, the monitor <NUM> may send the alarm signal to the control node <NUM>, to indicate that the processing performance of the data from the data transmission pipe <NUM> is excessively low. After receiving the alarm signal, the control node <NUM> reduces a data amount that is from the data transmission pipe <NUM> and that is allocated to the operation node <NUM>. For the data from the data transmission pipe <NUM>, when the monitor <NUM> detects that processing performance of the data from the data transmission pipe <NUM> is lower than the first threshold, the monitor <NUM> may send the alarm signal to the control node <NUM>, to indicate that the processing performance of the data from the data transmission pipe <NUM> is excessively low. After receiving the alarm signal, the control node <NUM> reduces a data amount that is from the data transmission pipe <NUM> and that is allocated to the operation node <NUM>.

It should be noted that processes of monitoring and the traffic control performed on data from different data transmission pipes are independent. In other words, monitoring on the data from the data transmission pipe <NUM> does not affect monitoring on the data from the data transmission pipe <NUM>, and does not affect monitoring on the data from the data transmission pipe <NUM>. The traffic control performed in the data transmission pipe <NUM> does not affect the data from the data transmission pipe <NUM>, and does not affect the data from the data transmission pipe <NUM>. In addition, first thresholds set for the data from the different data transmission pipes may be the same or may be different. This is not limited in this embodiment of this application.

As shown in <FIG>, the three monitors are respectively used to monitor data from the three data transmission pipes in the operation node <NUM>. If the processing performance of the data from one data transmission pipe in the three data transmission pipes is excessively low, the control node <NUM> performs the traffic control only on the data from the data transmission pipe with the excessively low data processing performance, and does not perform the traffic control on data in another data transmission pipe. The performance monitoring and the traffic control are performed at the finer granularity level. Therefore, this can reduce a case in which traffic of data from different partitions is reduced during the traffic control, improve an overall data processing speed, and reduce fluctuation in processing performance of the data processing system during the traffic control.

It may be understood that this example is merely used to explain this embodiment of this application, and shall not be construed as a limitation. A quantity of monitors <NUM> and a quantity of data transmission pipes <NUM> are not limited to three, and may also be another quantity.

The data processing system <NUM> may be a distributed system, or may be a centralized system. This is not limited in this embodiment of this application. All the control node <NUM>, the data transmission pipe <NUM>, the monitor <NUM>, and the operation node <NUM> may be implemented by a virtual machine. For specific descriptions of the control node <NUM>, the data transmission pipe <NUM>, the monitor <NUM>, and the operation node <NUM>, refer to the architecture of the data processing system described in <FIG>.

There may be a plurality of operation nodes <NUM> in the data processing system <NUM>. Refer to <FIG> is a schematic diagram of a structure of another data processing system <NUM> according to an embodiment of this application. In the data processing system <NUM>, there may be a plurality of operation nodes <NUM>. In the plurality of operation nodes <NUM>, data processing performance monitoring for distinguishing data transmission pipes may be performed in each operation node. Alternatively, the data processing performance monitoring for distinguishing the data transmission pipes may be performed on some operation nodes that need the data processing performance monitoring. That the processing performance monitoring is performed in which operation node is determined based on a service requirement. This is not limited in this embodiment of this application.

As shown in <FIG>, two operation nodes <NUM> are used as an example in <FIG>: a database <NUM> and a processing node <NUM>. In the data processing system <NUM>, the database <NUM> may be set as a monitoring object. A monitor <NUM> is disposed to detect processing performance of data from different data transmission pipes <NUM> in the database, but does not perform the data processing performance monitoring on the processing node <NUM>. A monitor <NUM> is disposed to perform data processing performance monitoring on data from a data transmission pipe <NUM>. A monitor <NUM> is disposed to perform data processing performance monitoring on data from a data transmission pipe <NUM>. A monitor <NUM> is disposed to perform data processing performance monitoring on data from a data transmission pipe <NUM>. The data processing performance monitoring may not be performed on the processing node <NUM> based on a service requirement.

For specific descriptions of a control node <NUM>, the data transmission pipe <NUM>, the monitor <NUM>, and the operation node <NUM>, refer to embodiments described in <FIG> and <FIG>.

Optionally, the data processing system <NUM> may include a plurality of control nodes <NUM>. Partitions of data received by the plurality of control nodes may be totally different, may be the same, or may be partially the same. Refer to <FIG> is a schematic diagram of a structure of still another data processing system <NUM> according to an embodiment of this application.

As shown in <FIG>, the data processing system <NUM> may include a plurality of control nodes. Two control nodes are used as an example in <FIG>: a control node <NUM> and a control node <NUM>'. Partitions to which data received by different control nodes belongs may be partially the same. As shown in <FIG>, the control node <NUM> may receive data from a partition A, data from a partition B, and data from a partition C, and allocate the data from the partition A, the data from the partition B, and the data from the partition C to an operation node <NUM>. The control node <NUM>' may receive the data from the partition A, the data from the partition B, and data from a partition D, and allocate the data from the partition A, the data from the partition B, and the data from the partition D to an operation node <NUM>'. A monitor <NUM> is used to monitor data processing performance in a data transmission pipe <NUM> in the operation node <NUM>. A monitor <NUM> is used to monitor data processing performance in a data transmission pipe <NUM> in the operation node <NUM>. A monitor <NUM> is used to monitor data processing performance in a data transmission pipe <NUM> in the operation node <NUM>. For the operation node <NUM>', a monitor <NUM><NUM>' is used to monitor data processing performance in a data transmission pipe <NUM>' in the operation node <NUM>'. A monitor <NUM>' is used to monitor data processing performance in a data transmission pipe <NUM>' in the operation node <NUM>'. A monitor <NUM>' is used to monitor data processing performance in a data transmission pipe <NUM>' in the operation node <NUM>'.

For specific descriptions of the control node, the data transmission pipe, the monitor, and the operation node, refer to embodiments described in <FIG> and <FIG>.

Optionally, a quantity of data transmission pipes <NUM> disposed in the control node <NUM> may be different from a quantity of partitions to which the received data belongs. Refer to <FIG> is a schematic diagram of a structure of yet still another data processing system <NUM> according to an embodiment of this application. As shown in <FIG>, a control node <NUM> may establish two data transmission pipes for received data: a data transmission pipe <NUM> and a data transmission pipe <NUM>. The data transmission pipe <NUM> is configured to transmit the received data from a partition A to an operation node <NUM>. The data transmission pipe <NUM> is configured to transmit the received data from both a partition B and a partition C to the operation node <NUM>. A monitor <NUM> is configured to monitor processing performance of data from the data transmission pipe <NUM>, in the operation node <NUM>. A monitor <NUM> is configured to monitor processing performance of data from the data transmission pipe <NUM>, in the operation node <NUM>.

For specific descriptions of the control node <NUM>, a data transmission pipe <NUM>, a monitor <NUM>, and the operation node <NUM>, refer to embodiments described in <FIG> and <FIG>.

Optionally, a function of the monitor <NUM> may be implemented by the operation node <NUM>. In other words, the monitor <NUM> is located in the operation node <NUM>. The function of the monitor <NUM> may further be implemented by the control node <NUM>. In other words, the monitor <NUM> is located in the control node <NUM>. The function of the monitor <NUM> may further be implemented by an independent device. In other words, the monitor <NUM> is implemented as the independent device.

Refer to <FIG> is a schematic diagram of a structure of still another data processing system <NUM> according to an embodiment of this application. A monitor <NUM>, a monitor <NUM>, and a monitor <NUM> may be integrated into an operation node <NUM>, may be integrated into a control node <NUM>, or may be implemented by an independent device. In addition, the monitor <NUM>, the monitor <NUM>, and the monitor <NUM> may be integrated into one independent device, or the monitor <NUM>, the monitor <NUM>, and the monitor <NUM> may be integrated into three independent devices respectively. This is not limited in this embodiment of this application.

When a monitor <NUM> detects that processing performance of data from a data transmission pipe is lower than a first threshold, the monitor <NUM> may send an alarm signal to the control node <NUM>, or the monitor <NUM> may instruct the operation node <NUM> to send an alarm signal to the control node <NUM>. This is not limited in this embodiment of this application.

For specific descriptions of the control node <NUM>, a data transmission pipe <NUM>, the monitor <NUM>, and the operation node <NUM>, refer to embodiments described in <FIG> and <FIG>.

The foregoing examples of the data processing system <NUM> are merely used to explain this embodiment of this application, and should not constitute a limitation.

Based on the data processing system, an embodiment of this application further provides a data processing method. The method is applied to the data processing system described in any one of <FIG>. Refer to <FIG> is a schematic flowchart of a data processing method according to this embodiment of this application. As shown in <FIG>, the data processing method includes but is not limited to the following steps: S101: In the data processing system, a monitor is used to monitor data processing performance of data from each data transmission pipe.

S102: Send an alarm signal to a control node when the monitor detects that processing performance of data from a first data transmission pipe is lower than a first threshold. The first data transmission pipe may be any data transmission pipe in a plurality of data transmission pipes. The alarm signal is used to indicate that the processing performance of the data from the first data transmission pipe is excessively low.

S103: When receiving the alarm signal, the control node reduces a data amount that is from the first data transmission pipe and that is allocated to an operation node.

Optionally, before the monitor is used to monitor the data processing performance for the data from each data transmission pipe in the operation node, the control node may construct the plurality of data transmission pipes according to partitioning policy information. The plurality of data transmission pipes are used to separately transmit data received by the control node from a user or another node. A quantity of data partitions obtained according to the partitioning policy information may be the same as or different from a quantity of constructed data transmission pipes. For example, as shown in <FIG>, one data transmission pipe may be constructed for each type of partition data, and three data transmission pipes are constructed to respectively transmit the three types of partition data. For another example, as shown in <FIG>, the data transmission pipe <NUM> is constructed for the data from the partition A, and is configured to transmit the data from the partition A to the operation node. The data from both the partition B and the partition C is constructed with the same data transmission pipe <NUM>. The quantity of data transmission pipes created according to a partitioning policy is not limited in this embodiment of this application.

Optionally, the monitor may be created by the operation node for each data transmission pipe. To be specific, in the operation node, one monitor is created for the data from each data transmission pipe to perform performance monitoring in a processing process. A quantity of monitors may be the same as the quantity of data transmission pipes.

Optionally, that each monitor detects the processing performance of data from the data transmission pipe may include two implementations, which are separately described below:
A first manner may be that the monitor is used to monitor a data processing speed of a processor in the operation node. A second threshold may be preset for the data processing speed of the processor. When the monitor detects that the data processing speed of the processor is less than or equal to the second threshold, the operation node sends the alarm signal to the control node.

A second manner may be the monitor is used to monitor a response speed of a data processing request in the operation node, or a delay of a data processing request is used to represent a response speed of the data processing request. A third threshold may be preset for the response speed of the data processing request. When the monitor detects that the response speed of the data processing request is less than or equal to the third threshold, the operation node sends, to the control node, the alarm signal used to indicate that the processing performance of the data from the first data transmission pipe is excessively low.

The monitor may monitor the processing performance of the data from the data transmission pipe in any one or two of the foregoing manners. This is not limited in this embodiment of this application.

Optionally, that the control node reduces the data amount that is from the first data transmission pipe and that is allocated to the operation node may also include two implementations, which are separately described below.

A first manner is that the control node may reduce a speed of placing data into the first data transmission pipe, to reduce a speed of data transmission in the first data transmission pipe. In other words, a speed of the data that is from the first data transmission pipe and that is flowing into the operation node may be reduced. This may reduce congestion of processing the data from the first data transmission pipe by the operation node, and improve processing performance of the data processing system.

A second manner is that the control node may keep a speed of placing data into the first data transmission pipe unchanged, but reduce a speed at which the first data transmission pipe transmits the data to the operation node. This may reduce congestion of processing the data from the first data transmission pipe by the operation node, and improve processing performance of the data processing system. Reducing the speed at which the first data transmission pipe transmits the data to the operation node may be reducing the speed at which the first data transmission pipe sends the data to the operation node, or may be reducing a speed at which the operation node extracts the data from the first data transmission pipe. This is not limited in this embodiment of this application.

The control node may reduce, in any one or two of the foregoing manners, the data amount that is from the first data transmission pipe and that is allocated to the operation node. This is not limited in this embodiment of this application.

Optionally, there may be a plurality of operation nodes in the data processing system. There may be one or more operation nodes that perform the data processing performance monitoring. When the plurality of operation nodes need to perform the data processing performance monitoring and data traffic control, the data processing performance monitoring and data traffic control between the plurality of operation nodes may be separately performed, without affecting each other. The data processing performance monitoring and the data traffic control between the plurality of operation nodes may also be performed in the first data transmission pipe. The control node determines by comprehensively considering results of the monitoring performed by the monitor in the plurality of operation nodes, a data amount that is from the first data transmission pipe and that is allocated to each operation node.

For example, if both operation nodes A and B need to perform the data processing performance monitoring and the data traffic control, a process in which an operation node A performs the monitoring and the traffic control in the first data transmission pipe and a process in which an operation node B performs the monitoring and the traffic control in the first data transmission pipe may be separately performed, without affecting each other. That the control node reduces the data amount that is from the first data transmission pipe and that is allocated to the operation node may be implemented by reducing the speed at which the first data transmission pipe transmits the data to the operation node.

In the foregoing example, the control node may also perform the traffic control by comprehensively considering results detected on the operation node A and the operation node B by using the monitor. Specifically, for example, the speed of placing the data into the first data transmission pipe may be reduced only when the control node receives a first alarm signal sent by the operation node A and receives a second alarm signal sent by the operation node B. The first alarm signal is used to indicate that the processing performance of the data from the first data transmission pipe is excessively low. The second alarm signal is used to indicate that the processing performance of the data from the first data transmission pipe is excessively low. The control node does not respond to an alarm signal when receiving the alarm signal that is sent by either of the operation node A and the operation node B and that indicates that the processing performance of the data from the first data transmission pipe is excessively low.

The foregoing example is merely used to explain this embodiment of this application. When there are the plurality of operation nodes, a specific policy for the data processing performance monitoring and the data traffic control may be designed based on a user requirement or a service requirement for using the data processing system. This is not limited in this embodiment of this application.

A function of the monitor may be implemented by the operation node, may be implemented by the control node, or may be implemented by an independent device. This is not limited in this embodiment of this application.

When the data processing system is a real-time data processing system, the real-time data processing system requires a fast data processing response speed. Compared with another processing node (for example, a processing node configured to perform an operation such as data calculation or deletion/modification), a database processes a larger data amount, and has a more complex data processing process. The data processing process in the database can better reflect the data processing performance of the entire data processing system. Therefore, if the data processing system includes the plurality of operation nodes, the operation node selected for the monitoring may be the database. In a scenario of the real-time data processing system, an operation node selected for the monitoring is the database, so that the data processing performance of the data processing system is monitored more accurately, and the data traffic control is performed more accurately.

The foregoing describes in detail the method in the embodiments of this application. The following provides apparatuses in the embodiments of this application.

Refer to <FIG> is a schematic diagram of structures of an operation node <NUM> and a control node <NUM> according to an embodiment of this application. As shown in <FIG>, the operation node <NUM> and the control node <NUM> are applied to a data processing system <NUM>. The data processing system <NUM> may be the data processing system described in any one of <FIG>.

The control node <NUM> may include a receiving module <NUM>, a control module <NUM>, and a plurality of data transmission pipes <NUM>. The receiving module <NUM> is configured to receive data. The data may be data from different partitions.

As shown in <FIG>, the data received by the receiving module <NUM> may include data from a partition A, data from a partition B, and data from a partition C.

The control module <NUM> is configured to separately send the data received by the receiving module <NUM> to the operation node <NUM> through the plurality of data transmission pipes <NUM>.

As shown in <FIG>, the plurality of data transmission pipes <NUM> may include a data transmission pipe <NUM>, a data transmission pipe <NUM>, and a data transmission pipe <NUM>.

The operation node <NUM> may include a processing module <NUM>, a monitor module <NUM>, and a sending module <NUM>. The processing module <NUM> is configured to receive the data from the plurality of data transmission pipes <NUM> for processing.

The monitor module <NUM> is configured to monitor processing performance of the data from the plurality of data transmission pipes.

The sending module <NUM> is configured to send an alarm signal to the control node <NUM> when the monitor module <NUM> detects that processing performance of data from a first data transmission pipe is lower than a first threshold. The alarm signal is used to indicate that the processing performance of the data from the first data transmission pipe is excessively low. The first data transmission pipe is any data transmission pipe in the plurality of data transmission pipes. To be specific, as shown in <FIG>, the first data transmission pipe may be any data transmission pipe in the data transmission pipe <NUM>, the data transmission pipe <NUM>, and the data transmission pipe <NUM>.

Specifically, the alarm signal may be sent by the sending module <NUM> to the control module <NUM>.

The control module <NUM> is further configured to, when receiving the alarm signal, reduce a data amount that is from the first data transmission pipe and that is allocated to the processing module <NUM>.

Optionally, the monitor module <NUM> is specifically configured to perform at least one of the following for data from each data transmission pipe: monitoring a data processing speed of the processing module <NUM>, and monitoring a processing response speed of the processing module <NUM>.

Optionally, the sending module <NUM> is specifically configured to send the alarm signal to the control module <NUM> when the monitor module <NUM> detects that a processing speed of the data from the first data transmission pipe is lower than a second threshold.

Optionally, the sending module <NUM> is specifically configured to send the alarm signal to the control module <NUM> when the monitor module <NUM> detects that a processing response speed of the data from the first data transmission pipe is lower than a third threshold.

Optionally, the control module <NUM> is specifically configured to perform at least one of the following when receiving the alarm signal: reducing a rate at which the first data transmission pipe receives data, and reducing a rate at which the first data transmission pipe transmits data to the processing module <NUM>.

The plurality of data transmission pipes <NUM> are created by the control module <NUM> according to partitioning policy information, and a partition of the data received by the receiving module <NUM> is determined according to the partitioning policy information.

Optionally, when the data processing system <NUM> is a real-time data processing system, and when there are a plurality of operation nodes and the operation node includes at least one database, the operation node <NUM> is a database. The monitor module <NUM> is configured to monitor processing performance of data separately from the plurality of data transmission pipes <NUM>, in the database.

In this embodiment, for functions of the modules, refer to corresponding descriptions in the embodiment of the data processing method shown in <FIG>.

Refer to <FIG> is a schematic diagram of structures of an operation node <NUM>, a monitor <NUM>, and a control node <NUM> according to an embodiment of this application. As shown in <FIG>, the operation node <NUM>, the monitor <NUM>, and the control node <NUM> are applied to a data processing system <NUM>. The data processing system <NUM> may be the data processing system described in any one of <FIG>.

The operation node <NUM> may include a processing module <NUM>, configured to receive data from the plurality of data transmission pipes <NUM> for processing.

The monitor <NUM> may include a monitor module <NUM> and a sending module <NUM>.

The monitor module <NUM> is configured to monitor processing performance of the processing module <NUM> on the data from the plurality of data transmission pipes <NUM>.

Optionally, the sending module <NUM> may further be included in the operation node <NUM>, and is configured to send the alarm signal to the control module <NUM> when the monitor module <NUM> detects that a processing speed of the data from the first data transmission pipe is lower than a first threshold.

Optionally, the sending module <NUM> specifically sends the alarm signal to the control module <NUM> when the monitor module <NUM> detects that a processing response speed of the data from the first data transmission pipe is lower than a third threshold.

Refer to <FIG> is a schematic diagram of a structure of still another data processing system <NUM> according to this embodiment of this application. The data processing system <NUM> includes a control node <NUM> and an operation node <NUM>. The control node <NUM> establishes a communication connection to the operation node <NUM>. A plurality of data transmission pipes <NUM> are established between the control node <NUM> and the operation node <NUM>. The control node <NUM> transmits received data to the operation node <NUM> through the plurality of data transmission pipes <NUM>. A monitor <NUM> is separately disposed in the data processing system <NUM> for data from the plurality of data transmission pipes <NUM>, and is configured to monitor processing performance of the data from the plurality of data transmission pipes <NUM>.

As shown in <FIG>, the control node <NUM> provided in this embodiment of this application includes one or more processors <NUM>, a communications interface <NUM>, and a memory <NUM>. The processor <NUM>, the communications interface <NUM>, and the memory <NUM> may be connected through a bus or in another manner. In this embodiment of the present invention, for an example, the processor <NUM>, the communications interface <NUM>, and the memory <NUM> are connected through a bus <NUM>. The processor <NUM> may include one or more general purpose processors, for example, one or more central processing units (CPU).

The processor <NUM> may be configured to run related program code <NUM>, and the related program code <NUM> includes the plurality of data transmission pipes <NUM>. In other words, the processor <NUM> may execute the program code <NUM> to implement functions of the plurality of data transmission pipes <NUM>. The data transmission pipe <NUM> is a logical concept and corresponds to a storage area.

The communications interface <NUM> may be a wired interface (for example, an Ethernet interface) or a wireless interface (for example, a cellular network interface or a wireless local area network interface), and is configured to communicate with another node. In this embodiment of this application, the communications interface <NUM> may be specifically configured to receive an alarm signal sent by the operation node <NUM>, and further send data from the data transmission pipe <NUM> to the operation node <NUM>.

The memory <NUM> may include a volatile memory, for example, a random access memory (RAM). The memory <NUM> may further include a non-volatile memory, for example, a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD). Alternatively, the memory <NUM> may include a combination of the foregoing types of memories. The memory <NUM> may be configured to store a group of program code <NUM>, so that the processor <NUM> invokes the program code <NUM> stored in the memory <NUM> to implement a method for implementing the data processing method in the embodiments of the present invention on a side of the control node <NUM>.

In a specific application scenario, the data transmission pipe <NUM> and the like are software modules. The software modules may be deployed on a server, a virtual machine on a server, or a container on a server.

It should be noted that the control node <NUM> shown in <FIG> is only an implementation of this embodiment of this application. In an actual application, the control node <NUM> may further include more or fewer components. This is not limited herein.

As shown in <FIG>, the operation node <NUM> provided in this embodiment of this application includes one or more processors <NUM>, a communications interface <NUM>, and a memory <NUM>. The processor <NUM>, the communications interface <NUM>, and the memory <NUM> may be connected through a bus or in another manner. In this embodiment of the present invention, for an example, the processor <NUM>, the communications interface <NUM>, and the memory <NUM> are connected through a bus <NUM>. The processor <NUM> may include one or more general purpose processors, for example, one or more central processing units CPUs.

The processor <NUM> may be configured to run related program code <NUM>, and the related program code <NUM> includes the monitor <NUM>. In other words, the processor <NUM> may execute the program code <NUM> to implement any one or more functions of the monitor <NUM>.

The communications interface <NUM> may be a wired interface (for example, an Ethernet interface) or a wireless interface (for example, a cellular network interface or a wireless local area network interface), and is configured to communicate with another node. In this embodiment of this application, the communications interface <NUM> may be specifically configured to send the alarm signal to the control node <NUM>, and may further receive the data transmitted in the data transmission pipe <NUM>.

The memory <NUM> may include a volatile memory, for example, a RAM. The memory <NUM> may further include a non-volatile memory, for example, a ROM, a flash memory, an HDD, or a solid-state drive SSD. Alternatively, the memory <NUM> may include a combination of the foregoing types of memories. The memory <NUM> may be configured to store a group of program code <NUM>, so that the processor <NUM> invokes the program code <NUM> stored in the memory <NUM> to implement a method for implementing the data processing method in the embodiments of the present invention on a side of the operation node <NUM>.

In a specific application scenario, the monitor <NUM> and the like are software modules.

The software modules may be deployed on a server, a virtual machine on a server, or a container on a server.

It should be noted that the operation node <NUM> shown in <FIG> is only an implementation of this embodiment of this application. In an actual application, the operation node <NUM> may further include more or fewer components. This is not limited herein.

Refer to <FIG> is a schematic diagram of a structure of still another data processing system <NUM> according to this embodiment of this application. The data processing system <NUM> includes a control node <NUM>, a monitor <NUM>, and an operation node <NUM>. A plurality of data transmission pipes <NUM> are established between the control node <NUM> and the operation node <NUM>. The control node <NUM> transmits received data to the operation node <NUM> through the plurality of data transmission pipes <NUM>. The monitor <NUM> is separately disposed in the data processing system <NUM> for data from the plurality of data transmission pipes <NUM>, and is configured to monitor processing performance of the data from the plurality of data transmission pipes <NUM>. There may be a plurality of monitors <NUM>. One monitor <NUM> may be disposed for each data transmission pipe. The monitor <NUM> is configured to monitor processing performance of data from the data pipe. The control node <NUM> establishes a communication connection to the operation node <NUM>. The control node <NUM> establishes a communication connection to each monitor <NUM>. Each monitor <NUM> establishes a communication connection to the operation node <NUM>. The plurality of monitors <NUM> may be implemented by one device, or may be implemented by a plurality of devices.

As shown in <FIG>, the control node <NUM> provided in this embodiment of this application includes one or more processors <NUM>, a communications interface <NUM>, and a memory <NUM>. The processor <NUM>, the communications interface <NUM>, and the memory <NUM> may be connected through a bus or in another manner. In this embodiment of this application, for an example, the processor <NUM>, the communications interface <NUM>, and the memory <NUM> are connected through a bus <NUM>. The processor <NUM> may include one or more general purpose processors, for example, one or more CPUs.

The communications interface <NUM> may be a wired interface (for example, an Ethernet interface) or a wireless interface (for example, a cellular network interface or a wireless local area network interface), and is configured to communicate with another node. In this embodiment of this application, the communications interface <NUM> may be specifically configured to receive an alarm signal sent by the monitor <NUM>, and further send data from the data transmission pipe <NUM> to the operation node <NUM>.

The memory <NUM> may include a volatile memory, for example, a RAM. The memory <NUM> may further include a non-volatile memory, for example, a ROM, a flash memory, an HDD, or a solid-state drive SSD. Alternatively, the memory <NUM> may include a combination of the foregoing types of memories. The memory <NUM> may be configured to store a group of program code <NUM>, so that the processor <NUM> invokes the program code <NUM> stored in the memory <NUM> to implement a method for implementing the data processing method in the embodiments of this application on a side of the control node <NUM>.

As shown in <FIG>, the monitor <NUM> provided in this embodiment of this application includes one or more processors <NUM>, a communications interface <NUM>, and a memory <NUM>. The processor <NUM>, the communications interface <NUM>, and the memory <NUM> may be connected through a bus or in another manner. In this embodiment of this application, for an example, the processor <NUM>, the communications interface <NUM>, and the memory <NUM> are connected through a bus <NUM>. The processor <NUM> may include one or more general purpose processors, for example, one or more central processing units CPUs. The processor <NUM> may be configured to run related program code <NUM>, and the related program code <NUM> includes the monitor <NUM>. In other words, the processor <NUM> may execute the program code <NUM> to implement a function on a side of the monitor <NUM>.

The communications interface <NUM> may be a wired interface (for example, an Ethernet interface) or a wireless interface (for example, a cellular network interface or a wireless local area network interface), and is configured to communicate with another node. In this embodiment of this application, the communications interface <NUM> may be specifically configured to send an alarm signal to the control node <NUM>, and may further monitor data processing performance of the operation node <NUM>.

The memory <NUM> may include a volatile memory, for example, a RAM. The memory <NUM> may further include a non-volatile memory (, for example, a ROM, a flash memory, an HDD, or a solid-state drive SSD. Alternatively, the memory <NUM> may include a combination of the foregoing types of memories. The memory <NUM> may be configured to store a group of program code <NUM>, so that the processor <NUM> invokes the program code <NUM> stored in the memory <NUM> to implement a method for implementing the data processing method in the embodiments of this application on a side of the monitor <NUM>. In a specific application scenario, the monitor <NUM> may be software modules. The software modules may be deployed on a server, a virtual machine on a server, or a container on a server.

It should be noted that the operation node <NUM> shown in <FIG> is only an implementation of this embodiment of this application. In an actual application, the monitor <NUM> may further include more or fewer components. This is not limited herein. An embodiment of this application further provides a chip system. The chip system includes at least one processor, a memory, and an interface circuit. The memory, the transceiver, and the at least one processor are interconnected through a line. The at least one memory stores an instruction. When the instruction is executed by the processor, the method procedure shown in <FIG> is implemented.

An embodiment of this application further provides a computer-readable storage medium. The computer-readable storage medium stores an instruction. When the instruction is run on a processor, the method procedure shown in <FIG> is implemented. An embodiment of this application further provides a computer program product. When the computer program product is run on a processor, the method procedure shown in <FIG> is implemented.

It may be understood that, in this application, technical terms and technical solutions in different embodiments may be mutually referenced and mutually cited based on logic of the embodiments. The embodiments to which the technical terms and the technical solutions are applicable are not limited in this application. The technical solutions in the different embodiments are combined with each other, and a new embodiment may further be formed.

All or some of the foregoing embodiments may be implemented by using software, hardware, firmware, or any combination thereof. When software is used to implement the embodiments, all or some of the embodiments may be implemented in a form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the procedure or functions according to the embodiments of this application are all or partially generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or another programmable apparatus. The computer instruction may be stored in a computer-readable storage medium, or may be transmitted by using the computer-readable storage medium. The computer instruction may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired (for example, a coaxial cable, an optical fiber, or a digital subscriber line (DSL)) or wireless (for example, infrared, radio, or microwave) manner. The computer-readable storage medium may be any usable medium accessible by the computer, or a data storage device, such as a server or a data center, integrating one or more usable media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium (for example, a DVD), a semiconductor medium (for example, a solid-state drive (SSD)), or the like.

Claim 1:
A data processing method, wherein the method is applied to a data processing system (<NUM>), the data processing system comprises a control node (<NUM>) and an operation node (<NUM>), a plurality of data transmission pipes (<NUM>, <NUM>, <NUM>) are established between the control node and the operation node, the control node transmits received data to the operation node through the plurality of data transmission pipes, a monitor (<NUM>) is disposed in the data processing system for the plurality of data transmission pipes, and the method comprises:
sending (S <NUM>), by the operation node (<NUM>), an alarm signal to the control node (<NUM>) when the monitor (<NUM>) detects that processing performance of data from a first data transmission pipe is lower than a first threshold, wherein the alarm signal is used to indicate that the processing performance of the data from the first data transmission pipe is excessively low, and the first data transmission pipe is any data transmission pipe in the plurality of data transmission pipes (<NUM>, <NUM>, <NUM>); and
when receiving the alarm signal, reducing (S103), by the control node (<NUM>), a data amount that is from the first data transmission pipe and that is allocated to the operation node (<NUM>) without affecting traffic of data from another data transmission pipe of the plurality of data transmission pipes; and
wherein the plurality of data transmission pipes (<NUM>, <NUM>, <NUM>) established between the control node (<NUM>) and the operation node (<NUM>) are created by the control node according to partitioning policy information, and the partitioning policy information is used by the control node to determine a partition used to store data received by the data processing system, and each data transmission pipe is created for data from one or more partitions indicated by the partitioning policy.