Patent Description:
The present invention relates to the field of computing technologies, and in particular, to a tree topology based computing system and method.

Artificial intelligence (AI) application grows explosively. The AI application is based on a deep neural network. Recently, the deep neural network has been applied to fields such as speech recognition, image recognition, and a complex game in a breakthrough manner, and is deployed in many fields such as face recognition, a safe city, automatic driving, medical image detection, an Al intelligent Go, and a conference recording system. Performance of the deep neural network is good and even better than that of a human. This benefits from that the deep neural network can extract a higher-layer feature from raw data and can effectively learn from massive data.

To further improve performance of the deep neural network, a depth of the network, a quantity of network parameters, calculation algorithm strength, and a quantity of training datasets are all increased. Consequently, computing complexity and a training time are both greatly increased. Atypical ResNet-<NUM> network is used as an example. <NUM> hours are required to complete <NUM> epochs of training based on an ImageNet training dataset by using a high-performance server including eight common K80s. Even if a high-performance server including eight V100s that are quickest currently is used, about eight hours are required to complete the <NUM> epochs of training. This training time is still very long, and deep neural network model and algorithm research personnel need to wait for a long time to obtain a feedback. This severely affects development efficiency of a model and an algorithm. Especially for a new field, a new model, and a new algorithm, a plurality of groups of hyperparameters usually need to be tried, and adjustment and optimization are repeatedly performed to obtain an ideal result. This process is longer, and has become a key bottleneck in a process of development -> verification -> deployment.

Therefore, promotion, deployment, and application of the deep neural network in a large-scale manner in many fields impose a higher and faster requirement for training efficiency. Training efficiency of a single server node is far from enough to meet a requirement of a production environment. To resolve this problem, large-scale distributed training is usually used in the prior art. For this model, a training process is distributed to a plurality of computing nodes for execution, and a final training result is obtained through aggregation, to alleviate computing pressure on the single server node and improve computing efficiency. However, because bandwidth between computing nodes in the large-scale distributed training is limited, when there is a large amount of training data, an aggregation process may be slow, and computing efficiency is low. <CIT> discloses methods and systems for parallel computation of an algorithm using a plurality of nodes configured as a Howard Cascade. A home node of a Howard Cascade receives a request from ahost system to compute an algorithm identified in the request. The request is distributed to processing nodes of the Howard Cascade in a time sequence order in a manner to minimize the time to so expand the Howard Cascade. The participating nodes then perform the designated portion of the algorithm in parallel. Partial results from each node are agglomerated upstream to higher nodes of the structure and then returned to the host system. The nodes each include a library of stored algorithms accompanied by data template information defining partitioning of the data used in the algorithm among the number of participating nodes. <CIT> discloses parallel computer system for use in high performance computing applications e.g. life science, makes server to use rolling checksum algorithm to collect parallel checkpoint.

Embodiments of the present invention provide a tree topology based computing system and method, so as to resolve a problem of low computing efficiency of a computing system in large-scale distributed training.

According to a first aspect, an embodiment of the present invention provides a tree topology based computing system, where the system may include:
a plurality of node clusters, where the plurality of node clusters constitute a multi-layer network structure in a tree topology manner, any minimum tree in the network structure includes a second node cluster serving as a parent node and at least one first node cluster serving as a child node, and the second node cluster is connected to the at least one first node cluster through a physical link, where each of the at least one first node cluster is configured to: obtain a first computing result based on a first computing input, and send the first computing result to the second node cluster through the physical link; and the second node cluster is configured to: receive, through the physical link, at least one first computing result sent by the at least one first node cluster, and aggregate the at least one first computing result and a second computing result to obtain a third computing result, where the second computing result is a result obtained by the second node cluster based on a second computing input.

According to the computing system provided in this embodiment of the present invention, each node cluster is responsible for aggregating computing results of the node cluster and is also responsible for aggregating computing results of a lower-layer node cluster connected to the node cluster, so that not only transmission of data from a lower layer to an upper layer is completed, but also data aggregation between node clusters is completed layer by layer in a transmission process, thereby reducing an amount of data that is to be aggregated and that is transmitted in bandwidth. In addition, because a tree networking topology is used in this embodiment of the present invention, computing and aggregation are performed between different node clusters at a same layer in parallel, thereby further improving computing and aggregation efficiency. In this way, a problem of low computing efficiency in large-scale distributed training is resolved.

The second node cluster includes at least one second computing node, and the second computing node is a neural network accelerator; and the first node cluster includes at least one first computing node, and the first computing node is a neural network accelerator. In this embodiment of the present invention, one or more neural network accelerators are disposed in a node cluster, so as to implement parallel computing in a neural network.

The second node cluster is further configured to send the third computing result to a third node cluster for aggregation, where the third node cluster is a parent node of the second node cluster. In this embodiment of the present invention, the second node cluster aggregates computing results of the first node cluster at a lower layer, and then sends an aggregated third result to a parent node of the second node cluster serving as a child node in a minimum tree, so as to perform upper-layer aggregation.

In a possible implementation, any minimum tree in the network structure includes one second node cluster and k first node clusters, where k is an integer greater than or equal to <NUM>. In this embodiment of the present invention, it is set that each minimum tree is converged according to a proportion of k:<NUM>, to facilitate management and expansion.

In a possible implementation, the second node cluster includes k second computing nodes, and any one of the k first node clusters includes k first computing nodes; and in any minimum tree in the network structure, the k second computing nodes in the second node cluster one-to-one correspond to the k first node clusters, and any one of the k second computing nodes is connected to the k first computing nodes in the corresponding first node cluster through the physical link. In this embodiment of the present invention, each node cluster includes k computing nodes, to facilitate distributed computing and distributed aggregation. In addition, the k second computing nodes in the second node cluster serving as a parent node one-to-one correspond to the k first node clusters, to be specific, one second computing node is responsible for performing upstream aggregation on one first node cluster, to balance an aggregation process. This helps further improve computing efficiency of a computing system.

In a possible implementation, any one of the k first node clusters is specifically configured to:
distribute the first computing input to the k first computing nodes for distributed computing, to obtain k first distributed computing results; perform distributed aggregation on the k first computing nodes based on the k first distributed computing results respectively, to obtain one slice of the first computing result on each first computing node; and synchronously or asynchronously send, by using the k first computing nodes, k slices of the first computing result to a corresponding second computing node for aggregation. In this embodiment of the present invention, computing tasks of the first node cluster serving as a child node are distributed to the k first computing nodes for parallel processing, and after a computing result of each first computing node is obtained, parallel aggregation is performed between the k first computing nodes, thereby greatly improving computing and aggregation efficiency.

In a possible implementation, the second node cluster is specifically configured to: distribute the second computing input to the k second computing nodes for distributed computing, to obtain k second distributed computing results, where the k second distributed computing results are the second computing result; receive, respectively by using the k second computing nodes, the k slices of the first computing result that are sent by the k first computing nodes in the corresponding first node cluster; aggregate, respectively by using the k second computing nodes, the second distributed computing result obtained through computation by each second computing node and the k slices of the first computing result of the corresponding first node cluster; and perform distributed aggregation on results obtained through aggregation by using all of the k second computing nodes, to obtain one slice of the third computing result on each second computing node. In this embodiment of the present invention, computing tasks of the second node cluster serving as a parent node are distributed to the k second computing nodes for parallel processing, and after a computing result of each second computing node is obtained, the second computing node aggregates computing results sent by the corresponding k first node clusters. In addition, the process between the k second computing nodes is a parallel operation. Finally, distributed aggregation between nodes is performed once again between the k second computing nodes, to obtain a final aggregation result of the second node cluster, thereby greatly improving computing and aggregation efficiency.

In a possible implementation, any one of the k first node clusters is specifically configured to:
distribute the first computing input to the k first computing nodes for distributed computing, to obtain k first distributed computing results; perform aggregation on a specified first computing node in the k first computing nodes based on the k first distributed computing results, to obtain the first computing result; and send, by using the specified first computing node, the first computing result to a corresponding second computing node for aggregation. In this embodiment of the present invention, computing tasks of the first node cluster serving as a child node are distributed to the k first computing nodes for parallel processing, after a computing result of each first computing node is obtained, aggregation is performed on the specified first computing node in the k first computing nodes, and then an aggregation result is sent to the second computing node for upper-layer aggregation, thereby greatly improving computing and aggregation efficiency.

In a possible implementation, the second node cluster is specifically configured to: distribute the second computing input to the k second computing nodes for distributed computing, to obtain k second distributed computing results; receive, by using each of the k second computing nodes, the first computing result sent by the specified first computing node in the corresponding first node cluster, and aggregate the first computing result and the obtained second distributed computing results; and aggregate, by using a specified second computing node in the k second computing nodes, results obtained through aggregation by using all of the k second computing nodes, to obtain the third computing result. In this embodiment of the present invention, computing tasks of the second node cluster serving as a parent node are distributed to the k second computing nodes for parallel processing, and after a computing result of each second computing node is obtained, the second computing node aggregates computing results sent by the corresponding k first node clusters. In addition, the process between the k second computing nodes is a parallel operation. Finally, distributed aggregation between nodes is performed once again by using the specified second computing node in the k second computing nodes, to obtain a final aggregation result of the second node cluster, thereby greatly improving computing and aggregation efficiency.

In a possible implementation, the first computing input includes a first parameter; and the second node cluster is further configured to:
send the first parameter to the k first node clusters respectively by using the k second computing nodes. In this embodiment of the present invention, the second node cluster serving as a parent node delivers, in parallel, related computing input parameters of the k first computing nodes to the corresponding first node cluster by using the first computing nodes, so as to increase a speed of obtaining the related parameters by the first node cluster, thereby improving parameter synchronization efficiency of an entire system.

In a possible implementation, the second node cluster is specifically configured to: send, by using each second computing node, the first parameter divided into k slices respectively to the k first computing nodes in the corresponding first node cluster, so that the first parameter is broadcast between the k first computing nodes; or send the first parameter to the k first computing nodes in the corresponding first node cluster in parallel respectively by using the k second computing nodes; or send the first parameter to one first computing node in the corresponding first node cluster by using the k second computing nodes, so that the one first computing node broadcasts the first parameter between other first computing nodes in the same cluster. In this embodiment of the present invention, in a process of delivering a related parameter of the computing system, the first parameter is divided into k slices and the k slices are sent to the k first computing nodes in parallel; or the first parameter is simultaneously sent to the k first computing nodes; or the first parameter is directly sent to a first computing node, and then the first computing node broadcasts the first parameter between other first computing nodes in the same cluster, so as to implement a process of delivering the first parameter.

In a possible implementation, the second node cluster is directly connected to the at least one first node cluster through the physical link. In this embodiment of the present invention, in each minimum tree in the computing system, a second node cluster may be directly connected to a first node cluster through a physical link.

In a possible implementation, the computing system further includes a switch, and the switch and each of the plurality of node clusters are directly connected through the physical link; and the second node cluster is connected to the at least one first node cluster through the switch. In this embodiment of the present invention, in each minimum tree in the computing system, a second node cluster may be indirectly and physically connected to a first node cluster through a switch.

In a possible implementation, the computing system is a neural network computing system; and the first computing input and the second computing input include a weight, training data, an offset, and a hyperparameter, and the first computing result, the second computing result, and the third computing result are gradients. In this embodiment of the present invention, the computing system is applied to a neural network training model, a corresponding computing input is a related parameter in the neural network training model, and a corresponding computing result is a gradient value.

According to a second aspect, an embodiment of the present invention provides a computing method, where the method may include:.

In a possible implementation, the second node cluster includes k second computing nodes, and any one of the k first node clusters includes k first computing nodes; and in any minimum tree in the network structure, the k second computing nodes in the second node cluster one-to-one correspond to the k first node clusters, and any one of the k second computing nodes is connected to the k first computing nodes in the corresponding first node cluster through a physical link.

In a possible implementation, the aggregating, by the second node cluster, the first computing result and the second computing result, to obtain a third computing result includes: distributing, by the second node cluster, the second computing input to the k second computing nodes for distributed computing, to obtain k second distributed computing results, where the k second distributed computing results are the second computing result; receiving, by the second node cluster respectively by using the k second computing nodes, k slices of the first computing result that are sent by the k first computing nodes in the corresponding first node cluster; aggregating, by the second node cluster respectively by using the k second computing nodes, the second distributed computing result obtained through computation by each second computing node and the k slices of the first computing result of the corresponding first node cluster; and performing, by the second node cluster, distributed aggregation on results obtained through aggregation by using all of the k second computing nodes, to obtain one slice of the third computing result on each second computing node.

In a possible implementation, the aggregating, by the second node cluster, the first computing result and the second computing result, to obtain a third computing result includes: distributing, by the second node cluster, the second computing input to the k second computing nodes for distributed computing, to obtain k second distributed computing results; receiving, by the second node cluster by using each of the k second computing nodes, the first computing result sent by the specified first computing node in the corresponding first node cluster, and aggregating the first computing result and the obtained second distributed computing results; and aggregating, by the second node cluster by using a specified second computing node in the k second computing nodes, results obtained through aggregation by using all of the k second computing nodes, to obtain the third computing result.

In a possible implementation, the method further includes: sending, by the second node cluster, the first parameter to the k first node clusters respectively by using the k second computing nodes.

In a possible implementation, the sending, by the second node cluster, the first parameter to the k first node clusters respectively by using the k second computing nodes includes: sending, by the second node cluster by using each second computing node, the first parameter divided into k slices respectively to the k first computing nodes in the corresponding first node cluster, so that the first parameter is broadcast between the k first computing nodes; or sending, by the second node cluster, the first parameter to the k first computing nodes in the corresponding first node cluster in parallel respectively by using the k second computing nodes; or sending, by the second node cluster, the first parameter to one first computing node in the corresponding first node cluster by using the k second computing nodes, so that the one first computing node broadcasts the first parameter between other first computing nodes in the same cluster.

In a possible implementation, the first computing input and the second computing input include a weight, training data, an offset, and a hyperparameter, and the first computing result, the second computing result, and the third computing result are gradients.

According to a third aspect, this application provides a computer storage medium, configured to store a computer software instruction used by the computing system provided in the first aspect. The computer software instruction includes a program designed for performing the foregoing aspects.

According to a fourth aspect, an embodiment of the present invention provides a computer program. The computer program includes an instruction. When the computer program is executed by a computer, the computer is enabled to execute a procedure in the computing system in the first aspect.

According to a fifth aspect, this application provides a node cluster. The node cluster is configured to support a function implemented by the first node cluster or the second node cluster in the computing system in the first aspect.

To describe the technical solutions in the embodiments of the present invention or in the background more clearly, the following briefly describes the accompanying drawings required for describing the embodiments of the present invention or the background.

In this specification, claims, and accompanying drawings of this application, the terms such as "first", "second", "third", and "fourth" are intended to distinguish between different objects but do not indicate a particular order. In addition, the terms "include", "have", or any other variant thereof, are intended to cover a non-exclusive inclusion. For example, a process, a method, a system, a product, or a device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes an unlisted step or unit, or optionally further includes another inherent step or unit of the process, the method, the product, or the device.

Mentioning an "embodiment" in this specification means that a particular characteristic, structure, or feature described with reference to the embodiment may be included in at least one embodiment of this application. The phrase shown in various locations in this specification may not necessarily refer to a same embodiment, and is not an independent or optional embodiment exclusive from another embodiment. It is explicitly and implicitly understood by persons skilled in the art that the embodiments described in this specification may be combined with another embodiment.

Terminologies such as "component", "module", and "system" used in this specification are used to indicate computer-related entities, hardware, firmware, a combination of hardware and software, software, or software being executed. For example, a component may be, but is not limited to, a process that runs on a processor, a processor, an object, an executable file, a thread of execution, a program, and/or a computer. As shown in figures, both a computing device and an application that runs on a computing device may be components. One or more components may reside within a process and/or a thread of execution, and a component may be located on one computer and/or distributed between two or more computers. In addition, these components may be executed from various computer-readable media that store various data structures. For example, the components may communicate by using a local and/or remote process and according to, for example, a signal having one or more data packets (for example, data from two components interacting with another component in a local system, a distributed system, and/or across a network such as the internet interacting with other systems by using the signal).

Some terms in this application are first described, so as to help persons skilled in the art have a better understanding.

Next, a technical problem that needs to be resolved in this application and an application scenario are proposed. In the prior art, in a large-scale distributed training system, a training manner widely applied in an academic circle and an industrial circle refers to synchronous stochastic gradient descent and data parallelism, and key points of a training algorithm are as follows:.

It can be learned from the foregoing process that deep neural network training is a high-strength computation process that is intensive in computing and network bandwidth and is very sensitive to a latency. The following Table <NUM> shows a neural network model of a typical deep neural network, and a parameter quantity and a parameter size list that correspond to the model. K = <NUM>, and M = <NUM> x <NUM> = <NUM>.

It can be learned from the foregoing list that there is a relatively large difference in parameter quantities and parameter sizes of different network models. VGG19-<NUM> is used as an example. A parameter size of the VGG19-<NUM> is up to <NUM> Mbytes. With reference to the key point (<NUM>) and the key point (<NUM>) in the foregoing training process, it can be learned that in the training process, gradients and parameters are exchanged very frequently between computing nodes, and traffic is also very high. This becomes worse with an increase of a computing node scale in the group. Therefore, how to effectively and quickly aggregate and synchronize these parameters between all computing nodes in an entire group system is a key problem that needs to be resolved in large-scale distributed training. There are related solutions in the prior art, for example:.

Currently, a fully connected structure (computing node-parameter server or worker-parameter server) is widely applied to a distributed training system. A topology of the distributed training system is shown in <FIG> is a schematic diagram of a fully connected architecture. A working principle of the fully connected architecture is as follows:.

An entire group includes many workers, and the worker is responsible for gradient computation of the local node. A parameter server (PS) is responsible for collecting gradient data computed by all workers in the entire group, aggregating the gradient data, calculating a new weight based on the aggregated gradient data, and then delivering the new weight to all the workers. To share pressure of network bandwidth and weight calculation, a plurality of PSs are generally used to constitute a group to bear workload. Assuming that a quantity of PSs in the group is P and a quantity of workers in the group is W, a working mechanism is as follows:.

Disadvantages of the prior art <NUM> are as follows:.

To resolve the foregoing problem of the "many-to-one traffic pattern", another solution is to restrict a sending/receiving relationship between worker nodes in a group, to form a logical ring, that is, a ring structure. <FIG> is a schematic diagram of a ring networking architecture. A working mechanism of the ring structure is as follows:.

To resolve the problem of the Ring length in the ring topology, an optimization solution is to adopt the following fat-tree networking. <FIG> is a schematic diagram of a fat-tree networking architecture. A working mechanism of a Fat-Tree structure is as follows:.

In a group, a node worker and a parameter server PS constitute a tree structure, and the tree structure is converged in a k: <NUM> manner (as shown in <FIG>, k = <NUM>). To implement non-blocking switching, a node closer to a root node requires higher network bandwidth.

In conclusion, the existing solutions have disadvantages regarding a scalability capability of a group. When a group scale increases and a quantity of nodes in the group increases, performance of the group degrades, linearity of the group deteriorates, deployment costs of the group increase, and network performance optimization and group operation overheads increase. This is unfavorable to construction of a large-scale AI group. Therefore, a technical problem to be resolved in this application is how to effectively and quickly aggregate and synchronize related parameters between all computing nodes in an entire group system in a large-scale distributed training system, to implement scalability of the group system, facilitate construction of a large-scale AI group, and improve training efficiency.

Based on the foregoing description, the following first describes an architecture of a computing system provided in an embodiment of the present invention. <FIG> is an architectural diagram of a tree topology based computing system according to an embodiment of the present invention. The computing system <NUM> may include a plurality of node clusters (each block in <FIG> represents one node cluster), and the plurality of node clusters constitute a multi-layer network structure in a tree topology manner (a layer N = <NUM> is used as an example in <FIG>), including an L0 layer, an L1 layer, an L2 layer, an L3 layer, and an L4 layer. Any minimum tree in the network structure includes a second node cluster serving as a parent node and at least one first node cluster serving as a child node, and the second node cluster is connected to the at least one first node cluster through a physical link. For example, <FIG> shows some minimum trees (a minimum tree <NUM>, a minimum tree <NUM>, and a minimum tree <NUM>). In the minimum tree <NUM>, the second node cluster is a parent node L4 layer-cluster <NUM>, and there are four first node clusters, including an L3 layer-cluster <NUM>, an L3 layer-cluster <NUM>, an L3 layer-cluster <NUM>, and an L3 layer-cluster <NUM>. By analogy, the minimum tree <NUM> and the minimum tree <NUM> each include one parent node and four child nodes. That is, it may be understood that a minimum tree in this application refers to a tree including one parent node at an upper layer and all child nodes of the parent node that are at a lower layer in two adjacent layers in a network architecture, and each child node is connected to the parent node through a physical link in the minimum tree.

The first node cluster is configured to: obtain a first computing result based on a first computing input, and send the first computing result to the second node cluster through the physical link. The first computing input is a related parameter, training data, or the like of a computing task that is assigned by the computing system to each first node cluster in an initial or iterative case, and the first computing result is a result obtained through computation by the first node cluster based on the first computing input. After completing the computation, the first node cluster needs to send the first computing result to the parent node of the first node cluster through the physical link between the first node cluster and the parent node, namely, the second node cluster for aggregation. It may be understood that the first node cluster in this application refers to all child nodes in each minimum tree in the computing system. In other words, in the network structure in <FIG>, except the L4 layer-cluster <NUM>, each of other node clusters may serve as a role of a child node in a minimum tree, and therefore, also needs to perform the foregoing actions in the minimum tree to which the node cluster belongs.

The second node cluster is configured to: receive, through the physical link, at least one first computing result sent by the at least one first node cluster, and aggregate the at least one first computing result and a second computing result to obtain a third computing result, where the second computing result is a result obtained by the second node cluster based on a second computing input. To be specific, the second node cluster serving as the parent node not only needs to perform a computing task assigned by the computing system, to obtain the second computing result, but also needs to aggregate the second computing result and one or more first computing results obtained through computation by all child nodes in the minimum tree to which the second node cluster belongs. Further, when the second node cluster is not a root node, the second node cluster further needs to send the third computing result to a parent node in a corresponding minimum tree in which the second node cluster serves as a child node, that is, send the third computing result to a third node cluster for upper-layer aggregation. It may be understood that the second node cluster in this application refers to a parent node in each minimum tree in the computing system. In other words, in the network structure in <FIG>, each of node clusters other than <NUM> clusters at the L0 layer may serve as a role of a parent node in a minimum tree.

It should be noted that a root node (for example, the L4 layer-cluster <NUM> in <FIG>) serves as only a parent node, node clusters at a lowest layer (for example, the <NUM> clusters at the L0 layer in <FIG>) serve as only child nodes, and each of other node clusters may serve as a first node cluster in a minimum tree, and serve as a second node cluster in another minimum tree.

In a possible implementation, any minimum tree in the network structure includes one second node cluster and k first node clusters, where k is an integer greater than or equal to <NUM>. In other words, any minimum tree in the network structure is converged in a k: <NUM> manner. In <FIG>, k = <NUM>. Therefore, in <FIG>, the L0 layer has <NUM> node clusters, the L1 layer has <NUM> clusters, the L2 layer has <NUM> clusters, the L3 layer has four clusters, and the L4 layer has one cluster. However, it may be understood that convergence proportions of all minimum trees may be the same or may be different. This is not specifically limited in this application.

Optionally, the first computing input and the second computing input include a weight, training data, an offset, and a hyperparameter, and the first computing result, the second computing result, and the third computing result are gradients. When the foregoing computing system is applied to an AI neural network, each node cluster in the computing system <NUM> is configured to: obtain a gradient of the node cluster through computation based on a weight, training data, an offset, and a hyperparameter that are allocated, and perform gradient aggregation between the node cluster and a parent node in a minimum tree to which the node cluster belongs. Finally, a final aggregated gradient is obtained on the root node. The root node calculates a new weight based on the final aggregated gradient and a hyperparameter such as a learning rate, and then distributes the new weight to each node cluster in the computing system, to start a next round of iterative computation.

Optionally, the second node cluster includes at least one second computing node, and the second computing node is a neural network accelerator; and the first node cluster includes at least one first computing node, and the first computing node is a neural network accelerator. In this embodiment of the present invention, one or more neural network accelerators are disposed in a node cluster, so as to implement parallel computing in a neural network.

In the computing system <NUM>, each node cluster is responsible for aggregating computing results of the node cluster and is also responsible for aggregating computing results of a lower-layer node cluster connected to the node cluster, so that not only transmission of data from a lower layer to an upper layer is completed, but also data aggregation between node clusters is completed layer by layer in a transmission process, thereby reducing an amount of data that is to be aggregated and that is transmitted in bandwidth. In addition, because a tree networking topology is used in this embodiment of the present invention, computing and aggregation are performed between different node clusters at a same layer in parallel, thereby further improving computing and aggregation efficiency. In this way, a problem of low computing efficiency in large-scale distributed training is resolved.

<FIG> is a schematic structural diagram of a connection relationship between node clusters in a minimum tree according to an embodiment of the present invention. As shown in <FIG>, a second node cluster (for example, an L1-cluster <NUM>) includes k (k = <NUM> is used as an example in <FIG>) second computing nodes (for example, an NNA <NUM>, an NNA <NUM>, an NNA <NUM>, and an NNA <NUM> in the L <NUM>-cluster <NUM>), and any one first node cluster (using an L0-cluster <NUM> as an example) of k first node clusters (for example, the L0-cluster <NUM>, an L0-cluster <NUM>, an LO-cluster <NUM>, and an L0-cluster <NUM>) includes k first computing nodes (for example, an NNA <NUM>, an NNA <NUM>, an NNA <NUM>, and an NNA <NUM> in the L0-cluster <NUM>). In any minimum tree in a network structure, the k second computing nodes in the second node cluster one-to-one correspond to the k first node clusters, and any one of the k second computing nodes is connected to the k first computing nodes in the corresponding first node cluster through a physical link. In <FIG>, the NNA <NUM> in the L1-cluster <NUM> corresponds to the L0-cluster <NUM>, and the NNA <NUM> in the L1-cluster <NUM> is connected to the L0-cluster <NUM> through a physical link. The NNA2 in the L1-cluster <NUM> corresponds to the L0-cluster <NUM>, and the NNA <NUM> in the L1-cluster <NUM> is connected to the L0-cluster <NUM> through a physical link. The NNA <NUM> in the L1-cluster <NUM> corresponds to the L0-cluster <NUM>, and the NNA <NUM> in the L1-cluster <NUM> is connected to the L0-cluster <NUM> through a physical link. The NNA4 in the L1-cluster <NUM> corresponds to the L0-cluster <NUM>, and the NNA4 in the L1-cluster <NUM> is connected to the L0-cluster <NUM> through a physical link.

It may be understood that the connection relationship between the node clusters in <FIG> is merely an example implementation in this embodiment of the present invention. A structure of a node cluster and a connection relationship between node clusters in this embodiment of the present invention include but are not limited to the foregoing structure and connection relationship.

The following uses a parent node and one child node in the foregoing minimum tree as an example, for example, a connection between an NPU <NUM> in the L1-cluster <NUM> and the L0-cluster <NUM>, to describe a structure and a connection relationship of the first node cluster and the second node cluster. <FIG> and <FIG> are a schematic diagram of a parent-child node structure and a connection relationship in a minimum tree according to an embodiment of the present invention. In <FIG> and <FIG>, any node cluster (including the foregoing first node cluster or second node cluster) may include the following functional modules:.

A main control CPU is responsible for management and control of a computing task on a node, control of interaction between nodes, and preprocessing and post-processing of data (if preprocessing or post-processing needs to be performed). For example, the main control CPU may be X86.

An SSD and an NVMe are local high-speed storage, and are configured to store a system and training data such as a first computing input and a second computing input.

A NIC <NUM>, a NIC <NUM>, a NIC <NUM>, and a NIC <NUM> are network interfaces, and each are configured to be directly connected, through a physical link, to a child node in a node cluster to which the network interface belongs. For example, in <FIG> and <FIG>, a NIC <NUM> in an L1-cluster <NUM> is directly connected to a network adapter NIC <NUM> in a child node L0-cluster <NUM> of the L1-cluster <NUM> through a physical link. Optionally, in <FIG> and <FIG>, any second computing node is directly connected to k first computing nodes in a corresponding first node cluster through a physical link, and a first computing result sent by each first node cluster is received through the physical link.

A NIC <NUM> and a NIC <NUM> are network interfaces on a neural network accelerator, and each are configured to perform interaction and communication between the computing node and the outside. To be specific, the NIC <NUM> is configured to: when a node cluster serves as a child node, to be directly and physically connected to one of a corresponding NIC <NUM>, NIC <NUM>, NIC <NUM>, and NIC <NUM> on a parent node, and send the first computing result to the second node cluster through the physical link. The NIC <NUM> is mainly configured to serve as an interface of another network plane (for example, a user plane, a control plane, or a management plane).

APCIe switch is a PCIe bus switch, and is configured to interconnect PCIe devices and interconnect X86 main control CPUs.

An NN accelerator is a neural network accelerator, may also be referred to as an accelerated NNA, and is usually mounted to a PCIe bus by using a PCIe endpoint (EP) device.

An NN accelerator/DDR is a memory on the neural network accelerator, and is used for local storage in a computing process.

An NN accelerator/PCIe is a PCIe bus interface on the neural network accelerator, and is used for interconnection and communication inside the computing node.

An NN accelerator/link is a high-speed interconnection link between neural network accelerators, and is configured to accelerate high-speed data exchange between NNAs.

An NN accelerator/NPU is an embedded neural network processor on the neural network accelerator, and is used for computation of various neural network operators.

When the computing system in this application is applied to an AI neural network field, in an AI training process, a processing process of each functional module in the foregoing node cluster is as follows:.

In a specific computing process, based on the structure of the node cluster and the connection relationship between the node clusters in <FIG> and <FIG>, an embodiment of the present invention provides a distributed computing solution, to be specific, computing tasks on each node cluster are distributed to a plurality of computing nodes (for example, the NN accelerators in <FIG> and <FIG>) in the node cluster for distributed computing. Further, distributed aggregation may be performed after each computing node completes a computing task. Specifically, the following two implementations may be included.

In a possible implementation, each of k first node clusters in any minimum tree is specifically configured to: distribute a first computing input to the k first computing nodes for distributed computing, to obtain k first distributed computing results; perform distributed aggregation on the k first computing nodes based on the k first distributed computing results respectively, to obtain one slice of the first computing result on each first computing node; and finally, synchronously or asynchronously send, by using the k first computing nodes, k slices of the first computing result to a corresponding second computing node for aggregation. Correspondingly, a second node cluster is specifically configured to: distribute a second computing input to k second computing nodes for distributed computing, to obtain k second distributed computing results, where the k second distributed computing results are the second computing result; receive, respectively by using the k second computing nodes, the k slices of the first computing result that are sent by the k first computing nodes in the corresponding first node cluster; aggregate, respectively by using the k second computing nodes, the second distributed computing result obtained through computation by each second computing node and the k slices of the first computing result of the corresponding first node cluster; and finally, perform distributed aggregation on results obtained through aggregation by using all of the k second computing nodes, to obtain one slice of the third computing result on each second computing node.

In this embodiment of the present invention, computing tasks of the first node cluster serving as a child node are distributed to the k first computing nodes for parallel processing, and after a computing result of each first computing node is obtained, parallel aggregation is performed between the k first computing nodes. Moreover, computing tasks of the second node cluster serving as a parent node are distributed to the k second computing nodes for parallel processing, and after a computing result of each second computing node is obtained, the second computing node locally aggregates computing results sent by the corresponding k first node clusters. In addition, the process between the k second computing nodes is a parallel operation. Finally, distributed aggregation between nodes is performed once again between the k second computing nodes, to obtain a final aggregation result of the second node cluster, thereby greatly improving computing and aggregation efficiency.

In the foregoing implementation, based on the interconnection relationship in <FIG>, the L1-cluster <NUM> and the L0-cluster <NUM> constitute a hierarchical structure. To be specific, five computing nodes: {an L1-cluster <NUM>. NPU <NUM>, an L0-cluster <NUM>. NPU <NUM>, an L0-cluster <NUM>. NPU <NUM>, an L0-cluster <NUM>. NPU <NUM>, and an L0-cluster <NUM>. NPU <NUM>} constitute one computing and aggregation unit in a minimum tree. The L1-cluster <NUM>. NPU <NUM> is one second computing node in a second node cluster in this embodiment of the present invention, and the other four NPUs serve as four first computing nodes in a first node cluster corresponding to the second computing node. Each NPU in the L0-cluster <NUM> completes gradient computation, and after distributed gradient aggregation is completed between the four NPUs in the L0-cluster <NUM>, each NPU sends aggregated gradient data to the aggregation node L1-cluster <NUM>. NPU <NUM> of the NPU. Upstream transmission paths are shown by dashed lines in <FIG> and <FIG>. <FIG> and <FIG> are a schematic diagram of an upstream data transmission path between node clusters according to an embodiment of the present invention. There are a total of four transmission paths:.

In another possible implementation, any one of k first node clusters is specifically configured to: distribute a first computing input to the k first computing nodes for distributed computing, to obtain k first distributed computing results; perform aggregation on a specified first computing node in the k first computing nodes based on the k first distributed computing results, to obtain the first computing result; and send, by using the specified first computing node, the first computing result to a corresponding second computing node for aggregation. Correspondingly, the second node cluster is specifically configured to: distribute a second computing input to k second computing nodes for distributed computing, to obtain k second distributed computing results; receive, by using each of the k second computing nodes, the first computing result sent by the specified first computing node in the corresponding first node cluster, and aggregate the first computing result and the obtained second distributed computing results; and finally aggregate, by using a specified second computing node in the k second computing nodes, results obtained through aggregation by using all of the k second computing nodes, to obtain the third computing result.

A difference from the distributed computing and distributed aggregation in the foregoing implementation lies in that, in this implementation, computing tasks of the first node cluster serving as a child node are distributed to the k first computing nodes for parallel processing, and after a computing result of each first computing node is obtained, parallel aggregation is performed on the specified first computing node in the k first computing nodes. Moreover, computing tasks of the second node cluster serving as a parent node are distributed to the k second computing nodes for parallel processing, and after a computing result of each second computing node is obtained, the second computing node aggregates computing results sent by the corresponding k first node clusters. In addition, the process between the k second computing nodes is a parallel operation. Finally, distributed aggregation between nodes is performed once again between the k second computing nodes, to obtain a final aggregation result of the second node cluster.

It should be noted that in the first computing result, the second computing result, and the third computing result, the first computing result is a result obtained after the first node cluster completes computing and aggregation, and the third computing result is obtained by the second node cluster by aggregating the first computing result and the second computing result. Therefore, the third computing result is also an aggregated result. The second computing result is one or more computing results that are obtained through computation by the second node cluster or all the second computing nodes in the second node cluster but have not been aggregated.

<FIG> is a schematic diagram of upstream aggregation in a computing system according to an embodiment of the present invention. In <FIG>, aggregation may be performed in each minimum tree according to the foregoing procedure, gradient aggregation between all minimum trees at a same layer is performed in parallel, and finally, aggregation of the entire computing system, that is, an entire tree, is completed. For a specific aggregation manner in each minimum tree in the computing system, refer to the foregoing aggregation procedure of the minimum tree in one of <FIG>, <FIG> and <FIG>, and <FIG> and <FIG>.

Based on the foregoing data, in the computing system, in a process of aggregation from a node cluster at a lower layer to a node cluster at an upper layer, an embodiment of the present invention further provides a solution for delivering an initial or updated related parameter (for example, a first computing input or a second computing input) from the node cluster at the upper layer to the node cluster at the lower layer. <FIG> and <FIG> are a schematic diagram of a downstream data transmission path between node clusters according to an embodiment of the present invention.

Based on the interconnection relationship in <FIG>, the L1-cluster <NUM> and the L0-cluster <NUM> constitute a hierarchical structure. To be specific, five computing nodes: {an L1-cluster <NUM>. NPU <NUM>, an L0-cluster <NUM>. NPU <NUM>, an L0-cluster <NUM>. NPU <NUM>, an L0-cluster <NUM>. NPU <NUM>, and an L0-cluster <NUM>. NPU <NUM>} constitute one computing and aggregation unit in a minimum tree. The L1-cluster <NUM>. NPU <NUM> is one second computing node in a second node cluster in this embodiment of the present invention, and the other four NPUs serve as four first computing nodes in a first node cluster corresponding to the second computing node. When receiving a new weight parameter (for example, a first parameter in this application), the L1-cluster <NUM>. NPU <NUM> needs to synchronize the new weight parameter with all first computing nodes connected to the L1-cluster <NUM>. To improve efficiency, the L1-cluster <NUM>. NPU <NUM> scatters the weight to all the first computing nodes. The weight may be sent to all the first computing nodes in the same node cluster in a broadcast manner through an internal high-speed physical link. As shown by dashed lines in <FIG> and <FIG>, there are a total of four downstream transmission paths for weight data:.

For example, in a parameter delivering process, when a first computing input includes the first parameter, the second node cluster is further configured to respectively send the first parameter to the k first node clusters by using the k second computing nodes. A specific delivering process may include the following three implementations:.

Implementation <NUM>: The second node cluster sends the first parameter to the k first computing nodes in the corresponding first node cluster in parallel respectively by using the k second computing nodes. That is, the second node cluster simultaneously sends the first parameter to the k first computing nodes in the corresponding first node cluster. Therefore, the k first computing nodes simultaneously receive the first parameter sent by the corresponding second computing nodes. In this implementation, a parameter delivering speed is high, but the second computing node needs to send the complete first parameter for k times.

Implementation <NUM>: The second node cluster sends the first parameter to one first computing node in the corresponding first node cluster by using the k second computing nodes, so that the one first computing node broadcasts the first parameter between other first computing nodes in the same cluster. That is, the second computing node first sends the first parameter to a specified first computing node in the corresponding first node cluster. Therefore, the specified first computing node in the k first computing nodes first receives the first parameter, then, the first computing node that receives the first parameter broadcasts the first parameter through a high-speed physical link between computing nodes in the same cluster, and finally, each first computing node obtains the complete first parameter.

Implementation <NUM>: The second node cluster sends, by using each second computing node, the first parameter divided into k slices respectively to the k first computing nodes in the corresponding first node cluster, so that the first parameter is broadcast between the k first computing nodes. That is, the second computing node divides the first parameter into k slices, and then sends each slice to a different first computing node. Therefore, all the k first computing nodes can receive a slice of the first parameter, and then the k first computing nodes each synchronizes a slice with each other through a high-speed physical link. In this way, transmission bandwidth between the second computing node and the first node cluster can be reduced, a parameter delivering time can also be reduced, and parameter delivering efficiency can be improved.

<FIG> is a schematic diagram of delivering of a parameter in a computing system according to an embodiment of the present invention. In <FIG>, a parameter may be delivered in each minimum tree according to the foregoing procedure, parameters are synchronized between different minimum trees in parallel, and finally, delivering of an initial parameter or an updated parameter, is completed in the entire computing system, that is, an entire tree. For a specific parameter delivering procedure in each minimum tree in the computing system, refer to the procedures in the foregoing implementation <NUM>, implementation <NUM>, and implementation <NUM>.

In this application, to reduce a time consumed in end-to-end (E2E) transmission and improve efficiency, in a possible implementation, a pipeline algorithm is used to increase an overlap ratio for computing and transmission. The time consumed in transmission is hidden as much as possible in a computing process. A processing manner is as follows:.

<FIG> is a schematic diagram of an aggregation and synchronization pipeline algorithm according to an embodiment of the present invention. Gradient aggregation and weight synchronization form a multi-level pipeline, to perform transmission in parallel. A time consumed in transmission is hidden in a time consumed in computing. A size of each small slice is m, a time consumed for processing each small slice on a node is t1, and a time consumed for transmitting each small slice between nodes is t2 = m/B, where B is effective bandwidth for transmission between the nodes.

Based on the foregoing procedures for aggregation and delivering of the parameter in the computing system, in a possible implementation, this application further provides another system connection manner. <FIG> is a schematic architectural diagram of a minimum tree in another tree topology based computing system according to an embodiment of the present invention. A minimum tree in the computing system <NUM> further includes a top of rack switch. The top of rack switch is directly connected to each of a plurality of node clusters through a physical link. A second node cluster is connected to at least one first node cluster through the top of rack switch. To be specific, minimum trees in the computing system <NUM> may be connected by using the top of rack ToR switch, to form a slim-tree networking topology. As shown in <FIG>, a total of six clusters: an L1-cluster <NUM>, an L0-cluster <NUM>, an L0-cluster <NUM>, an L0-cluster <NUM>, an L0-cluster <NUM>, and an L0-cluster <NUM> are connected to a same ToR, to constitute L0 and L1 layers of a slim tree according to a convergence ratio of <NUM>:<NUM> (in an actual deployment process, another convergence ratio may also be selected based on an actual situation). The L1-cluster <NUM> is a parent node, and the L0-cluster <NUM>, the L0-cluster <NUM>, the L0-cluster <NUM>, the L0-cluster <NUM>, and the L0-cluster <NUM> are five child nodes. It may be understood that for an internal structure of each node cluster and a specific connection relationship between node clusters, refer to the structure and the connection manner in <FIG> and <FIG>.

It may be understood that in the foregoing minimum tree architecture in <FIG>, a slim tree may be formed through layered stacking. <FIG> is a schematic architectural diagram of a large-scale computing system according to an embodiment of the present invention. In the figure, five minimum trees provided in <FIG> are included. If a scale of a group is large, a plurality of layers may be expanded to form a multi-layer slim tree. Gradient aggregation and weight synchronization may be completed based on related descriptions of an aggregation algorithm and a parameter delivering algorithm in the embodiments corresponding to <FIG> in this application.

It should be noted that in this embodiment of the present invention, scalability of a large-scale distributed neural network training group is implemented based on a tree networking topology, a layer-by-layer accumulation algorithm, and a multi-layer pipeline algorithm. The networking topology and algorithm are also applicable to other similar computing fields. With reference to the idea and algorithm implementation, the networking topology and algorithm are used to implement high-performance computing in this field.

When the computing system in this application is applied to the field of the large-scale distributed neural network training group, existing network plane planning may remain unchanged, to meet requirements of management, access, storage, control, and the like of a group node. Based on an existing network plane, only a distributed deep neural network (DDN) plane needs to be newly planned, and gradient data aggregation and weight parameter synchronization are dedicated by using the plane.

Next, NICs between computing nodes in a node cluster are connected in a back-to-back physical direct connection manner, to form a high-bandwidth low-latency channel between the nodes. A plurality of node clusters are converged in a k: <NUM> manner, to form a tree topology. A tree group system may be formed through layered stacking. Then, the DDN plane is mapped to a physical direct connection topology between the node clusters.

In addition, in this application, a plurality of NNAs, that is, computing nodes (for example, four computing nodes) in this application, may be configured for each node cluster, to form one cluster. In a data parallel training method, each NNA(for example, a first computing unit) in a cluster (for example, a first node cluster) first independently completes gradient computing, and then completes gradient aggregation between the plurality of NNAs in the cluster. After the aggregation is completed, an aggregated gradient is transmitted to a parent node (for example, a second node cluster) of the cluster. Each NNA (for example, a second computing unit) in the parent node cluster not only needs to complete gradient computing responsible by the NNA, but also needs to obtain gradient data aggregated by all child nodes, and aggregate the gradient data and a gradient obtained through computation by the NNA to obtain one piece of data. Then, aggregation is performed again between all NNAs in the parent node cluster. After the aggregation is completed, an aggregated gradient is transmitted to a parent node (for example, a third node cluster) of the parent node cluster.

Therefore, each node cluster in this application is responsible for not only gradient computing, but also gradient aggregation, routing, and transfer. Because each node cluster sends an aggregated gradient, amounts of data transmitted between layers in a tree are uniform. In this manner, in the entire group, execution is performed in parallel, and one piece of complete gradient data including computing results of all nodes can be obtained after aggregation is completed on a root node.

A new weight parameter can be calculated based on the foregoing gradient data. Once a new weight is obtained through calculation, the new weight may be transmitted downward along the tree topology, and the new weight is synchronized with all worker nodes (that is, the first computing node and the second computing node in this application). To improve downstream transmission efficiency, a parent node delivers the weight to a child node of the parent node, and after receiving the weight, the child node broadcasts the weight in the cluster. In this case, sending from the parent node to the child node and sending from the child node to a child node are performed in parallel, and a high-speed link in the cluster is fully used. In addition, to increase a ratio of computing to a consumed time and reduce a stall time of a NNA, the foregoing aggregation and synchronization are performed in a pipeline manner. For a deep neural network, after back propagation is performed, gradient data is computed, and dependency is canceled, the gradient data can enter a pipeline. Processing of the gradient aggregation and synchronization and the back propagation are overlapped.

In conclusion, when the computing system in this application is applied to the large-scale distributed neural network training group, the following beneficial effects are achieved:.

Therefore, the computing system provided in this application can obtain a stable speed-up ratio in a large-scale group, and is suitable for constructing a large-scale distributed training group.

<FIG> is a schematic flowchart of a computing method according to an embodiment of the present invention. The computing method may be applied to the computing system shown in <FIG>. The following provides a description from a second node cluster side with reference to <FIG>. The method may include the following step S101 to step S103.

Step S101: A second node cluster receives a first computing result sent by at least one first node cluster, where the first computing result is a result obtained by each of the at least one first node cluster based on a first computing input, the first node cluster and the second node cluster are in any minimum tree of a same tree network structure, and the second node cluster is a parent node of the at least one first node cluster.

Step S102: The second node cluster aggregates the first computing result and a second computing result, to obtain a third computing result, where the second computing result is a result obtained by the second node cluster based on a second computing input.

Step S103: The second node cluster sends the third computing result to a third node cluster for aggregation, where the third node cluster is in the tree network topology, and the third node cluster is a parent node of the second node cluster.

In a possible implementation, that the second node cluster aggregates the first computing result and the second computing result, to obtain a third computing result is specifically: distributing, by the second node cluster, the second computing input to the k second computing nodes for distributed computing, to obtain k second distributed computing results, where the k second distributed computing results are the second computing result; receiving, by the second node cluster respectively by using the k second computing nodes, k slices of the first computing result that are sent by the k first computing nodes in the corresponding first node cluster; aggregating, by the second node cluster respectively by using the k second computing nodes, the second distributed computing result obtained through computation by each second computing node and the k slices of the first computing result of the corresponding first node cluster; and performing, by the second node cluster, distributed aggregation on results obtained through aggregation by using all of the k second computing nodes, to obtain one slice of the third computing result on each second computing node.

In a possible implementation, that the second node cluster aggregates the first computing result and the second computing result, to obtain a third computing result is specifically: distributing, by the second node cluster, the second computing input to the k second computing nodes for distributed computing, to obtain k second distributed computing results; receiving, by the second node cluster by using each of the k second computing nodes, the first computing result sent by the specified first computing node in the corresponding first node cluster, and aggregating the first computing result and the obtained second distributed computing results; and aggregating, by the second node cluster by using a specified second computing node in the k second computing nodes, results obtained through aggregation by using all of the k second computing nodes, to obtain the third computing result.

In a possible implementation, the computing method further includes a step: sending, by the second node cluster, the first parameter to the k first node clusters respectively by using the k second computing nodes.

In a possible implementation, the sending, by the second node cluster, the first parameter to the k first node clusters respectively by using the k second computing nodes specifically includes the following three implementations:.

It should be noted that, for a specific procedure of the computing method described in this embodiment of the present invention and a related function of the second node cluster serving as an execution body, refer to related descriptions in the embodiments of the computing system in <FIG>.

It should be noted that, for brief description, the foregoing method embodiments are represented as a series of actions. However, persons skilled in the art should appreciate that this application is not limited to the described order of the actions, because according to this application, some steps may be performed in other orders or simultaneously. It should be further appreciated by persons skilled in the art that the embodiments described in this specification all belong to example embodiments, and the involved actions and modules are not necessarily required by this application.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus may be implemented in other manners. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be implemented through some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic or other forms.

The foregoing units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on a plurality of network units.

When the foregoing integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable storage medium. Based on such an understanding, the technical solutions of this application essentially, or the part contributing to the prior art, or all or some of the technical solutions may be implemented in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for instructing a computer device (which may be a personal computer, a server, or a network device) to perform all or some of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes: any medium that can store program code, such as a USB flash drive, a removable hard disk, a magnetic disk, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

Claim 1:
A tree topology based computing system for large-scale distributed training, comprising: a plurality of node clusters, wherein the plurality of node clusters constitute a multi-layer network structure in a tree topology manner, any minimum tree in the network structure comprises a second node cluster serving as a parent node and at least one first node cluster serving as a child node, and the second node cluster is connected to the at least one first node cluster through a physical link, the second node cluster comprises at least one second computing node, each first node cluster comprises at least one first computing node, and the second computing node and the first computing node are neural network accelerators, wherein
each of the at least one first node cluster is configured to: obtain a first computing result based on a first computing input, and send the first computing result to the second node cluster through the physical link; and
the second node cluster is configured to: receive, through the physical link, at least one first computing result sent by the at least one first node cluster, and aggregate the at least one first computing result and a second computing result to obtain a third computing result, wherein the second computing result is a result obtained by the second node cluster based on a second computing input, and the at least one first computing result, the second computing result, and the third computing result are gradients,
wherein the second node cluster is further configured to send the third computing result to a third node cluster for aggregation, wherein the third node cluster is a parent node of the second node cluster.