Patent ID: 12242952

DETAILED DESCRIPTION

In general, according to one embodiment, a system includes a first node and a second node of a first group, and a third node and a fourth node of a second group. In a case where the first node and the second node perform nth (n is a natural number) parallel distributed processing, the first node is configured to calculate a first gradient to update a first weight of objective function to a second weight and the second node is configured to calculate a second gradient to update the first weight of the objective function to the second weight. In a case where the third node and the fourth node perform mth (m is a natural number) parallel distributed processing, the third node is configured to calculate a third gradient to update a third weight of the objective function to a fourth weight and the fourth node is configured to calculate a fourth gradient to update the third weight of the objective function to the fourth weight. If the calculation of gradient by the first node and the second node is faster than the calculation of gradient by the third node and the fourth node, in n+1th parallel distributed processing performed by the first node and the second node, the second weight updated from the first weight is further updated using the first and second gradients, and, in m+1th parallel distributed processing performed by the third node and the fourth node, the fourth weight updated from the third weight is further updated using the first to fourth gradients.

Various embodiments will be described hereinafter with reference to accompanying drawings.

First Embodiment

A system of the present embodiment performs parallel distributed learning processing using an objective function as a reference in deep learning using massive data, for example. Note that the parallel distributed learning processing using an objective function as a reference may be any type of processing which uses an objective function as a feedback (evaluation value) of results of learning and performs learning by a plurality of processing subjects, and includes, for example, parallel distributed learning processing which optimizes an objective function.

Note that, in the deep learning, a stochastic gradient decent (SGD) method is used as a method to optimize an objective function, for example. In the SGD method, parameters of objective function are updated repeatedly using a vector to an optimized solution direction, which will be referred to as a gradient (vector). Note that the parameters of objective function include a weight, for example.

When a weight (weight vector) indicative of a current condition of SGD, gradient vector, and learning rate are represented as W(t), ∇W(t), and ε(t), weight Wt+1after the update will be obtained through the following formula.
W(t+1)=W(t)−ε(t)∇W(t)Formula (1)

Note that the learning rate ε(t)which determines an update width is adoptively determined corresponding to the progress of learning, and, for example, is decreased corresponding to the progress of learning.

The gradient is obtained by inputting training data to the objective function, and in general, a mini batch method in which a plurality of training data are input together to obtain an average gradient is used in consideration of calculation costs. The number of training data used to obtain the average gradient will be referred to as batch size. As the learning processes shared in parallel distributed optimization by the SGD, gradient will be used, for example.

Here, as a main algorithm of parallel distributed learning, a synchronized parallel distributed learning method and a non-synchronized parallel distributed learning method are available, for example.

In the parallel distributed learning process, a plurality of nodes perform calculation of gradient, and in the synchronized parallel distributed learning method, the nodes perform the calculation of gradient in synchronization. Specifically, the synchronized parallel distributed learning method is, for example, Synchronous-SGD in which the calculation of gradient of the mini batch method is dispersed in the nodes and an average value of the gradients calculated by the all nodes is used for update of weight. Note that there are several types of Synchronous-SGD, and there are, for example, a collective communication type in which gradients are shared in all nodes and a parameter server type in which gradients are gathered in a node which is referred to as a parameter server to perform updating of the weight and the updated weight is distributed to each node.

In the synchronized parallel distributed learning method, when the number of nodes (parallel numbers) calculating the gradient increases, costs for synchronization increase and the process throughput decreases, and when the number of nodes increases, the batch size increases and generalization performance decreases, and the whole process speed is influenced by the process speed of nodes of slow process speed.

On the other hand, in the non-synchronized parallel distributed learning method, a plurality of nodes perform calculation of gradient in a non-synchronized manner. Specifically, the non-synchronized parallel distributed learning method is, for example, Asynchronous-SGD which is an algorithm sharing the gradients as in the Synchronous-SGD. However, in the Asynchronous-SGD, gradients are not averaged by synchronization, and the weight is updated using the gradients calculated by the nodes as they are. Note that a parameter server type is mainly used as the Asynchronous-SGD.

In the non-synchronized parallel distributed learning method, high process throughput is obtained as compared to the synchronized parallel distributed learning method while there is a limit Lo its scalability since the convergence speed decreases because of a different between process speeds of the nodes.

Note that Asynchronous-SGD which has a different approach than Synchronous-SGD is, unlike the Synchronous-SGD, a parallel distributed learning algorithm which does not depend on batch size, and thus, may be referred to as a batch size independent parallel method (processing) or the like. Note that the batch size independent parallel method is basically the non-synchronized parallel distributed learning method.

Now, the outline of the system of the present embodiment (hereinafter referred to as present system) will be explained with reference toFIG.1. The present system is configured to perform different learning processes (for example, Synchronous-SD and other batch size independent parallel methods) in some levels, in the parallel distributed learning process.

Specifically, as shown inFIG.1, mini batch method parallel distributed processing is performed by Synchronous-SGD in each group of nodes in a first level and batch size independent parallel distributed processing is performed by representative nodes of the groups of the first level in a second level. Hereinafter, the present system will be explained in detail.

FIG.2shows an example of the structure of the present system. As shown inFIG.2, the present system10includes a server node20, a plurality of worker nodes30, and a plurality of worker nodes40.

In the present embodiment, the worker nodes30are in a group1, and the worker nodes40are in a group2.

The server node20is communicatively connected to one of the worker nodes30of group1(hereinafter referred to as representative node30of group1). Furthermore, the server node20is communicatively connected to one of the worker nodes40of group2(hereinafter referred to as representative node40of group2).

Note that, in the worker nodes30, the worker nodes30which are not communicatively connected to the server node20(that is, the worker nodes30other than the representative node30of group1) will be referred to as non-representative node30of group1. Furthermore, in the worker nodes40, the worker nodes40which are not communicatively connected to the server node20(that is, the worker nodes40other than the representative node40of group2) will be referred to as non-representative node40of group2.

In group1, the worker nodes30(representative node30and non-representative nodes30) are communicatively connected together. Similarly, in group2, the worker nodes40(representative node40and non-representative nodes40) are communicatively connected together.

In the present embodiment, in the first level, parallel distributed learning processing of mini batch method is performed by Synchronous-SGD in group1(worker nodes30) and group2(worker nodes40). Furthermore, in the second level, parallel distributed learning processing of batch size independent parallel distributed method is performed between the representative node30of group1and the representative node40of group2through the server node20.

Note that, whileFIG.2shows an example in which three worker nodes are included in each of groups1and2, the number of worker nodes may be two or more in each of the groups1and2. Furthermore, whileFIG.2shows only two groups (groups1and2), the number of groups may be three or more in the present system.

FIG.3shows an example of the system structure of the server node20ofFIG.2. The server node20includes, for example, CPU201, system controller202, main memory203, BIOS-ROM204, nonvolatile memory205, communication device206, and embedded controller (EC)207.

CPU201is a hardware processor configured to control operations of various components in the server node20. CPU201executes various programs loaded to the main memory203from the nonvolatile memory205which is a storage device. The programs include operating system (OS)203a,and various application programs. The application programs include a parallel distributed learning program203bfor server node.

Furthermore, CPU201executes basic input/output system (BIOS) stored in BIOS-ROM204. BIOS is a program for hardware control.

The system controller202is a device connecting between a local bus of CPU201and various components. The system controller202includes a memory controller for access control of the main memory203.

The communication device206is a device configured to execute wired or wireless communication. The communication device206includes a transmitter configured to transmit signals and a receiver configured to receive signals. The EC207is a one-chip microcomputer including an embedded controller for power management.

FIG.4shows an example of the system structure of the worker node30. Note that, although only the system structure of the worker node30will be explained, the worker node40has a similar structure.

The worker node30includes, for example, CPU301, system controller302, main memory303, BIOS-ROM304, nonvolatile memory305, communication device306, and embedded controller (EC)307.

CPU301is a hardware processor configured to control operations of various components in the worker node30. CPU301executes various programs loaded to the main memory303from the nonvolatile memory305which is a storage device. The programs include operating system (OS)303a,and various application programs. The application programs include a parallel distributed learning program303bfor worker node.

Furthermore, CPU301executes basic input/output system (BIOS) stored in BIOS-ROM304. BIOS is a program for hardware control.

The system controller302is a device connecting between a local bus of CPU301and various components. The system controller302includes a memory controller for access control of the main memory303.

The communication device306is a device configured to execute wired or wireless communication. The communication device306includes a transmitter configured to transmit signals and a receiver configured to receive signals. The EC307is a one-chip microcomputer including an embedded controller for power management.

FIG.5is a block diagram of an example of the functional structure of server node20. As shown inFIG.5, the server node20includes a training data storage21, data allocator22, transmission controller23, weight storage24, reception controller25, and calculator26.

In the present embodiment, the training data storage21and the weight storage24are stored in the nonvolatile memory205shown inFIG.3or the like. Furthermore, in the present embodiment, the data allocator22, transmission controller23, reception controller25, and calculator26are achieved by, for example, CPU201shown inFIG.3(that is, computer of server node20) executing the parallel distributed learning program203b(that is, software) stored in the nonvolatile memory205. Note that the parallel distributed learning program203bcan be distributed as preliminarily being stored in a computer readable memory medium. Furthermore, the parallel distributed learning program203bmay be downloaded in the server node20through a network, for example.

In this example, the components22,23,25, and26are achieved by software; however, the components22,23,25, and26may be achieved by hardware or by a combination of software and hardware.

The training data storage21stores training data used for calculation of gradient by each node (worker node) in the parallel distributed learning processing.

The data allocator22determines training data to be allocated to each of the worker nodes30and40from the training data stored in the training data storage21. The data allocator22divides the training data stored in the training data storage21into two parts, and allocates the divided training data to group1(specifically, the worker nodes30thereof) and group2(specifically, the worker nodes40thereof).

The transmission controller23includes a function to transmit various data through the communication device206. The transmission controller23transmits the training data allocated to group1(the worker nodes30thereof) by the data allocator22to the representative node30of group1. Furthermore, the transmission controller23transmits the training data allocated to group2(the worker nodes40thereof) by the data allocator22to the representative node40of group2.

The weight storage24stores the weight of objective function. Note that the weight stored in the weight storage24(that is, the weight managed by the server node20) is referred to as a master parameter.

The reception controller25includes a function to receive various data through the communication device206. The reception controller25receives gradient indicative of learning process on each of the worker nodes30and40. The gradient received by the reception controller25is calculated by each of the worker nodes30and40to update weight. The gradient calculated by each of the worker nodes30of group1is received from the representative node30of group1. The gradient calculated by each of the worker nodes40of group2is received from the representative node40of group2.

The calculator26updates the master parameter using the weight (master parameter) stored in the weight storage24and the gradient received from the reception controller25. In that case, the calculator26calculates the weight after update using the above Formula (1). The weight (weight after update) calculated by the calculator26is stored in the weight storage24as the master parameter, and is transmitted to the representative node30of group1or the representative node40of group2by the transmission controller23.

Hereinafter, the functional structure of the worker nodes30and40will be explained. Now, an example of the functional structure of the representative node30of group1will be explained with reference toFIG.6.

As shown inFIG.6, the representative node30of group1includes a reception controller31, training data storage32, weight storage33, calculator34, and transmission controller35.

In the present embodiment, the reception controller31, calculator34, and transmission controller35are achieved by, for example, CPU301ofFIG.4(that is, computer of representative node30) executing the parallel distributed learning program303bstored in the nonvolatile memory305(that is, software). Note that the parallel distributed learning program303bcan be distributed as preliminarily being stored in a computer readable storage medium. Furthermore, the parallel distributed learning program303bmay be downloaded in the representative node30through a network, for example.

In this example, the components31,34, and35are achieved by software; however, the components31,34, and35may be achieved by hardware or by a combination of software and hardware.

Furthermore, in the present embodiment, the training data storage32and the weight storage33are stored in the nonvolatile memory305shown inFIG.4or the like.

The reception controller31includes a function to receive various data through the communication device306. The reception controller31receives training data transmitted from the transmission controller23included in the server node20. In the training data received by the reception controller31, the training data allocated to the representative node30of group1are stored in the training data storage32. On the other hand, in the training data received by the reception controller31, the training data allocated to the non-representative nodes30of group1are transmitted from the representative node30of group1to the non-representative nodes30.

Furthermore, the reception controller31receives the gradient calculated by the non-representative nodes30of group1therefrom.

The weight storage33stores the weight of objective function. Note that the weight stored in the weight storage33(that is, the weight managed by the representative node30) will be referred to as the weight of representative node30of group1for easier understanding.

The calculator34calculates the gradient used for updating of the weight of objective function using the training data stored in the training data storage32and the weight stored in the weight storage33.

The transmission controller35includes a function to transmit various data through the communication device306. The transmission controller35transmits the gradient received by the reception controller31(gradient calculated by the non-representative nodes30) and the gradient calculated by the calculator34to the server node20.

Note that, as described above, if the weight calculated by the calculator26included in the server node20is transmitted from the server node20(transmission controller23), the weight is received by the reception controller31and is replaced with the weight stored in the weight storage33(weight before update). Thus, the weight of the representative node30of group1is updated. Furthermore, the weight is transmitted to the non-representative nodes30through the transmission controller35.

Now, an example of the functional structure of the non-representative node30of group1will be explained. The functional structure of the non-representative node30of group1will be explained with reference toFIG.6for easier understanding, and the structures different from those of the above representative node30of group1will be mainly explained.

The non-representative node30of group includes, as in the above representative node30of group1, a reception controller31, training data storage32, weight storage33, calculator34, and transmission controller35.

The reception controller31receives training data transmitted from the representative node30of group1. The training data received by the reception controller31are stored in the training data storage32.

The weight storage33stores the weight of objective function. Note that the weight stored in the weight storage33(that is, the weight managed by the non-representative node30) will be referred to as the weight of non-representative node30of group1for easier understanding.

As described above, if the weight (weight after update) is transmitted from the representative node30of group1, the weight is received by the reception controller31and is replaced with the weight stored in the weight storage33(weight before update). Thus, the weight of the non-representative node30of group1is updated.

The calculator34calculates the gradient used for updating of the weight of objective function using the training data stored in the training data storage32and the weight stored in the weight storage33. The gradient calculated by the calculator34is transmitted to the representative node30by the transmission controller35.

In this example, the representative node30and the non-representative node30of group1are explained, and the same functional structures apply to the representative node40and the non-representative node40of group2. Thus, in the following description, the functional structure of the representative node40and the non-representative node40of group2will be explained with reference toFIG.6.

Hereinafter, an example of the process flow of present system will be explained with reference to the sequence chart ofFIG.7. Note that the process between the server node20, group1(worker nodes30), and group2(worker nodes40) will be explained mainly, and the process of worker nodes of each group (group1and group2) will be explained later.

Note that, the training data allocated to each of the worker nodes30of group1are stored in the training data storage32included in the worker nodes30. The same applies to the worker nodes40of group2.

Furthermore, the same weight (hereinafter referred to as weight W0) is stored in the weight storage24included in the server node20and the weight storage33included in each of the worker nodes30and40.

In that case, a gradient calculation process is performed in group1(the worker nodes30thereof) (Step S1). In this gradient calculation process, each of the worker nodes30of group1calculates the gradient to update the weight of objective function using the training data stored in the training data storage32included in the worker nodes30and the weight W0stored in the weight storage33. Note that the worker nodes30of group1execute the gradient calculation process in synchronization.

The gradient calculated by the worker nodes30in step S1is transmitted to the server node20from the representative node30of group1(step S2).

The server node20(the reception controller25therein) receives the gradient transmitted in step S2. The server node20(the calculator26therein) calculates a new weight (hereinafter referred to as weight W1) using the received gradient and the weight W0stored in the weight storage24included in the server node20. Thus, the weight W0stored in the weight storage24is updated to the calculated weight W1(step S3).

The server node20(the transmission controller23) distributes the weight W1updated from the weight W0in step S3(master parameter after update) to group1(step S4).

As above, the weight W1distributed from the server node20is stored in the weight storage33included in each of the worker nodes30of group1. In that case, in group1, the following gradient calculation process can be performed using the weight to which the gradient calculated by group1is reflected.

On the other hand, in group2(the worker nodes40thereof), the gradient calculation process is performed as in group1(step S5). Through the gradient calculation process, each of the worker nodes40of group2calculates the gradient to update the weight of objective function using the training data stored in the training data storage32included in the worker nodes40and the weight W0stored in the weight storage33. Note that the worker nodes40of group2execute the gradient calculation process in synchronization.

The gradient calculated in step S5is transmitted from the representative node40of group2to the server node20(step S6).

The server node20receives the gradient transmitted in step S6. Here, the weight (master parameter) stored in the weight storage24included in the server node20is the weight W1updated in step S3.

Thus, the server node20calculates a new weight (hereinafter referred to as weight W2) using the received gradient and the weight W1. Thus, the weight W1stored in the weight storage24is updated to the calculated weight W2(step S7).

The server node20distributes the weight W2(master parameter) updated from the weight W1in step S7to group2(step S8).

The weight W2distributed from the server node20is stored in the weight storage33included in each of the worker nodes40of group2.

Here, the weight W2is updated using the gradient calculated in group2from the weight W1which is updated using gradient calculated in group1. That is, the weight W2is a weight calculated using the gradient calculated in group1(gradient calculated in step S1) and the gradient calculated in group2(gradient calculated in step S5). As in this case, when the calculation of gradient by group1is faster than the calculation of gradient by group2, parallel distributed learning processing is performed in group2using the weight updated using the gradient calculated by group1.

Therefore, in group2, the following gradient calculation process can be performed using the weight to which not only the gradient calculated by group2but also the gradient calculated by group1are reflected.

Furthermore, when steps S1to S4are performed, in group1, steps S9to S12corresponding to steps S1to S4are performed. In these steps, the weight W2is updated to a new weight (hereinafter referred to as weight W3) using the gradient calculated in the gradient calculation process in group1and the weight W2stored in the weight storage24included in the server node20. The weight W3is distributed to the worker nodes30of group1. Note that, in step S9, the gradient is calculated using training data which are different from the training data used in the gradient calculation process in step S1.

Here, the weight W3is further updated using the gradient calculated in group1from the weight W2which is updated using gradient calculated in group2. As in this case, when the calculation of gradient by group2is faster than the calculation of gradient by group1, parallel distributed learning processing is performed in group1using the weight updated using the gradient calculated by group2.

Therefore, in group1, the following gradient calculation process can be performed using the weight to which not only the gradients calculated by group1(gradients calculated in steps S1and S9) but also the gradient calculated by group2(gradient calculated in step S5) are reflected.

On the other hand, when steps S5to S8are performed, in group2, steps S13to S16corresponding to steps S5to S8are performed. In these steps, the weight W3is updated to a new weight (hereinafter referred to as weight W4) using the gradient calculated in the gradient calculation process in group2and the weight W3stored in the weight storage24included in the server node20. The weight W4is distributed to the worker nodes40of group2. Note that, in step S13, the gradient is calculated using training data which are different from the training data used in the gradient calculation process in step S5.

Here, the weight W4is updated using the gradient calculated in group2from the weight W3which is updated using gradient calculated in group1.

Therefore, in group2, the following gradient calculation process can be performed using the weight to which not only the gradients calculated by group2(gradients calculated in steps S5and S13) but also the gradients calculated by group1(gradients calculated in steps S1and S9) are reflected.

FIG.7shows steps S1to S16; however, the process ofFIG.7is performed repeatedly until the gradient calculation process (that is, parallel distributed learning process) is performed to all of the training data stored in the training data storage32of each of the worker nodes30and40.

As described above, in the present embodiment, the process is performed in groups1and2in synchronization while the process between the server node20and group1(the representative node30thereof) and the server node20and group2(the representative node40thereof) are performed in non-synchronization.

Now, the processes of the representative node and the non-representative node of each group when the process ofFIG.7is performed will be explained.

Initially, an example of the process flow of the representative node will be explained with reference to the flowchart ofFIG.8. Here, the process flow of the representative node30of group1will be explained.

The calculator34included in the representative node30calculates a gradient using the training data stored in the training data storage32and the weight (for example, weight W0) stored in the weight storage33(step S21). Hereinafter, the gradient calculated by the representative node30will be referred to as gradient of representative node30.

Note that, when the representative node30of group1performs the process of step S21, the non-representative nodes30of group1calculate a gradient in synchronization with the representative node30. Hereinafter, the gradient calculated by the non-representative nodes30will be referred to as gradient of non-representative node30.

In that case, the reception controller31receives the gradient of non-representative node30therefrom (step S22). Note that, in the present system, if the non-representative nodes30are in group1, the reception controller31receives a gradient from each of the non-representative nodes30.

Then, the calculator34calculates an average value of the gradient calculated in step S21(gradient of representative node30) and the gradient received in step S22(gradient of non-representative node30) (step S23). Hereinafter, the average value of the gradients calculated in step S23will be referred to as average gradient.

The transmission controller35transmits the average gradient of group1to the server node20(step S24).

Note that steps S21to S24are performed by the representative node30of group1in steps S1and S2(of steps S9and S10) ofFIG.7.

In that case, the process of steps S3and S4shown inFIG.7are performed by the server node20. That is, in the server node20, the master parameter is updated with the average gradient of group1transmitted in step S24, and the master parameter (for example, weight W1) after the update is transmitted from the server node20to the representative node30of group1.

When the master parameter is transmitted form the server node20, the reception controller31receives the master parameter (step S25).

The transmission controller35transmits the master parameter received in step S25to the non-representative node30(step S26).

The weight (for example, weight W0) stored in the weight storage33is replaced with the master parameter received in step S25(for example, weight W1) (step S27). Thus, the weight of representative node30of group1is updated to the master parameter (the weight corresponding thereto).

Note that steps S25to S27are performed by the representative node30after the process of step S4(or step S12) ofFIG.7.

When the process ofFIG.8is performed, the weight of representative node30of group1is updated to the weight calculated using the average gradient of group1, and in the following calculation of gradient, the updated weight can be used.

Note that, although this is not shown, the process ofFIG.8is performed repeatedly while the process ofFIG.7is continued.

Now, an example of the process flow of non-representative node will be explained with reference to the flowchart ofFIG.9. Here, the process flow of non-representative node30of group1will be explained.

The calculator34included in the non-representative node30calculates a gradient using the training data stored in the training data storage32and the weight (for example, weight W0) stored in the weight storage33in synchronization with the calculation of gradient in the representative node30(step S31).

When the process of step S31is performed, the transmission controller35transmits the gradient calculated in step S31(gradient of non-representative node30) to the representative node30(step S32).

Note that the process of steps S31and S32are performed by the non-representative node30in steps S1and S2(or in steps S9and S10) ofFIG.7.

When the process of step S32is performed, in the representative node30, the process of steps S22to S26ofFIG.8are performed. In that case, the master parameter (for example, weight W1) transmitted from the server node20is transmitted from the representative node30of group1to the non-representative node30.

When the master parameter is transmitted from the representative node30, the reception controller31receives the master parameter (step S33).

The weight (for example, weight W0) stored in the weight storage33is replaced with the master parameter received in step S33(step S34). Thus, the weight of non-representative node30of group1is updated to the master parameter (the weight corresponding thereto).

Note that the process of steps S33and S34is performed by the non-representative node30after the process of step S4(or step S12) ofFIG.7.

Through the process ofFIG.9, the weight of non-representative node30of group1is updated to the weight calculated using the average gradient of group1, and in the following calculation of gradient, the updated weight can be used.

Note that, although this is not shown, the process ofFIG.9is performed repeatedly while the process ofFIG.7is continued.

As described above, in group1, the gradients of all worker nodes30of group1are gathered to the representative node30, and an average gradient is calculated in the representative node30. In that case, for example, a collective communication algorithm which is referred to as Reduce (MPI_Reduce) defined by Message Passing Interface (MPI) can be used to effectively perform the transmission of gradient to the representative node30from the non-representative node30and the calculation process of the average gradient (sum of gradients of all worker nodes30). Here, the case where MPI_Reduce is used is explained; however, a different process equivalent to MPI_Reduce may be used instead.

In this example, the process of group1(representative node30and non-representative node30) is explained, and a similar process is performed in group2(representative node40and non-representative node40).

As described above, in the present embodiment, the system includes a plurality of worker nodes30(representative node and non-representative nodes) of group1and a plurality of worker nodes (representative node and non-representative nodes)40of group2. When the worker nodes30perform nth parallel distributed processing using an objective function as a reference, for example, the representative node (first node)30of group1calculates a first gradient to update a first weight of objective function to a second weight, and the non-representative node (second node)30of group1calculates a second gradient to update the first weight of objective function to the second weight.

On the other hand, when the worker nodes40perform mth parallel distributed processing in non-synchronization manner with the parallel distributed processing by the worker nodes30, for example, the representative node (third node)40of group2calculates a third gradient to update a third weight of objective function to a fourth weight, and the non-representative node (fourth node)40of group2calculates a fourth gradient to update the third weight of objective function to the fourth weight.

Here, in the present embodiment, if the calculation of gradient by group1(representative node30and non-representative node30) is faster than the calculation of gradient by group2(representative node40and non-representative node40), the second weight updated from the first weight is further updated in n+1th parallel distributed processing by group1using the first and second gradients, and the fourth weight updated from the third weight is further undated in m+1th parallel distributed processing by group2using the first to fourth gradients.

On the other hand, if the calculation of gradient by group2(representative node40and non-representative node40) is faster than the calculation of gradient by group1(representative node30and non-representative node30), the second weight updated from the first weight is further updated in n+1th parallel distributed processing by group1using the first to fourth gradients, and the fourth weight updated from the third weight is further updated in m+1th parallel distributed processing by group2using the third and fourth gradients.

As describe above, in the present embodiment, the worker nodes30and40are divided into a plurality of groups (group1and group2), and parallel distributed learning processing by Synchronous-SGD is performed in groups as a first level. In the first level, since synchronization is performed in each group, costs for synchronization and batch size can be suppressed as compared to a case where the worker nodes30and40are all synchronized for processing.

Furthermore, in the second level, parallel distributed learning processing is performed between representative nodes of the groups in the first level through the batch size independent parallel method, In the second level, the representative nodes do not need to work in synchronization, and thus, a high throughput can be obtained.

That is, in the present embodiment, since, for example, Synchronous-SGD and the batch size independent parallel method are combined in levels, a high scalability in parallel distributed learning processing can be achieved, and the parallel distributed learning processing with greater parallel number can be performed.

Now,FIG.10shows a time (learning time) required to obtain a predetermined generalization performance of each learning method.

FIG.10shows, as learning methods, non-parallel distributed method (single node learning method), Synchronous-SGD, batch size independent parallel method, and the method of the present embodiment (Synchronous-SGD+batch size independent parallel method).

As shown inFIG.10, if a learning time required to obtain a predetermined generalization performance in a learning process through a non-parallel distributed method (hereinafter referred to as learning time of non-parallel distributed method) is given 1.0, a learning time required to obtain a predetermines generalization performance in a learning process through a parallel distributed learning processing by Synchronous-SGD (hereinafter referred to as learning time of Synchronous-SGD) is given 0.6.

Similarly, if the learning time of non-parallel distributed method is given 1.0, a learning time required to obtain a predetermined generalization performance in parallel distributed learning processing through a batch size independent parallel method (hereinafter referred to as learning time of batch size independent parallel method) is given 0.5.

As compared to these examples, in the method of the present embodiment (parallel distributed method in levels), theoretically, the upper limits of each scalability of Synchronous-SGD and the batch size independent parallel method are multiplied as the scalability achieved thereby.

Specifically, a learning time required to obtain a predetermined generalization performance in the parallel distributed learning processing of the method of the present embodiment (a ratio of learning time thereof to a learning time of non-parallel distributed method) is a multiplied value of a ratio of the learning time of Synchronous-SGD to the learning time of non-parallel distributed method (0.6) and a ratio of the of learning time of batch size independent parallel method to the learning time of non-parallel distributed method (0.5), that is, 0.3.

As can be understood from the above, when Synchronous-SGD is used, a predetermined generalization performance can be achieved in 60% of the learning time required in the non-parallel distributed method, and when the batch size independent parallel method is used, a predetermined generalization performance is achieved in 50% of the learning time required in the non-parallel distributed method, and when the method of the present embodiment is used, a predetermined performance can be achieved in only 30% of the learning time required in the non-parallel distributed method.

Therefore, in the present embodiment, a high scalability can be achieved as compared to a case where the parallel distributed learning processing by Synchronous-SGD or parallel distributed learning processing by batch size independent parallel method is simply performed.

Note that, in the present embodiment, the second weight is calculated using a fifth gradient calculated from the first gradient calculated by the representative node30of group1and the second gradient calculated by the non-representative node30of group1(for example, the average value of the first and second gradients). Similarly, the fourth weight is calculated using a sixth gradient calculated from the third gradient calculated by the representative node40of group2and the fourth gradient calculated by the non-representative node40of group2(for example, the average value of the third and fourth gradients).

Furthermore, in the present embodiment, the second weight and the fourth weight are calculated in the server node20which is communicatively connected to the representative node30of group1and the representative node40of group2.

When the second weight is calculated in the server node20, the server node20transmits the second weight to the representative node30of group1and the representative node30transmits the second weight to the non-representative node30. In the present embodiment, with such a structure, the weights of the representative node30and the non-representative node30of group1can be updated to the weight calculated by the server node20.

Furthermore, when the fourth weight is calculated in the server node20, the server node20transmits the fourth weight to the representative node40of group2and the representative node40transmits the fourth weight to the non-representative node40. In the present embodiment, with such a structure, the weights of the representative node40and the non-representative node40or group2can be updated to the weight calculated by the server node20.

Note that, in the present embodiment, if the calculation of gradient by group1is faster than the calculation of gradient by group2, the weight is updated in n+1th parallel distributed processing by group1using the first and second gradients, and the weight is further updated in m+1th parallel distributed processing by group2using the first to fourth gradients.

Here, a case where “the calculation of gradient by group1is faster than the calculation of gradient by group2” includes a case where the server node20receives a gradient calculation result of group1(gradient transmitted from the representative node30of group1) before receiving a gradient calculation result of group2(gradient transmitted from the representative node40of group2).

That is, if the server node20receives the gradient calculation result of group1before receiving the gradient calculation result of group2, for example, the weight is updated using the gradient calculation result of group1(that is, the first and second gradients) in the n+1th parallel distributed processing by group1(a later parallel distributed processing), and the weight is further updated using the gradient calculation result of group1(and the weight updated based thereon) and the gradient calculation result of group2(that is, the first to fourth gradients) in the m+1th parallel distributed processing by group2.

Furthermore, in the present, embodiment, if the calculation of gradient by group2is faster than the calculation of gradient by group1, the weight is updated in m+1th parallel distributed processing by group2using the third and fourth gradients, and the weight is further updated in n+1th parallel distributed processing by group1using the first to fourth gradients.

Here, a case where “the calculation of gradient by group2is faster than the calculation of gradient by group1” includes a case where the server node20receives a gradient calculation result of group2(gradient transmitted from the representative node40of group2) before receiving a gradient calculation result of group1(gradient transmitted from the representative node30of group1).

That is, if the server node20receives the gradient calculation result of group2before receiving the gradient calculation result of group1, for example, the weight is updated using the gradient calculation result of group2(that is, the third and fourth gradients) in the m+1th parallel distributed processing by group2(a later parallel distributed processing), and the weight is further updated using the gradient calculation result of group2(and the weight updated based thereon) and the gradient calculation result of group1(that is, the first to fourth gradients) in the n+1th parallel distributed processing by group1.

That is, in the present embodiment, the update of the weight may be performed not on the basis of the order of the gradient calculation processes by the groups (which group can perform the gradient calculation process faster) but on the basis of the order of reception of the gradient calculation results from the groups by the server node20(which result is received by the server node20faster).

Note that, as described above, in Synchronous-SGD, the worker nodes30of group1performs the process in synchronization, for example; however, if a difference of process performances of the worker nodes30(process speeds based on the performances) is great, the process speed of group1is influenced by the process speed of the worker node30of low process performance (that is, the process speed of the worker node30of low process performance becomes dominant). The same applies to group2.

Therefore, the process speeds of worker nodes of the same group are adjusted to be substantially the same. Specifically, a difference of process speeds between the worker nodes30of group1(representative node30and non-representative nodes30) is set to a first threshold or less, and a difference of process speeds between the worker nodes40of group2(representative node40and non-representative nodes40) is set to a second threshold or less. Note that the first threshold and the second threshold may be the same value or different values.

Furthermore, as shown inFIG.11, if the process speed of the worker nodes40of group2is slower than the process speed of the worker nodes30of group1, for example, the number of the worker nodes30of group1may be set to be less than the number of the worker nodes40of group2. Note that, a case where the process speed of the worker nodes40of group2is slower than the process speed of the worker nodes30may include a case where an average value of the process speeds of the worker nodes40is less than an average value of the process speeds of the worker nodes30, or a case where the slowest process speed of the process speeds of the worker nodes40is slower than the slowest process speed of the process speeds of the worker nodes30. Furthermore, the process speed of each worker node may be calculated from a hardware performance or the like of the worker node, for example.

Furthermore, if the process speed of the worker nodes40of group2is slower than the process speed of the worker nodes30of group1, the process amount of group2in the parallel distributed processing (that is, training data allocated to group2) is set to be less than the process amount of group1(that is, training data allocated to group1). In that case, the number of worker nodes30of group1and the number of worker nodes40of group2may be the same.

That is, with the structure described above, a process time required in each group can be substantially the same (that is, influence caused by the difference in process speeds can be canceled) by adjusting the number of worker nodes of each group or the process amount (workload) of each group.

In the present embodiment, the server node20, each of the worker nodes30, and each of the worker nodes40are realized in a single device (machine) (that is, each node and the device is in relationship of one-to-one); however, each node may be achieved as a process or a thread executed in one device. That is, the entire system of the present embodiment (server node20, worker nodes30, and worker nodes40) can be achieved by a single device. Furthermore, the system of the present embodiment may be achieved by a plurality of devices number of which is different from the number of nodes.

That is, in the present embodiment, one node may be one computer (server), or a plurality of nodes may be mounted on one computer, or one node may be implemented by a plurality of computers. Note that, in the present embodiment, as described above, two or more groups can be included in one system, and two or more nodes can be included in one group.

Furthermore, in the present embodiment, Asynchronous-SGD which is a non-synchronized parallel distributed learning method is explained as a batch size independent parallel method; however, other methods such as Elastic Averaging SGD may be adopted.

Furthermore, in the present embodiment, different methods (algorithms) of parallel distributed learning processing are used between the first level and the second level; however, depending on algorithms to be combined, the parallel distributed learning processing may be performed in three or more levels.

Second Embodiment

Now, the second embodiment will be explained. Note that the system of the present embodiment is configured to perform different learning processes in some levels as in the first embodiment described above.

That is, parallel distributed processing of mini batch method by Synchronous-SGD in each group including a plurality of worker nodes is performed in the first level, and parallel distributed processing of batch size independent method (non-synchronization parallel distributed method) is performed between the representative nodes of groups in the second level.

FIG.12shows an example of the structure of the system of the present embodiment (hereinafter referred to as present system). As shown inFIG.12, the present system10includes a plurality of worker nodes30and a plurality of worker nodes40.

In the first embodiment described above, a server node20is included; however, the present embodiment does not include a server node20, and in this respect, the present embodiment is different from the first embodiment. Note that the worker nodes30are included in group1and the worker nodes40are included in group2as in the first embodiment described above.

One worker node30of the worker nodes30of group1(hereinafter referred to as representative node of group1) is communicatively connected to one worker node40of the worker nodes40of group2(hereinafter referred to as representative node of group2).

Note that, in the worker nodes30, the worker nodes30other than the representative node30of group1will be referred to as non-representative nodes30of group1. Similarly, in the worker nodes40, the worker nodes40other than the representative node40of group2will be referred to as non-representative nodes40of group2.

In the present embodiment, in the first level, parallel distributed learning processing is performed by Synchronous-SGD in group1(worker nodes30) and group2(worker nodes40). Furthermore, in the second level, parallel distributed learning processing of batch size independent parallel distributed method is performed between the representative node30of group1and the representative node40of group2.

Note that, whileFIG.12shows an example in which three worker nodes are included in each of groups1and2, the number of worker nodes may be two or more in each of the groups1and2. Furthermore, whileFIG.12shows only two groups (groups1and2), the number of groups may be three or more in the present system.

The system structure of the worker nodes30and40is similar to the first embodiment, and thus, the detailed explanation thereof will be omitted.

Hereinafter, an example of functional structure of the representative node30of group1of the worker nodes30and40will be explained. Note that the functional structure of the representative node30of group1of the present embodiment will be explained with reference toFIG.6for easier understanding, and the parts different from the representative node30of group1of the first embodiment will be mainly explained.

As shown inFIG.6, the representative node30includes a reception controller31, training data storage32, weight storage33, calculator34, and transmission controller35.

The reception controller31receives a gradient calculated in the non-representative nodes30of group1therefrom.

The training data allocated to the representative node30of group1are stored in the training data storage32. The weight of objective function is stored in the weight storage33.

The calculator34calculates the gradient to update the weight of objective function using the training data stored in the training data storage32and the weight stored in the weight storage33.

The calculator34updates the weight using the gradient received by the reception controller31(that is, gradient calculated by the non-representative nodes30), gradient calculated by the calculator34, and the weight stored in the weight storage33. In that case, the calculator34calculates the weight after update using the above Formula (1). The weight calculated by the calculator34is replaced with the weight stored in the weight storage33. Thus, the weight of the representative node30of group1is updated.

The transmission controller35transmits the gradient calculated by the calculator34to the non-representative nodes30of group1.

Furthermore, the transmission controller35transmits the gradient calculated by the non-representative nodes30of group1and gradient calculated by the calculator34(that is, gradient calculated by the representative node30) to a representative node of another group (for example, group2).

Here, the gradient calculated by the non-representative nodes30of group1and the gradient calculated by the representative node30of group1are transmitted to the representative node40of group2as mentioned above, and similarly, a gradient calculated by the non-representative nodes40of group2and a gradient calculated by the representative node40of group2are transmitted to the representative node30of group1.

When the gradient calculated by the non-representative nodes40of group2and the gradient calculated by the representative node40are received by the representative node30of group1(reception controller31), the calculator34updates the weight stored in the weight storage33using the gradients, and the transmission controller35transmits the gradients to the non-representative nodes30of group1.

Now, an example of functional structure of the non-representative node30of group1will be explained. Note that the functional structure of the non-representative node30of group1of the present embodiment will be explained with reference toFIG.6for easier understanding, and the parts different from the representative node30of group1of the first embodiment will be mainly explained.

The non-representative node30of group1includes, as in the representative node30of group1, a reception controller31, training data storage32, weight storage33, calculator34, and transmission controller35.

The reception controller31receives a gradient calculated in the representative node30of group1and a gradient calculated in other non-representative nodes30from the representative node30and the non-representative nodes30.

The training data allocated to the non-representative node30of group1are stored in the training data storage32. The weight of objective function is stored in the weight storage33.

The calculator34calculates the gradient to update the weight of objective function using the training data stored in the training data storage32and the weight stored in the weight storage33.

The calculator34updates the weight using the gradients received by the reception controller31(that is, gradient calculated by the representative node30and the gradient calculated by other non-representative nodes30), gradient calculated by the calculator34, and the weight stored in the weight storage33. In that case, the calculator34calculates the weight after update using the above Formula (1). The weight calculated by the calculator34is replaced with the weight stored in the weight storage33. Thus, the weight of the non-representative node30of group1is updated.

Note that, as described above, when the gradient calculated by the non-representative nodes40of group2and the gradient calculated by the representative node40of group2are transmitted by the transmission controller35included in the representative node30of group1, the gradients are received by the reception controller31and used for update of the weight.

While the representative node30and non-representative nodes30of group1are explained, in the present embodiment, the same functional structure is applied to the representative node40and the non-representative nodes40of group2. Thus, when the functional structure of the representative node40and the non-representative nodes40of group2are explained below,FIG.6will be used for reference.

Hereinafter, an example of the process flow of the present system will be explained with reference to the sequence chart ofFIG.13. Here, the process between group1(worker nodes30) and group2(worker nodes40) will be mainly explained, and the process of each worker node of each of the groups (group1and group2) will be explained later.

In this example, for example, a weight W10is stored in the weight storage33included in each of the worker nodes30of group1, and a weight W20is stored in the weight storage33included in each of the worker nodes40of group2. Note that the weight W10and the weight W20may be the same value-or difference values.

Initially, in group1(the worker nodes30thereof), a gradient calculation process by Synchronous-SGD is performed (step S41). Through the gradient calculation process, each of the worker nodes30of group1calculates a gradient to update a weight of objective function using the training data stored in the training data storage32included in the worker node30and the weight W10stored in the weight storage33. Note that the worker nodes30of group1perform the gradient calculation process in synchronization.

Each of the worker nodes30calculates a new weight (hereinafter referred to as weight W11) using the gradient calculated in step S41and the weight W10stored in the weight storage33included in the worker node30. Thus, the weight W10stored in the weight storage33included in each of the worker nodes30is updated to the calculated weight W11(step S42).

Here, when the process of step S42is performed, the gradient calculated in step S41is transmitted from the representative node30of group1to the representative node40of group2(step S43).

The representative node40of group2receives the gradient transmitted in step S43. The received gradient is shared in the worker nodes40of group2. Thus, each of the worker nodes40calculates a new weight (hereinafter referred to as weight W21) using the gradient received by the representative node40of group2and the weight W20stored in the weight storage33included in the worker node40. Thus, the weight W20stored in the weight storage33included in the worker node40is updated to the calculated weight W21(step S44).

Then, in group2(the worker nodes40thereof), as in group1, a gradient calculation process by Synchronous-SGD) is performed (step S45). Through the gradient calculation process, each of the worker nodes40of group2calculates a gradient to update a weight of objective function using the training data stored in the training data storage32included in the worker node40and the weight W21stored in the weight storage33. Note that the worker nodes40of group2perform the gradient calculation process in synchronization.

Each of the worker nodes40calculates a new weight (hereinafter referred to as weight W22) using the gradient calculated in step S45and the weight W21stored in the weight storage33included in the worker node40. Thus, the weight W21stored in the weight storage33included in each of the worker nodes40is updated to the calculated weight W22(step S46).

Here, when the process of step S46is performed, the gradient calculated in step S45is transmitted from the representative node40of group2to the representative node30of group1(step S47).

The representative node30of group1receives the gradient transmitted in step S47. The received gradient is shared in the worker nodes30of group1. Thus, each of the worker nodes30calculates a new weight (hereinafter referred to as weight W12) using the gradient received by the representative node30and the weight W11stored in the weight storage33included in the worker node30. Thus, the weight W11stored in the weight storage33included in the worker node30is updated to the calculated weight W12(step S48).

FIG.13shows steps S41to S48; however, the process ofFIG.13is performed repeatedly until the gradient calculation process (that is, parallel distributed learning process) is performed to all of the training data stored in the training data storage32of each of the worker nodes30and40.

As described above, in the present embodiment, the process is performed in groups1and2in synchronization while the process between group1and group2is performed in non-synchronization.

That is, in step S43ofFIG.13, the gradient calculated in step S41is transmitted from the representative node30of group1to the representative node40of group2, and a timing of transmitting the gradient is after the process of steps S41and S42, for example, and is not influenced by the process of group2(the worker nodes40thereof). Similarly, a timing of transmitting the gradient in step S47ofFIG.13is after the process of steps S45and S46, for example, and is not influenced by the process of group1(the worker nodes30thereof).

Hereinafter, the processes of the representative node and the non-representative nodes of each group when the process ofFIG.13is performed will be explained.

Initially, an example of the process flow of the representative node will be explained with reference to the flowchart ofFIG.14. Here, the process flow of the representative node30of group1will be explained.

The calculator34included in the representative node30of group1calculates a gradient using the training data stored in the training data storage32and the weight (for example, weight W11) stored in the weight storage33(step S51), Hereinafter, the gradient calculated by the representative node30will be referred to as gradient of representative node30.

Note that, when the representative node30of group1performs the process of step S51, the non-representative nodes30of group1calculate a gradient in synchronization with the representative node30as described later. Hereinafter, the gradient calculated by the non-representative nodes30will be referred to as gradient of non-representative node30.

In that case, the gradient of representative node30of group1and the gradient of the non-representative node30are distributed in group1(step S52). That is, the gradient of representative node30of group1is transmitted from the representative node30to each of the non-representative nodes30, and the gradient of each of the non-representative nodes30of group1is received by the representative node30of group1(the reception controller31therein).

Then, the calculator34calculates an average value of the gradient calculated in step S51(gradient of representative node30) and the gradient received by the reception controller31(gradient of non-representative node30) (step S53). Hereinafter, the average value of the gradients calculated in step S53will be referred to as average gradient of group1.

When the process of step S53is performed, the calculator34calculates a new weight using the average gradient of group1, and updates the weight stored in the weight storage33to the calculated weight (for example, weight W11) (step S54). Thus, the weight representative node30of group1is updated to the weight using the gradient calculated by each of the worker nodes30of group1.

When the process of step S54is performed, the transmission controller35transmits the average gradient of group1to the representative node40of group2(step S55).

The above process of steps S51to S55is performed by the representative node30of group1in steps S41to S43shown inFIG.13.

Note that, as described later, the process ofFIG.14is performed similarly by the representative node40of group2. Thus, for example, if the process corresponding step S55is performed by the representative node40of group2, the reception controller31included in the representative node30of group1can receive an average gradient of group2.

Here, whether or not the reception controller31receives the average gradient of group2is determined (step S56).

If it is determined that the average gradient of group2is received (YES in step S56), the transmission controller35transmits a reception flag “True” to the non-representative nodes30of group1(step S57).

Furthermore, the transmission controller35transmits the average gradient of group2received by the reception controller31to the non-representative nodes30of group1(step S58).

When the process of step S58is performed, the calculator34calculates a new weight using the average gradient of group2, and updates the weight stored in the weight storage33to the calculated weight (for example, weight W12) (step S59). Thus, the weight of representative node30of group1is updated to the weight using the gradient calculated by each of the worker nodes40of group2.

The above process of steps S56to S59is performed by the representative node30of group1in step S48shown inFIG.13.

Note that, if it is determined that the average gradient of group2is not received in step S56(NO in step S56), the transmission controller35transmits a reception flag “False” to the non-representative nodes30(step S60).

Through the above process ofFIG.14, the weight of the representative node30of group1is updated using the gradient calculated by each of the worker nodes30of group1(average gradient of group1) and is further updated using the gradient calculated by each of the worker nodes40of group2(average gradient of group2).

Note that, although this is not shown, the process ofFIG.14is performed repeatedly while the process ofFIG.13is continued.

Now, an example of the process flow of non-representative node will be explained with reference to the flowchart ofFIG.15. Here, the process flow of non-representative node30of group1will be explained.

The calculator34included in the non-representative node30calculates a gradient using the training data stored in the training data storage32and the weight (for example, weight W10) stored in the weight storage33in synchronization with the calculation of gradient in the representative node30(step S71).

In that case, the gradient of representative node30and the gradient calculated in step S71(gradient of the non-representative node30) are distributed in group1(step S72). That is, the gradient of representative node30of group1is transmitted from the non-representative node30to the representative node30(and other non-representative nodes30) of group1, and the gradient of the representative node30(and gradients of other non-representative nodes30) are received by the non-representative node30(the reception controller31therein).

Then, the calculator34calculates an average value of the gradient calculated in step S71(gradient of non-representative node30) and the gradients received by the reception controller31(gradients of representative node30and other non-representative nodes30) (step S73). Note that the average value of the gradients calculated in step S73corresponds to the average gradient of group1calculated in step S53ofFIG.14.

When the process of step S73is performed, the calculator34calculates a new weight using the average gradient of group1, and updates the weight stored in the weight storage33to the calculated weight (for example, weight W11) (step S74). Thus, the weight of non-representative node30of group1is updated to the weight using the gradient calculated by each of the worker nodes30of group1.

The above process of steps S71to S74is performed by the non-representative node30of group1in steps S41to S42shown inFIG.13.

Here, the reception flag transmitted from the representative node30in step S57or S60ofFIG.14is received by the reception controller31included in the non-representative node30.

In that case, whether or not the reception controller31receives the reception flag “True” is determined (step S76).

If it is determined that the reception flag “True” is received (YES in step S76), the reception controller31receives the average gradient of group2transmitted from the representative node30of group1in step S58ofFIG.14(step S77).

When the process of step S77is performed, the calculator34calculates a new weight using the average gradient of group2, and updates the weight stored in the weight storage33to the calculated weight (for example, weight W12) (step S78). Thus, the weight of non-representative node30of group1is updated to the weight using the gradient calculated by each of the worker nodes40of group2.

The above process of steps S75to S78is performed by the non-representative node30of group1in step S48shown inFIG.13.

Note that, if the reception flag “True” is not received in step S76(that is, the reception flag “False” is received) (NO in step S76), the average gradient of group2is not received, and thus, the process of steps S77and S78are not performed.

Through the above process ofFIG.15, the weight of the non-representative node30of group1is updated using the gradient calculated by each of the worker nodes30of group1(average gradient of group1) and is further updated using the gradient calculated by each of the worker nodes40of group2(average gradient of group2).

Note that, although this is not shown, the process ofFIG.15is performed repeatedly while the process ofFIG.13is continued.

As described above, in group1, the gradients are shared between all worker nodes30of group1, and an average gradient is calculated in each worker node30of group1. In that case, a collective communication algorithm which is referred to as Allreduce (MPI_Allreduce) defined by MPI can be used to effectively perform the transmission of gradient between the worker nodes30and the calculation process of the average gradient (sum of gradients of all worker nodes30). Here, the case where MPI_Allreduce is used is explained; however a different process equivalent to MPI_Allreduce may be used instead.

In this example, the process of the representative node30and the non-representative nodes30of group1is explained, and a similar process is performed as to the representative node40and the non-representative nodes40of group2.

Here, in the present embodiment, if the calculation of gradient by group1(representative node30and non-representative node30) is faster than the calculation of gradient by group2(representative node40and non-representative node40), the gradient calculated in group1is transmitted to the representative node40of group2. In that case, the representative node40of group2(and non-representative nodes40) calculate (update) the weight in the parallel distributed learning processing using the average gradient of group1.

Furthermore, if the calculation of gradient by group2is faster than the calculation of gradient by group1, the gradient calculated by group2is transmitted to the representative node30of group1. In that case, the representative node30of group1(and non-representative nodes30) calculate (update) the weight in the parallel distributed learning processing using the average gradient of group2.

As describe above, in the present embodiment, the worker nodes30and40are divided into a plurality of groups (group1and group2), and parallel distributed learning processing by collective communication type Synchronous-SGD is performed in groups as a first level. In the first level, the gradient is shared between the worker nodes of the groups, an average gradient is calculated in each of the worker nodes, and the weight is updated. In the first level, costs for synchronization and batch size can be suppressed.

Furthermore, in the second level, parallel distributed learning processing is performed between representative nodes of the groups in the first level through the batch size independent parallel method. In the second level, the representative nodes do not need to work in synchronization, and thus, a high throughput can be obtained.

That is, in the present embodiment, as in the first embodiment, since Synchronous-SGD and the batch size independent parallel method, for example, are combined in levels, a high scalability in parallel distributed learning processing can be achieved, and the parallel distributed learning processing with greater parallel number can be performed.

Note that, in the present embodiment, a case where “the calculation of gradient by group1is faster than the calculation of gradient by group2” includes a case where the representative node40of group2receives a gradient calculated by group1(gradient calculation result from group1) before calculation of gradient by group2.

That is, if the representative node40of group2receives the gradient calculated by group1before calculation of gradient by group2, for example, in the group1, the weight is updated using the gradient calculated by group1(that is, the first and second gradients). On the other hand, in group2, after the update of the weight using the gradients calculated by group1, the weight is further updated using the gradients calculated by group2(that is, the third and fourth gradients). In other words, if the representative node40of group2receives the gradient calculated by group1before the calculation of gradient by group2, the weight is updated using the first and second gradients in group1and the weight is updated using the first to fourth gradients in group2.

Furthermore, in the present embodiment, a case where “the calculation of gradient by group2is faster than the calculation of gradient by group1” includes a case where the representative node30of group1receives a gradient calculated by group2(gradient calculation result from group2) before calculation of gradient by group1.

That is, if the representative node30of group1receives the gradient calculated by group2before calculation of gradient by group1, for example, in the group2, the weight is updated using the gradient calculated by group2(that is, the third and fourth gradients). On the other hand, in group1, after the update of the weight using the gradients calculated by group2, the weight is further updated using the gradients calculated by group1(that is, the first and second gradients). In other words, if the representative node30of group1receives the gradient calculated by group2before the calculation of gradient by group1, the weight is updated using the third and fourth gradients in group2and the weight is updated using the first to fourth gradients in group1.

That is, in the present embodiment, the update of the weight may be performed not on the basis of the order of the gradient calculation processes by the groups (which group can perform the gradient calculation process faster) but on the basis of the order of reception of the gradient calculation results from the groups by the representative nodes of groups1and2.

Note that, in the present embodiment, the gradient is shared between groups; however, the weight updated in each of the worker nodes in the groups may be shared between the groups.

According to at least one of the above-described embodiments, a system, program, and method which can achieve a high scalability in the parallel distributed learning processing can be provided.

While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.