Patent Publication Number: US-2019188570-A1

Title: Methods and apparatus for model parallelism in artificial neural networks

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is based on and claims the benefit of European Application No. 17208970.8, filed Dec. 20, 2017, in the European Intellectual Property Office, the disclosure of which is incorporated herein by reference. 
     BACKGROUND 
     Field 
     Embodiments discussed herein relate to methods and apparatus for model parallelism in artificial neural networks. 
     Description of the Related Art 
     Computational units in an artificial neural network (ANN) are modelled after neurons in the human brain, the neurons in the ANN being grouped by layers. Typically there is an input layer of neurons, an output layer of neurons, and hidden layers of neurons, for example convolution, pooling, rectified linear units, fully connected layers, etc. A Deep Neural Network (DNN) is an ANN with multiple hidden layers of computational units between input and output layers. Each computational unit combines different inputs, which are weighted, to compute a function. This function may be a linear combination of the weighted inputs, or something more elaborate such as a sigmoid function. When training an ANN, the outputs of the network are compared with a desired output using a loss function and an error value is calculated for each neuron in the output layer. The error values are then back-propagated until each neuron in the network has an error value. These error values are used to calculate the gradients of the loss function with respect to the weights in the network, the gradients in turn being used to update the weights in order to minimize the loss function. 
     DNNs offer the potential to achieve significant advancements in speech and image recognition, with accuracy performance exceeding those recorded by other sophisticated methods in Machine Learning (ML). However, the training process of DNNs is an extremely computationally intensive task, which typically requires large computational resources, including training (execution) time, and memory (RAM). To address the long training times, state-of-the-art techniques make use of hardware accelerators, including, for example, CPUs or Intel® Xeon Phi™, exploiting their vast computational power. 
     However, these accelerators have memory restrictions, as they usually include a limited amount of in-device memory. Such memory restriction poses a problem in situations where the DNN to be trained requires more memory than that available within a single accelerator. In other words, where the parameters and the activations required to train the DNN do not fit into a single accelerator&#39;s memory, the process responsible for the training process cannot be performed straightaway. 
     In order to solve this problem, one proposed solution has been to split the parameters of a layer of neurons of the DNN and distribute such parameters across different accelerators, changing the training process accordingly to accommodate the distributed allocation of the weights. This is what is generally called ‘model parallelism’ (as opposed to ‘data parallelism’, where the entire DNN is replicated and stored on all accelerators, processing samples of the training data in parallel, for example as disclosed in WO2015003436). 
     In some circumstances, as discussed for example in Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama and T. Darrell, “Caffe: Convolutional Architecture for Fast Feature Embedding,”  arXiv preprint arXiv: 1408.5093, 2014 (hereafter “Caffe™”), such a training process with distributed parameters is not feasible. A training process with distributed parameters is disclosed in M. Abadi, A. Agarwal and P. Barham, “Large-Scale Machine Learning on Heterogeneous Distributed Systems,” arXiv:1603.04467v2, 2015 and S. Tokui, K. Oono, S. Hido and J. Clayton, “Chainer: a Next-Generation Open Source Framework for Deep Learning,”  Proceedings of Workshop on Machine Learning Systems  ( LearningSys )  in The Twenty - ninth Annual Conference on Neural Information Processing Systems  ( NIPS ), 2015, but the distribution has to be manually defined. As discussed in T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang and Z. Zhang, “MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems,”  Neural Information Processing Systems, Workshop on Machine Learning Systems,  2015, discloses another training process, in which the actual distribution is not done by splitting a particular layer, but by placing different layers at different accelerators, for example. 
     W. Wang, G. Chen, H. Chen, T. T. A. Dinh, J. Gao, O. Beng Chin, K.-L. Tan and S. Wang, “Deep Learning at Scale and at Ease,” ACM Trans. Multimedia Comput. Commun. Appl., Vol. 12, No. 4s, Article 69, November 2016 (hereafter “SINGA”) proposes a framework that partitions a neural network at the granularity of the layers, the allocation to the different resources being static, i.e. it is not possible to change or adapt the allocation during the execution of a DNN. Moreover, it is still for a user to decide how the layers are partitioned, and hence there is not a complete automatic handling of how the layers are distributed. 
     Another limitation seen across different proposals is that, once separated, there is no way to recombine parameters corresponding to distributed layers (for example for serial execution or testing purposes). It is desirable to provide an improved method and apparatus for model parallelism in artificial neural networks. 
     SUMMARY 
     According to an embodiment of an aspect there is provided a computer-implemented method comprising: automatically controlling allocation, to memories of available hardware resources, of parameters defining computational operations required to calculate an output of at least one layer of neurons of an artificial neural network, ANN, wherein: the allocation is controlled on the basis of previously-defined allocation data specifying how the operations required to calculate the output of the at least one layer of neurons are to be allocated to hardware resources to perform the operations, and the allocation data has been pre-defined using, at least partly, an automatic computer-implemented process. 
     This method has the technical effect of making the set-up and execution of an ANN using the memories and processing capabilities of multiple hardware resources simpler and more efficient. In an embodiment the details of how the parameters of a distributed layer in an ANN, such as a DNN, are to be split across different hardware resources, such as accelerators, are defined automatically, at least in part. This allocation information, which is shared by all processes or threads assigned to process each subpart of a particular layer, is used to automatically control the logic of how these distributed parameters are actually split. This allows a user to focus on the actual design of the architecture, regardless of how the layers will later be distributed across different hardware resources. 
     Such a method may realize dynamic and flexible high-level model parallelism. In particular, an embodiment may realize model parallelism for DNNs, hiding the details and the complexity of the distribution. As a result, this solution may be applied to any framework to provide model parallelism capabilities. These model parallelism capabilities allow ML practitioners to train DNNs with a larger number of parameters, overcoming the limitation of the memory available in the accelerators typically used. Having unlocked this possibility, larger problems may be tackled, improving the response from current artificial intelligence (AI) systems. 
     The allocation data may specify the number and identity of hardware resources to be used, how the parameters are to be split into groups, and how the groups of parameters are to be distributed amongst the hardware resources. The allocation data may be initially defined on the basis of at least some information that has been obtained automatically by the computer-implemented process. The initial definition of the allocation data may also take into account additional information that has been input by a user of the ANN. That is, optionally, an embodiment allows for a personalised distribution, by taking user preferences as an input. 
     The information used to define the allocation data may relate to at least one of the definition of the ANN, the system to be used to execute the ANN, and the available hardware resources. 
     The automatic computer-implemented process to pre-define the allocation data may include checking before each iteration of the network which of the hardware resources are available to execute that iteration of the network and, if necessary, re-defining the allocation data for that iteration accordingly. 
     All or different subsets of the network of hardware resources available at the particular machine in which the ANN is executed may be used, and how allocation of the different subparts of the distributed layer is done may be changed dynamically, from one iteration of the network to another. 
     For example, in cloud computing or virtual computing environments, where the underlying hardware may change, it may be beneficial to have a DNN solution that works regardless of changes in, or current availability of, hardware resources. As a result, users of cloud computing services may be able to experiment with different DNN configurations more quickly, since users would not need to deal with the details of the actual distribution of the DNN, but would be able to focus on the actual design and tuning of the designed network architecture. 
     Controlling allocation of parameters may comprise carrying out a set-up process to set up the ANN for execution and a subsequent execution process to execute the ANN. The allocation data may be initially defined before the set-up process. 
     The set-up process may comprise verifying that hardware resources specified by the allocation data for execution of the ANN are available for use. If at least one of the hardware resources is not available for use, the allocation data may be updated so as to exclude allocation of parameters to memory of the unavailable hardware resource. Allocation of the parameters to the memories of hardware resources may be carried out in accordance with the current allocation data. The set-up process may further comprise allocating a copy of all parameters to memory in a predetermined hardware resource. 
     Therefore, an embodiment may achieve an automatic dynamic distribution of layer parameters of an ANN, which allows for changes from one iteration of layer computation to another, depending on the availability of the underlying hardware resources. 
     The execution process may include verifying that hardware resources specified by the allocation data for execution of the ANN are available for use. If at least one of the hardware resources is no longer available for use, the parameters previously allocated to memory of the hardware resource that is no longer available may be reallocated to memory of at least another one of the hardware resources that is available for use. The allocation data may be updated so as to correspond to the reallocation of parameters. The execution process may further include creating processes or threads to execute respective computational operations as defined in the current allocation data and causing the computational operations to be performed. When a backward propagation phase of a layer has been executed, the execution process may further include updating the parameters of the layer in the memories of the relevant hardware resources in accordance with the result of the backward propagation. 
     Such a method may allow dynamic reallocation of layer parameters of an ANN to different available hardware resources. CPU and accelerator memory may be linked in a seamless manner so that, from the ANN perspective, the details of how the layers parameters are distributed, as well as the details of the necessary sub-operations, are hidden. 
     An embodiment may allow changes to be made in how a particular layer of a DNN is executed even during the same training process. In particular, fault-tolerant execution of a DNN, restarting the execution of the DNN from the last successful iteration, may be possible. 
     According to an embodiment of an aspect there is provided a computer program which, when run on a computer, causes that computer to carry out a method. 
     According to an embodiment of a third aspect there is provided apparatus comprising: a processor to automatically control allocation, to memories of available hardware resources, of parameters defining computational operations required to calculate an output of at least one layer of neurons of an artificial neural network, ANN; and memory storing allocation data specifying how the operations required to calculate the output of the at least one layer of neurons are to be allocated to hardware resources to perform the operations, the allocation data having been defined using, at least partly, an automatic computer-implemented process; the processor controlling allocation on the basis of the allocation data. The automatic computer-implemented process to pre-define the allocation data may include checking before each iteration of the network which of the hardware resources are available to execute that iteration of the network and, if necessary, re-defining the allocation data for that iteration accordingly. 
     Apparatus according to an embodiment, hereafter sometimes referred to as a layer controller, may perform an automatic, dynamic, and flexible distribution of the layer parameters according to the allocation data shared by all processes or threads assigned to process each subpart of a particular layer. The achieved distribution of layer parameters is flexible since it may change according to the hardware resources available. 
     The allocation data may specify the number and identity of hardware resources to be used, how the parameters are to be split into groups, and how the groups of parameters are to be distributed amongst the hardware resources. 
     The allocation data may be initially defined on the basis of at least some information that has been obtained automatically by the computer-implemented process. Thus, an embodiment may realize an automatic flexible distribution of layer parameters of an ANN, depending on the underlying hardware resources, without the need for any user contribution. 
     The initial definition of the allocation data may also take into account additional information that has been input by a user of the ANN. For example, the definition of the allocation data may be guided by the user via an input file with information about the underlying topology (how many accelerators, memory, etc.). This may allow ML practitioners to experiment with different distributions with the aim of finding which one may work for a particular combination of DNN and hardware settings. The information may relate to at least one of the definition of the ANN, the system to be used to execute the ANN, and the available hardware resources. 
     The processor may carry out a set-up process to set up the ANN, the set-up process comprising verifying that hardware resources specified by the allocation data for execution of the ANN are available for use. If at least one of the hardware resources is not available for use, the allocation data may be updated so as to exclude allocation of parameters to memory of the unavailable hardware resource. Allocation of the parameters to the memories of hardware resources may be carried out in accordance with the current allocation data. The set-up process may further comprise allocating a copy of all parameters to memory in a predetermined hardware resource. 
     An embodiment of the layer controller may be able to use all or various subsets of the accelerators available at the particular machine in which a DNN is executed, and change dynamically, from one iteration of the network to another, how allocation of the different subparts of the distributed layer is done. Thus dynamic model parallelism, i.e. changing from one distribution of layer parameters to another depending on the availability of accelerators at any given time, may be achieved. 
     The processor may carry out an execution process to execute the ANN, the execution process including verifying that hardware resources specified by the allocation data for execution of the ANN are available for use. If at least one of the hardware resources is no longer available for use, the parameters previously allocated to memory of the hardware resource that is no longer available may be reallocated to memory of at least another one of the hardware resources that is available for use, and updating the allocation data so as to correspond to the reallocation of parameters. The execution process may further include creating processes or threads to execute respective computational operations as defined in the current allocation data and causing the computational operations to be performed. When a backward propagation phase of a layer has been executed, the execution process may further include updating the parameters of the layer in the memories of the relevant hardware resources in accordance with the result of the backward propagation. 
     Thus, the actual distribution of layer parameters may change from one iteration to another. This dynamism may play a crucial role in fault-tolerant scenarios, as well as cloud and virtual computing environments, in which the conditions and availability of the accelerators may change. For example, higher priority jobs may reclaim some of the accelerators in use during training of a DNN, forcing the training framework to stop. In that case an embodiment may dynamically rebalance the workload to the remaining available accelerators. As a result, the training framework will not stop or crash in such circumstances. In another example, if one or more accelerators being used in a DNN training process were to fail, the training framework would continue the training from the last successful iteration, instead of having to re-start from the last snapshot of the layer parameters (if taken), which might lead to repeating many more iterations, or even, in the absence of snapshots, having to repeat the training process from the beginning. 
     These together with other aspects and advantages which will be subsequently apparent, reside in the details of construction and operation as more fully hereinafter described and claimed, reference being had to the accompanying drawings forming a part hereof, wherein like numerals refer to like parts throughout. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       These and/or other aspects and advantages will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings. Reference will now be made, by way of example, to the accompany drawings, in which: 
         FIG. 1 a    is a flowchart of a method in accordance with an embodiment; 
         FIG. 1 b    is a block diagram illustrating apparatus in accordance with an embodiment; 
         FIG. 2 a    is a flowchart of a previously-proposed method of training a DNN; 
         FIG. 2 b    is a flowchart of a method of training a DNN in accordance with an embodiment; 
         FIG. 3  is a diagram for use in explaining how a layer of a DNN is distributed in accordance with an embodiment; 
         FIG. 4  is a flowchart of a process for use with a method in accordance with an embodiment; 
         FIGS. 5 a , 5 b  and 5 c    are diagrams for use in explaining how the parameters of a layer may be partitioned; 
         FIG. 6  is a flowchart of a DNN set up process in a method in accordance with an embodiment; 
         FIG. 7  is a flowchart of a DNN execution process in a method in accordance with an embodiment; 
         FIG. 8  is a diagram for use in explaining the use of pointers to memory in an embodiment; 
         FIG. 9  is a diagram for use in explaining serial and parallel execution of DNN processes; 
         FIG. 10  is a diagram for use in explaining an embodiment; 
         FIGS. 11 a  and 11 b    show respective examples of computer code illustrating the definition of a network and how a user defines and launches training; 
         FIG. 12  is a diagram for use in explaining an application of an embodiment; 
         FIGS. 13 a , 13 b , 13 c , 13 d  and 13 e    diagrams for use in explaining another application of an embodiment; and 
         FIG. 14  is a block diagram of a computing device suitable for carrying out a method of an embodiment. 
     
    
    
     DETAILED DESCRIPTION 
     Reference will now be made in detail to the present embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments are described below to explain the present invention by referring to the figures. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended, such alterations and further modifications in the illustrated device, and such further applications of the principles of the invention as illustrated therein being contemplated as would normally occur to one skilled in the art to which the invention relates. 
     The flowchart of  FIG. 1 a    shows a method in accordance with an embodiment which comprises, in operation S 100 , automatically controlling allocation, to memories of available hardware resources, of parameters defining computational operations required to calculate an output of at least one layer of neurons of an ANN. The method may comprise sub-operation S 10  in which a set-up process to set up the ANN for execution is carried out and sub-operation S 20  in which an execution process to execute the ANN is carried out. The allocation is controlled on the basis of previously-defined allocation data specifying how the operations required to calculate the output of the at least one layer of neurons are to be allocated to hardware resources to perform the operations. The allocation data is pre-defined using an automatic computer-implemented process that may additionally be customized by user specifications. The automatic computer-implemented process to pre-define the allocation data may include checking before each iteration of the network which of the hardware resources are available to execute that iteration of the network and, if necessary, re-defining the allocation data for that iteration accordingly. 
     Apparatus in accordance with an embodiment is shown in  FIG. 1 b   . Layer controller  10  comprises a processor  1  and memory  2 . Processor  1  is configured to automatically control allocation, to memories of available hardware resources, of parameters defining computational operations required to calculate an output of at least one layer of neurons of an ANN in accordance with pre-defined allocation data stored in memory  2 . 
     An application of an embodiment in the training of a DNN will now be explained in comparison to a previously-proposed method. In the previously-proposed method illustrated in  FIG. 2 a   , an original (i.e. a previously-constructed) DNN is set up for execution on a network of accelerators at operation S 1  and is executed in operation S 2 . The configuration of the network is defined by a user, i.e. the user must be aware of the underlying accelerators, and is static, i.e. cannot be changed during execution of the DNN. As discussed above, in previously-proposed methods such as that of FIG.  2   a,  a training process with distributed parameters was either not possible, or the actual distribution was not done by splitting a particular layer, but by placing different layers at different accelerators, or it had to be manually defined, which required knowledge of the underlying hardware, and a certain level of user expertise. 
     In contrast, in an embodiment such as that illustrated in  FIG. 2 b   , the original DNN is set up for execution in operation S 1 A and then executed in operation S 2 A under the control of a layer controller  10 . Because of this the set up does not require the user to have any knowledge of the underlying accelerators. Furthermore, use of such a layer controller  10  allows the DNN to be flexible and dynamically executable using different accelerators. 
     The diagram of  FIG. 3  shows how a current layer, e.g. layer L n−1 , layer L n , layer L n+1 , of a DNN to be executed is distributed on underlying accelerators, e.g. accelerator A# 0  and/or accelerator A# 1 , under the control of a layer controller  10  in accordance with an embodiment. The process of calculating the output of a particular layer may be done in several different ways, and by using different accelerators. The proposed layer controller  10  is operable to ensure that a particular layer of the DNN is distributed and executed in accordance with previously-defined allocation data, hereafter referred to as “global state”, that specifies how the necessary operations to calculate the output of the layer are distributed and offloaded to different accelerators. This global state is obtained prior to set up of the DNN, as will be explained with reference to  FIG. 4 . For example, as shown in  FIG. 3 , in accordance with the global state read by layer controller  10 , layer controller  10  ensures that the operations necessary to calculate the output of layer L n−1  are allocated to accelerator A# 0 , that the operations necessary to calculate the output of layer L n  are distributed between accelerator A# 0  and accelerator A# 1 , and that the operations necessary to calculate the output of layer L n+1  are allocated to accelerator A# 1 . 
       FIG. 4  illustrates an embodiment of an initial set up process of a DNN (operation S 1 A in  FIG. 1 b   ) in which the global state is defined. The process starts in operation S 41  by reading the definition of a DNN, i.e. the listing of the different layers, their characteristics, and their connections, etc. (in Caffe™, for example, DNNs are defined following a prototxt format). In operation S 42  system information is read. This may comprise reading and parsing system files (such as/proc/cpuinfo), or the output of certain commands, such as nvidia-smi and/or dmesg, to extract information that assists in automatically building a comprehensive view of the underlying hardware. In operation  43  the availability of the hardware resources, in particular the accelerators, and the specifications of those resources, is checked, for example how many accelerators are available, how much memory they have, their interconectivity, etc. In operation S 44  of the process, how the parameters of each layer of the DNN are to be distributed at execution time is automatically determined, using the definition of the DNN, the system information, and the knowledge of the underlying hardware acquired in operations S 41  to  43  and (optionally) any user preferences which have been input. In operation S 45  the global state, which specifies the distribution properties, e.g. the number of accelerators to be used, the number of blocks to be created, the axis of splitting, per layer to be distributed, is created and stored. This information is stored in an internal data structure that may be exported to an interchangeable format (JSON) if needed. This may also be useful for logging purposes. The global state may comprise a descriptor, which may be implemented as an independent file or as a data structure in accordance with that used in the system. It will typically be handled as a data structure, but the creation of an independent file is useful to log the different distributions when profiling multiple experiments, for example. This global state descriptor may hold the distribution properties as elements in a multiple-entry list, one per layer to be distributed. Each entry may hold a set of key-value pairs that describe such properties (block ID, accelerator, block size, pointer to CPU memory, pointer to accelerator memory). As subsequently explained with reference to  FIG. 6 , these values will be read later by the layer controller  10  to determine how the layer parameters are to be distributed at each iteration of the DNN training. 
       FIG. 5  shows different potential partitioning of a Binary Large Object (BLOB) representing a particular layer&#39;s parameters. While the layer parameters remain unsplit at the CPU&#39;s memory MC (lefthand side of  FIG. 5 ), the corresponding copy of the parameters at accelerators&#39; memories MA may be split in one of several different ways. Visualizing the layer parameters as multi-dimensional arrays, for example, as shown in  FIG. 5 , the layer parameters may be split along one axis ( FIGS. 5 b  and 5 c   ) or multiple axes ( FIG. 5 a   ) and may be split one or more times along the splitting axis (once along two axes in  FIG. 5 a    creating eight blocks, once along one axis in  FIG. 5 b    creating two blocks, and three times along one axis creating four blocks). The blocks resulting from the splitting may be allocated to different accelerators&#39; memories, for example in  FIG. 5 a    the eight blocks are allocated respectively to the memories of eight accelerators GPU# 0 , GPU# 1 , GPU# 2 , GPU# 3 , GPU# 5 , GPU# 6 , GPU# 7 , GPU# 8 . In an embodiment, how the layer parameters are to be split may be automatically decided. For example, in the case that there are two accelerators (GPU# 0 , GPU# 1  as shown in  FIG. 5 b   ), that the number of inputs to the layer is y, and the number of outputs is x, the layer parameters may be split into two blocks each of size 
     
       
         
           
             
               
                 x 
                 * 
                 y 
               
               2 
             
             . 
           
         
       
     
     Splitting may also be customized in accordance with input user preferences, which may state how many accelerators are to be used, how many blocks are to be created, and/or the axis of splitting, per layer to be distributed. 
       FIG. 6  and  FIG. 7  illustrate processes followed by the layer controller  10  at different stages in accordance with embodiments. 
       FIG. 6  shows the process followed by the layer controller  10  in the set up phase of the DNN (operation S 1 A of  FIG. 1 b   ), after the initial set-up process of  FIG. 4  has been carried out to create the global state, to allocate the necessary memory for the layer parameters prior to execution of the DNN (operation S 2 A of  FIG. 1 b   ). In operation S 61  the layer controller  10  reads the stored global state, which was created during the initial set up of the DNN ( FIG. 4 ). Then, in operation S 62 , the status of the available accelerators is checked, in order to validate the defined distribution of the layer parameters according to the global state. In operation S 63  it is determined whether the result of the check on the status of the accelerators indicates that one or more of the accelerators to be used has failed, is absent, or has provided no response. If that is the case (No, operation S 63 ), the method proceeds to operation S 64  in which the global state is updated accordingly, following a process similar to the one described with respect to  FIG. 4 , but under the new conditions. Once the global state has been verified in operation S 63  (Yes) or updated in operation S 64 , the method allocates memory at the CPU for the unsplit layer parameters (operation S 65 ), and at the different accelerators (operation S 66 ). As a result, the method produces a list of pointers to the different memory locations of each block of layer parameters at each accelerator. These pointers will later be used to locate such blocks and calculate the layer&#39;s output accordingly.  FIG. 7  illustrates the process followed by the layer controller  10  during forward and backward phases of the execution of the DNN (operation S 2 A of  FIG. 1 b   ). In the process, for each iteration of the training (one cycle of the forward and backward phases), training data is received in operation S 70  and the stored global state is read in operation S 71 . In operation S 72  the status of the accelerators is checked in operation S 72  to determine whether it is in accordance with that of the global state. In the case of failure, absence, or lack of response from one or several of these accelerators (Yes, operation S 73 ), in operation S 74  blocks of layer parameters are reallocated amongst the memories of the remaining accelerators accordingly, returning new values for the pointers to the different memory locations of each block for each split layer. In most cases it will be necessary to reallocate all the parameters, not just those previously allocated to the failed accelerator(s), in order to balance the workload of each accelerator, as shown in  FIG. 10 , which illustrates reallocation and update of the global state to go from a distribution over four CPUs (GPU# 0  to GPU# 3 ) to a distribution over three CPUs (GPU# 0  to GPU# 2 ). In addition, the parameters located at each accelerator are updated, data being moved from the reference copy at the CPU, for example as shown in  FIG. 10 . This movement is facilitated by storing pointers to the corresponding sub-parts at the CPU&#39;s memory MC, as well as the size of each sub-part, for example as shown in  FIG. 8 . Original layer parameters allocated to memory of the CPU are shown on the lefthand side of  FIG. 8 , and partitioned layer parameters allocated to the different accelerators&#39; memory location (using four CPUs as an example) are shown to the right. Offsets to the corresponding sub-parts at the CPU&#39;s memory are also stored, to aid the movement of data from the CPU to the accelerators and vice versa. In operation S 75  the global state is updated according to the changes made in operation S 74 , following a process similar to the one described with reference to  FIG. 4 , but under the new conditions. 
     The process of  FIG. 7  continues at operation S 76 , after updating of the global state in operation S 75  or if all accelerators are verified as operational in operation S 73  (No), by reading the list of pointers to the accelerators&#39; memories and determining the sub-operations that are necessary to calculate the layer&#39;s output as if the layer is unsplit (that is, the output of the distributed layer should be equivalent to the serial layer&#39;s output, regardless of how the layer has been actually split). These sub-operations involve the multiplication of split multi-dimensional matrices, which are stored at different accelerators (memory locations are kept in the list of pointers previously mentioned), and therefore, the logic of such sub-operations depends on the actual distribution, including how many blocks there are and the axis along which the layer parameters are split. As a consequence, each of the different sub-operations required to calculate the output may have different requirements, e.g. dimensionality of the submatrices involved, or operations before and/or after the multiplication. These latter operations may involve the addition and/or concatenation of matrices, in order to produce a result which is equivalent to the one resulting from a serial unsplit execution of the layer. In any case, the actual mathematical operations performed with such matrices may be done by a linear algebra library, such as cuBLAS. 
     After determination of the sub-operations and parameters in operation S 76 , new processes/threads are created in operation S 77 . As shown in  FIG. 9 , which illustrates serial execution on the lefthand side and model-parallel execution on the righthand side, the execution of each sub-operation of a layer is done by a different process or thread, which is created dynamically prior to the actual execution of the required sub-operations. 
     In this way, whenever there is a change in the conditions for the layer operation, only the necessary processes or threads are created to handle the resulting sub-operations. After creation of the new processes/threads, the sub-operations are performed at operation S 78 . Finally, at operation S 79 , the layer controller  10  returns the output of the layer, which means that, from the network perspective, an input was given, and an output was calculated, without any more detail regarding how the actual operations were executed. In the case that the layer is executing its backward propagation phase, there is one additional task that is performed, at operation S 78 A, which is the update of the layer parameters, both those located at the memory MC of the CPU and those located at the memories MA of the accelerators. 
     Embodiments may be implemented as an additional module to potentially any framework, providing it with model parallel capabilities, and encapsulating the details and complexity of the distribution. For example, the proposed method may be implemented within the Caffe™ framework. Caffe™ implements a mechanism in which the CPU and the GPU memory are related by a wrapper. This wrapper considers that the representation of a particular multi-dimensional array is identical in both the CPU and GPU memories. A module in accordance with an embodiment may be attached to this wrapper, modifying the wrapper to take into account the pointers and offset explained with reference to  FIG. 8 . In this way, when a function within Caffe TM  tries to move data from CPU to GPU memory (or vice versa), this movement would now be limited to the accelerators to which this data has been allocated. The same would happen when returning pointers to the position in memory of the layer parameters. In Caffe TM , only one position is returned, as the layer&#39;s operation is not split. By implementing the proposed method within Caffe™, it would be necessary to create new processes/threads, and each one of these would receive a different pointer, corresponding to the sub-part required for its sub-operation. 
       FIGS. 11 a  and 11 b    show respective examples of computer code illustrating the definition of a network and how a user defines and launches training in accordance with SINGA&#39;s approach to model distribution ( FIG. 2 a   ) and an embodiment if implemented in Caffe™ ( FIG. 2 b   ). The code shown in  FIG. 2 a    was extracted from github project page: https://github.com/apache/incubator-single/blob/master/examples/cifar 10 /alexnet-parallel.cc). As can be seen from  FIG. 2 a   , SINGA&#39;s approach is actually restricted by the user contribution, as it forces the user to define the distribution explicitly for each different model and underlying architecture. 
       FIG. 12  illustrates how the dynamic capabilities of the proposed method would allow different executions to be prepared when Caffe™ is going through its test and train phases. The train iterations of Caffe™ may be performed in parallel, while the test iterations may be executed serially, at the CPU or using only one CPU&#39;s memory. This is also useful when Caffe™ is trained using specific hardware, but the inference/test phase of the DNN is done in another system which is only able to handle serial execution. 
       FIG. 13  shows another example of an application of the proposed method, in this case to the recovery of a training process when one of the accelerators in use fails. Since embodiments allow for dynamic allocation of the layer parameters across different iterations, fault recovery is possible.  FIG. 13 a    illustrates the intended configuration at a certain iteration i. If at the beginning of iteration i there is a failure at one of the accelerators ( FIG. 13 b   ), or simply a lack of response, an embodiment may either make use of another available accelerator to reallocate the corresponding portion of layer parameters ( FIG. 13 c   ), or reallocate all layer parameters to one or more other accelerators already in use ( FIG. 13 d   ), or simply fall back to an execution using only the memory and capabilities of the CPU ( FIG. 13 e   ). 
       FIG. 14  is a block diagram of a computing device, such as a data storage server, which may be used to implement some or all of the operations of a method of an embodiment, and perform some or all of the tasks of apparatus of an embodiment. For example, the computing device of  FIG. 14  may be used to implement operation S 100 , operation S 10  or operation S 20  of the method illustrated in  FIG. 1 a   , and to perform some or all of the tasks of the layer controller  10  shown in  FIG. 1   b.    
     The computing device comprises a processor  993 , and memory,  994 . Optionally, the computing device also includes a network interface  997  for communication with other such computing devices, for example with other computing devices of invention embodiments. 
     For example, an embodiment may be composed of a network of such computing devices. Optionally, the computing device also includes one or more input mechanisms such as keyboard and mouse  996 , and a display unit such as one or more monitors  995 . The components are connectable to one another via a bus  992 . 
     The memory  994 , which may for example serve as memory  2  of the layer controller  10 , or memory MC of the CPU, or memory MA of an accelerator A, may include a computer readable medium, which term may refer to a single medium or multiple media (e.g., a centralized or distributed database and/or associated caches and servers) configured to carry computer-executable instructions or have data structures stored thereon. Computer-executable instructions may include, for example, instructions and data accessible by and causing a general purpose computer, special purpose computer, or special purpose processing device (e.g., one or more processors) to perform one or more functions or operations. Thus, the term “computer-readable storage medium” may also include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methods of the present disclosure. The term “computer-readable storage medium” may accordingly be taken to include, but not be limited to, solid-state memories, optical media and magnetic media. By way of example, and not limitation, such computer-readable media may include non-transitory computer-readable storage media, including Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid state memory devices). 
     The processor  993 , which may for example serve as processor  1  of the layer controller  10 , is configured to control the computing device and execute processing operations, for example executing computer program code stored in the memory  994  to implement some or all of the methods described with reference to  FIGS. 1 a , 2 b   ,  3 ,  4 ,  5 ,  6 ,  7 ,  8 ,  9 ,  10 ,  12  and/or  13  and defined in the claims. For example, processor  993  may execute computer program code to implement each of operations S 10  and S 20  of  FIG. 1 b   , or only operation S 10  of  FIG. 1 b    in whole or in part, or only operation S 20  of  FIG. 1 b    in whole or in part. 
     The memory  994  stores data being read and written by the processor  993 . As referred to herein, a processor may include one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. The processor may include a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. The processor may also include one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. In one or more embodiments, a processor is configured to execute instructions for performing the operations and operations discussed herein. 
     The display unit  995  may display a representation of data stored by the computing device and may also display a cursor and dialog boxes and screens enabling interaction between a user and the programs and data stored on the computing device. The input mechanisms  996  may enable a user to input data and instructions to the computing device. 
     The network interface (network I/F)  997  may be connected to a network, such as the Internet, and is connectable to other such computing devices via the network. The network I/F  997  may control data input/output from/to other apparatus via the network. 
     Other peripheral devices such as microphone, speakers, printer, power supply unit, fan, case, scanner, trackerball etc may be included in the computing device. 
     Methods embodying the present invention may be carried out on a computing device such as that illustrated in  FIG. 14 . Such a computing device need not have every component illustrated in  FIG. 14 , and may be composed of a subset of those components. A method embodying the present invention may be carried out by a single computing device in communication with one or more data storage servers via a network. The computing device may be a data storage itself storing at least a portion of the data. 
     A method embodying the present invention may be carried out by a plurality of computing devices operating in cooperation with one another. One or more of the plurality of computing devices may be a data storage server storing at least a portion of the data. 
     Embodiments may be implemented in hardware, or as software modules running on one or more processors, or on a combination thereof. That is, those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functionality described above. The invention may also be embodied as one or more device or apparatus programs (e.g. computer programs and computer program products) for carrying out part or all of the methods described herein. Such programs embodying the present invention may be stored on computer-readable media, or could, for example, be in the form of one or more signals. Such signals may be data signals downloadable from an Internet website, or provided on a carrier signal, or in any other form. 
     The above-described embodiments of the present invention may advantageously be used independently of any other of the embodiments or in any feasible combination with one or more others of the embodiments. 
     The many features and advantages of the embodiments are apparent from the detailed specification and, thus, it is intended by the appended claims to cover all such features and advantages of the embodiments that fall within the true spirit and scope thereof. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the inventive embodiments to the exact construction and operation illustrated and described, and accordingly all suitable modifications and equivalents may be resorted to, falling within the scope thereof.