Patent Publication Number: US-11042797-B2

Title: Accelerating parallel processing of data in a recurrent neural network

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit of U.S. Provisional Patent Application No. 62/789,863 filed Jan. 8, 2019, the entire contents of which are incorporated herein by reference. 
    
    
     BACKGROUND OF THE DISCLOSURE 
     The disclosure relates generally to artificial neural networks. Artificial neural networks (hereinafter, “neural networks”) are computing systems that have been inspired by biological neural networks such as in the human brain for learning to recognize and process information, perform tasks, etc. based on examples and information that have been previously received, or “seen”. Accordingly, the performance of neural networks may be enhanced by persisting information for continued use in future processes. One known model for persisting information is a recurrent neural network. A recurrent neural network includes feedback for passing information from one step of the neural network to the next. However, the ability to connect and use past information for present processes is limited in typical recurrent neural networks. For example, a typical recurrent neural network may not be able to learn long-term dependencies. Long-term dependencies generally refer to circumstances in which a relatively large gap exists between a time at which relevant information is established and a time at which the relevant information is needed for a process. Typical recurrent neural networks may not be able to learn how to use the past information if the gap becomes too large. 
     A Long Short-Term Memory neural network (referred to herein as an “LSTM”) is a kind of recurrent neural network that is capable of learning long-term dependencies. A typical LSTM and associated algorithm is described further below with respect to  FIG. 1 . LSTMs capture and leverage temporal behaviors in data by remembering features representing the state of past events. Unfortunately, this temporal dependence (e.g., dependence between processing frames of an input video) at a fine grain may limit the parallelism of LSTMs on parallel hardware that might enhance the speed of LSTMs. LSTMs are known for their lack of significant speedups on parallel hardware due to memory bandwidth bottlenecks and tight phase-dependencies that prevent trivially exploiting parallelism. Accelerating LSTMs remains increasingly important due to their application in a wide range of processes that change over time, for example, speech recognition and video processing. 
     In general, the LSTM algorithm, as it is carried out for inference (as opposed to training), simply performs a set of four vector-matrix multiplications, then a set of pointwise operations on the resulting vector outputs. For purposes of this disclosure, “pointwise operation” means an algebraic operation on a first set (such as a matrix) of function values with a set of corresponding function values in a second set. In machine-learning lingo, these four matrix-multiplications are referred to as gates, because their goal is to let only the correct/relevant information on to the outputs or next phase of the algorithm. At each step (‘i’,  FIG. 2 ) of the algorithm, there are neuron activation vectors C i  and H i  (also referred to as candidates and outputs, respectively, or as state vectors) which capture past state, and an input vector X i  which captures the current incoming information to process. 
     Two basic phases, C i+1  and H i+1 , are carried out to compute the next state vectors. First, X i  is concatenated with H i , and this combined vector is fed through a series of four independent vector-matrix multiplications. These represent four different responsibilities of memory: understanding inputs, forgetting old remembered values, computing new candidate information, and preparing outputs. After matrix multiplication, a series of simple pointwise operations are performed on the four output vectors, the result of which is C i+1  and H i+1 . Since these are pointwise operations, this phase is much shorter and less performance critical. The parallelization challenge is how to carry out the above computations in parallel, e.g., across a multicore system, without causing too much synchronization overhead. 
     Similar to LSTM, a Gated Recurrent Unit (GRU) neural network (referred to herein as a “GRU”) is another type of recurrent neural network that is more apt than a standard recurrent neural network for learning long-term dependencies. The main difference between a GRU and an LSTM is that an additional feedback loop is required for each iteration (or, “timestep”) of the algorithm in a GRU. GRUs suffer many of the processing challenges that are described above for LSTMs. 
     In view of at least the above considerations, a general architecture and technique for parallelizing the inference phase of LSTMs, GRUs, and similar recurrent neural networks is needed. 
     BRIEF DESCRIPTION OF THE EXEMPLARY EMBODIMENTS 
     An exemplary embodiment of a method of accelerating a neural network may include distributing from a first master core to each of a plurality of processing cores a same relative one or more columns of weight matrix data for each of a plurality of gates in the neural network and broadcasting a current input vector from the first master core to each of the processing cores. The processing cores may respectively process in parallel the same relative one or more columns of weight matrix data. 
     An exemplary embodiment of a processor configured for accelerating a recurrent neural network may include a first master core and a plurality of processing cores. The first master core may be configured for distributing to each respective processing core a current input vector and a same relative column of weight matrix data for each of a plurality of gates in the recurrent neural network. The plurality of processing cores may be configured for processing each column of weight matrix data in parallel, at each of the respective processing cores. 
     An exemplary embodiment of a system configured for accelerating a recurrent neural network may include a host system, a processor including a master core and a plurality of processing cores, a plurality of data vaults, and an interface for maintaining data communication between the processor and the host system. Each data vault may include a local cache and the master core or the processing core to which the local cache is assigned. Each vault may be assigned to a contiguous region of an addressable storage space of the host system, and the contiguous region of the addressable storage space for each vault is stored (at least in part) in the local cache of each of each vault. The host system may be configured for controlling a flow of data to and from each vault. The master core may be configured for distributing to each vault a same relative column of weight matrix data for each of a plurality of gates in the recurrent neural network, based at least in part on instructions from the host system. The plurality of processing cores may be configured for processing each column of weight matrix data in parallel, at each of the respective processing cores. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       A more particular description will be rendered by reference to specific embodiments thereof that are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments thereof and are not therefore to be considered to be limiting of its scope, exemplary embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings in which: 
         FIG. 1  illustrates an architecture and algorithm for a typical LSTM according to the prior art; 
         FIG. 2  illustrates an architecture and algorithm for an LSTM according to an exemplary embodiment; 
         FIG. 3  illustrates an exemplary distribution of weight matrices; 
         FIG. 4  illustrates an exemplary algorithm for the exemplary LSTM; 
         FIG. 5  illustrates an exemplary chip architecture for the exemplary LSTM; 
         FIG. 6  illustrates an architecture and algorithm for a GRU according to an exemplary embodiment. 
     
    
    
     Various features, aspects, and advantages of the embodiments will become more apparent from the following detailed description, along with the accompanying figures in which like numerals represent like components throughout the figures and text. The various described features are not necessarily drawn to scale but are drawn to emphasize specific features relevant to some embodiments. 
     The headings used herein are for organizational purposes only and are not meant to limit the scope of the description or the claims. To facilitate understanding, reference numerals have been used, where possible, to designate like elements common to the figures. 
     DETAILED DESCRIPTION 
     A typical LSTM architecture and algorithm  100  according to the prior art is shown in  FIG. 1 , which is taken from the website colah.github.io/posts/2015-08-Understanding-LSTMs/. With reference to  FIG. 1 , the LSTM algorithm  100 , as it is carried out for inference (as opposed to training) at a particular timestep  112  of the LSTM network  100 , performs a set of four vector-matrix operations  110   a ,  110   b ,  110   c ,  110   d  and then a set of pointwise operations  120   a ,  120   b ,  120   c ,  120   d ,  120   e  on the resulting vector outputs, as explained in additional detail, below. The four vector-matrix multiplications  110   a ,  110   b ,  110   c ,  110   d  are referred to as gates, because their goal is to let only the correct/relevant information on to the outputs or next phase of the algorithm. Each gate is associated with and/or includes a vector of values according to a particular operation of the neural network, and the four vectors together make up a weight matrix of the neural network. Certain inputs to the subject timestep  112  are provided by a candidate vector C t−1  and an output vector h t−1  that capture a past state from a previous timestep  111  of the LSTM network. An input vector X t  to the subject timestep  112  captures the current incoming information to process and is concatenated with the previous output vector h t−1 . The concatenated vector will serve as an input for the vector-matrix multiplication of each gate  110   a ,  110   b ,  110   c ,  110   d . The candidate vector C t−1 , also referred to as the cell state, persists through the LSTM neural network, including the previous timestep  111 , the subject timestep  112  (as C t ), and a future timestep  113 , and represents preserved information regarding past states and information that has been updated at each timestep  111 ,  112 ,  113  of the LSTM network. 
     With continuing reference to  FIG. 1  and the gates  110   a ,  110   b ,  110   c ,  110   d , each gate performs, alone or in part, four different responsibilities of memory in the LSTM network: 1) understanding inputs ( 110   a ); 2) forgetting old remembered values ( 110   a ); 3) computing new candidate information ( 110   b  and  110   c ); and, 4) preparing outputs ( 110   d ). The first gate  110   a  is both an “understand” and a “forget” gate that, first, evaluates the current inputs X t  and, second, determines what, if any, information in the cell state C t−1  should be modified based on changing conditions between the previous output h t−1  and the current inputs X t . The forget gate portion of  110   a  is a sigmoid function that outputs a number between 0 and 1 for each corresponding aspect represented as a number in the cell state C t−1 , where 0 represents “completely get rid of this” while 1 represents “completely keep this.” The cell state C t−1  is multiplied by the output from the forget gate  110   a  at the pointwise multiplication operation  120   a.    
     The second gate  110   b  and the third gate  110   c  together represent an “input” gate and respectively decide which values of the cell state C t−1  to update and create a vector of new candidate values that is added to the state. The second gate  110   b  is a sigmoid function that again outputs a value between 0 and 1, where, e.g., 0 represents “do not update” while 1 represents “update completely.” The third gate  110   c  is a hyperbolic tangent function (tan h) that outputs values between −1 and 1 representing each candidate value, and the results of the tan h function  110   c  are multiplied by the output from the sigmoid function  110   b , at the pointwise multiplication operation  120   b , to scale the degree to which each cell state value is updated. The result of the pointwise multiplication operation  120   b  is then added to the cell state C t−1  (as modified by the forget gate  110   a ) at pointwise addition operation  120   c.    
     Finally, the LSTM network timestep  112  at the fourth gate  110   d  determines what to output from the timestep  112 . First, a sigmoid function determines what parts of the cell state to output (at C t ) based on, for example, the cell state values that have been forgotten and/or updated and the likely functions that may be required by the future timestep  113  as a result. Then, the cell state C t−1  (as modified by the previous gates  110   a ,  110   b ,  110   c ) is put through the tan h pointwise function  120   d  to push the cell state values to between −1 and 1. The output of the tan h pointwise function  120   d  is multiplied by the output of the sigmoid function at pointwise multiplication operation  120   e  to create a filtered cell state as output h t  from the subject timestep  112 . The full cell state C t  is also output from the subject timestep  112  and each of the cell state C t  and the output h t  serve as state inputs to the future LSTM timestep  113  for similar processing. In addition, output h t  may be provided to a memory core that stores outputs representing past states of the LSTM network. 
     For purposes of illustrating features of the exemplary embodiments, an example will now be introduced and referenced throughout the disclosure. This example is illustrative and not limiting and is provided purely for explanatory purposes. 
     With reference to  FIG. 2 , an exemplary parallel LSTM network architecture and algorithm  200  according to the disclosure is shown. In the exemplary embodiment shown in  FIG. 2 , the LSTM  200  may be implemented as, for example and without limitation, a processing chip or integrated circuit, implemented on a host  210  which may be any known computing device consistent with this disclosure. The LSTM  200  includes, among other things, a master core  220  and a plurality of processing cores  211 ,  212 ,  213 ,  214 . The master core  220  may be a memory and broadcasting device for broadcasting a current input vector X i  and/or current state vectors C i , H i  to each of the processing cores  211 ,  212 ,  213 ,  214  (the current state vector H i  is computed in parallel during a previous timestep of the algorithm, as described below). In some embodiments, the processing cores may include highly parallel processing logic, such as a Coarse-Grained Reconfigurable Architecture (CGRA). In the same or other embodiments, the processing cores may include a Mixed-Grained Reconfigurable Architecture (MGRA) or other parallel processing logic consistent with this disclosure. In operation, the host system  210  may slice a weight matrix  300  ( FIG. 3 ) including four gate matrices  301 ,  302 ,  303 ,  304  ( FIG. 3 ) into subsets W 1 , W 2 , W 3 , W 4  that each include the same relative column from each of the gate matrices  301 ,  302 ,  303 ,  304 . Each of the subsets W 1 , W 2 , W 3 , W 4  may have a substantially equal number of columns. The host system  210  may then distribute, via the master core  220 , each subset W 1 , W 2 , W 3 , W 4  to a respective processing core  211 ,  212 ,  213 ,  214 . The master core  220  broadcasts a concatenated vector X i /H i  of the current input X i  and current state H i  to each of the processing cores  211 ,  212 ,  213 ,  214 . 
     With reference now to  FIG. 3 , an exemplary distribution of the weight matrix  300  is shown. For demonstration purposes, the input vectors are shown as row vectors. In the example shown in  FIG. 3 , each row r 1 , r 2 , . . . , r n  from a same relative one or more columns  301   c1 ,  302   c2 ,  303   c2 ,  304   c2  of each gate matrix  301 ,  302 ,  303 ,  304  is made available to the processing core  212  along with the concatenated vector X i /H i  and the current state vector as previously discussed. Each of the other processing cores  211 ,  213 ,  214  would similarly receive each row r 1 , r 2 , . . . , r n  from a corresponding one or more columns (a slice) of each gate matrix  301 ,  302 ,  303 ,  304 . In operation, each of the processing cores  211 ,  212 ,  213 ,  214  may then compute, in parallel, a subset of the matrix multiplication and pointwise operation(s) on each corresponding subset W 1 , W 2 , W 3 , W 4  of elements from each gate matrix  301 ,  302 ,  303 ,  304 . The parallel computation results in the final computation of fragments of the C i+1  vector, which are stored locally at each processing core, and the H i+1  vector, which are needed by the processing cores  211 ,  212 ,  213 ,  214  during the next timestep of the algorithm. Accordingly, and with reference back to  FIG. 2 , the H i+1  vector fragments H 1   i+1 , H 2   i+1 , H 3   i+1 , H 4   i+1  are written (i.e., uploaded) to the array controlled by the master core  220  for later broadcast. In some embodiments, the slice of columns loaded into a given processing core represents adjacent columns of the gate matrix, while in other embodiments the slice of columns represents non-adjacent columns. Further, in some embodiments the gate matrices are stored in memory in column-major order, while in other embodiments, the gate matrices are stored in row-major order. For example, with four processing cores  211 ,  212 ,  213 ,  214  and with the gate matrices stored in a row-major order, a first processing core  211  would receive the first (consecutive) ¼ th  of the columns, the second processing core  212  would receive the second (consecutive) ¼ th  of the columns, the third processing core  213  would receive the third (consecutive) ¼ th  of the columns, and the fourth processing core  214  would receive the fourth (consecutive) ¼ th  of the columns. 
     In the exemplary embodiments, the LSTM  200  is shown with four processing cores  211 ,  212 ,  213 ,  214 . In other embodiments, any number of processing cores consistent with this specification may exist, so long as each processing core  211 ,  212 ,  213 ,  214  (assuming processing cores with similar capabilities) is given a roughly equal number of columns from the weight matrix  300  for each gate  301 ,  302 ,  303 ,  304 . In other words, for M columns per gate and N processing cores, each processing core would receive roughly M/N columns. Distributing the same relative columns to each processing core  211 ,  212 ,  213 ,  214  allows the pointwise operations to be performed without requiring additional synchronization. 
     With reference now to  FIG. 4 , a more detailed illustration of the exemplary algorithm  400  is shown according to processing timesteps  411 ,  412 ,  413 . Before the timestep  411 , the weight matrix  300  of the LSTM  200  is sliced into subsets W 1 , W 2 , W 3 , W 4 , at step LW, and each subset W 1 , W 2 , W 3 , W 4  respectively of the weight matrix  300  is preloaded into a processing core (or, in some embodiments, a cache (or alternatively a simple buffer))  522 ,  523 ,  524 ,  525  ( FIG. 5 ) associated with the processing core, as explained below with respect to  FIG. 5 ). For purposes of this disclosure, a “cache” may refer to a scratchpad (local memory with programmer-controlled allocation) or a traditional cache (local memory with microarchitecture-controlled allocation). In the exemplary embodiment shown in  FIG. 5 , the set of inputs for an inference problem associated with the algorithm are copied in their entirety to a contiguous location  531 ,  532 ,  533 ,  534 ,  535  ( FIG. 5 ) corresponding to each processing core  211 ,  212 ,  213 ,  214 , as discussed with respect to  FIG. 5 . In various other embodiments, the weight matrix may be preloaded, in its entirety or in part, into the processing cores. In an exemplary aspect, the weight matrix is preloaded in its entirety. The weight allocation and loading step LW need only be performed once per inference problem, as weights are shared both across timesteps within the same inference and across subsequent inferences. The allocation and loading step LW is a performance-non-critical step that may be performed by the host  210 , as described below with respect to  FIG. 5 . 
     At a timestep  411 , the current input vector X i  is loaded into each processing core at step Lx 1 . In one aspect, X i  may be concatenated with H i−1  (not shown) from, e.g., a previous inference. In another aspect, X i  may be concatenated with a zero vector during an initial timestep. Further, as shown in  FIG. 4 , the input vector X i+1  for the next timestep can be preloaded, at step Lx 2 , from memory while the current state vectors C i  and H i  are being computed. According to this aspect, an exemplary chip  500  ( FIG. 5 ) according to the disclosure may include a double buffer at each cache  522 ,  523 ,  524 ,  525 —one buffer for the current input vector X i    522   a ,  523   a ,  524   a ,  525   a  ( FIG. 5 ) and one buffer for the “next” input vector X i+1    522   b ,  523   b ,  524   b ,  525   b  ( FIG. 5 ) for the next timestep. At the transition between timesteps  411 ,  412 ,  413  the “next”  522   b ,  523   b ,  524   b ,  525   b  and “current”  522   a ,  523   a ,  524   a ,  525   a  buffers are swapped for all processing cores. A barrier at each transition point may be used to synchronize all processing cores. In the exemplary embodiment, the next input vector X i+1  and the computed state vector H i  respectively are multicast from the master core  220  to each processing core, at steps Bx 2  and Bh 1 , which reduces processing overhead by obviating the need for each core to send a request to the master core  220 . In the exemplary embodiment, steps Lx 2  and Bx 2  (and corresponding Lx 3 , Bx 3  and Lx 4 , Bx 4 , as discussed below) may be considered as part of the same phase, because they operate in lockstep as the master core  220  performs a standard memory load of the next input vector X i+1  and then uses a broadcast (e.g., a cache that stores and forwards information) to broadcast the data into the next input buffer  522   b ,  523   b ,  524   b ,  525   b  of each processing core. 
     With continuing reference to  FIG. 4 , at the timestep  411 , processing at each core proceeds as described above with respect to  FIGS. 1-3 . The vector-matrix multiplications for each gate are performed at step VM 1  and the pointwise operations are performed at step PO 1 . The resulting computed state vector C i  is written to a local memory  410  and the computed output (state) vector H i  is written to a “next” output (state) buffer  220   a  of the master core  220 , for use in further timesteps. At step Bh 1 , the next output (state) vector H i  is multicast to the processing cores for use in the subsequent timestep. It will be understood that, depending on the capabilities of the processing cores, the steps may all be performed sequentially, or various of these steps may overlap in their execution. For example, step PO 1  may execute as outputs are produced from step VM 1 . Similarly, the processing cycle at the timestep  412  uses input vector X i+1  and output (state) vector H i  for performing the same processing steps, including vector-matrix multiplication at step VM 2  and pointwise operations at step PO 2 , while the next input vector X i+2  is loaded into the next input buffer at step Lx 3 . As previously described, the next input vector is multicast to all processing cores, as indicated by step Bx 3 . The resulting computed state vector C i+1  is written to the local memory  410  and the computed output (state) vector H i+1  is written to the next output (state) buffer  220   a  of the master core  220 . At step Bh 2 , the next output (state) vector H i+1  is multicast to the processing cores. At the timestep  413 , the process continues with the vector-matrix multiplication VM 3  and pointwise operations PO 3  based on vector input X i+2  and computed (output) state vector H i+1 . Input vector X i+3  is preloaded into the next input buffer at step Lx 4  and broadcast to the processing cores at step Bx 4 . Once again, the resulting computed state vector C i+2  is written to the local memory  410  and the computed output (state) vector H i+2  is written to the next output (state) buffer  220   a  of the master core  220  for broadcast to the processing cores. At each timestep, the master core  220  waits until it receives the partial outputs (H i ) from all cores before performing the broadcast. In some embodiments, the master core  220  may use a barrier to synchronize the operation. The process continues in this fashion for as many timesteps are required for the particular inference operation. 
     In further aspects of the exemplary algorithm depicted in  FIG. 4 , each processing core maintains its own input (X i ), state (C i ), and output (state) (H i ) vectors corresponding to its distributed weight columns. The host  210  allocates the per-core memory (as described below with respect to  FIG. 5 ) for such storage. While the exemplary embodiment includes a double buffer  522   a ,  523   a ,  524   a ,  525   a / 522   b ,  523   b ,  524   b ,  525   b  at every processing core cache  522 ,  523 ,  524 ,  525  for current X i  and next X i+1  input vectors, each cache needs only one buffer for output (state) H i  values because each processing core independently enforces the dependence between current and next values using barrier synchronization. Similarly, for each state vector C i , the dependence between timesteps is enforced naturally by the processing/computation, so each processing core needs only one buffer for its state vector C i . The master core  220 , on the other hand, receives the values for the next output (state) vector H i  while it is performing synchronization, so the master core  220  includes the next buffer  220   a  for the next output (state) vector H i . 
     With reference now to  FIG. 5 , an exemplary architecture for the chip  500  on which the LSTM  200  is implemented is shown. The chip includes, among other things, a master core  520 , a plurality of processing cores  511 ,  512 ,  513 ,  514 , a physical memory  510 , and a plurality of local caches  521 ,  522 ,  523 ,  524 ,  525 , wherein each local cache is respectively assigned to the master core  520  or a processing core  511 ,  512 ,  513 ,  514 . As previously discussed with respect to  FIG. 4 , the local cache  522 ,  523 ,  524 ,  525  of each processing core  511 ,  512 ,  513 ,  514  may be a double-buffered cache, and may include assigned storage locations for the current input vector X i  and the next input vector X i+1 . For example, local cache  522 ,  523 ,  524 ,  525  may include the current input buffer  522   a ,  523   a ,  524   a ,  525   a  for the current input vector X i  and the next input buffer  522   b ,  523   b ,  524   b ,  525   b  for the next input vector X i+1 . In the exemplary embodiment, the physical memory  510  of the chip maintains correspondence with a portion of a virtual address space  530  on the host  210 , wherein data transfers between the physical memory  510  and the virtual address space  530  on the host  210  are initiated by the host  210 . Further, the master core  520  and each of the processing cores  511 ,  512 ,  513 ,  514  is assigned a contiguous region  531 ,  532 ,  533 ,  534 ,  535  of the virtual address space  530 , and that region is stored on the corresponding local cache  521 ,  522 ,  523 ,  524 ,  525  of the master core  520  and each processing core  511 ,  512 ,  513 ,  514 . As such, the master core  520  and each processing core  511 ,  512 ,  513 ,  514  uses the data allocated to its particular address range  521 ,  522 ,  523 ,  524 ,  525 . The combination of a processing core and a local cache is referred to herein as a “vault”  550 ,  550   a ,  550   b ,  550   c ,  550   d  (for brevity for purposes of this disclosure, reference to a “vault  550 ” refers to all vaults that may be present, including, e.g.,  550 ,  550   a ,  550   b ,  550   c , and  550   d ). In certain embodiments, multiple processing cores may be assigned to the same vault  550 . 
     With continuing reference to  FIG. 5 , the exemplary chip  500  may be connected to a Central Processing Unit (CPU)  540  of the host. The CPU  540  may perform performance-non-critical tasks such as the allocation and loading step LW ( FIG. 4 ), coarse grain memory allocation among vaults, initialization and coordination of parallel tasks, and phases of the algorithm. The host CPU  540  is also responsible for pushing and pulling relevant data to the vaults  550 . Other components that the exemplary chip  500  may include are, without limitation, a broadcast mechanism for multicasting replicated data across processing elements of the chip  500 , a barrier mechanism for forcing synchronization of the parallel computing operations, one or more scratchpad memories or circuits for improving cache bandwidth by, e.g., storing weights (across batch elements) or inputs (across columns) for reuse, debugging facilities, and other known facilities, functions, and mechanisms consistent with this disclosure. 
     In further aspects of the exemplary embodiments for, e.g., handling relatively large inputs, the host  210  may stream inputs to the master core  220  in a manner that is synchronized with the progress of the exemplary algorithm and the demand for additional inputs. In the same or other embodiments, the exemplary LSTM  200  may include two or more master cores wherein, for example, the operations of synchronizing output (state) vectors H i  and broadcasting input vectors X i  are decoupled and handled by different master cores. The master core dedicated to broadcasting the inputs may also be rotated among processing cores  211 ,  212 ,  213 ,  214  throughout the course of the algorithm. 
     In additional aspects of the exemplary embodiment, certain techniques may be implemented to reduce broadcast latency and the impact on bandwidth across the chip  500 . In a first technique, if the inputs are small enough, data may be duplicated for one or more groups of processing cores and the inputs broadcast only to processing cores within a particular group. Such a technique is described in commonly-owned U.S. Patent Application No. 62/781,644 filed on Dec. 19, 2018, which is incorporated herein by reference in its entirety. In such a technique, trail-buffer simultaneous read-reuse mechanisms become relevant, which has the additional benefit of obviating a double-buffered cache  522 ,  523 ,  524 ,  525  for the current input vector X i  and the next input vector X i+1 . In the same or other embodiments, each processing core  211 ,  212 ,  213 ,  214  may directly broadcast its output H i  to all other processing cores using a prethrow interface. The prethrow interface technique may eliminate the step of writing the outputs H i  to the master core  220  and the barrier synchronization delay prior to broadcasting the outputs Hi at step Bh i . 
     With reference now to  FIG. 6 , one timestep  630  of an exemplary Gated Recurrent Unit (GRU)  600  according to the disclosure is shown in isolation. A GRU generally is similar to an LSTM, one difference being that for each timestep  630  of the GRU an additional feedback loop  605  including a sigmoid function at a first gate  610   a  and a pointwise multiplication  620   a  is required. Without the additional feedback loop  605 , the algorithm for the exemplary LSTM embodiments described above can be implemented directly. The GRU  600  also requires additional communication capacities for assembling and distributing the information r t  from the feedback loop  605 . The additional capacities may be a prethrow feature as discussed above. 
     With continuing reference to  FIG. 6 , the algorithm for one timestep  630  in the exemplary GRU  600  begins, similar to the timesteps  411 ,  412 ,  413 , by distributing a weight matrix to a plurality of processing cores (not shown). An input vector x t  and a state vector h t−1  are concatenated and the result is provided to a first gate  610   a  and a second gate  610   b  which are sigmoid functions. The information r t  from the first sigmoid function  610   a  is provided to the pointwise multiplication  620   a  along with the state vector h t−1 . The result of the pointwise multiplication  620   a  between the state vector h t−1  and the information r t  from the first sigmoid function  610   a  is then concatenated with the input vector x t  and the result is provided to a third gate  610   c  which is a tan h function. 
     The result z t  of the sigmoid function of the second gate  610   b  is input into a Rectified Linear Unit (ReLU)  620   b  that thresholds at zero the values of the vector z t , by replacing negative values with zero. The result of the ReLU is then provided as an input into a pointwise multiplication operation  620   c , along with state vector h t−1 . The result of the multiplication operation  620   c  is subsequently input into a pointwise addition operation  620   d , along with the result of a pointwise multiplication operation  620   e  on an output vector h t′  of the tan h function of the fourth gate  610   c.    
     The input to the tan h function of the fourth gate  610   c  is the concatenation of the input vector x t  and the feedback loop  605 . The output vector h t′  of the tan h function is input into the pointwise multiplication operation  620   e , along with the output vector z t  of the sigmoid function of the second gate  610   b . The result of the pointwise multiplication operation  620   e  is then provided to the pointwise addition operation  620   d , along with the state vector h t−1 . The result of the pointwise addition operation  620   d  is output as a partial state vector h t , thus ending the timestep  630 . 
     In another aspect of the exemplary GRU algorithm  600 , the respective sigmoid functions of the first gate  610   a  and the second gate  610   b  may be calculated simultaneously and the tan h function may be calculated after obtaining the result from the feedback loop  605 . In a further aspect of the exemplary GRU algorithm  600 , the feedback loop  605  and the sigmoid function of the second gate  610   b  ReLU  620   b  pointwise multiplication operation  620   c  path may proceed in parallel. 
     The present disclosure, in various embodiments, configurations and aspects, includes components, methods, processes, systems and/or apparatus substantially developed as depicted and described herein, including various embodiments, sub-combinations, and subsets thereof. Those of skill in the art will understand how to make and use the present disclosure after understanding the present disclosure. The present disclosure, in various embodiments, configurations and aspects, includes providing devices and processes in the absence of items not depicted and/or described herein or in various embodiments, configurations, or aspects hereof, including in the absence of such items as may have been used in previous devices or processes, e.g., for improving performance, achieving ease and/or reducing cost of implementation. 
     The phrases “at least one”, “one or more”, and “and/or” are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B and C”, “at least one of A, B, or C”, “one or more of A, B, and C”, “one or more of A, B, or C” and “A, B, and/or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together. 
     In this specification and the claims that follow, reference will be made to a number of terms that have the following meanings. The terms “a” (or “an”) and “the” refer to one or more of that entity, thereby including plural referents unless the context clearly dictates otherwise. As such, the terms “a” (or “an”), “one or more” and “at least one” can be used interchangeably herein. Furthermore, references to “one embodiment”, “some embodiments”, “an embodiment” and the like are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. Approximating language, as used herein throughout the specification and claims, may be applied to modify any quantitative representation that could permissibly vary without resulting in a change in the basic function to which it is related. Accordingly, a value modified by a term such as “about” is not to be limited to the precise value specified. In some instances, the approximating language may correspond to the precision of an instrument for measuring the value. Terms such as “first,” “second,” “upper,” “lower” etc. are used to identify one element from another, and unless otherwise specified are not meant to refer to a particular order or number of elements. 
     As used herein, the terms “may” and “may be” indicate a possibility of an occurrence within a set of circumstances; a possession of a specified property, characteristic or function; and/or qualify another verb by expressing one or more of an ability, capability, or possibility associated with the qualified verb. Accordingly, usage of “may” and “may be” indicates that a modified term is apparently appropriate, capable, or suitable for an indicated capacity, function, or usage, while taking into account that in some circumstances the modified term may sometimes not be appropriate, capable, or suitable. For example, in some circumstances an event or capacity can be expected, while in other circumstances the event or capacity cannot occur—this distinction is captured by the terms “may” and “may be.” 
     As used in the claims, the word “comprises” and its grammatical variants logically also subtend and include phrases of varying and differing extent such as for example, but not limited thereto, “consisting essentially of” and “consisting of.” Where necessary, ranges have been supplied, and those ranges are inclusive of all sub-ranges therebetween. It is to be expected that variations in these ranges will suggest themselves to a practitioner having ordinary skill in the art and, where not already dedicated to the public, the appended claims should cover those variations. 
     The terms “determine”, “calculate” and “compute,” and variations thereof, as used herein, are used interchangeably and include any type of methodology, process, mathematical operation or technique. 
     The foregoing discussion of the present disclosure has been presented for purposes of illustration and description. The foregoing is not intended to limit the present disclosure to the form or forms disclosed herein. In the foregoing Detailed Description for example, various features of the present disclosure are grouped together in one or more embodiments, configurations, or aspects for the purpose of streamlining the disclosure. The features of the embodiments, configurations, or aspects of the present disclosure may be combined in alternate embodiments, configurations, or aspects other than those discussed above. This method of disclosure is not to be interpreted as reflecting an intention that the present disclosure requires more features than are expressly recited in each claim. Rather, as the following claims reflect, the claimed features lie in less than all features of a single foregoing disclosed embodiment, configuration, or aspect. Thus, the following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of the present disclosure. 
     Advances in science and technology may make equivalents and substitutions possible that are not now contemplated by reason of the imprecision of language; these variations should be covered by the appended claims. This written description uses examples to disclose the method, machine and computer-readable medium, including the best mode, and also to enable any person of ordinary skill in the art to practice these, including making and using any devices or systems and performing any incorporated methods. The patentable scope thereof is defined by the claims, and may include other examples that occur to those of ordinary skill in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal language of the claims.