Patent Publication Number: US-10768856-B1

Title: Memory access for multiple circuit components

Description:
BACKGROUND 
     Artificial neural networks are computing systems with an architecture based on biological neural networks. Artificial neural networks can be trained, using training data, to learn about how to perform a certain computing task. 
     A neural network may include a set of processing nodes. Each processing node can perform computations on an input data element to generate an output data element, and the final decision can be generated based on a combination of the output data elements generated by the set of processing nodes. As part of the processing, each processing node can perform a set of arithmetic operations such as floating-point multiplications and additions to generate an intermediate output. Each processing node can also perform post-processing operations on the intermediate output to generate a final output. A neural network may be implemented by an integrated circuit with arithmetic circuitries and data paths to perform the arithmetic operations and post-processing operations, as well as memory devices to store the input data, intermediate outputs and final outputs to support the arithmetic and post-processing operations. Different components of the circuitries and data paths may access the memory devices to read the input data, to store and read the intermediate outputs, and to store the final outputs for the arithmetic operations and post-processing operations. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Various embodiments in accordance with the present disclosure will be described with reference to the drawings, in which: 
         FIG. 1  illustrates an example classifier device that uses techniques disclosed herein to process data; 
         FIGS. 2A-2F  are simplified block diagrams illustrating a prediction model and the operations that use techniques disclosed herein, according to certain aspects of the present disclosure; 
         FIGS. 3A-3F  are simplified block diagrams for some of the internal components of an apparatus for implementing the prediction model of  FIGS. 2A-2E , according to certain aspects of the present disclosure; 
         FIG. 4  illustrates an example flow diagram of performing multi-layer neural network processing of multiple sets of data, according to certain aspects of the present disclosure; and 
         FIG. 5  illustrates an example of a computing device, according to certain aspects of the disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     In the following description, various embodiments will be described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will also be apparent to one skilled in the art that the embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified in order not to obscure the embodiments being described. 
     Embodiments of the present disclosure relate to a computing system for performing neural-network processing of data. More specifically, the computing system may include an access engine to provide an interface between a memory device and one or more computation components. Each of the computation components may be configured to perform a sequential access operation of data (e.g., a sequence of write operations or read operations) at the memory device. The access engine can convert the sequential access operation into a single access operation at the memory device, to reduce the operation time of the memory device and to reduce the wait time to access the memory device for each computation component, which can reduce power consumption and improve the performance of the computing system. 
     An artificial neural network (herein after “neural network”) may include multiple processing nodes. The processing nodes can be divided into layers including, for example, an input layer, a number of intermediate layers (also known as hidden layers), and an output layer. Each processing node of one layer (e.g., an input layer, an intermediate layer, etc.) may receive a sequential stream of input data elements, multiply each input data element with a weight, compute a weighted sum of the input data elements, and forward the weighted sum to the next layer. The next layer may perform post-processing on the weighted sums. The post-processing may include a pooling operation to generate subsamples of the weighted sums to reduce the data size, applying an activation function to the subsamples, etc. to generate output data. The output layer may also compute a weighted sum of the output data, and generate a binary output (e.g., “yes” or “no”) based on whether the weighted sum of the output data exceeds a threshold. 
     As discussed above, a neural network may be implemented by data processing circuitries and a memory device. The data processing circuitries may include, for example, a systolic array to multiply the input data elements with weights and to accumulate the multiplication products to generate a set of weighted sums. A systolic array may include an array of data processing units (DPU) connected in a network. Each DPU may perform arithmetic operations for a neural network layer. Each row of DPUs may be configured to process one input data set comprising multiple input data elements, and each DPU of the row can be configured to perform arithmetic operations on the multiple input data elements sequentially by processing one input data element at a time. For example, each DPU may include sequential processing logics that operate based on a continuous clock signal. In one clock cycle, a DPU can process a first input data element of the input data set to generate a first output. In the next clock cycle, the same DPU can process a second input data element of the input data set to generate a second output. 
     The memory device can provide temporary storage for the input data sets, and the memory device can also be accessed sequentially in multiple access operations, to provide each row of DPUs with one input data element (from one input data set) per access operation. The sequential accesses of the memory device (e.g., to retrieve the input data sets, storing of intermediate output data sets, storing of final output data sets, etc.) may require operating the memory device continuously for a large number of clock cycles, which can lead to huge power consumption. The sequential access operations are also undesirable in an environment where the memory device is accessed by multiple circuit components. For example, in a case where the memory device is a single-port static random access memory (SRAM) device and only allows access by one circuit component at one time, each circuit component may need to wait for the sequential access operations by another circuit component to finish before accessing the memory device, which can lead to long wait time and reduced throughput of the neural network processing. Although a multi-port SRAM device can be provided to allow concurrent access by different circuit components, such arrangements can lead to substantial increase in the physical dimension of the memory device and chip area. 
     Embodiments of the present disclosure relate to a computing system for performing neural-network processing of data. More specifically, the computing system may include a state buffer comprising static random access memory (SRAM) devices and configured to store input data and final output data of computations for a neural network layer. The computing system may also include an array of computing elements and a set of post-processing circuitries. The array is configured to receive the input data sequentially, whereas the set of post-processing circuitries is configured to output the final output data sequentially. The computing system further includes a read access engine configured to perform a read operation in the state buffer to obtain the input data, store the input data obtained from the state buffer in a first local buffer, and transmit the input data from the first local buffer to the array sequentially. The computing system further includes a write access engine configured to receive the final output data sequentially from the post-processing circuitries, store the final output data in a second local buffer, and perform a write operation in the state buffer to store the final output data obtained from the second local buffer. 
     With embodiments of the present disclosure, read and write access engines can be provided as interfaces between the memory device and each of different processing circuitries (e.g., the array of computing elements, the post-processing circuitries, etc.) of a neural network processing system. The read and write access engines can provide sequential access (e.g., sequential reads or writes) of data at the memory device for the different components. Further, the read and write access engines can also perform a single read (or write) operation to acquire (or store) the data at the memory devices. The single read/write operation can be performed in a single clock cycle. Therefore, the durations of access to the memory device can be shortened. As an illustrative example, instead of operating the memory device for 16 consecutive clock cycles to read one input data element per clock cycle for the array of computing elements (to access a total of 16 input data elements), the memory device can be operated to read the 16 input data elements in a single clock cycle. In the remaining 15 clock cycles, the memory device can be either idle, or can be operated to perform a single write operation to store 16 output data elements for the post-processing circuitries (instead of storing one output data element per clock cycle for 16 consecutive clock cycles). With such arrangements, both the power consumption of the memory device as well as the wait time to access the memory device can be reduced. As a result, the performance of the computing system can be improved. 
       FIG. 1  illustrates an example classifier device  100  that uses techniques disclosed herein to process data. Classifier device  100  can be, for example, a computing device operating a software application  102  and a prediction model  103  to predict information included in a data sequence, and perform a pre-determined function based on the prediction. For example, classifier device  100  can be part of an image recognition service provided to identify certain objects (e.g., texts, a person, etc.) from an image. It is understood that the image recognition service is merely provided as an illustrative example, and that techniques disclosed herein can be used for other data processing applications including, for example, text-based data processing (e.g., processing of search queries), audio data processing, etc. 
     The image recognition service can be provided in a multi-tenant compute service system. The multi-tenant compute service system may typically include a plurality of servers that can host data and can be used by multiple clients or organizations to run instances, such as virtual machine instances or bare-metal instances (e.g., operating systems that run directly on the server hardware). In most cases, instances, such as bare-metal or virtual machine instances, in a multi-tenant compute service system, may be allocated to a client when the client needs them and decommissioned when they are no longer needed, such that the resources can be reallocated to other clients. In the present disclosure, the terms “tenant,” “client,” and “customer” may be used interchangeably, although such terms do not necessarily imply the existence of any particular business arrangement. The term “instance” may refer to, for example, an instance that is executed directly on server hardware or as a virtual machine. Different types of instances generally correspond to different hardware functions and/or arrangements of hardware (e.g., different amounts of available memory and/or processing hardware). In the example of  FIG. 1 , the multi-tenant compute service system may provide the image recognition service when the client needs it and decommissioned when it is no longer needed, such that the resources supporting the image recognition service (e.g., access to software application  102 , and the underlying hardware resources for processing software application  102 ) can be reallocated to other clients. 
     As shown in  FIG. 1 , software application  102  can receive pixel data of an image  104  from a user. Image  104  may include an array of pixels. Software application  102  can perform analysis on the pixel data, and predict one or more objects  106  depicted in image  104 . The analysis may include, for example, comparing the pixel data against a set of pre-determined image features. As to be discussed in more detail below, software application  102  may employ prediction model  103  to compute a set of scores based on the pixel data of image  104 . The set of scores may represent, for example, the likelihood of image  104  including the pre-determined image features. Software application  102  can then determine other information about the content of image  104  based on the scores. For example, based on the scores, software application  102  can determine that image  104  is an image of a panda. 
     Prediction model  103  can be in the form of an artificial neural network. The artificial neural network may include a plurality of processing nodes, with each processing node configured to process part of the input pixel data, or to further process the intermediate outputs from other processing nodes.  FIG. 2A  illustrates an example of prediction model  103  that uses techniques disclosed herein. In the example of  FIG. 2A , prediction model  103  may be a multi-layer neural network such as a deep neural network (DNN), a convolutional neural network (CNN), etc. Prediction model  203  may include an input layer  207 , a set of intermediate layers including intermediate layers  209  and  211 , and an output layer (not shown in  FIG. 2 ). 
     Layer  207  may process pixel data representing different portions of image  204 . For example, in the example of  FIG. 2A , layer  207  may process the pixel data of image  204 . Each processing node of layer  207  is assigned to receive a pixel value (e.g., x 0 , x 1 , x 2 , . . . x n ) corresponding to a pre-determined pixel within image  204 , and transmit one or more weights with the received pixel value to layer  209 . In a case where prediction model  203  is a DNN, each processing node of layer  207  can be assigned a set of weights defined based on a matrix W 1 . Each processing node of layer  207  can send the received pixel value and the assigned weights to each processing node of layer  209 . In a case where prediction model  103  is a CNN, groups of the processing nodes of layer  207  may share a set of weights, and each group may send the set of weights and the pixel values received by the group of processing nodes to a single processing node of layer  209 . 
     Layer  209  may process the scaled outputs from layer  207  to generate a set of intermediate outputs. For example, assuming processing node  210   a  of layer  209  is connected to n processing nodes in layer  207 , processing node  210   a  may generate a sum of the scaled outputs received from layer  207  based on the following equation: 
     
       
         
           
             
               
                 
                   
                     sum 
                     
                       210 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       a 
                     
                   
                   = 
                   
                     
                       ∑ 
                       
                         i 
                         = 
                         0 
                       
                       n 
                     
                     ⁢ 
                     
                       ( 
                       
                         W 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           1 
                           i 
                         
                         × 
                         
                           x 
                           i 
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   
                     Equation 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     1 
                   
                   ) 
                 
               
             
           
         
       
     
     Here, sum 210a  represents a sum generated by processing node  210   a . W 1   i ×x i  represents a scaling of a particular pixel value (e.g., x 0 ) with the associated weight (e.g., W 1   0 ) by a processing node of layer  207 . In a case where prediction model  103  is a DNN, each processing node of layer  209  may generate the sum based on the scaling of pixel values from each processing node of layer  207 , and then generate a sum (e.g., Sum 210a ) by summing the scaled pixel values. The sum may also represent a dot-product between an input vector comprising a number of elements (e.g., pixel values) and a weight vector (e.g., W 1 ). 
     On the other hand, in a case where prediction model is a 103 CNN, each processing node of layer  209  may generate the sum based on the scaling of pixel values from a group of processing nodes of layer  207 . The sum may represent a convolution result between a group of pixel values and a filter comprising the weight values.  FIG. 2B  illustrates an example of a convolution operation layer  209  may perform. In  FIG. 2B , filter  230  may include a two-dimensional array of weights. The weights in filter  230  may represent a spatial distribution of pixels for certain features to be detected from the image. The two-dimensional array may have a height of R rows and a width of S columns, and is typically smaller than an input image with a height of H pixels and a width of W pixels. Each weight may be mapped to a pixel in a rectangular block of pixel values with the same R rows and S columns. A processing node of layer  209  (e.g., processing node  210   a ) can receive, from a group of processing nodes of input layer  207 , a group  240  of pixel values corresponding to a first rectangular block of pixels from the input image, and generate a convolution output  242  based on a summation of multiplication results between each weight of filter  230  and each corresponding pixel in group  240  according to Equation 1, to generate a dot-product between a matrix represented by filter  230  and a matrix represented by group  240 . Another processing node of layer  209  can also receive, from another group of processing nodes of input layer  207 , a group  244  of pixel values corresponding to a second rectangular block of pixels from the input image, and generate a convolution output  246  based on a summation of multiplication results between each weight of filter  230  and each corresponding pixel in group  244  according to Equation 1, to generate a dot-product between the matrix of filter  230  and a matrix represented by group  240 . In some examples, each convolution output in  FIG. 2B  (e.g., convolution output  242 , convolution output  246 , etc.) can correspond to the output of a processing node of layer  209 . In some examples, the pixel data in the input image may be referred to as an input feature map to indicate that the pixels are processed by the same filter (or same sets of filters) corresponding to certain feature(s). The convolution outputs may be referred to as an output feature map to indicate that the output is the result of processing an input feature map with the filter. 
     As shown in  FIG. 2B , the convolution operations can be arranged in a sliding-window such that the second rectangular block overlaps, or is otherwise adjacent to, the first rectangular block in the input image. For example, in the example of  FIG. 2B , D may be a distance of stride (in pixel) of the sliding-window for each convolution operations, such that the block of pixels corresponding to group  244  may be situated at a distance D (in terms of pixels) from the block of pixels corresponding to group  240 , and the next block of pixels may also be situated at the same distance D from group  244 . Other processing nodes of layer  209  may also receive groups of pixels corresponding to other rectangular blocks and generate other intermediate outputs. The convolution outputs can be part of a convolution output array  280  with a height of E rows and a width of F columns. The array of convolution outputs can have a smaller height and a smaller width than the input image. Rectangular blocks of the convolution outputs can be further grouped, and convolution operations can be performed at layer  211  between the groups of convolution outputs and another set of filter weights to generate another set of convolution outputs. 
     In some examples, the convolution operations can be performed between multiple images and multiple filters. For example, referring to  FIG. 2C , a set of C filters  260  may correspond to a number (C) of images  270 , and convolution operations can be performed between each filter of the set of filters  260  and blocks of pixels on the corresponding image of images  270 . The convolution results for each filter-image pair can be summed to generate a convolution output as follows: 
     
       
         
           
             
               
                 
                   
                     O 
                     
                       e 
                       , 
                       f 
                     
                   
                   = 
                   
                     
                       ∑ 
                       
                         r 
                         = 
                         0 
                       
                       
                         R 
                         - 
                         1 
                       
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           0 
                         
                         
                           S 
                           - 
                           1 
                         
                       
                       ⁢ 
                       
                         
                           ∑ 
                           
                             c 
                             = 
                             0 
                           
                           
                             C 
                             - 
                             1 
                           
                         
                         ⁢ 
                         
                           
                             X 
                             
                               
                                 eD 
                                 + 
                                 r 
                               
                               , 
                               
                                 
                                   f 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   D 
                                 
                                 + 
                                 s 
                               
                             
                             c 
                           
                           × 
                           
                             W 
                             
                               r 
                               , 
                               s 
                             
                             c 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   
                     Equation 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     2 
                   
                   ) 
                 
               
             
           
         
       
     
     Here, the convolution operation involves the images (or pixel arrays). X c   eD+r,fD+s  may refer to the value of a pixel at an image of index c, within the number (C) of images  270 , with a horizontal pixel coordinate of eD+r and a vertical pixel coordinate of fD+s. D is the sliding-window stride distance, whereas e and f correspond to the location of the output in the convolution output array, which can also correspond to a particular sliding window. Further, r and s correspond to a particular location within the sliding window. A pixel at an (r,s) location and of an image of index c can also correspond to a weight W c   r,s  in a corresponding filter of the same index c at the same (r,s) location. Equation 2 indicates that to compute a convolution output O e,f , each pixel within a sliding window (indexed by (e,f)) may be multiplied with a corresponding weight W c   r,s . A partial sum of the multiplication products within each sliding window for each of the images within the image set can be computed. And then a sum of the partial sums for all images of the image set can be computed. 
     Moreover, in some examples, multiple sets of filters can be used to perform convolution operations with a set of images to generate a set of convolution output arrays, with each convolution output array corresponding to a set of filters. For example, the multiple sets of filters may correspond to multiple image features to be detected from the set of images, and each convolution output array corresponds to the detection results for each image feature from the set of images. For example, where M sets of filters are applied to C images to generate M convolution output arrays, Equation 2 can be updated as follows: 
     
       
         
           
             
               
                 
                   
                     O 
                     
                       e 
                       , 
                       f 
                     
                     m 
                   
                   = 
                   
                     
                       ∑ 
                       
                         r 
                         = 
                         0 
                       
                       
                         R 
                         - 
                         1 
                       
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           s 
                           = 
                           0 
                         
                         
                           S 
                           - 
                           1 
                         
                       
                       ⁢ 
                       
                         
                           ∑ 
                           
                             c 
                             = 
                             0 
                           
                           
                             C 
                             - 
                             1 
                           
                         
                         ⁢ 
                         
                           
                             X 
                             
                               
                                 eD 
                                 + 
                                 r 
                               
                               , 
                               
                                 
                                   f 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   D 
                                 
                                 + 
                                 s 
                               
                             
                             c 
                           
                           × 
                           
                             W 
                             
                               r 
                               , 
                               s 
                             
                             
                               c 
                               , 
                               m 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   
                     Equation 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     3 
                   
                   ) 
                 
               
             
           
         
       
     
     Here, convolution output O e,f   m  and weight W c,m   r,s  has an index m corresponding to one of the M sets of filters. 
       FIG. 2D  illustrates an example of C sets of input data sets (with C=3) to be convolved with M sets of filters (with M=2). Each set of input data corresponds to the entries of a pixel group. For example, each of pixel groups  282 ,  286 , and  290  may correspond to one input data set. Each of the M sets of filters includes a set of C filters which correspond to the C sets of input pixel arrays. In the example of  FIG. 2D , there are two filter sets where the first filter set comprises filter arrays  284   a ,  284   b , and  284   c  and the second filter set comprises filter arrays  288   a ,  288   b , and  288   c . The convolution operations generate M sets of output data sets, with each output data set corresponding to a convolution output array. In the example of  FIG. 2D , two convolution output arrays  294  and  296  are generated. Each convolution output array corresponds to convolving one set (of the M sets) of filters with the input pixel arrays. For example, first element O 0,0   0  of convolution output array  294  can be generated by a sum of a dot-product between pixel group  282  and filter array  284   a , a dot-product between pixel group  286  and filter array  284   b , and a dot-product between pixel group  290  and filter array  284   c.    
     Referring back to  FIG. 2A , one processing node of layer  209  may be configured to generate one convolution output array, and a set M of processing nodes of layer  209  can correspond to a set M of convolution output arrays. The processing node of layer  209  can also post-process each convolution output with a pooling operation followed by an activation function to generate a final output for layer  209 . As discussed above, a pooling operation may include generating subsamples of the convolution output to reduce the output data size, which can reduce the computation complexity and improve system performance, while trading-off on the processing accuracy. In some cases, the pooling operation can also be skipped to avoid the degradation in the processing accuracy. Reference is now made to  FIG. 2E , which illustrates examples of post-processing of the convolution output. In the examples of  FIG. 2E , the convolution output comprises an 4×4 array  294 . Array  294  can be divided into four partition arrays including arrays  294   a ,  294   b ,  294   c , and  294   d . Two alternative examples of pooling operations on the partitions are illustrated. In a maximum pooling (“max pooling” of  FIG. 2E ) operation, the maximum value of each partition array can be selected as a subsample representing each partition array. In the max pooling example of  FIG. 2E , a maximum value of 23 is selected from partition array  294   a , a maximum value of 9 is selected from partition array  294   b , a maximum value of 5 is selected from partition array  294   c , whereas a maximum value of 12 is selected from partition array  294   d . In the average pooling example (“avg pooling” of  FIG. 2E ), an average value of 10 can be computed from partition array  2394   a , an average value of 6 is computed rom partition array  294   b , an average value of 4 is computed from partition array  294   c , whereas an average value of 9 is computed from partition array  294   d . In both of the maximum pooling and average pooling examples of  FIG. 2E , the convolution output array has been downsized from an 4×4 array to a 2×2 array. 
     Following the pooling operation, the subsamples of the convolution output (or the convolution output if the pooling operation is skipped) can be processed using an activation function. The activation function may translate the convolution output (or subsamples) into a decision of whether to forward the convolution output (or subsamples) to upper layers. The generation of the decision can be analogous to the firing of an actual biological neuron. An example of an activation function can be a rectified linear unit (ReLu) defined according to the following equation:
 
ReLu( y )=max(0, y )  (Equation 4)
 
       FIG. 2F  illustrates another example of post-processing of the convolution output. In the example of  FIG. 2F , each element of the convolution output array  294  can be processed by an activation function (e.g., ReLu) to generate intermediate output array  298 . Array  298  can be divided into four partition arrays including arrays  298   a ,  298   b ,  298   c , and  298   d . Pooling operations (e.g., maximum pooling, average pooling, etc.) can then be performed on array  298  to generate the subsample outputs. 
     A processing node of layer  209  (e.g., processing node  210   a ) may process the convolution output subsamples with the ReLu function to generate intermediate outputs based on Equation 4. Layer  211  may further process the intermediate outputs from layer  209  by, for example performing additional convolution operations based on different sets of filters. The outputs from each processing node of layer  211  may be forwarded to other higher intermediate layers, or to an output layer (not shown in  FIG. 2A ). The output layer may form an output vector representing, for example, a probability that a certain image feature is included in image  104 , and/or a probability that image  104  includes an image of a panda. For example, the output vector may be compared against a reference vector associated with a nose object of a panda, or a reference vector associated with a panda. A decision about whether image  104  is an image of a panda can be determined based on the comparison result. 
       FIG. 3A  shows an apparatus  300  according to some embodiments of the present disclosure. Apparatus  300  may be part of a computer system, e.g., a host server. Apparatus  300  may be part of a multi-tenant compute service system and can communicate with a host device (not shown in  FIG. 3A ) to provide computing and memory resources for a computing service. For example, referring back to  FIG. 1 , apparatus  300  may provide computing and memory resources for computations with prediction model  103 . A host device can operate software application  202  and communicate with apparatus  300  to perform one or more image recognition tasks based on computations with prediction model  103 . 
     In the example of  FIG. 3A , apparatus  300  may include a neural network processor  302  coupled to memory  312 , a direct memory access (DMA) controller  316 , and a host interface  314  via an interconnect  318 . As to be discussed in more detail, neural network processor  302  can provide the computing resources to support the computations with prediction model  103 . Memory  312  may be configured to store the instructions, input data (e.g., pixel data of image  204 ) and the weights (e.g., the filter data) received from the host device. Memory  312  may also be configured to store the output of neural network processor  302  (e.g., one or more image recognition decisions on the input images) at memory  312 . Memory  312  may include any suitable memory, e.g., dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate DRAM (DDR DRAM), storage class memory (SCM), flash memory devices, etc. 
     DMA controller  316  may be configured to perform DMA operations to transfer data between neural network processor  302  and the host device. For example, as discussed above, the host device can store the instructions, input data, and the weights at memory  312 . The host device can provide the memory addresses for the stored instructions, data, and weights to neural network processor  302  (e.g., in the form of memory descriptors). Neural network processor  302  can then obtain the stored instructions, data, and weights based on the memory addresses provided by the host device. Neural network processor  302  can also store the results of computations (e.g., one or more image recognition decisions) at memory  312 , and provide the memory addresses for the stored results to the host device. 
     Host interface  314  may be configured to enable communication between the host device and neural network processor  302 . For example, host interface  314  may be configured to transmit the memory descriptors including the memory addresses of the stored data (e.g., input data, weights, results of computations, etc.) between the host device and neural network processor  302 . Host interface  314  may include, for example, a peripheral component interconnect express (PCIe) interface or any suitable interface for communicating with the host device. 
     Neural network processor  302  can provide the computing resources to support the neural network computations for prediction model  103 . In the example of  FIG. 3A , neural network processor  302  may be an integrated circuit, such as a system on chip (SoC), and can include a number of circuit components, a state buffer  322 , a computing engine  324 , an output buffer  326 , and a post-processor  328 . In addition, neural network processor  302  may also include a read access engine  336  and a write access engine  338  to provide read and write access to state buffer  322  for computing engine  342  and post-processor  328 , as to be discussed in detail below. 
     State buffer  322  may be configured to provide caching of data used for computations at computing engine  324 . The data cached at state buffer  322  may include, for example, the input data and weights obtained from memory  312 , output data of computing engine  324 , as well as output data of post-processor  328 . The caching can reduce the effect of memory access bottleneck (e.g., caused by the latencies at memory  312 , DMA controller  316 , interconnect  318 , etc.) on the performance of computing engine  324 . State buffer  322  can be an on-chip memory device and may include, for example, static random access memory (SRAM). State buffer  322  can also be partitioned based on the organization of computing engine  324 . For example, state buffer  322  can include multiple SRAM banks, with each bank configured to store input data and weights for a row of computing engine  324 . 
     Computing engine  324  may include a set of processing elements (PE) configured to perform one or more arithmetic operations involved in neural network computations. Computing engine  324  may include a two-dimensional array of processing elements arranged in multiple rows and columns similar to a systolic array. Reference is now made to  FIG. 3B , which illustrates an example of computing engine  324 . In the example of  FIG. 3B , computing engine  324  includes a 3×3 array with three PEs in each row and three PEs in each column. Each PE may include a row input bus  352 , a column input bus  354 , a column output bus  356 , and a row output bus  358 . A PE may receive inputs from a left PE of the same row (or from external circuitries) via row input bus  352 . The PE may also receive inputs from an above PE of the same column (or from external circuitries) via column input bus  354 . The PE may perform arithmetic operations based on the inputs, and transmit the result of the arithmetic operations to a lower PE of the same column below (or to external circuitries) via column output bus  356 . The PE may also forward the inputs to a right PE of the same row, via row output bus  358 . 
     Each row of computing engine  324  may process one input data set comprising multiple input data elements, whereas each column of computing engine  324  generates a weighted sum of input data elements of different input data sets. As an illustrative example, in a case where computing engine  324  is to process pixel groups  282 ,  286 , and  290  of  FIG. 2D , a first row may receive elements of pixel group  282 , a second row may receive elements of input pixel array  286 , and a third row may receive elements of input pixel array  290 . Each PE includes a multiplier and an adder to handle one input data element at a time. A PE may receive one input data element and a weight (e.g., from row input bus  352 ) and generate, using the multiplier, a multiplication product to represent a weighted input data element. Moreover, the PE also receives a partial weighted sum from the PE above. The partial weighted sum represents the weighted sum of input data elements of input data sets received by each row above that PE. The PE adds the weighted input data element to the partial weighted sum, and passes the updated partial weighted sum to the PE below, and the PEs at the third row can generate a weighted sum of input data elements received by the three rows. 
     The operations of each PE of computing engine  324  can be synchronized to a continuous clock signal to improve the interoperability between computing engine  324  and other components of neural network processor  302 . Each PE may include sequential logic circuitries (e.g., registers, state machines, etc.) to store input data, weights, and output data for the adder and multiplier circuitries, and to synchronize the flow of the data into and out of the circuitries. The sequential logic circuitries of each PE can be clocked by either the same continuous clock signal or a replica of the clock signal. For example, in a first clock cycle, PE  360   b  of the second row may receive a first input data element of pixel group  386  (e.g., X 0,0   1 ), as well as a partial sum comprising weighted first input data element of pixel group  382  (e.g., W 0,0   0,0 ×X 0,0   0 ) from PE  360   a  of the first row. Within the first clock cycle, PE  360   b  may multiply the input data element X 0,0   1  with weight W 0,0   1,0 , add the multiplication product to the partial sum to generate an updated partial sum of W 0,0   0,0 ×X 0,0   0 +W 0,0   1,0 ×X 0,0   1 , and store the updated first partial sum in the set of internal registers. In the second clock cycle, PE  360   b  may forward the updated first partial sum to PE  360   c  below. In the third clock cycle, PE  360   c  can output a partial sum of W 0,0   0,0 ×X 0,0   0 +W 0,0   1,0 ×X 0,0   1 +W 0,0   2,0 ×X 0,0   2 . 
     Each column of computing engine  324  may correspond to a different processing node of a neural network layer, and each column can apply a different set of weights to generate different weighted sums for different output data sets. For example, the first column may apply weights of filter arrays  284   a ,  284   b , and  284   c  of  FIG. 2D  to generate a weighted sum of W 0,0   0,0 ×X 0,0   0 +W 0,0   1,0 ×X 0,0   1 +W 0,0   2,0 ×X 0,0   2  for the computation of the first element (O 0,0   0 ) of convolution output array  294  of  FIG. 2D . The second column may apply weights of filter arrays  288   a ,  288   b , and  288   c  of  FIG. 2D  to generate a weighted sum of W 0,0   0,1 ×X 0,0   0 +W 0,0   1,1 ×X 0,0   1 +W 0,0   2,1 ×X 0,0   2  for the computation of the first element (O 0,0   1 ) of convolution output array  296  of  FIG. 2D . Moreover, each column can also operate sequentially to generate additional weighted sums. For example, the first column may generate a weighted sum of first input data elements of input pixel groups  282 ,  286 , and  290  (W 0,0   0,0 ×X 0,0   0 +W 0,0   1,0 ×X 0,0   1 +W 0,1   2,0 ×X 0,1   2 ) in a first pass. In a second pass, the first column may generate a weighted sum of second input data elements of input pixel groups  282 ,  286 , and  290  (W 0,1   0,0 ×X 0,1   0 +W 0,1   1,0 ×X 0,1   1 +W 0,1   2,0 ×X 0,1   2 ). The weighted sums can be accumulated at output buffer  326  to generate the first element (O 0,0   0 ) of convolution output array  294  according to Equation 3. The second column may also generate, in different passes, the weighted sums for the first element (O 0,0   1 ) of convolution output array  296  according to Equation 3. While each column may generate the weighted sum in sequential passes, the generation of the weighted sums by each column can be performed in parallel to improve the rate of output data generation as well as the throughput of the neural network processing. 
     Referring back to  FIG. 3A , post-processor  328  can be configured to perform post-processing on the elements of the convolution output arrays provided by output buffer  326  to generate final outputs for the neural network layer. In the example of  FIG. 3A , post-processor  328  includes a pooling engine  328   a  and an activation engine  328   b . Pooling engine  328   a  can perform, for example, maximum pooling, average pooling, etc., on the convolution output array elements to generate subsamples, and store the subsamples at state buffer  322 . Pooling engine  328   a  can also be controlled to skip the pooling operation and store each element at state buffer  322 . The subsamples (or each element) of the convolution output arrays can be stored at state buffer  322  as intermediate outputs of post-processor  328 . Activation engine  328   b  can retrieve the subsamples (or each element) of the convolution output arrays from state buffer  322 , and apply one or more activation functions (e.g., ReLu function) on the retrieved data, to generate the final output data. Activation engine  328   b  may include one or more lookup tables (e.g., in the form of multiplexer circuits) to implement the activation functions. 
     Both pooling engine  328   a  and activation engine  328   b  may be configured to retrieve and data (e.g., from state buffer  322 ) to perform the post-processing (e.g., pooling and activation function processing) in batches. A post-processing batch can start as soon as output buffer  326  generates a set of new convolution output array elements. Compared with in a case where the post-processing is not started until each element of the convolution output arrays is generated, batch processing can speed up the post-processing and reduces the storage space requirements at output buffer  326  and post-processor  328  to support the post-processing operations. For example, referring to the example of  FIG. 3E , pooling engine  328   a  may start a new batch processing whenever four new elements of a convolution output array are generated at output buffer  326  (for each column of computing engine  324 ). Pooling engine  328   a  may either perform a pooling operation (e.g., average pooling, maximum pooling, etc.), or perform no pooling operation, on the four new elements. Depending on whether a pooling operation is performed, pooling engine  328   a  may store one subsample of the four elements, or the four elements, at state buffer  322 . Pooling engine  328   a  then starts a new batch processing after another four new elements of the convolution output array are generated. 
     Activation engine  328   b  can also perform the activation function processing in batches. For example, after new subsamples (or the four new elements) are stored at state buffer  322 , activation engine  328   b  can retrieve the subsamples (or the four new elements) from state buffer  322  and apply the activation function processing to generate the final output data elements. The final output data elements can be stored as the input data for the next neural network layer computations at state buffer  322 . 
     Read access engine  336  can provide read access to state buffer  322  for a read access requester device including, for example, computing engine  324  and post-processor  328 . Moreover, write access engine  338  can provide write access to state buffer  322  for a write access requester device including, for example, post-processor  328 . Each of read access engine  336  and write access engine  338  can convert a sequential series of access operations (e.g., multiple read or write operations across multiple clock cycles) to a single access operation to reduce power and reduce wait latency, as discussed above. Each of read access engine  336  and write access engine  338  may be organized based on state buffer  322 . For example, each of read access engine  336  and write access engine  338  may include multiple sub-engines corresponding to multiple SRAM banks of state buffer  322 , with each sub-engine providing access to the corresponding SRAM bank. A sub-engine of read access engine  336  can convert a sequential series of read access operations to the corresponding SRAM bank for multiple data elements (e.g., by a row of computing engine  324  or by post-processor  328 ) to a single read access for the multiple data elements. A sub-engine of write engine  338  can also convert a sequential series of write accesses for storing multiple data elements at the corresponding SRAM bank (e.g., by post-processor  328 ) to a single write access for the multiple data elements. Moreover, a sub-engine, coupled with a SRAM bank comprising single-port SRAM devices, can create a memory with multiple read and write access ports. 
     Reference is now made to  FIG. 3C , which illustrates an example of a read access sub-engine  336   a  (of read access engine  336 ). Read access sub-engine  336   a  may include read access interfaces  370   a  and  370   b  for interfacing with, respectively, row  324   a  of computing engine  324  and post-processor  328 . Read access sub-engine  436   a  also includes a memory read access interface  372  for interfacing with SRAM bank  322   a . SRAM bank  322   a  may include single port SRAM devices and includes a single read data port RDATA[15:0]. Read access sub-engine  336   a  and SRAM bank  322   a  together can form memory  375  with multiple read data ports  377   a  and  377   b.    
     Each read access interface includes a read request processor (e.g., one of read request processors  374   a  or  374   b ) and a read data register (e.g., one of read data registers  376   a  or  376   b ). The read data register can be a shift-register that can shift out stored data (e.g., in parallel form) to form a sequential data stream. In some examples, the read request processor can receive a sequence of read requests from a read access requester device, with each read request for a data element. For example, read request processor  374   a  may receive a sequence of read requests from row  324   a  of computing engine  324 , with each read request for an input data element of an input data set (e.g., X 0,0   0 , X 0,1   0 , X 0,2   0 , etc.), to compute the weighted sums. Moreover, in a case where pooling engine  328   a  skips the pooling operation, read request processor  374   b  may receive a sequence of read requests from post-processor  328 , with each read request being for a data element of an convolution output array (e.g., O 0,0   0 , O 0,1   0 , O 0,2   0 , etc.), to perform post-processing (e.g., activation function processing, pooling operation, etc.). In some examples, the read request processor can accumulate a pre-determined number of read requests (and the corresponding read addresses), and then initiate a single read operation at state buffer  322  when the pre-determined number of read requests has been accumulated. The pre-determined number can be based on, for example, a size of a read data element requested by each read request, and a size of a read data element to be provided by SRAM bank  322   a  in a single read operation. For example, in a case where a data element for a read request has a size of 1 byte, and SRAM bank  322   a  can provide 16 bytes of data element for each read operation (e.g., limited by the output data bus width), the read request processor can accumulate up to 16 read requests and the corresponding read addresses. 
     In some examples, the read request processor may also receive a single read request from each read access requester device for multiple data elements, or from state buffer access controller  440  on behalf of each read access requester device. The read request may include a starting read address and indicate the pre-determined number of data elements (e.g., 16 input data elements, etc.) to be read from SRAM bank  322   a.    
     After accumulating the pre-determined number of read requests and read addresses (or receiving the single read request including a staring read address and a number of read data pieces to be acquired), the read request processor can transmit a read enable signal and the starting read address to memory read interface  372 , which can forward the read enable signal and the read address to SRAM bank  322   a  to perform a single read operation at SRAM bank  322   a . SRAM bank  322   a  can return the requested read data to memory read interface  372 , which can forward the requested read data to the read access interface. The requested read data can be stored in the read data register, which can then provide the requested read data sequentially back to the requester device. In the example of  FIG. 3C , read request processor  374   a  can accumulate 16 read requests and read addresses for one-byte input data elements B[0], B[1], B[2], . . . B[15] from row  324   a  of computing engine  324 , and transmit a read enable signal and the starting read addresses to SRAM bank  322   a  (via memory read access interface  372 ), which then returns a 16-byte data B[15:0], between times T 1  and T 2 . The 16-byte data B[15:0] can be stored at read data register  376   a . Moreover, read request processor  374   b  can accumulate 16 read requests and read addresses for one-byte convolution output array elements C[0], C[1], C[2], . . . C[15] from post-processor  328  (one of pooling engine  328   a  or activation engine  328   b ), and transmit a read enable signal and the starting read addresses to SRAM bank  322   a  (via memory read access interface  372 ), which then returns a 16-byte data C[15:0], between times T 3  and T 4 . Here, the convolution output array elements C[0], . . . C[15] and the input data elements B[0], . . . B[15] may correspond to different batches of computations, and the convolution output array elements C[0], . . . C[15] are not generated from input data elements B[0], . . . B[15]. Between times T 4  and T 5 , read data register  376   a  can transmit the input data elements sequentially (e.g., start with B[0], followed by B[1], B[2], etc.) back to row  324   a  of computing engine  324  via read data port  377   a . Simultaneously (or at different time period), read data register  376   b  can also transmit the convolution output data elements sequentially (e.g., start with C[0], followed by C[1], C[2], etc.) back to post-processor  328  via read data port  377   b . Such arrangements allow post-processor  328  and computing engine  324  to operate in parallel to process data for different batches of computations, which can reduce the total time needed to complete the different batches of the neural network computations. As a result, the performance of neural network processor  302  can be improved. Moreover, by reducing the access time of SRAM bank  322   a , power consumption due to memory access can be reduced as well. 
       FIG. 3D  provides an illustrative example of the operations of read access sub-engine  336   a . Memory read interface  372  may transmit a read enable signal for read request processor  354   a  (corresponding to row  324   a  of computing engine  324 ) at time T 1 , and receive a 16-byte data element B[15:0] (e.g., input data elements) from SRAM bank  322   a  in one clock cycle between times T 1  and T 2 . The input data elements can be stored at read data register  376   a . Between times T 3  and T 4 , read data register  356   a  can transmit  16  one-byte data elements (B[0], B[1], . . . B[15]) sequentially to row  324   a  of computing engine  324  in 16 consecutive clock cycles, with one input data element being transmitted for each clock cycle. Memory read interface  372  may also transmit a read enable signal for read request processor  354   b  (corresponding to activation engine  328   b ) at time T 5 , and receive a 16-byte data element C[15:0] (e.g., convolution array output elements generated by row  324   a  of computing engine  324  and output buffer  326 ) from SRAM bank  322   a  in one clock cycle between times T 5  and T 6 . The convolution array output elements can be stored at read data register  356   b . Between times T 6  and T 7 , read data register  356   b  can transmit  16  one-byte data elements C[0], C[1], . . . C[15] sequentially to activation engine  326   b  in 16 consecutive clock cycles, with one output element being transmitted for each clock cycle. 
     Reference is now made to  FIG. 3E , which illustrates an example of a write access sub-engine  338   a  (of write engine  338 ). Write access sub-engine  338   a  may include write access interfaces  380   a  and  380   b  for interfacing with post-processor  328  to accept multiple sets of data from post-processor  328  for storage at SRAM bank  322   a . Write access sub-engine  338   a  also includes a memory read access interface  382  for interfacing with SRAM bank  322   a . As discussed above, SRAM bank  322   a  may include single port SRAM devices. Here, SRAM bank  322   a  may also include a single write data port WDATA[15:0]. Write access sub-engine  338   a  and SRAM bank  322   a  together can form memory  375  with multiple write data ports  387   a  and  387   b.    
     Each write access interface includes a write request processor (e.g., one of write request processors  384   a  or  384   b ) and a write data register (e.g., one of write data registers  386   a  or  386   b ). The write data register can be a shift-register that can accept a sequential stream of data and store the data in parallel form. In some examples, the write request processor can receive a sequence of write requests and a corresponding sequence of write data elements from a write access requester device for storing the write data elements at state buffer  322 . For example, write request processor  364   a  may receive a sequence of write requests from post-processor  328  together with a corresponding sequence of data elements of an convolution output array (e.g., O 0,0   0 , O 0,1   0 , O 0,2   0 , etc.), or subsamples of these data elements, to be stored at state buffer  322 . Write request processor  384   b  may also receive a sequence of write requests from activation engine  328   b , and another corresponding sequence of data elements of the convolution output array (e.g., O 0,5   0 , O 0,6   0 , O 0,7   0 , etc.), or subsamples of these data elements, to be stored at state buffer  322 . 
     In some examples, the write request processor can accumulate a pre-determined number of write requests (and the corresponding write addresses and write data elements) for a write access requester device, and then initiate a single write operation at state buffer  322  when the pre-determined number of write requests has been accumulated. The pre-determined number can be based on, for example, a size of a write data element to be stored for each write request, and a size of a write data element to be stored into SRAM bank  322   a  in a single write operation. For example, in a case where a write data element for a write request has a size of 1 byte, and SRAM bank  322   a  can store 16 bytes of write data element for each write operation (e.g., limited by the input data bus width), the write request processor can buffer up to 16 write requests and the corresponding write addresses and write data elements. 
     In some examples, the write request processor may also receive a write request from each write access requester device, or from state buffer access controller  340  on behalf of each write access requester device. The write request processor may also receive, from state buffer access controller  340  and with the write request, the write data elements to be stored in SRAM bank  322   a.    
     After accumulating the pre-determined number of write data elements, the write request processor can signal memory write interface  382  to perform a single write operation at SRAM bank  322   a  to store the write data elements (e.g.,  16  convolution output array elements, etc.) at a first write address provided by the requester device. SRAM bank  322   a  can then store the write data elements. In the example of  FIG. 3E , write request processor  384   a  can accumulate 16 write requests, write addresses, and one-byte data elements D[0], D[1], D[2], . . . D[15] (received via write data port  387   a ) from post-processor  328 , and then transmit a write enable signal, the starting write address, and 16-byte data elements D[15:0] to SRAM bank  322   a  (via memory write access interface  382 ), which then stores the 16-byte data D[15:0]. Moreover, write request processor  384   b  can accumulate 16 write requests, write addresses, and one-byte data elements E[0], E[1], E[2], . . . E[15] (received via write data port  387   b ) from post-processor  328 , and then transmit a write enable signal, the starting write address, and 16-byte data elements E[15:0] to SRAM bank  322   a  (via memory read access interface  382 ), which then stores the 16-byte data E[15:0]. The data elements D[0], D[1], D[2], . . . D[15] and E[0], E[1], E[2], . . . E[15] may correspond to different columns of computing engine  324  and/or different processing nodes of a neural network layer, and can represent the results of pooling operations (e.g., by pooling engine  328   a ) or activation processing (e.g., by activation engine  328   b ). 
       FIG. 3F  provides an illustrative example of the operations of write sub-engine  438   a . Between times T 0  to T 1 , write request processor  384   a  may receive  16  one-byte data elements D[0], D[1], . . . D[15] and  16  one-byte data elements E[0], E[1], E[15] sequentially from post-processor  328  in 16 consecutive clock cycles. The data elements are to be stored at SRAM bank  322   a . Between times T 1  and T 2 , write request processor  384   a  may transmit a write enable signal to memory write interface  382 , which causes memory write interface  382  to perform a write operation to store a 16-byte data element (D[15:0]) at SRAM bank  322   a  in one clock cycle between times T 1  and T 2 . Between times T 3  to T 4 , write request processor  384   b  may transmit a write enable signal to memory write interface  382  at time T 4  to cause memory write interface  382  to perform a write operation to store the 16-byte data element (E[15:0]) at SRAM bank  322   a  in one clock cycle between times T 3  and T 4 . With the arrangements of  FIG. 3F , post-processor  328  can transmit a larger number of output data sets in parallel to SRAM bank  322   a , which allows post-processor  328  to generate more output data sets within a time period, which can reduce the total time needed to complete the different batches of the neural network computations. As a result, the performance of neural network processor  302  can be improved. Moreover, by reducing the access time of SRAM bank  322   a , power consumption due to memory access can be reduced as well. 
     In some examples, state buffer access controller  340  can schedule the performances of the read operations and write operations by, respectively, read access engine  336  and write access engine  338 . The scheduling can be based on data dependency. For example, based on a sequence of operations among state buffer  322 , pooling engine  328   a , and activation engine  328   b  to generate output data for a neural network layer computation, state buffer access controller  340  can schedule a read operation for computing engine  324  first (e.g., to obtain the input data sets for computations), followed by a write operation by pooling engine  328   a  to store the outputs (or the subsamples of the outputs) of computing engine  324 , followed by a read operation for activation engine  328   b  to obtain the outputs (or the subsamples of the outputs) of computing engine  324  stored by pooling engine  328   a , and followed by a write operation by activation engine  328   b  to store the results of the activation function processing. 
     On the other hand, in a case where there is no data dependency, state buffer access controller  340  can interleave read and write operations for different access requester devices. For example, state buffer access controller  340  can control read access sub-engine  336   a  to perform a read operation at state buffer  322  to obtain a new set of weights for the next neural network layer computations. That read operation can be scheduled to be performed between the read or write operations performed by post-processor  328 , based on an assumption that post-processor  328  does not update the weights. Further, state buffer access controller  340  can also control a write access engine to obtain the new weights from DMA controller  316  (and from memory  312 ) and perform a write operation to store the new weights at state buffer  322 . The write operation can also be scheduled to be performed between the read or write operations performed by post-processor  328 , based on an assumption that post-processor  328  does not update the new weights. 
       FIG. 4  illustrates an example flow diagram of a process  400  for operating an array of computing elements. Process  400  may be implemented by, for example, read access engine  336  and write access engine  338  to support neural network computations at an integrated circuit (e.g., neural network processor  302 ). The computations may include, for example, computations for a deep neural network (DNN), a convolutional neural network (CNN), etc. 
     At operation  402 , read access engine  336  may receive, from a read access requester device, a first request for receiving first data. The read access requester device may include, for example, computing engine  324 , activation function engine  328   b , etc. The first data may include, for example, pixel data, convolution output, etc. The first request may include multiple requests sequentially received over multiple clock cycles, with each of the multiple request being for reading a data element of the first data. The first request may also include (or being associated with) a read address. 
     At operation  404 , read access engine  336  may perform a read operation at a memory device (e.g., state buffer  322 ) to receive the first data based on the first request. Read access engine  336  may transmit a read enable signal and the read address to state buffer  322  to receive the first data in a single clock cycle. Read access engine  336  can store the first data in a shift register (e.g., read data register  376   a ). 
     At operation  406 , read access engine  336  can convert the first data into a first sequential data stream. The conversion can be performed by the shift register. As each data element of the first sequential data stream is being generated (e.g., by the shifting action of the shift register), the first sequential data stream can also be transmitted to the read access requester device over multiple clock cycles, with one data element being transmitted per clock cycle, at operation  408 . 
     At operation  410 , write access engine  338  may receive, from a write access requester device (e.g., pooling engine  328   a , activation function engine  328   b , etc.), a second request for storing second data. The second data to be stored may include, for example, output of pooling engine  328   a , output of activation function engine  328   b , etc. The second request may include multiple requests transmitted sequentially over multiple clock cycles. The second request may also include (or be associated with) a write address. 
     At operation  412 , write access engine  338  may receive, from the write access requester device, a second sequential data stream comprising the second data. The second sequential data stream may include data elements of the second data transmitted sequentially over multiple clock cycles, with one data element being received per clock cycle, at operation  412 . 
     At operation  414 , write access engine  338  may convert the second sequential data stream into the second data. The conversion can be performed by the shifting action of a shift register (e.g., write data register  386   a ). 
     At operation  416 , write access engine  338  may perform a write operation at the memory device (e.g., state buffer  322 ) to store the second data based on the second request. Write access engine  338  may transmit a write enable signal, the write address, as well as the second data converted at operation  416  to state buffer  322  to store the second data in a single clock cycle. 
       FIG. 5  illustrates an example of a computing device  500 . Functionality and/or several components of the computing device  500  may be used without limitation with other embodiments disclosed elsewhere in this disclosure, without limitations. A computing device  500  may perform computations to facilitate processing of a task. As an illustrative example, computing device  500  can be part of a server in a multi-tenant compute service system. Various hardware and software resources of computing device  500  (e.g., the hardware and software resources associated with provision of an image recognition service) can be allocated to a client upon request. 
     In one example, the computing device  500  may include processing logic  502 , a bus interface module  508 , memory  510 , and a network interface module  512 . These modules may be hardware modules, software modules, or a combination of hardware and software. In certain instances, modules may be interchangeably used with components or engines, without deviating from the scope of the disclosure. The computing device  500  may include additional modules, not illustrated here. In some implementations, the computing device  500  may include fewer modules. In some implementations, one or more of the modules may be combined into one module. One or more of the modules may be in communication with each other over a communication channel  514 . The communication channel  514  may include one or more busses, meshes, matrices, fabrics, a combination of these communication channels, or some other suitable communication channel. 
     The processing logic  702  may include one or more integrated circuits, which may include application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), systems-on-chip (SoCs), network processing units (NPUs), processors configured to execute instructions or any other circuitry configured to perform logical arithmetic and floating point operations. Examples of processors that may be included in the processing logic  702  may include processors developed by ARM®, MIPS®, AMD®, Intel®, Qualcomm®, and the like. In certain implementations, processors may include multiple processing cores, wherein each processing core may be configured to execute instructions independently of the other processing cores. Furthermore, in certain implementations, each processor or processing core may implement multiple processing threads executing instructions on the same processor or processing core, while maintaining logical separation between the multiple processing threads. Such processing threads executing on the processor or processing core may be exposed to software as separate logical processors or processing cores. In some implementations, multiple processors, processing cores or processing threads executing on the same core may share certain resources, such as for example busses, level 1 (L1) caches, and/or level 2 (L2) caches. The instructions executed by the processing logic  502  may be stored on a computer-readable storage medium, for example, in the form of a computer program. The computer-readable storage medium may be non-transitory. In some cases, the computer-readable medium may be part of the memory  510 . Processing logic  502  may also include hardware circuitries for performing artificial neural network computation including, for example, neural network processor  302 , etc. 
     The access to processing logic  502  can be granted to a client to provide the personal assistant service requested by the client. For example, computing device  500  may host a virtual machine, on which an image recognition software application can be executed. The image recognition software application, upon execution, may access processing logic  502  to predict, for example, an object included in an image. As another example, access to processing logic  502  can also be granted as part of bare-metal instance, in which an image recognition software application executing on a client device (e.g., a remote computer, a smart phone, etc.) can directly access processing logic  502  to perform the recognition of an image. 
     The memory  510  may include either volatile or non-volatile, or both volatile and non-volatile types of memory. The memory  510  may, for example, include random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, and/or some other suitable storage media. In some cases, some or all of the memory  510  may be internal to the computing device  500 , while in other cases some or all of the memory may be external to the computing device  500 . The memory  510  may store an operating system comprising executable instructions that, when executed by the processing logic  702 , provides the execution environment for executing instructions providing networking functionality for the computing device  500 . The memory  510  may also store, for example, software applications for performing artificial neural network computation. For example, memory  510  may store software routines related to the computations of equations above. In a case where processing logic  502  is in the form of FPGA, memory  510  may store netlists data representing various logic circuit components of processing logic  502 . 
     The bus interface module  508  may enable communication with external entities, such as a host device and/or other components in a computing system, over an external communication medium. The bus interface module  508  may include a physical interface for connecting to a cable, socket, port, or other connection to the external communication medium. The bus interface module  708  may further include hardware and/or software to manage incoming and outgoing transactions. The bus interface module  708  may implement a local bus protocol, such as Peripheral Component Interconnect (PCI) based protocols, Non-Volatile Memory Express (NVMe), Advanced Host Controller Interface (AHCI), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), Serial AT Attachment (SATA), Parallel ATA (PATA), some other standard bus protocol, or a proprietary bus protocol. The bus interface module  708  may include the physical layer for any of these bus protocols, including a connector, power management, and error handling, among other things. In some implementations, the computing device  500  may include multiple bus interface modules for communicating with multiple external entities. These multiple bus interface modules may implement the same local bus protocol, different local bus protocols, or a combination of the same and different bus protocols. 
     The network interface module  512  may include hardware and/or software for communicating with a network. This network interface module  512  may, for example, include physical connectors or physical ports for wired connection to a network, and/or antennas for wireless communication to a network. The network interface module  512  may further include hardware and/or software configured to implement a network protocol stack. The network interface module  512  may communicate with the network using a network protocol, such as for example TCP/IP, Infiniband, RoCE, Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless protocols, User Datagram Protocol (UDP), Asynchronous Transfer Mode (ATM), token ring, frame relay, High Level Data Link Control (HDLC), Fiber Distributed Data Interface (FDDI), and/or Point-to-Point Protocol (PPP), among others. In some implementations, the computing device  900  may include multiple network interface modules, each configured to communicate with a different network. For example, in these implementations, the computing device  500  may include a network interface module for communicating with a wired Ethernet network, a wireless 802.11 network, a cellular network, an Infiniband network, etc. In some embodiments, computing device  500  may receive a set of parameters, such as the aforementioned weight vectors for generation of forget gate factor, input factor, output factor, etc. from a server through network interface module  512 . 
     The various components and modules of the computing device  500 , described above, may be implemented as discrete components, as a System on a Chip (SoC), as an ASIC, as an NPU, as an FPGA, or any combination thereof. In some embodiments, the SoC or other component may be communicatively coupled to another computing system to provide various services such as traffic monitoring, traffic shaping, computing, etc. In some embodiments of the technology, the SoC or other component may include multiple subsystems as disclosed herein. 
     The modules described herein may be software modules, hardware modules or a suitable combination thereof. If the modules are software modules, the modules can be embodied on a non-transitory computer readable medium and processed by a processor in any of the computer systems described herein. It should be noted that the described processes and architectures can be performed either in real-time or in an asynchronous mode prior to any user interaction. The modules may be configured in the manner suggested in  FIG. 5  and/or functions described herein can be provided by one or more modules that exist as separate modules and/or module functions described herein can be spread over multiple modules. 
     The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereunto without departing from the broader spirit and scope of the disclosure as set forth in the claims. 
     Other variations are within the spirit of the present disclosure. Thus, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the disclosure to the specific form or forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the disclosure, as defined in the appended claims. 
     The use of the terms “a” and “an” and “the” and similar referents in the context of describing the disclosed embodiments (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms “comprising,” “having,” “including,” and “containing” are to be construed as open-ended terms (i.e., meaning “including, but not limited to,”) unless otherwise noted. The term “connected” is to be construed as partly or wholly contained within, attached to, or joined together, even if there is something intervening. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illuminate embodiments of the disclosure and does not pose a limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure. 
     Disjunctive language such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is intended to be understood within the context as used in general to present that an item, term, etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and/or Z). Thus, such disjunctive language is not generally intended to, and should not, imply that certain embodiments require at least one of X, at least one of Y, or at least one of Z to each be present. 
     Various embodiments of this disclosure are described herein, including the best mode known to the inventors for carrying out the disclosure. Variations of those embodiments may become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate and the inventors intend for the disclosure to be practiced otherwise than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.