Patent Publication Number: US-2023153621-A1

Title: Arithmetic unit for deep learning acceleration

Description:
BACKGROUND 
     Technical Field 
     The present disclosure generally relates to structures that support and advance deep machine learning, for example in convolutional neural networks (CNN). More particularly, but not exclusively, the present disclosure relates to an arithmetic unit for a deep learning acceleration engine. 
     Description of the Related Art 
     Known computer vision, speech recognition, and signal processing applications benefit from the use of learning machines. Learning machines discussed in this disclosure may fall under the technological titles of machine learning, artificial intelligence, neural networks, probabilistic inference engines, accelerators, and the like. Such machines are arranged to quickly perform hundreds, thousands, and millions of concurrent operations. Conventional learning machines can deliver hundreds of TeraFlops (i.e., one million millions (10 12 ) floating-point operations per second) of computing power. 
     In some cases, learning machines are organized as deep convolutional neural networks (DCNN). A seminal work in the DCNN arts is “Gradient-Based Learning Applied To Document Recognition,” by Y. LeCun et al., Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998, which led to winning the 2012 ImageNet Large Scale Visual Recognition Challenge with “AlexNet.” AlexNet, as described in “ImageNet Classification With Deep Convolutional Neural Networks,” by Krizhevsky, A., Sutskever, I., and Hinton, G., NIPS, pp. 1-9, Lake Tahoe, Nev. (2012), is a DCNN that significantly outperformed classical approaches for the first time. 
     A DCNN is a computer-based tool that processes large quantities of data and adaptively “learns” by conflating proximally related features within the data, making broad predictions about the data, and refining the predictions based on reliable conclusions and new conflations. The DCNN is arranged in a plurality of “layers,” and different types of predictions are made at each layer. 
     For example, if a plurality of two-dimensional pictures of faces is provided as input to a DCNN, the DCNN will learn a variety of characteristics of faces such as edges, curves, angles, dots, color contrasts, bright spots, dark spots, etc. These one or more features are learned at one or more first layers of the DCNN. Then, in one or more second layers, the DCNN will learn a variety of recognizable features of faces such as eyes, eyebrows, foreheads, hair, noses, mouths, cheeks, etc.; each of which is distinguishable from all of the other features. That is, the DCNN learns to recognize and distinguish an eye from an eyebrow or any other facial feature. In one or more third and then subsequent layers, the DCNN learns entire faces and higher order characteristics such as race, gender, age, emotional state, etc. The DCNN is even taught in some cases to recognize the specific identity of a person. For example, a random image can be identified as a face, and the face can be recognized as Orlando Bloom, Andrea Bocelli, or some other identity. 
     In other examples, a DCNN can be provided with a plurality of pictures of animals, and the DCNN can be taught to identify lions, tigers, and bears; a DCNN can be provided with a plurality of pictures of automobiles, and the DCNN can be taught to identify and distinguish different types of vehicles; and many other DCNNs can also be formed. DCNNs can be used to learn word patterns in sentences, to identify music, analyze individual shopping patterns, play video games, create traffic routes, and DCNNs can be used for many other learning-based tasks too. 
       FIGS.  1 A- 1 J  may collectively be referred to herein as  FIG.  1   . 
       FIG.  1 A  is a simplified illustration of a convolutional neural network (CNN) system  10 . In the CNN system, a two-dimensional array of pixels is processed by the CNN. The CNN analyzes a 10×10 input object plane to determine if a “1” is represented in the plane, if a “0” is represented in the plane, or if neither a “1” nor a “0” is implemented in the plane. 
     In the 10×10 input object plane, each pixel is either illuminated or not illuminated. For the sake of simplicity in illustration, illuminated pixels are filled in (e.g., dark color) and unilluminated pixels are not filled in (e.g., light color). 
       FIG.  1 B  illustrates the CNN system  10  of  FIG.  1 A  determining that a first pixel pattern illustrates a “1” and that a second pixel pattern illustrates a “0.” In the real world, however, images do not always align cleanly as illustrated in  FIG.  1 B . 
     In  FIG.  1 C , several variations of different forms of ones and zeroes are shown. In these images, the average human viewer would easily recognize that the particular numeral is translated or scaled, but the viewer would also correctly determine if the image represented a “1” or a “0.” Along these lines, without conscious thought, the human viewer looks beyond image rotation, various weighting of numerals, sizing of numerals, shifting, inversion, overlapping, fragmentation, multiple numerals in the same image, and other such characteristics. Programmatically, however, in traditional computing systems, such analysis is very difficult. A variety of image matching techniques are known, but this type of analysis quickly overwhelms the available computational resources even with very small image sizes. In contrast, however, a CNN system  10  can correctly identify ones, zeroes, both ones and zeroes, or neither a one nor a zero in each processed image with an acceptable degree of accuracy even if the CNN system  10  has never previously “seen” the exact image. 
       FIG.  1 D  represents a CNN operation that analyzes (e.g., mathematically combines) portions of an unknown image with corresponding portions of a known image. For example, a 3-pixel portion of the left-side, unknown image B5-C6-D7 may be recognized as matching a corresponding 3-pixel portion of the right-side, known image C7-D8-E9. In these and other cases, a variety of other corresponding pixel arrangements may also be recognized. Some other correspondences are illustrated in Table 1. 
     
       
         
           
               
             
               
                 TABLE 1 
               
             
            
               
                   
               
               
                 Corresponding known to unknown  
               
               
                 images segments 
               
            
           
           
               
               
               
            
               
                   
                 FIG. 1D 
                 FIG. 1D 
               
               
                   
                 Left-side, unknown image 
                 Right-side, known image 
               
               
                   
                   
               
               
                   
                 C3-B4-B5 
                 D3-C4-C5 
               
               
                   
                 C6-D7-E7-F7-G6 
                 D8-E9-F9-G9-H8 
               
               
                   
                 E1-F2 
                 G2-H3 
               
               
                   
                 G2-H3-H4-H5 
                 H3-I4-I5-I6 
               
               
                   
                   
               
            
           
         
       
     
     Recognizing that segments or portions of a known image may be matched to corresponding segments or portions of an unknown image, it is further recognized that by unifying the portion matching operation, entire images may be processed in the exact same way while achieving previously uncalculated results. Stated differently, a particular portion size may be selected, and a known image may then be analyzed portion-by-portion. When a pattern within any given portion of a known image is mathematically combined with a similarly sized portion of an unknown image, information is generated that represents the similarity between the portions. 
       FIG.  1 E  illustrates six portions of the right-side, known image of  FIG.  1 D . Each portion, also called a “kernel,” is arranged as a 3-pixel-by-3-pixel array. Computationally, pixels that are illuminated are represented mathematically as a positive “1” (i.e., +1); and pixels that are not illuminated are represented mathematically as a negative “1” (i.e., −1). For the sake of simplifying the illustration in  FIG.  1 E , each illustrated kernel is also shown with the column and row reference of  FIG.  1 D . 
     The six kernels shown in  FIG.  1 E  are representative and selected for ease of understanding the operations of CNN system  10 . It is clear that a known image can be represented with a finite set of overlapping or non-overlapping kernels. For example, considering a 3-pixel-by-3-pixel kernel size and a system of overlapping kernels having a stride of one (1), each 10×10 pixel image may have 64 corresponding kernels. 
     A first kernel spans the 9 pixels in columns A, B, C, and rows 1, 2, 3. 
     A second kernel spans the 9 pixels in columns B, C, D, and rows 1, 2, 3. 
     A third kernel spans the 9 pixels in columns C, D, E, and rows 1, 2, 3. 
     This pattern of kernels continues until an eighth kernel spans the 9 pixels in columns H, I, J, and rows 1, 2, 3. 
     Kernel alignment continues in this way until a 57 th  kernel spans columns A, B, C, and rows 8, 9, 10, and so on until a 64 th  kernel spans columns H, I, J, and rows 8, 9, 10. 
     In other CNN systems, kernels may be overlapping or not overlapping, and kernels may have strides of 2, 3, or some other number. The different strategies for selecting kernel sizes, strides, positions, and the like are chosen by a CNN system designer based on past results, analytical study, or in some other way. 
     Returning to the example of  FIGS.  1 D and  1 E , a total of 64 kernels are formed using information in the known image. The first kernel starts with the upper-most, left-most 9 pixels in a 3×3 array. The next seven kernels are sequentially shifted right by one column each. The ninth kernel returns back to the first three columns and drops down to the second row, similar to the carriage return operation of a text-based document, which concept is derived from a twentieth-century manual typewriter. In following this pattern, 
       FIG.  1 E  shows the 7 th , 18 th , 24 th , 32 nd , 60 th , and 62 nd  kernels of the 10×10 image in  FIG.  1 D (b). 
     Sequentially, or in some other known pattern, each kernel of the known image is aligned with a correspondingly sized set of pixels of the image under analysis. In a fully analyzed system, for example, the first kernel of the known image is conceptually overlayed on the unknown image in each of the kernel positions. Considering  FIGS.  1 D and  1 E , the first kernel is conceptually overlayed on the unknown image in the position of Kernel No. 1 (left-most, top-most portion of the image), then the first kernel is conceptually overlayed on the unknown image in the position of Kernel No. 2, and so on, until the first kernel is conceptually overlayed on the unknown image in the position of Kernel No. 64 (bottom-most, right-most portion of the image). The procedure is repeated for each of the 64 kernels, and a total of 4096 operations are performed (i.e., 64 kernels in each of 64 positions). In this way, it is also shown that when other CNN systems select different kernel sizes, different strides, and different patterns of conceptual overlay, then the number of operations will change. 
     In the CNN system  10 , the conceptual overlay of each kernel on each portion of an unknown image under analysis is carried out as a mathematical process called convolution. Each of the nine pixels in a kernel is given a value of positive “1” (+1) or negative “1” (−1) based on whether the pixel is illuminated or unilluminated, and when the kernel is overlayed on the portion of the image under analysis, the value of each pixel in the kernel is multiplied by the value of the corresponding pixel in the image. Since each pixel has a value of +1 (i.e., illuminated) or −1 (i.e., unilluminated), the multiplication will always result in either a +1 or a −1. Additionally, since each of the 4096 kernel operations is processed using a 9-pixel kernel, a total of 36,864 mathematical operations (i.e., 9×4096) are performed at this first stage of a single unknown image analysis in a very simple CNN. Clearly, even simple CNN systems require tremendous computational resources, and the computational requirements for more complex CNN systems grow exponentially. 
     As just described, each of the 9 pixels in a kernel is multiplied by a corresponding pixel in the image under analysis. An unilluminated pixel (−1) in the kernel, when multiplied by an unilluminated pixel (−1) in the subject unknown image will result in a +1 indicated a “match” at that pixel position (i.e., both the kernel and the image have an unilluminated pixel). Similarly, an illuminated pixel (+1) in the kernel multiplied by an illuminated pixel (+1) in the unknown image also results in a match (+1). On the other hand, when an unilluminated pixel (−1) in the kernel is multiplied by an illuminated pixel (+1) in the image, the result indicates no match (−1) at that pixel position. And when an illuminated pixel (+1) in the kernel is multiplied by an unilluminated pixel (−1) in the image, the result also indicates no match (−1) at that pixel position. 
     After the nine multiplication operations of a single kernel are performed, the product results will include nine values; each of the nine values being either a positive one (+1) or a negative one (−1). If each pixel in the kernel matches each pixel in the corresponding portion of the unknown image, then the product result will include nine positive one (+1) values. Alternatively, if one or more pixels in the kernel do not match a corresponding pixel in the portion of the unknown image under analysis, then the product result will have at least some negative one (−1) values. If every pixel in the kernel fails to match the corresponding pixel in the corresponding portion of the unknown image under analysis, then the product result will include nine negative one (−1) values. Considering the mathematical combination (i.e., the multiplication operations) of pixels, it is recognized that the number of positive one (+1) values and the number of negative one (−1) values in a product result represents the degree to which the feature in the kernel matches the portion of the image where the kernel is conceptually overlayed. Thus, by summing all of the products (e.g., summing the nine values) and dividing by the number of pixels (e.g., nine), a single “quality value” is determined. The quality value represents the degree of match between the kernel and the portion of the unknown image under analysis. The quality value can range from negative one (−1) when no kernel pixels match and positive one (+1) when every pixel in the kernel has the same illuminated/unilluminated status as its corresponding pixel in the unknown image. 
     The acts described herein with respect to  FIG.  1 E  may also collectively be referred to as a first convolutional process in an operation called “filtering.” In a filter operation, a particular portion of interest in a known image is searched for in an unknown image. The purpose of the filter is to identify if and where the feature of interest is found in the unknown image with a corresponding prediction of likelihood. 
       FIG.  1 F  illustrates twelve acts of convolution in a filtering process.  FIG.  1 G  shows the results of the twelve convolutional acts of  FIG.  1 F . In each act, a different portion of the unknown image is processed with a selected kernel. The selected kernel may be recognized as the twelfth kernel in the representative numeral one (“1”) of  FIG.  1 B . The representative “1” is formed in  FIG.  1 B  as a set of illuminated pixels in a 10-pixel-by-10-pixel image. Starting in the top-most, left-most corner, the first kernel covers a 3-pixel-by-3-pixel portion. The second through eighth kernels sequentially move one column rightward. In the manner of a carriage return, the ninth kernel begins in the second row, left-most column. Kernels 10-16 sequentially move one column rightward for each kernel. Kernels 17-64 may be similarly formed such that each feature of the numeral “1” in  FIG.  1 B  is represented in at least one kernel. 
     In  FIG.  1 F (a), a selected kernel of 3-pixels by 3-pixels is conceptually overlayed on a left-most, top-most section of an unknown image. The selected kernel in this case is the twelfth kernel of the numeral “1” of  FIG.  1 B . The unknown image in  FIG.  1 F (a) may appear to a human observer as a shifted, poorly formed numeral one (i.e., “1”). In the convolutional process, the value of each pixel in the selected kernel, which is “+1” for illuminated pixels and “−1” for unilluminated pixels, is multiplied by each corresponding pixel in the unknown image. In  FIG.  1 F (a), five kernel pixels are illuminated, and four kernel pixels are unilluminated. Every pixel in the unknown image is unilluminated. Accordingly, when all nine multiplications are performed, five products are calculated to be “−1,” and four products are calculated to be “+1.” The nine products are summed, and the resulting value of “−1” is divided by nine. For this reason, the corresponding image of  FIG.  1 G (a) shows a resulting kernel value of “−0.11” for the kernel in the left-most, top-most section of the unknown image. 
     In  FIGS.  1 F (b),  1 F(c), and  1 F(d), the kernel pixel is sequentially moved rightward across the columns of the image. Since each pixel in the area of the first six columns and first three rows spanning the first six columns is also unilluminated,  FIGS.  1 G (b),  1 G(c), and  1 G(d) each show a calculated kernel value of “−0.11.” 
       FIGS.  1 F (e) and  1 G(e) show a different calculated kernel value from the earlier calculated kernel values of “−0.11.” In  FIG.  1 F (e), one of the illuminated kernel pixels matches one of the illuminated pixels in the unknown image. This match is shown by a darkened pixel in  FIG.  1 F (e). Since  FIG.  1 F (e) now has a different set of matched/unmatched characteristics, and further, since another one of the kernel pixels matches a corresponding pixel in the unknown image, it is expected that the resulting kernel value will increase. Indeed, as shown in  FIG.  1 G (e), when the nine multiplication operations are carried out, four unilluminated pixels in the kernel match four unilluminated pixels in the unknown image, one illuminated pixel in the kernel matches one illuminated pixel in the unknown image, and four other illuminated pixels in the kernel do not match the unilluminated four pixels in the unknown image. When the nine products are summed, the result of “+1” is divided by nine for a calculated kernel value of “+0.11” in the fifth kernel position. 
     As the kernel is moved further rightward in  FIG.  1 F (f), a different one of the illuminated kernel pixels matches a corresponding illuminated pixel in the unknown image.  FIG.  1 G (f) represents the set of matched and unmatched pixels as a kernel value of “+0.11.” 
     In  FIG.  1 F (g), the kernel is moved one more column to the right, and in this position, every pixel in the kernel matches every pixel in the unknown image. Since the nine multiplications performed when each pixel of the kernel is multiplied by its corresponding pixel in the unknown image results in a “+1.0,” the sum of the nine products is calculated to be “+9.0,” and the final kernel value for the particular position is calculated (i.e., 9.0/9) to be “+1.0,” which represents a perfect match. 
     In  FIG.  1 F (h), the kernel is moved rightward again, which results in a single illuminated pixel match, four unilluminated pixel matches, and a kernel value of “+0.11,” as illustrated in  FIG.  1 G (h). 
     The kernel continues to be moved as shown in  FIGS.  1 F (i),  1 F(j),  1 F(k), and  1 F(l), and in each position, a kernel value is mathematically calculated. Since no illuminated pixels of the kernel are overlayed on illuminated pixels of the unknown image in in  FIGS.  1 F (i) to  1 F(l), the calculated kernel value for each of these positions is “−0.11.” The kernel values are shown in  FIGS.  1 G (i),  1 G(j),  1 G(k), and  1 G(l) as “−0.11” in the respective four kernel positions. 
       FIG.  1 H  illustrates a stack of maps of kernel values. The topmost kernel map in  FIG.  1 H  is formed when the twelfth kernel of the numeral “1” in  FIG.  1 B  is moved into each position of the unknown image. The twelfth kernel will be recognized as the kernel used in each of  FIGS.  1 F (a) to  1 F(l) and  FIGS.  1 G (a) to  1 G(l). For each position where the selected kernel is conceptually overlayed on the unknown image, a kernel value is calculated, and the kernel value is stored in its respective position on the kernel map. 
     Also in  FIG.  1 H , other filters (i.e., kernels) are also applied to the unknown image. For simplicity in the discussion, the 29th kernel of the numeral “1” in  FIG.  1 B  is selected, and the 61st kernel of the numeral “1” in  FIG.  1 B  is selected. For each kernel, a distinct kernel map is created. The plurality of created kernel maps may be envisioned as a stack of kernel maps having a depth equal to the number of filters (i.e., kernels) that are applied. The stack of kernel maps may also be called a stack of filtered images. 
     In the convolutional process of the CNN system  10 , a single unknown image is convolved to create a stack of filtered images. The depth of the stack is the same as, or is otherwise based on, the number of filters (i.e., kernels) that are applied to the unknown image. The convolutional process in which a filter is applied to an image is also referred to as a “layer” because they can be stacked together. 
     As evident in  FIG.  1 H , a large quantity of data is generated during the convolutional layering process. In addition, each kernel map (i.e., each filtered image) has nearly as many values in it as the original image. In the examples presented in  FIG.  1 H , the original unknown input image is formed by 100 pixels (10×10), and the generated filter map has 64 values (8×8). The simple reduction in size of the kernel map is only realized because the applied 9-pixel kernel values (3×3) cannot fully process the outermost pixels at the edge of the image. 
       FIG.  1 I  shows a pooling feature that significantly reduces the quantity of data produced by the convolutional processes. A pooling process may be performed on one, some, or all of the filtered images. The kernel map in  FIG.  1 I  is recognized as the top-most filter map of  FIG.  1 H , which is formed with the 12th kernel of the numeral “1” in  FIG.  1 B . 
     The pooling process introduces the concepts of “window size” and “stride.” The window size is the dimensions of a window such that a single, maximum value within the window will be selected in the pooling process. A window may be formed having dimensions of m-pixels by n-pixels wherein “m” and “n” are integers, but in most cases, “m” and “n” are equal. In the pooling operation shown in  FIG.  1 I , each window is formed as a 2-pixel-by-2-pixel window. In the pooling operation, a 4-pixel window is conceptually overlayed onto a selected portion of the kernel map, and within the window, the highest value is selected. 
     In the pooling operation, in a manner similar to conceptually overlaying a kernel on an unknown image, the pooling window is conceptually overlayed onto each portion of the kernel map. The “stride” represents how much the pooling window is moved after each pooling act. If the stride is set to “two,” then the pooling window is moved by two pixels after each pooling act. If the stride is set to “three,” then the pooling window is moved by three pixels after each pooling act. 
     In the pooling operation of  FIG.  1 I , the pooling window size is set to 2×2, and the stride is also set to two. A first pooling operation is performed by selecting the four pixels in the top-most, left-most corner of the kernel map. Since each kernel value in the window has been calculated to be “−0.11,” the value from the pooling calculation is also “−0.11.” The value of “−0.11” is placed in the top-most, left-most corner of the pooled output map in  FIG.  1 I . 
     The pooling window is then moved rightward by the selected stride of two pixels, and the second pooling act is performed. Once again, since each kernel value in the second pooling window is calculated to be “−0.11,” the value from the pooling calculation is also “−0.11.” The value of “−0.11” is placed in the second entry of the top row of the pooled output map in  FIG.  1 I . 
     The pooling window is moved rightward by a stride of two pixels, and the four values in the window are evaluated. The four values in the third pooling act are “+0.11,” “+0.11,” “+0.11,” and “+0.33.” Here, in this group of four kernel values, “+0.33” is the highest value. Therefore, the value of “+0.33” is placed in the third entry of the top row of the pooled output map in  FIG.  1 I . The pooling operation does not care where in the window the highest value is found, the pooling operation simply selects the highest (i.e., the greatest) value that falls within the boundaries of the window. 
     The remaining 13 pooling operations are also performed in a like manner so as to fill the remainder of the pooled output map of  FIG.  1 I . Similar pooling operations may also be performed for some or all of the other generated kernel maps (i.e., filtered images). Further considering the pooled output of  FIG.  1 I , and further considering the selected kernel (i.e., the twelfth kernel of the numeral “1” in  FIG.  1 B ) and the unknown image, it is recognized that the highest values are found in the upper right-hand corner of the pooled output. This is so because when the kernel feature is applied to the unknown image, the highest correlations between the pixels of the selected feature of interest (i.e., the kernel) and the similarly arranged pixels in the unknown image are also found in the upper right-hand corner. It is also recognized that the pooled output has values captured in it that loosely represent the values in the un-pooled, larger-sized kernel map. If a particular pattern in an unknown image is being searched for, then the approximate position of the pattern can be learned from the pooled output map. Even if the actual position of the feature is not known with certainty, an observer can recognize that the feature was detected in the pooled output. The actual feature may be moved a little bit left or a little bit right in the unknown image, or the actual feature may be rotated or otherwise not identical to the kernel feature, but nevertheless, the occurrence of the feature and its general position may be recognized. 
     An optional normalization operation is also illustrated in  FIG.  1 I . The normalization operation is typically performed by a Rectified Linear Unit (ReLU). The ReLU identifies every negative number in the pooled output map and replaces the negative number with the value of zero (i.e., “0”) in a normalized output map. The optional normalization process by one or more ReLU circuits helps to reduce the computational resource workload that may otherwise be required by calculations performed with negative numbers. 
     After processing in the ReLU layer, data in the normalized output map may be averaged in order to predict whether or not the feature of interest characterized by the kernel is found or is not found in the unknown image. In this way, each value in a normalized output map is used as a weighted “vote” that indicates whether or not the feature is present in the image. In some cases, several features (i.e., kernels) are convolved, and the predictions are further combined to characterize the image more broadly. For example, as illustrated in  FIG.  1 H , three kernels of interest derived from a known image of a numeral “1” are convolved with an unknown image. After processing each kernel through the various layers, a prediction is made as to whether or not the unknown image includes one or more pixel patterns that show a numeral “1.” 
     Summarizing  FIGS.  1 A- 1 I , kernels are selected from a known image. Not every kernel of the known image needs to be used by the CNN. Instead, kernels that are determined to be “important” features may be selected. After the convolution process produces a kernel map (i.e., a feature image), the kernel map is passed through a pooling layer, and a normalization (i.e., ReLU) layer. All of the values in the output maps are averaged (i.e., sum and divide), and the output value from the averaging is used as a prediction of whether or not the unknown image contains the particular feature found in the known image. In the exemplary case, the output value is used to predict whether the unknown image contains a numeral “1.” In some cases, the “list of votes” may also be used as input to subsequent stacked layers. This manner of processing reinforces strongly identified features and reduces the influence of weakly identified (or unidentified) features. Considering the entire CNN, a two-dimensional image is input to the CNN and produces a set of votes at its output. The set of votes at the output are used to predict whether the input image either does or does not contain the object of interest that is characterized by the features. 
     The CNN system  10  of  FIG.  1 A  may be implemented as a series of operational layers. One or more convolutional layers may be followed by one or more pooling layers, and the one or more pooling layers may be optionally followed by one or more normalization layers. The convolutional layers create a plurality of kernel maps, which are otherwise called filtered images, from a single unknown image. The large quantity of data in the plurality of filtered images is reduced with one or more pooling layers, and the quantity of data is reduced further by one or more ReLU layers that normalize the data by removing all negative numbers. 
       FIG.  1 J  shows the CNN system  10  of  FIG.  1 A  in more detail. In  FIG.  1 J (a), the CNN system  10  accepts a 10-pixel-by-10-pixel input image into a CNN. The CNN includes a convolutional layer, a pooling layer, a rectified linear unit (ReLU) layer, and a voting layer. One or more kernel values are convolved in cooperation with the unknown 10×10 image, and the output from the convolutional layer is passed to the pooling layer. One or more max pooling operations are performed on each kernel map provided by the convolutional layer. Pooled output maps from the pooling layer are used as input to a ReLU layer that produces normalized output maps, and the data contained in the normalized output maps is summed and divided to determine a prediction as to whether or not the input image includes a numeral “1” or a numeral “0.” 
     In  FIG.  1 J (b), another CNN system  10   a  is illustrated. The CNN in the CNN system  10   a  includes a plurality of layers, which may include convolutional layers, pooling layers, normalization layers, and voting layers. The output from one layer is used as the input to a next layer. In each pass through a convolutional layer, the data is filtered. Accordingly, both image data and other types data may be convolved to search for (i.e., filter) any particular feature. When passing through pooling layers, the input data generally retains its predictive information, but the quantity of data is reduced. Since the CNN system  10   a  of  FIG.  1 J (b) includes many layers, the CNN is arranged to predict that the input image contains any one of many different features. 
     One other characteristic of a CNN is the use of back propagation to reduce errors and improve the quality of the neural network to recognize particular features in the midst of vast quantities of input data. For example, if the CNN arrives at a prediction that is less than 1.0, and the prediction is later determined to be accurate, then the difference between the predicted value and 1.0 is considered an error rate. Since the goal of the neural network is to accurately predict whether or not a particular feature is included in an input data set, the CNN can be further directed to automatically adjust weighting values that are applied in a voting layer. 
     Back propagation mechanisms are arranged to implement a feature of gradient descent. Gradient descent may be applied on a two-dimensional map wherein one axis of the map represents “error rate,” and the other axis of the map represents “weight.” In this way, such a gradient-descent map will preferably take on a parabolic shape such that if an error rate is high, then the weight of that derived value will be low. As error rate drops, then the weight of the derived value will increase. Accordingly, when a CNN that implements back propagation continues to operate, the accuracy of the CNN has the potential to continue improving itself automatically. 
     The performance of known object recognition techniques that use machine learning methods is improved by applying more powerful models to larger datasets, and implementing better techniques to prevent overfitting. Two known large datasets include LabelMe and ImageNet. LabelMe includes hundreds of thousands of fully segmented images, and more than 15 million high-resolution, labeled images in over 22,000 categories are included in ImageNet. 
     To learn about thousands of objects from millions of images, the model that is applied to the images requires a large learning capacity. One type of model that has sufficient learning capacity is a convolutional neural network (CNN) model. In order to compensate for an absence of specific information about the huge pool of data, the CNN model is arranged with at least some prior knowledge of the data set (e.g., statistical stationarity/non-stationarity, spatiality, temporality, locality of pixel dependencies, and the like). The CNN model is further arranged with a designer selectable set of features such as capacity, depth, breadth, number of layers, and the like. 
     Early CNN&#39;s were implemented with large, specialized super-computers. Conventional CNN&#39;s are implemented with customized, powerful graphic processing units (GPUs). As described by Krizhevsky, “current GPUs, paired with a highly optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting.” 
       FIGS.  2 A- 2 B  may collectively be referred to herein as  FIG.  2   . 
       FIG.  2 A  is an illustration of the known AlexNet DCNN architecture. As described by Krizhevsky,  FIG.  1    shows the “delineation of responsibilities between [the] two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The GPUs communicate only at certain layers. The network&#39;s input is 150,528-dimensional, and the number of neurons in the network&#39;s remaining layers is given by 253,440-186,624-64,896-64,896-43,264-4096-4096-1000.” 
     Krizhevsky&#39;s two GPUs implement a highly optimized two-dimensional (2D) convolution framework. The final network contains eight learned layers with weights. The eight layers consist of five convolutional layers CL 1 -CL 5 , some of which are followed by max-pooling layers, and three fully connected layers FC with a final 1000-way softmax, which produces a distribution over 1000 class labels. 
     In  FIG.  2 A , kernels of convolutional layers CL 2 , CL 4 , CL 5  are connected only to kernel maps of the previous layer that are processed on the same GPU. In contrast, kernels of convolutional layer CL 3  are connected to all kernel maps in convolutional layer CL 2 . Neurons in the fully connected layers FC are connected to all neurons in the previous layer. 
     Response-normalization layers follow the convolutional layers CL 1 , CL 2 . Max-pooling layers follow both the response-normalization layers as well as convolutional layer CL 5 . The max-pooling layers summarize the outputs of neighboring groups of neurons in the same kernel map. Rectified Linear Unit (ReLU) non-linearity is applied to the output of every convolutional and fully connected layer. 
     The first convolutional layer CL 1  in the AlexNet architecture of  FIG.  1 A  filters a 224×224×3 input image with 96 kernels of size 11×11×3 with a stride of 4 pixels. This stride is the distance between the receptive field centers of neighboring neurons in a kernel map. The second convolutional layer CL 2  takes as input the response-normalized and pooled output of the first convolutional layer CL 1  and filters the output of the first convolutional layer with 256 kernels of size 5×5×48. The third, fourth, and fifth convolutional layers CL 3 , CL 4 , CL 5  are connected to one another without any intervening pooling or normalization layers. The third convolutional layer CL 3  has 384 kernels of size 3×3×256 connected to the normalized, pooled outputs of the second convolutional layer CL 2 . The fourth convolutional layer CL 4  has 384 kernels of size 3×3×192, and the fifth convolutional layer CL 5  has 256 kernels of size 3×3×192. The fully connected layers have 4096 neurons each. 
     The eight layer depth of the AlexNet architecture seems to be important because particular testing revealed that removing any convolutional layer resulted in unacceptably diminished performance. The network&#39;s size is limited by the amount of memory available on the implemented GPUs and by the amount of training time that is deemed tolerable. The AlexNet DCNN architecture of  FIG.  1 A  takes between five and six days to train on two NVIDIA GEFORCE GTX 580 3 GB GPUs. 
       FIG.  2 B  is a block diagram of a known GPU such as the NVIDIA GEFORCE GTX 580 GPU. The GPU is a streaming multiprocessor containing 32 unified device architecture processors that employ a flexible scalar architecture. The GPU is arranged for texture processing, shadow map processing, and other graphics-centric processing. Each of the 32 processors in the GPU includes a fully pipelined integer arithmetic logic unit (ALU) and floating point unit (FPU). The FPU complies with the IEEE 754-2008 industry standard for floating-point arithmetic. The GPU in this case is particularly configured for desktop applications. 
     Processing in the GPU is scheduled in groups of 32 threads called warps. Each of the 32 threads executes the same instructions simultaneously. The GPU includes two warp schedulers and two instruction dispatch units. In this arrangement, two independent warps can be issued and executed at the same time. 
     All of the subject matter discussed in the Background section is not necessarily prior art and should not be assumed to be prior art merely as a result of its discussion in the Background section. Along these lines, any recognition of problems in the prior art discussed in the Background section or associated with such subject matter should not be treated as prior art unless expressly stated to be prior art. Instead, the discussion of any subject matter in the Background section should be treated as part of the inventor&#39;s approach to the particular problem, which in and of itself may also be inventive. 
     BRIEF SUMMARY 
     A device may be summarized as including an integrated circuit; a reconfigurable stream switch formed in the integrated circuit; and an arithmetic unit having a plurality of inputs coupled to the reconfigurable stream switch and at least one output, the arithmetic unit solely dedicated to performance of a plurality of parallel operations, wherein each one of the plurality of parallel operations carries out a portion of a formula, the formula being: output=AX+BY+C. The integrated circuit may be arranged for convolutional neural network operations. The arithmetic unit may be arranged to receive streaming data passed through the reconfigurable stream switch, and the arithmetic unit may be arranged to pass a stream of resultant data through the reconfigurable stream switch via the at least one output. At least one of the plurality of inputs may be arranged to pass vector data into the arithmetic unit. At least one of the plurality of inputs may be arranged to pass scalar data into the arithmetic unit. 
     The arithmetic unit may include multiplexor circuitry arranged as a plurality of multiplexor circuits communicatively coupled to at least some of the plurality of inputs; multiplier circuitry arranged as a plurality of multiplier circuits, each multiplier circuit arranged to accept multiplicand data, multiplier data, or both multiplicand data and multiplier data from the multiplexor circuitry, the multiplier circuitry further arranged to perform at least some multiplication operations of the formula; temporary storage circuitry arranged as a plurality of temporary storage repositories, each of the plurality of temporary storage repositories arranged to receive product data from a multiplier circuit; data shift circuitry arranged as a plurality of shift circuits, the plurality of shift circuits communicatively coupled the temporary storage circuitry; and adder circuitry, the adder circuitry arranged as at least one adder circuit to perform at least some summation operations of the formula. The arithmetic unit may include post-summation circuitry arranged to receive sum data from the adder circuitry, the post-summation circuitry communicatively coupled to the at least one output of the arithmetic unit. The post-summation circuitry may include configurable direction shift circuitry. 
     The arithmetic unit may include a first multiplexor circuit coupled to receive an “A” input, the “A” input corresponding to the “A” in the formula; a second multiplexor circuit coupled to receive a “B” input, the “B” input corresponding to the “B” in the formula; a third multiplexor circuit coupled to receive a “C” input, the “C” input corresponding to the “C” in the formula; a first multiplier circuit arranged to receive the “A” input from the first multiplexor circuit and further arranged to receive an “X” input, the first multiplier circuit further arranged to produce a first product by multiplying the “A” input and the “X” input; a second multiplier circuit arranged to receive the “B” input from the second multiplexor circuit and further arranged to receive a “Y” input, the second multiplier circuit further arranged to produce a second product by multiplying the “B” input and the “Y” input; and an adder circuit arranged to sum information generated by first and second multiplier circuits and information passed through the third multiplexor circuit, the adder circuit arranged to produce a sum corresponding to the output in the formula. The first multiplexor circuit may be arranged to pass first scalar data or first vector data, and the second multiplexor circuit may be arranged to pass second scalar data or second vector data. 
     A machine learning method may be summarized as including streaming input data through a reconfigurable stream switch to an arithmetic unit, the arithmetic unit solely dedicated to performance of a formula, the formula being: output=AX+BY+C; generating output data corresponding to the output of the formula with the arithmetic unit; streaming the output data through the reconfigurable stream switch; and identifying at least one feature in the input data based on information in the stream of output data. The identifying may be performed within a deep convolutional neural network procedure. The input data may be image sensor data. 
     The method may include defining a feature volume in the image sensor data, the feature volume having a feature volume height, a feature volume width, and a feature volume depth; defining an input tile having an input tile height, an input tile width, and an input tile depth; and walking the input tile through the feature volume. 
     The method may include configuring a first convolution accelerator to perform a max pooling operation; configuring a second convolution accelerator to perform an average pooling operation; streaming image sensor data through the reconfigurable stream switch to a third convolutional accelerator; and convolving the image sensor data to produce a stack of kernel maps. And the method may further include streaming the stack of kernel maps through the reconfigurable stream switch to the first convolution accelerator and to the second convolution accelerator; performing the max pooling operation with the first convolution accelerator and streaming max pool data through the reconfigurable stream switch as the input data to the arithmetic unit; performing the average pooling operation with the second convolution accelerator and streaming average pool data through the reconfigurable stream switch as the input data to the arithmetic unit; and performing a max-average pooling operation with the arithmetic unit and streaming max-average pool data through the reconfigurable stream switch. 
     The method may include performing, via the arithmetic unit, at least one of a bias operation, a mean operation, a scaling operation, a max-average pooling operation, and a branch merge operation. 
     The method may include selecting, via a selection circuit, at least one of vector data and scalar data; passing the selected at least one of vector data and scalar data into a second arithmetic unit, the second arithmetic unit solely dedicated to performance of the formula; and performing, via the second arithmetic unit, a linear data combination operation. 
     An integrated circuit may be summarized as including on-board memory, which may be random access memory (RAM); an applications processor; a digital signal processor (DSP) cluster; a configurable accelerator framework (CAF); and at least one communication bus architecture communicatively coupling the applications processor, the DSP cluster, and the CAF to the on-board memory, wherein the CAF includes: a reconfigurable stream switch; and at least one arithmetic unit having a plurality of inputs coupled to the reconfigurable stream switch and at least one output coupled to the reconfigurable stream switch, each arithmetic unit solely dedicated to performance of a plurality of parallel operations, wherein each one of the plurality of parallel operations carries out a portion of a formula, the formula being: output=AX+BY+C. The integrated circuit may be formed as a system on chip. The integrated circuit may include a power manager circuit, the power manager circuit electrically coupleable to a battery. 
     This Brief Summary has been provided to introduce certain concepts in a simplified form that are further described in detail below in the Detailed Description. Except where otherwise expressly stated, the Brief Summary does not identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. 
    
    
     
       BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 
       Non-limiting and non-exhaustive embodiments are described with reference to the following drawings, wherein like labels refer to like parts throughout the various views unless otherwise specified. The sizes and relative positions of elements in the drawings are not necessarily drawn to scale. For example, the shapes of various elements are selected, enlarged, and positioned to improve drawing legibility. The particular shapes of the elements as drawn have been selected for ease of recognition in the drawings. One or more embodiments are described hereinafter with reference to the accompanying drawings in which: 
         FIG.  1 A  is a simplified illustration of a convolutional neural network (CNN) system; 
         FIG.  1 B  illustrates the CNN system of  FIG.  1 A  determining that a first pixel pattern illustrates a “1” and that a second pixel pattern illustrates a “0”; 
         FIG.  1 C  shows several variations of different forms of ones and zeroes; 
         FIG.  1 D  represents a CNN operation that analyzes (e.g., mathematically combines) portions of an unknown image with corresponding portions of a known image; 
         FIG.  1 E  illustrates six portions of the right-side, known image of  FIG.  1 D ; 
         FIG.  1 F  illustrates 12 acts of convolution in a filtering process; 
         FIG.  1 G  shows the results of the 12 convolutional acts of  FIG.  1 F ; 
         FIG.  1 H  illustrates a stack of maps of kernel values; 
         FIG.  1 I  shows a pooling feature that significantly reduces the quantity of data produced by the convolutional processes; 
         FIG.  1 J  shows the CNN system of  FIG.  1 A  in more detail; 
         FIG.  2 A  is an illustration of the known AlexNet DCNN architecture; 
         FIG.  2 B  is a block diagram of a known GPU; 
         FIG.  3    is an exemplary mobile device having integrated therein a DCNN processor embodiment illustrated as a block diagram; 
         FIG.  4    is an embodiment depicting a configurable accelerator framework (CAF), such as the image and deep convolutional neural network (DCNN) co-processor subsystem of  FIG.  3   ; 
         FIG.  5    is a stream switch embodiment in more detail; 
         FIG.  6    is a convolution accelerator (CA) embodiment; 
         FIG.  7 A  is a first high level block diagram illustrating certain data paths supported by arithmetic units for deep learning acceleration in a neural network; 
         FIG.  7 B  is a second high level block diagram illustrating certain data paths supported by arithmetic units for deep learning acceleration in a neural network; 
         FIG.  7 C  is a third high level block diagram illustrating convolutional accelerators and an arithmetic unit arranged for an exemplary branch-merge operation in a neural network; 
         FIG.  7 D  is a first data flow diagram illustrating structures configurable to execute a single dedicated formula (i.e., AX+BY+C) with a arithmetic unit for deep learning acceleration in a neural network; 
         FIG.  7 E  is a second data flow diagram illustrating one-by-one (1×1) support operations of an arithmetic unit for deep learning acceleration in a neural network; 
         FIG.  7 F  is a third data flow diagram illustrating operations of an arithmetic unit for deep learning acceleration in a neural network in a final stage of a convolutional process. 
     
    
    
     DETAILED DESCRIPTION 
     The present invention may be understood more readily by reference to the following detailed description of the preferred embodiments of the invention. It is to be understood that the terminology used herein is for the purpose of describing specific embodiments only and is not intended to be limiting. It is further to be understood that unless specifically defined herein, the terminology used herein is to be given its traditional meaning as known in the relevant art. 
     It has been recognized by the inventors that learning machines can be improved if additional dedicated hardware structures are integrated with or otherwise made available to the architectures that implement the learning machine. One such improvement that can be made includes structures and acts that implement an arithmetic unit as described herein. The inventive arithmetic unit of the present disclosure may be implemented on a wide variety of learning machines. For brevity, however, the present disclosure includes implementations of the inventive arithmetic unit in a particular deep convolutional neural network disclosed in U.S. patent application Ser. No. 15/423,272 to DESOLI et aI., and entitled DEEP CONVOLUTIONAL NETWORK HETEROGENEOUS ARCHITECTURE, which application is incorporated by reference into the present application. This particular deep convolutional network heterogeneous architecture learning machine discloses a system on chip (SoC) having a system bus, a plurality of addressable memory arrays coupled to the system bus, at least one applications processor core coupled to the system bus, and a configurable accelerator framework coupled to the system bus. The configurable accelerator framework is an image and deep convolutional neural network (DCNN) co-processing system. The SoC also includes a plurality of digital signal processors (DSPs) coupled to the system bus, wherein the plurality of DSPs coordinate functionality with the configurable accelerator framework to execute the DCNN. 
       FIGS.  3 - 6    and the accompanying detailed description thereof illustrate and present elements of an exemplary system on chip (SoC)  110  configurable as a high-performance, energy efficient hardware accelerated DCNN processor.  FIGS.  7 A- 7 F  and the accompanying detailed description thereof illustrate and present structures and data flow diagrams of arithmetic units for deep learning acceleration  700  integrated with the hardware accelerated DCNN processor of  FIGS.  3 - 6   . The exemplary SoC  110 , which is particularly useful for machine learning applications, implements an image and DCNN co-processor subsystem  400  ( FIG.  4   ), which may interchangeably be referred to as a configurable accelerator framework; an architecturally efficient stream switch  500  ( FIG.  5   ), which creates data locality at previously unprecedented levels; a set of convolution accelerators  600  ( FIG.  6   ), which perform a convolution of input feature data with kernel data derived from the training of the neural network; and a set of arithmetic units particularly arranged for deep learning acceleration  700  ( FIG.  7   ). 
       FIG.  3    is an exemplary mobile device  100  having integrated therein a DCNN processor embodiment illustrated as a block diagram. The mobile DCNN processor is arranged as a system on chip (SoC)  110 , however other arrangements are also contemplated (e.g., multiple chips, several chip die in a single integrated circuit, and the like). The illustrated SoC  110  includes a plurality of SoC controllers  120 , a configurable accelerator framework (CAF)  400  (e.g., an image and DCNN co-processor subsystem), an SoC global memory  126 , an applications (e.g., a host) processor  128 , and a plurality of DSPs  138 , each of which are communicatively coupled, directly or indirectly, to a primary (e.g., system) communication bus  132  and a secondary communications (e.g., DSP) bus  166 . 
     The configurable accelerator framework (CAF)  400  is communicatively coupled to the system bus  166 , which provides a mechanism for convolution accelerators of the CAF  400  to access the SoC global memory  126  as needed and to communicate with the DSPs  138  as needed. The CAF  400  is illustrated in more detail in  FIG.  4   . 
     The SoC  110  includes various SoC controllers  120 , some of which control the SoC  110 , and others of which control one or more peripheral devices. SoC controllers  120  include an applications (e.g., a host) processor  128  (e.g., an ARM processor or some other host processor), a clock generator  168  (e.g., a clock manager), a reset controller  170 , and a power manager  172  to provide additional support, control, and management of various timing, power consumption, and other aspects of the SoC  110  and other components. Other SoC controllers  120  that control peripherals include a low speed peripheral I/O interface  130  and an external memory controller  174  to communicate with or otherwise access external chips, components, or memory of the exemplary device  100  in which the SoC  110  is embedded. 
     The applications processor  128  may act as an intermediate module or as an interface to other programs or components of the exemplary electronic device  100  with which the SoC  110  is integrated. In some embodiments, the applications processor  128  may be referred to as an applications processor core. In various embodiments, the applications processor  128  loads an SoC configuration file at boot time and configures DSPs  138  and the CAF  400  according to the configuration file. As the SoC  110  processes one or more batches of input data (e.g., an image), the applications processor  128  may coordinate the reconfiguration of the CAF  400  or DSPs  138  based on the configuration file, which itself may be based on the DCNN layers and topology. 
     The SoC  110  also includes a primary communications bus  132  (e.g., an AXI—Advanced eXtensible Interface) that facilitates communications between the SoC controllers  120  and the DSPs  138  and between the SoC controllers  120  and the CAF  400 . For example, the DSPs  138  or the CAF  400  can communicate, via the primary communications bus  132  with the applications processor  128 , one or more peripheral controllers/peripheral communications interface (low speed peripheral I/O)  130 , an external memory (not shown) via an external memory controller  174 , or other components. The SoC controllers  120  may also include other supporting and cooperative devices such as a clock manager (e.g., a clock generator)  168 , a reset controller  170 , a power manager  172  to provide additional timing and power management to the SoC  110 , and other components. 
     In some embodiments, and as illustrated in  FIG.  3   , the plurality of DSPs  138  are arranged in a plurality of DSP clusters, such as a first DSP cluster  122 , a second DSP cluster  140 , and several other DSP clusters that are not referenced for simplification of the illustration. 
     Each DSP cluster  122 ,  140  includes a plurality (e.g., two) of DSPs  142 ,  152 , a plurality (e.g., two) of local DSP crossbar switches  144 ,  154 , and a DSP cluster crossbar switch  145 ,  155 . Each DSP  142 ,  152  in a particular cluster is capable of communicating with other DSP&#39;s  142 ,  152  via the DSP cluster crossbar switch  145 ,  155 . Each DSP  142 ,  152  has access to a corresponding instruction cache  146 ,  156 , and local DSP memory  148 ,  158  via its corresponding local DSP crossbar switch  144 ,  154 . In one non-limiting embodiment, each instruction cache  146 ,  156  is a 4-way 16 kB instruction cache and each local DSP memory  148 ,  158  is 64 kB of local RAM storage for its corresponding DSP. Each DSP cluster  122 ,  140  also includes a shared DSP cluster memory  160 ,  159  and a cluster DMA  162 ,  164  for accessing the SoC global memory  160 ,  159 . 
     Each DSP cluster  122 ,  140  is communicatively coupled to a global DSP cluster crossbar switch  150  via the DSP cluster crossbar switch  145 ,  155  to enable each DSP  142 ,  152  in each DSP cluster  122 ,  140  to communicate with one another and other components on the SoC  110 . The global DSP cluster crossbar switch  150  enables each DSP to communicate with other DSPs in the plurality of DSP clusters  138 . 
     Additionally, the global DSP cluster crossbar switch  150  is communicatively coupled to a system bus  166  (e.g., secondary communications bus, xbar—SoC crossbar switch, or the like), which enables each DSP to communicate with other components of the SoC  110 . For example, each DSP  142 ,  152  can communicate with one or more components (e.g., one or more convolution accelerators) of the CAF  400  or access an SoC global memory  126  via the system bus  166 . In some embodiments, each DSP  142 ,  152  can communicate with the SoC memory  126  via the DMA  162 ,  164  of its corresponding DSP cluster  122 ,  140 . Moreover, DSP  142 ,  152  may communicate with the controllers  120  or other modules of the SoC  110  as needed via the system bus  166 . Each DSP accesses the system bus  166  via its local DSP crossbar switch  144 ,  154 , its DSP cluster crossbar switch  145 ,  155 , and the global DSP cluster crossbar switch  150 . 
     The plurality of DSPs  138  can be assigned or allocated to perform specific instructions to accelerate other operations of the DCNN. These other operations may include non-convolutional operations performed during a DCNN process, which are in some cases primarily performed by the CAF  400 . Examples of these non-convolutional operations include, but are not limited to, max or average pooling, nonlinear activation, cross-channel response normalization, classification representing a small fraction of the total DCNN computation but more amenable to future algorithmic evolutions, or other operations, e.g., Min, Max, Sqrt, Mac, Butterfly, Average, 2-4 SIMD ALU. In some cases, operations that previously have been performed using one or more of the DSPs  138  are now performed using the arithmetic unit for deep learning acceleration structures described herein with reference to  FIG.  7   . Accordingly, improved operations of the processors and their associated computing devices described herein may be realized by the arithmetic unit structures described herein. 
     DSPs  138  can operate concurrently (e.g., in parallel) with the operations of CA&#39;s in the CAF  400  and concurrently (e.g., in parallel) with data transfers, which may be synchronized by way of interrupts, mailboxes, or some other synchronization mechanism for concurrent execution. 
     In various embodiments, the SoC memory  126  includes a plurality of memory components for storing data that is accessible to the components of the CAF  400  or the DSPs  138 . In at least one embodiment, the SoC memory  126  is configured in a hierarchical-type memory structure. In one non-limiting example, the SoC memory  126  includes four SRAM banks each with 1 Mbyte of storage space. 
     In at least one embodiment, the configurable accelerator framework (CAF)  400  may be organized as an image and DCNN co-processor subsystem of the SoC  110 . As described herein, the CAF  400  includes a reconfigurable dataflow accelerator fabric connecting high-speed camera interfaces with any one or more of arithmetic units for deep learning acceleration ( FIG.  6   ), sensor processing pipelines, croppers, color converters, feature detectors, video encoders, eight channel digital microphone interface, streaming DMAs, and a plurality of convolution accelerators. 
     Additional details regarding the CAF  400  are described in conjunction with  FIG.  4   . Briefly, the CAF  400  receives incoming data (e.g., image data in  FIG.  4   , but other types of streaming data in different embodiments), such as from the camera interface, or other sensors, and distributes the incoming data to the various components of the CAF  400  (e.g., convolution accelerators, which are described in more detail in conjunction with  FIG.  6   , arithmetic units for deep learning acceleration  700  described in more detail in conjunction with  FIG.  7   , and the like) and/or one or more of the plurality of DSPs  138  to employ the DCNN and recognize objects in the incoming images. 
     The CAF  400  utilizes unidirectional links to transport data streams via a configurable, fully connected switch to or from different kinds of source or sink components. For example, the configurable fully connected switch, which is described in more detail in conjunction with  FIG.  5   , can transport data via direct memory accesses (DMAs) to the SoC global memory  126 , I/O interfaces (e.g., cameras), and various types of accelerators (e.g., convolution accelerator (CA)  600 , arithmetic units for deep learning acceleration  700 , etc.). In some cases, the CAF  400  is configured at boot time based on information received from a particular SoC configuration tool, and the CAF  400  is re-configured during run time based on defined DCNN layers and topology or information received from one or more DSPs  138 , applications processor  128 , or the like. 
     The CAF  400  allows for the definition of a selectable number of concurrent, virtual processing chains at run time. The CAF  400  also includes a full featured back pressure mechanism to control data flow to the various components of the framework. The CAF  400  is arranged for stream multicasting operations, which enable the reuse of a data stream at multiple block instances. Linked lists control the fully autonomous processing of an entire convolution layer. Multiple accelerators, grouped or chained together, handle varying sizes for feature maps data and multiple kernels in parallel. Grouping the convolutional accelerators (CA&#39;s)  600  to achieve larger computational entities enables choosing an acceptably optimal balancing of the available data bandwidth, budget power, and available processing resources. Each CA  600  includes a line buffer to fetch up to a predetermined number (e.g., 12) of feature map data words in parallel with a single memory access. Further supporting the CA  600  structures are the arithmetic units for deep learning acceleration  700 , which perform math functions conformable to the formula of Equation 1 with a data locality heretofore unknown. 
         AX+BY+C →Output  (1)
 
     Rather than passing data for interim math functions out of the CAF  400  to a separate device such as a DSP, data is retained within the CAF  400  architecture thereby achieving significant speed and data throughput gains. 
     In each CA ( 600 ), a register-based kernel buffer provides multiple read ports (e.g., 36 read ports), while multiple fixed-point multiply-accumulate (MAC) units (e.g., 36 16-bit MAC units) perform multiple MAC operations per clock cycle (e.g., up to 36 operations per clock cycle). An adder tree accumulates MAC results for each kernel column. The overlapping, column based calculation of the MAC operations allows an acceptably optimal reuse of the feature maps data for multiple MACs, thereby reducing power consumption associated with redundant memory accesses. 
     Kernel sets are partitioned in batches processed sequentially and intermediate results can be stored in the SoC global memory  126 . Various kernel sizes (e.g., up to 12×12), various batch sizes (e.g., up to 16), and parallel kernels (e.g., up to 4) can be handled by a single CA  600  instance but any size kernel can be accommodated with the accumulator input. 
     The configurable batch size and a variable number of parallel kernels enable acceptably optimal trade-offs for the available input and output bandwidth sharing across different units and the available computing logic resources. 
     A different acceptably optimal configuration of CA&#39;s  600  in the CAF  400  is determined for each DCNN layer. These configurations may be determined or adjusted using a holistic tool that starts with a DCNN description format, such as Caffe&#39; or TensorFlow. The CA  600  supports on-the-fly kernel decompression and rounding when the kernel is quantized nonlinearly with 8 or fewer bits per weight with top-1 error rate increases up to 0.3% for 8 bits. 
       FIG.  4    is an embodiment depicting a configurable accelerator framework (CAF)  400 , such as the image and deep convolutional neural network (DCNN) co-processor subsystem  400  of  FIG.  3   . The CAF  400  may be configured for image processing, audio processing, prediction analysis (e.g., games of skill, marketing data, crowd behavior prediction, weather analysis and prediction, genetic mapping, disease diagnosis, and other scientific, commercial, consumer, and such processing) or some other type of processing; particularly processing that includes convolutional operations. 
     The CAF  400  is also arranged with a number of configurable modules. Some modules are optional, and some modules are required. Many optional modules are commonly included in embodiments of a CAF  400 . One required module of a CAF  400  is, for example, the stream switch  500 . The stream switch  500  provides a design time parametric, run-time reconfigurable accelerator interconnect framework to support dataflow based processing chains. Another required module is, for example, a set of CAF control registers  402 . Other modules may be required as well. Optional modules of the CAF  400  include a system bus interface module  404 , a selected number of DMA engines  406  (e.g., DMA controllers), a selected number of external device interfaces  408 , a selected number of processing modules  410 , a selected number of convolution accelerators (CA&#39;s)  600 , and a selected number of arithmetic units for deep learning acceleration  700 . 
     The stream switch  500  is a reconfigurable unidirectional interconnection structure formed with a plurality of unidirectional “stream links.” The stream links are arranged to transport multibit data streams from accelerators, interfaces, and other logic modules to the stream switch  500  and from the stream switch  500  to accelerators, interfaces, and other logic modules. 
     In addition to the stream switch  500 , the CAF  400  may also include a system bus interface module  404 . The system bus interface module  404  provides an interface to other modules of SoC  110 . As shown in the exemplary embodiment of  FIG.  3   , the CAF  400  is coupled to the secondary communication bus  166 . In other cases, the CAF  400  may be coupled to the primary communication bus  132  or some other communication mechanism. Control information may be passed unidirectionally or bidirectionally through the system bus interface module  404  of the CAF  400 . Such interface is used to provide a host processor (e.g., DSP of DSP cluster  130 , applications processor  128 , or another processor) access to all of the CAF control registers  402 , which are used to control, operate, or otherwise direct particular features of the framework. In some embodiments, each DMA engine  406 , external device interface  408 , processing module  410 , convolution accelerator  600 , and arithmetic unit for deep learning acceleration  700  has an interface to the configuration network with a defined set of configuration registers (e.g., formed in CAF control registers  402 ). 
     The CAF  400  includes a plurality of DMA engines  406 . In  FIG.  4   , sixteen DMA engines  406   a  to  406   p  are illustrated, but some other number of DMA engines may be included in other embodiments of SoC  110  according to one or more choices made by a semiconductor practitioner at design time. The DMA engines  406  are arranged to provide bidirectional channels for input data flow, output data flow, or input and output data flow. In these cases, substantial quantities of data are passed into the CAF  400 , out from the CAF  400 , or into and out from the CAF  400 . For example, in some cases, one or more DMA engines  406  are used to pass streaming video data from memory or from a data source device (e.g., a high-definition (HD) video camera) that produces substantial quantities of video data. Some or all of the video may be passed in from the source device, in from or out to SoC global memory  126 , and the like. 
     In one exemplary embodiment, one or more DMA engines  406  are connected to the stream switch  500  with one input port  504  ( FIG.  5   ) and one output stream port  516  ( FIG.  5   ). The DMA engines  406  can be configured in either input or output mode. The DMA engines  406  can be configured to pack and send data to any address location accessible on the primary communication bus  132 , the secondary communication bus  166 , or some other address location. The DMA engines  406  can also additionally or alternatively be configured to unpack fetched data and translate the unpacked data into a data stream. 
     The CAF  400  of  FIG.  4    includes a design-time selectable, run-time configurable plurality of external device interfaces  408 . The external device interfaces  408  provide a connection to external devices which produce (i.e., source devices) or consume (i.e., sink devices) data. In some cases, the data that passes through an external device interface  408  includes streaming data. The amount of streaming data that is passed through an external device interface  408  may be predetermined in some cases. Alternatively, the amount of streaming data passed through an external device interface  408  may be indeterminate, and in such cases, the external device may simply produce or consume data whenever the particular external device is enabled and so directed. External devices coupled through the external device interfaces  408  may include image sensors, digital microphones, display monitors, or other source and sink devices. In  FIG.  4   , external device interface  408  includes a digital visual interface (DVI) external device interface  408   a , a first image sensor interface and image signal processor (ISP) external device interface  408   b , and a second image sensor interface and ISP external device interface  408   c . Other interfaces are also contemplated, though for simplicity in illustration, only three external device interfaces  408  are shown. 
     A plurality of processing modules  410  are integrated in the CAF  400 . Three processing modules  410  are illustrated for simplicity, but another selected number (e.g., two, four, eight, sixteen) of processing modules  410  may also be integrated in a CAF  400  at design time by a semiconductor practitioner. A first processing module  410  is an MPEG/JPEG processing module  410   a  arranged to perform certain video (i.e., MPEG) processing and certain image (i.e., JPEG) processing. A second processing module  410  is an H264 processing module  410   b , which is arranged to perform particular video encoding/decoding operations. A third processing module  410  is a color converter processing module  410   n , which is arranged to perform color-based operations on certain multimedia data. 
     In many cases, the DMA controllers  406 , the external device interfaces  408 , the processing modules  410 , the convolution accelerators  600 , the arithmetic units for deep learning acceleration  700 , and other modules integrated in a CAF  400  are IP modules selected from a library by a semiconductor practitioner at design time. The semiconductor practitioner may specify the number of modules, features of particular modules, bus widths, power parameters, layout, memory availability, bus access, and many other parameters. 
     Table 2 is a non-exhaustive exemplary list of IP modules in a library; any of which may be incorporated into CAF  400  by a semiconductor practitioner. In many cases, as new modules are designed, and as existing modules are modified, the new IPs will be added to a library such as the library of Table 2. 
     
       
         
           
               
             
               
                 TABLE 2 
               
             
            
               
                   
               
               
                 CAF Library of IP modules 
               
            
           
           
               
               
            
               
                 Functional Unit 
                 Application 
               
               
                   
               
               
                 RGB/YUV Sensor Interface 
                 Interface 
               
               
                 Bayer Sensor Interface 
                 Interface 
               
               
                 Video Out Interface (DVI) 
                 Interface 
               
               
                 Functional Unit 
                 Application 
               
               
                 Enhanced I/O (Sensor Interface,  
                 Interface 
               
               
                 Video Out, Overlay) 
                   
               
               
                 ISP (Image Signal Processor) 
                 Signal Processing 
               
               
                 Mini ISP (Image Signal Processor) 
                 Signal Processing (Bayer -&gt; RGB) 
               
               
                 GP Color Converter Unit 
                 General Purpose 
               
               
                 Image Cropper and Resizer Unit 
                 General Purpose 
               
               
                 Morph Filter Unit 
                 General Purpose 
               
               
                 Background Remove Unit (+shadow 
                 Background/Foreground  
               
               
                 remove) 
                 segmentation 
               
               
                 Reference Frame Update Unit 
                 Background/Foreground  
               
               
                   
                 segmentation 
               
               
                 JPEG Encoder 
                 Encoder 
               
               
                 JPEG Decoder 
                 Decoder 
               
               
                 H264 Encoder 
                 Encoder 
               
               
                 H264 Encoder 
                 Encoder (Baseline, Intra Only) 
               
               
                 Rectification and Lens Distortion 
                 Stereo Vision 
               
               
                 Correction 
                   
               
               
                 Census Transformation Unit (BRIEF) 
                 Stereo Vision 
               
               
                 Stereo Vision Depth Map Generator 
                 Stereo Vision 
               
               
                 Feature Point Detector (FAST) 
                 Feature Detection 
               
               
                 Feature Detection (Viola Jones) 
                 Face Detection (e.g.. Integral  
               
               
                   
                 Image, ISA Extension) 
               
               
                 Feature Detection (Optical Flow) 
                 Facial Tracking 
               
               
                 Feature Point Extractor  
                 Feature Detection - Difference of 
               
               
                 (DoG + SIFT) 
                 Gaussian plus Scale Invariant  
               
               
                   
                 Feature Transform 
               
               
                 Feature Extraction 
                 Edge Extraction (Sobel, Canny) 
               
               
                 Clock and Interrupt Manager 
                 System Control 
               
               
                 Debug Support Unit 
                 Debug 
               
               
                 GP IO Unit 
                 General Purpose 
               
               
                 3D convolution accelerator for  
                 Processing 
               
               
                 neural networks 
                   
               
               
                 Functional Unit 
                 Application 
               
               
                 Arithmetic Units for Deep Learning 
                 Processing Support 
               
               
                 Acceleration 
               
               
                   
               
            
           
         
       
     
     In the configurable accelerator framework (CAF)  400  of  FIG.  4   , eight convolution accelerators  600  are represented, CA0 to CA7. In other CAF  400  embodiments, a different number of convolution accelerators are formed. The number of convolution accelerators  600  and the particular features available in each convolution accelerator  600  are, in some cases, based on parameter values selected by a semiconductor practitioner at design time. 
     The convolution accelerators (CA&#39;s)  600  are data processing units with a selected number (e.g., one, two, four, eight) of input and output stream link ports. One or more configuration registers (e.g., a set of configuration registers) are arranged to control operations of the CA  600 . In some cases, configuration registers are included in the CAF control registers  402 , and in these or other cases, certain configuration registers are formed as part of the CA  600 . 
     One or more convolution accelerator template modules may be included in an IP modules library such as the library described with respect to Table 2. In these cases, data stored in the IP modules library includes relevant building blocks that reduce the work required to build a new accelerator that implements an accelerator&#39;s core functionality. A predefined set of configuration registers can be extended. Configurable FIFOs formed or otherwise located at the stream link ports can be used to absorb data rate fluctuations and provide some buffering margin required to relax certain flow control constraints in a processing chain. 
     Typically, each CA  600  either consumes data, generates data, or both consumes data and generates data. Data that is consumed passes through a first stream link of the reconfigurable stream switch  500 , and data that is streamed passes through a second stream link of the stream switch  500 . In at least some embodiments, CA&#39;s have no direct access to memory address space accessible by the primary communications bus  132  ( FIG.  3   ), the secondary communications bus  166  ( FIG.  3   ), or other bus addresses. However, if random memory access to data passed on a system bus is required, a CA  600  may also use an optional bus port interface, which may be along the lines of the system bus interface module  404  of  FIG.  4   , which is used for several things including permitting DMA engines to access memory locations on the system bus. As discussed above, some CA  600  implementations are part of a library, which can be used in other CAF  400  embodiments to simply instantiate the CA  600  in a global system definition file. 
     One or more arithmetic unit for deep learning acceleration template modules may also be included in the IP modules library such as the library described with respect to Table 2. Here, the predefined set of configuration registers can be further extended to provide parameter storage for configuration of the included arithmetic units. The parameters are associated with configuration of any desirable number of multiplexor circuits, multiplier circuits, adder circuits, temporary storage circuits, data shift circuits, and other circuits. 
     Each arithmetic unit for deep learning acceleration  700  is solely dedicated to execution of the formula in Equation 1. Scalar data, vector data, streaming data, constant data, interleaved data, and any other desirable data available within the CAF  400  framework may be passed into the arithmetic units for deep learning acceleration as operands, and generated resultant data will be passed out from the respective arithmetic unit. The data passed into the arithmetic unit for deep learning acceleration  700  may be sourced from a stream switch  500 , a memory inside or outside of the CAF  400  frame work, a sensor or particular interface, or from some other source. Along these lines, each of these types of data sources may, in some cases, consume the data generated in the arithmetic unit. As discussed herein, some arithmetic unit for deep learning acceleration  700  implementations are part of a library, which can be used in other CAF  400  embodiments to simply instantiate the arithmetic unit in a global system definition file. 
     System level programmers of machine learning systems desire flexibility to choose a desirable programming model for their particular implementation. To support this high level of flexibility, the CAF  400  is arranged with a reconfigurable stream switch  500 . As described in the present disclosure, the stream switch  500  acts as a data transfer fabric to improve logic block (IP) reuse, data reuse, and the reuse of other components and logic, which in turn allows a reduction of on-chip and off-chip memory traffic, and which provides a much greater flexibility to leverage the same logic blocks in different application use cases. Integrated in the stream switch  500  is a plurality of unidirectional links arranged to transport data streams via a configurable fully connected switch to, from, and to and from different kinds of data sources, data sinks, and data sources and data sinks such as direct memory access (DMA) controllers, I/O interfaces (e.g., cameras), and various types of accelerators. 
     The transported data may take any desired format such as a stream of raster scan image frames, a stream of macroblock oriented images, audio streams, raw data blocks, a stream of input or output arithmetic unit values, or any other format. The stream switch  500  can also transport messages, commands, or other like control information along a processing chain forwarded by each unit to one or more or more targeted units where the control information is processed. The control information may be used to signal events, to reconfigure the processing chain itself, or to direct other operations. 
       FIG.  5    is a stream switch embodiment  500  in more detail. The stream switch  500  includes a user-selectable, design-time configurable first number of stream link input ports  504  and a user-selectable, design-time configurable second number of stream link output ports  516 . In some cases, there is the same number of input ports as there are output ports. In other cases, there are more input ports than output ports, and in still other cases, there are more output ports than input ports. The number of input ports and the number of output ports are defined at design time. 
     In the stream switch  500  embodiment of  FIG.  5   , one stream link  502  embodiment is shown in detail. Other stream links  502   a ,  502   b , are also illustrated without detail for simplicity in the illustration. The stream links  502   a ,  502   b  are generally arranged along the lines of the stream link  502 , and for the sake of clarity in the disclosure any of the illustrated stream links may be identified as stream link  502 . 
     At run-time, stream switch  500  communicatively couples input stream link ports to output stream link ports through a stream link  502  according to configuration data written to certain ones of the CAF control registers  402  ( FIG.  4   ). In the embodiment, one or more of the input stream link ports  504  may be desirably arranged to concurrently forward received data streams to one or multiple (multicast) output ports  516  on the same clock cycle. Thus, one input stream link port can be communicatively coupled (e.g., electrically connected for the passage of data) to one or more output stream link interfaces, which results in a physical replication of the input data stream. The stream link  502  provides a straightforward, unidirectional interface to transport data streams and control information associated with the data streams. In such embodiments, a single control signal, which may in some cases be propagated on a single dedicated or shared data path, provides flow control. 
     Some conductors of the stream link are used to pass data; some other conductors may include a data validity indicator, a first pixel indicator, a last pixel indicator, a line type definition, and a stall signal. The stall signal is used as a back pressure (e.g., flow control) mechanism. In some embodiments of the stream link, image data, command data, control information, messages, and the like are passed in a frame-based protocol along the processing chain though the stream switch  500 . 
     In the stream switch  500 , each output port  516  is associated with a particular stream link  502 . In  FIG.  5   , for example, output port X is associated with stream link  502 . In addition, one or more input ports  504  are associated with each stream link. In some cases, for example, each and every input port  504  is associated with each and every stream link  502 . In this way, each input port  504  may pass data to any and all output ports  516  at the same time or at different times. 
     Individual communication path conduits of the stream link are unidirectional. That is, signals on each communication path conduit flow in only one direction. In some cases, a plurality of communication path conduits unidirectionally accept data received from an input port and pass the data to one or more output ports. In these cases, and in other cases, a single communication path conduit unidirectionally receives command information (e.g., flow control information) from an output port and passes the command information to one or more input ports. In some other cases, the command information received from an output port and passed to one or more input ports is passed on two or more communication path conduits. 
     As shown in the detailed stream link  502  of  FIG.  5   , the set of unidirectional communication path conduits from a plurality of input ports  504  are passed into a data switch  506 . In some cases, the set of unidirectional communication path conduits from every input port  504  are passed into the data switch  506 . In other cases, the unidirectional communication path conduits of one or more, but less then all, input ports  504  are passed into a data switch  506  of a particular stream link  502 . The data switch  506  may include multiplexor logic, demultiplexor logic, or some other form of switching logic. As shown in  FIG.  5   , data passed into stream link  502  from a plurality of input ports  504  may be concurrently present at input nodes of the data switch  506 . A selection mechanism  508  is arranged to determine which input data is passed through the data switch  506 . That is, based on the selection mechanism  508 , the input data from one of input ports A, B, C, D is passed through the data switch  506  to an output of the data switch  506 . The output data will be passed on N A . . . D  unidirectional communication path conduits, which will match the number of unidirectional communication path conduits of the selected input port. 
     The selection mechanism  508  is directed according to stream switch configuration logic  510 . The stream switch configuration logic  510  determines at run time which input port  504  shall supply data to the associated output port, and based on the determination, the stream switch configuration logic  510  forms an appropriate selection signal that is passed to the data switch  506 . The stream switch configuration logic  510  operates at run time and in real time. The stream switch  510  may take direction from CAF control registers, from a DSP of the DSP cluster  122  ( FIG.  3   ), from the application processor  128 , or from some other control device. In addition, the stream switch configuration logic  510  may also take direction from message/command logic  512 . 
     In some embodiments, data is passed uniformly through each particular stream link  502 . That is, in some cases, one stream link  502  is configured (e.g. stream switch configuration logic  510 , CAF control registers, or the like) to cooperatively pass any number N of first datums (e.g., bits, bytes, words, nibbles, tuples, or some other data samples, etc.), and one or more other stream links  502  are similarly configured to pass corresponding second datums. In this configuration, for each datum passed through the first stream link  502 , there is a corresponding datum passed through each of the other one or more stream links  502 . 
     In other embodiments, data is not passed uniformly through each particular stream link  502 . Data may be interleaved, for example, or passed in another non-uniform way. In an interleaved embodiment, the various stream links  502  may be configured to interleave data. In one such interleaved example, a first stream link  502  may be arranged to pass “M” datums from a first source (e.g., input port  504 ), and then the first stream link  502  may be arranged to pass “N” datums from a second source (e.g., a different input port  504 ). 
     Alternatively, in yet one more interleaving embodiment, two stream links  502  may be arranged to pass different numbers of datums in a non-uniform way. That is, while a first stream link  502  is passing “M” datums, a second stream link  502  is simultaneously or concurrently passing “N” datums. In the examples, described herein, “M” and “N” are integers. In some cases, “M” and “N” are different integers. 
     In some stream switch  500  embodiments, certain specific messages that are passed through an input port  504 , for example by an interface or an accelerator, are recognized by command logic  512  in one or more stream links  502  of the stream switch  500  and used to reprogram one or more stream links  502  in real time. In these or in other embodiments, the stream switch  500  is configured to merge data streams according to fixed patterns. For example, in at least one case, a stream switch  500  may be arranged to select and pass data to an output port  516  by switching between input streams passed on two or more input ports  504 . For example, after each line, each frame, each N transactions, or by some other measure, the stream switch  500  may be configured to pass data from a different input port  504  to a selected output port  516 . 
     Data passed from the data switch  506  may, in some cases, pass through one or more optional output synchronization logic stages  514 . The output synchronization logic stages  514  may be used to store or otherwise buffer a selected amount (e.g., one or more bits, a few or many bytes, etc.) of data passed from a data source coupled to an input port  504  toward a data sink device coupled to an output port  516 . Such buffering, synchronizing, and other such operations may be implemented when data source devices and data sink devices operate at different rates, different phases, using different clock sources, or in other manners that may be asynchronous to each other. 
     The stream switch  500  includes a back pressure stall signal mechanism, which is used to pass flow control information from a sink device to a source device. The flow control information is passed from a sink device to inform a data stream source device to lower its data rate. Lowering the data rate will help to avoid a data overflow in the sink device. 
     One portion of the back pressure stall signal mechanism includes a back pressure stall signal path that is included in each input port. The back pressure stall signal path is arranged as a back pressure unidirectional communication path conduit. In  FIG.  5   , four back pressure input port mechanisms are illustrated, BP A , BP B , BP C , BP D ; one each for each of the illustrated input ports. In other embodiments, the back pressure mechanism of each input port may include one or more unidirectional communication path conduits. In some embodiments, the back pressure mechanism of each input port has the same number of unidirectional communication path conduits, which may be, for example, a single conduit. In these cases, for example, when a data source device coupled to the particular input port detects that a signal on the back pressure mechanism is asserted, the particular data source device will slow or stop the amount of data passed to the associated input port. 
     Each output port  516  includes another portion of a back pressure mechanism. One output port back pressure mechanism for each of the three illustrated output ports X, Y, Z, of  FIG.  5    are illustrated, BPx, BPy, BPz. In some cases, each output port back pressure mechanism includes a same number of unidirectional communication path conduits (e.g., one). In other cases, at least one output port has a back pressure mechanism with a different number of unidirectional communication path conduits than another back pressure mechanism of another output port. 
     The output port back pressure mechanism conduits are passed to combinatorial back pressure logic  518  in each stream link  502 . In  FIG.  5   , back pressure logic  518  receives back pressure control signals BPx, BPy, BPz. The combinatorial back pressure logic  518  also receives control information from the stream switch configuration logic  510 . The combinatorial back pressure logic  518  is arranged to pass relevant flow control information back through the input port back pressure mechanism of an input port  504  to a particular data source device. 
       FIG.  6    is a convolution accelerator (CA) embodiment  600 . The CA  600  may be implemented as any one or more of the convolution accelerators  600  of  FIG.  4   . 
     The CA  600  includes three input data interfaces and one output data interface that are each arranged for coupling to a stream switch  500  ( FIG.  5   ). A first CA input data interface  602  is arranged for coupling to a first stream switch output port  516 , a second CA input data interface  604  is arranged for coupling to a second stream switch output port  516 , and a third CA input data interface  606  is arranged for coupling to a third stream switch output port  516 . A CA output data interface  608  is arranged for coupling to a selected stream switch input port  504 . The specific stream switch  500  port that each CA input data interface  602 ,  604 ,  606  and output data interface  608  is coupled to may be determined by default, at boot time, or at run time, and the specific coupling may be programmatically changed at run time. 
     In an exemplary embodiment, the first CA input data port  602  is arranged to pass a stream of batch data into the CA  600 , the second CA input data port  604  is arranged to pass a stream of kernel data into the CA  600 , and the third CA input data port  606  is arranged to pass a stream of feature data into the CA  600 . The output data port  608  is arranged to pass an output data stream from the CA  600 . 
     The CA  600  includes several internal memory buffers. The internal memory buffers may share a common memory space in some embodiments. In other embodiments, some or all of the internal memory buffers may be separate and distinct from each other. The internal memory buffers may be formed as registers, flip flops, static or dynamic random access memory (SRAM or DRAM), or in some other structural configuration. In some cases, the internal memory buffers may be formed using a multiport architecture that lets, for example, one device perform data “store” operations in the memory while another device performs data “read” operations in the memory. 
     A first CA internal buffer  610  is physically or virtually arranged in line with the first CA input data interface  602 . In this way, batch data streamed into the CA  600  may be automatically stored in the first CA internal buffer  610  until the data is passed to a particular math unit in the CA  600  such as an adder tree  622 . The first CA internal buffer  610  may be fixed with a size that is determined at design time. Alternatively, the first CA internal buffer  610  may be defined with a variable size that is determined programmatically at boot time or run time. The first CA internal buffer  610  may be 64 bytes, 128 bytes, 256 bytes, or some other size. 
     A second CA internal buffer  612  and a third CA internal buffer  614  are formed along the lines of the first CA internal buffer  610 . That is, the second and third CA internal buffers  612 ,  614  may each have their own fixed size that is determined at design time. Alternatively, the second and third CA internal buffers  612 ,  614  may have a variable size that is determined programmatically at boot time or run time. The second and third CA internal buffers  612 ,  614  may be 64 bytes, 128 bytes, 256 bytes, or some other size. The second CA internal buffer  612  is physically or virtually arranged in line with the second CA input data interface  604  to automatically store streamed kernel data until the kernel data is passed to a dedicated fourth CA internal buffer  616  that is dedicated to storing kernel buffer data. The third CA internal buffer  614  is physically or virtually arranged in line with the adder tree  622  to automatically store summed data until it can be passed through the CA output interface  604 . 
     The fourth CA internal buffer  616  is a dedicated buffer arranged to desirably store kernel data and apply the stored kernel data to a plurality of CA multiply-accumulate (MAC) units  620 . 
     The fifth CA internal buffer  618  is a feature line buffer that is arranged to receive streamed feature data passed through the third CA input interface  606 . Once stored in the feature line buffer, the feature data is applied to the plurality of CA MAC units  620 . Feature and kernel buffer data applied to the CA MAC units  620  is mathematically combined according to the convolutional operations described herein, and the resulting output products from the CA MAC units  620  are passed to the CA adder tree  622 . The CA adder tree  622  mathematically combines (e.g., sums) the incoming MAC unit data and batch data passed through the first CA input data port. 
     In some cases, the CA  600  also includes an optional CA bus port interface  624 . The CA bus port interface  624 , when it is included, may be used to pass data into or out from the CA  600  from SoC global memory  126  or some other location. In some cases, the applications processor  128 , a DSP of the DSP cluster  122 , or some other processor directs the passage of data, commands, or other information to or from the CA  600 . In these cases, the data may be passed through the CA bus port interface  624 , which may itself be coupled to the primary communications bus  132 , the secondary communication bus  166 , or some other communications structure. 
     In some cases, the CA  600  may also include CA configuration logic  626 . The CA configuration logic  626  may be fully resident with the CA  600 , partially resident with the CA  600 , or remote from the CA  600 . The configuration logic  600  may, for example, be fully or partially embodied in the CAF control registers  402 , the SoC controllers  120 , or some other structures of the SoC  110 . 
       FIGS.  7 A- 7 F  may collectively be referred to herein as  FIG.  7   . 
     Embodiments of an arithmetic unit for deep learning acceleration  700 , configured in one way or another for a particular purpose, may be interchangeably referred to herein as any one of arithmetic unit for deep learning acceleration  700 A to  700 F. 
       FIG.  7 A  is a first high level block diagram  700 A illustrating certain data paths supported by arithmetic units for deep learning acceleration  700  in a neural network. It has been recognized that many supporting operations in a neural network, and particularly in a convolutional neural network arranged for deep learning, can be accomplished with one or more affine transformations. In such transformations, a plurality of related and co-equal data points organized as a frame are co-linearly transformed and translated by a scalar, a vector, a plurality of scalars or vectors, or the like. These transformations may be applied, for example, in image processing to accomplish rotation, scale, shear, or the like. And along these lines, these affine transformations may be applied to support deep learning sub-processes such as biasing, batch normalization, scaling, mean subtraction, element-wise addition, and other linear combinations of vector type operations such as max-average pooling, and the like. 
     In  FIG.  7 A , an arithmetic unit for deep learning acceleration  700  is structured in hardware circuitry to perform Equation 1. 
         AX+BY+C →Output  (1)
 
     The arithmetic unit for deep learning acceleration  700  can be configured and directed to perform certain data path computational operations that support memory bandwidth intensive state-of-the-art convolutional neural network algorithms. Rather than performing these operations with digital signal processors (DSPs), for example, which has heretofore been conventional, these data path computations can instead be merged and mapped onto one or more arithmetic units for deep learning acceleration  700 . The arithmetic unit for deep learning acceleration  700  executes the affine transformation according to the algorithm that defines the operations of the convolutional neural network. 
     The hardware circuitry of the arithmetic unit for deep learning acceleration  700  includes dedicated circuits to retrieve data, accept data, route data, multiply operands to produce products, add values to produce sums, shift values right or left by any number of places, combine data, serialize data, interleave data, and perform other like operations. The arithmetic unit for deep learning acceleration  700  may include registers, volatile memory arrays, non-volatile memory arrays, buffers, and other types of data repositories. Each arithmetic unit for deep learning acceleration  700  may have one or more circuits to perform the operations. For example, Equation 1 calls out two multiplication operations. An arithmetic unit for deep learning acceleration  700  in some embodiments may include a single multiplication circuit that is used and re-used to carry out the two multiplication operations. Alternatively, the arithmetic unit for deep learning acceleration  700  in other embodiments may include two or more multiplication units. Along these lines some embodiments of the arithmetic unit for deep learning acceleration  700  may include a single adder circuit, and other embodiments include two or more adder circuits. 
     In  FIG.  7 A , the arithmetic unit for deep learning acceleration  700  is arranged with a first stream input X  702  and a second stream input Y  704 . The first and second stream inputs  702 ,  704  may be selectively coupleable through the stream switch  500  to any type of streaming data source such as a convolutional accelerator  600 , an external device interface  408  (e.g., an image sensor interface), a direct memory access (DMA) engine  406 , or some other streaming data source. As illustrated in  FIG.  7 A , the first and second stream inputs  702 ,  704  are arranged to provide scalar data, vector data, or scalar and vector data that are processed as the “X” and “Y” operands, respectively, in Equation 1. 
     In some cases, the arithmetic unit for deep learning acceleration  700  is arranged to receive constant vector data from a vector constant memory  706 . The constant vector data may be provided once per iteration of Equation 1, and in other cases the vector constant data may be provided multiple times such as once for each iteration of Equation 1. That is, the constant vector data may be provided statically or as a stream of data that corresponds to streaming data X, streaming data Y, streaming data X and Y, interleaved data X and Y, or in some other way. In some cases, constant vector data is provided on one or more inputs as a single static datum for multiple iterations of Equation 1, and other constant vector data is provided on one or more inputs as streaming dynamic constant data across the multiple iterations of Equation 1. 
     In the embodiment of  FIG.  7 A , a first constant input A  706 A passes the constant vector data that is processed as the “A” operand in Equation 1, a second constant input B  706 B passes the constant vector data that is processed as the “B” operand in Equation 1, and a third constant input C  706 C passes the constant vector data that is processed as the “C” operand in Equation 1. 
     The vector constant memory  706  may, in some cases, be coupled to address generation circuitry  708 . The address generation circuitry  708  is arranged to provide timed and sequenced addresses in the vector constant memory  706  repository so that constant vector data may be timely provided to the arithmetic unit for deep learning acceleration  700  at vector constant inputs A, B, C,  706 A,  706 B,  706 C, respectively. The address generation unit  708  may include auto-sequencing circuitry to increment addresses, decrement addresses, offset addresses, apply constant or variable strides, interleave or generate addresses in some other pattern, or perform still other memory address configuration operations. 
     A set of configuration parameter circuits  710  are provided in the first high level block diagram  700 A of  FIG.  7 A . The configuration parameter circuits  710  may include memory, registers, buffers, and other repository circuitry that direct operations of the arithmetic unit for deep learning acceleration  700 . In some cases, the configuration parameter circuits  710  are arranged as part of the CAF control registers  402  ( FIG.  4   ). 
     Exemplary parameter repositories (e.g., sub-circuits of configuration parameter circuits  710 ) include a repository for fixed point arithmetic shift constants  710 A, a repository for an operating mode  710 B, and a repository for scalar constants A, B, C,  710 C. 
     The fixed point arithmetic shift constants  710 A may include any number of right-shift values, left-shift values, or some other shift arrangement. The shift operations may be performed on multiply values, divide values, align values, scale values, sums, products, or shift data for another reason. 
     The operation mode  710 B may include circuitry that directs operations of the address generation unit  708 . Alternatively, or in addition, the operation mode  710 B repository stores data that directs operations of the arithmetic unit for deep learning acceleration  700 . The operation mode may include one or more parameters. The one or more parameters may, for example, direct selections lines of multiplexor circuits, provide constant values to various circuits in the arithmetic unit, direct operations of shift circuits (e.g., direction, number of positions, and the like), provide directions to combinatorial logic (e.g., input values, inversion circuitry, or the like), support these modules in different ways, and support other modules within the arithmetic unit for deep learning acceleration  700 . 
     The operation mode  710 B configuration parameter circuits may be arranged to work cooperatively with the address generation unit  708  in ways that flexibly support convolution operations of a machine learning algorithm. For example, operation mode  710 B parameters can be arranged to support feature input streams of any desirable shape. Because of this support, the feature volume is not limited to specific Width-by-Height-by-Depth (W×H×D) shapes as in at least some conventional cases. Instead, the operation mode  710 B configuration parameter circuits can direct the address generation unit  708  to support nearly any scalable and flexible three-dimensional (3D) tiling scheme and feature volume ‘walking’ scheme. By specifying the input geometry (i.e., W×H×D) of the feature volume, and by specifying the width-by-height-by-depth (w×h×d) dimensions of the 3D tile, and by specifying the walking order, the 3D tile can be walked through the feature volume in any order. For example, in one case, the 3D tile is walked through the feature volume across depth, followed by width, followed by height. In another case, the 3D tile is walked through the feature volume across width, followed by depth, followed by height. In a third case, the 3D tile is walked through the feature volume across depth, followed by height, followed by width. As evident in the examples of 3D tile walking, the feature volume can be navigated in many different ways by setting appropriate parameters in the operation mode  710 B configuration parameter circuits. Hence, the arithmetic unit for deep learning acceleration  700  can operate with incoming data in nearly any particular format. The address generation unit  708  is arranged to properly address the “A,” “B,” and “C” values to apply to the incoming data as specified by the input 3D tile walking order, the 3D tile geometry, and the feature volume geometry. 
     One advantage of the cooperative configuration parameter circuits  710  and the address generation unit  708  is that the convolution accelerator  600  that feeds the arithmetic unit can operate at any desirable granularity of data that is most acceptably optimal in terms of balancing performance, power, memory bandwidth, and data reuse; and in these cases, the convolution accelerator  600  can feed data directly to the arithmetic unit through stream switch  500  without any need to rearrange or reassemble the data in any other way. The machine learning algorithm specifies the expected 3D tile and feature volume input geometry using, for example, registers of the configuration parameter circuits  710  that cooperate with the address generation unit  708 . 
     In some cases, the address generation unit  708  may be implemented with a plurality of auto-reloading counters (e.g., four auto-reloading counters). The counters may be prioritized, cascaded, interleaved, or the counters may cooperate in other ways. Each of the counters may, for example, be used to “count” the directional walking of the 3D tile through the feature volume. For example, addresses can be auto-generated in the address generation unit  708  using values from the counters. By configuring the counters to cooperatively provide output values in a desired arrangement, a 3D tile can be controlled to “walk” across the height, width, and depth of the feature volume in the order specified by the machine learning algorithm. 
     Scalar constants A, B, C,  710 C may include bits, bytes, nibbles, words, or differently formed scalar values for application within the arithmetic unit for deep learning acceleration  700 . The scalar constants A, B, C,  710 C may include values stored in memory, buffers, registers, or some other repository. The circuitry forming the scalar constants A, B, C,  710 C may include timing circuitry, latches, interface structures, and the like. The scalar constants  710 C may be provided into the deep learning accelerator  700  in any desirable relationship with the streaming data X, Y, provided at first and second stream inputs  702 ,  702 , respectively, and with vector constant data provided at vector constant inputs A, B, C,  706 A,  706 B,  706 C. 
     An output  712  of the arithmetic unit for deep learning acceleration  700  is arranged to pass sums, products, or other data generated in the arithmetic unit for deep learning acceleration  700 . The output  712  may pass discrete values. Alternatively in these or other embodiments, the output  712  may pass a stream of values. 
     Having now described the arithmetic unit for deep learning acceleration  700  and supporting circuit modules arranged around the arithmetic unit for deep learning acceleration  700  in some embodiments, the disclosure now describes a non-limiting set of use-cases for the unit. There are many uses for the arithmetic unit for deep learning acceleration  700  in machine learning systems, however cases described herein are directed to deep learning for convolutional networks for brevity. In these cases, the arithmetic units for deep learning acceleration  700  are not used for convolution, but they are instead used to support the convolutional features with increased data locality. In this way, data remains streaming rather than moving into and out of memory, which is an improvement over known convolutional systems. That is, as convolutional processes are carried out in the convolution accelerators  600 , data is streamed from and to one or more CA  600  structures, through stream switch  500 , and in to or out from one or more arithmetic units for deep learning acceleration  700 . 
     For example, some operations used in deep learning using convolutional neural networks include, biasing operations, batch normalization operations, scaling operations, mean subtraction operations, element-wise addition operations, and other linear combinations of vector-type operations used in max-average pooling operations, for example. An exemplary biasing operation is depicted in Equation 2; an exemplary batch normalization operation is depicted in Equation 3; an exemplary scaling operation is depicted in Equation 4; an exemplary mean subtraction operation is depicted in Equation 5; an exemplary element-wise addition operation is depicted in Equation 6, and an exemplary max-average pooling operation is depicted in Equation 7. 
     
       
         
           
             
               
                 
                   
                     X 
                     + 
                     bias 
                   
                   → 
                   Output 
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
       
         
           
             
               
                 
                   
                     
                       ( 
                       
                         X 
                         - 
                         mean 
                       
                       ) 
                     
                     
                       vari 
                       ⁢ 
                       a 
                       ⁢ 
                       n 
                       ⁢ 
                       c 
                       ⁢ 
                       e 
                     
                   
                   → 
                   Output 
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
       
         
           
             
               
                 
                   
                     
                       ( 
                       
                         X 
                         * 
                         scale_factor 
                       
                       ) 
                     
                     + 
                     scale_bias 
                   
                   → 
                   Output 
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
       
         
           
             
               
                 
                   
                     
                       ( 
                       
                         X 
                         - 
                         mean 
                       
                       ) 
                     
                     * 
                     mean_scaling 
                     ⁢ 
                     _factor 
                   
                   → 
                   Output 
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
       
         
           
             
               
                 
                   
                     
                       X 
                       ⁢ 
                       1 
                     
                     + 
                     
                       X 
                       ⁢ 
                       2 
                     
                   
                   → 
                   Output 
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
           
         
       
       
         
           
             
               
                 
                   
                     
                       ( 
                       
                         A 
                         * 
                         X 
                       
                       ) 
                     
                     + 
                     
                       ( 
                       
                         B 
                         * 
                         Y 
                       
                       ) 
                     
                   
                   → 
                   Output 
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
     The nature of the operations represented in Equations 2-7 are vector operations depicted with “X” and “Y,” while constants such as “A,” “B,” “bias,” “mean,” “variance,” “scale factor,” and the like can be vectors, scalars, or a both vectors and scalars. What is more, considering the nature of the transformations involved in deep learning and supported by the arithmetic unit for deep learning acceleration  700 , the inventors have recognized that substantial benefits may be achieved by combining multiple operations into a single affine transform operating on one or two input streams. In this way, data is not processed in a first operation, stored, and the retrieved for a second operation. Instead, a first operation is performed on first data that passes through the stream switch  500 , and the second data that is output from the operation is passed through the stream switch  500  directly into a second operation. 
     For example, a first linear operation is depicted in Equation 8, where operands in the linear operation are formed by the sub-operations of Equations 9 and 10. 
     
       
         
           
             
               
                 
                   
                     
                       ( 
                       
                         A 
                         * 
                         X 
                       
                       ) 
                     
                     + 
                     B 
                   
                   → 
                   Output 
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
           
         
       
       
         
           
             
               
                 
                   
                     
                       ( 
                       scale_factor 
                       ) 
                     
                     
                       vari 
                       ⁢ 
                       a 
                       ⁢ 
                       n 
                       ⁢ 
                       c 
                       ⁢ 
                       e 
                     
                   
                   → 
                   A 
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
       
         
           
             
               
                 
                   
                     
                       
                         
                           ( 
                           
                             ( 
                             
                               bias 
                               - 
                               
                                 m 
                                 ⁢ 
                                 ean 
                               
                             
                           
                           ) 
                         
                         * 
                         A 
                       
                       ) 
                     
                     + 
                     scale_bias 
                   
                   → 
                   B 
                 
               
               
                 
                   ( 
                   10 
                   ) 
                 
               
             
           
         
       
     
     In this case, the affine transform realizes significant time savings because data is processed in a plurality of operations without requiring temporary storage of interim values. Instead, the “interim values” are passed through stream switch  500  into a concurrently operating arithmetic unit for deep learning acceleration  700 . In this way, several modes of operation are supported by the arithmetic units for deep learning acceleration  700 . That is, the arithmetic unit for deep learning acceleration  700  is structured for substantial flexibility providing any type of desirable constants, such as passing A, B, and C as vectors, scalars, or vectors and scalars in various use-cases. In one biasing case, for example, A and C are scalar constants where A is equal to one (i.e., A=1) and C is equal to zero (i.e., C=0), and B is a vector. This use case may, for example, be implemented where “length” is equal to the “depth” of an input feature three-dimensional (3D) volume. In another machine learning convolutional processing case where batch normalization is performed, C is equal to zero (i.e., C=0), and A and B are vectors. In mean subtraction embodiments with scaling, A and B are scalar constants and C is equal to zero (i.e., C=0). And in yet one more machine learning operation, that implements element-wise addition operations, scalar constants A and B are equal to one (i.e., A=1; B=1), so output is simply achieved as vector addition of X and Y (i.e., X+Y). 
     Various test cases have been performed by the inventors where, for example, vector data has been pre-loaded into vector constant memory  706  and scalar data is preloaded into the scalar constants A, B, C,  710 C repository. In other test cases, vector data is streamed in via stream switch  500  via vector constant input A  706 A, vector constant input B  706 B, and vector constant input C  706 C. In various ones of these test cases, where normalization, scaling, and biasing operations were performed, the inventors have realized two and one-half times (i.e., 2.5×) savings in parameter storage memory, faster execution of neural network convolutional operations, and reduced power consumption realized by avoiding intermediate memory storage of temporary data. Rather than storing five vectors, for example, only two vectors are stored and other parameters are streamed through the arithmetic units for deep learning acceleration  700 . 
       FIG.  7 B  is a second high level block diagram  700 B illustrating certain data paths supported by arithmetic units for deep learning acceleration  700  in a neural network. An algorithm implementing a deep learning program operating on an SoC  110  ( FIG.  3   ) embedded in a mobile device  100  configures the configurable accelerator framework  400  to cooperatively perform certain affine transformations using one or more arithmetic units for deep learning acceleration  700  to support concurrent convolution operations performed by one or more convolution accelerators  600 . Data locality is provided by passing data to and from the convolution accelerators  600  through stream switch  500 , and by concurrent processing in an arithmetic unit for deep learning acceleration  700  using data also passed through the stream switch  500 . One portion of the deep learning program in  FIG.  7 B  represents data paths to implement a max-average pooling operation. 
     A camera sensor, in one embodiment, provides a stream of image data  150  passed through a first image sensor interface and image signal processor (ISP) external device interface  408   b  and to a convolution accelerator  600 A via stream switch  500 . The convolution accelerator  600 A walks kernels of known data through the images in the stream of image data  750 . The data may be further processed in any number of additional convolution processes via convolution accelerator  600 A, another convolution accelerator, or some other structure of the CAF  400 . For the sake of simplifying the pertinent portions of the algorithm now under discussion, it is described that a stream of filtered image data  752  is passed from convolution accelerator  600  back through the stream switch  500  to a max pooling accelerator  600 B and an average pooling accelerator  600 C. 
     The max pooling accelerator  600 B produces a stream of max pooled image data  754  that is passed into the arithmetic unit for deep learning acceleration  700 , and the average pooling accelerator  600 C produces a stream of average pooled image data  756  that is also passed into the arithmetic unit for deep learning acceleration  700 . Both the stream of max pooled image data  754  and the stream of average pooled image data  756  are streams of vector data. In some embodiments, the stream of max pooled image data  754  corresponds to information passed on stream input X  702  ( FIG.  7 A ). In these or other embodiments, the stream of average pooled image data  756  corresponds to information passed on stream input Y  704  ( FIG.  7 A ). 
     In some cases, the streams of input data on stream input X and stream input Y are passed uniformly. That is, for each portion (e.g., datum) of max pooled image data  754  passed through stream input X  702 , a corresponding portion of average pooled image data  756  is passed through input Y  704 . In other cases, it may be desirable to pass data through stream inputs X, Y  702 ,  704 , non-uniformly. In at least one case, “M” data samples are passed through stream input X  702 , and “N” data samples are passed through stream input Y  704 . In this interleaved way, selected datums from the max pooled image data  754  can be desirably processed with selected datums from the average pooled image data  756 . Many other non-uniform data flow arrangements are contemplated; such non-uniformity may be enabled by parameters stored, for example, in CAF control registers  402  ( FIG.  4   ), configuration parameter circuits  710 , or some other source. 
     The configuration parameter circuits  710 , which may include a repository for vector constants A, B, C,  706 A,  706 B,  706 C or for scalar constants A, B, C  710 A,  710 B,  710 C, pass constant data  758  into the arithmetic unit for deep learning acceleration  700 . To assist the present discussion, exemplary data passed into arithmetic unit for deep learning acceleration  700  is represented as Equations 11-14. Specifically, the exemplary stream of max pooled image data  754  is depicted in Equation 11; the exemplary stream of average pooled image data  756  is depicted in Equation 12; and the constant data  758  is depicted in Equations 13 and 14. 
       MaxPoolingImageData={ mp 1, mp 2, . . .  mpn}   (11)
 
       AveragePoolingImageData={ av 1, av 2, . . .  avn}   (12)
 
       (ConstantData A )={ a 1, a 2 , . . . an}   (13)
 
       (ConstantData B )={ b 1, b 2, . . .  bn}   (14)
 
     To further assist the present discussion, the exemplary resulting data (i.e., the stream of max-average pooled data  760 ) that is generated by, and passed from, arithmetic unit for deep learning acceleration  700  is represented as Equation 15. In some embodiments, the stream of max-average pooled data  760  corresponds to information streamed on output  712  ( FIG.  7 A ). 
       MaxAvgPooledData={( a 1* mp 1+ b 1* av 1),( a 2* mp 2+ b 2* av 2) . . . ,( an*mpn+bn*avn )}  (15)
 
       FIG.  7 C  is a third high level block diagram  700 C illustrating convolutional accelerators and an arithmetic unit arranged for an exemplary branch-merge operation in a neural network. It has been recognized by the inventors that in some cases, known network topologies may often include branches in machine learning algorithm. Considering image recognition, for example, one first set of convolution operations may process feature map data with kernels devoted to a first color, and one second set of convolution operations may process the same feature map data with kernels devoted to a second color. By taking advantage of the configurable accelerator framework  400 , the convolution operations for both colors can occur in parallel. What is even more efficient, if the machine learning algorithm later desires to merge the convolution streams back together, the arithmetic unit for deep learning acceleration  700  can be used concurrently with the convolution operations to rejoin the streams as represented, for example, in  FIG.  7 C . 
     In the third high level block diagram  700 C, a camera sensor  408   b  provides streaming data to at least two convolution accelerators  600 A,  600 B, which perform convolution processing. In other cases, the input the convolution accelerators  600  may come from a different sensor, a different convolution accelerator  600 , memory, or some other device. In still other cases, any number of convolution accelerators  600  ( FIG.  6   ) may receive the input streaming data, or any other number of different devices may be communicatively coupled to the stream switch  500  to receive streaming data. In  FIG.  7 C , output  752 A,  752 B from the convolution accelerators  600 A,  600 B, respectively, is passed through stream switch  500  to the arithmetic unit for deep learning acceleration  700 . The arithmetic unit for deep learning acceleration  700  also receives input from vector constant memory  706  ( FIG.  7 A ), configuration parameter circuits  710  ( FIG.  7 A ), or some other data source (e.g., control registers  402  ( FIG.  4   )). Upon receiving input stream data from the plurality of branched sources, the arithmetic unit for deep learning acceleration  700  combines the data according to the particular formula described herein (i.e., AX 1 +BY 2 +C) and streams the resulting data  761  back through the stream switch  500 . It is shown that the streaming data is passed into a stream engine DMA engine  406  in the embodiment of  FIG.  7 C , but data may be streamed to other destinations in other cases. For example, in cases where more than two branches are merged, the output from the arithmetic unit for deep learning acceleration  700  may be streamed through the stream switch  500  to a second arithmetic unit for deep learning acceleration  700 , which also receives data streamed from a third branch. 
     The branch-merge operations illustrated in  FIG.  7 C  clearly illustrate more efficient operations than previously known. The branch-merge operation is performed with a single unit, and concurrently to other operations (e.g., convolution, bias, normalization, scaling, and any other operations) before the merge operation is conducted. In this way, data is re-used without having to be streamed into a remote memory device and streamed back out from the remote memory device. These operations save power, on-chip area, time, and the like. Desirably, the operations scale to perform as many concurrent operations as permitted by the available resources. 
       FIG.  7 D  is a first data flow diagram  700 D illustrating structures configurable to execute a single dedicated formula (i.e., AX+BY+C) with a arithmetic unit for deep learning acceleration  700  in a neural network. In view of the operations illustrated and described in the present disclosure, it can be appreciated that the arithmetic unit for deep learning acceleration  700  is highly configurable, and based on the configuration parameters, the arithmetic unit for deep learning acceleration  700  is capable of configuration for bias operations (e.g., (AX+C)), mean subtraction operations (e.g., (AX−C)), scaling/batch normalization operations (e.g., (AX+C)), element-wise addition/subtraction operations (e.g., (X+Y), (X−Y)), max-average pooling operations (e.g., (AX+BY)), branch-merge operation (e.g., (AX+BY+C)) and many other operations. The configuration may, for example, be supported using one or more of the configuration parameter circuits  710  ( FIG.  7 A ) such as the operation mode  710 B repositories. 
     The arithmetic unit for deep learning acceleration  700  in  FIG.  7 D  is solely dedicated to performance of a plurality of parallel operations wherein each one of the plurality of parallel operations carries out a portion of a formula, the formula being: output=AX+BY+C. The arithmetic unit includes a plurality of inputs  762  that are selectively coupleable to any one or more of the stream switch  500  ( FIG.  4   ), control registers  402  ( FIG.  4   ), vector constant memory  706  ( FIG.  7 A ), configuration parameter circuits  710  ( FIG.  7 A ), or some other data source. The arithmetic unit for deep learning acceleration  700  also includes multiplexor circuitry  764 , multiplier circuitry  766 , temporary storage circuitry  768 , data shift circuitry  770 , adder circuitry  772 , and post summation circuitry that may include latch circuitry  774  and configurable direction shift circuitry  776 . Information is streamed or otherwise passed from the arithmetic unit for deep learning acceleration  700  via an output  712 . In some cases, the resultant output data from the arithmetic unit is passed via output  712  to the stream switch  500 , a memory, or some other circuitry. 
     In some embodiments, the arithmetic unit for deep learning acceleration  700  circuitry of  FIG.  7 D  is formed in an integrated circuit. In some embodiments, the integrated circuit is arranged for convolutional neural network operations. The arithmetic unit circuitry may be formed in the same integrated circuit as the stream switch  500  and other circuits of the configurable accelerator framework  400  ( FIG.  4   ). In addition, or in the alternative, embodiments of the arithmetic unit for deep learning acceleration circuitry of  FIG.  7 D  may be formed in the same integrated circuit as other circuits of SoC  100  ( FIG.  3   ). 
     The arithmetic unit for deep learning acceleration  700  circuitry has one or more multi-input selection circuits  764  (e.g., multiplexors, multiplexor circuitry, multiplexor circuits, or the like) communicatively coupled to the plurality of inputs  762 . In the data flow diagram of  FIG.  7 D , a first multiplexor circuit  764 A is communicatively coupled to receive scalar data A from a scalar repository such as a register (e.g., scalar constants A, B, C,  710 C), and vector data A from a vector repository such as vector constant memory  706 , stream switch  500 , or some other vector data source. A selection input A-Type is directed to pass either the scalar data A or the vector data A through the first multiplexor circuit  764 A toward a first multiplier  766 A. The selection input A-Type may be communicatively coupled to and directed by data in the CAF control registers  402  ( FIG.  4   ), the configuration parameter circuits  710  ( FIG.  7 A ), or some other selection information source. 
     Also in the data flow diagram of  FIG.  7 D , a second multiplexor circuit  764 B is communicatively coupled to receive scalar data B from a scalar repository such as a register (e.g., scalar constants A, B, C,  710 C), and vector data B from a vector repository such as vector constant memory  706 , stream switch  500 , or some other vector data source. A selection input B-Type is directed to pass either the scalar data B or the vector data B through the second multiplexor circuit  764 B toward a second multiplier  766 B. The selection input B-Type may be communicatively coupled to and directed by data in the CAF control registers  402  ( FIG.  4   ), the configuration parameter circuits  710  ( FIG.  7 A ), or some other selection information source. 
     The data flow diagram of  FIG.  7 D  also depicts a third multiplexor circuit  764 C communicatively coupled to receive scalar constant data (e.g., “0”, “1”, or some other constant scalar data), scalar data C from a scalar repository such as a register (e.g., scalar constants A, B, C,  710 C), vector constant data (e.g., “0,0”; “0,1”; “1,0”; or some other constant vector data), and vector data C from a vector repository such as vector constant memory  706 , stream switch  500 , or some other vector data source. A selection input C-Type is directed to pass either the constant or variable scalar data C or the constant or variable vector data C through the third multiplexor circuit  764 C toward third data shift circuitry  770 C. The selection input C-Type may be communicatively coupled to and directed by data in the CAF control registers  402  ( FIG.  4   ), the configuration parameter circuits  710  ( FIG.  7 A ), or some other selection information source. 
     Based on the operation that the arithmetic unit for deep learning acceleration  700  is directed to perform, the plurality of inputs  762  are arranged to pass vector data into the arithmetic unit, scalar data into the arithmetic unit, or a combination of vector data and scalar data into the arithmetic unit. 
     The multiplier circuitry  766  is arranged as a plurality of multiplier circuits. A first multiplier circuit  766 A, and a second multiplier circuit  766 B are represented in  FIG.  7 D , but some other number of multiplier circuits may instead be formed. For example, in some cases, a single multiplier circuit is arranged in the arithmetic unit that is shared or otherwise used. In other cases, a plurality of three or more multiplier circuits is included. Each multiplier circuit is arranged to accept multiplicand data, multiplier data, or both multiplicand data and multiplier data from the multiplexor circuitry. In some cases, such as in the embodiment of  FIG.  7 D , multiplicand data, multiplier data, or both multiplicand data and multiplier data is passed directly into the multiplier circuitry  766  without first passing through the multiplexor circuitry  764 . The multiplier circuitry  766  is arranged to perform at least some multiplication operations of the formula implemented by the arithmetic unit for deep learning acceleration  700 . As arranged in  FIG.  7 D , a first multiplier circuit  766 A receives either scalar A data or vector A data and stream input X vector data, which may come, for example, from stream switch  500  ( FIG.  5   ). As further arranged in  FIG.  7 D , a second multiplier circuit  766 B receives either scalar B data or vector B data and stream input Y vector data, which may come, for example, from stream switch  500  ( FIG.  5   ). The multiplier circuitry  766  produces a product when so provided with multiplicand and multiplier data. 
     The product data produced by the multiplier circuitry  766  is passed into temporary storage circuitry  768 . The temporary storage circuitry  768  may be arranged as one or a plurality of temporary storage repositories. For example, product data from multiplier  766 A is passed into a temporary storage circuit  768 A, which may be a register, and product data from multiplier  766 B is passed into a temporary storage circuit  768 B, which may be a register. 
     Data passed through the third multiplexor circuit  764 C is received at third data shift circuitry  770 C. Based on the operations directed in the particular machine learning algorithm, the data received at third data shift circuitry  770 C may be shifted by a particular amount passed via input C-Shift. The parameter data passed into C-Shift may be retrieved from CAF registers  402  ( FIG.  4   ), configuration parameter circuits  710 , or some other source. In some cases, where the “C” operand in the formula (e.g., AX+BY+C) is constant, the third data shift circuitry  770 C may be disabled, optionally left out of the arithmetic unit, or otherwise rendered out of the data path. Shifted or otherwise desirable data passed through the third multiplexor circuit  764 C is optionally stored in a third temporary storage circuit  768 C. The optional storage may be implemented or otherwise used to align data operations according to a pipeline structure, a sequencing model, or for some other reasons. 
     Product data from the first multiplier circuit  766 A may be optionally stored in a first temporary storage circuit  768 A, and product data from the second multiplier circuit  766 B may be optionally stored in a second temporary storage circuit  768 B. The optional storage may be implemented or otherwise used to align data operations according to a pipeline structure, a sequencing model, or for some other reasons. In the data path of  FIG.  7 D , data passed from the first multiplier represents the “AX” product in the formula (e.g., AX+BY+C), and data passed from the second multiplier represents the “BY” product in the formula (e.g., AX+BY+C). 
     The data passed from first and second multipliers  766 A,  766 B, respectively, and through optional first and second temporary storage circuits  768 A,  768 B, respectively, is received at first and second data shift circuitry  770 A,  770 B, respectively. The product data may be shifted by a particular amount passed via input AX-Shift and BY-Shift inputs, respectively. The parameter data passed into AX-Shift and BY-Shift inputs may be retrieved from CAF registers  402  ( FIG.  4   ), configuration parameter circuits  710 , or some other source. 
     Adder circuitry  772  is arranged to sum data from the AX data path, the BY data path, and the C data path. The adder circuitry  772  may be formed as a single summing circuit, cascaded summing circuits, sequenced summing circuits, or in any other arrangement. In the data path of  FIG.  7 D , adder circuitry  772   n  is communicatively coupled to receive data from first data shift circuitry  770 A, second data shift circuitry  770 B, and third temporary storage circuitry  768 C. The adder circuitry  772   n  may be directed by an algorithm to perform at least some summation operations of the (AX+BY+C) formula. 
     In some cases, the sum data from the adder circuitry  772  is directly passed out from the arithmetic unit for deep learning acceleration  700 . In other cases, determined post-summation circuitry may be included in the arithmetic unit. In the data path of  FIG.  7 D , the post-summation circuitry includes latch circuitry  774  and directional shift circuitry  776 . These circuits may be optional. In some cases, the data produced by a formula that is implemented in the arithmetic unit is captured in a latch circuit  774   n  for timing purposes, alignment purposes, sequencing purposes, or for other purposes. And a directional shift circuit  776   n  is arranged to bit-wise align the output data from the adder circuitry  772   n , which may have optionally been captured in latch  774   n.    
     The directional shift circuitry  776  may be directed by shift direction parameters (i.e., “Shift-Direction” in  FIG.  7 D ) and shift amount parameters (i.e., “Result-Shift” in  FIG.  7 D ). The directional shift of the output sum data may be for timing alignment, bit-wise data alignment, normalization, scaling, or for some other reason. Data passed from the post-summation circuitry is communicated via output  712  through stream switch  500 , memory, or some other circuitry. 
     In some cases, the shift values (e.g., Result-Shift, Shift-Direction) are used to shift intermediate values (e.g., any one or more of AX, BY, C) and additionally or alternatively used to shift final values to provide flexible fixed point computations. With these parameters, the machine learning algorithm can flexibly use nearly any fixed point math format with selectable or variable integer and fractional parts for each value that is input to or passed out from the arithmetic units for deep learning acceleration  700 . The shift values permit alignment of intermediate results (e.g., any one or more of AX, BY, C) before the mathematical operation. In these or other cases, Result-Shift parameter permits generation of values the meet the output fixed point format directed by the machine learning algorithm. In some embodiments of the data flows of  FIGS.  7 D- 7 F , internal values are limited to “left” shifting, but in other embodiments, the shift direction may be selected. 
     In one exemplary case of use, a plurality of the arithmetic units for deep learning acceleration  700  are arranged according to an (AX+BY+C) data path. The data path is implemented as a set of acts of a particular machine learning method. For example, the method may be a portion of a deep convolutional neural network procedure or otherwise arranged to implement such a procedure. The procedure may be carried out by SoC  100  to identify at least one feature in certain input data, such as, for example, identifying a particular feature in an image or set of images to an acceptable level of certainty. 
     In the procedure to identify a feature in a set of streaming image sensor data, a feature volume is defined in the image sensor data. The feature volume has a feature volume height, a feature volume width, and a feature volume depth. An input tile is also defined having an input tile height, an input tile width, and an input tile depth. The input tile is walked through the feature volume. Such a process includes streaming the image sensor data through the reconfigurable stream switch  500  to a convolution accelerator unit  600 . In the convolution accelerator unit  600 , the image sensor data is convolved to produce a stack of kernel maps. 
     Also in the procedure, a first convolution accelerator  600  is configured to perform a max pooling operation, a second convolution accelerator  600  is configured to perform an average pooling operation, and an arithmetic unit for deep learning acceleration  700  is configured to perform a max-average pooling operation (e.g., AX+BY+0). The arithmetic unit is solely dedicated to performance of the mathematical operations that can be implemented according to the (AX+BY+C) formula. In yet other cases, the arithmetic units for deep learning acceleration  700  may be configured to perform bias operations, mean operations, scaling operations, branch merge operations, or still other operations. 
     Continuing the procedure, the stack of kernel maps is streamed through the reconfigurable stream switch  500  to the first and second convolution accelerators  600 . The max pooling operation is performed with the first convolution accelerator  600 , and max pool data is streamed through the reconfigurable stream switch  500  as the input data to the arithmetic unit for deep learning acceleration  700 . The average pooling operation is performed with the second convolution accelerator  600 , and average pool data is streamed through the reconfigurable stream switch  500  as the input data to the arithmetic unit for deep learning acceleration  700 . The arithmetic unit for deep learning acceleration  700  performs the max-average pooling operation, and streaming max-average pool data is passed through the reconfigurable stream switch. 
     In more detail, considering the procedures now under discussion, streaming input data is passed through a reconfigurable stream switch  500  ( FIG.  5   ) to convolution accelerators  600  and to the arithmetic units for deep learning acceleration  700 . The arithmetic units are solely dedicated to performance of a formula, the formula being: output=AX+BY+C. Streaming data passed into the arithmetic unit may be, for example, streaming image sensor data (e.g., camera sensor data  408 B in  FIG.  7 B ) from an image sensor coupled to the stream switch  500 , interim data generated during a machine learning process (e.g., during a convolution process), or some other data. 
       FIG.  7 E  is a second data flow diagram  700 E illustrating one-by-one (1×1) support operations of an arithmetic unit for deep learning acceleration  700  in a neural network. Final stage circuitry  778  at the output of the second data flow diagram  700 E is represented in  FIG.  7 F . 
       FIG.  7 F  is a third data flow diagram  700 F illustrating operations of an arithmetic unit for deep learning acceleration  700  in a neural network in a final stage  778  of a convolutional process. 
     The final stage circuitry  778  in  FIG.  7 E  is in, at least some embodiments, a second arithmetic unit for deep learning acceleration  700  cascaded to the first arithmetic unit for deep learning acceleration  700  depicted in  FIG.  7 E . This second arithmetic unit depicted in  FIG.  7 F  is configured differently from the first arithmetic unit depicted in  FIG.  7 E . 
     A description of structures in  FIG.  7 E  that correspond to structures in  FIG.  7 D  may be introduced without repeating a further detailed description for brevity. The arithmetic unit includes a plurality of inputs  762  that are selectively coupleable to the stream switch  500  ( FIG.  4   ) or another data source. The arithmetic unit for deep learning acceleration  700  includes multiplexor circuitry  764 , multiplier circuitry  766 , temporary storage circuitry  768 , data shift circuitry  770 , and final stage circuitry  778 . Other circuitry of the arithmetic unit (e.g., adder circuitry, post summation circuitry, latch circuitry, and the like) are optionally not used in the one-by-one support operations and not shown in  FIG.  7 E . Information is streamed or otherwise passed from the arithmetic unit for deep learning acceleration  700  via output  712  to the stream switch  500 , a memory, or some other circuitry. 
     Two additional multiplexor circuits  764 D,  764 E in  FIG.  7 E  are arranged to selectively pass, uniformly (i.e., data at stream X has a one-to-one correspondence with data at stream Y) or non-uniformly (e.g., interleaved or some other non-uniform arrangement), streaming vector data, such as from stream input X  702  and stream input Y  704 , to respective multiplier circuitry  766 . Based on an algorithm that selects which products are directed for generation, an XY-Mode selection signal will direct passage of either constant data A or streaming vector data Y through multiplexor circuit  764 D to multiplier  766 A. Along these lines, the same or a different machine learning algorithm can select which products are directed for generation, using a Cony-Mode selection signal to direct passage of either streaming vector data X or streaming vector data Y to multiplier  766 B. Yet additional programming choices are available to a machine learning algorithm that will direct product generation in a multiplier circuit  766 C of constant scalar or vector data with streaming vector data X. The algorithm may further direct, via the Cony-Mode signal, whether the product data from multiplier circuit  766 C or whether the constant scalar or vector data will be passed through a multiplexor circuit  764 F to the third temporary storage circuitry  768 C. 
     In the arithmetic unit of  FIG.  7 E , multiplier-produced product data will be stored in the first and second temporary storage circuitry  768 A,  768 B. Shift circuitry  770  coupled to the temporary storage circuits  768  is controlled by a machine learning algorithm to arrange the data in any desired way. In some cases, of course, no shifting is necessary. The data from the shift circuitry  770  is passed through final stage circuitry  778  before passage on output  712  to the stream switch  500  ( FIG.  5   ), a storage repository such as a memory arranged for vector storage, or some other circuitry. 
     A description of structures in the arithmetic unit for deep learning acceleration  700  of  FIG.  7 F  that correspond to structures in  FIG.  7 D  may be introduced without repeating a further detailed description for brevity. The arithmetic unit includes a plurality of inputs  762  that are selectively coupleable to the stream switch  500  ( FIG.  4   ) or another data source. Parametric inputs, for example, such as XY-Mode and Cony-Mode, may be sourced from CAF control registers  402 , configuration parameter circuits  710 , or another source. The arithmetic unit for deep learning acceleration  700  of  FIG.  7 F  includes multiplexor circuitry  764 , adder circuitry  772 , temporary storage circuitry  768 , data shift circuitry  770 , and serializer circuitry  780 . Other circuitry of the arithmetic unit are optionally not used in the embodiment of  FIG.  7 F . 
     In the third data flow embodiment of  FIG.  7 F , parametric input including an XY-Mode signals, a Cony-Mode signals, Result-Shift signals, Shift-Direction signals, Active Kernel Count signals, and other such parametric information is made via inputs  762 . The inputs may draw information from a machine learning algorithm, CAF control registers  402 , configuration parameter circuits  710 , or some other source. In  FIG.  7 F , combinatorial logic  782 , and in at least one embodiment a logical OR-gate  782   n , is arranged to pass signal information representative of the XY-Mode and Cony-Mode parameters. The signal information is used in multiplexor circuitry  764 . Based on the XY-Mode and Cony-Mode parameters, a streaming input representing certain dot product values (BY) or cached recurring data are passed through multiplexor circuit  764 G to adder circuitry  772 G. The same XY-Mode and Cony-Mode signal information is also used as selection information for multiplexor circuit  764   n  to selection values that will be passed through output  712  of the final stage  778  circuitry. 
     The Cony-Mode signal information is applied to selection inputs of multiplexor circuit  764 H and multiplexor circuit  764 I. The Cony-Mode signal here determines whether constant information (e.g., zero) or cached recurring data from temporary storage circuitry  768 H is passed through multiplexor circuit  764 H to adder circuitry  772 H. And the Cony-Mode signal also determines whether summation data from adder circuit  772 G or recurring data from temporary storage circuitry  768 I is passed through multiplexor circuit  764 I to adder circuitry  772 I. 
     Adder circuits  772  in the final stage  778  of  FIG.  7 F  perform summing operations according to a machine learning algorithm and pass output data to temporary storage circuits  768 . More specifically, the output of adder circuits  772 G,  772 H, and  772 I is respectively loaded into temporary storage circuits  768 G,  768 H, and  768 I, respectively. Data from the temporary storage circuits  768  is passed through optional shift circuitry  770  before being passed to serializer circuitry  780 . More specifically, the data from temporary storage circuits  768 G,  768 H, and  768 I is loaded into data shift circuits  770 G,  770 H, and  770 I, respectively. A serializer circuit  780   n  groups, packetizes, or otherwise forms the output data that will be selective passed through multiplexor circuit  764   n  to output  712 . 
     In at least one instance of the second and third data flow diagrams of  FIGS.  7 E,  7 F , a stream of constant vectors enters the arithmetic unit for deep learning acceleration  700  in data flow  700 E of  FIG.  7 E . The stream of constant vectors enters on the stream input Y, and a stream of vector data enters the arithmetic unit on the stream input X. The arithmetic unit is directed by a machine learning algorithm to perform a one-by-one (1×1) convolution or dot-product operation. Various parameters are programmed by the machine learning algorithm into the CAF control registers  402 , the configuration parameter circuits  710 , or another structure. These parameters are used to direct the operations of the arithmetic units for deep learning acceleration  700  in accordance with the first and second data flows of  FIGS.  7 E and  7 F . 
     Still considering processing in the second data flow  700 E of  FIG.  7 E , for example, the XY-Mode parameter is arranged to specify a one-by-one (1×1) convolution operation using both X and Y inputs, and the Cony-Mode parameter is arranged to specify one-by-one (1×1) convolution using vector data passed on stream input X and kernel vectors stored in vector constant memory  706  ( FIG.  7 A ) or scalar constants A, B, C, stored in configuration parameter circuits  710  ( FIG.  7 A ). 
     In a non-limiting example, when XY-Mode is “1,” and when vector data is streamed on input X from stream switch  500 , and when convolution kernel weight vectors are streamed on input Y from stream switch  500 , the arithmetic unit will perform a one-by-one (1×1) convolution of the data, which may be streamed through output  712  back into stream switch  500  and into the third data flow  700 F of  FIG.  7 F . What is more, if Cony-Mode is “1,” it is possible to cache/load the kernel weights in local memory such as vector constant memory  706  ( FIG.  7 A ) and pass the constants via vector constant inputs  706 A,  706 B,  706 C ( FIG.  7 A ) so that up to three one-by-one-by-D kernels can be cached, wherein “D” is the line length of the internal line buffers for storing A, B and C constants. Further still, in a heretofore unavailable way, the cached data can be reused if needed to operate on the vector data streamed on input X and corresponding to convolution feature data. In this way, the operations selectively performed in the second data flow  700 E of  FIG.  7 E  can include those identified in Equation 16. 
       output=( X*Y )|( A*X )|( B*X )|( C*X )  (16)
 
     The equations performed depend on settings of the XY-Mode and Cony-Mode parameters. The performance of the formula may also include use of internal accumulators for to store partial convolution results before generating the output. 
     The present disclosure refers to a “semiconductor practitioner.” A semiconductor practitioner is generally one of ordinary skill in the semiconductor design and fabrication arts. The semiconductor practitioner may be a degreed engineer or another technical person or system having such skill as to direct and balance particular features of a semiconductor fabrication project such as geometry, layout, power use, included intellectual property (IP) modules, and the like. The semiconductor practitioner may or may not understand each detail of the fabrication process carried out to form a die, an integrated circuit, or other such device. 
       FIGS.  7 A- 7 F  include data flow diagrams illustrating non-limiting processes that may be used by embodiments of the mobile computing device  100 . In this regard, each described process may represent a module, segment, or portion of software code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some implementations, the functions noted in the process may occur in a different order, may include additional functions, may occur concurrently, and/or may be omitted. 
     The figures in the present disclosure illustrate portions of one or more non-limiting computing device embodiments such as mobile device  100 . The computing devices may include operative hardware found in conventional computing device apparatuses such as one or more processors, volatile and non-volatile memory, serial and parallel input/output (I/O) circuitry compliant with various standards and protocols, wired and/or wireless networking circuitry (e.g., a communications transceiver), one or more user interface (UI) modules, logic, and other electronic circuitry. 
     Amongst other things, the exemplary mobile devices of the present disclosure (e.g., mobile device  100  of  FIG.  3   ) may be configured in any type of mobile computing device such as a smartphone, a tablet, a laptop computer, a wearable device (e.g., eyeglasses, jacket, shirt, pants, socks, shoes, other clothing, hat, helmet, other headwear, wristwatch, bracelet, pendant, other jewelry), vehicle-mounted device (e.g., train, plane, helicopter, unmanned aerial vehicle, unmanned underwater vehicle, unmanned land-based vehicle, automobile, motorcycle, bicycle, scooter, hover-board, other personal or commercial transportation device), industrial device (e.g., factory robotic device, home-use robotic device, retail robotic device, office-environment robotic device), or the like. Accordingly, the mobile devices include other components and circuitry that is not illustrated, such as, for example, a display, a network interface, memory, one or more central processors, camera interfaces, audio interfaces, and other input/output interfaces. In some cases, the exemplary mobile devices may also be configured in a different type of low-power device such as a mounted video camera, an Internet-of-Things (IoT) device, a multimedia device, a motion detection device, an intruder detection device, a security device, a crowd monitoring device, or some other device. 
     Processors, as described herein, include central processing units (CPU&#39;s), microprocessors, microcontrollers (MCU), digital signal processors (DSP), application specific integrated circuits (ASIC), state machines, and the like. Accordingly, a processor as described herein includes any device, system, or part thereof that controls at least one operation, and such a device may be implemented in hardware, firmware, or software, or some combination of at least two of the same. The functionality associated with any particular processor may be centralized or distributed, whether locally or remotely. A processors may interchangeably refer to any type of electronic control circuitry configured to execute programmed software instructions. The programmed instructions may be high-level software instructions, compiled software instructions, assembly-language software instructions, object code, binary code, micro-code, or the like. The programmed instructions may reside in internal or external memory or may be hard-coded as a state machine or set of control signals. According to methods and devices referenced herein, one or more embodiments describe software executable by the processor, which when executed, carries out one or more of the method acts. 
     In some cases, the processor or processors described in the present disclosure, and additionally more or fewer circuits of the exemplary mobile devices described in the present disclosure, may be provided in an integrated circuit. In some embodiments, all of the elements shown in the processors of the present figures (e.g., SoC  110 ) may be provided in an integrated circuit. In alternative embodiments, one or more of the arrangements depicted in the present figures (e.g., SoC  110 )  6  may be provided by two or more integrated circuits. Some embodiments may be implemented by one or more dies. The one or more dies may be packaged in the same or different packages. Some of the depicted components may be provided outside of an integrated circuit or die. 
     The processors shown in the present figures and described herein may be fixed at design time in terms of one or more of topology, maximum available bandwidth, maximum available operations per unit time, maximum parallel execution units, and other such parameters. Some embodiments of the processors may provide re-programmable functionality (e.g., reconfiguration of SoC modules and features to implement a DCNN) at run-time. Some or all of the re-programmable functionality may be configured during one or more initialization stages. Some or all of the re-programmable functionality may be configured on the fly with no latency, maskable latency, or an acceptable level of latency. 
     As known by one skilled in the art, a computing device as described in the present disclosure, and mobile device  100  being such a computing device, has one or more memories, and each memory comprises any combination of volatile and non-volatile computer-readable media for reading and writing. Volatile computer-readable media includes, for example, random access memory (RAM). Non-volatile computer-readable media includes, for example, read only memory (ROM), magnetic media such as a hard-disk, an optical disk, a flash memory device, and/or the like. In some cases, a particular memory is separated virtually or physically into separate areas, such as a first memory, a second memory, a third memory, etc. In these cases, it is understood that the different divisions of memory may be in different devices or embodied in a single memory. The memory in some cases is a non-transitory computer medium configured to store software instructions arranged to be executed by a processor. 
     In the present disclosure, memory may be used in one configuration or another. The memory may be configured to store data. In the alternative or in addition, the memory may be a non-transitory computer readable medium (CRM) wherein the CRM is configured to store instructions executable by a processor. The instructions may be stored individually or as groups of instructions in files. The files may include functions, services, libraries, and the like. The files may include one or more computer programs or may be part of a larger computer program. Alternatively or in addition, each file may include data or other computational support material useful to carry out the computing functions of the systems, methods, and apparatus described in the present disclosure. 
     The computing devices illustrated and described herein, of which mobile device  100  is one example, may further include operative software found in a conventional computing device such as an operating system or task loop, software drivers to direct operations through I/O circuitry, networking circuitry, and other peripheral component circuitry. In addition, the computing devices may include operative application software such as network software for communicating with other computing devices, database software for building and maintaining databases, and task management software where appropriate for distributing the communication and/or operational workload amongst various processors. In some cases, the computing device is a single hardware machine having at least some of the hardware and software listed herein, and in other cases, the computing device is a networked collection of hardware and software machines working together in a server farm to execute the functions of one or more embodiments described herein. Some aspects of the conventional hardware and software of the computing device are not shown in the figures for simplicity, but are well understood by skilled practitioners. 
     When so arranged as described herein, each computing device may be transformed from a generic and unspecific computing device to a combination device comprising hardware and software configured for a specific and particular purpose. Along these lines, the features of the combination device bring improvements to the technological computing arts heretofore unseen and unknown. 
     Database structures, if any are present in the mobile devices or supporting network devices described herein, may be formed in a single database or multiple databases. In some cases hardware or software storage repositories are shared amongst various functions of the particular system or systems to which they are associated. A database may be formed as part of a local system or local area network. Alternatively, or in addition, a database may be formed remotely, such as within a “cloud” computing system, which would be accessible via a wide area network or some other network. 
     In at least one embodiment, mobile devices described herein may communicate with other devices via communication over a network. The network may involve an Internet connection or some other type of local area network (LAN) or wide area network (WAN). Non-limiting examples of structures that enable or form parts of a network include, but are not limited to, an Ethernet, twisted pair Ethernet, digital subscriber loop (DSL) devices, wireless LAN, WiFi, cellular-based networks, or the like. 
     Buttons, keypads, computer mice, memory cards, serial ports, bio-sensor readers, touch screens, and the like may individually or in cooperation be useful to an operator of the mobile device or other such devices as described herein. The devices may, for example, input control information into the system. Displays, printers, memory cards, LED indicators, temperature sensors, audio devices (e.g., speakers, piezo device, etc.), vibrators, and the like are all useful to present output information to the operator of these mobile devices. In some cases, the input and output devices are directly coupled to the control systems described herein and electronically coupled to a processor or other operative circuitry. In other cases, the input and output devices pass information via one or more communication ports (e.g., RS-232, RS-485, infrared, USB, etc.) 
     Unless defined otherwise, the technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, a limited number of the exemplary methods and materials are described herein. 
     In the foregoing description, certain specific details are set forth to provide a thorough understanding of various disclosed embodiments. However, one skilled in the relevant art will recognize that embodiments may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In other instances, well-known structures associated with electronic and computing systems including client and server computing systems, as well as networks, have not been shown or described in detail to avoid unnecessarily obscuring descriptions of the embodiments. 
     Unless the context requires otherwise, throughout the specification and claims which follow, the word “comprise,” and variations thereof, such as “comprises” and “comprising,” are to be construed in an open, inclusive sense, e.g., “including, but not limited to.” 
     Reference throughout this specification to “one embodiment” or “an embodiment” and variations thereof means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. 
     As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the content and context clearly dictates otherwise. It should also be noted that the conjunctive terms, “and” and “or” are generally employed in the broadest sense to include “and/or” unless the content and context clearly dictates inclusivity or exclusivity as the case may be. In addition, the composition of “and” and “or” when recited herein as “and/or” is intended to encompass an embodiment that includes all of the associated items or ideas and one or more other alternative embodiments that include fewer than all of the associated items or ideas. 
     In the present disclosure, conjunctive lists make use of a comma, which may be known as an Oxford comma, a Harvard comma, a serial comma, or another like term. Such lists are intended to connect words, clauses or sentences such that the thing following the comma is also included in the list. 
     The headings and Abstract of the Disclosure provided herein are for convenience only and do not limit or interpret the scope or meaning of the embodiments. 
     The various embodiments described above can be combined to provide further embodiments. Aspects of the embodiments can be modified, if necessary to employ concepts of the various patents, application and publications to provide yet further embodiments. 
     These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.