Source: https://patents.google.com/patent/US9986187B2/en
Timestamp: 2019-02-17 14:43:55
Document Index: 544545679

Matched Legal Cases: ['art 1', 'art 2', 'art 1', 'art 2', 'art 1', 'art 2']

US9986187B2 - Block operations for an image processor having a two-dimensional execution lane array and a two-dimensional shift register - Google Patents
Block operations for an image processor having a two-dimensional execution lane array and a two-dimensional shift register Download PDF
US9986187B2
US9986187B2 US15/628,527 US201715628527A US9986187B2 US 9986187 B2 US9986187 B2 US 9986187B2 US 201715628527 A US201715628527 A US 201715628527A US 9986187 B2 US9986187 B2 US 9986187B2
US15/628,527
US20180007303A1 (en
2016-07-01 Priority to US15/201,237 priority Critical patent/US20180007302A1/en
2017-06-20 Application filed by Google LLC filed Critical Google LLC
2017-06-20 Priority to US15/628,527 priority patent/US9986187B2/en
2017-06-22 Assigned to GOOGLE INC. reassignment GOOGLE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MEIXNER, ALBERT, REDGRAVE, JASON RUPERT, FINCHELSTEIN, Daniel Frederic, MARK, WILLIAM R., PATTERSON, DAVID, SHACHAM, Ofer
2018-01-04 Publication of US20180007303A1 publication Critical patent/US20180007303A1/en
2018-05-29 Publication of US9986187B2 publication Critical patent/US9986187B2/en
A method is described that includes, on an image processor having a two dimensional execution lane array and a two dimensional shift register array, repeatedly shifting first content of multiple rows or columns of the two dimensional shift register array and repeatedly executing at least one instruction between shifts that operates on the shifted first content and/or second content that is resident in respective locations of the two dimensional shift register array that the shifted first content has been shifted into.
This application is a continuation application of, and claims priority to pending U.S. application Ser. No. 15/201,237, filed on Jul. 1, 2016. The entirety of the disclosure of the prior application is herein incorporated by reference.
The field of invention pertains generally to image processing, and, more specifically, to block operations for an image processor having a two-dimensional execution lane array and a two-dimensional shift register.
An apparatus is described that includes means for, on an image processor having a two dimensional execution lane array and a two dimensional shift register array, repeatedly shifting first content of multiple rows or columns of the two dimensional shift register array and repeatedly executing at least one instruction between shifts that operates on the shifted first content and/or second content that is resident in respective locations of the two dimensional shift register array that the shifted first content has been shifted into.
FIG. 9b shows an embodiment of an instruction word of the stencil processor;
As such, the larger overall image processing sequence may take the form of an image processing pipeline or a directed acyclic graph (DAG) and the development environment may be equipped to actually present the developer with a representation of the program code being developed as such. Kernels may be developed by a developer individually and/or may be provided by an entity that supplies any underlying technology (such as the actual signal processor hardware and/or a design thereof) and/or by a third party (e.g., a vendor of kernel software written for the development environment). As such, it is expected that a nominal development environment will include a “library” of kernels that developers are free to “hook-up” in various ways to effect the overall flow of their larger development effort. Some basic kernels that are expected to be part of such a library may include kernels to provide any one or more of the following basic image processing tasks: convolutions, denoising, color space conversions, edge and corner detection, sharpening, white balance, gamma correction, tone mapping, matrix multiply, image registration, pyramid construction, wavelet transformation, block-wise discrete cosine, and Fourier transformations.
In an embodiment, as index values are typically used to define a desired look-up table entry, the look-up table information region is accessed using a normal linear accessing scheme. In an embodiment the look-up region of memory is read only (i.e., the processor cannot change information in a look-up table and is only permitted to read information from it). For simplicity FIG. 4a suggests only one look-up table is resident within the look-up table region 405 but the virtual environment permits for multiple, different look-up tables to be resident during the simulated runtime. Embodiments of the virtual ISA instruction format for instructions that perform look-ups into the look-up table are provided further below.
FIG. 4c shows each of the virtual processors reading 414 a constant value from a constant look-up table 415 within the constant memory region 407. Here, e.g., it is expected that different threads 401 may need a same constant or other value on the same clock cycle (e.g., a particular multiplier to be applied against an entire image). Thus, accesses into the constant look-up table 415 return a same, scalar value to each of the virtual processors as depicted in FIG. 4c . Because look-up tables are typically accessed with an index value, in an embodiment, the constant look-up table memory region is accessed with a linear random access memory address. In an embodiment the constant region of memory is read only (i.e., the processor cannot change information in a constant table and is only permitted to read information from it). For simplicity FIG. 4c only shows a single constant look-up table 415 in the constant memory region 407. As threads may make use of more than one such table memory region 407 is configured to be large enough to hold as many constant tables are needed/used.
In an embodiment, instructions for loads/stores from/to the input/output arrays have the following format:
[OPCODE]LINEGROUP_(name)[(((X*XS+X0)/XD);((Y*YS+Y0)/YD);Z]
[OPCODE] STATS_(name)[(((X*XS+X0)/XD);((Y*YS+Y0)/YD);Z]
A simple blur kernel that averages together the pixel values for the X,Y location along with its left and right neighbors may therefore be written in pseudo-code as depicted in FIG. 5a . As observed in FIG. 5a , the location ((X); (Y)) corresponds to the position of the virtual processor that is writing to the output array. In the above pseudo-code, LOAD corresponds to the opcode for a load from the input array and STORE corresponds to the opcode for the store to the output array. Note that there exists a LINEGROUP_1 in the input array and a LINEGROUP_1 in the output array.
R1 <= LOAD LINEGROUP_1[((3X)−1);3(Y);0]
R2 <= LOAD LINEGROUP_1[3(X);3(Y);0]
R3 <= LOAD LINEGROUP_1[((3X)+1);3(Y);0]
R2 <= ADD R1, R2
R2 <= ADD R2, R3
R2 <= DIV R2, 3
R1 <= LOAD LINEGROUP_1[(X−1)/3;(Y)/3;0]
R2 <= LOAD LINEGROUP_1[(X)/3;(Y)/3;0]
R3 <= LOAD LINEGROUP_1[(X+1)/3;(Y)/3;0]
STORE LINEGROUP_1[(X);(Y);(0)]; R2
A similar format with similarly minded opcode may be utilized for instructions that target the constant and the private memory regions (e.g., LOAD CNST_(name)[(A*B+C)]; LOAD PRVT_(name)[(A*B+C)]. In an embodiment, look-up table and the constant table accesses are read-only (a processor cannot change the data that has been placed there). As such no STORE instructions exist for these memory regions. In an embodiment the private region of memory is read/write. As such a store instruction exists for that memory region (e.g., STORE PRVT[(A*B+C)].
FIGS. 8a through 8e illustrate at high level embodiments of both the parsing activity of a line buffer unit 701, the finer grained parsing activity of a sheet generator unit 703, as well as the stencil processing activity of the stencil processor 702 that is coupled to the sheet generator unit 703.
Some notable architectural features of the data computation unit 1001 include the shift register structure 1006 having wider dimensions than the execution lane array 1005. That is, there is a “halo” of registers 1009 outside the execution lane array 1005. Although the halo 1009 is shown to exist on two sides of the execution lane array, depending on implementation, the halo may exist on less (one) or more (three or four) sides of the execution lane array 1005. The halo 1005 serves to provide “spill-over”space for data that spills outside the bounds of the execution lane array 1005 as the data is shifting “beneath” the execution lanes 1005. As a simple case, a 5×5 stencil centered on the right edge of the execution lane array 1005 will need four halo register locations further to the right when the stencil's leftmost pixels are processed. For ease of drawing, FIG. 10 shows the registers of the right side of the halo as only having horizontal shift connections and registers of the bottom side of the halo as only having vertical shift connections when, in a nominal embodiment, registers on either side (right, bottom) would have both horizontal and vertical connections.
In a first iteration of machine level operation, depicted in FIG. 16b , the R0 register space locations are shifted right one location into the destination location's R1 register space. A subsequent ADD operation adds the R0 content with either the R1 content or the R2 content depending on the location of the lane relative to the iteration count. Specifically, the first row location selects the null in R2 (and not the shifted content in R1) because its location (0) is equal to or less than 2N−1 where N is the iteration count (20−1=0), adds the null to the content in R0 and stores the resultant back in R0. The first row location, therefore will maintain a value of A0 in R0.
which corresponds to the elements of row 1903 in matrix A being multiplied with the corresponding elements of column 1902 in matrix B in FIG. 19a . As can be seen from these two examples, the resultant for any coordinate location x,y in the resultant matrix C can be expressed as:
For simplicity, FIG. 21 represents the first complex term 2101 as Re1+jIm1 and represents the second complex 2102 term as Re2+jIm2. As is known in the art, the real part of (Re1+jIm1)*(Re2+jIm2) can be expressed as (Re1*Re2)−(Im1*Im2) while the imaginary part can be expressed as j((Re1*Im2)+(Re2*Im1)). The summations of the 2D DFT over 2D space, just like the matrix multiply discussed at length immediately above, add the products of elements in a row of a coordinate location by corresponding elements in the column of the coordinate location.
FIG. 24b and FIG. 24c depict 2 stage and 4 stage butterfly operations. Processing is similar to the 1 stage butterfly operation described just above, except that in the case of the 2 stage butterfly the shift register array shifts the R0 register content two units to the right and then four units to the left. Every other even lane and its rightmost neighbor select from one of R1 and R0″ whereas the other even lanes and their rightmost neighbor select from the other of R1 and R0″. In the case of the 4 stage butterfly, depicted in FIG. 25c , the shift register array shifts the contents of R0 four units to the right and then selects all values as they reside in R0 (labeled R0′).
a two-dimensional array of processing elements; and
multiple shift-register planes, each shift-register plane comprising a separate two-dimensional shift-register array, wherein each shift register of each shift-register array is dedicated to one of the processing elements,
wherein the processor is configured to execute instructions to perform a matrix multiplication operation on a first matrix stored in a first shift-register plane and a second matrix stored in a second shift-register plane to generate a result of the matrix multiplication operation stored in a third shift-register plane, wherein the first matrix comprises N columns, and wherein the instructions cause each processing element to perform, for N iterations, operations on shift registers dedicated to the processing element, the operations comprising:
performing a multiplication operation between a first value stored in a first shift register of the first shift-register plane and a second value stored in a second shift register of the second shift-register plane,
performing an addition operation between a result of the multiplication operation and a summation value stored in a third shift register of the third shift-register plane,
updating the summation value stored in the third shift register of the third shift-register plane using a result of the addition operation, and
based on determining that a current iteration is not a last iteration:
shifting data in the first shift-register plane one unit in a first dimension corresponding to rows of the first matrix, and
shifting data in the second shift-register plane one unit in a second dimension corresponding to columns of the second matrix.
2. The processor of claim 1, wherein the first matrix comprises R rows and the second matrix comprises S columns, and wherein the operations further comprise:
performing a row-wise rotational shearing transformation on the first matrix including shifting each row i, from row 0 to row R−1, by i units; and
performing a column-wise rotational shearing transformation on the second matrix including shifting each column j, from column 0 to column S−1, by j units.
3. The processor of claim 1, wherein the operations further comprise:
loading an initial condition for the matrix multiplication by loading initial values into shift registers of the third shift-register plane.
4. The processor of claim 1, wherein the operations further comprise computing a two-dimensional discrete Fourier transform (2D DFT) using a first matrix of real values, a first matrix of imaginary values, a second matrix of real values, and a second matrix of imaginary values including:
computing a real portion of the 2D DFT including:
performing a first matrix multiplication operation between the first matrix of real values and the second matrix of real values,
performing a second matrix multiplication operation between the first matrix of imaginary values and the second matrix of imaginary values, and
subtracting a result of the second matrix multiplication operation from the first matrix multiplication operation; and
computing an imaginary portion of the 2D DFT including:
performing a third matrix multiplication operation between the first matrix of real values and the second matrix of imaginary values,
performing a fourth matrix multiplication operation between the second matrix of real values and the first matrix of imaginary values, and
adding a result of the third matrix multiplication operation and a result of the fourth matrix multiplication operation.
5. The processor of claim 4, wherein the operations further comprise:
moving the computed real portion of the 2D DFT to another shift-register plane or to memory;
reloading the first matrix of real values and the first matrix of imaginary values back into respective shift-register planes; and
performing a row-wise rotational shearing transformation on the first matrix of real values and the first matrix of imaginary values.
6. A computer program product, encoded on one or more non-transitory computer storage media, comprising instructions that when executed by a processor comprising:
a two-dimensional array of processing elements, and
cause the processor to perform a matrix multiplication operation on a first matrix stored in a first shift-register plane and a second matrix stored in a second shift-register plane to generate a result of the matrix multiplication operation stored in a third shift-register plane, wherein the first matrix comprises N columns, and wherein the instructions cause each processing element to perform, for N iterations, operations on shift registers dedicated to the processing element, the operations comprising:
7. The computer program product of claim 6, wherein the first matrix comprises R rows and the second matrix comprises S columns, and wherein the operations further comprise:
9. The computer program product of claim 6, wherein the operations further comprise computing a two-dimensional discrete Fourier transform (2D DFT) using a first matrix of real values, a first matrix of imaginary values, a second matrix of real values, and a second matrix of imaginary values including:
11. A method performed by a processor having a two-dimensional array of processing elements and multiple shift-register planes,
wherein the method causes the processor to perform a matrix multiplication operation on a first matrix stored in a first shift-register plane and a second matrix stored in a second shift-register plane to generate a result of the matrix multiplication operation stored in a third shift-register plane, wherein the shift-register planes each comprise a separate two-dimensional shift register array, wherein each shift register of each shift register array is dedicated to one processing element in the two-dimensional array of processing elements, and wherein the first matrix comprises N columns, the method comprising:
performing, by each processing element, for N iterations, operations on shift registers dedicated to the processing element, the operations comprising:
12. The method of claim 11, wherein the first matrix comprises R rows and the second matrix comprises S columns, and wherein the method further comprises:
14. The method of claim 11, further comprising computing a two-dimensional discrete Fourier transform (2D DFT) using a first matrix of real values, a first matrix of imaginary values, a second matrix of real values, and a second matrix of imaginary values including:
US15/628,527 2016-07-01 2017-06-20 Block operations for an image processor having a two-dimensional execution lane array and a two-dimensional shift register Active US9986187B2 (en)
US15/201,237 US20180007302A1 (en) 2016-07-01 2016-07-01 Block Operations For An Image Processor Having A Two-Dimensional Execution Lane Array and A Two-Dimensional Shift Register
US15/628,527 US9986187B2 (en) 2016-07-01 2017-06-20 Block operations for an image processor having a two-dimensional execution lane array and a two-dimensional shift register
US15/946,095 US20180234653A1 (en) 2016-07-01 2018-04-05 Block operations for an image processor having a two-dimensional execution lane array and a two-dimensional shift register
US15/201,237 Continuation US20180007302A1 (en) 2016-07-01 2016-07-01 Block Operations For An Image Processor Having A Two-Dimensional Execution Lane Array and A Two-Dimensional Shift Register
US15/946,095 Division US20180234653A1 (en) 2016-07-01 2018-04-05 Block operations for an image processor having a two-dimensional execution lane array and a two-dimensional shift register
US20180007303A1 US20180007303A1 (en) 2018-01-04
US9986187B2 true US9986187B2 (en) 2018-05-29
ID=59153293
US15/201,237 Pending US20180007302A1 (en) 2016-07-01 2016-07-01 Block Operations For An Image Processor Having A Two-Dimensional Execution Lane Array and A Two-Dimensional Shift Register
US15/628,527 Active US9986187B2 (en) 2016-07-01 2017-06-20 Block operations for an image processor having a two-dimensional execution lane array and a two-dimensional shift register
US15/946,095 Pending US20180234653A1 (en) 2016-07-01 2018-04-05 Block operations for an image processor having a two-dimensional execution lane array and a two-dimensional shift register
US (3) US20180007302A1 (en)
CN (1) CN107563953A (en)
DE (2) DE102017113859A1 (en)
GB (1) GB2560208A (en)
TW (2) TWI625697B (en)
WO (1) WO2018005036A1 (en)
EP0293701A2 (en) 1987-06-01 1988-12-07 Applied Intelligent Systems, Inc. Parallel neighborhood processing system and method
US5751864A (en) 1993-09-08 1998-05-12 Sony Corporation Image data processing apparatus
US20050270412A1 (en) 2004-06-04 2005-12-08 Konica Minolta Holdings, Inc. Image sensing apparatus
US20070047828A1 (en) 2005-08-31 2007-03-01 Hideki Ishii Image data processing device
US20070080969A1 (en) 2002-02-28 2007-04-12 Shinichi Yamaura Image processing circuit, combined image processing circuit, and image forming apparatus
WO2007071883A2 (en) 2005-12-19 2007-06-28 Dxo Labs Digital data processing method and system
US20090002390A1 (en) 2007-06-29 2009-01-01 Brother Kogyo Kabushiki Kaisha Image processor
US7583851B2 (en) 2004-12-28 2009-09-01 Fujitsu Microelectronics Limited Apparatus and method for processing an image
US20090317009A1 (en) 2005-10-21 2009-12-24 Mobilic Technology (Cayman) Corp. Universal fixed-pixel-size isp scheme
US20100188538A1 (en) 2007-09-05 2010-07-29 Tohoku University Solid-state image sensor and drive method for the same
US20110087867A1 (en) 2005-06-23 2011-04-14 Jacobson Quinn A Primitives to enhance thread-level speculation
US20110125768A1 (en) 2008-06-30 2011-05-26 Canon Kabushiki Kaisha Information processing apparatus, method of controlling the same, and storage medium
US20110153925A1 (en) 2009-12-22 2011-06-23 Bains Kuljit S Memory controller functionalities to support data swizzling
US20120320070A1 (en) 2011-06-20 2012-12-20 Qualcomm Incorporated Memory sharing in graphics processing unit
JP2013090070A (en) 2011-10-15 2013-05-13 Konica Minolta Business Technologies Inc Image processing apparatus, linkage method and linkage program
US20130202051A1 (en) 2012-02-02 2013-08-08 Texas Instruments Incorporated Sub-Pictures for Pixel Rate Balancing on Multi-Core Platforms
US20130243329A1 (en) 2012-03-15 2013-09-19 Herta Security, S.L. Parallel object detection method for heterogeneous multithreaded microarchitectures
US20130318544A1 (en) 2011-01-31 2013-11-28 Panasonic Corporation Program generation device, program generation method, processor device, and multiprocessor system
US20130314428A1 (en) 2012-05-23 2013-11-28 Mstar Semiconductor, Inc. Image processing apparatus and associated method
US20140136816A1 (en) 2012-11-09 2014-05-15 Scott Krig Scalable computing array
US8970884B2 (en) 2011-12-27 2015-03-03 Kyocera Document Solutions Inc. Image forming apparatus
US20150086134A1 (en) * 2013-09-20 2015-03-26 The Board Of Trustees Of The Leland Stanford Junior University Low power programmable image processor
US20150106596A1 (en) 2003-03-21 2015-04-16 Pact Xpp Technologies Ag Data Processing System Having Integrated Pipelined Array Data Processor
CN103034524B (en) 2011-10-10 2016-05-04 辉达公司 Paravirtualized virtual gpu
US20160219225A1 (en) 2015-01-22 2016-07-28 Google Inc. Virtual linebuffers for image signal processors
US20160316094A1 (en) 2015-04-23 2016-10-27 Google Inc. Sheet Generator For Image Processor
US20160313980A1 (en) 2015-04-23 2016-10-27 Google Inc. Virtual Image Processor Instruction Set Architecture (ISA) And Memory Model And Exemplary Target Hardware Having A Two-Dimensional Shift Array Structure
US20160316157A1 (en) 2015-04-23 2016-10-27 Google Inc. Line buffer unit for image processor
US20160316107A1 (en) 2015-04-23 2016-10-27 Google Inc. Two dimensional shift array for image processor
US20160313999A1 (en) 2015-04-23 2016-10-27 Google Inc. Energy efficient processor core architecture for image processor
US20160313984A1 (en) 2015-04-23 2016-10-27 Google Inc. Compiler for translating between a virtual image processor instruction set architecture (isa) and target hardware having a two-dimensional shift array structure
US20160314555A1 (en) 2015-04-23 2016-10-27 Google Inc. Architecture for high performance, power efficient, programmable image processing
2016-07-01 US US15/201,237 patent/US20180007302A1/en active Pending
2017-06-08 WO PCT/US2017/036563 patent/WO2018005036A1/en unknown
2017-06-20 GB GB1709786.6A patent/GB2560208A/en active Pending
2017-06-20 US US15/628,527 patent/US9986187B2/en active Active
2017-06-22 DE DE102017113859.5A patent/DE102017113859A1/en active Pending
2017-06-22 DE DE202017103725.8U patent/DE202017103725U1/en active Active
2017-06-30 TW TW106121882A patent/TWI625697B/en active
2017-06-30 TW TW107112864A patent/TW201826219A/en unknown
2017-07-03 CN CN201710534277.6A patent/CN107563953A/en active Search and Examination
2018-04-05 US US15/946,095 patent/US20180234653A1/en active Pending
US20090228677A1 (en) 2005-12-19 2009-09-10 Dxo Labs Digital data processing method and system
US20170310855A1 (en) * 2015-04-23 2017-10-26 Google Inc. Two Dimensional Shift Array for Image Processor
US9769356B2 (en) * 2015-04-23 2017-09-19 Google Inc. Two dimensional shift array for image processor
"ChimeraTM: The NVIDIA Computational Photography Architecture" Whitepaper, NVIDIA Corporation 2013, 15 pages.
Adams, et al. "The Frankencamera: An Experimental Platform for Computational Photography", ACM Transactions on Graphics, vol. 29, No. 4, Article 29, Publication Jul. 2010, 12 pages.
Ahn, et al., "Evaluating the Image Stream Architecture." In ACM SIGARCH Computer Architecture News, vol. 32, No. 2, IEEE Computer Society, Jun. 2004, 14 pages.
Arasan "Silicon Hive Announces New Image Signal Processor", Eindhoven, the Netherlands, May 16, 2006, http://www.design-reuse.com/news/13362/silicon-hive-image-singal-processor.html, 3 pages.
Balfour, et al., "An Energy-Efficient Processor Architecture for Embedded Systems" IEEE Computer Architecture Letters 7, No. 1 p. 29-32, May 2008.
Barry, et al., "Always-On Vision Processing Unit for Mobile Applications", IEEE Micro, Mar./Apr. 2015, pp. 56-66.
Bolotoff, Paul V., "Alpha-The History in Facts and Comments" http://alasir.com/articles/alpha_history/alpha_21164_21164pc.html, Last modification date Apr. 22, 2007, 5 pages.
Bolotoff, Paul V., "Alpha—The History in Facts and Comments" http://alasir.com/articles/alpha_history/alpha_21164_21164pc.html, Last modification date Apr. 22, 2007, 5 pages.
Brunhaver, John S. "Design and Optimization of a Stencil Engine", Stanford University, Jun. 2015, 133 pages.
Cardells-Tormo et al., "Area-efficient 2-D Shift-variant Convolvers for FPGA-based Digital Image Processing," IEEE Workshop on Signal Processing Systems Design and Implementation, Nov. 2, 2005, 5 pages.
Cardells-Tormo, F.; Molinet, P.-L., "Area-efficient 2-D shift-variant convolvers for FPGA-based digital image processing," Circuits and Systems II: Express Briefs, IEEE Transactions on, vol. 53, No. 2, pp. 105-109, 2006. *
CEVA-MM3101: An Imaging-Optimized DSP Core Swings for an Embedded Vision Home Run, http://www.bdti.com/InsideDSP/2012/01/24/CEVA, Jan. 19, 2012, 3 pages.
Chao et al., "Pyramid Architecture for 3840×2160 Quad Full High Definition 30 Frames/s Video Acquisition," IEEE Transactions on Circuits and Systems for Video Technology, Nov. 1, 2010, 10 pages.
Chen et al., "CRISP: Coarse-Grained Reconfigurable Image Stream Processor for Digital Still Cameras and Camcorders." IEEE Transactions on Circuits and Systems for Video Technology, Sep. 2008, 14 pages.
Chen, et al., "DaDianNao: A Machine-Learning Supercomputer." 47m Annual IEEE/ACM International Symposium, pp. 609-622, IEEE, Dec. 2014.
Chenyun, et al., "A Paradigm Shift in Local Interconnect Technology Design in the Era of Nanoscale Multigate and Gate-All-Around Devices," IEEE Electron Device Letters, vol. 36, No. 3, pp. 274-26, Mar. 2015.
Clearspeed Whitepaper: CSX Processor Architecture, www.clearspeed.com, 2007, 16 pages.
Codrescu, et al., "Hexagon DSP: An Architecture Optimized for Mobile Multimedia and Communications." IEEE Micro vol. 34, Issue 2, pp. 34-43, Mar. 2014.
Dally, William J., "Computer architecture is all about interconnect." Proceedings of 8th International Symposium High-Perf. Comp. Architecture, Feb. 2002, 11 pages.
Datta et al. "Stencil computation optimization and auto-tuning on state-of-the-art multicore architectures," International Conference for High Performance Computing, Networking, Storage and Analysis, Nov. 15, 2008, 12 pages.
de Dinechin, et al., "A Clustered Manycore Processor Architecture for Embedded and Accelerated Applications." In High Performance Extreme Computing Conference (HPEC), IEEE, pp. 1-6, Sep. 2013.
DeVito, et al., "Terra: A Multi-Stage Language for High-Performance Computing", PLDI'13, Jun. 16-22, 2013, Seattle, Washington, 11 pages.
DeVito, et al.,"First-class Runtime Generation of High-Performance Types using Exotypes", PLDI'14, Jun. 9-11, ACM, 12 pages.
Dias et al. "A Configurable Window-based Processing Element for Image Processing on Smart Cameras," MVA 2007 IAPR Conference on Machine Vision Applications, May 16, 2007, 15 pages.
Dykes et al., "Communication and Computation Patterns of Large Scale Image Convolutions on Parallel Architectures," Parallel Processing Symposium, Jan. 1, 1994, 6 pages.
Eichenberger, et al., "Optimizing Compiler for the Cell Processor" PACT, Tuesday, Sep. 20, 2005, Part 1, pp. 1-16.
Eichenberger, et al., "Optimizing Compiler for the Cell Processor" PACT, Tuesday, Sep. 20, 2005, Part 2, pp. 17-32.
EyeQ2TM, "Vision System on a Chip", Mobileye, Sep. 2007, 4 pages.
Farabet, et al., "Hardware Accelerated Convolutional Neural Networks for Synthetic Vision Systems." Proceedings of 2010 IEEE International Symposium, pp. 257-260, Jun. 2010.
Farabet, et al., "Neuflow: A Runtime Reconfigurable Dataflow Processor for Vision." IEEE Computer Society Conference, pp. 109-116, Jun. 2011.
Galal, et al., "FPU Generator for Design Space Exploration." 21st IEEE Symposium on Computer Arithmetic (ARITH), Apr. 2013, 10 pages.
Gentile, et al., "Image Processing Chain for Digital Still Cameras Based on SIMPil Architecture." ICCP International Conference Workshops, pp. 215-222, IEEE, Jun. 2005.
Goldstein, et al., "PipeRench: A Coprocessor for Streaming Multimedia Acceleration", Carnegie Mellow University, Research Showcase @CMU, Appears in the 26th Annual International Symposium on Computer Architecture, May 1999, Atlanta, Georgia, 14 pages.
Gupta, et al., "A VLSI Architecture for Updating Raster-Scan Displays", Computer Graphics, vol. 15, No. 3, Aug. 1981, pp. 71-78.
Hameed, et al., "Understanding Sources of Inefficiency in General-Purpose Chips." ACM SIGARCH Computer Architecture News, vol. 38, No. 3, pp. 37-47, 2010.
Hanrahan, Pat, "Domain-Specific Languages for Heterogeneous GPU Computing", NVIDIA Technology Conference, Oct. 2, 2009, 30 pages.
Hegarty, et al., "Darkroom: Compiling High-Level Image Processing Code into Hardware Pipelines", Proceedings of ACM SIGGRAPH, Jul. 2014, 11 pages.
Henretty, et al., "A Stencil Compiler for Short-Vector SIMD Architectures", ACM, ICS'13, Jun. 10-14, 2013, Eugene, Oregon, pp. 13-24.
Horowitz, Mark, "Computing's Energy Problem: (and what we can do about it)", IEEE, International Solid-State Circuits Conference 2014, 46 pages.
International Search Report and Written Opinion issued in International Application No. PCT/US2017/036563, dated Aug. 31, 2017, 29 pages.
Kapasi, et al. "The Imagine Stream Processor", IEEE, International Conference on Computer Design: VLSI in Computers and Processors (ICCD'02), 2002, 17 pages.
Khailany, et al., "A Programmable 512 GOPS Stream Processor for Signal, Image, and Video Processing", IEEE Journal of Solid-State Circuits, vol. 43, No. 1, pp. 202-213, Jan. 2008.
Khawam, et al., "The Reconfigurable Instruction Cell Array", IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 16, No. 1, Jan. 2008, pp. 75-85.
Khronos, SPIR 2.0 "Provisional Specification to Enable Compiler Innovation on Diverse Parallel Architectures", Aug. 11, 2014, 2 pages.
Levinthal, "Parallel Computers for Graphics Applications", ACM, 1987, pp. 193-198.
Levinthal, et al., "Chap-A SIMD Gaphics Processor", Computer Graphics, vol. 18, No. 3, Jul. 1984, pp. 77-82.
Levinthal, et al., "Chap—A SIMD Gaphics Processor", Computer Graphics, vol. 18, No. 3, Jul. 1984, pp. 77-82.
Liu et al. "Data reuse in sliding window operations based on storing block targeted hardware compilation," IEEE 2nd International Conference on Future Computer and Communication, May 21, 2010, 5 pages.
M.J. Duff, "CLIP 4: A Large Scale Integrated Circuit Array Parallel Processor," Proc. IEEE Int'I Joint Conf. Pattern Recognition, . . . Jan. 2004, pp. 728-733.
Mandal, et al., "An Embedded Vision Engine (EVE) for Automotive Vision Processing." In Circuits and Systems (ISCAS), 2014 IEEE Symposium, pp. 49-52, IEEE, 2014.
Mcintosh-Smith, "A Next-Generation Many-Core Processor With Reliability, Fault Tolerance and Adaptive Power Management Features Optimized for Embedded and High Performance Computing Applications" in Proceedings of the High Performance Embedded Computing Conference (HPEC) 2008, pp. 1-2, 5.
Mody, et al., "High Performance and Flexible Imaging Sub-System." In Advances in Computing, Communications and Informatics (ICACCI), 2014 International Conference, pp. 545-548. IEEE, 2014.
Molnar, et al., "PixelFlow: High-Speed Rendering Using Image Composition" proceedings of Siggraph 92, Computer Graphics, 26, Jul. 2, 1992, 10 pages.
Moloney, David, "1 TOPS/W Software Programmable Media Processor." Hot Chips 23 Symposium (HCS), IEEE, Aug. 2011, 24 pages.
Moloney, et al., "Myriad 2: Eye of the Computational Vision Storm", Hot Chips 26 Symposium (HCS), IEEE, Aug. 2014, 18 pages.
MPPA-MANYCORE, Product Family Overview http://www.kalray.eu/IMG/pdf/FLYER_MPPA_MANYCORE-4.pdf, Feb. 2014, 2 pages.
MPPA—MANYCORE, Product Family Overview http://www.kalray.eu/IMG/pdf/FLYER_MPPA_MANYCORE-4.pdf, Feb. 2014, 2 pages.
Multioutput Scaler Reference Design Altera Corporation, Application Note AN-648-1.0, Aug. 2012, 18 pages.
Nightingale, Edmund B., et al., "Helios: Heterogeneous Multiprocessing with Satellite Kernels," SOSP '09, Oct. 11-14, 2009, Big Sky, Montana, U.S.A., (Oct. 11, 2009), 14 pages.
NVIDIA Tegra K1-A New Era in Mobile Computing-Whitepapers-Jan. 2014, 26 pages.
NVIDIA Tegra K1—A New Era in Mobile Computing—Whitepapers—Jan. 2014, 26 pages.
NVIDIA Tegra X1-NVIDIA's New Mobile Superchip-Whitepapers-Jan. 2015, 41 pages.
NVIDIA Tegra X1—NVIDIA's New Mobile Superchip—Whitepapers—Jan. 2015, 41 pages.
NVIDIA, "PTX:Parallel Thread Execution ISA Version 1.4", Aug. 23, 2010, 135 pages.
NVIDIA-NVIDIA's Next Generation CUDA™ Compute Architecture: Kepler™, GK110/210, 2014, 23 pages, http://international.download.nvidia.com/pdf/kepler/NVIDIA-Kepler-GK110-GK210-Architecture-Whitepaper.pdf.
NVIDIA—NVIDIA's Next Generation CUDA™ Compute Architecture: Kepler™, GK110/210, 2014, 23 pages, http://international.download.nvidia.com/pdf/kepler/NVIDIA-Kepler-GK110-GK210-Architecture-Whitepaper.pdf.
Olofsson, et al., "A 25 GFLOPS/Watt Software Programmable Floating Point Accelerator" High Performance Embedded Computing Conference, 2010, 2 pages.
Oosterhout, Optimized Pixel Template Image Correlator, Master Thesis, Aug. 19, 1992, 74 pages.
Parker, Richard, "Embedding Computer Vision in Electronic Devices: How New Hardware and Software Choices Have Changed the Designer's Approach", Future Technology Magazine, pp. 22-23, Feb. 2013.
Pelc, Oscar, "Multimedia Support in the i.MX31 and i.MX31 L Applications Processors", Freescale Semiconductor, Inc., Feb. 2006, http://cache.freescale.comfiles32bit/doc/white_paper/IMX31MULTIWP.pdf, 12 pages.
Pham, et al., "NeuFlow: Dataflow Vision Processing System-On-A-Chip." IEEE 55m International Midwest Symposium, Aug. 2012, 4 pages.
Qadeer, et al., "Convolution Engine: Balancing Efficiency & Flexibility in Specialized Computing", ISCA '13 Tel-Aviv, Israel, ACM 2013, pp. 24-35.
Ragan-Kelley et al., "Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines," Jun. 16, 2013, 12 pages.
S.F. Reddaway,"DAP-A Distributed Processor Array", ISCA '73 Proceedings of the First Ann. Symposium on Computer Architecture, pp. 61-65.
S.F. Reddaway,"DAP—A Distributed Processor Array", ISCA '73 Proceedings of the First Ann. Symposium on Computer Architecture, pp. 61-65.
SCP2200: Image Cognition Processors Family [Product Brief], https://www.element14.com/community/docs/DOC-50990, Oct. 26, 2012, 2 pages.
SCP2200: Image Cognition Processors, https://www.element14.com/community/docs/DOC-50984/I/scp2200-image-cognition-processors, Oct. 25, 2012, 2 pages.
Sedaghati, et al., "SIVEC: A Vector Instruction Extension for High Performance Stencil Computation," Parallel Architectures and Compilation Techniques, Oct. 10, 2011, 12 pages.
SemiWiki.com-New CEVA-ZM4 Vision IP Does Point clouds and More: Published Feb. 27, 2015, https://www.semiwiki.com/forum/content/4354-new-ceva-xm4-vision-ip-does-point-clouds-more.html.
SemiWiki.com—New CEVA-ZM4 Vision IP Does Point clouds and More: Published Feb. 27, 2015, https://www.semiwiki.com/forum/content/4354-new-ceva-xm4-vision-ip-does-point-clouds-more.html.
Shacham, et al. "Smart Memories Polymorphic Chip Multiprocessor", Proceedings of the 46th Design Automation Conference (OAC), San Francisco, CA, Jul. 2009.
Shacham, et al., "Rethinking Digital Design: Why Design Must Change", IEEE micro Magazine, Nov./Dec. 2010.
Shacham, Ofer, "Chip Multiprocessor Generator: Automatic Generation of Custom and Heterogeneous Complete Platforms" Dissertation-Stanford University, May 2011, 190 pages. Part 1.
Shacham, Ofer, "Chip Multiprocessor Generator: Automatic Generation of Custom and Heterogeneous Complete Platforms" Dissertation-Stanford University, May 2011, 190 pages. Part 2.
Shacham, Ofer, "Chip Multiprocessor Generator: Automatic Generation of Custom and Heterogeneous Complete Platforms" Dissertation—Stanford University, May 2011, 190 pages. Part 1.
Shacham, Ofer, "Chip Multiprocessor Generator: Automatic Generation of Custom and Heterogeneous Complete Platforms" Dissertation—Stanford University, May 2011, 190 pages. Part 2.
Silicon Hive: "Silicon System Design of Media and Communications Applications", Jun. 13, 2007, 22 pages.
Spampinato, et al., "A Basic Linear Algebra Compiler", ACM, CG0'14, Feb. 15-19, 2014, Orlando Fl, pp. 23-32.
SPIR, The SPIR Specification, Version 2.0, Revision Date Jun. 5, 2014, 40 pages.
Stein, et al. "A Computer Vision System on a Chip: A Case Study From the Automotive Domain," IEEE Computer Society Conference, p. 130, 2005.
Stojanov, et al., "Abstracting Vector Architectures in Library Generators: Case Study Convolution Filters", ARRAY 14, ACM, Jun. 11, 2014, UK, pp. 14-19.
Stream Processors, Inc. Announces Storm-1 Family of Data-Parallel Digital Signal Processors, ISSCC 207, Feb. 12, 2007, 3 pages. http://www.businesswire.com/news/home/20070212005230/en/Stream-Processors-Announces-Storm-1-Family-Data-Parallel-Digital.
Stream Processors, Inc., Company History-Foundational Work in Stream Processing initiated in 1995, https://en.wikipedia.org/wiki/Stream_Processors,_inc, 5 pages.
Stream Processors, Inc., Company History—Foundational Work in Stream Processing initiated in 1995, https://en.wikipedia.org/wiki/Stream_Processors,_inc, 5 pages.
Tanabe, et al., "Visconti: multi-VLIW image Recognition Processor Based on Configurable Processor [obstacle detection applications]", Custom Integrated Circuits Conference, IEEE, 2003, http://ieeexplore.ieee.org/document/1249387/?arnumber=1249387&tag=1.
Van der Wal, et al., "The Acadia Vision Processor", IEEE proceedings of International Workshop on Computer Applications for Machine Perception, Sep. 2000, http://citeseerx.ist.psu.edu/ciewdoc/download?doi=10.1.1.32.3830&rep=rep1&type=pdf, 10 pages.
Wahib et al., "Scalable Kernel Fusion for Memory-Bound GPU Applications," SC14: International Conference for High Performance Computing, Networking, Storage and Analysis, Nov. 16, 2014, 12 pages.
Yu et al., "Optimizing Data Intensive Window-based Image Processing on reconfigurable Hardware Boards," Proc. IEEE Workshop on Signal Processing System Design and Implementation, Nov. 2, 2005, 6 pages.
Zhou, Minhua, et al., "Parallel Tools in HEVC for High-Throughput Processing," Applications of Digital Processing, XXXV, Proc. of SPI, vol. 8499, (Jan. 1, 2012), pp. 1-13.
WO2018005036A1 (en) 2018-01-04
DE102017113859A1 (en) 2018-01-04
GB2560208A (en) 2018-09-05
US20180007303A1 (en) 2018-01-04
US20180234653A1 (en) 2018-08-16
DE202017103725U1 (en) 2017-10-04
TWI625697B (en) 2018-06-01
TW201802767A (en) 2018-01-16
CN107563953A (en) 2018-01-09
TW201826219A (en) 2018-07-16
US20180007302A1 (en) 2018-01-04
GB201709786D0 (en) 2017-08-02
Qadeer et al. 2013 Convolution engine: balancing efficiency & flexibility in specialized computing
US20050125369A1 (en) 2005-06-09 System and method for accelerating and optimizing the processing of machine learning techniques using a graphics processing unit
Hegarty et al. 2014 Darkroom: compiling high-level image processing code into hardware pipelines.
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MEIXNER, ALBERT;FINCHELSTEIN, DANIEL FREDERIC;PATTERSON, DAVID;AND OTHERS;SIGNING DATES FROM 20160711 TO 20160715;REEL/FRAME:042792/0211