Semiconductor device and method of controlling the semiconductor device

A semiconductor device includes a dynamic reconfiguration processor that performs data processing for input data sequentially input and outputs the results of data processing sequentially as output data, an accelerator including a parallel arithmetic part that performs arithmetic operation in parallel between the output data from the dynamic reconfiguration processor and each of a plurality of predetermined data, and a data transfer unit that selects the plurality of arithmetic operation results by the accelerator in order and outputs them to the dynamic reconfiguration processor.

CROSS-REFERENCE TO RELATED APPLICATIONS

The disclosure of Japanese Patent Application No. 2018-114861 filed on Jun. 15, 2018 including the specification, drawings and abstract is incorporated herein by reference in its entirety.

BACKGROUND

The present invention relates to a semiconductor device and a control method thereof, and more particularly relates to, for example, a semiconductor device and a control method thereof which are suitable for realizing efficient arithmetic processing.

In addition to Central Processing Units (CPU) there are dynamic reconfiguration processors that perform high processing performance. The dynamic reconfiguration processors is referred Dynamically Reconfigurable Processor (DRP) or array-type processor. The dynamic reconfiguration processor is a processor capable of dynamically reconfiguring a circuit by dynamically switching the operation content of each of a plurality of processor elements and the connection relationship between the plurality of processor elements in accordance with operation instructions sequentially given. A technique related to a dynamic reconfiguration processor is disclosed in, for example, Japanese Patent No. 3674515 (Patent Document 1) as an array processor.

In addition, “SIMD”, <ja.wikipedia.org/wiki/SIMD> (Non-Patent Document 1) and “Mechanisms for 30 times faster mechanical learning with Google Tensor Processing Unit”, <cloudplatform-jp.googleblog.com/2017/05/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu.html> (Non-Patent Document 2) disclose techniques related to parallel arithmetic processing.

SUMMARY

However, the processing performance of the dynamic reconfiguration processor disclosed in Patent Document 1 is insufficient to perform large-scale arithmetic processing such as, for example, deep learning processing. Other objects and novel features will become apparent from the description of this specification and the accompanying drawings.

According to one embodiment, the semiconductor device includes a data processing unit that performs data processing on sequentially input first input data and sequentially outputs the result of data processing as first output data, a parallel arithmetic unit that performs arithmetic processing in parallel between the first output data sequentially output from the data processing unit and each of a plurality of predetermined data, a holding circuit that holds the results of the arithmetic processing, and a first data transfer unit that sequentially selects a plurality of arithmetic processing results held by the accelerator in order and sequentially outputs the results of the arithmetic processing as the first input data.

According to another embodiment, a control method of a semiconductor device performs arithmetic processing on first input data sequentially input using a data processing unit, sequentially outputs the result of arithmetic processing as first output data, performs arithmetic processing in parallel between the first output data sequentially output from the data processing unit and each of a plurality of predetermined data using an accelerator, sequentially selects a plurality of arithmetic processing results output from the accelerator, and sequentially outputs the same as the first input data.

According to the above-mentioned embodiment, it is possible to provide a semiconductor device capable of realizing efficient arithmetic processing and a control method thereof.

DETAILED DESCRIPTION

For clarity of explanation, the following description and drawings are appropriately omitted and simplified. The respective elements described in the drawings as functional blocks for performing various processes can be configured by a CPU (Central Processing Unit), a memory, and other circuits in terms of hardware, and are realized by programs loaded in the memory in terms of software. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof, and the present invention is not limited to any of them. In the drawings, the same elements are denoted by the same reference numerals, and a repetitive description thereof is omitted as necessary.

The programs described above may be stored and provided to a computer using various types of non-transitory computer readable media. Non-transitory computer readable media includes various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R/W, solid-state memories (e.g., masked ROM, PROM (Programmable ROM), EPROM (Erasable PROM, flash ROM, RAM (Random Access Memory)). The program may also be supplied to the computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable medium may provide the program to the computer via wired or wireless communication paths, such as electrical wires and optical fibers.

First Embodiment

FIG. 1is a block diagram showing a configuration example of a semiconductor system SYS1on which a semiconductor system1according to a first embodiment of the present invention is mounted. The semiconductor device1according to the present embodiment includes an accelerator having a parallel arithmetic part that performs parallel arithmetic operation, a data processing unit such as a dynamic reconfiguration processor that sequentially performs data exchange, and a data transfer unit that sequentially selects and sequentially outputs from a plurality of arithmetic processing results by the accelerator to the data processing unit. As a result, the semiconductor device1and the semiconductor system SYS1including the semiconductor device1according to the present embodiment can perform a large amount of regular data processing using an accelerator, and perform other data processing using a data processing unit, thereby realizing efficient arithmetic processing. Hereinafter, a concrete description will be given.

As shown inFIG. 1, the semiconductor system SYS1includes a semiconductor device1, a CPU2, and an external memory3. The semiconductor device1includes a dynamic reconfiguration processor (hereinafter referred to as a DRP)11, an accelerator12, a data transfer unit13, a data transfer unit14, and a Direct Memory Access (DMA)15.

The DRP11executes arithmetic processing on data sequentially inputted from the external memory3, for example, and sequentially outputs the result of the arithmetic processing as a data DQout. In this manner, the DRP11can transmit and receive data every cycle. Here, the DRP11is a data processor capable of dynamically reconfiguring circuits by dynamically switching the operation content of each of a plurality of processor elements and the connections between the plurality of processor elements in accordance with operation instructions read from a configuration data memory provided in the DRP11.

For example, the DRP11includes a plurality of processor elements provided in array, a plurality of switching elements provided corresponding to the plurality of processor elements, and a status managing unit. The state management unit issues an instruction pointer determined in advance by a program to each of the processor elements. Each of the processor elements comprises, for example, at least an instruction memory and an arithmetic unit. The arithmetic unit performs arithmetic processing in accordance with an operation instruction specified by an instruction pointer from the state management unit, among a plurality of operation instructions stored in the instruction memory. The arithmetic unit may be, for example, a 16-bit arithmetic unit that performs arithmetic processing on 16-bit width data, or an arithmetic unit that performs arithmetic processing on other bit width data. Alternatively, the arithmetic unit may be configured by a plurality of arithmetic units. Each of the switch elements sets a connection relationship between a corresponding processor element and another processor element in accordance with an operation instruction read from an instruction memory of the corresponding processor element. Thereby, the DRP11can dynamically switch the circuitry in accordance with the sequentially applied operation instructions.

In this embodiment, the DRP11is provided in the semiconductor device1, but it is not limited thereto. For example, a central processing unit (CPU) may be provided instead of the DRP11, as long as the CPU performs arithmetic processing on sequentially inputted data.

The data transfer unit13distributes or serializes the data DQout in accordance with, for example, the degree of parallelism of the arithmetic processing required for the parallel arithmetic part121, and outputs the data as data DPin.

The accelerator12performs an arithmetic operation between the data DPin sequentially outputted from the data transfer unit13and n (n is an integer equal to or greater than 2) pieces of predetermined data D_0to D_(n−1) in parallel. In the following description, the predetermined data D_0to D_(n−1) are not distinguished and may be simply referred to as predetermined data D.

Specifically, the accelerator12includes a parallel arithmetic part121and a local memory122. The local memory122stores, for example, a plurality of pieces of predetermined data D_0to D_(n−1) read from the external memory3and initial setting information such as a bias value b.

For example, when k×m elements constituting matrix data with k rows and m columns are successively input to the accelerator12as data DPin, k rows each having m data are input to the accelerator12sequentially, i.e., k×m data. However, regardless of the value of k, the accelerator12uses the predetermined data D_0to D_(n−1) for each of the m data, which are input data for one row, for arithmetic processing. Therefore, n pieces of predetermined data D_0to D_(n−1), i.e., m×n pieces of data corresponding to m data corresponding to one row of input data are stored in the local memory122. The parallel arithmetic part121is configured by a plurality of arithmetic units that perform arithmetic processing in parallel. The parallel arithmetic part121performs arithmetic operation in parallel between the data DPin and each of the plurality of predetermined data D_0to D_(n−1), and outputs n arithmetic processing results as a data DPout.

The data transfer unit14sequentially selects n pieces of data DPout output in parallel from the accelerator12, and sequentially outputs the selected pieces of data as a data DQin.

The DRP11performs arithmetic processing on the data DQin sequentially outputted from the data transfer unit14, and sequentially outputs the results of the arithmetic processing to, for example, the external memory3.

The CPU2controls the operation of the semiconductor device1in accordance with a control instruction read from the external memory3, for example. More specifically, the CPU2prepares data strings (descriptors) for instructing operations of the accelerators12and the data transfer units13and14in detail, and stores the data strings (descriptors) in the external memory3.

The DMA15reads the descriptors from the external memory3, interprets the contents, and issues an operation instruction to the accelerator12and the data transfer units13and14. For example, the DMA15transfers an initial setting information stored in the external memory3to the local memory122in accordance with an instruction described in the descriptor. The DMA15instructs the data transfer unit13to distribute or serialize the data DPin in accordance with the degree of parallelism of the arithmetic processing by the parallel arithmetic part121. The DMA15instructs the data transfer unit14to combine or serialize the n pieces of data DPout outputted in parallel in accordance with the degree of parallelism of the arithmetic processing by the parallel arithmetic part121.

When the operation specified by one descriptor is completed, the DMA15reads the next descriptor from the external memory3, and issues an operation instruction to the accelerator12and the data transfer units13and14. It is preferable that the descriptor is read prior to the completion of the operation by the descriptor read immediately before. Thereby, the processing latency can be hidden.

The descriptors may be applied from programs operating in the DRP11instead of the CPU2, or may be generated in advance.

FIG. 2is a block diagram showing a specific configuration example of the semiconductor device1. InFIG. 2, the DRP11outputs data DQout of 64-bit width of 4 channels as data DQout_0to DQout_3. The DRP11is not limited to outputting data DQout_0to DQout_3of four channels, and can be appropriately changed to a configuration for outputting data of any number of channels and any number of bit widths.

The parallel arithmetic part121includes, for example, parallel arithmetic units MAC256_0to MAC256_3. Each of the parallel arithmetic units MAC256_0to MAC256_3includes 256 arithmetic units that perform arithmetic processing in parallel. The data DPin_0to DPin_3are input to the parallel arithmetic units MAC256_0to MAC256_3, respectively.

The parallel arithmetic unit MAC256_0outputs a maximum of 256 arithmetic processing results by executing arithmetic processing in parallel using a maximum of 256 arithmetic units (four sets of 64 units) with respect to the 64-bit width (16-bit width×4 sets) data DPin_0.

Similarly, the parallel arithmetic unit MAC256_1outputs a maximum of 256 arithmetic processing results by executing arithmetic processing in parallel using a maximum of 256 arithmetic units (four sets of 64 units) with respect to the 64-bit width (16-bit width×4 sets) data DPin_1. The parallel arithmetic unit MAC256_2outputs a maximum of 256 arithmetic processing results by executing arithmetic processing in parallel using a maximum of 256 arithmetic units (four sets of 64 units) with respect to the 64-bit width (16-bit width×4 sets) data DPin_2. The parallel arithmetic unit MAC256_3outputs a maximum of 256 arithmetic processing results by executing arithmetic processing in parallel using a maximum of 256 arithmetic units with respect to data DPin_3having a width of 64 bits (16 bits width×4 sets).

FIG. 3is a block diagram showing a configuration example of the parallel arithmetic unit MAC256_0.FIG. 3also shows the data transfer units13and14provided before and after the parallel arithmetic unit MAC256_0.

As shown inFIG. 3, the parallel arithmetic unit MAC256_0includes parallel arithmetic units MAC64_0to MAC64_3. Each of the parallel arithmetic units MAC64_0to MAC64_3is composed of arithmetic units that perform arithmetic processing in parallel.

The 0th bit to 15th bit (hereinafter referred to as data DPin_00) of the 64-bit width data DPin_0are input to the parallel arithmetic unit MAC64_0. The 16th bit to 31st bit (hereinafter referred to as data DPin_01) of the 64-bit width data DPin_0are input to the parallel arithmetic unit MAC64_1. The 32nd bit to 47th bit (hereinafter, referred to as data DPin_02) of the 64-bit width data DPin_0are input to the parallel arithmetic unit MAC64_2. The 48th bit to 63rd bit of the 64-bit width data DPin_0(hereinafter referred to as the data DPin_03) are input to the parallel arithmetic unit MAC64_3.

The parallel arithmetic unit MAC64_0performs arithmetic processing on 16-bit width data DPin_00in parallel using a maximum of 64 arithmetic units and output an arithmetic processing result of a maximum of 64 of the arithmetic processing results each having 16-bit width. The parallel arithmetic unit MAC64_1performs arithmetic processing on 16-bit width data DPin_01in parallel using a maximum of 64 arithmetic units and output a maximum of 64 arithmetic processing results each having 16-bit width. The parallel arithmetic unit MAC64_2can perform arithmetic processing on 16-bit width data DPin_02in parallel using a maximum of 64 arithmetic units and output a maximum of 64 arithmetic processing results each having 16-bit width. The parallel arithmetic unit MAC64_3can perform arithmetic processing on 16-bit width data DPin_03in parallel using a maximum of 64 arithmetic units and output a maximum of 64 arithmetic processing results each having 16-bit width.

The parallel arithmetic units MAC256_1to MAC256_3have the same configuration as that of the parallel arithmetic unit MAC256_0, and therefore description thereof is omitted.

Returning toFIG. 2, the description will be continued. The parallel arithmetic unit MAC256_0performs arithmetic processing on data DPin_0having a 64-bit width (16-bit width×4 sets), and outputs four sets of a maximum of 64 arithmetic processing results each having 16-bit width as data DPout_0.

Similarly, the parallel arithmetic unit MAC256_1performs arithmetic processing on the data DPin_1, and outputs four sets of a maximum of 64 arithmetic processing results each having 16-bit width as data DPout_1. The parallel arithmetic unit MAC256_2performs arithmetic processing on the data DPin_2, and outputs four sets of a maximum of 64 arithmetic processing results each having 16-bit width as data DPout_2. The parallel arithmetic unit MAC256_3performs arithmetic processing on the data DPin_3, and outputs four sets of a maximum of 64 arithmetic processing results each having 16-bit width as data DPout_3.

The data transfer unit14, for example, selects one by one from each of the four sets each having 64 16-bit width data included in the data DPout_0output in parallel from the parallel processor MAC256_0and sequentially outputs the data DQin_0which comprises the four sets each having 16-bit width data (i.e., the data DQin_0of the 64 bit width). As described above, the data transfer unit14may select and sequentially output 16-bit width data one by one from each set, or may sequentially output all data for each set so as to output 64 16-bit width data in one set and then output 64 16-bit width data in the next set, but the present invention is not limited thereto. The data output method of the data transfer unit14may be switched depending on the mode.

Similarly, the data transfer unit14sequentially selects, for example, one by one from each of four sets each having 64 16-bit width data included in the data DPout_1output in parallel from the parallel arithmetic unit MAC256_1, and sequentially outputs the data DQin_1which comprises four sets of 16-bit width data (i.e., 64-bit width data DQin_1). In addition, the data transfer unit14selects one by one from each of four sets each having 64 16-bit width data included in the data DPout_2output in parallel from the parallel processor MAC256_2, and outputs the data DQin_2which comprises four sets of the 16-bit width data (i.e., the 64-bit width data DQin_2) in sequence. The data transfer unit14sequentially selects, for example, one by one from each of four sets each having 64 16-bit width data in DPout_3output in parallel from the parallel processor MAC256_3, and outputs the data DQin_3which comprises four sets of 16-bit width data (i.e., a 64-bit width data DQin_3).

These 64-bit width data DQin_0to DQin_3are inputted to the DRP11. The DRP11performs arithmetic processing on the data DQin_0to DQin_3, and sequentially outputs the arithmetic processing results to the external memory3. The data DQin_0to DQin_3may be used for calculation of the data DQout_0to DQout_3.

As described above, the semiconductor device1according to the present embodiment includes an accelerator having a parallel arithmetic part that performs arithmetic processing in parallel, a data processing unit such as DRP that sequentially transfers data, and a data transfer unit that sequentially selects and outputs a plurality of arithmetic processing results by the accelerator to the data processing unit. As a result, the semiconductor device according to the present embodiment and the semiconductor system including the same can perform a large amount of regular data processing using an accelerator and perform other data processing using a data processing unit, so that efficient arithmetic processing can be realized even in a large-scale arithmetic processing such as, for example, a deep learning processing.

Hereinafter, a calculation method of a neural network using the semiconductor device1according to the present embodiment will be described with reference toFIGS. 4 and 5.FIG. 4is a diagram showing an example of a neural network structure.FIG. 5is a diagram schematically showing the flow of the operation processing of the neural network.

As shown inFIG. 4, the operation of the neural network takes a procedure of performing a multiply-and-accumulate calculation operation of multiplying the input data with the weight w(w′), performing an operation such as activation on the result, and outputting the operation result.

As shown inFIG. 5, the DRP11reads out data required for the arithmetic processing of the accelerator12from the external memory3(step S1), and rearranges the calculator and the data as necessary (step S2). Thereafter, the data read from the external memory3is sequentially outputted from the DRP11to the accelerator12as data inputted to the accelerator12. The accelerator12performs a parallel multiply-and-accumulate calculation operation by multiplying the data sequentially outputted from the DRP11with the data (corresponding to the weight) stored in the local memory in order from the received data from the DRP11(step S4). Then, the arithmetic results by the accelerator12are sequentially outputted to the DRP11(step S5). The DRP11performs operations such as addition and activation on the data received from the accelerator12as required (step S6). The processing results of the DRP11are stored in the external memory3(step S7). By realizing the processing of the neural network by such processing and repeating this processing, it is possible to execute the arithmetic processing required for the deep learning.

In this manner, in the neural network, it is possible to realize high-speed operation by executing regular parallel multiply-and-accumulate calculation operation using the accelerator12among required operations. Further, the DRP11which is a data processor capable of dynamically reconfiguring circuits executes the arithmetic processing other than the regular parallel multiply-and-accumulate calculation operation, it becomes possible to flexibly set the processing such as activation in different layers (first layer and second layer in the example ofFIG. 5). In addition, the DRP11can reconfigure the circuit configuration so that the input data required for the multiply-and-accumulate calculation operation is divided and read out from the external memory3to be output to the accelerator12in accordance with the parallel operation size which can be simultaneously processed by the accelerator12. Thereby, the degree of freedom of the operation format of the parallel arithmetic part121can be provided.

Next, the operation of the semiconductor system SYS1will be described with reference toFIG. 6.FIG. 6is a timing chart showing a processing flow of the semiconductor system SYS1.

Hereinafter, a case where matrix operation is performed by the accelerator12will be described as an example.FIG. 7is a diagram schematically showing a matrix arithmetic expression. InFIG. 7, multiplication operation of matrix data In composed of elements of k rows×m columns and matrix data W composed of elements of m rows×n columns is performed, and the result of adding the bias value b to each element of the multiplication result is output as matrix data Out composed of elements of k rows×n columns.

When the accelerator12performs calculation operation on the matrix data In of the first layer, initial setting information including the matrix data W and the bias value b corresponding to the matrix data In of the first layer is stored in the local memory122of the accelerator12(times t1to t2inFIG. 6) (seeFIG. 8). More specifically, the DMA15transfers the initial setting information read from the external memory3to the local memory122in accordance with the instruction of the descriptor generated by the CPU2. Note that a DMA (not shown) dedicated to the accelerator12may be provided separately from the DMA15, and initial setting information read from the external memory3may be transferred to the local memory122using the DMA dedicated to the accelerator12.

Thereafter, the first row data of the matrix data In (hereinafter, also referred to as row data In1) is read from the external memory3(time t2inFIG. 6). The DRP11outputs the row data In1read from the external memory3to the accelerator12after performing a predetermined process as needed (time t3ofFIG. 6).

The accelerator12sets the bias value b read from the local memory122as an initial value in the respective arithmetic units of the parallel arithmetic part121, and then performs an arithmetic process of the row data In1(corresponding to the data DPin) and the matrix data W (corresponding to the predetermined data D) read from the local memory122(time t4inFIG. 6).

FIG. 9is a diagram showing a specific example of a multiplication expression of row data In1(the first row data of matrix data In) and matrix data W. InFIG. 9, it is assumed that the row data In1is composed of 20 columns of elements b0to b19. In the matrix data W, it is assumed that the first row data is composed of elements a0,0a0,1. . . a0,19of 20 columns, the second row data is composed of elements a1,0a1,1. . . a1,19of 20 columns, and the 20th row data, which is the last row, is composed of elements a19,0a19,1. . . a19,19of 20 columns.

Here, the accelerator12performs multiplication operation in parallel on the elements (e.g., b0) of each column of the row data In1and the elements (e.g., a0,0a0,1. . . a0,19) of the 20 columns of each row of the matrix data W, and then adds the multiplication operation results of 20 pieces in each column to calculate the elements of each column of the matrix data Out.

FIG. 10is a diagram showing a specific configuration example of the accelerator12. In the example ofFIG. 10, 20 arithmetic units121_0to121_19among a plurality of arithmetic units provided in the parallel arithmetic part121are used. Each of the arithmetic units121_0to121_19includes a multiplier MX1, an adder AD1, a register RG1and a register RG2.

In the arithmetic unit121-j(j is any one of 0 to 19), the bias value b read from the local memory122is set as an initial value in the register RG1(the bias value b is not shown inFIG. 10).

Thereafter, the multiplier MX1multiplies the element b0of the first column data in the row data In1(corresponding to the 16-bit width data DPin) by the elements a0,jof first row in the matrix data W read from the local memories122(corresponding to the predetermined data D_j of 16-bit width). The adder AD1adds the multiplication result (a0,j×b0) by the multiplier MX1and the value (bias value b) stored in the register RG1and transfers the addition result to the register RG1.

After that, the multiplier MX1multiplies the element b1of the second column in the row data In1input subsequently by the element a1,jof the second row in the matrix data W read from the local memory122. The adder AD1adds the multiplication result (a1,j×b1) by the multiplier MX1and the value (a0,j×b0) stored in the register RG1and transfers the addition result to the register RG1.

Since the operations of multiplying, adding, and storing as described above are repeated for 20 cycles, the register RG1stores the element of the first row in the matrix data Out ((a0,j×b0)+(a1,j×b1)+·+·+(a19,j×b19)). Thereafter, the value stored in the register RG1is transferred to the register RG2, and the value stored in the register RG2is output as an element of the first row of the matrix data Out after time t5inFIG. 6.

When the data transfer from the register RG1to the register RG2is completed (time t5ofFIG. 6), it is possible to start the arithmetic operation by the arithmetic unit121_jfor the data of the second row (also called row data In2), which is the next row in the matrix data In (time t6ofFIG. 6). Thereby, the accelerator12can execute the parallel arithmetic operation on the row data In2(times t6to t9inFIG. 6) while transferring the arithmetic operation result stored in the register RG2to the data transfer unit14(corresponding to times t7to t10inFIG. 6). As a result, the efficiency of the parallel arithmetic operation can be increased.

Therefore, it is preferable for the DRP11to receive the arithmetic operation result of row data Int by the accelerator12during an output period of the second row data In2, which is a period from the completion of the output of the first row data In1in the matrix data In to the start of the output of the third row data In3(seeFIG. 11).

The data transfer unit14sequentially selects the 20 arithmetic operation results each having 16-bit width (corresponding to the data DPout) outputted from the arithmetic units121_0to121_19, and sequentially outputs them as a 16-bit width data DQin. In other words, the data transfer unit14sequentially outputs the elements of the twenty columns of the first row of the matrix data Out as the data DQin. The sequentially outputted data DQin is received by the DRP11at times t7to t10inFIG. 6.

In the DRP11, for example, the adder AD2performs addition processing on the data DQin sequentially outputted from the data transfer unit14, the arithmetic unit TN1performs predetermined arithmetic operation based on the hyperbolic tangent functions, and the multiplier MX2performs multiplication operation. The operation result is written to the external memory3, for example, at times t8to t11inFIG. 6.

When the accelerator12completes the arithmetic operation for all the row data from the first row to the k-th row of the matrix data In of the first layer, the same arithmetic operation is subsequently performed for the matrix data In of the second layer. Before the arithmetic operation is performed on the matrix data In of the second layer, initial setting information (matrix data W and bias value b) corresponding to the matrix data In of the second layer is stored in the local memory122. The accelerator12repeats such parallel arithmetic operation.

It is preferable that the local memory122has a storage area to store initial setting information corresponding to the matrix data In of at least two layers, i.e., the matrix data W and the bias value b. Thereby, during execution of the matrix operation on the matrix data In of the first layer, the initial setting information used for the operation on the matrix data In of the second layer can be transferred to the free area of the local memory122. Thereby, after completion of the arithmetic operation for the matrix data of the first layer, the matrix calculation for the matrix data of the second layer can be quickly executed without waiting for the transfer of the initial setting information, as shown inFIG. 12. In this case, it is preferable that the local memory122is configured to be capable of reading and writing data at the same time.

On the other hand, even if the local memory122does not have enough storage space to store the initial setting information corresponding to one layer of matrix data In, or has storage space to store the initial setting information corresponding to one layer of matrix data In, the initial setting information may be divided and stored. Hereinafter, a brief description will be given with reference toFIG. 13.

FIG. 13is a flow chart showing the operation of the semiconductor system SYS1. In the example ofFIG. 13, it is assumed that the local memory122does not have a storage area sufficient to store the initial setting information corresponding to the matrix data In of the third layer.

As shown inFIG. 13, in step S101, initial setting information corresponding to the matrix data In of the first layer is stored in the local memory122. Thereafter, in step S102, the parallel arithmetic part121performs an arithmetic operation on the matrix data In of the first layer. Thereafter, the initial setting information corresponding to the matrix data In of the second layer is stored in the local memory122in step S103. Thereafter, in step S104, the parallel arithmetic part121performs an arithmetic operation on the matrix data In of the second layer. Thereafter, in step S105, initial setting information corresponding to a part of the matrix data In of the third layer is stored in the local memory122. In step S106, the parallel arithmetic part121performs an arithmetic operation on a part of the matrix data In of the third layer. In step S107, the initial setting information corresponding to the remaining matrix data In of the third layer is stored in the local memory122. In step S108, the parallel arithmetic part121performs an arithmetic operation on the remaining matrix data In of the third layer. Thereafter, the result of the arithmetic operation executed in step S106and the result of the arithmetic processing executed in step S108are added in the DRP11in step S109. Thereby, it is possible to realize the arithmetic operation on the matrix data In of the third layer.

As described above, the semiconductor device1according to the present embodiment includes an accelerator having a parallel arithmetic part that performs arithmetic operation in parallel, a data processing unit such as DRP that sequentially transfers data, and a data transfer unit that sequentially selects and outputs a plurality of arithmetic operation results by the accelerator to the data processing unit. As a result, the semiconductor device according to the present embodiment and the semiconductor system including the semiconductor device perform a large amount of regular data processing using an accelerator and perform other data processing using a data processing unit, so that efficient arithmetic processing can be realized even in a large-scale arithmetic processing such as, for example, a deep learning processing.

In the present embodiment, the case where each of the arithmetic units121_0to121_19includes the register RG2in addition to the multiplier MX1, the adder AD1, and the register RG1has been described as an example, but the present invention is not limited thereto. Each of the arithmetic units121_0to121_19may include the multiplier MX1, the adder AD1, and the register RG1, and may not include the register RG2. This further suppresses the circuit scale.

In the present embodiment, the case where the bias value b is stored in the local memory122has been described as an example, but the present invention is not limited to this. For example, the bias value b may be stored in a register or the like provided separately from the local memory122, or the bias value b may be a fixed value such as 0 and may not be stored in the local memory122.

FIG. 14is a diagram showing a configuration example of the accelerator52according to the comparative example. As shown inFIG. 14, in the accelerator52, each of the arithmetic units121_0to121_19includes a multiplier MX1, an adder AD1, a register RG1, an adder AD2, an arithmetic unit TN1, and a multiplier MX2. That is, in the accelerator52, the adder AD2, the arithmetic unit TN1and the multiplier MX2which are provided in the DRP11in the accelerator12, are provided in the arithmetic units121_0to121_19. However, in the accelerator52, after the arithmetic operation processing by the multiplier MX1, the adder AD1and the register RG1is repeated for 20 cycles in each arithmetic unit, the arithmetic operation processing by the adder AD2, the arithmetic unit TN1and the multiplier MX2is executed for only one cycle. That is, in the accelerator52, since the adder AD2, the arithmetic unit TN1and the multiplier MX2which are used less frequently are provided in all of the plurality of arithmetic units, there is a problem that the circuit scale increases.

On the other hand, in the accelerator12, the arithmetic units121_0to121_19do not include the adder AD2, the arithmetic unit TN1and the multiplier MX2, which are used infrequently, and these arithmetic units are configured and commonly used in the preceding stage of the DRP11. Thereby, an increase in the circuit scale can be suppressed.

Configuration Example of the Parallel Arithmetic Units

Next, a specific configuration example of a plurality of arithmetic units provided in the parallel arithmetic part121will be described.FIG. 15is a diagram showing a specific configuration example of the parallel arithmetic unit MAC64_0. As shown inFIG. 15, the parallel arithmetic unit MAC64_0includes 64 arithmetic units121_0to121_63that perform arithmetic operation processing in parallel. Each of the arithmetic units121_0to121_63includes a multiplier MX1, an adder AD1, a register RG1, and a register RG2. Here, the paths of the multiplier MX1, the adder AD1, the register RG1and the register RG2in the arithmetic units121_0to121_63perform predetermined arithmetic operation processing on the 16-bit width data, and output 16-bit width data.

Since the parallel arithmetic units MAC64_1to MAC64_3have the same configuration as that of the parallel arithmetic unit MAC64_0, their descriptions are omitted.

First Modification of the Parallel Arithmetic Units

FIG. 16is a diagram showing a first modification of the parallel arithmetic unit MAC64_0as the parallel operator MAC64a_0. As shown inFIG. 16, the parallel arithmetic unit MAC64a_0includes 64 arithmetic units121a_0to121a_63. Each of the arithmetic units121a_0to121a_63includes a selector SL1, a multiplier MX1, an adder AD1, a register RG1, and a register RG2.

The selector SL1sequentially selects and outputs 16-bit data read from the local memory122bit by bit. The paths of the multiplier MX1, the adder AD1, the register RG1, and the register RG2perform arithmetic operation processing using the 1-bit width data output from the selector SL1and the 16-bit width data from the data transfer unit13, and output 16-bit width data.

In this way, even when the parallel arithmetic unit MAC64a_0carries out the arithmetic operation process for the data having 1-bit width read out from the local memory122, it is possible to suppress the increase in the number of readings from the local memory122by reading the data having a 16-bit width from the local memory122and then sequentially selecting one bit from the data having a 16-bit width and performing the arithmetic operation processing. As a result, power consumption can be reduced.

The parallel arithmetic units MAC64a_1to MAC64a_3have the same configuration as that of the parallel arithmetic unit MAC64a_0, and therefore description thereof is omitted.

It should be noted that when the arithmetic operation processing is performed on the 1-bit width data read out from the local memory122, the multiplication processing means multiplying the data from the data transfer unit13by either +1 or −1. Therefore, the multiply and accumulate calculation operation adds or subtracts the data from the data transfer unit13to or from the data stored in the register RG1. This can also be realized by the configuration of the parallel arithmetic unit as shown inFIG. 17.

Second Modification of the Parallel Arithmetic Units

FIG. 17is a diagram showing a second modification of the parallel operator MAC64__0as the parallel operator MAC64b_0. As shown inFIG. 17, the parallel arithmetic unit MAC64b_0includes 64 arithmetic units121b_0to121b_63. Each of the arithmetic units121b_0to121b_63includes a selector SL1, an adder AD1, a subtractor SB1, a selector SL2, a register RG1, and a register RG2.

Here, the selector SL1sequentially selects and outputs 16-bit data read from the local memory122bit by bit. The adder AD1adds the 16-bit width data from the data transfer unit13and the data stored in the register RG1. The subtractor SB1subtracts the data stored in the register RG1from the 16-bit width data from the data transfer unit13. The selector SL2selects and outputs either the addition result by the adder AD1or the subtraction result by the subtractor SB1based on the value of the 1-bit width data output from the selector SL1. The data output from the selector SL2is stored in the register RG1. Thereafter, the data stored in the register RG1is stored in the register RG2and then output to the data transfer unit14.

The parallel arithmetic unit MAC64b_0can realize the same operation as the parallel arithmetic unit MAC64a_0.

The parallel arithmetic units MAC64b_1to MAC64b_3have the same configuration as that of the parallel arithmetic unit MAC64b_0, and therefore description thereof is omitted.

Third Modification of the Parallel Arithmetic Units

FIG. 18(a third modification of a plurality of arithmetic units comprising a parallel arithmetic unit) shows a third modification of the parallel arithmetic unit MAC64_0as the parallel operator MAC64c_0. As shown inFIG. 18, the parallel arithmetic unit MAC64c_0includes 64 arithmetic units121c_0to121c_63. Each of the arithmetic units121c_0to121c_63performs arithmetic operation processing between 16 pieces of 1-bit data from the data transfer unit13and 16 pieces of 1-bit data read from the local memory122in units of 1 bit.

Each of the arithmetic units121c_0to121c_63includes 16 paths including a multiplier MX1, an adder AD1, a register RG1, and a register RG2. Here, each path performs arithmetic operation processing by using one of 16 pieces of 1-bit data from the data transfer unit13and one of 16 pieces of 1-bit data read from the local memory122, and outputs 1-bit data. The 1-bit data is represented by binary values of 1 and 0 in hardware, and these values of 1 and 0 are used for calculation as +1 and −1, respectively, in meaning.

As described above, the parallel calculator MAC64c_0can perform 16 arithmetic operation processes for 1-bit data by transferring and reading data using a 16-bit data path, even when the calculation process is performed using 1-bit data from the data transfer unit131and 1-bit data read from the local memory122.

The operation of the configuration shown inFIG. 18can also be realized by the configuration of the parallel arithmetic unit as shown inFIG. 19.

Fourth Modification of the Parallel Arithmetic Units

FIG. 19is a diagram showing a fourth modification of the parallel operator MAC64_0as the parallel operator MAC64d_0. As shown inFIG. 19, the parallel arithmetic unit MAC64d_0includes 64 arithmetic units121d_0to121d_63. The arithmetic units121d_0to121d_63include XNOR circuits XNR1, pop counters CNT1, an adder AD1, a register RG1, and a register RG2.

The XNOR circuit XNR1performs a negative exclusive OR operation on 16 pieces of 1-bit data from the data transfer unit13and 16 pieces of 1-bit data read from the local memory122in units of 1 bit. The pop counter CNT1counts the number of “1” output values when the output values of the XNOR circuits XNR1are viewed in binary units. Here, the output value of the pop counter CNT1represents the number of bits having the same output value when the output value of the pop counter CNT1represents the number of bits having the same value when the 16-bit data from the data transfer unit13and the 16-bit data read from the local memory122are viewed as binary numbers. The output data of the pop counter CNT1is added to the data stored in the register RG1by the adder AD1. However, since the values to be +1 and −1 are originally calculated as 1 and 0, it is necessary to correct the output value. It is also possible to cope with this problem by processing the bias value necessary for correction in advance.

As described above, the parallel arithmetic unit MAC64d_0performs arithmetic operation processing in units of 1-bit between 16 pieces of 1-bit data from the data transfer unit13and 16 pieces of 1-bit data read from the local memory122in parallel by 16 pieces, adds these pieces of arithmetic operation processing, and outputs the result as 16-bit data. Thereby, the parallel arithmetic unit MAC64d_0can realize the same operation as that of the parallel arithmetic unit MAC64d_0.

The parallel arithmetic units MAC64d_1to MAC64d_3have the same configuration as that of the parallel arithmetic unit MAC64d_0, and therefore description thereof is omitted. Fifth modification of the parallel arithmetic units

FIG. 20(a fifth modification of a plurality of operators comprising a parallel calculator) shows a fifth modification of the parallel calculator MAC64_0as the parallel calculator MAC64e_0. The parallel operator MAC64e_0includes 64 operators121e_0to121e_63.

The arithmetic units121e_0to121e_63further include 1-bit conversion circuits CNV1for converting 16-bit width data stored in the register RG1into 1-bit width data, as compared with the arithmetic units121d_0to121d_63. The 1-bit conversion circuit CNV1can output the activated value as a 1-bit value by outputting 0 when the operation result is negative and 1 otherwise, for example, by using the bias value. In this case, 64 pieces of 1-bit data from the arithmetic units121e_0to121e_63are input to the data transfer unit14. It should be noted that the data transfer unit14can also output 64 pieces of 1-bit data as 16-bit width data by bundling them. Thus, the data transfer unit14can output 64 pieces of 1-bit data in four cycles.

Sixth Modification of the Parallel Arithmetic Units

FIG. 21is a diagram showing a sixth modification of the parallel operator MAC64_0as the parallel arithmetic unit MAC64f_0. The parallel arithmetic unit MAC64f_0includes 64 arithmetic units121e_0to121e_63.

The arithmetic unit121e_0includes arithmetic units121_0,121a_0,121c_0, and121e_0and a selector SL3. The selector SL3selects one of the arithmetic units121_0,121a_0,121c_0, and121e_0according to the mode and outputs the selected one. The arithmetic units121e_1to121e_63have the same configuration as that of the arithmetic unit121e_0, and therefore description thereof is omitted. Note that a part of the arithmetic unit121e_0and a part of the arithmetic unit121c_0can have a common circuit, and whether to output 16 bits as it is or via a 1-bit conversion circuit may be selected. The mode may be fixedly specified, for example, by setting a register by the CPU, or may be specified for each descriptor by describing information of the mode to be specified in the descriptor.

In this manner, the parallel arithmetic unit MAC64f_0can switch the content of the arithmetic operation processing according to the required arithmetic accuracy, memory usage, and throughput. The parallel arithmetic units MAC64e_1to MAC64e_3have the same configuration as that of the parallel arithmetic unit MAC64e_0, and therefore description thereof is omitted.

Example of Data Transfer by the Data Transfer Unit13

Next, an example of data transfer from the DRP11to the accelerator12by the data transfer unit13will be described. Hereinafter, examples of data transfer by the data transfer unit13in accordance with a mode of an operation in which data is input from the DRP11to the accelerator12via the data transfer unit13, hereinafter referred to as an input mode, will be described.

FIG. 22is a diagram showing the parallel arithmetic unit MAC256_0, the data transfer unit13and the accelerator12when the input mode is the first input mode. In this case, the data transfer unit13outputs 64-bit (16-bit×4) data DQout_0as it is as data DPin_0using the selection circuit131. The 16-bit data DPin_00to DPin_03constituting the 64-bit data DPin_0are input to the parallel arithmetic units MAC64_0to MAC64_3, respectively.

The relationship between the data transfer unit13and the parallel arithmetic units MAC256_1to MAC256_3is the same as the relationship between the data transfer unit13and the parallel arithmetic unit MAC256_0, the description thereof is omitted.

FIG. 23is a diagram showing the parallel arithmetic unit MAC256_0, the data transfer unit13and the accelerator12when the input mode is the second input mode. In this case, the data transfer unit13uses the selection circuit131to divide the data DQout_00into two pieces of 16-bit data DQout_00and DQout_02constituting the data DQout_0of 32 bits (16 bits×2) and output the divided pieces of 16-bit data DPin_00and DPin_01, and also divides the data DQout_02into two pieces and outputs the divided pieces of 16-bit data DPin_02and DPin_03. These 16-bit data DPin_00to DPin_03are input to the parallel arithmetic units MAC64_0to MAC64_3, respectively.

The relationship between the data transfer unit13and the parallel arithmetic units MAC256_1to MAC256_3is the same as the relationship between the data transfer unit13and the parallel arithmetic unit MAC256_0, the description thereof is omitted.

FIG. 24is a diagram showing the parallel arithmetic unit MAC256_0, the data transfer unit13and the accelerator12when the input mode is the third input mode. In this case, the data transfer unit13uses the selection circuit131to distribute the 16-bit data DQout_0to four pieces of data, and outputs the divided pieces of data as 16-bit data DPin_00to DPin_03. These 16-bit data DPin_00to DPin_03are input to the parallel arithmetic units MAC64_0to MAC64_3, respectively.

The relationship between the data transfer unit13and the parallel arithmetic units MAC256_1to MAC256_3is the same as the relationship between the data transfer unit13and the parallel arithmetic unit MAC256_0, the description thereof is omitted.

FIG. 25is a diagram showing the parallel arithmetic unit MAC256_0, the data transfer unit13and the accelerator12when the input mode is the fourth input mode. In this case, the data transfer unit13alternately selects the 16-bit data DQout_00and DQout_01out of the 16-bit data DQout_00to DQout_03composing the 64-bit data DQout_0(16-bit×4) using the selection circuit131(in the example shown inFIG. 25, select B1, B2, B3, and B4in this order), distributes the selection result to two, and outputs the data DPin_00and DPin_01of 16-bit. The remaining 16-bit data DQout_02and DQout_03are alternately selected (in the example ofFIG. 25, A1, A2, A3, and A4are selected in this order), and the selection result is divided into two and output as 16-bit data DPin_02and DPin_03. These 16-bit data DPin_00to DPin_03are input to the parallel arithmetic units MAC64_0to MAC64_3, respectively. The relationship between the data transfer unit13and the parallel arithmetic units MAC256_1to MAC256_3is the same as the relationship between the data transfer unit13and the parallel arithmetic unit MAC256_0, the description thereof is omitted.

At this time, two pieces of data to be output in one output process of the DRP11are input to each input terminal of the accelerator12. Therefore, the processing speed of the accelerator12is balanced by doubling the processing speed of the DRP11. In order to maximize the processing performance of the accelerator12, it is preferable to adjust the processing speed of the accelerator12to be slightly slower than twice the processing speed of the DRP11. When data is intermittently outputted from the DRP11, it is preferable to increase the processing rate of the DRP11in accordance with the degree of intermittency of the data because the processing performance of the accelerator12can be maximized.

FIG. 26is a diagram showing the parallel arithmetic unit MAC256_0, the data transfer unit13and the accelerator12when the input mode is the fifth input mode. In this case, the data transfer unit13uses the selection circuit131to alternately select the 16-bit data DQout_00and DQout_01constituting the 32-bit data DQout_0(16 bits×2) (in the example ofFIG. 26, A1, A2, A3, and A4are selected in this order), distributes the selection result to four, and outputs 16-bit data DPin_00to DPin_03. These 16-bit data DPin_00to DPin_03are input to the parallel arithmetic units MAC64_0to MAC64_3, respectively.

The relationship between the data transfer unit13and the parallel arithmetic units MAC256_1to MAC256_3is the same as the relationship between the data transfer unit13and the parallel arithmetic unit MAC256_0, the description thereof is omitted.

At this time, two pieces of data to be output in one output process of the DRP11are input to each input terminal of the accelerator12. Therefore, the processing speed of the accelerator12is balanced by doubling the processing speed of the DRP11. In order to maximize the processing performance of the accelerator12, it is preferable to adjust the processing speed of the accelerator12to be slightly slower than twice the processing speed of the DRP11. When data is intermittently outputted from the DRP11, it is preferable to increase the processing rate of the DRP11in accordance with the degree of intermittency of the data because the processing performance of the accelerator12can be maximized.

FIG. 27is a diagram showing the parallel arithmetic unit MAC256_0, the data transfer unit13and the accelerator12when the input mode is the sixth input mode. In this case, the data transfer unit13selects the 16-bit data DQout_00to DQout_02composing the data DQout_0of 48 bits (16 bits×3) in order using the selection circuit131(in the example shown inFIG. 27, A1, A2, A3, A4, A5, and A6are selected in order), and distributes the selection result to four and outputs the data DPin_00to DPin_03of 16 bits. These 16-bit data DPin_00to DPin_03are input to the parallel arithmetic units MAC64_0to MAC64_3, respectively.

The relationship between the data transfer unit13and the parallel arithmetic units MAC256_1to MAC256_3is the same as the relationship between the data transfer unit13and the parallel arithmetic unit MAC256_0, the description thereof is omitted.

At this time, three pieces of data to be output in one output process of the DRP11are input to each input terminal of the accelerator12. Therefore, if the processing speed of the accelerator12is three times the processing speed of the DRP11, it is well balanced. In order to maximize the processing performance of the accelerator12, it is preferable to adjust the processing speed of the accelerator12to be slightly slower than three times the processing speed of the DRP11. When data is intermittently outputted from the DRP11, it is preferable to increase the processing rate of the DRP11in accordance with the degree of intermittency of the data because the processing performance of the accelerator12can be maximized.

FIG. 28is a diagram showing the parallel arithmetic unit MAC256_0, the data transfer unit13and the accelerator12when the input mode is the seventh input mode. In this case, the data transfer unit13selects the 16-bit data DQout_00to DQout_03composing the data DQout_0of 64 bits (16 bits×4) sequentially using the selection circuit131(in the example shown inFIG. 28, A1, A2, A3, A4, A5, A6, A7, and A8are selected in this order), and distributes the selection result to four and outputs the data DPin_00to DPin_03of 16 bits. These 16-bit data DPin_00to DPin_03are input to the parallel arithmetic units MAC64_0to MAC64_3, respectively.

The relationship between the data transfer unit13and the parallel arithmetic units MAC256_1to MAC256_3is the same as the relationship between the data transfer unit13and the parallel arithmetic unit MAC256_0, the description thereof is omitted.

At this time, four pieces of data to be output in one DRP output process are input to each input terminal of the accelerator12. Therefore, if the processing speed of the accelerator12is four times the processing speed of the DRP11, it is well balanced. In order to maximize the processing performance of the accelerator12, it is preferable to adjust the processing speed of the accelerator12to be slightly slower than four times the processing speed of the DRP11. When data is intermittently outputted from the DRP11, it is preferable to increase the processing rate of the DRP11in accordance with the degree of intermittency of the data because the processing performance of the accelerator12can be maximized.

As described above, the semiconductor device1according to the present embodiment can arbitrarily change the degree of parallelism of the parallel arithmetic process on the data inputted from the DRP11to the accelerator12via the data transfer unit13. It should be noted that data processing is efficient when the data output rate from the DRP11is adjusted to match the processing throughput of the accelerator12. In particular, if the data output rate from the DRP11is set to be slightly higher than the processing throughput of the accelerator12, the processing performance of the accelerator12can be maximized.

Example of Data Transfer by the Data Transfer Unit14

Next, an example of data transfer from the accelerator12to the DRP11by the data transfer unit14will be described. Hereinafter, examples of data transfer by the data transfer unit14in accordance with the mode of operation in which data is output from the accelerator12to the DRP11via the data transfer unit14, hereinafter referred to as the output mode, will be described. The data DPout_0is composed of data DPout_00to DPout_03, which will be described later.

FIG. 29is a diagram showing the parallel arithmetic unit MAC256_0of the accelerator12and the data transfer unit14when the output mode is the first output mode. In this case, the data transfer unit14uses the selection circuit141to sequentially select one data from a maximum of 64 16-bit data DPout_00output in parallel from the parallel arithmetic unit MAC64_0, and sequentially output the selected data as 16-bit data DQin_00. In addition, 16-bit data DQin_01is output sequentially by selecting one data from DPout_01having a maximum of 64 16-bit data output in parallel from the parallel processor MAC64_1. In addition, 16-bit data DQin_02is output sequentially by selecting one data from DPout_02having a maximum of 64 16-bit data output in parallel from the parallel processor MAC64_2. Further, a maximum of 64 16-bit data DPout_03output in parallel from the parallel arithmetic unit MAC64_3are sequentially selected, and sequentially outputs the selected data as 16-bit data DQin_03. That is, the data transfer unit14sequentially outputs 64-bit width data DQin_0composed of 16-bit data DQin_00to DQin_03.

The relationship between the parallel arithmetic units MAC256_1to MAC256_3and the data transfer unit14is the same as the relationship between the parallel arithmetic unit MAC256_0and the data transfer unit14, and a description thereof will be omitted.

FIG. 30is a diagram showing the parallel arithmetic unit MAC256_0of the accelerator12and the data transfer unit14when the output mode is the second output mode. In this case, the data transfer unit14includes a selection circuit141composed of a first selection circuit141_1and a second selection circuit141_2.

First, the selection circuit141_1sequentially selects one data from a maximum of 64 16-bit data DPout_00output in parallel from the parallel arithmetic unit MAC64_0, and sequentially outputs the selected data as 16-bit data DQin_00. In addition, 16-bit data DQin_01is output sequentially by selecting one by one from DPout_01having a maximum of 64 16-bit data output in parallel from the parallel processor MAC64_1. In addition, 16-bit data DQin_02is output sequentially by selecting one by one from DPout_02having a maximum of 64 16-bit data output in parallel from the parallel processor MAC64_2. In addition, 16-bit data DQin_03is output sequentially by selecting one by one from DPout_03with a maximum of 64 16-bit data output in parallel from the parallel processor MAC64_3.

After that, the selection circuit141_2outputs 16-bit data DQin_00, and subsequently outputs 16-bit data DQin_01. In parallel, 16-bit data DQin_02is output, followed by 16-bit data DQin_03. That is, the data transfer unit14sequentially outputs data DQin_0having a 32-bit width composed of one of data DQin_00and DQin_01and one of data DQin_02and DQin_03output from the selection circuit141_2.

The data transfer unit14may alternately output 16-bit data DQin_00and 16-bit data DQin_01using the selection circuit141_2. The 16-bit data DQin_02and the 16-bit data DQin_03may be alternately output.

The relationship between the parallel arithmetic units MAC256_1to MAC256_3and the data transfer unit14is the same as the relationship between the parallel arithmetic unit MAC256_0and the data transfer unit14, and a description thereof will be omitted.

FIG. 31is a diagram showing the parallel arithmetic unit MAC256_0of the accelerator12and the data transfer unit14in the case where the output mode is the third output mode. In this case, the data transfer unit14includes a selection circuit141composed of a first selection circuit141_1and a second selection circuit141_2.

First, the selection circuit141_1sequentially selects one data from a maximum of 64 16-bit data DPout_00output in parallel from the parallel arithmetic unit MAC64_0, and sequentially outputs the selected data as 16-bit data DQin_00. In addition, 16-bit data DQin_01is output sequentially by selecting one by one from DPout_01having a maximum of 64 16-bit data output in parallel from the parallel arithmetic unit MAC64_1. In addition, 16-bit data DQin_02is output sequentially by selecting one by one from DPout_02having a maximum of 64 16-bit data output in parallel from the parallel arithmetic unit MAC64_2. In addition, 16-bit data DQin_03is output sequentially by selecting one by one from DPout_03with a maximum of 64 16-bit data output in parallel from the parallel arithmetic unit MAC64_3.

Thereafter, the selection circuit141_2sequentially selects one data from the 16-bit data DQin_00to DQin_03, and sequentially outputs the selected data as the 16-bit width data DQin_0.

The relationship between the parallel arithmetic units MAC256_1to MAC256_3and the data transfer unit14is the same as the relationship between the parallel arithmetic unit MAC256_0and the data transfer unit14, and a description thereof will be omitted.

FIG. 32is a diagram showing the parallel arithmetic unit MAC256_0of the accelerator12and the data transfer unit14in the case where the output mode is the fourth output mode. In this case, the data transfer unit14includes a selection circuit141composed of a first selection circuit141_1and a second selection circuit141_2.

First, the selection circuit141_1sequentially selects one data from a maximum of 64 16-bit data DPout_00output in parallel from the parallel arithmetic unit MAC64_0, and sequentially outputs the selected data as 16-bit data DQin_00(C1, C2, C3, C4, . . . in the example ofFIG. 32). In addition, 16-bit data DPout_01are sequentially selected one by one from a maximum of 64 16-bit data DPout_01output in parallel from the parallel arithmetic unit MAC64_1, and are sequentially output as 16-bit data DQin_01(D1, D2, D3, D4, . . . in the example ofFIG. 32). In addition, a maximum of 64 16-bit data DPout_02output in parallel from the parallel arithmetic unit MAC64_2are sequentially selected one by one, and the selected data are sequentially output as 16-bit data DQin_02(E1, E2, E3, E4, . . . ). In addition, a maximum of 64 16-bit data DPout_03output in parallel from the parallel arithmetic unit MAC64_3are sequentially selected one by one, and the selected data are sequentially output as 16-bit data DQin_03(in the example ofFIG. 32, F1, F2, F3, F4, . . . ).

Thereafter, the selection circuit141_2alternately outputs the 16-bit data DQin_00and the 16-bit data DQin_01as 32-bit data. In parallel with this, 16-bit data DQin_02and 16-bit data DQin_03are output in order (in this example, four elements in order) and two pieces of data are collectively output as 32-bit data. That is, the data transfer unit14sequentially outputs 64-bit width data DQin_0.

The relationship between the parallel arithmetic units MAC256_1to MAC256_3and the data transfer unit14is the same as the relationship between the parallel arithmetic unit MAC256_0and the data transfer unit14, and a description thereof will be omitted.

At this time, the data is inputted to the DRP11at a rate of ½ of the data outputted from the accelerator12. Therefore, when the processing speed of the accelerator12is about twice the processing speed of the DRP11, the data transfer speed of the data output from the accelerator12can be reduced to the DRP11processing speed after the accelerator12executes the parallel arithmetic operation processing efficiently without being rate-limited by the DRP11processing.

FIG. 33is a diagram showing the parallel arithmetic unit MAC256_0of the accelerator12and the data transfer unit14in the case where the output mode is the fifth output mode. In this case, the data transfer unit14includes a selection circuit141composed of a first selection circuit141_1and a second selection circuit141_2.

First, the selection circuit141_1sequentially selects one by one from a maximum of 64 16-bit data DPout_00output in parallel from the parallel arithmetic unit MAC64_0, and sequentially outputs the selected data as 16-bit data DQin_00(C1, C2, C3, C4, . . . in the example ofFIG. 33). In addition, a maximum of 64 16-bit data DPout_01output in parallel from the parallel arithmetic unit MAC64_1are sequentially selected one by one, and are sequentially output as 16-bit data DQin_01(D1, D2, D3, D4, . . . in the example ofFIG. 33). In addition, a maximum of 64 16-bit data DPout_02output in parallel from the parallel arithmetic unit MAC64_2are sequentially selected one by one, and are sequentially output as 16-bit data DQin_02(E1, E2, E3, E4, . . . in the example ofFIG. 33). In addition, a maximum of 64 16-bit data DPout_03output in parallel from the parallel arithmetic unit MAC64_3are sequentially selected one by one, and are sequentially output as 16-bit data DQin_03(in the example ofFIG. 33, F1, F2, F3, F4, . . . ).

Thereafter, the selection circuit141_2sequentially outputs 16-bit data DQin_00to DQin_03in order (in this example, in order of four elements) and collects two pieces of data as 32-bit width data DQin_0.

The relationship between the parallel arithmetic units MAC256_1to MAC256_3and the data transfer unit14is the same as the relationship between the parallel arithmetic unit MAC256_0and the data transfer unit14, and a description thereof will be omitted.

At this time, the data is inputted to the DRP11at a rate of ½ of the data outputted from the accelerator12. Therefore, in particular, when the processing speed of the accelerator12is about twice the processing speed of the DRP11, the data transfer speed of the data output from the accelerator12can be reduced to the DRP11processing speed after the accelerator12executes the parallel arithmetic processing efficiently without being rate-limited by the DRP11processing.

FIG. 34is a diagram showing the parallel arithmetic unit MAC256_0of the accelerator12and the data transfer unit14in the case where the output mode is the sixth output mode. In this case, the data transfer unit14includes a selection circuit141composed of a first selection circuit141_1and a second selection circuit141_2.

First, the selection circuit141_1sequentially selects one by one from a maximum of 64 16-bit data DPout_00output in parallel from the parallel arithmetic unit MAC64_0, and sequentially outputs the selected data as 16-bit data DQin_00(C1, C2, C3, C4, . . . in the example ofFIG. 34). In addition, a maximum of 64 16-bit data DPout_01output in parallel from the parallel arithmetic unit MAC64_1are sequentially selected one by one, and are sequentially output as 16-bit data DQin_01(D1, D2, D3, D4, . . . in the example ofFIG. 34). In addition, a maximum of 64 16-bit data DPout_02output in parallel from the parallel arithmetic unit MAC64_2are sequentially selected one by one, and are sequentially output as 16-bit data DQin_02(E1, E2, E3, E4, . . . in the example ofFIG. 34). In addition, a maximum of 64 16-bit data DPout_03output in parallel from the parallel arithmetic unit MAC64_3are sequentially selected one by one, and are sequentially output as 16-bit data DQin_03(F1, F2, F3, F4, . . . in the example ofFIG. 34).

Thereafter, the selection circuit141_2sequentially outputs 16-bit data DQin_00to DQin_03in order (in this example, in order of four elements) and collects three pieces of data as 48-bit width data DQin_0.

The relationship between the parallel arithmetic units MAC256_1to MAC256_3and the data transfer unit14is the same as the relationship between the parallel arithmetic unit MAC256_0and the data transfer unit14, and a description thereof will be omitted.

At this time, data is inputted to the DRP11at a rate of one third of the data outputted from the accelerator12. Therefore, when the processing speed of the accelerator12is about three times the processing speed of the DRP11, the data transfer speed of the data outputted from the accelerator12can be reduced to the DRP11processing speed after the accelerator12executes the parallel arithmetic processing efficiently without being rate-limited by the DRP11processing.

FIG. 35is a diagram showing the parallel arithmetic unit MAC256_0of the accelerator12and the data transfer unit14in the case where the output mode is the seventh output mode. In this case, the data transfer unit14includes a selection circuit141composed of a first selection circuit141_1and a second selection circuit141_2.

First, the selection circuit141_1sequentially selects one by one from a maximum of 64 16-bit data DPout_00output in parallel from the parallel arithmetic unit MAC64_0, and sequentially outputs the selected data as 16-bit data DQin_00(C1, C2, C3, C4, . . . in the example ofFIG. 35). In addition, a maximum of 64 16-bit data DPout_01output in parallel from the parallel arithmetic unit MAC64_1are sequentially selected one by one, and are sequentially output as 16-bit data DQin_01(D1, D2, D3, D4, . . . in the example ofFIG. 35). In addition, 16-bit data DPout_02are sequentially selected one by one from a maximum of 64 16-bit data DPout_02output in parallel from the parallel arithmetic unit MAC64_2, and are sequentially output as 16-bit data DQin_02(E1, E2, E3, E4, . . . in the example ofFIG. 35). In addition, 16-bit data DPout_03are sequentially selected one by one from a maximum of 64 16-bit data DPout_03output in parallel from the parallel arithmetic unit MAC64_3, and are sequentially output as 16-bit data DQin_03(in the example ofFIG. 35, F1, F2, F3, F4, . . . ).

Thereafter, the selection circuit141_2sequentially outputs 16-bit data DQin_00to DQin_03in order (in this example, in order of four elements) and collects four pieces of data as 64-bit width data DQin_0.

The relationship between the parallel arithmetic units MAC256_1to MAC256_3and the data transfer unit14is the same as the relationship between the parallel arithmetic unit MAC256_0and the data transfer unit14, and a description thereof will be omitted.

At this time, data is inputted to the DRP11at a rate of ¼ of the data outputted from the accelerator12. Therefore, when the processing speed of the accelerator12is about four times the processing speed of the DRP11, the data transfer speed of the data outputted from the accelerator12can be reduced to the DRP11processing speed after the accelerator12executes the parallel arithmetic processing efficiently without being rate-limited by the DRP11processing.

As described above, in the semiconductor device1according to the present embodiment, the data outputted from the accelerator12to the DRP11via the data transferring unit14can be changed to data of any bit-width. In order to maximize the performance of the accelerator12, it is preferable that the data rate received by the DRP11is slightly higher than the data rate outputted from the accelerator12.

FIG. 36is a diagram showing the flow of the arithmetic operation processing of the parallel arithmetic part121when the arithmetic operation is performed on the input data with the maximum degree of parallelism. As shown inFIG. 36, the data DQout_0outputted from the DRP11is distributed and supplied as data DPin_0to DPin_3by the data transferring unit13to the parallel arithmetic units MAC64_0to MAC64_3provided in the parallel arithmetic units MAC256_0to MAC256_3, respectively. At this time, the parallel arithmetic part121can perform arithmetic operation on the data DQout_0(data DPin_0to DPin_3) in parallel by using up to 1024 arithmetic units. Note that the data transferring unit14is configured to selectively output the arithmetic operation results output in parallel from each of the 1024 arithmetic units, so that these arithmetic operation results can be converted into data of a desired bit-width and output to the DRP11.

FIG. 37is a diagram showing the flow of the arithmetic operation of the parallel arithmetic part121in the case where the arithmetic operation is performed on the input data with the degree of parallelism as the minimum unit. As shown inFIG. 37, the data DQout_0outputted from the DRP11is supplied to the parallel arithmetic unit MAC64_0provided in the parallel arithmetic unit MAC256_0as the data DPin_0by the data transferring unit13. At this time, the parallel operation part121can execute the arithmetic operation on the data DQout_0(data DPin_0) in parallel by using one to 64 arithmetic units out of the 64 arithmetic units provided in the parallel arithmetic unit MAC64_0.

FIG. 38is a diagram showing the flow of the arithmetic operation processing of the parallel arithmetic part121when the arithmetic operation is performed on the input data with the degree of parallelism set at a medium level. In the embodiment ofFIG. 38, the data DQout_0outputted from the DRP11is distributed and supplied as data DPin_0and DPin_1by the data transferring unit13to the parallel arithmetic units MAC64_0to MAC64_3provided in the parallel arithmetic unit MAC256_0and the parallel arithmetic units MAC64_0to MAC64_2provided in the parallel arithmetic unit MAC256_0, respectively. Here, the parallel arithmetic part121can perform arithmetic operation processing on the data DQout_0(data DPin_0and DPin_1) in parallel using, for example, 400 arithmetic units.

FIG. 39is a diagram showing the flow of the arithmetic operation of the parallel arithmetic part121when the parallel arithmetic operation is performed on each of the two input data. In the embodiment ofFIG. 39, the data DQout_0outputted from the DRP11is distributed and supplied as data DPin_0and DPin_1by the data transferring unit13to the parallel arithmetic units MAC64_0to MAC64_3provided in the parallel arithmetic unit MAC256_0and the parallel arithmetic units MAC64_0to MAC64_2provided in the parallel arithmetic unit MAC256_1, respectively. Further, the data DQout_2outputted from the DRP11is distributed and supplied as data DPin_2by the data transferring unit13to the parallel arithmetic units MAC64_0and MAC64_1provided in the parallel arithmetic unit MAC256_2. At this time, the parallel arithmetic part121can execute the arithmetic operation on the data DQout_0(data DPin_0and DPin_1) in parallel using, for example, 400 arithmetic units, and execute operation processing on the data DQout_2(data DPin_2) in parallel using, for example, 120 different arithmetic units.

In the case of executing arithmetic operation using a plurality of arithmetic units different from each other for two or more input data, for example, a plurality of arithmetic units used for arithmetic operation processing for one input data and a plurality of arithmetic units used for arithmetic operation processing for the other input data may be supplied with individual predetermined data read out from the local memory122, or may be supplied with common predetermined data.

Second Embodiment

FIG. 40is a block diagram showing an exemplary configuration of a semiconductor system SYS1aon which the semiconductor system1aaccording to the second embodiment is mounted. The semiconductor device1ashown inFIG. 40has a DRP11ainstead of a DRP11as compared with the semiconductor device1shown inFIG. 1.

The DRP11ahas, for example, two state management units (STCs; State Transition Controller)111and112, performs arithmetic operation on data read out from the external memory3using one state management unit111, outputs the arithmetic operation result to the accelerator12, and performs arithmetic operation on data output from the accelerator12using the other state management unit112, and writes the arithmetic operation result to the external memory3. That is, the DRP11aoperates the processing of the data to be transmitted to the accelerator12and the processing of the data received from the accelerator12independently of each other. As a result, in the DRP11a, it is possible to make the operation instruction (application) given when performing the dynamic reconfiguration simpler than the dynamic reconfiguration instruction (application) when performing the dynamic reconfiguration operation (DRP11). It also allows the DRP11ato reconfigure circuits more easily than with DRP11.

In addition, the DRP11ais provided with two state management units for independently operating the processing of the data to be transmitted to the accelerator12and the processing of the data received from the accelerator12, whereby, for example, the degree of flexibility of arrangement of an external input terminal to which the data read from the external memory3is input, an external output terminal to which the data directed to the accelerator12is output, an external input terminal to which the data from the accelerator12is input, and an external output terminal to which the write data directed to the external memory3is output can be increased.

As described above, the semiconductor device according to first and second embodiments includes an accelerator having a parallel arithmetic part that performs arithmetic operation in parallel, a data processing unit such as DRP that sequentially transfers data, and a data transfer unit that sequentially selects and outputs a plurality of arithmetic operation processing results by the accelerator to the data processing unit. As a result, the semiconductor device according to the first and second embodiments and the semiconductor system including the same can perform a large amount of regular data processing by using the accelerator, and perform other data processing by using the data processing unit, so that efficient arithmetic operation can be performed even in a large-scale arithmetic processing such as, for example, a deep learning processing.

Although the invention made by the inventor has been specifically described based on the embodiment, the present invention is not limited to the embodiment already described, and it is needless to say that various modifications can be made without departing from the gist thereof.

In the first and second embodiments described above, the case where the individual predetermined data read out from the local memory122is supplied to the plurality of arithmetic units constituting the parallel arithmetic part121is described, but the present invention is not limited thereto. The common predetermined data read from the local memory122may be supplied to all or a group of the plurality of arithmetic units constituting the parallel arithmetic part121. In this case, the circuit scale and power consumption of the local memory122can be reduced.

Some or all of the above-described embodiments may be described as the following appendix, but the present invention is not limited to the following.

A semiconductor device, comprising: a data processing unit that performs data processing on sequentially input first input data and sequentially outputs the result of data processing as first output data; a parallel arithmetic unit that performs arithmetic processing in parallel between the first output data sequentially output from the data processing unit and each of a plurality of predetermined data; a holding circuit that holds the results of the arithmetic processing; and a first data transfer unit that sequentially selects a plurality of arithmetic processing results held by the accelerator and sequentially outputs the results of the arithmetic processing as the first input data.

The semiconductor device according to Appendix 1, wherein the data processing unit is a processor that can be dynamically reconfigured based on an operation command that is sequentially given.

A semiconductor system comprising: a semiconductor device as described in Appendix 3; an external memory; and a control unit that controls the operation of the semiconductor device based on a control instruction read from the external memory.