Patent Description:
In recent years, due to the advancement of deep learning, the accuracy of image recognition is improving. A convolutional neural network (CNN) is known as a method that is used in deep learning. In CNN, a plurality of layers are hierarchically connected, and a plurality of feature images are included in each layer. <FIG> shows an example of a network in which there are four layers (layers <NUM> to <NUM>) and four feature images in each layer. In <FIG>, a feature image (i, j) represents a j-th feature image in a layer i. A filter processing result is calculated by using learned filter coefficients and feature image pixels (feature data). The filter processing is a multiply-accumulate (MAC) operation and includes a plurality of multiplications and a cumulative sum. Each arrow shown in <FIG> indicates a MAC operation.

Feature images of a current layer are calculated by using feature images of a preceding layer and filter coefficients corresponding to the preceding layer. To calculate one feature image of the current layer, the information of a plurality of feature images of the preceding layer is required. The equation of the convolution operation for calculating each feature image of the current layer is as follows. <MAT>
where Oi,j(n) is a variable representing a MAC operation result corresponding to a position (i, j) in an n-th feature image in the current layer. In equation (<NUM>), there are M feature images in the preceding layer, and Ii,j(m) represents feature data at the position (i, j) in an m-th feature image. There are X × Y filter coefficients C<NUM>,<NUM>(m, n) to CX,Y(m, n), and the filter coefficient differs for each feature image. The MAC operations for calculating the n-th feature image in the current layer are performed M × X × Y times. After the convolution operation has been executed, the feature images of the current layer are calculated by executing processing such as activation and pooling by using the MAC operation result Oi,j(n).

Since CNN requires a large number of MAC operations, an efficient data parallel processing apparatus is needed when CNN is to be applied to an embedded system such as a mobile terminal, an onboard device, or the like. Since reducing the bit width of the processing date will reduce the cost of an arithmetic operation unit which calculates the convolution operation result, the degree of parallelism (DOP) of the arithmetic operation unit can be increased. A hardware arrangement that processes a network with a different data bit width for each layer is proposed in <NPL>.

In the method described in <NPL>, different kinds of arithmetic operators are used to process a CNN with a different bit width for each layer. In a case in which the bit width of the feature data of an input layer is <NUM> bits and the bit width of the feature data of an intermediate layer is <NUM> bits, a convolution operation unit dedicated to <NUM>-bit data and a convolution operation unit dedicated to <NUM>-bit data will be required.

Although the <NUM>-bit data layer and the <NUM>-bit data layer can be processed by pipelining to process data in parallel, the hardware use efficiency is reduced when the calculation amount of the convolution processing differs for each layer. In addition, when feature data having a bit width (such as <NUM> bits) which is between <NUM> bits and <NUM> bits is to be processed, the efficiency will degrade because the convolution operation unit dedicated to <NUM>-bit data will have to be used since there is no convolution operation unit for this bit width.

In the method described in <NPL>, there is proposed an RNN (Recurrent Neural Network) dedicated hardware having an SIMD (single instruction multiple data) configuration capable of processing feature data sets that have plurality of bit widths. Although <NUM>-bit data, <NUM>-bit data, and <NUM>-bit data can be processed by using the same hardware, this will increase the processing time when the total sum of data output in parallel is to be calculated because an SIMD command will need to be executed once again after the data has been temporarily held in a memory. In the method described in <NPL>, an information processing apparatus for setting shift amounts corresponding to a plurality of shift operation means based on a bit width of data, is disclosed: said apparatus however resorts to hierarchic processing, without respect to the number of MACS and SHIFTS as in the present invention.

The present invention provides a technique for implementing efficient processing even if there are data sets having a plurality of bit widths in a multi-layer network. The present invention in an aspect provides an information processing apparatus as specified in claims <NUM> to <NUM>. The present invention in another aspect provides a non-transitory computer-readable storage medium as specified in claim <NUM>.

The invention is defined in the appended independent claims <NUM>, <NUM> and <NUM>. Preferred embodiments of the invention are set out in the appended dependent claims. Note that each embodiment to be described below is an example of detailed implementation of the present invention and is a detailed embodiment of the arrangement described in the appended claims.

An example of the hardware arrangement of an information processing apparatus according to the embodiment will be described first with reference to the block diagram of <FIG>. A computer apparatus such as a PC (personal computer), a tablet terminal device, a smartphone, or the like is applicable as the information processing apparatus. In addition, this information processing apparatus may be an embedded device which is to be embedded in such devices.

An input unit <NUM> is formed by a user interface such as a keyboard, a mouse, a touch panel, or the like, and can input various kinds of instructions to a CPU <NUM> when operated by a user.

A data storage unit <NUM> is a large-capacity information storage device such as a hard disk drive device or the like. The data storage unit <NUM> stores various kinds of information to be used in the information processing apparatus such as an OS (Operating System), various kinds of computer programs executed by the CPU <NUM>, data to be used when the CPU <NUM> executes various kinds of processing, and the like. The data stored in the data storage unit <NUM> include images to be processed by an image processing unit <NUM>. Note that information to be described below as "known information" is also stored in the data storage unit <NUM>. The computer programs and data stored in the data storage unit <NUM> are loaded to a RAM <NUM> or the like by the CPU <NUM>, a data processing unit <NUM>, and the image processing unit <NUM> and become processing targets of the CPU <NUM>, a data processing unit <NUM>, and the image processing unit <NUM>.

Note that the data storage unit <NUM> may be a storage medium (for example, a flexible disk, a CD-ROM, a CD-R, a DVD, a memory card, a CF card, a smart media, an SD card, a memory stick, an xD picture card, a USB memory, or the like). In this case, the information processing apparatus needs to include a device to read out and write the information from/to such a storage medium.

A communication unit <NUM> functions as a communication interface for performing data communication with an external device. It may be set so that the communication unit <NUM> will obtain information necessary for executing processing in the information processing apparatus from an external device. The communication unit <NUM> may transmit the result of the processing performed by the information processing apparatus to an external device.

A display unit <NUM> is formed by a liquid-crystal screen or a touch panel screen, and can display images, characters, and the like to display a processing result obtained by the CPU <NUM>, the data processing unit <NUM>, or the image processing unit <NUM>. Note that the display unit <NUM> may be a projection device such as a projector. The input unit <NUM> and the display unit <NUM> may be integrated and form a device, such as a touch screen device, which has both an instruction input accepting function and a display function.

The data processing unit <NUM> executes CNN calculation by executing processing in accordance with the flowchart of <FIG> by using an image written into the RAM <NUM> by the image processing unit <NUM>, and outputs the obtained calculation result to the data storage unit <NUM>, the RAM <NUM>, or the like. Note that an image which is to be a processing target of the data processing unit <NUM> is not limited to an image written into the RAM <NUM> by the image processing unit <NUM> and may be, for example, an image input by another apparatus. The data processing unit <NUM> will be described later with reference to <FIG>.

The CPU <NUM> executes various kinds of processing by using computer programs and data stored in a ROM <NUM> or the RAM <NUM>. This allows the CPU <NUM> to control the overall operation of the information processing apparatus.

The ROM <NUM> stores information that need not be rewritten such as setting data, an activation program, and the like of the information processing apparatus. The RAM <NUM> includes an area for storing computer programs and data loaded from the data storage unit <NUM> and the ROM <NUM>, information received by the communication unit <NUM> from an external device, and the like. The RAM <NUM> includes a work area used by the CPU <NUM>, the data processing unit <NUM>, and the image processing unit <NUM> to execute various kinds of processing. The RAM <NUM> can appropriately provide various kinds of areas in this manner.

The image processing unit <NUM> reads out an image stored in the data storage unit <NUM> and writes the image in the RAM <NUM> after executing pixel value range adjustment on each pixel of the image under the instruction from the CPU <NUM>.

The input unit <NUM>, the data storage unit <NUM>, the communication unit <NUM>, the display unit <NUM>, the data processing unit <NUM>, the CPU <NUM>, the ROM <NUM>, the RAM <NUM>, and the image processing unit <NUM> described above are all connected to a bus <NUM>.

Note that the hardware arrangement of the information processing apparatus is not limited the arrangement shown in <FIG>. For example, the arrangement of <FIG> may be implemented by a plurality of apparatuses. In addition, the information processing apparatus need not always include devices such as the input unit <NUM>, the data storage unit <NUM>, and the display unit <NUM>, and it may be set so that these devices will be connected to the information processing apparatus via a communication path.

In addition, some or all of the pieces of information described as being stored in the RAM <NUM> may be stored in the data storage unit <NUM>, and some or all of the pieces of information described as being stored in the data storage unit <NUM> may be stored in the RAM <NUM>. Alternatively, it may be set so that a part of the RAM <NUM> will be used as the data storage unit <NUM> or it may be virtually arranged so that storage device of a communication partner device of the communication unit <NUM> will be used as the data storage unit via the communication unit <NUM>.

Also, although only one CPU <NUM> is shown in <FIG>, the number of CPU <NUM> included in the information processing apparatus is not limited to one, and a plurality of CPUs may be included in the information processing apparatus. In addition, the data processing unit <NUM> and the image processing unit <NUM> may be implemented as hardware or may be implemented as computer programs. In the case of the latter, these computer programs will be stored in the data storage unit <NUM>, and the functions of the data processing unit <NUM> and the image processing unit <NUM> will be executed by the CPU <NUM> executing the corresponding computer programs.

Note that based on the processing result of the data processing unit <NUM>, the CPU <NUM> will perform image processing and/or image recognition on each frame of a moving image obtained from the communication unit <NUM> or the data storage unit <NUM>. The result of the image processing or image recognition by the CPU <NUM> is stored in the RAM <NUM> or the data storage unit <NUM> or output to an external device via the communication unit <NUM>. Also, the result of the image processing or image recognition by the CPU <NUM> may be displayed as an image or characters on the display unit <NUM> or output as audio if the information processing apparatus has an audio output function.

This embodiment uses CNN as the processing target network. <FIG> shows an example of the arrangement of the processing target network. The details of the processing target network of <FIG> are as described above. Note that pieces of information such as the calculation amount of each MAC operation, the size of each feature image, the number of feature images, the number of bits of each feature image, and the like of the processing target network are stored in the data storage unit <NUM> or the like.

The number of layers in the processing target layer network shown in <FIG> is four (layers <NUM> to <NUM>), and there are four feature images in each layer. As described above, a feature image (i, j) represents a j-th feature image of a layer i. Also, the bit width of each feature image of a layer changes depending on the layer. The bit width of each feature image of the layer <NUM> is <NUM> bits, the bit width of each feature image of the layer <NUM> is <NUM> bits, the bit width of each feature image of the layer <NUM> is <NUM> bits, and the bit width of each feature image of the layer <NUM> is <NUM> bits. Since the first layer (layer <NUM>) and the final layer (layer <NUM>) hold input/output image information, the bit widths (<NUM> bits) of the first layer and the final layer tend to be larger than the bit widths (<NUM> bits and <NUM> bits) of intermediate layers (layer <NUM> and layer <NUM>). Each feature image is formed by a plurality of pixels (feature data).

The calculation (generation) of feature images of each of the layers <NUM> to <NUM> executed by the data processing unit <NUM> will be described hereinafter. MAC operations according to equation (<NUM>) described above are executed by using filter coefficients and <NUM>-bit feature images (<NUM>, <NUM>), (<NUM>, <NUM>), (<NUM>, <NUM>), and (<NUM>, <NUM>) of the layer <NUM>. Subsequently, <NUM>-bit feature images (<NUM>, <NUM>), (<NUM>, <NUM>), (<NUM>, <NUM>), and (<NUM>, <NUM>) of the layer <NUM> are generated as a result of the MAC operations.

Next, the <NUM>-bit feature images (<NUM>, <NUM>), (<NUM>, <NUM>), (<NUM>, <NUM>), and (<NUM>, <NUM>) of the layer <NUM> and the filter coefficients are used to perform MAC operations in accordance with equation (<NUM>) described above. Subsequently, <NUM>-bit feature images (<NUM>, <NUM>), (<NUM>, <NUM>), (<NUM>, <NUM>), and (<NUM>, <NUM>) of the layer <NUM> are generated as a result of the MAC operations.

An example of the arrangement of the data processing unit <NUM> is shown in <FIG>. A data memory <NUM> holds the feature data of each feature image of each layer, and a coefficient memory <NUM> holds the filter coefficients. A multiplier-accumulator <NUM> calculates each MAC operation result by executing a MAC operation by using the filter coefficients held in the coefficient memory <NUM> and the feature data held in the data memory <NUM>. A shift operator <NUM> shifts each MAC operation result obtained by the multiplier-accumulator <NUM>, and an adder <NUM> obtains "a total sum of the shifted MAC operation results" by adding the plurality of shifted MAC operation results. Based on "the total sum of the shifted MAC operation results" obtained by the adder <NUM>, a processing unit <NUM> calculates an activation/pooling processing result and stores the calculated activation/pooling processing result in the data memory <NUM>. A control unit <NUM> controls the overall operation of the data processing unit <NUM>.

The data processing by the data processing unit <NUM> will be described in accordance with the flowchart of <FIG>. In step S101, the control unit <NUM> reads out feature data of a plurality of input feature images and the filter coefficients from the RAM <NUM>, stores the read-out feature data in the data memory <NUM>, and stores the read-out feature coefficients in the coefficient memory <NUM>.

In step S102, the control unit <NUM> starts the loop of the layers and sets one of the unprocessed layers as the processing target layer. Since the layers <NUM> to <NUM> will be set sequentially as a processing target in this example, the layer <NUM> will be the processing target layer first.

In step S103, the control unit <NUM> sets the shift parameters, defining shift amounts, of the shift operator <NUM> in accordance with the layer information. In step S104, the control unit <NUM> starts the loop of the output feature images and sequentially calculates the output feature data. In step S105, the control unit <NUM> initializes the MAC operation result stored in the adder <NUM> to set the MAC operation result to zero. The adder <NUM> has a total sum calculation function.

In step S106, the control unit <NUM> starts the loop of input feature images and sequentially processes the input feature data. In step S107, under the control of the control unit <NUM>, the multiplier-accumulator <NUM> and the shift operator <NUM> perform the MAC operation and the shift operation, respectively, described above. The details (step S115 to step S117) of the process of step S107 will be described later.

In step S108, the adder <NUM> adds the plurality of MAC operation results to obtain "the total sum of the shifted MAC operation results". In step S109, the control unit <NUM> determines the completion of the input feature image loop. If the processing of all of the input feature images has been completed, the process advances to step S110. Otherwise, the process returns to step S107, and the processing of the next unprocessed input feature image is started. In step S110, the processing unit <NUM> calculates an activation processing result based on "the total sum of the shifted MAC operation results" obtained by the adder <NUM> in accordance with <MAT>.

In this case, f(▪) is an activation function, and x is input data. Although ReLU (Rectified Linear Unit) is used to implement the activation function in this example, the present invention is not limited to ReLU, and the activation function may be implemented by another non-linear function or a quantization function. Note that the bit width of the activation processing result will be adjusted as needed.

In step S111, the processing unit <NUM> calculates the activation/pooling processing result by executing pooling processing based on the activation processing result in accordance with the layer information. In step S112, the processing unit <NUM> stores the activation/pooling processing result calculated in step S111 as a feature image of the next layer in the data memory <NUM>.

In step S113, the control unit <NUM> determines the completion of the loop of the output feature images. If the processing of all of the output feature images has been completed, the process advances to step S114. Otherwise, the process returns to step S105 to start the processing of an unprocessed output feature image.

In step S114, the control unit <NUM> determines the completion of the loop of the layers. If the processing of all of the layers has been completed, the processing according to the flowchart of <FIG> ends. Otherwise, the process returns to step S103, and the processing of an unprocessed layer is started.

The MAC operation and the shift operation (steps S115 to S117) of step S107 will be described. In step S115, in addition to reading out the feature data from the data memory <NUM> and transferring the read-out feature data to the multiplier-accumulator <NUM>, the control unit <NUM> reads out the filter coefficients from the coefficient memory <NUM> and transfers the read-out filter coefficients to the multiplier-accumulator <NUM>. The number of the filter coefficients and the transfer count will vary depending on the bit widths of the feature data.

In step S116, the multiplier-accumulator <NUM> calculates the MAC operation results based on the feature data and the filter coefficients. In step S117, the shift operator <NUM> shifts the MAC operation results obtained in step S116 based on the shift amounts indicated by the shift parameters set in step S103.

This embodiment can process data of different bit widths. <FIG> shows the operations of the multiplier-accumulator <NUM>, the shift operator <NUM>, and adder <NUM> in a case in which <NUM>-bit feature data is to be processed. <FIG> shows the operations of the multiplier-accumulator <NUM>, the shift operator <NUM>, and adder <NUM> in a case in which <NUM>-bit feature data is to be processed.

In a case in which the feature data is <NUM> bits, the multiplier-accumulator <NUM> divides the <NUM>-bit feature data <NUM> (value: <NUM>) into sets of data of <NUM> bits (<NUM>-bit data) as shown in <FIG>. The multiplier-accumulator <NUM> uses the four sets of <NUM>-bit data (values: <NUM>, <NUM>, <NUM>, <NUM>) obtained from the division and a shared filter coefficient to calculate four MAC operation results, and the shift operator <NUM> shifts the four MAC operation results based on four shift parameters. Subsequently, the adder <NUM> adds the four shifted MAC operation results to calculate one set of feature data (the MAC operation results of <NUM>-bit input feature data). The data processing unit <NUM> can process one set of <NUM>-bit input feature data in this manner.

In a case in which the feature data is <NUM> bits, the multiplier-accumulator <NUM> uses the four sets of <NUM>-bit data <NUM> (values: <NUM>, <NUM>, <NUM>, <NUM>) and four filter coefficients to calculate four MAC operation results as shown in <FIG>. The shift operator <NUM> shifts the four MAC operation results based on one shift parameter. Since the shift parameter is zero, the states of the MAC operation results are the same before and after the shift operation. Subsequently, the adder <NUM> calculates one set of feature data (the total sum of the MAC operation results of the four <NUM>-bit input feature data sets) by adding the four MAC operation results. The data processing unit <NUM> can process the four sets of <NUM>-bit input feature data in parallel in this manner.

Let M be the number of input feature images and <NUM> × <NUM> be the filter size. Since the filter size equal to one pixel, and the values of variables x and y are constants, Oi,j(n) will be calculated by using Ii,j(n). The calculation (equation <NUM>) of the MAC operation can be simplified as <MAT>.

Although the multiplier-accumulator <NUM> will calculate each convolution result of the filter coefficients and the input feature data in a case in which the filter size is more than <NUM> × <NUM>, the multiplier-accumulator <NUM> will calculate the product of I(m) and C(m, n) in a case in which the filter size is equal to <NUM> × <NUM>.

Assume that there are two kinds of feature processing target data, feature processing data whose bit width is α bits and feature processing data whose bit width is β bits. The multiplier-accumulator <NUM> shown in <FIG> includes P α-bit data MAC operation units for calculating the MAC operation results, and the shift operator <NUM> includes P α-bit data shift operation units for calculating the shift operation results. α, β, and P satisfy the following condition.

In a case in which the bit width of input feature data I'(β) is β bits, the output of the adder <NUM> is represented as equation (<NUM>) below based on the premise of equations (<NUM>), (<NUM>), and (<NUM>). A MAC operation result O(n) of the n-th output image is given by <MAT>
where I(α), P(m) is input data of the α-bit data MAC operation unit, Cp(m, n) is a filter coefficient, and S(p) is a shift parameter. A variable m is the number (processing number of the multiplier-accumulator <NUM>) of an α-bit input feature image group (<NUM> group = P images), a variable p is the MAC operation unit number and the shift operation unit number, and a variable n is the output feature image number. The shift operation is expressed by processing by the power of <NUM>.

The filter coefficient Cp(m, n) is, as shown in equation (<NUM>), a filter coefficient C'(m, n) corresponding to an m-th β-bit feature image. Since a shared filter coefficient is used for the α-bit input feature image group, the variable p can be omitted. The number of filter coefficients to be supplied in parallel to the P MAC operation units is <NUM>, and the transfer count is <NUM>.

In this case, the input data I'(β) is divided into P sets of α-bit data I(α), P(m). The value of the shift parameter S(p) is calculated based on the MAC operation unit number p and the bit width α of the divided data by <MAT>.

The β-bit input feature data I'(β) is represented by the divided P sets of α-bit data I(α), P(m) as <MAT>.

In this case, a substitution of equations (<NUM>), (<NUM>), and (<NUM>) into equation (<NUM>) yields the equation of the output data O(n) as <MAT>.

On the other hand, in a case in which the bit width of input feature data I'(α) is α bits, the output of the adder <NUM> is represented as equation (<NUM>) below based on the premise of equations (<NUM>), (<NUM>), and (<NUM>). The MAC operation result O(n) of the n-th output image is given by <MAT>
where I(α), P(m) is the input data of the α-bit data MAC operation unit, Cp(m, n) is the filter coefficient, and S(p) is the shift parameter. The variable m is the number (the processing number of the multiplier-accumulator <NUM>) of the α-bit input feature image group (<NUM> group = P images), the variable p is the MAC operation unit number and the shift operation unit number, and the variable n is the output feature image number. The shift operation is expressed by processing by the power of <NUM>.

The filter coefficient Cp(m, n) is a filter coefficient C'((m-<NUM>) × P + p, n) corresponding to an {(m-<NUM>) × P + p}-th α-bit feature image. Since the filter coefficient differs depending on the MAC operation unit number p, the number of filter coefficients to be supplied in parallel to the P MAC operation units is P, and the transfer count is P.

The input feature data I'(α) becomes the input data I(α), P(m) of the α-bit data MAC operation unit, and the value of the shift parameter S(p) is constantly <NUM> as shown by <MAT>.

Although the P sets of the α-bit input feature data I'(α) are directly input to the MAC operation units, the P sets of input data are feature data of different feature images. The feature image number is expressed as shown in equation (<NUM>) below by the MAC operation unit number p, the number P of shift operation units, and the processing number m of the multiplier-accumulator <NUM>.

A substitution of equations (<NUM>), (<NUM>), and (<NUM>) into equation (<NUM>) yields the equation of the output data O(n) as <MAT>.

By changing the value of the shift parameter S(p) and the number of filter coefficients, the feature data I'(α) whose bit width is α bits and the feature data I'(β) whose bit width is β bits can be processed by using the same operators (the multiplier-accumulator <NUM>, the shift operator <NUM>, and the adder <NUM>).

<FIG> and <FIG> and <FIG> show an example of an arrangement when P = <NUM>, β = <NUM>, and α = <NUM>. The bit width of input data for the multiplier-accumulator <NUM> is <NUM> bits, the bit width of the input data for the shift operator <NUM> is <NUM> bits, and the bit width of the input data for the adder <NUM> is <NUM> bits.

<FIG> show an example of the processing time of a case in which the processing target network shown in <FIG> is processed by using the hardware arrangement shown in <FIG>. <FIG> and <FIG> show an example of when the layer <NUM> (<NUM>-bit data, the number M of input feature images = <NUM>) is processed. Feature data I'(<NUM>)(<NUM>) of the feature image (<NUM>, <NUM>) is <NUM> bits, and four sets of data I(<NUM>), <NUM>(<NUM>) to I(<NUM>), <NUM>(<NUM>) obtained by dividing the feature data by four based on equation (<NUM>) are input to the multiplier-accumulator <NUM>. Shift operation results are calculated by using the input feature data sets, the shift parameters, and the filter coefficient C(m, n), the calculated shift operation results are input to the adder <NUM>, and an initial value of zero is added to the obtained result. The calculation result is set as the shift operation result and held by the adder <NUM>. The duration of this process is <NUM>.

Feature data I'(<NUM>)(<NUM>) of the feature image (<NUM>, <NUM>) is <NUM> bits, and four sets of data I(<NUM>), <NUM>(<NUM>) to I(<NUM>), <NUM>(<NUM>) obtained by dividing the feature data by four based on equation (<NUM>) are input to the multiplier-accumulator <NUM>. Shift operation results are calculated by using the input feature data sets, the shift parameters, and the filter coefficient C(m, n), and the calculated shift operation results are input to the adder <NUM> and added to the previous result. The duration of this process is <NUM>.

The feature images (<NUM>, <NUM>) and (<NUM>, <NUM>) are sequentially processed in a manner similar to the feature image (<NUM>, <NUM>), the shift operation results are accumulated, and the addition result is calculated. The duration of the process is <NUM>. Finally, the feature data of the feature image (<NUM>, <NUM>) is output via the processing unit <NUM>. The processing time of the four feature images is <NUM>.

<FIG> and <FIG> show an example of when the layer <NUM> (<NUM>-bit data, the number M of input feature images = <NUM>) is processed. Each set of feature data I'(<NUM>)(<NUM>) to <NUM>'(<NUM>)(<NUM>) of the feature images (<NUM>, <NUM>) to (<NUM>,<NUM>) is <NUM> bits, and the four sets of data I(<NUM>), <NUM>(<NUM>) to I(<NUM>), <NUM>(<NUM>) are input in parallel to the multiplier-accumulator <NUM> based on equation (<NUM>). The shift operation results are calculated by using the input feature data, the shift parameter, and the filter coefficient Cp (m, n), the obtained results are input to the adder <NUM> and added with the initial value of zero, and the calculation result becomes the shift operation result. Finally, the feature data of the feature image (<NUM>, <NUM>) is output via the processing unit <NUM>. The processing time of the four feature images is <NUM>.

As shown in <FIG> and <FIG> and <FIG>, the processing time per output data is <NUM> when the input feature data is <NUM> bits, and the processing time per output data is <NUM> when the input feature data is <NUM> bits. Data of different bit widths can be processed efficiently by the common data processing unit <NUM>.

Differences from the first embodiment will be described below. Matters not particularly mentioned below are similar to those of the first embodiment.

The first embodiment described an example in which the shift operation is performed after the MAC operation. However, the same processing result can be obtained even if the order of the MAC operation and the shift operation is switched. A part of the flowchart of <FIG> will change when the order of the MAC operation and the shift operation is switched. Step S107 changes to steps S901 to S903 in <FIG>.

<FIG> shows an example of the arrangement of a data processing unit <NUM> according to this embodiment. A shift operator <NUM> shifts the feature data stored in a data memory <NUM> based on a shift parameter, and the multiplier-accumulator <NUM> calculates the MAC operation results based on the shifted feature data and the filter coefficient.

The MAC operation and the shift operation (steps S901 to S903) performed in step S107 will be described. In step S901, a control unit <NUM> reads out feature data from the data memory <NUM> and reads out a filter coefficient from a coefficient memory <NUM>. In step S902, the shift operator <NUM> shifts the feature data based on the shift parameter set in step S103. In step S903, the multiplier-accumulator <NUM> calculates the MAC operation results based on the shifted feature data and the filter coefficient.

In this embodiment, the shift operator <NUM> includes P α-bit data shift operation units for calculating shift operation results, and the multiplier-accumulator <NUM> includes P MAC operation units for calculating MAC operation results. The output of the multiplier-accumulator <NUM> is represented by equation (<NUM>) below and is equivalent to the output of a shift operator <NUM> shown in equation (<NUM>).

<FIG> shows an example of a case in which P = <NUM>, β = <NUM>, and α = <NUM>. The bit width of the input data for the shift operator <NUM> is <NUM> bits, and the bit width of the input data for the multiplier-accumulator <NUM> is <NUM> bits, and the bit width of the input data for an adder <NUM> is <NUM> bits. Since the circuit scale of the shift operator <NUM> and the circuit scale of the multiplier-accumulator <NUM> are different due to the difference in the bit widths, it is possible to reduce the overall circuit scale by switching the order of the shift operator <NUM> (shift operator <NUM>) and the multiplier-accumulator <NUM> (multiplier-accumulator <NUM>).

The first and second embodiments described an example in which the bit widths of input feature data are α bits (the bit width of each MAC operation unit) and β bits (the product of the bit width of each MAC operation unit and the number of MAC operation units). However, the present invention is not limited to these, and bit widths other than α and β may be used.

Input feature data whose bit width is γ bits can be processed in this embodiment. <FIG> shows an example in which the feature data is <NUM> bits. In a case in which the feature data is <NUM> bits, a multiplier-accumulator <NUM> divides each of two sets of <NUM>-bit feature data <NUM> (values: <NUM>, <NUM>) by <NUM> bits as shown in <FIG>. The multiplier-accumulator <NUM> uses the four sets of <NUM>-bit data (values: <NUM>, <NUM>, <NUM>, <NUM>) obtained by the division and two filter coefficients to calculate four MAC operation results. A shift operator <NUM> shifts the four MAC operation results based on two shift parameters. An adder <NUM> adds the four shifted MAC operation results to calculate one set of feature data (the total sum of the MAC operation results of two sets of <NUM>-bit input feature data). In this manner, a data processing unit <NUM> can process two sets of <NUM>-bit input feature data in parallel. γ is the bit width of the input feature data, and the value of γ differs from the value of β. The definitions of α, β, and P are the same as in the first embodiment, and γ, α, and P' satisfy the following condition. <MAT>
where γ is smaller than β, and P is a multiple of P'. In a case in which the bit width of input feature data I'(γ) is γ bits, output data O(n) of the adder <NUM> is expressed as equation (<NUM>) below based on the premise of equations (<NUM>), (<NUM>), and (<NUM>). The MAC operation result O(n) of an n-th output feature image is given by <MAT>
where I(α), p(m) is the input data of the α-bit data MAC operation unit, Cp(m, n) is a filter coefficient, and S(p) is a shift parameter. A variable m is the number (the processing number of the multiplier-accumulator <NUM>) of an α-bit input feature image group (<NUM> group = P images). The MAC operation units are divided into P/P' sets and the shift operation units are divided into P/P' sets, and a variable q is a set number of the MAC operation unit. A variable p is the MAC operation unit number and the shift operation unit number in the set, and a variable n is the output feature image number. The shift operation is expressed by processing by the power of <NUM>.

A filter coefficient Cp, q(m, n) is a filter coefficient C'((m-<NUM>) × P/P' + q, n) corresponding to an {(m-<NUM>) × P/P' + q }-th γ-bit feature image. The filter coefficient is calculated based on the set number q of the MAC operation unit. Since a part of the filter coefficient is shared, the number of filter coefficients to be supplied in parallel to the P MAC operation units is P/P' and the transfer count is P/P'.

In this case, the input feature data I'(γ) is divided into P' sets of α-bit data I(α), p(m). A shift parameter S(. ) is calculated based on the bit width α of the MAC operation unit and the MAC operation unit number p.

The γ-bit input feature data I'(γ) is expressed by the divided P' sets of α-bit data I(α), p, q(m).

A substitution of equations (<NUM>), (<NUM>), and (<NUM>) into equation (<NUM>) yields the equation of the output data O(n) as
<MAT>.

By setting the value of the shift parameter S(p, q) and the number of filter coefficients, the feature data I'(γ) whose bit width is γ bits can be processed by using the same operators (the multiplier-accumulator <NUM>, the shift operator <NUM>, and the adder <NUM>) as in the first embodiment.

<FIG> and <FIG> show an example of an arrangement when P = <NUM>, β = <NUM>, and α = <NUM>, and <FIG> show an example of the processing time when the processing target network shown in <FIG> is processed by using the hardware arrangement shown in <FIG>.

<FIG> and <FIG> show an example of a case in which P' = <NUM> and γ = <NUM>, and a layer <NUM> (<NUM>-bit data, the number M of input feature images = <NUM>) is processed. Each of feature data I'(<NUM>), (<NUM>) and I'(<NUM>), (<NUM>) of feature images (<NUM>, <NUM>) and (<NUM>, <NUM>) is <NUM> bits, and four sets of data I(<NUM>), <NUM>(<NUM>) to I(<NUM>), <NUM>(<NUM>) divided based on equation (<NUM>) are input to the multiplier-accumulator <NUM>. The shift operation results are calculated by using the input feature data, the shift parameter, and a filter coefficient C(m, n), the calculated shift operation results are input to the adder <NUM>, and an initial value of zero is added to the calculated results. The calculation result becomes the shift operation result and is held by the adder <NUM>. The duration of the process is <NUM>.

Each of feature data I'(<NUM>), (<NUM>) and I'(<NUM>), (<NUM>) of feature images (<NUM>, <NUM>) and (<NUM>, <NUM>) is <NUM> bits, and four sets of data I(<NUM>), <NUM>(<NUM>) to I(<NUM>), <NUM>(<NUM>) divided based on equation (<NUM>) are input to the multiplier-accumulator <NUM>. The shift operation results are calculated by using the input feature data, the shift parameter, and a filter coefficient C(m, n), the calculated shift operation results are input to the adder <NUM>, and the results are added to the preceding result. The duration of the operation is <NUM>. Finally, the feature data of a feature image (<NUM>, <NUM>) is output via the processing unit <NUM>. The processing time of four feature images is <NUM>.

In this manner, the embodiment is advantageous in that it is highly flexible since feature data other than data whose bit width is α bits (the bit width of each MAC operation unit) or β bits (the product of the bit width α of each MAC operation unit and the number P of MAC operation units) can be processed.

Although the first embodiment described an example in which activation processing is executed by a processing unit <NUM>, the execution of the activation processing is not limited to the processing unit <NUM>, and it may be set so that another device, for example, a CPU <NUM> will execute the activation processing. This is also similarly applicable to other processing operations, and the above embodiments have shown merely an example of the main body of various kinds of processing, and a main body different from the main body described in the above embodiments may be used.

In addition, activation/pooling processing was executed in accordance with the layer information in the first embodiment. However, the activation/pooling processing may be omitted depending on the case.

Also, although the first to third embodiments described a case in which the filter size (the height and the width of each filter) is <NUM> × <NUM>, the filter size is not limited to <NUM> × <NUM> and may be another size. The numerical values used in the description of the above embodiments are merely examples used to make a more specific explanation and are not intended to limit the numerical values to be used to those described in the above embodiments.

In a case in which the filter size is small, there is an advantage in that the capacity of a memory (a coefficient memory <NUM> or <NUM>) for holding filter coefficients can be made smaller. The minimum value to be set as the filter width and the filter height is <NUM>.

Also, the first to third embodiments set the number of input feature images to be M and the number of output feature images to be N. However, numerical values applicable to M and N are not limited to specific numerical values. In this manner, the numerical values applicable to various kinds of variables described above are not limited to specific numerical values.

In addition, although the filter coefficients were held in the coefficient memory <NUM> or <NUM> and the feature data were held in a data memory <NUM> in the first to third embodiments, the memories for holding the filter coefficients and the feature data are not limited to specific memories. For example, the filter coefficients and the feature data may be held in a memory included in a multiplier-accumulator <NUM> or <NUM> or may be held in a RAM <NUM>.

In addition, the bit width of each filter coefficient is not limited to a specific bit width. Furthermore, although CNN has been used as the processing target network in the first to third embodiments, the processing target network is not limited to CNN and may be a network to which a plurality of other kinds of layers are hierarchically connected such as RNN, MLP (multilayer perceptron), or the like.

Claim 1:
An information processing apparatus, characterized by comprising:
control means (<NUM>) for setting shift amounts corresponding to a plurality of shift operation means based on a bit width of data, for each layer of a network including a plurality of layers;
a multiplier-accumulator (<NUM>) comprising a plurality of MAC (multiply-accumulate) means for executing MAC operations on a plurality of data and a plurality of filter coefficients of the layer;
a shift operator (<NUM>) comprising the plurality of shift operation means for shifting a plurality of MAC operation results obtained by the plurality of MAC means by multiplying each MAC operation result by a power of <NUM> based on the shift amount; and
adding means (<NUM>) for calculating a total sum of the plurality of MAC operation results shifted by the plurality of shift operation means,
wherein the number (P) of MAC means comprised in the multiplier-accumulator is the same as the number (P) of shift operation means comprised in the shift operator,
wherein, in a case where the bit width of data input to a layer is equal to a bit width (β) which is a product of the bit width (α) of each MAC means with the number (P) of MAC means, the input data is divided into pieces of data (<NUM>) corresponding to the number (P) of MAC means, and the pieces of data are operated in parallel by the multiplier-accumulator and the shift operator.