QUANTIZATION METHOD AND APPARATUS FOR ARTIFICIAL NEURAL NETWORK

Provided are a quantization method and a quantization apparatus for an artificial neural network. The quantization method for the artificial neural network may include estimating sample scale factors of first sample parameters that are part of first parameters within the artificial neural network, determining a prediction scale factor based on the sample scale factors, and quantizing first parameters based on the prediction scale factor.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119 to Korean Patent Application No. 10-2023-0111513, filed on Aug. 24, 2023, in the Korean Intellectual Property Office, the disclosure of which is herein incorporated by reference in its entirety.

BACKGROUND

The inventive concept relates to an artificial neural network, and more particularly, to a quantization method, a quantization apparatus, and a quantization system for an artificial neural network.

An artificial neural network may refer to a computing device or a method performed by the computing device to implement interconnected sets of artificial neurons (or neuron models). Artificial neurons may generate output data by performing simple operations on input data, and the output data may be transmitted to other artificial neurons. As an example of an artificial neural network, a deep neural network or deep learning may have a multi-layered structure.

When dynamic quantization is used in a deep learning inference operation, a scale factor may be required for the input of each layer of a model. Deep learning interference may require a considerable amount of computations when a quantization operation is performed by obtaining scale factors for every input for every layer.

SUMMARY

The inventive concept provides a quantization method and a quantization apparatus for an artificial neural network, in which high accuracy and low computational complexity of the artificial neural network may be achieved.

According to one or more embodiments, there is provided a quantization method for an artificial neural network, the quantization method including estimating sample scale factors of first sample parameters, the first sample parameters being part of first parameters within the artificial neural network, determining a prediction scale factor based on the sample scale factors, and quantizing first parameters based on the prediction scale factor.

According to one or more embodiments, there is provided a quantization system for an artificial neural network, the quantization system including at least one processor, and a storage medium configured to store commands executable by the at least one processor to perform a quantization process of the artificial neural network, wherein the quantization process of the artificial neural network may include estimating sample scale factors of first sample parameters, the first sample parameters being part of first parameters within the artificial neural network, determining a prediction scale factor based on the sample scale factors, and quantizing first parameters based on the prediction scale factor.

According to one or more embodiments, there is provided a quantization apparatus for an artificial neural network, the quantization apparatus including a scale factor estimator configured to estimate sample scale factors of first sample parameters, the first sample parameters being part of first parameters within the artificial neural network and to determine a prediction scale factor based on the sample scale factors, and a quantizer configured to quantize the first parameters based on the prediction scale factor.

DETAILED DESCRIPTION

Hereinafter, example embodiments of the inventive concept will be described in detail with reference to the accompanying drawings.

FIG.1is a block diagram illustrating a quantization system according to an example embodiment.

FIG.1is a block diagram illustrating a quantization system100according to an example embodiment. An artificial neural network may refer to a computing system that attempts to simulate a biological neural network in the brain of an animal. In the artificial neural network, performing of a task may be learned by considering a plurality of samples (or examples), which is different from a classical algorithm such as rule-based programming that performs a task according to a pre-defined condition. The artificial neural network may have a structure in which artificial neurons (or neurons) are connected to one another, and connection between the neurons may be referred to as a synapse. The neuron may process a received signal and may transmit the processed signal to another neuron through the synapse. An output of the neuron may be referred to as an activation. The neuron and/or synapse may have a variable weight, and the influence of the signal processed by the neuron may be increased or decreased according to the weight. Also, a bias associated with an individual neuron is applied.

A deep neural network or a deep learning architecture may have a layer structure, and an output of a certain layer may be an input of a subsequent layer. In such a multi-layered structure, each of layers may be trained according to a plurality of samples. The artificial neural network, such as a deep neural network, may be implemented by a number of processing nodes that correspond to artificial neurons, respectively, and high computational complexity may be required to obtain satisfactory results, for example, high accuracy results. Thus, many computing resources may be required.

In order to reduce computational complexity, the artificial neural network may be quantized. Quantization may refer to a process in which input values are mapped to values of a number smaller than the number of the input values, such as mapping real numbers to integers through rounding off. In the artificial neural network, quantization may include a process of converting a floating decimal point neural network into an integer neural network. For example, in the artificial neural network, quantization may be used in an activation, a weight of a layer, or the like. A floating decimal point number may include a sign, an exponent, and a significant, wherein an integer number may include an integer part. In some embodiments, the integer part of the integer number may include a sign bit. Referring toFIG.1, an input data set IN is expressed in floating decimal point numbers, and an output data set OUT that undergoes a quantization process may be expressed in integer numbers. The artificial neural network using floating decimal point numbers may have high accuracy and computational complexity, while the artificial neural network using integer numbers may have reduced accuracy and less computational complexity.

In the artificial neural network, quantization for the artificial neural network may result in a decrease in accuracy due to the trade-off relationship between the accuracy of results and the computational complexity, and the degree of reduction in accuracy may depend on a method of quantization. Hereinafter, as described below with reference to the accompanying drawings, the quantization system100according to an example embodiment may provide quantization according to requirements while minimizing the reduction of accuracy, and thus, a quantized neural network having reduced complexity while having sufficiently high performance may be provided.

The quantization system100may be any type of a system that performs quantization according to example embodiments and may also be referred to as a quantization apparatus. For example, the quantization system100may be a computing system including at least one processor and at least one memory. As a non-limiting example, the quantization system100may be a mobile computing system, such as a laptop computer, a smartphone, or the like, as well as a stationary computing system, such as a desktop computer and a server, or the like. As shown inFIG.1, the quantization system100may include an estimator120and a quantizer140, and each of the estimator120and the quantizer140may be implemented by a logic block implemented by logic synthesis, a software block performed by a processor, or a combination thereof. In some embodiments, each of the estimator120and the quantizer140may be a procedure as a set of a plurality of commands executed by a processor, or may be stored in a memory that is accessible by the processor.

Herein, the scale factor estimator120and the quantizer140may each be analog and/or digital circuits including one or more of a logic gate, an integrated circuit, a microprocessor, a microcontroller, a memory circuit, a passive electronic component, an active electronic component, an optical component, and the like, and may be configured to execute software and/or firmware to perform the corresponding functions or operations described above.

Referring toFIG.1, the quantization system100may receive the input data set IN and may generate the quantized output data set OUT. The input data set IN may include a plurality of pieces of input data, and the quantized output data set OUT may include a plurality of pieces of output data. As shown inFIG.1, pieces of matrix-type data as the input data set IN may be input to the quantization system100according to an example embodiment, and the quantization system100may output pieces of matrix-type data as the quantized output data set OUT. Pieces of data according to an example embodiment may include pieces of data of each pixel extracted from an image. Each piece of input data that constitutes the input data set IN may include a floating decimal point number. Each piece of output data that constitutes the output data set OUT to be output through the quantization system100may include an integer number. As described below, in the quantization system100according to an example embodiment, a computation for determining scale factors for all of inputs in a quantization operation may be omitted and accordingly, the total amount of computations may be reduced.

The estimator120may receive the input data set IN and may determine a prediction scale factor such as an activation, a weight, or the like to provide the prediction scale factor to the quantizer140. In order to quantize a floating decimal point number-type variable, a scale factor may be required. The scale factor may indicate a value for mapping a range of a value of data that is being quantized to a quantized range that corresponds to the maximum value and the minimum value that can be represented by the number of bits used in a quantization process. Detailed descriptions of the scale factor will be provided below in detail with reference toFIGS.3A and3B. A process of quantizing a variable may be classified into dynamic quantization and static quantization. The quantization system100according to an example embodiment may quantize pieces of data by using the dynamic quantization method in which a weight, an activation, or the like in the artificial neural network is quantized in real time in an interference step after learning. Pieces of data to be quantized such as a weight, an activation, or the like in the artificial neural network may be referred to as a parameter for convenience in the present specification. When scale factors are calculated or estimated for all of the parameters to be quantized, the scale factors may have high accuracy but an excessive amount of computations may be required. As described below, the estimator120according to an example embodiment may estimate a scale factor for a portion of the parameters to be quantized and may determine a prediction scale factor from the estimated scale factors to provide the determined prediction scale factor to the quantizer140. That is, the estimator120may estimate a scale factor for only a portion of the parameters to be quantized and skip estimating a scale factor for the other portion of the first parameters. Since the quantization system100may estimate the scale factor only for a portion of the parameters and not all of the parameters, determine a prediction scale factor based on the estimated scale factors from the portion of the parameters, and perform quantization for the parameters using the prediction scale factor, computation for determining scale factors for all of inputs in a quantization operation during deep learning interference may be omitted and accordingly, the total amount of computations may be reduced.

The quantizer140may receive the prediction scale factor corresponding to the parameters from the estimator120and may quantize the parameters based on the prediction scale factor to generate the quantized output data set OUT. A detailed quantization process of the quantizer140will be described below in detail with reference toFIG.7or the like.

FIG.2is a view illustrating an example of an artificial neural network according to an example embodiment.

Referring toFIG.2, the artificial neural network may have a structure including an input layer L1, hidden layers (L2, L3), and an output layer Ln (where n is a natural number of 3 or more), and operations may be performed based on the received input data (e.g., I1, I2), and output data (e.g., O1, O2) may be generated based on the result of operations.

The artificial neural network may be a deep neural network including one or more hidden layers, or n-layers neural networks. For example, as shown inFIG.2, the artificial neural network may be a deep neural network including the input layer L1, the hidden layers L2and L3, and the output layer Ln. The deep neural network may include a Convolutional Neural Networks (CNN), a Recurrent Neural Networks (RNN), a Deep Belief Network, a Restricted Boltzman Machine, or the like, and example embodiments are not limited thereto.

When the artificial neural network has a deep neural network (DNN) structure, the artificial neural network includes more layers through which valid information may be extracted, such that the artificial neural network may process more complex data sets than in other types of an artificial neural network according to the related art. The artificial neural network may also include layers having various different structures from those shown inFIG.2.

Each of the layers L1to Ln included in the artificial neural network may include a plurality of artificial nodes, which are also known as neurons, units or similar terms. For example, as shown inFIG.2, an input layer L1may include two nodes ND and hidden layers L2and L3may include three nodes ND. However, this is only an example, and each of the layers included in the artificial neural network may include various numbers of nodes ND, and the numbers of the nodes ND included in each layer may be different from each other.

Nodes included in each of the layers included in the artificial neural network may be connected to each other to exchange data. For example, one node ND may receive data from other nodes ND to perform computations and may output the result of computations to other nodes ND.

An input and an output of each of the nodes ND may be referred to as an activation. The activation may be an output value of one node ND and may be an input value of the nodes ND included in the next layer. Each of the nodes ND may determine its own activation based on activations and weights received from the nodes ND included in a previous layer. The weights are network parameters used to calculate the activation in each node ND and may be values allocated to the connection relationship between the nodes ND. For example, in the second layer L2, the nodes ND may determine their own activations based on activations (a11, a12), which are received from the previous layer L1, weights (w21, w22, w23, w24, w25, w26), and biases (b21, b22, b23). Each of the nodes ND may be a computational unit that receives an input and outputs an activation and may perform input-output mapping.

The artificial neural network may include an activation function between the layers. An activation function may convert the output of the previous layer into an input of the next layer. For example, the activation function may be a non-linear function, such as Rectified Linear Unit (ReLU), Parametric Rectified Linear Unit (PReLU), hyperbolic tangent (tan h), sigmoid function, and may convert the output of the second layer L2non-linearly between the second layer L2and the third layer L3. The activation may be a value obtained by applying the activation function to a weighted sum of activations received from the previous layer.

Subsequently, referring toFIGS.1and2, an object to be quantized by the quantization system100may include activations and weights of the artificial neural network. For example, the activations (a11, a12) received from the first layer L1which are input to the second layer L2and the weights (w21, w22, w23, w24, w25, w26), may be data in floating decimal point number format. The quantization system100may quantize the activations (a11, a12), and the weights (w21, w22, w23, w24, w25, w26) into data in an integer number format through the quantization process described above. The weighted sum may be calculated using the quantized activations and weights and thus the size of a deep learning model may be reduced and the amount of computations may be reduced. Subsequently, as the activation function is applied to a value of the calculated weighted sum, activations (a21, a22, a23) that are input to the third layer L3from the second layer L2may be data in a floating decimal point number format again. That is, the object to be quantized by the quantization system100may be parameters of a specific layer or may be parameters corresponding to each of the layers.

FIGS.3A and3Bare views illustrating a type of quantization method according to an example embodiment.

Referring toFIG.3A, in the Affine quantization method, the maximum value and the minimum value of data before being quantized may linearly correspond to the maximum value and the minimum value of a data format to be quantized, respectively. For example, as shown inFIG.3A, the maximum value (α) of data before being quantized may be 7 and the minimum value (β) of data before being quantized may be −5. When data is quantized into an integer number format of 8 bits, the quantized data may have a value in a range of −128 to 127. According to the Affine quantization method, the maximum value (α) of data before being quantized may correspond to 127 and the minimum value (β) of data before being quantized may correspond to −128. Data values between the maximum value (α) and the minimum value (β) may linearly correspond to data values between 127 and −128. Thus, when the maximum value (α) of data before being quantized and the minimum value (β) of data before being quantized have different values, value 0 before being quantized may correspond to another value that is not 0 after being quantized. In the process of linearly corresponding values, the ratio of the range of values of data after being quantized with respect to the range of values of data before being quantized may be a scale factor. That is, in the Affine quantization method, the scale factor may be calculated as the following [Equation 1].

In [Equation 1], b represents a number of bits of integer number data after quantization. Referring to the values illustrated inFIG.3, the scale factor of the Affine quantization method ofFIG.3Amay be 255/12.

Referring toFIG.3B, in the scale quantization method, similarly to the Affine quantization method, the maximum value and the minimum value of data before being quantized may linearly correspond to the maximum value and the minimum value of a data format to be quantized, respectively. However, in the scale quantization method, in order to make the value of 0 before being quantized to correspond to the value of 0 after being quantized, the relationship α=−β may be satisfied. That is, in the scale quantization method, the maximum value (α) of an absolute value of data before being quantized may be obtained, and α and −α may linearly correspond to the maximum value and the minimum value of a data format to be quantized, respectively. In the process of linearly corresponding values, the ratio of the range of values of data after being quantized with respect to the range of values of data before being quantized may be a scale factor. That is, in the scale quantization method, the scale factor may be calculated as the following [Equation 2].

In [Equation 2], b represents a number of bits of integer number data after quantization. Referring to the values illustrated inFIG.3, the scale factor of the scale quantization method ofFIG.3bmay be 127/7.

In both the quantization methods ofFIGS.3A and3B, the scale factor may be multiplied by data before being quantized and values obtained by the multiplication may be located in the range of data after being quantized. The quantization method may include a round operation of expressing the result of multiplying the scale factor by data to be quantized in the format of integers, and a clip operation of correcting values that are out of the range, as described below in detail with reference toFIG.7. In both the quantization methods ofFIGS.3A and3B, a process of calculating the maximum value and the minimum value of data may be required to calculate or estimate the scale factor. When various data sets are input and scale factors are calculated for all data sets, an excessive amount of computations may be required. As described below, according to an example embodiment, a computation for determining scale factors for all the inputs in a quantization operation during deep learning inference may be omitted and accordingly, the total amount of computations may be reduced.

FIG.4is a flowchart illustrating a quantization method for an artificial neural network according to an example embodiment.

For example, the quantization method ofFIG.4may be performed by using the quantization system100ofFIG.1. Hereinafter,FIG.4will be described with reference toFIG.1.

Referring toFIG.4, in operation S410, an operation of estimating sample scale factors of first sample parameters may be performed. The first sample parameters may be part of first parameters. The first parameters may include input variables that are input to a first layer, or weights of the first layer, that are to be quantized. The input variables input to the first layer may be activation outputs from the previous layer of the first layer.

The first sample parameters that are part of the first parameters may be parameters for determining a prediction scale factor to be described below. The first sample parameters may be selected by sampling part of input first parameters (not a pre-defined sample). For example, when the input first parameters include A number of data sets, the first sample parameters may include first B number of data sets of the input first parameters (where B is a natural number less than natural number A). That is, in an example embodiment, the first sample parameters may refer to a series (e.g., consecutive) of first parameters input among the first parameters. The first B number of data sets of the input first parameters may be first sample parameters, and an operation of estimating or calculating a scale factor about each of the first sample parameters may be performed on the first B number of data sets of the input first parameters. The scale factor corresponding to the first sample parameter may be estimated by using the method described above with reference toFIGS.1through3. For example, when the input data set IN is input as the first sample parameter as shown inFIG.1, the maximum value or the minimum value in the data set may be obtained and the scale factor may be calculated or estimated based thereon, as described above with reference toFIG.3. In order to obtain the maximum value or the minimum value in the data set, a comparator may be utilized or a comparison operation may be performed, as described below in detail with reference toFIG.7.

Referring toFIG.4, in operation S420, an operation of determining a prediction scale factor based on sample scale factors may be performed. In the previous operation S410, after sample scale factors are estimated for the first sample first parameters, a prediction scale factor that may represent all first parameters may be determined based on the sample scale factors. As described above, when scale factors are calculated for all of parameters, an excessive amount of computations may be required. Thus, in the quantization method of the present disclosure, the prediction scale factor may be utilized in a quantization calculation. In a method of determining the prediction scale factor based on the sample scale factors, various equations may be utilized, and different equations may be selected according to the characteristics or relation of data sets. The method of determining the prediction scale factor according to an example embodiment may use the following [Equation 3].

[Equation 3] is a computational equation to be performed by a computer and/or program, and is a computational equation indicating a computation method of the prediction scale factor PSF as the sample scale factors SF are sequentially input to the quantizer140. In [Equation 3], specifically, the prediction scale factor may be calculated by reflecting an exponential moving average. [Equation 3] shows the case where a proportional coefficient is ½, in the computation method of the prediction scale factor PSF, and embodiments are not limited to [Equation 3]. Another example formula based on [Equation 3] is shown in Equation 4 below.

[Equation 4] is a computational equation to be performed by a computer and/or program, and is a computational equation indicating a computation method of the prediction scale factor PSF as the sample scale factors SF are sequentially input to the quantizer140. k is a real number between 0 and 1, and the larger the value k is, the larger the effect of a previous observation value may be reduced. When k is ½, [Equation 3] is established.

The method of determining the prediction scale factor according to another example embodiment may use the following [Equation 5].

[Equation 5] is a computational equation to be performed by a computer and/or program, and is a computational equation indicating a computation method of the prediction scale factor PSF as the sample scale factors SF are sequentially input to the quantizer140. In [Equation 5], specifically, the prediction scale factor may be calculated by reflecting an arithmetic mean.

The method of determining the prediction scale factor according to another example embodiment may use the following [Equation 6].

[Equation 6] is a computational equation to be performed by a computer and/or program, and is a computational equation indicating a computation method of the prediction scale factor PSF as the sample scale factors SF are sequentially input to the quantizer140. In [Equation 6], specifically, the prediction scale factor may be calculated by reflecting the minimum value of the sample scale factor.

The method of determining the prediction scale factor according to another example embodiment may use the following [Equation 7].

[Equation 7] is a computational equation to be performed by a computer and/or program, and is a computational equation indicating a computation method of the prediction scale factor PSF as the sample scale factors SF are sequentially input to the quantizer140. In [Equation 7], specifically, the prediction scale factor may be calculated by reflecting the maximum value of the sample scale factor.

Equations such as [Equation 3] to [Equation 7] may be calculated by calculating the sample scale factors sequentially for sample parameters input in the sequence in time, and reflecting the sample scale factors sequentially calculated in the prediction scale factor value accumulatively. In this case, the sample scale factor that is firstly calculated (or estimated) may be determined as an initial prediction scale factor. In addition, after sample scale factors corresponding to sample parameters are calculated, a prediction scale factor value may also be calculated based on all of sample scale factors.

Through equations such as [Equation 3] to [Equation 7], in operation S420, an operation of determining a prediction scale factor based on the sample scale factors may be performed. However, this is only an example equation, and a method of determining the prediction scale factor based on the sample scale factors is not limited thereto. For example, a value of the prediction scale factor calculated from the sample scale factors may be adjusted by adding or multiplying a proper coefficient to Equations such as [Equation 3] to [Equation 7]. In addition, the value of the prediction scale factor may also be calculated by combining a plurality of equations including one or more of the above equations. By utilizing an appropriate equation, a quantization operation, in which accuracy may be secured while reducing the amount of computations of the artificial neural network, may be performed.

Referring toFIG.4, in operation S430, an operation of quantizing first parameters based on the prediction scale factor may be performed. The quantization operation may be performed by the operation of the quantizer140ofFIG.1and may also be performed by a quantization apparatus including the configuration ofFIG.7, which will be described below. A method of quantizing first parameters may include the Affine quantization method or the scale quantization method described above with reference toFIG.3. Specifically, the prediction scale factor calculated in the previous operation S420may be a factor calculated instead of a process of calculating scale factors for all parameters and may be used in a quantization process of the first parameters. Instead of calculating the scale factor of each of the first parameters, the calculated prediction scale factor may be used to quantize the first parameters. According to the above-described embodiment, by using the prediction scale factor, a computation for determining scale factors for all of inputs in a quantization operation during deep learning inference may be omitted and accordingly, the total amount of computations may be reduced. In addition, computing resources for implementing an artificial neural network based on the quantized artificial neural network having high accuracy may be reduced, and the application range of the artificial neural network may be extended.

FIG.5is a block diagram illustrating a quantization apparatus for an artificial neural network according to an example embodiment.

Referring toFIG.5, a quantization apparatus500for the artificial neural network may include a memory510and a processor520. Hereinafter,FIG.5will be described with reference toFIGS.1and4, and redundant descriptions with descriptions with reference to the previous drawings are omitted.

The memory510of the quantization apparatus500may store a program for quantization for the artificial neural network according to an example embodiment and may store activation or data quantized by the quantization method. In addition, the memory510may store the quantized output data set OUT ofFIG.1and may store bias or data generated in a quantization process. The memory510may store a series of commands related to performance of quantization. The memory510may communicate with the processor520to exchange data with each other.

The processor520may be one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a Neural Processing Unit (NPU). However, this is an example, and the processor520is not limited to the description above. The processor520illustrated inFIG.5may include a data fetcher522, a vector processing unit524, and an inner product array526. The data fetcher522may receive data such as weights, activation, bias or the like from the memory510and may transmit the data to the vector processing unit524. The vector processing unit524may include a Quantization with Scale factor Estimation (QSE) unit524_1therein. The illustration ofFIG.5is an example, and the QSE unit524_1may function independently outside the vector processing unit524. The QSE unit524_1may obtain a prediction scale factor corresponding to parameters such as input weights, activation, and the like and may perform quantization based on the prediction scale factor. A detailed operation of the QSE unit524_1will be described below with reference toFIG.7.

The vector processing unit524may transmit quantized parameters to the data fetcher522, and the data fetcher522may transmit the quantized parameters to the inner product array526. The inner product array526may calculate a weighted sum based on data such as the input quantized weights, quantized activation, quantized bias, and the like.FIG.5illustrates that the quantization apparatus100includes N processing units526_1,526_2, . . .526_N. However, this is an example, and the quantization apparatus100may also include processing units PE in different numbers. The processing unit PE may calculate a weighted sum between quantized weights and quantized activations, and a detailed operation of the processing unit PE will be described below in detail with reference toFIG.6. When calculating the weighted sum based on unquantized parameters, the inner product array526requires a large amount of computations, and much time and many resources may be required in an interference operation of deep learning. According to an example embodiment, a computation for estimating scale factors for all of inputs in a quantization operation during deep learning inference may be omitted and accordingly, the total amount of computations may be reduced.

Data such as activations output through the inner product array526may be transmitted to the vector processing unit524, and the vector processing unit524may receive additional data from the data fetcher522to perform a quantization operation of the next layer. In addition, the vector processing unit524may transmit the output activation data transmitted from the inner product array526to the memory510so as to store the output activation data.

Although not shown inFIG.5, the configuration of a buffer or a register for storing data therein may be additionally included, and the configuration such as an adder, a multiplexer of selecting or adding data, may be additionally included.

FIG.6is a block diagram illustrating a processing device inside the quantization apparatus500ofFIG.5according to an example embodiment.

Referring toFIGS.5and6, the inner product array526included in the apparatus500may include a plurality of processing devices PE,526_1,526_2, . . .526_N. Each of the processing devices PE, e.g.,526_N as illustrated inFIG.6, may include a plurality of multipliers MPR and an adder tree. The processing devices PE526_N may perform the computation shown in the following [Equation 8].

In [Equation 8], I represents an input quantized activation, W represents an input quantized weight, Bias represents a bias, and y represents an output. The processing device PE526_N may receive X number of quantized activations, quantized weights corresponding to the quantized activations, and a bias. X may represent a number of channels of a layer on which computation is performed and may have different values for each layer. In addition,FIG.6illustrates that X number of multipliers MPR are present. However, this is an example, and the processing devices PE526_N may include different numbers of multipliers MPR according to the hardware specification. Quantized activations and quantized weights multiplied by multipliers may be added by the adder tree along with a bias. As a result, the processing devices PE526_N may calculate a weighted sum by receiving X quantized activations, quantized weights, and a bias value.

Although not shown inFIG.6, the processing devices PE526_N according to an example embodiment may include a register or a memory for storing data according to embodiments, and may additionally include a component for applying an activation function.

FIG.7is a block diagram illustrating a QSE unit inside the quantization apparatus ofFIG.5according to an example embodiment.

Referring toFIGS.5and7, the vector processing unit524inside the quantization apparatus500may include a QSE unit524_1. The QSE unit524_1may include a scale factor estimator710and a quantizer720. The scale factor estimator710and the quantizer720may be examples of the estimator120and the quantizer140ofFIG.1and may perform partially common functions. Regarding the operation of the quantization method, redundant descriptions with the descriptions ofFIGS.1and4are omitted.

The QSE unit524_1may perform scale factor estimation and quantization of parameters such as activations, weights, and the like. The QSE unit524_1may receive first parameters and may output quantized parameters. In this case, each of the parameters may refer to data sets, and each of the data sets may include N data. For example, in the case of an image having 3×3 pixels, one image data set may include 9 pieces of data corresponding to 9 pixels, and in a quantization operation in a first layer, N may be 9. In this case, each of the first parameters may include 9 pieces of data, and the QSE unit524_1may receive input data 0_1, 0_2, . . . , and 0_N that constitutes one input parameter and may output quantized output data Q_1, Q_2, . . . , and Q_N. The quantized output data Q_1, Q_2, . . . , and Q_N may constitute one quantized output parameter. That is, the input and output operations illustrated inFIG.7are operations corresponding to one parameter, that is, one data set. For example, when 1,000 images, each image having 3×3 pixels, are processed, the first layer may receive 9 pieces of input data 0_1, 0_2, . . . , and 0_N and may output quantized output data Q_1, Q_2, . . . , and Q_N, with respect to one image. In addition, input and output of corresponding data may be performed on the 1,000 images.

The scale factor estimator710inside the QSE unit524_1may estimate sample scale factors of first sample parameters. The first sample parameters may be part of first parameters. In an embodiment, the QSE unit524_1may estimate a scale factor for only part of the first parameters to be quantized and skip estimating a scaling factor for the other part of the first parameters. As described above with reference toFIG.4, the first sample parameters according to an example embodiment may refer to a certain number of first parameters among the first parameters. The first sample parameters may be consecutive first parameters and may be a group of first parameters that first appears. For each of the first sample parameters, the scale factor estimator710may calculate the maximum value and the minimum value of a sample parameter by using a plurality of comparators CPRs to estimate a sample scale factor for the corresponding sample parameter. As described above with reference toFIGS.3A and3B, the scale factor estimator710may calculate a sample scale factor based on the maximum value and the minimum value calculated through the plurality of comparators CPRs.

The scale factor estimator710may include a register711configured to store a sample scale factor and a prediction scale factor therein. The scale factor estimator710may calculate the sample scale factor of each first sample parameter by using the plurality of comparators CPRs described above until B number of first sample parameters, e.g., B data sets, are input to the scale factor estimator710. A value of B may be preset. The calculated sample scale factors may be delivered to the quantizer720and may be stored in the register711. The scale factor estimator710may calculate the prediction scale factor based on equations such as [Equation 3] through [Equation 6] and the calculated sample scale factors, as described above with reference toFIG.4.

After all of first sample parameters, e.g., B data sets, are input to the QSE unit524_1, the register711of the scale factor estimator710may calculate and store the prediction scale factor. Subsequently, when additional first parameters are input to the QSE unit524_1, the scale factor estimator710may not perform computations for calculating the maximum value or the minimum value through a comparison computation between pieces of data of each of the additional first parameters. That is, the scale factor estimator710may output the prediction scale factor stored in the register711to the quantizer720, without estimating the scale factor according to an input of the first parameters, after calculation of the prediction scale factor is completed. Scale factor computation may not be performed on all parameters, e.g., all data sets, and a quantization computation may be performed using the prediction scale factor. Thus, according to an example embodiment, a computation for estimating scale factors for all of inputs in a quantization operation during deep learning inference may be omitted and accordingly, the total amount of computations may be reduced.

The quantizer720may output quantized output data Q_1, Q_2, . . . , and Q_N after input data O_1, O_2, . . . , and O_N and the scale factor SF from the scale factor estimator710are input to the quantizer720. In a section where the first sample parameters are input to the quantizer720, the calculated sample scale factor may be input from the scale factor estimator710to the quantizer720.FIG.7shows that the scale factor SF is input to the quantizer720. However, this is a comprehensive notation, and after the calculation of the predictive scale factor is completed, as described above, the prediction scale factor stored in the register711may be input to the quantizer720. That is, while first sample parameters are input, the sample scale factor corresponding to each of the first sample parameters may be sequentially input to the quantizer720through a series of calculations of the scale factor estimator710. After all of sample parameters are input, the prediction scale factor may be calculated and stored in the register711and may be input to the quantizer720.

The quantizer720may multiply the prediction scale factor by each of the input data O_1, O_2, . . . , and O_N, through multipliers MPR. Subsequently, the quantizer720may round the multiplied value and may output finally quantized output data Q_1, Q_2, . . . , and Q_N through a clip calculation. A round calculation unit in the quantizer720may indicate the output data in the form of integer numbers.

Calculation of the clip calculation unit may be expressed as [Equation 9].

In [Equation 9], x is an input data value, 1 is the minimum value of quantized data to be expressed, and u is the maximum value of quantized data to be expressed. The clip calculation unit in the quantizer720may adjust a data value such that the output data may be between the maximum value and the minimum value.

According to the above-described embodiment, a computation for estimating scale factors of all of inputs in a quantization operation during deep learning inference may be omitted and accordingly, the total amount of computations may be reduced. In addition, computing resources for implementing an artificial neural network based on the quantized artificial neural network having high accuracy may be reduced, and the application range of the artificial neural network may be extended.

Herein, the scale factor estimator710and the quantizer710may each be analog and/or digital circuits including one or more of a logic gate, an integrated circuit, a microprocessor, a microcontroller, a memory circuit, a passive electronic component, an active electronic component, an optical component, and the like, and may be configured to execute software and/or firmware to perform the corresponding functions or operations described above.

FIG.8is a block diagram illustrating a computing system2000according to an example embodiment.

In some embodiments, the quantization system100ofFIG.1and/or the quantization apparatus500ofFIG.5may be implemented in the computing system2000ofFIG.8. As shown inFIG.8, the computing system2000may include a system memory2100, a processor2300, a storage2500, input/output devices2700, and communication connections2900. Components included in the computing system2000may be communicatively connected to one another via a bus, for example.

The system memory2100may include a program2120. The program2120may allow the processor2300to perform quantization of an artificial neural network according to example embodiments. For example, the program2120may include a plurality of commands executable by the processor2300, and the plurality of commands included in the program2120may be executed by the processor2300such that quantization of the artificial neural network may be performed. The system memory2100that is a non-limiting example, may include a volatile memory such as a Static Random Access Memory (SRAM) and a Dynamic Random Access Memory (DRAM), or a nonvolatile memory such as a flash memory, etc.

The processor2300may include at least one core for executing arbitrary command sets (e.g., Intel Architecture-32 (IA-32), 64-bit extension IA-32, x86-64, PowerPC, Sparc, MIPS, ARM, IA-64, etc.). The processor2300may execute the commands stored in the system memory2100and may execute the program2120such that quantization of the artificial neural network may be performed.

The storage2500may not lose the stored data even if power supplied to the computing system2000is blocked. For example, the storage2500may include a non-volatile memory such as an Electrically Erasable Programmable Read-Only Memory (EEPROM), a flash memory, a Phase Change Random Access Memory (PRAM), a Resistance Random Access Memory (RRAM), a Nano Floating Gate Memory (NFGM), a Polymer Random Access Memory (PoRAM), a Magnetic Random Access Memory (MRAM), a Ferroelectric Random Access Memory (FRAM), or a storage medium such as a magnetic tape, an optical disc, a magnetic disc, or the like. In some embodiments, the storage2500may be detachable from the computing system2000.

In some embodiments, the storage2500may store the program2120for quantization of the artificial neural network according to an example embodiment, and before the program2120is executed by the processor2300, the program2120or at least a part thereof may be loaded into the system memory2100from the storage2500. In some embodiments, the storage2500may store files written in program languages, and the program2120generated by a compiler or the like or at least a part thereof may be loaded into the system memory2100from the files.

In some embodiments, the storage2500may store data to be processed by the processor2300and/or data processed by the processor2300. For example, the storage2500may store quantized activations or data according to the quantization method described above, may store the quantized output data set OUT ofFIG.1, or may store a bias or data generated in quantization.

The input/output devices2700may include an input device such as a keyboard, a pointing device, or the like and may include an output device such as a display device, a printer, or the like. For example, the user may trigger execution of the program2120by the processor2300, may input the input data ofFIG.2(e.g., I1, I2), or may check the quantized output data set OUT ofFIG.1and/or an error message through the input/output devices2700.

The communication connections2900may provide access about a network outside the computing system2000. For example, the network may include a plurality of computing systems and communication links, and the communication links may include wired links, optical links, wireless links or links having other arbitrary formats.

FIG.9is a block diagram illustrating a portable computing system according to an example embodiment.

In some embodiments, the quantized artificial neural network according to an example embodiment may be implemented in a portable computing device3000. The portable computing device3000that is a non-limiting example may be any portable electronic device that supplies power through battery or self-power, such as mobile phones, tablet personal computers (PCs), wearable devices, and Internet of Things (IoT) devices.

As shown inFIG.9, the portable computing device3000may include a memory subsystem3100, input/output devices3300, a processing unit3500, and a network interface3700. The memory subsystem3100, the input/output devices3300, the processing unit3500, and the network interface3700may communicate with one another via a bus3900. In some embodiments, at least two of the memory subsystem3100, the input/output devices3300, and the processing unit3500, and the network interface3700may be included in one package as a System-on-a-Chip (SoC).

The memory subsystem3100may include a random access memory (RAM)3120and a storage3140. The RAM3120and/or the storage3140may store commands to be executed by the processing unit3500and data to be processed by the processing unit3500. For example, the RAM3120and/or the storage3140may store variables such as signals, weights, biases of the artificial neural network and may also store parameters of an artificial neuron (or a calculation node) of the artificial neural network. In some embodiments, the storage3140may include non-volatile memory.

The processing unit3500may include a CPU3520, a GPU3540, a Digital Signal Processor (DSP)3560, and an NPU3580. Unlike inFIG.9, in some embodiments, the processing unit3500may include only at least part of the CPU3520, the GPU3540, the DSP3560, and the NPU3580.

The CPU3520may control an overall operation of the portable computing device3000, and perform a specific work itself, or may direct other components of the processing unit3500to perform the specific work in response to, for example, an external input received through the input/output devices3300. The GPU3540may generate data for an image to be output through a display apparatus included in the input/output devices3300or may encode data received from a camera included in the input/output devices3300. The DSP3560may process digital signals, for example, digital signals provided from the network interface3700such that valid data may be generated.

The NPU3580that is executive hardware for the artificial neural network may include a plurality of calculation nodes corresponding to at least part of artificial neurons that constitute the artificial neural network, and at least part of the plurality of calculation nodes may process signals in a parallel manner. According to an example embodiment, the quantized artificial neural network, such as a deep neural network, has a high accuracy as well as low computational complexity and thus may be easily implemented in the portable computing device3000ofFIG.9, and the quantized artificial neural network has fast processing speed and may be implemented by the NPU3580that is simple and small, for example.

The input/output devices3300may include input devices such as a touch input device, a sound input device, a camera, and the like, and output devices such as a display device, a sound output device, and the like. The network interface3700may provide access about a mobile communication network such as Long Term Evolution (LTE), 5th Generation (5G), and the like, to the portable computing device3000, and may also provide access about a local network such as Wireless Fidelity (WiFi).