Neural network device and method of quantizing parameters of neural network

A neural network device includes a quantization parameter calculator configured to quantize parameters of a neural network that is pre-trained, so that the quantized parameters are of mixed data types, analyze a statistical distribution of parameter values of an M-bit floating-point type, the parameter values being associated with at least one layer of the neural network, M being a natural number greater than three, obtain a quantization level of each of the parameters statistically covering a distribution range of the parameter values, based on the analyzed statistical distribution, and quantize input data and weights of the M-bit floating-point type into asymmetric input data of an N-bit fixed-point type and weights of an N-bit floating-point type, using quantization parameters that are obtained based on the obtained quantization level of each of the parameters, N being a natural number greater than one and less than M.

CROSS-REFERENCE TO THE RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119 to Korean Patent Application No. 10-2019-0080543, filed on Jul. 4, 2019, in the Korean Intellectual Property Office (KIPO), the disclosure of which is incorporated by reference herein in its entirety.

BACKGROUND

The disclosure relates to a neural network, and more particularly to a neural network device and a method of quantizing parameters of a neural network, which use mixed-type data.

2. Description of the Related Art

A neural network refers to a computational architecture that models a biological brain. Recently, with the development of neural network technology, various kinds of electronic systems have been actively studied for analyzing input data and extracting valid information, using a neural network device. The neural network device may use a large amount of computations for complex input data. In order for the neural network device to analyze high-quality input in real time and extract information, technology capable of efficiently processing neural network operations may be used.

SUMMARY

According to embodiments, a neural network device includes a quantization parameter calculator configured to quantize parameters of a neural network that is pre-trained, so that the quantized parameters are of mixed data types, and a processor configured to apply the quantized parameters to the neural network. The quantization parameter calculator is further configured to analyze a statistical distribution of parameter values of an M-bit floating-point type, the parameter values being associated with at least one layer of the neural network, M being a natural number greater than three, obtain a quantization level of each of the parameters statistically covering a distribution range of the parameter values, based on the analyzed statistical distribution, and quantize input data and weights of the M-bit floating-point type into asymmetric input data of an N-bit fixed-point type and weights of an N-bit floating-point type, using quantization parameters that are obtained based on the obtained quantization level of each of the parameters, N being a natural number greater than one and less than M.

According to embodiments, a neural network device includes a quantization parameter calculator configured to quantize parameters of a neural network that are pre-trained, so that the quantized parameters are of mixed data types, and a processor configured to apply the quantized parameters to the neural network. The quantization parameter calculator includes a logic circuit configured to quantize input data and weights of an M-bit floating-point type into asymmetric input data of an N-bit fixed-point type and weights of an N-bit floating-point type, using quantization parameters, a control circuit configured to control the logic circuit, an input register and an weight register configured to store the quantization parameters, and a result register configured to store the asymmetric input data of the N-bit fixed-point type and the weights of the N-bit floating-point type.

According to embodiments, a method of quantizing parameters of a neural network, includes analyzing a statistical distribution of parameter values of an M-bit floating-point type, the parameter values being associated with at least one layer of the neural network, M being a natural number greater than three. The method further includes obtaining a quantization level of each of the parameters statistically covering a distribution range of the parameter values, based on the analyzed statistical distribution, and quantizing input data and weights of the M-bit floating-point type into asymmetric input data of an N-bit fixed-point type and weights of an N-bit floating-point type, using quantization parameters that are obtained based on the obtained quantization level of each of the parameters, N being a natural number greater than one and less than M. The method further includes applying, to the neural network, the asymmetric input data of the N-bit fixed-point type and the weights of the N-bit floating-point type.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiments provide a neural network device capable of increasing a a dynamic range and an accuracy.

Embodiments provide a method of quantizing parameters of a neural network, capable of increasing a dynamic range and an accuracy.

In detail, a quantization parameter calculator quantizes an M-bit floating-point type of input data and weights into N-bit fixed-point type of asymmetric input data and N-bit floating-point type of weights, respectively, based on a quantization level that minimizes quantization error in advance. A processor applies the quantized input data and the quantized weights to a neural network. Therefore, a neural network device may increase a dynamic range and an accuracy, and may increase a computing speed and reduce a power consumption by separating quantization and application of the quantized data to the neural network.

The embodiments will be described more fully hereinafter with reference to the accompanying drawings. In the drawings, like numerals refer to like elements throughout. The repeated descriptions may be omitted.

FIG.1is a flowchart illustrating a method of quantizing parameters of a neural network, according to embodiments.

Referring toFIG.1, in operation S100, a statistical distribution of M-bit floating-point type of parameter values is analyzed. The parameter values are associated with at least one layer of the neural network from data of a pre-trained floating-point neural network, and M is a natural number greater than three. A neural network is one of machine learning scheme, which is an operation model of software and/or hardware emulating a biological system using many artificial neurons connected through connection lines. The neural network uses the artificial neurons having simplified functions of the biological neurons and the artificial neurons are connected through connection lines of predetermined connection strengths to perform recognition or learning of human being. Recently deep learning is being studied to overcome limitation of the neural network. The deep learning neural network structure will be described below with reference toFIGS.3A and3B.

In operation S200, a quantization level of each of the parameters statistically covering a distribution range of the floating-point parameter values is determined based on the statistical distribution. The determination of the quantization level will be described below with reference toFIGS.9through11.

In operation S300, the M-bit floating-point type of input data and weights are quantized into N-bit fixed-point type of asymmetric input data and N-bit floating-point type of weights, respectively, using quantization parameters generated based on the quantization level. N is a natural number greater than one and smaller than M.

In operation S400, the quantized N-bit fixed-point type of asymmetric input data and the quantized N-bit floating-point type of weight are applied to the neural network, and a quantized neural network is generated.

In the conventional scheme for quantizing floating-point type of input data and weights, it is difficult to increasing both of accuracy and dynamic range. For increasing the accuracy, the dynamic range is degraded and for increasing the dynamic range, the accuracy is degraded. However, according to examples, the M-bit floating-point type of input data and weights are quantized into N-bit fixed-point type of asymmetric input data and N-bit floating-point type of weights, respectively, in advance, and the quantized N-bit fixed-point type of asymmetric input data and the quantized N-bit floating-point type of weight are applied to the neural network to generate the quantized neural network. Therefore, it is possible to increasing both of the accuracy and the dynamic range.

FIG.2is a block diagram illustrating a neural network device according to embodiments.

Referring toFIG.2, a neural network device100may include a plurality of processors110, a task manager TKMNG120, a quantization parameter calculator QPCL130and a memory MEM160.

The neural network device100may be driven by the plurality of processors110. For example, the plurality of processors110may include heterogeneous processors as illustrated inFIG.2. According to examples, the plurality of processors110may include at least two homogeneous processors. Various services (e.g., a task TK or an application) such as an image classify service, a user authentication service, an advanced driver assistance system (ADAS) service, and/or a voice assistant service may be executed and processed by the plurality of processors110. The task TK may include any one or any combination of a plurality of operations or arithmetic operations. For example, the task TK may represent applications such as an image classification service, a user authentication service based on biological information, ADAS service, a voice assistant service, etc. For example, the plurality of operations may include various operations such as a convolution operation, a rectified linear unit (RELU) operation, etc.

According to an embodiment, the quantization parameter calculator QPCL130and at least one of the plurality processors110may be, respectively, embodied as various numbers of hardware, software and/or firmware structures that execute respective functions described herein. For example, they may use a direct circuit structure, such as a memory, a processor, a logic circuit, a look-up table, etc. that may execute the respective functions through controls of one or more microprocessors or other control apparatuses. Also, they may be specifically embodied by a module, a program, or a part of code, which contains one or more executable instructions for performing specified logic functions, and executed by one or more microprocessors or other control apparatuses. Further, they may include or may be implemented by a microprocessor such as a central processing unit (CPU) that performs the respective functions. They may be combined into one single component which performs all operations or functions to be respectively performed by them.

According to an embodiment, the plurality of processors110may include a central processing unit (CPU)111, a graphic processing unit (GPU)112, a neural processing unit (NPU)113, a digital signal processor (DSP)114, an image signal processor (ISP)115and a dedicated hardware (DHW)116.

For example, the dedicated hardware116may include a vision processing unit (VPU), a vision intellectual property (VIP), etc. Each processor may be referred to as a processing element (PE).

AlthoughFIG.2illustrates only computing resources as examples of the plurality of processors110, the plurality of processors110may further include communication resources such as a direct memory access unit (DMA) for controlling access to the memory160, a connectivity for supporting various internal and/or external communications, or the like.

The task manager120receives the task TK that is to be performed from an external device or by a user, manages or schedules the received task TK, and assigns the task TK to the plurality of processors110. For example, the task manager120may assign operations included in the task TK to the plurality of processors110, and generate path information that indicates a computing path for the task TK. The computing path for the task TK may include a sequence of the operations included in the task TK and a driving sequence of processors for performing the operations included in the task TK.

The quantization parameter calculator130may analyze a statistical distribution of M-bit floating-point type of parameter values associated with at least one layer of the neural network from data of a pre-trained floating-point neural network, may determine a quantization level of each of the parameters statistically covering a distribution range of the floating-point parameter values based on the statistical distribution and quantize the M-bit floating-point type of input data and weights into N-bit fixed-point type of asymmetric input data and N-bit floating-point type of weights, respectively, using quantization parameters generated based on the quantization level, and may provide the processors110with a mixed type of quantized parameters MXPR including the quantized N-bit fixed-point type of asymmetric input data and the quantized N-bit floating-point type of weights. That is, the quantization parameter calculator130may perform operations S100, S200and S300inFIG.1.

Any one or any combination of the processors110may apply the quantized N-bit fixed-point type of asymmetric input data and the quantized N-bit floating-point type of weights to the neural network. That is, any one or any combination of the processors110may perform operation S400inFIG.1.

The memory160may store various data that are processed by the neural network device100. In examples, the memory160may include at least one volatile memory such as a dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), a static random access memory (SRAM), etc., and/or at least one nonvolatile memory such as an electrically erasable programmable read-only memory (EEPROM), a flash memory, a phase change random access memory (PRAM), a resistance random access memory (RRAM), a magnetic random access memory (MRAM), a ferroelectric random access memory (FRAM), a nano floating gate memory (NFGM), or a polymer random access memory (PoRAM), etc.

AlthoughFIG.2illustrates only data/signal transmission flows between some elements in the neural network device100, all elements in the neural network device100may be connected to one another via at least one bus, and thus all elements in the neural network device100may be communicate with one another via the at least one bus.

The neural network device100may further include software elements, e.g., a framework, a kernel or a device driver, a middleware, an application programming interface (API), an application program or an application, or the like. At least a portion of the software elements may be referred to as an operating system (OS).

FIGS.3and4are diagrams for describing examples of a deep learning neural network structure that is driven by a neural network device, according to embodiments.

Referring toFIG.3, a general neural network may include an input layer IL, a plurality of hidden layers HL1, HL2, . . . , HLn and an output layer OL.

The input layer IL may include i input nodes x1, x2, . . . , xi, where i is a natural number. Input data (e.g., vector input data) IDAT whose length is i may be input to the input nodes x1, x2, . . . , xi such that each element of the input data IDAT is input to a respective one of the input nodes x1, x2, . . . , xi.

The output layer OL may include j output nodes y1, y2, . . . , yj, where j is a natural number. Each of the output nodes y1, y2, . . . , yjmay correspond to a respective one of classes to be categorized. The output layer OL may output the output values (e.g., class scores or simply scores) associated with the input data IDAT for each of the classes. The output layer OL may be referred to as a fully-connected layer and may indicate, for example, a probability that the input data IDAT corresponds to a car.

A structure of the neural network illustrated inFIG.3Amay be represented by information on branches (or connections) between nodes illustrated as lines, and a weighted value assigned to each branch. Nodes within one layer may not be connected to one another, but nodes of different layers may be fully or partially connected to one another.

Each node (e.g., the node h11) may receive an output of a previous node (e.g., the node x1), may perform a computing operation, computation or calculation on the received output, and may output a result of the computing operation, computation or calculation as an output to a next node (e.g., the node h21). Each node may calculate a value to be output by applying the input to a function, e.g., a nonlinear function.

Generally, the structure of the neural network is set in advance, and the weighted values for the connections between the nodes are set appropriately using data having an already known answer of which class the data belongs to. The data with the already known answer is referred to as “training data,” and a process of determining the weighted value is referred to as “training.” The neural network “learns” during the training process. A group of an independently trainable structure and the weighted value is referred to as a “model,” and a process of predicting, by the model with the determined weighted value, which class the input data belongs to, and then outputting the predicted value, is referred to as a “testing” process.

The general neural network illustrated inFIG.3Amay not be suitable for handling input image data (or input sound data) because each node (e.g., the node h11) is connected to all nodes of a previous layer (e.g., the nodes x1, x2, . . . , xiincluded in the layer IL) and then the number of weighted values drastically increases as the size of the input image data increases. Thus, a convolutional neural network, which is implemented by combining the filtering technique with the general neural network, has been researched such that two-dimensional image (e.g., the input image data) is efficiently trained by the convolutional neural network.

Unlike the general neural network, each layer of the convolutional neural network may have three dimensions of width, height and depth, and thus data that is input to each layer may be volume data having three dimensions of width, height and depth. For example, if an input image inFIG.3Bhas a size of 32 widths (e.g., 32 pixels) and 32 heights and three color channels R, G and B, input data IDAT corresponding to the input image may have a size of 32*32*3. The input data IDAT inFIG.3Bmay be referred to as input volume data or input activation volume.

Each of convolutional layers CONV1, CONV2, CONV3, CONV4, CONV5 and CONV6 may perform a convolutional operation on input volume data. In an image processing, the convolutional operation represents an operation in which image data is processed based on a mask with weighted values and an output value is obtained by multiplying input values by the weighted values and adding up the total multiplied values. The mask may be referred to as a filter, window or kernel.

Parameters of each convolutional layer may consist of a set of learnable filters. Every filter may be small spatially (along width and height), but may extend through the full depth of an input volume. For example, during the forward pass, each filter may be slid (more precisely, convolved) across the width and height of the input volume, and dot products may be computed between the entries of the filter and the input at any position. As the filter is slid over the width and height of the input volume, a two-dimensional activation map that gives the responses of that filter at every spatial position may be generated. As a result, an output volume may be generated by stacking these activation maps along the depth dimension. For example, if input volume data having a size of 32*32*3 passes through the convolutional layer CONV1 having four filters with zero-padding, output volume data of the convolutional layer CONV1 may have a size of 32*32*12 (e.g., a depth of volume data increases).

Each of RELU layers RELU1, RELU2, RELU3, RELU4, RELU5 and RELU6 may perform a rectified linear unit operation that corresponds to an activation function defined by, e.g., a function f(x)=max(0, x) (e.g., an output is zero for all negative input x). For example, if input volume data having a size of 32*32*12 passes through the RELU layer RELU1 to perform the rectified linear unit operation, output volume data of the RELU layer RELU1 may have a size of 32*32*12 (e.g., a size of volume data is maintained).

Each of pooling layers POOL1, POOL2 and POOL3 may perform a down-sampling operation on input volume data along spatial dimensions of width and height. For example, four input values arranged in a 2*2 matrix formation may be converted into one output value based on a 2*2 filter. For example, a maximum value of four input values arranged in a 2*2 matrix formation may be selected based on 2*2 maximum pooling, or an average value of four input values arranged in a 2*2 matrix formation may be obtained based on 2*2 average pooling. For example, if input volume data having a size of 32*32*12 passes through the pooling layer POOL1 having a 2*2 filter, output volume data of the pooling layer POOL1 may have a size of 16*16*12 (e.g., width and height of volume data decreases, and a depth of volume data is maintained).

One convolutional layer (e.g., CONV1) and one RELU layer (e.g., RELU1) may form a pair of CONV/RELU layers in the convolutional neural network, pairs of the CONV/RELU layers may be repeatedly arranged in the convolutional neural network, and the pooling layer may be periodically inserted in the convolutional neural network, thereby reducing a spatial size of image and extracting a characteristic of image.

An output layer or a fully-connected layer FC may output results (e.g., class scores) of the input volume data IDAT for each of the classes. For example, the input volume data IDAT corresponding to the two-dimensional image may be converted into an one-dimensional matrix or vector as the convolutional operation and the down-sampling operation are repeated. For example, the fully-connected layer FC may represent probabilities that the input volume data IDAT corresponds to a car, a truck, an airplane, a ship and a horse.

The types and number of layers included in the convolutional neural network may not be limited to an example described with reference toFIG.3Band may be changed. In addition, the convolutional neural network may further include other layers such as a softmax layer for converting score values corresponding to predicted results into probability values, a bias adding layer for adding at least one bias, or the like.

As such, the deep learning neural network may include a plurality of layers, and the fixed-point format may be determined independently with respect to each of the plurality of layers. The nodes in each layer may perform the quantization based on the same fixed-point format. When the ANN includes a plurality of layers, the virtual overflow detection circuit140may generate a plurality of virtual overflow information corresponding to the plurality of layers, respectively, and the quantization parameter calculator130may determine the (k+1)-th fixed-point format for the next quantization with respect to each of the plurality of layers based on the plurality of virtual overflow information.

FIG.5Ais a block diagram illustrating an example of a quantization parameter calculator inFIG.2, according to embodiments.

Referring toFIG.5A, the quantization parameter calculator130may include a control circuit CTRL131, a logic circuit LOG132, an embedded memory eMEM133, an input register REGI134, a weight register REGW135and a result register REGR136.

The control circuit131controls overall operations of the quantization parameter calculator130. The control circuit131may control flows of instructions and data for quantizing parameters of the neural network.

The embedded memory133may store the instructions and the data of the quantization parameter calculator130. Input data and weight values to be quantized may be stored in the input register134and the weight register135, respectively.

The logic circuit132may analyze a statistical distribution of M-bit floating-point type of parameter values associated with at least one layer of the neural network from data of a pre-trained floating-point neural network, may determine a quantization level of each of the parameters statistically covering a distribution range of the floating-point parameter values based on the statistical distribution and quantize the M-bit floating-point type of input data and weights into N-bit fixed-point type of asymmetric input data and N-bit floating-point type of weights, respectively, using quantization parameters generated based on the quantization level, based on the input data and the weights stored in the input register134and the weight register135.

The logic circuit132may store quantization parameters in the input register134and the weight register135and may store the quantized N-bit fixed-point type of asymmetric input data and the quantized N-bit floating-point type of weights in the result register136. The logic circuit132may be connected to the input register134and the weight register135and may be also connected to the result register136.

The control circuit131may control the result register136to provide the quantized N-bit fixed-point type of asymmetric input data and the quantized N-bit floating-point type of weights to one of the processors110.

The input register134and the weight register135may be implemented with a memory that has operating speed faster than the embedded memory133, and the input data and the weights may be loaded to the input register134and the weight register135from the embedded memory133.

FIG.5Bis a block diagram illustrating one of processors inFIG.2, according to embodiments.

FIG.5Billustrates an example of the neural processing unit113and other processors may have a similar or same configuration of the neural processing unit113.

Referring toFIG.5B, the neural processing unit (or, a processor)113may include a control circuit171, a logic circuit172, an embedded memory173and a register174.

The control circuit171controls overall operations of the processor113. The control circuit171may control flows of instructions and data for quantizing parameters of the neural network.

The embedded memory173may store the instructions and the data of the processor113. The register174may store the quantized N-bit fixed-point type of asymmetric input data and the quantized N-bit floating-point type of weights provided from the result register136.

The logic circuit172may be connected to the register174and may apply the quantized N-bit fixed-point type of asymmetric input data and the quantized N-bit floating-point type of weights to the neural network to generate a quantized neural network QNN.

FIG.6is a block diagram of a hardware configuration of a neural network device according to embodiments.

Referring toFIG.6, the neural network device100may include the processor113, the quantization parameter calculator130and the memory160.

The neural network device100may correspond to a computing device having various processing functions such as generating a neural network, training (or learning) a neural network, quantizing a floating-point type neural network into a mixed-data type neural network, which includes fixed-point type data and floating-point type, or retraining a neural network.

The processor113may repeatedly train (learn) a given initial neural network to generate a pre-trained neural network161. The initial neural network may have parameters of a floating-point type, for example, parameters of 32-bit floating-point precision, to secure processing accuracy of the neural network. The floating-point type parameters may include, for example, various types of data that are input/output to/from a neural network, such as input/output activations, weights, and biases of a neural network. As the repetitive training of the neural network progresses, the floating-point type parameters of the neural network may be tuned to compute a more accurate output for a given input.

A floating point may use a relatively large amount of computations and a high memory access frequency compared to a fixed point. Therefore, in a mobile device such as a smart phone, a tablet PC, a wearable device or the like and an embedded device having relatively low processing performance, the processing of a neural network having floating-point type parameters may not be smooth. Thus, to drive a neural network with acceptable accuracy loss while sufficiently reducing the amount of computations in such devices, the floating-point type parameters that are processed in the neural network may be quantized.

The neural network device100may perform quantization163to convert parameters of a trained neural network into predetermined bits of a fixed-point type and a floating-point type taking into consideration the processing performance of a device (e.g., a mobile device, an embedded device, etc.) in which a neural network is to be deployed. In addition, the neural network device100may transmit the quantized neural network QNN to the device to be deployed. The device in which a neural network is to be deployed may be, for example, an autonomous vehicle that performs speech recognition and image recognition using a neural network, a robot, a smart phone, a tablet device, an augmented reality (AR) device, an Internet of Things (IOT) device, or the like.

The quantization parameter calculator130may obtain data of the pre-trained neural network161, using floating points stored in the memory160. The pre-trained neural network data may be data repeatedly trained with floating-point type parameters. Neural network training may be repeatedly performed by receiving training set data as an input, and then repeatedly performed with test set data again, but is not limited thereto. The training set data is input data for training a neural network, and the test set data is input data that does not overlap the training set data and may train the neural network trained with the training set data while measuring performance thereof.

The quantization parameter calculator130may analyze a statistical distribution of M-bit floating-point type of parameter values associated with at least one layer of the neural network from data of the pre-trained neural network161. The quantization parameter calculator130may analyze the statistical distribution by obtaining statistics for each channel of floating-point parameter values of weights, input activations, and output activations used in each layer during the pre-training of the neural network.

The quantization parameter calculator130may determine a quantization level of each of the parameters statistically covering a distribution range of the floating-point parameter values based on the statistical distribution, and quantize the M-bit floating-point type of input data and weights into N-bit fixed-point type of asymmetric input data and N-bit floating-point type of weights (163), respectively, using quantization parameters generated based on the quantization level. The quantization parameter calculator130may provide the processor113with the mixed-type parameters MXPR including the quantized N-bit fixed-point type of asymmetric input data and the quantized N-bit floating-point type of weights.

The processor113may apply the quantized N-bit fixed-point type of asymmetric input data and the quantized N-bit floating-point type of weights to the pre-trained neural network161to generate the quantized neural network QNN.

The memory160may store neural network-related data sets that have been processed or are to be processed by the processor113and/or the quantization parameter calculator130, for example, data of an untrained initial neural network, data of a neural network generated in a training process, data of a neural network for which training has been completed, and data of a quantized neural network. In addition, the memory160may store various programs related to training algorithms and quantization algorithms of a neural network to be executed by the processor113and/or the quantization parameter calculator130.

FIG.7is a diagram illustrating examples of a floating-point value and fixed-point values.

Referring toFIG.7, a floating-point value510is expressed as “a×2b”, in which “a” is a fractional part and “b” is an exponent part. The floating-point value510is expressed by 32 bits including a 1-bit sign part, an 8-bit exponent part, and a 23-bit fractional part.

Furthermore, fixed-point values520are expressed by “Qm.n”, where m and n are natural numbers. In the expression “Qm.n”, “m” denotes the number of bits indicating the exponent part, and “n” denotes the number of bits indicating the fractional part. Accordingly, a bit width of a fixed-point value is (1+m+n) obtained by summing a 1-bit sign part, an m-bit exponent part, and an n-bit fractional part. Because bits of the fixed-point bits indicating the fractional part are n bits, a fractional length is n. For example, “Q3.4” is a total 8-bit fixed-point value including a 1-bit sign part, a 3-bit exponent part, and a 4-bit fractional part, “Q1.30” is a total 32-bit fixed-point value including a 1-bit sign part, a 1-bit exponent part, and a 30-bit fractional part, and “Q15.16” is a total 32-bit fixed-point value including a 1-bit sign part, a 15-bit exponent part, and a 16-bit fractional part.

FIG.8is a diagram illustrating an example of a relationship between a fractional length and an accuracy of a fixed-point value.

Referring toFIG.8, assuming that the total bit width allotted to a fixed-point value is 3 bits, a fixed-point expression530of Q2.0 in which the fractional length is 0 and a fixed-point expression540of Q1.1 in which the fractional length is 1 are compared to each other.

For Q2.0, because the exponent part is 2 bits and the fractional part is 0 bits, fixed-point values from −4 to 3 may be expressed, and an interval between the possible fixed-point values is 1. For Q1.1, because the exponent part is 1 bit and the fractional part is 1 bit, fixed-point values from −2 to 1.5 may be expressed, and an interval between the possible fixed-point values is 0.5.

As can be seen from the comparison, although 3 bits are allotted to both the fixed-point expression530of Q2.0 and the fixed-point expression540of Q1.1, Q2.0 is able to express a wider range of fixed-point values than Q1.1, but has a lower accuracy because the interval between the fixed-point values is wider. On the other hand, Q1.1 is able to express a narrower range of fixed-point values than Q2.0, but has a higher accuracy because the interval between the fixed-point values is narrower. Consequently, it may be seen that the accuracy of a fixed-point value depends on the fractional length of the fixed-point value, that is, the number of fractional bits allotted to the fixed-point value.

The neural network device100may increase accuracy and dynamic range because the neural network device100quantizes M-bit floating-point type of input data into N-bit fixed-point type of input data and the quantized fixed-point type of input data is asymmetric.

FIG.9is a graph showing an example of a statistical distribution of parameters used in a layer.

Referring toFIG.9, after the repeated training of a neural network having floating-point parameters, a distribution of intrinsic floating-point values, that is, parameter values, is generated for a layer. The quantization parameter calculator130ofFIG.5Aanalyzes, from pre-trained neural network data, a statistical distribution for at least one layer based on statistics of floating-point parameter values for each channel. In detail, the quantization parameter calculator130obtains, from the pre-trained neural network data, statistics for floating-point parameter values of weights, input activations, and output activations, and normalizes the statistics for a layer with a probability density function (PDF) of a normal (Gaussian) distribution550. However, althoughFIG.9illustrates an example in which, for convenience of explanation, the quantization parameter calculator130normalizes the statistics with the PDF of the normal distribution550, the quantization parameter calculator130is not limited thereto. In other words, the quantization parameter calculator130may analyze the statistics by using various types of statistical distributions or probability distributions other than the normal distribution550. The normal distribution550ofFIG.9may be a distribution of floating-point activation values in a layer or a distribution of floating-point weight values in a layer.

The quantization parameter calculator130may determine a quantization level based on the normal distribution550so that parameters of the layer are quantized to a point expression having a fractional length. In detail, the quantization parameter calculator130obtains a statistical maximum value Xmax and a statistical minimum value Xmin of parameters in the normal distribution550, and determines a quantization level K capable of statistically covering floating-point values in a range between the statistical maximum value Xmax and the statistical minimum value Xmin.

The statistical maximum value Xmax and the statistical minimum value Xmin are values based on a range in which a quantization error is minimized. The statistical maximum value Xmax and the statistical minimum value Xmin may be defined in various ways. For example, the statistical maximum value Xmax and the statistical minimum value Xmin may be an actual maximum parameter value and an actual minimum parameter value of parameters in the normal distribution550. Alternatively, the statistical maximum value Xmax and the statistical minimum value Xmin may be upper and lower limits obtained based on the mean, variance, or standard deviation σ1of the normal distribution550.

FIG.10is a graph showing another example of a statistical distribution of parameters used in a layer.

Referring toFIG.10, instead of the normal distribution550ofFIG.9, the quantization parameter calculator130normalizes the statistics for each layer with the PDF of a Laplace distribution560. The quantization parameter calculator130, as described with reference toFIG.9, determines a quantization level of parameters of a corresponding layer by obtaining the statistical maximum value Xmax and the statistical minimum value Xmin appropriate to the Laplace distribution560.

FIG.11is a graph showing another example of a statistical distribution of parameters used in a layer.

Referring toFIG.11, the quantization parameter calculator130normalizes the statistics for each layer with the PDF of a normal distribution570. The quantization parameter calculator130, as described with reference toFIG.9, determines a quantization level of parameters of a corresponding layer by obtaining a statistical maximum value Ymax and a statistical minimum value Ymin appropriate to the normal distribution570. The statistical maximum value Ymax and the statistical minimum value Ymin may be upper and lower limits obtained based on the mean, variance, or standard deviation σ2of the normal distribution570.

FIG.12is a diagram illustrating that the neural network device ofFIG.2quantizes input data and weights, according to embodiments.

InFIG.12, it is assumed that M is 32 and N is 8.

Referring toFIG.12, the quantization parameter calculator130may quantize M-bit floating-point type 32-BIT floating point (FLP) of input data IDTA and weights WDTA into N-bit fixed-point type 8-BIT asymmetric (ASM) fixed point (FXP) of asymmetric input data QIDTA and N-bit floating-point type 8-BIT FLP of weights QWDTA, respectively. The processor113may apply the quantized N-bit fixed-point type 8-BIT ASM FXP of asymmetric input data QIDTA and the quantized N-bit floating-point type 8-BIT FLP of weights QWDTA to the neural network to generate the quantized neural network QNN.

FIG.13Aillustrates quantized input data provided from a quantization parameter calculator.

Referring toFIG.13A, the quantization parameter calculator130may quantize the M-bit floating-point type of input data IDTA into the N-bit fixed-point type of asymmetric input data QIDTA. The M-bit floating-point type of input data IDTA may be represented as e×2f and the N-bit fixed-point type of asymmetric input data QIDTA may be represented as Qm.n.

A sum of a fractional part e and an exponent part f of the M-bit floating-point type of input data IDTA corresponds to M−1 and m+n of the N-bit fixed-point type of asymmetric input data QIDTA corresponds to N−1. Although, it is assumed that M is 32 and N is 8, values of M and M are not limited thereto. The quantized input data QIDTA is asymmetric because the statistical maximum value Xmax and the statistical minimum value Xmin are asymmetric with respect to each other inFIG.9. When the quantized input data QIDTA is represented asymmetrically, the input data IDTA may be more accurately represented and the accuracy may be enhanced.

FIG.13Billustrates quantized weights provided from a quantization parameter calculator.

Referring toFIG.13B, quantized weights Wnew may include a fractional part a′ and an exponent part b′ and may further include an exponent bias bse.

When the quantization parameter calculator130quantizes the M-bit floating-point type of weights into the N-bit floating-point type of weights, a sum of a number of bits in the fractional part a′ and a number of bits in the exponent part b′ corresponds to N−1.

Because the quantized weights Wnew includes the exponent bias bse, the weights WDAT may be represented with highly dynamic range.

The exponent bias bse may be differently applied to negative weights and positive weights before the negative weights and the positive weights are quantized.

FIG.14is a flowchart illustrating an operation of quantizing input data and weights inFIG.1, according to embodiments.

Referring toFIGS.5A, and6through14, for quantizing input data and weights (operation S300ofFIG.5A), in operation S310, the quantization parameter calculator130may determine a first step size, a second step size, a first zero point and a second zero point based on the quantization level, a maximum value the statistical distribution and a minimum value of the statistical distribution. The first step size is associated with the input data and corresponds to a gap between the quantization level, the second step size corresponds to a gap between a quantization level of an output data obtained by performing node operation of the input data and the weights, the first zero point indicates a zero value of the input data and the second zero point indicates a zero value of the output data.

In operation S320, the quantization parameter calculator130may determine a new weight based on the first step size and the second step size. In operation S330, the quantization parameter calculator130may determine a bias associated with the output data based on the new weight.

The quantization parameter calculator130determines the first step size Δx, which is associated with associated with the input data x and corresponds to a gap between the quantization level based on the statistical maximum value Xmax and the statistical minimum value Xmin by

Here, K corresponds to the quantization level.

The input data x may be quantized by a following Expression 1.
x=Δx(qx−zx),  [Expression 1]

Δ⁢y=Ymax-YminK
for analyzing statistics for a next layer.

The output data y may be quantized by a following Expression 2.
y=Δy(ay−zy)=Σc,k{wc,kxc,k}+bias  [Expression 2]

Here, c denotes c-th layer, k denotes k-th node in the c-th layer and bias denotes a bias associated with the node.

The quantization parameter calculator130generates a new weight Wnew based on the first step size Δx and the second step size Δy by a following Expression 3.

may be represented by a following Expression 4.

Here, if a new bias biasnewis represented by

bias-Δy⁢Σc,k⁢{wnewc,k⁢zx},a
following Expression 5 is deduced and the quantization parameter calculator130generates a quantized bias qbiasnew.

Here, round is a round off function to round off predetermined numbers. In Expression 5, a floor function instead of the round function may be used and the floor function is a function that takes as a input a real number and gives as an output the greatest integer less than or equal to the real number.

Therefore, the quantization parameter calculator130may generate the new bias biasnewassociated with the output data based on the bias, the new weights Wnew, the first zero point zxand the second step size Δy. In addition, a quantized output qymay be generated by a following Expression 6.

If the new weights is quantized by Expression 7, Expression 6 is deduced into Expression 8.
qw=−1sign×23bit exponent-bse×4 bit mantissa with implicit bit  [Expression 7]

The quantization parameter calculator130may stretch the weights before quantizing the weights for accomplishing higher accuracy. Stretching the weights is represented by a following Expression 9.

Here, Swcorresponds to a multiple of two, which is greater than a maximum Wnewmax of the new weight Wnew and is nearest to the maximum Wnewmax and Wstretched corresponds to stretched weights. The quantization parameter calculator130generates the quantized N-bit floating-point type of weights by using a multiple of two, which is greater than a maximum of the quantized N-bit floating-point type of weights and which is nearest to the weights.

FIG.15is a diagram illustrating an example of quantization operation of a quantization parameter calculator, according to embodiments.

Referring toFIG.15, the quantization parameter calculator130receives 32-bit floating-point type of input data in operation S612, collects samples from the input data in operation S614, and computes quantization steps based on statistical distribution of the collected samples in operation S616. The quantization parameter calculator130receives 32-bit floating-point type of weights in operation S618, computes new weights based on the quantization steps in operation S622, and computes a new bias in operation S624. The quantization parameter calculator130computes zero points based on the quantization steps in operation S626, computes a minimum value of a statistical distribution to represent a real zero point in operation S632, and computes quantized 8-bit fixed-point type of asymmetric input data in operation S634. The quantization parameter calculator130computes an exponent bias based on the new weights in operation S628, and computes quantized 8-bit floating-point type of weights in operation S636.

FIG.16is a diagram illustrating another example of quantization operation of a quantization parameter calculator, according to embodiments.

FIG.16differs fromFIG.15in that operation S627of stretching the weights is added between operations S622and S628, and detailed description will be omitted.

FIG.17is a diagram illustrating an example operation of a processor in the neural network device ofFIG.2, according to embodiments.

FIG.17illustrates an example operation of the processor113for one layer in the neural network device100.

Referring toFIG.17, one layer in the neural network device100may include a plurality of multipliers711˜71m(m is an integer greater than two), a plurality of arithmetic shifters721˜72mand a plurality of adders731˜73pand741(p is an integer greater than two).

The multiplier711performs multiplication on a quantized input data QIDTA1and a fractional part a′1of the quantized weights and provides a result of the multiplication to the arithmetic shifter721. The arithmetic shifter721shifts an output of the multiplier711by a sum of the exponent part b′1of the quantized weights and an exponent bias bse1to output a shifted result. The multiplier71mperforms multiplication on a quantized input data QIDTAm and a fractional part a′m of the quantized weights and provides a result of the multiplication to the arithmetic shifter72m. The arithmetic shifter72mshifts an output of the multiplier71mby a sum of the exponent part b′m of the quantized weights and an exponent bias bsem to output a shifted result. The adders731˜73pand741add outputs of the arithmetic shifters721˜72mand a bias to provide quantized output data QODTA. For increasing accuracy, the quantization parameter calculator130may quantize the bias into M-bit asymmetric fixed-point type.

FIG.18is a diagram illustrating a multiplier and an arithmetic shifter in the operation of the processor inFIG.17.

Referring toFIG.18, the multiplier711performs fixed-point multiplication on the quantized N-bit fixed-point type of input data QIDTA and a fractional part a′ of the quantized N-bit floating-point type of weights to provide fixed-point type FXP of output MOUT. The arithmetic shifter721shifts the output MOUT of the multiplier711by a sum of the exponent part b′ of the quantized N-bit floating-point type of weights to provide fixed-point type of output SOUT. The arithmetic shifter721further shifts the output MOUT of the multiplier711by an exponent bias bse of the weights.

FIG.19is a block diagram illustrating a neural network system according to embodiments.

Referring toFIG.19, an neural network system may include a first electronic device1101and a second electronic device1201. The deep learning system may be driven by the first electronic device1101and the second electronic device1201.

The first electronic device1101may include a plurality of heterogeneous processors1110, a task manager TKMNG1120, a quantization parameter calculator1130, and a memory MEM1160. The second electronic device1201may include a plurality of heterogeneous processors1210, a task manager TKMNG1220, a quantization parameter calculator QPCL1230, and a memory MEM1260. The pluralities of heterogeneous processors1110and1210may respectively include CPUs1111and1211, GPUs1112and1212, NPUs1113and1213, DSPs1114and1214, ISPs1115and1215, and dedicated hardwares1116and1216.

The pluralities of heterogeneous processors1110and1210, the task managers1120and1220, the quantization parameter calculator1130and1230, and the memories1160and1260inFIG.19may be substantially the same as described with reference toFIGS.1through18.

In examples, some of the heterogeneous processors (e.g.,1111,1112,1113,1114,1115and1116) may be included in the first electronic device1101, and the other heterogeneous processors (e.g.,1211,1212,1213,1214,1215and1216) may be included in the second electronic device1201. The first electronic device1101may be an electronic device that directly interacts with a user (e.g., directly controlled by a user). The second electronic device1201may be physically separated from the first electronic device1101, and may be interoperable with the first electronic device1101.

In examples, the first electronic device1101may be any computing device and/or mobile device, such as a personal computer (PC), a laptop computer, a mobile phone, a smart phone, a tablet computer, a personal digital assistants (PDA), a portable multimedia player (PMP), a digital camera, a camcorder, a music player, a video player, a portable game console, a navigation device, a wearable device, an internet of things (IOT) device, an internet of everythings (IoE) device, a virtual reality (VR) device, an augmented reality (AR) device, etc.

In examples, the second electronic device1201may be any computing device and/or mobile device that is interoperable with the first electronic device1101. For example, the second electronic device1201may be a companion device that depends on the first electronic device1101, such as a wearable device (e.g., a smart watch). Alternatively, the second electronic device1201may be an in-house server (e.g., a home gateway) that controls an IoT device and/or an IoE device, or an outside server (e.g., a cloud server).

The inventive concepts may be applied to various devices and systems that include the neural network and/or machine learning systems.

The foregoing is illustrative of the embodiments and is not to be construed as limiting thereof. Although the embodiments have been described, those skilled in the art will readily appreciate that many modifications are possible in the embodiments without materially departing from the inventive concepts.