METHOD AND MACHINE LEARNING SYSTEM TO PERFORM QUANTIZATION OF NEURAL NETWORK

The present disclosure relates to a system and method of performing quantization of a neural network having multiple layers. The method comprises receiving a floating-point dataset as input dataset and determining a first shift constant for first layer of the neural network based on the input dataset. The method also comprises performing quantization for the first layer using the determined shift constant of the first layer. The method further comprises determining a next shift constant for next layer of the neural network based on output of a layer previous to the next layer, and performing quantization for the next layer using the determined next shift constant. The method further comprises iterating the steps of determining shift constant and performing quantization for all layers of the neural network to generate fixed point dataset as output.

TECHNICAL FIELD

The present disclosure relates to neural networks in general and more particularly, to method and system to perform quantization of neural networks.

BACKGROUND

Deep neural networks have outperformed several existing machine learning models and have become the state-of-the-art in many fields, namely, computer vision, medical imaging, natural language processing and speech recognition, even rivalling up to human cognizance in some. The success of the deep neural networks is significantly attributed to the depth of the model, number of parameters in the model and the complexity of the model. Despite the success of these networks, the training and inference are exceptionally computation-intensive and memory-consuming, requiring lots of computational power and storage. Due to the high resource requirements, many deep learning or neural network tasks are mainly done in the cloud (most of the computations are either performed on GPUs or special hardware such as neuronal network accelerators).

Owing to the computation and power constraints, in many cases the deep learning or neural networks cannot be deployed in resource constrained settings. Therefore, it is difficult and demanding to embed these deep learning or neural networks in low resource devices for utilization in real world tasks. Thus, it is desirous to have a machine learning system and associated method that reduce the number of parameters in the neural network which would not only reduce memory consumption but also the computation time.

The information disclosed in this background of the disclosure section is only for enhancement of understanding of the general background of the disclosure and should not be taken as an acknowledgement or any form of suggestion that this information forms prior art already known to a person skilled in the art.

SUMMARY

Embodiments of the present disclosure relate to a method of performing quantization of a neural network having multiple layers. The method comprises receiving a floating-point dataset as input dataset and determining a shift constant for each layer of the neural network based on the input dataset. The method also comprises performing quantization for each layer using the determined shift constant to generate the corresponding fixed-point dataset as output dataset. The method further comprises iterating the steps of determining shift constant and performing quantization for all layers of the neural network.

Another aspect of the present disclosure relates to a system to perform quantization of a neural network having multiple layers. The apparatus comprises a processor, wherein the processor is configured to receive floating-point dataset as input dataset and determine a shift constant for each layer of the neural network based on the input dataset. The processor is also configured to perform quantization for each layer using the determined shift constant to generate the corresponding fixed-point dataset as output dataset and iterate the steps: determination of shift constant and performance of quantization for all layers of the neural network.

Yet another aspect of the present disclosure relates to a non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to receive floating-point dataset as input dataset and determine a shift constant for each layer of the neural network based on the input dataset. The one or more processors is configured to perform quantization for each layer using the determined shift constant to generate the corresponding fixed-point dataset as output dataset and iterate the steps: determination of shift constant and performance of quantization for all layers of the neural network.

The aforementioned aspects of the present disclosure may overcome one or more of the shortcomings of the prior art. Additional features and advantages may be realized through the techniques of the present disclosure. Other embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed disclosure.

DETAILED DESCRIPTION

FIG.1is an exemplary block diagram of a machine learning system (MLS)100to perform quantization of a neural network in accordance with some embodiments of the present disclosure.

The machine learning system100(hereinafter referred to as system100) is configured to perform quantization of neural network using one or more components of the system100. In one embodiment, the system100quantizes a floating-point dataset associated with input of the neural network to a corresponding fixed-point dataset output with improved accuracy, wherein dataset refers to plurality of images. The system100computes a real value known as shift constant and propagates the shift constant across different layers of the neural network to achieve minimum accuracy loss post quantization.

In an embodiment, the MLS100may include an I/O interface102, a processor104, a memory106, and one or more modules108. The I/O interface102may be configured to receive one or more inputs such as an original validation dataset from one or more external data sources. The processor104may be configured to perform one or more functions of the MLS100for performing quantization of a neural network. The memory106may be communicatively coupled to the processor106and may store data110and other related data.

In some embodiments, the MLS100may include the modules108for performing various operations in accordance with embodiments of the present disclosure. In an embodiment, the modules108may include, without limiting to, a receiving module122, a dataset reduction module124, a shift constant determination module126, and a quantization module128. The MLS100may also comprise other modules130to perform various miscellaneous functionalities of the MLS100. It will be appreciated that such aforementioned modules may be represented as a single module or a combination of different modules. The modules may be implemented in the form of software implemented by a processor, hardware and or firmware.

The data110may be stored within the memory106and may include, without limiting to, original validation dataset112, reduced dataset114, shift constant data116, and other data118. In some embodiments, the data110may be stored within the memory106in the form of various data structures. Additionally, the data110may be organized using data models, such as relational or hierarchical data models. The other data118may comprises other temporary data generated by other module130for performing various functions of the MLS100.

The MLS100further comprises I/O devices (not shown) coupled with the processor104. The I/O device is configured to receive inputs via the I/O interface102and transmit outputs for displaying in the I/O device via the I/O interface102. In one embodiment, the I/O interface may include the keypad, switch, digital pen as the input devices and speaker, LEDs, LCD as the output devices.

As used herein, the term module refers to an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality. In an embodiment, the other modules130may be used to perform various miscellaneous functionalities of the MLS100. It will be appreciated that such other modules130may be represented as a single module or a combination of different modules.

In operation, the system100performs quantization of a floating-point neural network to a fixed-point neural network using the modules108. In one implementation, the receiving module122is configured to receive the original validation dataset112from any external data source and transmit to the dataset reduction module124through the I/O interface102. In one example, the original validation dataset112comprises a plurality of test images or test dataset obtained from an external system.

The dataset reduction module124is configured to receive the original validation dataset and obtain the reduced dataset114for computation of parameters required for quantization of the neural network. The dataset reduction module124is configured to obtain the reduced dataset114, which is subset of the original validation dataset112for quantization calibration, so that the reduced dataset114have representational images covering all combinations of dynamic ranges present in the original validation dataset112. In order to obtain the reduced dataset112, the dataset reduction module124is configured to determine true positive images from the received original validation dataset112in case if the original validation dataset112have been classified as true and false positive images by respective models. The dataset reduction module124considers only true positive images as test images for computation of effective vector, as false positive images will not degrade the accuracy after quantization. In one embodiment, the dataset reduction module124is configured to compute the effective vector for each test image, wherein the effective vector is a max_min ratio_vector which will be explained in detail in the forthcoming paragraphs.

In order to compute the effective vector of each test image, the dataset reduction module124is configured to compute maximum and minimum value at an output of each layer of neural network, wherein the maximum and minimum value at the output of each layer of neural network is also referred to as ‘intermediate features’. For example, X_maxiLrepresents maximum value at the output of LthLayer of ithimage in the reduced dataset, and X_miniLrepresents minimum value at the output of LthLayer of ithimage in the reduced dataset.

After computing the maximum and minimum value of each layer of all the true positive images, the dataset reduction module124is configured to compute the global maximum value of each layer as shown in eq. (1) and compute the global minimum value of each layer as shown in eq. (2)

In one embodiment, the effective vector of each image is computed by computing ratio of the global maximum value of all test images for each layer of the neural network and maximum value of a particular test image for each layer, and ratio of global minimum value of all test images for each layer of the neural network and minimum value of a particular test image for each layer, as shown in eq. (3).

Upon computing the effective vector for each test image, the dataset reduction module124is configured to apply k-means clustering and stratified sampling to the computed effective vector of each test image, to cluster the test images to k-clusters. In an exemplary embodiment, the value of k is chosen based on number of classes and total number of images.

Upon clustering and sampling the effective vector of each test image, the dataset reduction module124is configured to select one or more images from each cluster, to obtain the reduced dataset114. In one embodiment, the dataset reduction module124selects outliers of each cluster or group as they are most sensitive true positive images that get affected by quantization. Therefore, the dataset reduction module124selects the one or more images from each cluster along with outliers, effective vector falling far away from the centre of the clusters. In one embodiment, in order to obtain M images as reduced dataset114, the dataset reduction module124is configured to select M/k images from each cluster in the order of the increasing distance from centre of each of k clusters. On obtaining the reduced dataset114, the system100proceeds to perform quantization on the reduced dataset114.

In one embodiment, the shift constant determination module126is configured to receive the reduced dataset114from the dataset reduction module124and determine shift constant for each layer of the neural network based on the reduced dataset114. In one embodiment, the shift constant determination module126is configured to determine the shift constant based on type of the layer of the neural network, wherein the type of layer includes one of a parametric layer and a non-parametric layer, wherein the non-parametric layer is one of activation layer, pooling layer, layer with multiple inputs and non-operational single input single output layers.

In order to determine the shift constant of the parametric layer, the shift constant determination module126is configured to determine one or more parameters associated with an input for each parametric layer. In one embodiment, the input includes input data and weight data for that parametric layer. In one embodiment, the one or more parameters can be an absolute maximum value of the input for each parametric layer (XabsMax) as shown in eq. (4) and a quantization parameter (Qm.n) for each parametric layer. The value ‘m’ indicates a number of integer bits of a fixed-point dataset and the value ‘n’ indicates a number of fractional bits of a fixed point dataset.

wherein the input is uniformly distributed over (Xmin, Xmax).

The number of integer bits “m” is based on the value of XabsMax as shown in below eq. (5)

The number of fractional bits “n” is computed as shown in below eq. (6)

wherein Mbits is a maximum number of bits present to represent the input distribution in fixed point.

After determining the one or more parameters, the shift constant determination module126is configured to compute an initial shift constant (Csc_init) using the determined parameter as shown in eq. (7).

Upon determining the initial shift constant, the shift constant determination module126is configured to perform the quantization of parametric layer using the initial shift as shown in eq. (8) and (9).

Wherein Xfpis the floating-point data from the reduced dataset, and Mbits is a maximum number of bits present to represent the floating-point data in fixed-point e.g., 8-bit/16-bit.

The shift constant determination module126is configured to estimate a first output value using the quantization shown in eq. (8) and (9) at an output of the neural network in response to the quantization.

The shift constant determination module126is configured to compute a model-interference-error (MIE) at the output of the neural network, after the quantization is performed. The MIE is computed by comparing the estimated first output value with a second output value, wherein the second output value is determined as an output of the neural network prior to quantization.

The shift constant determination module126is configured to determine an updated shift constant (Csc_updt) as shown in eq. (10).

wherein Csc_updt(i)=Csc_initfor the 1stiteration,wherein “i” is an iteration number, Csc_updt(i) is a shift constant of ithiteration, and Csc_updt(i+1) is a shift constant of (i+1)thiteration.

The shift constant determination module126is then configured to iterate the step of obtaining Csc_updtand performing quantization till the MIE converges to a minimum value as shown inFIG.5. A plot between the model inference error for each iteration with respect to the iteration number is shown inFIG.5. From theFIG.5, the shift constant corresponding to the iteration number at which MIE is minimum is taken as the shift constant.

Upon obtaining the shift constant at which the MIE converges, the shift constant determination module is configured to find an updated final shift constant by performing above procedure using the below eq. (11)

wherein Csc_updt(1) is an updated shift constant obtained using the eq. (10) for which MLE converges to minimum error value.

The shift constant determination module126is configured to determine shift constant for the non-parametric layer by assigning the shift constant as ‘1’ when a first layer of the neural network is the non-parametric layer. For other non-parametric layers, the shift constant determination module126is configured to determine shift constant for the non-parametric layer having non-exponential input by assigning the shift constant of a previous layer, and determine shift constant for the non-parametric layer having exponential input by assigning the shift constant as ‘1’.

The shift constant determination module126is configured to determine shift constant for the layer with multiple inputs by assigning minimum of the shift constant of the previous layers, as the shift constant for the layer with multiple inputs. For example, when Add layer (i.e., layer with multiple inputs) is connected to conv1and conv2having sc1and sc2respectively, the shift constant determination module126determines the shift constant for Add layer as min (sc1, sc2).

The quantization module128is coupled to the shift constant determination module126and is configured to receive the shift constant value of each layer. The quantization module128performs quantization of input data associated with each layer using the below eq. (12) and (13), wherein the input data is input for each layer.

Wherein Xfpis the input data,

The quantization module128performs quantization of weight data associated with the first layer using the below eq. (14) and (15).

Wherein Xfpis the weight data, wherein Cscis shift constant of the first layer.

The quantization module128performs quantization of weight data associated with the layer other than first layer using the below eq. (16) and (17).

Here, by performing Csc/Csc_i, there will be improvement in performance of the MLS100as the additional divisions required at each layer are eliminated by performing a single operation of Csc/Csc_i.

Wherein Xfpis the weight data, wherein Cscis shift constant of the present layer, and wherein Csc_iis shift constant of previous parametric layer.

FIG.2illustrates a flow chart of an exemplary method of performing quantization of a neural network in accordance with some embodiments of the present disclosure.

The method200comprises one or more blocks implemented by the MLS100for performing quantization of the neural network. The method200may be described in the general context of a computer processor executable instructions. Generally, computer processor executable instructions can include scalar instructions, vector instructions, comparison and selection-based instructions etc.

The order in which the method200is described in not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method200. Additionally, individual blocks may be deleted from the method300without departing from the spirit and scope of the subject matter described herein. Furthermore, the method200can be implemented in any suitable hardware having parallel processing capability, software executed by a processor, firmware, or combination thereof.

At block202, receive a reduced dataset as input dataset at a first layer from the dataset reduction module124. In one embodiment, the shift constant determination module126is configured to receive the reduced dataset to perform quantization of the neural network. The method of obtaining reduced dataset from an original validation dataset will be explained in the forthcoming paragraphs with respect toFIG.3.

At block204, a shift constant for the first layer of the neural network is determined. In one embodiment, the shift constant determination module126is configured to determine shift constant for the first layer of the neural network. In one embodiment, the shift constant determination module126is configured to determine the shift constant based on type of the layer of the neural network, wherein the type of layer includes one of a parametric layer and a non-parametric layer, and wherein the non-parametric layer can be one of activation layer, pooling layer, and layer with multiple inputs.

In another embodiment, the shift constant determination module126is configured to determine shift constant for each parametric layer of the neural network will be explained in the forthcoming paragraphs with respect toFIG.4.

In yet another embodiment, the shift constant determination module126is configured to determine shift constant for the non-parametric layer by assigning the shift constant as ‘1’ when the first layer is the non-parametric layer. In another embodiment, if the first layer is a parametric layer, the shift constant determination module126determined the shift constant for the subsequent non-parametric layer having non-exponential input by assigning the shift constant of the previous layer as the shift constant for the activation layer, pooling layer, and non-operational single input single output layers.

In still further embodiment, the shift constant determination module126is configured to determine shift constant for the layer with multiple inputs by assigning minimum of the shift constant of the previous layers, as the shift constant for the layer with multiple inputs.

At block206, quantization for the first layer using the determined shift constant is performed by the quantization module128. The quantization module128performs quantization of input data associated with each layer using the below eq. (12) and (13). The quantization module128performs quantization of weight data associated with the first layer using the below eq. (14) and (15). The quantization module128performs quantization of weight data associated with the layer other than first layer using the below eq. (16) and (17).

At block208, a shift constant for next layer of the neural network is determined. In one embodiment, the shift constant determination module126is configured to determine shift constant for the next layer of the neural network based on output of a layer previous to next layer using the similar steps performed in step204.

At block210, quantization for the next layer using the determined next shift constant is performed by the quantization module128using the similar steps performed in step206.

At block212, quantization of all layers of the neural network is determined. In one embodiment, the quantization module128determines as to whether all layers in the neural network are quantized. If it is determined that all the layers are quantized, the method stops further processing along the YES block. Otherwise, the method proceeds to block208along the NO block and the steps in blocks208and210are iterated till all the layers of the neural network are quantized, thereby obtaining a fixed point dataset as an output of the neural network.

FIG.3illustrates a flow chart of an exemplary method for obtaining reduced dataset from an original validation dataset in accordance with some embodiments of the present disclosure.

The method300comprises one or more blocks implemented by the dataset reduction module124for obtaining reduced dataset. The method300may be described in the general context of a computer processor executable instructions. Generally, computer processor executable instructions can include scalar instructions, vector instructions, comparison and selection-based instructions etc.

The order in which the method300is described in not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method300. Additionally, individual blocks may be deleted from the method300without departing from the spirit and scope of the subject matter described herein. Furthermore, the method300can be implemented in any suitable hardware having parallel processing capability, software executed by a processor, firmware, or combination thereof.

At block302, receive an original validation dataset112from the receiving module122. In one embodiment, the original validation dataset112comprises a plurality of test images.

At block304, true positive images from the original validation dataset112are determined. The dataset reduction module124is configured to determine true positive images from the received original validation dataset112if the original validation dataset112comprises true positive and false positive images as classified by respective models. The dataset reduction module124retrieves true positive images as test images from the original validation dataset112for computing effective vector, as false positive images will not degrade the accuracy after quantization.

At block306, an effective vector for each test image is computed. The dataset reduction module124is configured to compute an effective vector for each test image. In order to compute the effective vector of each test image, the dataset reduction module124is configured to compute maximum and minimum value at an output of each layer of neural network. For example, X_maxiLrepresents maximum value at the output of LthLayer of ithimage, and X_miniLrepresents minimum value at the output of LthLayer of ithimage.

After computing the maximum and minimum value of each layer of all the test images, the dataset reduction module124is configured to compute the global maximum value of each layer as shown in eq. (1) and compute the global minimum value of each layer as shown in eq. (2).

In one embodiment, the effective vector of each image is computed by computing ratio of maximum value of all test images for each layer of the neural network and maximum value of a particular test image for each layer, and ratio of minimum value of all test images for each layer of the neural network and minimum value of a particular test image for each layer, as shown in eq. (3).

At block308, k-means clustering and stratified sampling to the computed effective vector is applied, upon computing the effective vector for each test image. The dataset reduction module124is configured to apply k-means clustering and stratified sampling to the computed effective vector of each test image, to cluster the test to k-clusters. In an exemplary embodiment, the value of k is chosen based on number of classes and total number of images.

At block310, one or more images from each cluster is selected to obtain the reduced dataset, upon clustering and sampling the effective vector of each true positive image. The dataset reduction module124is configured to select one or more images from each cluster, to obtain the reduced dataset114. In one embodiment, in order to M images of reduced dataset, the dataset reduction module124is configured to select M/k images from each cluster, in the order of the increasing distance from centre of each of k clusters.

FIG.4illustrates a flow chart of an exemplary method for determining a shift constant for parametric layer, in accordance with some embodiments of the present disclosure.

The method400comprises one or more blocks implemented by the shift constant determination module126to determine a shift constant for the parametric layer. The method400may be described in the general context of a computer processor executable instructions. Generally, computer processor executable instructions can include scalar instructions, vector instructions, comparison and selection-based instructions etc.

The order in which the method400is described in not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method400. Additionally, individual blocks may be deleted from the method400without departing from the spirit and scope of the subject matter described herein. Furthermore, the method400can be implemented in any suitable hardware having parallel processing capability, software executed by a processor, firmware, or combination thereof.

At block402, one or more parameters associated with an input for each parametric layer is determined. In one embodiment, the shift constant determination module126is configured to determine one or more parameters associated with an input for each parametric layer. In one embodiment, the input includes input data and weight data for that parametric layer. In one embodiment, the one or more parameters can be an absolute maximum value of the input for each parametric layer (XabsMax) as shown in eq. (4) and a quantization parameter (Qm.n) for each parametric layer. The value ‘m’ indicates a number of integer bits of a fixed-point dataset and the value ‘n’ indicates a number of fractional bits of a fixed point dataset.

At block404, an initial shift constant is computed using the determined parameters. In one embodiment, the shift constant determination module126is configured to compute an initial shift constant (Csc_init) using the determined parameter as shown in eq. (7).

At block406, quantization of each parametric layer is performed using the shift constant. In one embodiment, the shift constant determination module126is configured to perform the quantization using the initial shift as shown in eq. (8) and (9).

At block408, a first output value at an output of the neural network is estimated. In one embodiment, the shift constant determination module126is configured to estimate a first output value at an output of the neural network in response to the quantization.

At block410, a model-inference-error (MIE) at the output of the neural network is computed based on the first output value and a second output value. In one embodiment, the shift constant determination module126is configured to compute a model-interference-error (MIE) at the output of the neural network, after the quantization is performed by comparing the estimated first output value with a second output value, wherein the second output value is determined as an output of the neural network prior to quantization.

At block412, whether the MIE converges to a minimum error value is determined. In one embodiment, if the shift constant determination module126determines that the MIE converges to a minimum value, then the method stops further processing along the YES block. Otherwise, the method proceeds to block414along the NO block to determine an updated shift constant.

At block414, an updated shift constant is determined if the MIE does not converge to the minimum value. In one embodiment, the shift constant determination module126is configured to determine an updated shift constant (Csc_updt) as shown in eq. (10).

FIG.6illustrates an exemplary arrangement of different layers in neural network to perform quantization, in accordance with some embodiments of the present disclosure.

In order to perform quantization, shift constant for each layer needs to be determined.

The initial input dataset is passed through the conv1and conv2layer. By performing the step204for the conv1and conv2, a shift constant of sc1and sc2are obtained, respectively. Further, by performing the step206, the conv1and conv2layers are quantized using the shift constant sc1and sc2, respectively.

The output of the conv1layer is passed as input to the relu layer and shift constant for the relu layer is determined as shift constant of previous layer i.e., sc1. Similar, shift constant of another relu layer connected to the conv2is determined as shift constant of previous layer i.e., sc2.

For the conv3layer, the shift constant is determined as sc3and perform quantization using the shift constant sc3and previous parametric layer shift constant sc1i.e., sc3/sc1.

For the conv4layer, the shift constant is determined as sc4and perform quantization using the shift constant sc4and previous parametric layer shift constant sc2i.e., sc4/sc1.

For the Add layer connected to conv3and conv4, the shift constant is determined as minimum of sc3and sc4i.e., sc3as shown inFIG.6.

For the conv5layer connected to Add layer, the shift constant is determined as sc5and perform quantization using the shift constant sc5and previous parametric layer shift constant sc3i.e., sc5/sc3.

For the Tanh layer connected to Add layer, the shift constant is determined as 1 as the layer is exponential layer.

For the conv6layer connected to Tanh layer, the shift constant is determined as sc6and perform quantization using the shift constant sc6and previous layer shift constant 1. The similar procedure is repeated for all layers as shown inFIG.6.

The apparatus and method disclosed herein present an improved way of quantizing the neural network. The method presents improved way of computing an effective shift constant for each neural network that minimize model interference error, thereby making the quantized neural networks deployed even in resource constrained settings.

Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure of the embodiments of the disclosure is intended to be illustrative, but not limiting, of the scope of the disclosure.