Patent Description:
Technological automation of feature extraction, pattern recognition, and/or analyses, as only examples, has been implemented through processor implemented neural network models, as specialized computational architectures, that after substantial training may provide computationally intuitive feature extractions or recognitions, mappings between input patterns and output patterns, pattern recognitions of input patterns, categorization, or classification in various forms. The trained capability of extracting such information or recognitions, generating such mappings, performing such pattern recognitions, or performing such categorizations or classifications may be referred to as a learning capability of the neural network. Such trained capabilities may also enable the specialized computational architecture to classify an input pattern or object(s), or portions of the input pattern or object(s), e.g., as a member that belongs to one or more predetermined groups. Further, because of the specialized training, such specially trained neural network may thereby have a generalization capability of generating a relatively accurate or reliable output with respect to an input pattern that the neural network may not have been trained for, for example. However, because such operations are performed through such specialized computation architectures, and in different automated manners than they would have been performed in non-computer implemented or non-automated approaches, they also invite problems or drawbacks that only occur because of the automated and specialized computational architecture manner in which they are implemented.

Such neural network implementations also require many operations for complex input data, which may require large power consumption and require long processing times. Further, attempts to reduce such processing requirements that are implemented result in degradations in accuracy and precision.

The publication of <NPL>), refers to an optimization strategy for training neural networks which are called "BitNet". The parameters of neural networks are usually unconstrained and have a dynamic range dispersed over all real values. The expressive power of the network is limited by dynamically controlling the range and set of values that the parameters can take. An end-to-end approach is used that circumvents the discreet parameter space by optimizing a relaxed, continuous, and differentiable upper bound of the typical classification-loss function. For each layer of the network, this approach optimizes real valued translation and scaling factors and arbitrary precision integer valued parameters.

<NPL>) discloses a method to discretize both training and inference, where weights, activations, gradients, and errors among layers are shifted and linearly constrained to low-bitwidth integers. For tasks with small numbers of categories, Softmax layer is avoided and mean-square-error criterion is applied but mean operation is omitted to form a sum-square-error criterion. In an experiment, since ImageNet task is much difficult than CIFAR10 and has <NUM>,<NUM> categories, Softmax is added and quantizations are removed in the last layer.

It is the object of the present invention to provide an improved processor-implemented neural network method and neural network apparatus.

<FIG> is a diagram illustrating an example of a neural network <NUM> according to one or more embodiments.

Referring to <FIG>, the neural network <NUM> may have a structure including an input layer, hidden layers, and an output layer. The neural network <NUM> may perform an operation based on received input data (for example, I<NUM> and I<NUM>) and generate output data (for example, O<NUM> and O<NUM>) based on a result of the operation.

The neural network <NUM> may be a deep neural network (DNN) or an n-layers neural network including one or more hidden layers. For example, as illustrated in <FIG>, the neural network <NUM> may be a DNN including an input layer Layer <NUM>, two hidden layers (a second layer Layer <NUM> and a third layer Layer <NUM>), and an output layer Layer <NUM>. The DNN may be a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a restricted Boltzman machine, or examples may be implemented through other machine learning architectures, and thus, examples are not limited thereto. The use of the term 'may' herein with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented while all examples and embodiments are not limited thereto.

When the example neural network <NUM> includes a DNN structure, the neural network <NUM> may include more layers that may extract or generate new or inferred effective information. Thus, in various examples, the neural network <NUM> may handle more complex data sets than previous machine learning approaches. In <FIG>, the neural network <NUM> is illustrated as including four layers, but this is only an example, and the neural network <NUM> may include less or more layers. In addition, the neural network <NUM> may include layers of various structures different from those illustrated in <FIG>. In an embodiment, the neural network <NUM> may include, as a DNN, a convolutional layer, a pooling layer, or a fully connected layer.

Each of the layers included in the neural network <NUM> may include a plurality of nodes, which may also be referred to as nodes 'processing elements (PEs)', 'units', or similar computational architectural terms. In the embodiment illustrated in <FIG> the input layer Layer <NUM> may include two nodes and the second layer Layer <NUM> may include three nodes. However, this is only an example and each of the layers included in the neural network <NUM> may include various numbers of nodes. Accordingly, the neural network <NUM> includes a plurality of layers, and each of the layers includes a plurality of nodes. Depending on the architecture of the neural network <NUM>, nodes included within layers and/or in neighboring layers may be selectively connected according to respective connections, e.g., which may or may not be weighted. For example, the neural network <NUM> may be implemented by a processor, i.e., one or more processors, configured to generate a neural network structure/architecture with such a plurality of layers each including plural nodes and configured to apply such example weighted connections between neighboring nodes in neighboring layers of the neural network structure, and/or apply example kernels or weighted connections within layers, to interpret input data applied to the neural network structure. As only examples, herein such an 'interpretation' of input data may include a performed recognition, verification, or rejection, or input data binary or multi-class classification, clustering, pattern observation, transformation, and/or regression, as well as any other trained objective of the neural network in various examples.

The nodes included in each of the layers included in the neural network <NUM> may be connected to one another to exchange data. In an embodiment, one node may receive data from other nodes and operate according to the received data, and output operation results to other nodes.

An output value of each of the nodes may be referred to as an activation, i.e., as a respective result of one or more activation functions of a corresponding node applied with respect to at least input information to the corresponding node. For example, the activation may be as simple as rectified linear unit (ReLU), sigmoid function, or tanh applied to the summation of multiple weighted inputs, or the nodes may be more complex, such being gated or long short-term memory nodes, as non-limiting examples. Thus, an activation may be the output value of one node of one layer, and then may be an input value for one or more nodes included in one or more next layers, further subsequent layers, one or more previous layers or the same layer, or another neural network. Thus, as only an example, each of the nodes of the example neural network <NUM> may generate its activation based on weights and activations received from the nodes included in a previous layer. The weight may be a trained parameter of the neural network <NUM> that results from training of parameters of an initial or interim neural network, for example. Thus, a trained weight may be applied to a particular input (or resultant activation from a previous node/layer) to a node, with respective other trained weights being applied to other respective inputs to the node, such that the activation of the node with respect to such weighted inputs and may be a value or activation result that is then output and assigned to a connection relationship between that node and one or more next nodes. In addition to such weight parameters the neural network <NUM> may also apply biases for such activations. The biases may be set or trained during the training of the neural network <NUM>, for example.

Thus, each of the nodes may be a computational unit that receives one or more data values and outputs the activation, and ultimately may map inputs to outputs. For example, when σ is an activation function, <MAT>is a weight from the kth node in the (i-<NUM>)th layer to the jth node in the ith layer, and <MAT>is a bias value of the jth layer, and <MAT>is the activation of the jth node in the ith layer, the activation <MAT>may be expressed by Equation <NUM>.

As illustrated in <FIG>, the activation of the first node in the second layer Layer <NUM> may be expressed as <MAT>In addition, <MAT>may have a value of <MAT> according to Equation <NUM>. However, the above-described Equation <NUM> is only an example for describing the activation and the weight used for processing data in the neural network <NUM>, but is not limited thereto. As noted in the above example, the activation may be a value obtained by processing through a ReLU a value obtained by applying the activation function to a weighted sum of activations received from the previous layer.

As described above, in the neural network <NUM>, numerous data sets may be exchanged between a plurality of channels interconnected to one another, and may undergo numerous operations processes while passing through layers. In examples herein, it is found that a technology that may improve neural networks, as well as improve the training of such neural networks, by decreasing the amount of operations needed to process complicated input data and simultaneously reduce accuracy loss may be desired, thereby improving the computer functionality of the corresponding neural network apparatuses and methods.

Accordingly, one or more embodiments provide technological improvements that may include improving processing operations (e.g. helping to improve speed and helping to minimize any corresponding accuracy loss) of a processor, reduce memory requirements, improve memory access speeds, and/or improve the speed of classification determinations. Further, with one or more embodiments, more complex and sophisticated trained neural networks may be performed on processing systems that have lesser capabilities, such as in mobile examples, where such trained neural networks may not have been available for implementation or may not have been able to be performed with sufficient speed to operate in real-time during operation of such mobile apparatuses, as non-limiting examples.

<FIG> is a block diagram of a hardware configuration of a neural network apparatus <NUM> according to an embodiment.

Referring to <FIG>, the neural network apparatus <NUM> may include a processor <NUM> and a memory <NUM>. The neural network apparatus <NUM> may include the options and apparatuses of <NUM> of <FIG>, and additional hardware components such as in mobile apparatus examples.

In an example, the neural network apparatus <NUM> may correspond to a computing device comprising one or more processors having various processing functions such as functions to generate a neural network, train or learn the neural network, quantize a floating-point type neural network to a fixed-point type neural network, or retrain the neural network. In an embodiment, the neural network apparatus <NUM> may be implemented in various types of devices such as personal computers (PCs), server devices, mobile devices, etc. Also, the neural network apparatus <NUM> may include a hardware accelerator for driving a neural network. The hardware accelerator may correspond to, for example, a neural processing unit (NPU), a tensor processing unit (TPU), a neural engine, etc., which are dedicated modules for driving the neural network, but the present disclosure is not limited thereto.

The processor <NUM> may perform all control operations to control the neural network apparatus <NUM>. In an embodiment, the processor <NUM> controls all functions of the neural network apparatus <NUM> by executing complex readable instructions stored in the memory <NUM> in the neural network apparatus <NUM>. The processor <NUM> may be implemented by a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc., which are provided in the neural network apparatus <NUM>. However, the present disclosure is not limited thereto.

The memory <NUM> is hardware for storing various pieces of data processed in the neural network apparatus <NUM>. For example, the memory <NUM> may store data processed and to be processed in the neural network apparatus <NUM>. Furthermore, the memory <NUM> may store applications, drivers, etc. to be driven by the neural network apparatus <NUM>. The memory <NUM> may be DRAM, but the present disclosure is not limited thereto. The memory <NUM> may include at least one of volatile memory or nonvolatile memory. The memory <NUM> may also store trained parameters of the neural network.

The processor <NUM> may generate a trained neural network by repeatedly training (learning) a given initial neural network. In this regard, the processor <NUM> may train the neural network according to a predetermined precision. In other words, the processor <NUM> may set precision of the parameters processed in the neural network when trained. Precision refers to a degree to which detail the parameters in the neural network may be processed. Precision may be described by a bit width of the parameters in the neural network. In an embodiment, a parameter with high precision such as a parameter of <NUM> bit floating point precision and a parameter with low precision such as a parameter of <NUM> bit fixed point precision may be present. In this regard, the parameters may include various types of data input/output to/from the neural network such as input/output activations, as well as weights, biases, etc. of the neural network.

In examples, the processor <NUM> may process the parameters of the neural network according to a low precision. In other words, the processor <NUM> may use data having a relatively smaller bit width than typical neural networks as the parameters, thereby reducing the computational amount while reducing the precision. In an embodiment, unlike such typical neural networks that use <NUM>-bit or <NUM>-bit floating-point or fixed-point data, the processor <NUM> may use <NUM>-bit or <NUM>-bit floating-point or fixed-point data as the parameters according to the low precision. However, when the low precision is commonly set for all layers in the neural network, since a great accuracy loss due to the characteristics of some layers in the neural network may occur, it is found herein, in an example, to lessen or minimize the accuracy loss by setting a high precision for some layers in the neural network.

The processor <NUM> trains a neural network for classification. The neural network for classification may output an operation result as to which class of input data corresponds to which class. Specifically, the neural network for classification may output an operation result regarding the possibility that the input data corresponds to each of multiple classes as a result value for each of the classes. The neural network for classification includes a softmax layer and a loss layer. The softmax layer may convert the result value for each of classes into a probability value or other probabilistic information. The loss layer may be used to calculate loss as an objective function for learning.

<FIG> illustrates a neural network <NUM> for classification according to an embodiment.

The neural network <NUM> for classification includes a fully-connected (FC) layer <NUM>, a softmax layer <NUM>, and a loss layer <NUM>; and it may also include hidden layers <NUM>. Some of the hidden layers <NUM> may be FC layers, and thus the FC layer <NUM> may be a last FC layer of the neural network <NUM>. In other words, the FC layer <NUM> may be the FC layer of the last order among FC layers of the neural network <NUM> as a non-limiting example.

When input data is input to the neural network <NUM>, after the sequential operations of the hidden layers <NUM> and the FC layer <NUM>, the FC layer <NUM> may output an operation result(s) regarding the possibility that the input data is classified into each class. In other words, the FC layer <NUM> may output a result value of the possibility that the input data is classified into each class, as the operation result(s) for each class. Specifically, the FC layer <NUM> may include nodes corresponding to each class, and each of the nodes of the FC layer <NUM> may output the result value of the possibility that the input data is classified into each class. In an embodiment, when the neural network <NUM> is implemented for a classification job targeting five classes, an output value of each of first to fifth nodes of the FC layer <NUM> may be a result value of the possibility that the input data is classified into each of first to fifth classes.

The FC layer <NUM> outputs the operation result(s) to the softmax layer <NUM>. The softmax layer <NUM> may convert the operation result(s) into a probability value y. In other words, the softmax layer <NUM> may generate the probability value y by normalizing the result value of the possibility that the input data is classified into each class. The softmax layer <NUM> then ouputs the probability value y to the loss layer <NUM>. The loss layer <NUM> may calculate a cross-entropy loss L of the operation result(s) based on the probability value y. In other words, the loss layer <NUM> may calculate the cross-entropy loss L indicating an error of the operation result(s), e.g., based upon a known label for the input training data.

In an embodiment, the softmax layer <NUM> may convert the computation result(s) into the probability value y by using Equation <NUM> below, and the loss layer <NUM> may calculate the cross-entropy loss L of the operation result(s) by using Equation <NUM> below. <MAT> <MAT>.

In Equations <NUM> and <NUM>, si denotes an output value of an i-th node of the FC layer <NUM> (i.e., a result value for an i-th class among the classes), yi denotes an output value of the i-th node of the softmax layer <NUM> (i.e. a probability value for the i-th class among the classes), Nc denotes the number of classes, and ti denotes a ground truth (GT) label for the i-th class.

Then, as a back propagation learning process, the softmax layer <NUM> may calculate a gradient <MAT>of the cross-entropy loss L through the loss layer <NUM>.

In an embodiment, the softmax layer <NUM> may be considered to calculate the gradient <MAT> of the cross-entropy loss L by using Equation <NUM>.

In Equation <NUM>, si denotes the output value of the i-th node of the FC layer <NUM> (i.e., the result value for the i-th class among the classes), yi denotes the output value of the i-th node of the softmax layer <NUM> (i.e. the probability value for the i-th class among the classes), Nc denotes the number of classes, and ti denotes the GT label for the i-th class.

Then, a learning process is performed for the FC layer <NUM>, the learning process based on the gradient <MAT>of the cross-entropy loss L. For example, a weight of the FC layer <NUM> may be updated in accordance with a gradient decent algorithm. Also, the hidden layers <NUM> may also have their weights respectively updated in such a back propogation chain learning process.

<FIG> illustrates output values of each layer in a neural network for classification according to one or more embodiments.

In each layer in the neural network, values according to a predetermined precision may be output. Specifically, referring to <FIG>, output values of each layer in the neural network may be distributed in quantized levels. For example, when the predetermined precision is a bit width of <NUM> bits, the output values may be distributed to <NUM><NUM> quantized levels.

Referring to the upper graph of <FIG> a training or inference operation, output values si of the FC layer <NUM> which is a last FC layer may be distributed to quantized levels with respect to <NUM>, while output values yi of the softmax layer <NUM> may be normalized probability values, and distributed to quantized levels with respect to <NUM>/(number Nc of classes). For example, when the number of classes is <NUM>, the output values yi of the softmax layer <NUM> may be distributed to quantized levels with respect to <NUM>/<NUM>, and when the number of classes is <NUM>, the output values yi of the softmax layer <NUM> may be distributed to quantized levels with respect to <NUM>/<NUM>.

The lower graph of <FIG> shows, in a back-propagation learning process, the gradient <MAT> of the cross-entropy loss L for training the FC layer <NUM>. The gradient for a class corresponding to a GT label may have a value between -<NUM> and <NUM>, but the gradient for other classes may have a value between <NUM> and <NUM> due to the characteristics of the gradient according to Equation <NUM>. In particular, the greater the number of classes, the more the gradient for other classes may have a value close to <NUM>.

The gradient for training the FC layer <NUM> may be processed to have a value according to the predetermined precision. In other words, the gradient used to train the FC layer <NUM>, derived from the softmax layer, may be adjusted to the quantized levels according to a predetermined bit width. However, when the number of classes is large, during a process of adjusting the gradient to the quantized levels, the gradient for the class corresponding to the GT label may be adjusted to -<NUM>, but the gradient for the other classes may be adjusted to <NUM>. In an embodiment, referring to the right graph of <FIG>, when the number of classes is <NUM>, the gradient for classes that do not correspond to the GT label may be adjusted to levels between <NUM> and <NUM>, whereas when the number of classes is <NUM>, the gradient for the classes that do not correspond to the GT label may all be adjusted to <NUM>. Thus, when the number of classes is large, the sum of the gradients for all the classes may be biased to a negative value, and consequently may have a negative impact on learning or training of the neural network <NUM>.

Therefore, it is required to set precision in consideration of the number of classes with respect to the FC layer <NUM>, rather than setting precision (in particular, low precision) fixed to all the layers in the neural network <NUM> at one time.

Referring again to <FIG>, the processor <NUM> may determine precision for a layer (hereinafter referred to as an output layer) outputting the operation result(s) of the possibility that the input data is classified into each class based on information about the number of classes. Referring to <FIG>, the output layer is the FC layer <NUM>, which is the last FC layer. The output layer is connected to the softmax layer <NUM> and the loss layer <NUM>.

First, the processor <NUM> obtains the information about the number of classes into which the input data may be classified. According to an embodiment, since the number of classes is equal to the number of nodes of the output layer, the processor <NUM> may determine the number of nodes of the output layer and obtain the information about the number of classes. According to an embodiment, the processor <NUM> may also obtain the information about the number of classes based on user input or the number of classes may be determined.

The processor <NUM> then determines precision for the output layer based on the obtained information about the number of classes.

According to an embodiment which is not according to the invention, the processor <NUM> may determine the precision for the output layer in proportion to, or based on, the number of classes. In other words, the processor <NUM> may determine that the higher the number of classes, the higher the precision for the output layer. For example, when the number of classes is <NUM>, the processor <NUM> may determine the precision for the output layer as a bit width of <NUM> bits, and when the number of classes is <NUM>, determine the precision for the output layer as a bit width of <NUM> bits. Thus, the output layer precision may be variably set based on the number of classes.

The processor <NUM> determines the precision for the output layer, according to Equation <NUM> below. That is, the processor <NUM> may determine the bit width applied to the output layer as the precision for the output layer.

In Equation <NUM>, |class| denotes the size of classes (that is, the number of classes), α denotes a predetermined bias, and β denotes a proportional constant. For example, when α is set to <NUM>, β is set to <NUM>, and the number of classes is <NUM>, the processor <NUM> may determine the precision for the output layer as a bit width of <NUM> bits or more. Further, when α is set to <NUM> and the number of classes is <NUM>, the processor <NUM> may determine the precision for the output layer as a bit width of <NUM> bits or more. Further, when α is set to <NUM> and the number of classes is <NUM>, the processor <NUM> may determine the precision for the output layer as a bit width of <NUM> bits or more.

According to another embodiment which is not according to the invention, the processor <NUM> may determine the precision for the output layer to be higher than precision for other layers in the neural network when the number of classes is greater than or equal to a predetermined threshold. In an embodiment, when the number of classes is <NUM> or more, the processor <NUM> may determine the precision for the output layer to be higher than a bit width of <NUM> bits that is the precision for other layers in the neural network.

The processor <NUM> processes parameters in the output layer according to the predetermined precision. The processor <NUM> may process the parameters of the output layer to have the predetermined precision and perform a training/learning process of the neural network through the parameters having the predetermined precision. The parameters of the output layer are trained using a gradient of the cross-entropy loss derived with respect to a subsequent softmax layer. Accordingly, the processor <NUM> may adjust the gradient of the cross-entropy loss input to the output layer to a quantized level according to a predetermined bit width and perform the training/learning process of the neural network through the adjusted gradient. Further, the parameters of the output layer may further include weight and activation of the output layer.

Accordingly, the neural network apparatus <NUM> sets the precision for the output layer such as the last FC layer in consideration of the number of classes, e.g., rather than setting a fixed precision for all the layers in the neural network, thereby lessening or minimizing the accuracy loss in the training/learning process. In particular, the neural network apparatus <NUM> may process the parameters of the neural network with a low precision with respect to other layers of the neural network while processing the parameters of the output layer of the neural network with a high precision in consideration of the number of classes, thereby lessening or minimizing the accuracy loss that may occur due to the collective low precision setting.

<FIG> illustrates an example of determining a precision in the neural network <NUM> for classification according to an embodiment.

The neural network apparatus <NUM> obtains information on the class size of the neural network <NUM> for classification. That is, the neural network apparatus <NUM> obtains information about the number of classes into which input data may be classified in the neural network <NUM> for classification. The neural network apparatus <NUM> may obtain the information about the number of classes by determining the number of nodes of the FC layer <NUM> which is a last FC layer.

The neural network apparatus <NUM> determines precision for the FC layer <NUM>, which is an output layer, based on the obtained information about the number of classes. classes according to Equation <NUM> above.

The neural network apparatus <NUM> processes parameters in the FC layer <NUM> according to a predetermined precision. According to an embodiment, the neural network apparatus <NUM> may process a gradient of a cross-entropy loss output from the softmax layer <NUM> and used to train the FC layer <NUM>, to have a predetermined precision, and thus proceed with the learning/training of the neural network <NUM> through the gradient having the predetermined precision. Training of the neural network <NUM> may be continued with adjustments of the parameters of the neural network until the neural network is trained to a predetermined accuracy and/or predetermined inaccuracy.

<FIG> is a block diagram of an electronic system <NUM> according to an embodiment.

Referring to <FIG>, the electronic system <NUM> may extract effective information by analyzing input data in real time based on a neural network, determine a situation based on the extracted information, or control elements of the electronic device represented by or on which the electronic system <NUM> is mounted. For example, the electronic system <NUM> may be any one of, or applied to, or mounted in, robot devices such as drones, advanced drivers assistance system (ADAS), etc., smart TVs, smart phones, medical devices, mobile devices, image display devices, measuring devices, loT devices, etc., as non-limiting electronic device examples.

The electronic system <NUM> may include a processor <NUM>, RAM <NUM>, a neural network apparatus <NUM>, a memory <NUM>, a sensor module <NUM>, and a communication (TX/RX) module <NUM>. The electronic system <NUM> may further include an input/output module, a security module, a power control device, etc. Some hardware components of the electronic system <NUM> may be mounted on at least one semiconductor chip, for example. In examples, the neural network apparatus <NUM> may include the above-described neural network apparatus <NUM> or neural network dedicated hardware accelerator or an apparatus including the same.

The processor <NUM> may control some or all operations of the electronic system <NUM>. The processor <NUM> may include one processor core (Single Core), or a plurality of processor cores (Multi-Core). The processor <NUM> may process or execute instructions and programs and/or data stored in the memory <NUM>. In an embodiment, the processor <NUM> may control implementations of the neural network apparatus <NUM> by executing the instructions stored in the memory <NUM>. The processor <NUM> may also be implemented by a CPU, a GPU, an AP, etc..

The RAM <NUM> may temporarily store programs, data, or instructions. For example, the programs and/or data stored in the memory <NUM> may be temporarily stored in the RAM <NUM> according to a booting code or the control of the processor <NUM>. The RAM <NUM> may be implemented by memory such as dynamic RAM (DRAM) or static RAM (SRAM), etc. The ram may also temporarily store initial or interim parameters as during any of the training operations described herein and performed by the system <NUM>.

The neural network apparatus <NUM> may perform an operation of a neural network based on the received input data, and generate an information signal based on a result of the operation, e.g. using the neural network described herein. The neural network may include CNNs, RNNs, deep belief networks, restricted Boltzman machines, etc., but the present disclosure is not limited thereto. The neural network apparatus <NUM> is hardware that drives the above-described neural network for classification and may correspond to the neural network dedicated hardware accelerator.

The information signal may include one of various types of recognition signals such as voice recognition signal, object recognition signal, image recognition signal, biometric information recognition signal, etc. For example, the neural network apparatus <NUM> may receive frame data included in a video stream as input data, and generate from frame data a recognition signal for an object included in an image indicated by the frame data and operate or not operate based on the recognition, or lock or unlock access to the corresponding device. However, the present disclosure is not limited thereto, and the neural network apparatus <NUM> may receive various types of input data and generate a recognition or classification signal according to the input data, according to the type of an electronic device on which the electronic system <NUM> is represented by or mounted in.

The memory <NUM> is a storage for storing data, such as, an operating system (OS), various programs, and various pieces of data. In an embodiment, the memory <NUM> may store intermediate results generated in an operation performing process of the neural network apparatus <NUM>, such as, an output feature map, as an output feature list or an outer feature matrix. In an embodiment, the memory <NUM> may store a compressed output feature map. Furthermore, the memory <NUM> may store quantized neural network data, such as, parameters, weight maps, or a weight list, which are used by the neural network apparatus <NUM>.

The memory <NUM> may be DRAM, but the present disclosure is not limited thereto. The memory <NUM> may include at least one of a volatile memory and a nonvolatile memory. The nonvolatile memory may include ROM, PROM, EPROM, EEPROM, flash memory, PRAM, MRAM, RRAM, FRAM, etc. The volatile memory may include DRAM, SRAM, SDRAM, PRAM, MRAM, RRAM, FeRAM, etc. In an embodiment, the memory <NUM> may include at least one of HDD, SSD, CF, SD, Micro-SD, Mini-SD, xD, and Memory Stick.

The sensor module <NUM> may collect information about the periphery of the electronic device represented by, or on which the electronic system <NUM> is mounted. The sensor module <NUM> may sense or receive a signal, such as, an image signal, a voice signal, a magnetic signal, a biometric signal, a touch signal, etc., from the outside of the electronic device, and convert a sensed or received signal to data. To this end, the sensor module <NUM> may include at least one of various types of sensing devices, such as microphones, imaging devices, image sensors, light detection and ranging (LIDAR) sensors, ultrasonic sensors, infrared sensors, biosensors, touch sensors, etc..

The sensor module <NUM> may provide the neural network apparatus <NUM> with the converted data as input data. For example, the sensor module <NUM> may include an image sensor, and may generate a video stream by photographing the external environment of the electronic device, and provide the neural network apparatus <NUM> with consecutive data frames of the video stream in order as input data. However, the present disclosure is not limited thereto, and the sensor module <NUM> may provide various types of data to the neural network apparatus <NUM>.

The communication module <NUM> may include various wired or wireless interfaces capable of communicating with external devices. For example the communication module <NUM> may include a local area network (LAN), wireless local area network (WLAN) such as wireless fidelity (Wi-Fi), wireless personal area network (WPAN) such as Bluetooth, wireless universal serial bus (USB), ZigBee, near field communication (NFC), radio-frequency identification (RFID), power line communication (PLC), or a communication interface capable of connecting to a mobile cellular network such as 3rd generation (<NUM>), 4th generation (<NUM>), long-term evolution (LTE), etc..

<FIG> is a diagram for explaining a method of operating the neural network apparatus <NUM> according to an embodiment.

The method shown in <FIG> may be performed by the neural network apparatus <NUM> of <FIG> or each component of the electronic system of <FIG>, as non-limiting examples. Accordingly, redundant explanations are omitted below.

In operation S710, the neural network apparatus <NUM> obtains information about a number of classes into which input data may be classified. According to an embodiment, the neural network apparatus <NUM> may determine a number of nodes of a layer outputting an operation result indicating a possibility that the input data is classified into each class and obtain the information about the number of classes. According to an embodiment, the neural network apparatus <NUM> may obtain the information about the number of classes based on a user input.

In operation S720, the neural network apparatus <NUM> may determine precision for that layer (hereinafter referred to as an output layer) for outputting the operation result indicating the possibility that the input data is classified into each class, based on the information obtained in operation S710. According to an embodiment which is not according to the invention, the neural network apparatus <NUM> may determine the precision for the output layer to be in proportion to the number of classes. According to another embodiment, the neural network apparatus <NUM> may determine the precision for the output layer according to Equation <NUM>. According to another embodiment which is not according to the invention, the neural network apparatus <NUM> may determine the precision for the output layer to be higher than precision for other layers of the neural network when the number of classes is greater than or equal to a predetermined threshold.

In operation S730, the neural network apparatus <NUM> processes parameters in the layer according to the precision determined in operation S720. The neural network apparatus <NUM> may process the parameters of the output layer to have a predetermined accuracy and perform a training/learning process of the neural network through the parameters having the predetermined precision. The learned parameters may then be stored in memory.

According to embodiments, a neural network apparatus may selectively not set a fixed precision for all layers in a neural network and may set precision in consideration of the number of classes for an output layer such as a last FC layer, thereby lessening or minimizing the precision loss in a training/learning process of the neural network.

The neural network apparatus <NUM>, processor <NUM>, memory <NUM>, RAM <NUM>, Neural Network Device <NUM>, sensor module <NUM>, Tx/Rx module <NUM>, in <FIG> that perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application that are performed by the hardware components. One or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

The methods that perform the operations described in <FIG> this application are performed by computing hardware, for example, by one or more processors or computers, implemented as described above executing instructions or software to perform the operations described in this application that are performed by the methods.

Claim 1:
A processor-implemented neural network method, wherein the neural network comprises a last fully-connected layer (<NUM>) outputting an operation result, a softmax layer (<NUM>) connected to and following the last fully-connected layer, and a loss layer (<NUM>) connected to the softmax layer, the method comprising:
obtaining (S710) information about a number of classes into which input data is classified;
determining (S720) a precision for the last fully-connected layer as being greater than β × log<NUM>(|class|) + α wherein |class| denotes the number of classes, α denotes a predetermined bias, and β denotes a proportional constant; and
processing (S730) parameters of the last fully-connected layer according to the determined precision, wherein the parameters comprise a gradient of a cross-entropy loss derived from a loss generated by the loss layer with respect to the operation result, and using the gradient to train the last fully-connected layer.