Patent Description:
Machine-learning tools are being used for many complex applications to analyze large amounts of data, such as for image and speech recognition, and one of those tools is the deep neural network (DNN), which has demonstrated promising performance in many domains. DNNs are trained with sample data before they can act as classifiers. However, oftentimes, the DNNs have to be retrained to fine-tune performance or to assimilate larger amounts of training data.

In most situations, training a DNN involves solving a non-convex optimization problem with no analytical solution. Typically, solutions are based on solving this problem via iterative procedures, e.g., stochastic gradient descent (SGD). Despite recent progress in computing infrastructure and implementation optimization, it may still take hours, or even up to days or weeks to train a deep neural network, making hard to retrain and inflexible.

Recognizing people in images is a task that is easy for humans but much harder for computers. Being capable of recognizing a substantial number of individuals with high precision and high recall is of great value in many practical applications, such as surveillance, security, photo tagging, and celebrity recognition.

<NPL> relates to Convolutional Neural Networks (CNNs). CNNs consist of alternating layers of feature extraction and subsampling. CNNs are hierarchically structured as a stack of layers including convolutional layers, pooling layers and a fully-connected layer. To form a classification decision, the spatial order from the pooling layer's feature responses is removed and the feature responses are fed to a one-layered fully-connected neural network with a weight matrix W and bias vector b. The specifically discussed convolutional neural network architecture comprises a convolutional feature extraction layer followed by an average pooling layer and a fully-connected softmax output layer. Several different learning strategies are compared, one of which is transfer learning. Transfer learning is usually realized as a transfer of weights. The starting point of each transfer learning training is a classifier trained on one of two different data sets. Since both data sets differ in a number of classes, the output layer of the source network is capped and replaced by a randomly initialized output layer that has the appropriate number of output neurons.

It is the object of the present invention to provide an enhanced system and a method for training a deep neural network more efficiently.

Various ones of the appended drawings merely illustrate example embodiments of the present disclosure and cannot be considered as limiting its scope.

Example methods, systems, and computer programs are directed to training a deep neural network (DNN). Examples merely typify possible variations. Unless explicitly stated otherwise, components and functions are optional and may be combined or subdivided, and operations may vary in sequence or be combined or subdivided. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of example embodiments. It will be evident to one skilled in the art, however, that the present subject matter may be practiced without these specific details.

The term "image recognition" may sound like it only involves one task, but in reality, image recognition is often geared to specific tasks, such as recognizing people, flowers, animals, places, monuments, fruits, or items in a refrigerator. One method for recognizing images is by using deep neural networks (DNN) to classify (i.e., recognize) items in images.

A method used for initial condition construction of a DNN includes leveraging an existing, already-trained neural network with similar structure, which has been trained for general purpose tasks and then fine tune the new neural network to the particular desired task. In some previous implementations, at the initial stage, the lower-level layers of the model to be trained are typically set to have the same parameters as the pre-trained model, while the parameters of the last layer are initialized to be random numbers sampled from certain distributions (usually Gaussian). This is due mainly to the lower-level layers being more general, while the last (e.g., top) layer is focused on a specific task. For example, for tasks including Flickr-style estimation (e.g., utilizes the Flicker data set), flower recognition, and places recognition, data scientists have trained the network by fine tuning an existing neural network trained for general image recognition.

Embodiments presented herein show how to train classifiers quickly while maintaining or improving the accuracy of the classifiers, such as by training the neural network in a supervised learning scenario. As used herein, the initial network to be trained is called the predecessor network, and the second neural network, trained based on the predecessor network, is called the successor network. The predecessor network is modified by eliminating the use of bias in the classifier, and the classifier of the successor network is initialized during the training phase based on the parameters learned from the predecessor network. The results are neural networks that train much faster while maintaining accuracy.

In one example, a method includes training a predecessor network defined for image recognition of items, the training of the predecessor network further comprising initializing parameters of a predecessor classifier with random numbers sampled from a predetermined distribution, and utilizing, by the predecessor classifier, an image-classification probability function without bias. The method further includes an operation for training a successor network defined for image recognition of items in a plurality of classes, the training of the successor network further comprising initializing parameters of a successor classifier with parameters learned from the predecessor network, and utilizing, by the successor classifier, the image-classification probability function without bias. The method further includes operations for receiving an image for recognition, and for recognizing the image utilizing the successor classifier.

In another aspect, a system includes a memory comprising instructions, and one or more computer processors. The instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising: training a predecessor network defined for image recognition of items, the training of the predecessor network further comprising initializing parameters of a predecessor classifier with random numbers sampled from a predetermined distribution, and utilizing, by the predecessor classifier, an image-classification probability function without bias; training a successor network defined for image recognition of items in a plurality of classes, the training of the successor network further comprising initializing parameters of a successor classifier with parameters learned from the predecessor network, and utilizing, by the successor classifier, the image-classification probability function without bias; receiving an image for recognition; and recognizing the image utilizing the successor classifier.

In another aspect, a non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising: training a predecessor network defined for image recognition of items, the training of the predecessor network further comprising initializing parameters of a predecessor classifier with random numbers sampled from a predetermined distribution, and utilizing, by the predecessor classifier, an image-classification probability function without bias; training a successor network defined for image recognition of items in a plurality of classes, the training of the successor network further comprising initializing parameters of a successor classifier with parameters learned from the predecessor network, and utilizing, by the successor classifier, the image-classification probability function without bias; receiving an image for recognition; and recognizing the image utilizing the successor classifier.

<FIG> illustrates the training of a classifier, according to example embodiments. A machined-learning algorithm is designed for recognizing faces, and a training set <NUM> includes data that maps a sample to a class <NUM> (e.g., a class includes all the images from one person). The classes may also be referred to as labels. Although embodiments presented herein are presented with reference to face recognition, the same principles may be applied to train machine-learning programs used for recognizing any type of items.

The training set <NUM> includes a plurality of images of persons (e.g., image <NUM>), and each image is associated with one of the persons to be recognized (e.g., a class). The machine-learning program is trained with the training data to generate a classifier <NUM> operable to recognize images. In example embodiments, the machine-learning program is a DNN.

When an input image <NUM> is to be recognized, the classifier <NUM> analyzes the input image <NUM> to identify the class (e.g., person <NUM>) corresponding to the input image <NUM>.

<FIG> illustrates the feature-extraction process and classifier training, according to example embodiments. Training the classifier may be divided into feature extraction layers <NUM> and classifier layer <NUM>. Each image is analyzed in sequence by a plurality of layers <NUM>-<NUM> in the feature-extraction layers <NUM>.

With the development of deep convolutional neural networks, the focus in face recognition has been to learn a good face feature space, in which faces of the same person are close to each other, and faces of different persons are far away from each other. For example, the verification task with the LFW dataset has been often used for face verification.

Many face identification tasks (e.g., MegaFace and LFW) are based on a similarity comparison between the images in the gallery set and the query set, which is essentially a K-nearest-neighborhood (KNN) method to estimate the person's identity. In the ideal case, there is a good face feature extractor (inter-class distance is always larger than the intra-class distance), and the KNN method is adequate to estimate the person's identity.

Feature extraction is a process to reduce the amount of resources required to describe a large set of data. When performing analysis of complex data, one of the major problems stems from the number of variables involved. Analysis with a large number of variables generally requires a large amount of memory and computational power, and it may cause a classification algorithm to overfit to training samples and generalize poorly to new samples. Feature extraction is a general term describing methods of constructing combinations of variables to get around these large data-set problems while still describing the data with sufficient accuracy for the desired purpose.

In some example embodiments, feature extraction starts from an initial set of measured data and builds derived values (features) intended to be informative and non-redundant, facilitating the subsequent learning and generalization steps. Further, feature extraction is related to dimensionality reduction, such as be reducing large vectors (sometimes with very sparse data) to smaller vectors capturing the same, or similar, amount of information.

Determining a subset of the initial features is called feature selection. The selected features are expected to contain the relevant information from the input data, so that the desired task can be performed by using this reduced representation instead of the complete initial data. DNN utilizes a stack of layers, where each layer performs a function. For example, the layer could be a convolution, a non-linear transform, the calculation of an average, etc. Eventually this DNN produces outputs by classifier <NUM>. In <FIG>, the data travels from left to right and the features are extracted. The goal of training the neural network is to find the parameters of all the layers that make them adequate for the desired task.

In some example embodiments, the structure of each layer is predefined. For example, a convolution layer may contain small convolution kernels and their respective convolution parameters, and a summation layer may calculate the sum, or the weighted sum, of two pixels of the input image. Training assists in defining the weight coefficients for the summation.

One way to improve the performance of DNNs is to identify newer structures for the feature-extraction layers, and another way is by improving the way the parameters are identified at the different layers for accomplishing a desired task. The challenge is that for a typical neural network, there may be millions of parameters to be optimized. Trying to optimize all these parameters from scratch may take hours, days, or even weeks, depending on the amount of computing resources available and the amount of data in the training set.

As discussed in more detail below, training of a neural network may be greatly accelerated by leveraging the parameters from a pre-trained neural network (e.g., predecessor network) to find the parameters for a successor network.

In some methods, the last operation in training is finding the parameters for the classifier <NUM>, which is initialized by using random samplings because there is no knowledge on the function used by the classifier. In example embodiments, the classifier <NUM> is initialized during training with parameters copied from the predecessor network, which greatly accelerates the training process. For example, a DNN may have been pre-trained to recognize dogs, cats, and other animals, but to obtain a classifier, using the same feature extractor, to recognize different fruits, requires very different parameters.

In example embodiments, the Softmax function is used as the final layer (classifier) <NUM>. The Softmax function, or normalized exponential function, is a generalization of the logistic function that "compresses" a K-dimensional vector z of arbitrary real values to a K-dimensional vector σ(z) of real values in the range [<NUM>, <NUM>] and that add up to <NUM>. The function σ(z) is calculated with the following equation:
<MAT>.

In probability theory, the output of the Softmax function can be used to represent a categorical distribution, e.g., a probability distribution over K different possible outcomes. The Softmax function is used in various multiclass classification methods, such as multinomial logistic regression, multiclass linear discriminant analysis, naive Bayes classifiers, and artificial neural networks.

<FIG> illustrates the classification of classes when using bias and with parameters randomly initialized, according to example embodiments. As used herein, "with parameters randomly initialized" refers to a neural network with a classifier that is trained without parameters being initialized from a previously-trained neural network. With parameters randomly initialized means that the parameters of the last layer, which is responsible for classification, are initialized using random numbers, as discussed in more detail below. On the other hand, with parameters initialized to have values which are estimated to approximate the optimal solution, embodiments presented initialize the parameters of the last layer using approximately optimal values based on previous trainings of the neural network, so the training time is significantly reduced. More details are provided below for a neural network with retraining in reference to <FIG> and <FIG>.

<FIG> illustrates one example of the features extracted with Softmax. The graph illustrates a hyperspace where the vectors associated with each class as situated. Class A area <NUM> includes the vectors for class A, class B area <NUM> for the vectors of class B, and class C area <NUM> for the vectors of class C, which means that the vectors for one class are close in space and have similar norms (e.g., length or size of the feature vector). A hyper-sphere <NUM> separates class A from class B and C. Thus, features having vectors with the same norm are located on the same hyper-sphere.

A common method for initial condition construction is to leverage an existing neural network, having similar structure to the target neural network, where the existing neural network has been trained for a general purpose (e.g., a process called fine-tuning). More specifically, at the initial stage, the lower-level layers of the model to be trained are typically set to have the same parameters as the pre-trained model, while the parameters of the last layer are set to be random numbers sampled from certain distributions (usually Gaussian). This is mainly because the lower-level layers tend to be more general, while the last layer is tightly defined for a specific task. For example, for tasks including Flickr style estimation, flower recognition, and places recognition, the network may be trained by fine-tuning from a neural network for general image recognition trained with ImageNet.

In example embodiments, the predecessor network is defined as having two parts: a feature extractor Ø(·), which corresponds to a stack of layers, and a Softmax layer. Typically, the Softmax layer estimates the probability yk(xi,w,b), for the kth component of the probability output y, that the ith sample xi belongs to the kth class, as follows:
<MAT>.

In equation (<NUM>), Ø(xi) non-linearly transforms the sample xi to the embedded feature domain, wk is the weight vector for the kth class, bk is the bias for the kth class, and Ø(xi) includes the features extracted from image xi. As used herein, unless otherwise specified, wk and bk refer to the weight vector and the bias of the classifier. T refers to the transpose operation.

<FIG> is a flowchart of a method <NUM> for training a machine-learning program for a specific task without retraining, according to example embodiments. While the various operations in this flowchart are presented and described sequentially, one of ordinary skill will appreciate that some or all of the operations may be executed in a different order, be combined or omitted, or be executed in parallel.

At operation <NUM>, a copy of the parameters of the feature extractor layers from a DNN configured for a general task is made, as discussed above. From operation <NUM>, the method flows to operation <NUM> to set an initial condition for the classifier, where the initial condition includes random samples from the predetermined distribution (e.g., Gaussian distribution).

From operation <NUM>, the method flows to operation <NUM> to learn the classifier, e.g., train the classifier with the training data based on the initial condition set at operation <NUM>.

<FIG> illustrates the classification of classes without using bias and with initial weight-parameter setting, according to example embodiments. In example embodiments, the bias term is eliminated (e.g., set to zero in the probability function. More specifically, the probability yk<NUM>(x,w) that the sample xi belongs to the kth class is calculated with equation (<NUM>) <NUM> as:
<MAT>.

In some example embodiments, the predecessor network is retrained with AlexNet and Residual Network using the ImageNet training data. The Softmax implementations using equations (<NUM>) and (<NUM>) empirically showed similar performance in terms of top-<NUM> accuracy on the validation set (<NUM>% with equation (<NUM>) and <NUM>% with equation (<NUM>)).

A possible explanation of the similar performance without the bias term b is that, without the bias term, the multi-layer neural network is highly capable of learning features for different classes locating in different cones and with similar norms, and thus can separate the different classes based on the weighting vectors. For example, class A <NUM>, class B <NUM>, and class C <NUM> are situated on the same feature sphere <NUM>, but separated from each other in different respective cones of the hyperspace.

Further, the lower-level layers of the successor network share the same parameters as the lower-level layers of the predecessor network (e.g., feature extraction layers). In example embodiments, the initial condition for the classifier of the successor network is changed with reference to the embodiments illustrated in <FIG>. The weight vector <MAT> for the classifier layer of the successor network is initialized as follows:
<MAT>.

Where Ck defines the set of the indexes for the samples which belongs to the kth class, and the parameter αk sets the norm of the weight vector to an empirical value. Further, by using equation (<NUM>), the weight vector <MAT> is initialized to have the same direction as the average of the vectors Ø(xi), where i ∈ Ck.

The feature vector Ø(xi) is extracted, using the predecessor network, from the ith image xi for the new, specific task. Ck is used to denote the set of the index for images which belong to the kth class. In other words, the right side of equation (<NUM>) is the averaged features for the kth class in the new specific task for the successor network. The rationale behind this operation is that the initial value of the weight vector <MAT> should have the same direction as the averaged feature value for the corresponding classes, since we aim to have <MAT> in Ck have a large value.

The previous equation (<NUM>) is used to initialize the weight vector for DNNs with Softmax as the last layer. In some example embodiments, the weight vector is initialized for DNNs with multiple binary classifiers as the last layer for multi-label scenarios. In this case, xk, the average of feature vectors within the kth cluster is calculated as:
<MAT>.

In addition, x\k, the average of feature vectors for all the samples except the kth cluster is calculated as:
<MAT>.

Thus, the weight vector <MAT> for the kth class may be initialized utilizing xk and x\k, as follows:
<MAT>.

Where the parameter βk is used to set the norm of the weight vector to an empirical value.

Comparing equations (<NUM>) and (<NUM>), it can be observed that the difference is the elimination of the bias b. Further, xi is an image and Ø(xi) is the features extracted from this image, with a weighting vector w. The bias b here acts as a scaler used to separate classes. When b is present, the classes can be located anywhere in the feature space, e.g., inside the hyper-sphere <NUM> or outside the hyper-sphere <NUM>; they can be anywhere.

However, when the bias term is removed, all the classes tend to locate themselves on the same hyper-sphere <NUM>, and all the vectors go through the coordinates origin point. The weighting vector wk has to go through the origin because there is no bias. This is a useful property for the feature extractor because when the feature extractor is applied on the new task, trained using the new class function, the different classes will locate themselves on the hyper-sphere as illustrated in <FIG>. They are separated, but on the same hyper- sphere. Further, the feature vectors from the different classes will have similar norms in the feature space, making it easier to find a separation (e.g., the parameters) to separate classes.

It may appear that the feature extractor won't operate as well without the bias term, but given the great flexibility of the feature extractor, the feature extractor, in the form of multi-layer neural network, can be trained to compensate this simpler functional form by separating the classes without the bias.

The advantage with this approach is that the feature extractor has to be trained once and the feature extractor doesn't have to be trained again and again. After the feature extractor is trained once, the feature extractor may be used for different tasks, and the training time with this approach is greatly reduced. It is also believed that if the same amount of time were used to train with this approach (by using more iterations), then the accuracy would be better than with the algorithm illustrated in <FIG>.

<FIG> is a flowchart of a method <NUM> for training the machine-learning program with retraining, according to example embodiments. While the various operations in this flowchart are presented and described sequentially, one of ordinary skill will appreciate that some or all of the operations may be executed in a different order, be combined or omitted, or be executed in parallel.

The method <NUM> includes operations <NUM>-<NUM> to train the first classifier (the predecessor network) and operations <NUM>-<NUM> to train the second classifier (the successor network). At operation <NUM>, the parameters of the feature extractor layers are copied from a DNN configured for performing general tests.

From operation <NUM>, the method flows to operation <NUM> for selecting a cost function with zero bias, such as equation (<NUM>). From operation <NUM>, the method flows to operation <NUM> where the initial condition for the classifier is set. As discussed earlier, the initial condition is based on a random distribution.

From operation <NUM>, the method flows to operation <NUM> were the first classifier is learned by performing the training operation with the training data. From operation <NUM>, the method flows to operation <NUM> when training for the second classifier is started. At operation <NUM>, the parameters of the feature extractor layers are copied from the DNN for the general task.

From operation <NUM>, the method flows to operation <NUM> where the cost function with zero bias is selected, such as equation (<NUM>). From operation <NUM>, the method flows to operation <NUM>, where the initial condition is set based on the parameters of the first classifier. See for example equation (<NUM>). From operation <NUM>, the method flows to operation <NUM> to learn the second classifier.

<FIG> illustrates some experimental measurements showing the faster tool-training times with the successor network, according to example embodiments. It is noted that the experimental results illustrated in <FIG> are examples and do not describe every possible embodiment. The results should therefore not be interpreted to be exclusive or limiting, but rather illustrative.

The charts <NUM>, <NUM>, <NUM>, and <NUM> illustrate the accuracy results based on the number of iterations for the training, for both the "random" initial distribution for the classifier and the preset condition set based on the predecessor network, referred to as "mean.

The method was evaluated in a multi-class classification setup. The AlexNet DNN was trained with ImageNet data, with the modified Softmax, as illustrated in equation (<NUM>) as the predecessor network, and estimated the initial condition by setting the last layer of the successor network using equation (<NUM>).

The parameters of the lower-level layers were fixed. The tasks tested include Flickr-style estimation <NUM>, flower recognition <NUM>, and places recognition <NUM>. The method "mean" includes setting the parameters of the last layer using equation (<NUM>), while the method "random" includes setting the parameters of the last layer using random numbers following a certain distribution, which is the conventional method. As shown, the "mean" method reduces the number of iterations needed to achieve the same performance by three to five times.

Chart <NUM> also shows the result when using a different feature extractor for face recognition, as a reference to experiment with a different extractor, which showed similar results.

<FIG> is a flowchart of a method for training a deep neural network (DNN), according to example embodiments. While the various operations in this flowchart are presented and described sequentially, one of ordinary skill will appreciate that some or all of the operations may be executed in a different order, be combined or omitted, or be executed in parallel.

Operation <NUM> is for training, by one or more processors, a predecessor network defined for image recognition of items. The training of the predecessor network further comprises operations <NUM> and <NUM>. Operation <NUM> is for initializing parameters of a predecessor classifier with random numbers sampled from a predetermined distribution, and operation <NUM> is for utilizing, by the predecessor classifier, an image-classification probability function without bias.

From operation <NUM>, the method flows to operation <NUM> for training, by the one or more processors, a successor network defined for image recognition of items in a plurality of classes. The training of the successor network further comprises operations <NUM> and <NUM>. Operation <NUM> is for initializing parameters of a successor classifier with parameters learned from the predecessor network, and operation <NUM> is for utilizing, by the successor classifier, the image-classification probability function without bias.

From operation <NUM>, the method flows to operation <NUM> where an image is received for recognition. At operation <NUM>, the one or more processors recognize the image utilizing the successor classifier.

In one example, initializing parameters of the successor classifier further comprises initializing a weight vector wk for a class k based on a set of indexes for training samples which belong to the k class.

Initializing parameters of the successor classifier further comprises initializing the weight vector wk for the class k further based on an average direction of vectors for the class k in the predecessor network.

In one example, vectors for a same class are located substantially on a same hyper-sphere.

The predecessor network includes a plurality of feature-extraction layers and the predecessor classifier, where the successor network includes the plurality of feature-extraction layers and the successor classifier.

In one example, the image-classification probability function for a sample image x in class k is based on a weight vector w and features extracted from the sample image x by the feature-extraction layers.

In one example, the feature-extraction layers of the successor network share same parameters as the feature-extraction layers of the predecessor network.

<FIG> is a block diagram illustrating an example of a machine <NUM> upon which one or more example embodiments may be implemented. The machine <NUM> may be a personal computer (PC), a tablet PC, a set-top box (STB), a laptop, a mobile telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine <NUM> is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), or other computer cluster configurations.

Examples, as described herein, may include, or may operate by, logic or a number of components or mechanisms. Circuitry is a collection of circuits implemented in tangible entities that include hardware (e.g., simple circuits, gates, logic, etc.). Circuitry membership may be flexible over time and underlying hardware variability. Circuitries include members that may, alone or in combination, perform specified operations when operating. In an example, hardware of the circuitry may be immutably designed to carry out a specific operation (e.g., hardwired). In an example, the hardware of the circuitry may include variably connected physical components (e.g., execution units, transistors, simple circuits, etc.) including a computer-readable medium physically modified (e.g., magnetically, electrically, by moveable placement of invariant massed particles, etc.) to encode instructions of the specific operation. In connecting the physical components, the underlying electrical properties of a hardware constituent are changed, for example, from an insulator to a conductor or vice versa. The instructions enable embedded hardware (e.g., the execution units or a loading mechanism) to create members of the circuitry in hardware via the variable connections to carry out portions of the specific operation when in operation. Accordingly, the computer-readable medium is communicatively coupled to the other components of the circuitry when the device is operating. In an example, any of the physical components may be used in more than one member of more than one circuitry. For example, under operation, execution units may be used in a first circuit of a first circuitry at one point in time and reused by a second circuit in the first circuitry, or by a third circuit in a second circuitry, at a different time.

The machine (e.g., computer system) <NUM> may include a hardware processor <NUM> (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory <NUM>, and a static memory <NUM>, some or all of which may communicate with each other via an interlink (e.g., bus) <NUM>. The machine <NUM> may further include a display device <NUM>, an alphanumeric input device <NUM> (e.g., a keyboard), and a UI navigation device <NUM> (e.g., a mouse). In an example, the display device <NUM>, input device <NUM>, and UI navigation device <NUM> may be a touchscreen display. The machine <NUM> may additionally include a mass storage device (e.g., drive unit) <NUM>, a signal generation device <NUM> (e.g., a speaker), a network interface device <NUM>, and one or more sensors <NUM>, such as a GPS sensor, compass, accelerometer, or other sensor. The machine <NUM> may include an output controller <NUM>, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., a printer, card reader, etc.).

The mass storage device <NUM> may include a machine-readable medium <NUM> on which is stored one or more sets of data structures or instructions <NUM> (e.g., software) embodying or utilized by any one or more of the techniques or functions described herein. The instructions <NUM> may also reside, completely or at least partially, within the main memory <NUM>, within the static memory <NUM>, or within the hardware processor <NUM> during execution thereof by the machine <NUM>. In an example, one or any combination of the hardware processor <NUM>, the main memory <NUM>, the static memory <NUM>, or the mass storage device <NUM> may constitute machine-readable media.

While the machine-readable medium <NUM> is illustrated as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) configured to store the one or more instructions <NUM>.

The term "machine-readable medium" may include any medium that is capable of storing, encoding, or carrying instructions <NUM> for execution by the machine <NUM> and that causes the machine <NUM> to perform any one or more of the techniques of the present disclosure, or that is capable of storing, encoding or carrying data structures used by or associated with such instructions <NUM>. Non-limiting machine-readable medium examples may include solid-state memories, and optical and magnetic media. Specific examples of machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

The instructions <NUM> may further be transmitted or received over a communications network <NUM> using a transmission medium via the network interface device <NUM> utilizing any one of a number of transfer protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), mobile telephone networks (e.g., cellular networks), Plain Old Telephone Service (POTS) networks, wireless data networks (e.g., Institute of Electrical and Electronics Engineers (IEEE) <NUM> family of standards known as Wi-Fi®, IEEE <NUM> family of standards known as WiMax®), IEEE <NUM>. <NUM> family of standards, peer-to-peer (P2P) networks, among others. In an example, the network interface device <NUM> may include one or more physical jacks (e.g., Ethernet, coaxial, or phone jacks) or one or more antennas to connect to the communications network <NUM>. In an example, the network interface device <NUM> may include a plurality of antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) techniques. The term "transmission medium" shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions <NUM> for execution by the machine <NUM>, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.

Claim 1:
A system comprising:
a memory (<NUM>, <NUM>) comprising instructions (<NUM>); and
one or more computer processors (<NUM>), wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
training a predecessor deep neural network defined for image recognition of items, the training of the predecessor deep neural network further comprising:
initializing parameters of a predecessor softmax layer with random numbers sampled from a predetermined distribution; and
utilizing, by the predecessor softmax layer, an image-classification probability function without bias;
training a successor deep neural network defined for image recognition of items in a plurality of classes, the training of the successor deep neural network further comprising:
initializing parameters of a successor softmax layer with parameters learned from the trained predecessor deep neural network, the initializing parameters of the successor softmax layer comprising initializing a weight vector wk of the successor softmax layer for a class k (<NUM>) based on an average direction of feature vectors for the class k in the trained predecessor deep neural network; and
utilizing, by the successor softmax layer, the image-classification probability function without bias;
receiving an image (<NUM>) for recognition; and
recognizing the image utilizing the successor deep neural network, wherein the predecessor deep neural network includes a plurality of feature-extraction layers (<NUM>) and the predecessor softmax layer, wherein the successor deep neural network includes the plurality of feature-extraction layers and the successor softmax layer, and wherein the parameters of the feature-extraction layers of the predecessor and the successor deep neural network are copied from a deep neural network configured for a general task.