Patent Description:
Neural networks are machine learning models that employ one or more layers of operations to generate an output, e.g., a classification, for a received input. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer of the network. Some or all of the layers of the network generate an output from a received input in accordance with current values of a respective set of parameters.

Some neural networks include one or more convolutional neural network layers. Each convolutional neural network layer has an associated set of kernels. Each kernel includes values established by a neural network model created by a user. In some implementations, kernels identify particular image contours, shapes, or colors. Kernels can be represented as a matrix structure of weight inputs. Each convolutional layer can also process a set of activation inputs. The set of activation inputs can also be represented as a matrix structure.

<NPL> discloses a unified end-to-end approach to build a large scale Visual Search and Recommendation system for e-commerce.

<NPL> propose Knowledge Adaptation, an extension of Knowledge Distillation (Bucilua et al. , <NUM>; Hinton et al. , <NUM>) to a domain adaptation scenario. The authors show how a student model achieves state-of-the-art results on unsupervised domain adaptation from multiple sources on a standard sentiment analysis benchmark by taking into account the domain-specific expertise of multiple teachers and the similarities between their domains.

One aspect of the subject matter described in this specification is embodied in a computer-implemented method for generating a unified machine learning computing model according to claim <NUM>.

One aspect of the subject matter described in this specification is embodied in a system according to claim <NUM>.

One aspect of the subject matter described in this specification is embodied in one or more non-transitory machine-readable storage devices according to claim <NUM>.

The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages. Object recognition has received increased attention in vision research. In this context, the described teachings include processes for using a neural network to generate a unified machine learning model using an L2-loss function, where the unified model can be used to identify or recognize a variety of objects for multiple object verticals (e.g., dresses, handbags, shoes).

For example, given image data depicting representations of a piece of garment, the unified model generated according to the described teachings can be used to locate or retrieve the same or similar items. In some instances, an item's appearance can change with lighting, viewpoints, occlusion, and background conditions. Distinct object verticals can also have different characteristics such that images from a dress vertical may undergo more deformations than those from a shoe vertical. Hence, because of these distinctions, separate models are trained to identify items in each object vertical.

However, separate specialized models require substantial resources for model storage and added computational demands to support deployment of multiple models. These resource burdens may become more severe when multiple models are used on mobile platforms. Hence, a unified model for object recognition across different apparel verticals can reduce processor utilization and provide increases in computing efficiency of an example object recognition system. Moreover, object retrieval functions are performed efficiently by combining multiple specialized models in a single unified model that uses a smaller computational footprint. This results in a technological improvement in the technological area of model generation and use.

Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

Machine learning systems can be trained, using deep neural networks, to recognize particular categories of items based on learned inferences. Deep neural networks can generate inferences based on analysis of input data received by the machine learning system. Trained machine learning systems may produce or generate one or more specialized models that use particular sets of learned inferences for identification or recognition of particular categories of items.

For example, in the context of apparel recognition, specialized models can be trained to recognize items associated with a particular category or object vertical, such as dresses, pants, or handbags in image data to which the model is applied. In alternative implementations, a category or object vertical can correspond to a variety of items or objects, such as automobiles, animals, human individuals, and various physicals objects, for example represented in image data. Object vertical also corresponds to audio signal data.

In general, specialized models may significantly outperform general models, e.g., models that are trained to recognize items associated with a wide range of object verticals. Therefore, item recognition models, generated using deep neural networks, are often trained separately for different object verticals. An object vertical defines a distinct category for an object that belongs to the vertical, e.g., for apparel, object verticals may be the item categories of hats, shoes, shirts, jackets, etc.). However, an item recognition system that includes groups of specialized models for identifying items of different object verticals can be expensive to deploy and may not be sufficiently scalable.

In this context, the subject matter described in this specification includes systems and methods for generating a unified machine learning model using a neural network on a data processing apparatus. The unified embedding model is generated using a deep neural network that utilizes learning targets indicated by embedding outputs generated by respective specialized models. The neural network (or deep neural network) is used by an example machine learning system to learn inferences for identifying a variety of items across multiple object verticals, e.g., item categories corresponding to various apparel classes.

For example, a machine learning system that includes a neural network can access respective learning targets for each object vertical in a group of object verticals. The system can determine the respective learning targets based on two or more embedding outputs of the neural network. Each embedding output can be generated by respective specialized models that are individually trained using a triplet loss function and to identify data (e.g., an image of a luxury handbag) associated with a particular object vertical (e.g., handbags).

The data processing apparatus of the system trains the neural network to identify data associated with each vertical in the group of object verticals. The neural network can be trained using the respective learning targets of the specialized models and based on an L2-loss function. The data processing apparatus uses the trained neural network to generate a unified machine learning model configured to identify particular data items (e.g., Brand name sneakers, Luxury purse, Luxury blouse, etc.) associated with each object verticals in the group (e.g., shoes, handbags, tops/shirts, etc.).

<FIG> illustrates a neural network system architecture <NUM> ("system <NUM>") for generating an example machine learning model based on a first loss function. Generating a machine learning model can include system <NUM> performing neural network computations associated with inference workloads. In particular, computations for inference workloads can include processing neural network inputs (e.g., input activations) through layers of a neural network. Each layer of a neural network can include a set of parameters (e.g., weights) and processing an input through a neural network layer can include computing dot products using the input activations and parameters as operands for the computations.

System <NUM> generally includes an example neural network indicated by neural net architecture <NUM>. A neural network of architecture <NUM> can include a base network <NUM>, a pooling layer <NUM>, a first connected layer set <NUM>, a second connected layer set <NUM>, and embedding outputs <NUM>. Base network <NUM> can include a subset of neural network layers of architecture <NUM>.

For example, a deep neural network can include a base network <NUM> that includes multiple convolutional layers. These convolutional layers can be used to perform complex computations for computer based recognition of various items included in a variety of image data. In some implementations, base network <NUM> can be inception v2, an inception v3, an inception v4, or another related neural net structure. Although described in the context of image data, the processes of this specification can be applied to detection or recognition of audio signal data.

Architecture <NUM> can include a variety of additional neural network layers that perform various functions associated with inference computations for training a machine learning model. For example, pooling layer <NUM> can be an average pooling layer or max pooling layer that perform functions related to pooling output activations for down-sampling operations. The down-sampling operations can reduce a size of output datasets by modifying certain spatial dimensions that relate to an input dataset.

Connected layer sets <NUM>, <NUM> can be respective sets of fully connected layers that include artificial neurons that have full connections to all activations in a previous layer. Embedding outputs <NUM> can correspond to one or more output feature sets that include a vector of floats/parameters for given output dimension (<NUM>-d, <NUM>-d, etc.). As described in more detail below, embedding outputs <NUM> are formed, produced, or generated, when the example neural network of system <NUM> is trained to perform certain computational functions for object/item recognition or identification.

System <NUM> can include one or more processors and other related circuit components that form one or more neural networks. In general, methods and processes described in this specification can be implemented using a variety of processor architectures, such as Central Processing Units (CPUs), Graphics Processing Units (GPUs), digital signal processors (DSPs), or other related processor architectures.

System <NUM> can include multiple computers, computing servers, and other computing devices that each include processors and memory that stores compute logic or software instructions that are executable by the processors. In some implementations, system <NUM> includes one or more processors, memory, and data storage devices that collectively form one or more architecture <NUM>. Processors of architecture <NUM> process instructions for execution by system <NUM>, including instructions stored in the memory or on the storage devices. Execution of the stored instructions can cause performance of the machine learning processes described herein.

Referring again to <FIG>, system <NUM> is configured to perform a variety of computing operations related to machine learning processes. For example, system <NUM> performs learning operations <NUM> and <NUM> as well as a variety of other operations related to training a neural network to generate one or more specialized machine learning models. In some implementations, system <NUM> executes programmed code or software instructions to perform computations associated with learning operations <NUM> and <NUM>. As described in more detail below, learning operations <NUM> and <NUM> are executed by system <NUM> to train respective specialized learning models based on a triplet loss function indicated by computing logic <NUM>.

Learning operation <NUM> includes system <NUM> using the neural network of architecture <NUM> to generate model training data. The model training data corresponds to embedding outputs that are produced by system <NUM> when the system is trained to generate a particular specialized model. System <NUM> generates multiple distinct specialized models and produces individual sets of embedding outputs, where a particular set of embedding outputs corresponds to a particular specialized model.

For example, in the context of apparel recognition or apparel retrieval, separate specialized models can be generated to recognize and retrieve apparel items for different apparel categories (e.g., dresses, tops, handbags, etc.). In some implementations, embedding models for recognizing images from various websites or other user produced images (e.g., "street" or "real life" digital images captured using mobile devices/smartphones) can be learned using separate sub-networks.

During model training, a sub-network for each vertical such as dresses, handbags, eyewear, and pants can be fine-tuned independently. For these sub-networks, the results of the model training can enable a machine learning system to produce up to eleven separate specialized models that each correspond to one of eleven verticals. As used herein, a "vertical" or "object vertical" corresponds to an object or item category. For apparel recognition, an object vertical can be an apparel item category such as dresses, handbags, eyewear, pants, etc. As discussed in more detail below, using separate models for object recognition of items (e.g., apparel items) in a particular category/vertical can result in substantially accurate item recognition results.

In some implementations, system <NUM> trains a neural network of architecture <NUM> using image data for apparel items that are each associated with different apparel categories. System <NUM> uses image data for multiple different types of handbags to train the neural network. System <NUM> can then generate a specialized model to identify or recognize particular types of handbags based on embedding outputs that are produced in response to training the neural network.

As indicated above, a particular set of embedding outputs corresponds to a particular specialized model. For example, in apparel recognition, a first set of embedding outputs can correspond to neural network training data for learned inferences used to generate a first model for recognizing certain shirts/blouses or tops (e.g., see operation <NUM>). Similarly, a second set of embedding outputs can correspond to neural network training data for learned inferences used to generate a second model for recognizing certain jeans/pants/skirts or bottoms.

Each set of embedding outputs include embedding feature vectors that can be extracted and used for object or item retrieval. As described in more detail below, extracted sets of embedding feature vectors correspond to respective learning targets and an embedding output of a trained neural network model includes these embedding feature vectors.

One or more learning targets can be used to train a machine learning system (e.g., system <NUM>) to generate particular types of specialized computing models. For example, as discussed below with reference to features of <FIG>, multiple distinct learning targets can be used to train at least one unified machine learning model that recognizes items that are associated with multiple different verticals or categories.

Referring again to <FIG>, system <NUM> executes learning operation <NUM> to generate specialized learning models based on model training data determined at learning operation <NUM>. In some instances, determining or generating model training data corresponds to an example process of "learning" individual embedding models. In some implementations, for model training and feature vector extraction, system <NUM> uses a two-stage approach when training (e.g., a first stage) and when extracting embedding feature vectors for object retrieval (e.g., a second stage).

In the context of apparel retrieval from image data, the first stage can include localizing and classifying an apparel item of the image data. In some instances, classifying an apparel item of the image data includes system <NUM> determining an object class label for the apparel item of the image data. For example, system <NUM> can use an example object detector to analyze the image data. System <NUM> can then use the analysis data to detect an object or apparel item of the image data that includes object attributes that are associated with handbags. Based on this analysis and detection, system <NUM> can then determine that the object class label for the apparel item is a "handbag" class label.

In some implementations, system <NUM> includes object detection architecture that is a single-shot multi-box (SSD) detector for a base network <NUM> that is an inception V2 base network. In other implementations, system <NUM> can be configured to use or include a variety of other object detection architectures and base network combinations. The SSD can be an example computing module of system <NUM> that executes program code to cause performance of one or more object detection functions.

For example, this SSD detector module can provide bounding boxes that bound an object of the image data. The SSD can further provide apparel class labels that indicate whether the bounded object is a handbag, an eyewear item, or a dress. In some implementations, object pixels of the image data can be cropped and various features can then be extracted on the cropped image using a particular embedding model of system <NUM>. Sub-process steps associated with the first stage of the two-stage process can be used to train a specialized embedding model based on a variety of image data.

In response to determining an object class label at the first stage, system <NUM> can proceed to the second stage and train specialized embedding models to compute similarity features for object retrieval. For example, system <NUM> can perform embedding model training using a triplet loss function indicated by computing logic <NUM>. More specifically, system <NUM> uses triplet ranking loss to learn feature embeddings for each individual item/object vertical or category.

For example, a triplet includes an anchor image, a positive image, and a negative image. During triplet learning, system <NUM> seeks to produce embeddings such that the positive image gets close to the anchor image while the negative is pushed away from the anchor image in a feature space of the neural network. Embeddings learned from triplet training are used to compute image similarity.

For example, let <MAT> be a triplet, where <MAT> represent the anchor image, positive image and negative image respectively. A learning goal of system <NUM> is to minimize a computational output associated with the following loss function shown as equation (<NUM>), <MAT> where: i) α is the margin enforced between the positive and negative pairs, ii) f(I) is the feature embedding for image I, and iii) D(fx , fy) is the distance between the two feature embeddings fx and fy.

For implementations that involve training separate models for particular apparel object verticals, a positive image is the same product (e.g., Chanel handbag) as the anchor image, while the negative image is of another product but in the same apparel vertical (e.g., Luxury handbag). In some implementations, system <NUM> executes computing logic for semi-hard negative mining functions for obtaining negative image data. For example, system <NUM> can access online/web-based resources and use semi-hard negative mining to identify strong negative object images. System <NUM> can then use these object images to enhance or improve effectiveness of the training for a particular specialized model.

As indicated above, model training based on the triplet loss function of logic <NUM> produces training data such as embedding outputs that include feature vectors. Extracted sets of embedding feature vectors can correspond to respective learning targets. Hence, at learning operation <NUM>, system <NUM> can determine respective learning targets based on the triplet loss model training data. System <NUM> can then use two or more of the learning targets to train a machine learning system (e.g., system <NUM> of <FIG>) to generate at least one unified computing model, as described below.

<FIG> illustrates a neural network system architecture <NUM> ("system <NUM>") for generating an example machine learning model based on a second loss function, e.g., an L2-loss function. As shown, system <NUM> includes substantially the same features as system <NUM> described above. However, system <NUM> includes an L2 normalization layer <NUM> that is described in more detail below. In some implementations, system <NUM> is a sub-system of system <NUM> and can be configured to execute the various computing functions of system <NUM> described above.

System <NUM> is configured to learn or generate a unified embedding model that is trained based on learned inferences. These learned inferences enable object recognition of various item groupings, where each grouping corresponds to distinct object verticals or categories (e.g., apparel categories). System <NUM> learns one or more unified models by combining training data produced when system <NUM> is used to train respective specialized models as described above. In some related model learning/training scenarios, combining training data from separate models to generate a unified model can cause performance degradation when triplet loss is used to train the unified model.

For example, using the triplet loss function of logic <NUM> to train a unified model based on combined training data (e.g., embedding outputs <NUM>) of the different object verticals can lead to significant drops in recognition accuracy compared to the accuracy of models trained for each individual vertical. These decreases can occur as more object categories (or verticals) are accumulated into a single unified model. However, as described in more detail below, when combining embeddings from multiple specialized model to generate a single unified model, reducing training difficulty and training complexity can lead to substantial improvements in performance and recognition accuracy.

According to the described teachings, system <NUM> can generate a unified embedding model that achieves equivalent performance and recognition accuracy when compared to individual specialized models. Further, the unified model can have the same, or even less, model complexity as a single individual specialized model. Hence, this specification describes improved processes and methods for easing or reducing the difficulties in training model embeddings for multiple verticals such that a unified model can be generated.

For example, separate specialized models can be first trained to achieve a desired threshold level of accuracy for recognizing objects included in image data. As indicated above, the separate models can be trained using system <NUM> and based on the triplet loss function of computing logic <NUM>. Embedding outputs of each separately trained model are then used as the learning targets to train an example unified model.

In some implementations, a particular specialized model can have an example accuracy metric of <NUM>, where the model accurately identifies certain handbags <NUM>% (<NUM>) of the time. A unified model generated according to the described teachings achieves accurate object recognition results that exceed an accuracy metric of the objection results of the particular specialized model (e.g., <NUM>). For example, in the context of apparel recognition, a unified model generated according to the described teachings can have an object retrieval or recognition accuracy of <NUM> accuracy metric, or <NUM> percent accuracy, for a handbags apparel category.

In some implementations, accurately recognizing/identifying an object using the unified model includes determining a category of the object (e.g., "handbag"), determining an owner or designer of the object (e.g., "Chanel" or "Gucci"), and/or determining a type/style of the handbag (e.g., "Chanel <NUM> classic flap bag"). In some instances, identifying an object by the unified model an include the model retrieving (e.g., object retrieval) associated image data that includes a graphical representation of the object.

Referring again to <FIG>, system <NUM> is configured to generate a unified model that can execute multiple tasks for accurate object recognition and retrieval that, in prior systems, are performed by separated models but with reduced accuracy. Moreover, the described teachings include methods and processes for improvements in emulating separate model embeddings outputs (e.g., learning targets) through use of an L2 loss function.

For example, training a unified model based on triplet loss and by combining training data for two different object verticals (e.g., handbags and shoes) can generate a unified model that performs object recognition of items in those verticals with reasonable accuracy. However, using triplet loss when combining training data for three or more different object verticals may result in a unified model that performs with substantially poor object recognition accuracy. The poor accuracy results from difficult and complex computing challenges that occur when training a unified model based on a triplet loss function for several distinct verticals.

To ease this training difficulty, this specification proposes a learning scheme that uses embedding outputs from specialized models as learning targets such that L2-loss can be used instead of triplet loss. Use of the L2-loss function eases the training difficult with generating the unified model and provides for more efficient use of a neural network's feature space. The end result is a unified model that can achieve the same (or greater) retrieval accuracy as a number of separate specialized models, while having the model complexity of a single specialized model.

System <NUM> uses the respective learning targets of the separate models to learn a unified learning model such that embeddings generated from this unified model are the same as (or very close to) the embeddings of separate specialized models generated by system <NUM>. In some implementations, system <NUM> uses a neural network of architecture <NUM> to determine respective learning targets for each object vertical in a group of object verticals. Each of the respective learning targets is based on a particular embedding output of the neural network.

<FIG> shows computing operations for generating a unified machine learning model. For example, at learning operation <NUM> of <FIG>, system <NUM> accesses learning targets that correspond to feature embeddings for the respective specialized models. At learning operation <NUM>, system <NUM> generates unified model training data that correspond to feature embeddings for detecting objects of various verticals. The feature embeddings are based on neural network inference computations that occur during unified model learning.

For example, let <MAT>, where each Vi is a set of verticals whose data can be combined to train an embedding model. Let <MAT> be a set of embedding models, where each Mi is the model learned for vertical set Vi. Let <MAT> be a set of N training images. If the vertical-of- Ij ∈ Vs, s = <NUM>. K, its corresponding model Ms is used to generate embedding features for image Ij. Let fsj denote the feature embeddings generated from Ms for image Ij.

At learning operation <NUM>, system <NUM> generates a unified machine learning model configured to identify particular items included in example image data. The image data is associated with each object vertical of the group and the unified model is generated using a neural network trained based on a particular loss function (e.g., L2-loss). For example, system <NUM> is configured to learn a model U, such that the features produced from model U are the same as features produced from the separate specialized models generated by system <NUM>.

In particular, let fuj denote the feature embeddings generated from model U. A learning goal of system <NUM> is to determine a model U which can minimize a computational output associated with the following loss function shown as equation (<NUM>).

With reference to equation (<NUM>), features fuj are computed from model U, while features fsj can be computed from the different specialized models described above. The above model learning description uses the L2-loss function indicated by computing logic <NUM> and equation (<NUM>) above, instead of the triplet loss function indicated by computing logic <NUM> and equation (<NUM>) above.

In some implementations, system <NUM> is configured to generate a unified model that has an output dimension <NUM> that is <NUM>-d. In contrast, a single specialized model generated at learning operation <NUM> can have an output dimension <NUM>, e.g., <NUM>-d, that is larger than, or substantially larger than, the <NUM>-d output dimension of the unified model. As noted above, use of L2-loss provides for a less complex and less difficult training process than triplet loss.

Additionally, use of the L2-loss function provides for less complexity and difficulty in the application of learning techniques, such as batch normalization. For example, with batch normalization, neural network layer inputs can be normalized to allow for higher learning rates. For an example image classification model, batch normalization can be used to achieve desired threshold accuracy metrics (e.g., <NUM> or higher) with fewer training steps when compared to learning techniques used with other loss functions.

In some implementations, covariate shift can be minimized in response to system <NUM> performing batch normalization functions that are applied via L2 normalization layer <NUM>. For example, deep neural networks can include multiple layers in a sequence. Training deep neural networks is often complicated by the fact that a distribution of each layer's inputs can change during model training, for example, as parameters of previous layers in a sequence change.

Such changes can slow down a speed with which a model can be trained using a deep neural network, thereby requiring slower learning rates and careful parameter initialization. Parameter changes that adversely affect training speed can be described as neural network internal covariate shift. However, by using L2-loss, batch normalization processes for normalizing layer inputs can be performed to resolve or minimize adverse effects on training speed that are caused by covariate shift.

Further, a learning approach that uses L2-loss to generate a unified model allows for the use of increased amounts of unlabeled data relative to triplet loss. For example, with the triplet loss learning approach for training apparel recognition models, a product identity (e.g. "Chanel <NUM> classic flap bag") can be required to generate embeddings data for the training triplet. However, a model training and learning approach that uses L2-loss requires only the vertical labels, which can be generated automatically by an example localization/classification model. Hence, use of L2-loss can reduce processor utilization and increase system bandwidth for additional computations by foregoing computations for determining product identities.

Alternatively, the described teachings also include methods for selecting vertical data combinations for producing a particular model (e.g., a unified model or other related combined model forms) that can be used to reduce a number of specialized models. This particular combined model can be successfully learned and can have a comparable object recognition accuracy that is similar to, the same as, or greater than the recognition accuracy of each separate specialized model.

Selective or "smart" vertical combination is used to determine a combined model (e.g., an example unified model). System <NUM> includes computing logic for determining which embeddings data from different verticals for specialized models can be combined to produce an example combined model. In particular, starting with a first vertical, system <NUM> progressively adds embeddings data from other verticals. While adding embeddings data, system <NUM> can perform sample item recognition tasks to determine whether a model learned from the combined data causes observed accuracy degradation.

System <NUM> can steadily add embeddings data from other verticals until accuracy degradation is observed. In other instances, system <NUM> determines a particular combination of verticals for a number of specialized models, where each specialized model is used for item recognition across a subset of verticals. Further, system <NUM> can determine a particular combination of verticals for a number of specialized models while also maintaining a threshold level of accuracy. System <NUM> can then use feature embeddings for specialized models that correspond to particular verticals in the subset and produce a combined model based on the feature embeddings.

<FIG> shows graphical representations of embeddings data <NUM> for different object verticals in a feature space of an example neural network. The graphical representations generally indicate that a unified model trained (e.g., learned) based on the described teachings can provide more efficient and broader use of a feature space of the neural network. For example, the described learning approach that uses L2-loss can efficiently train a unified model by taking advantage of pre-established feature mappings (e.g., learning targets) learned for separate specialized models.

Embeddings data <NUM> includes t-distributed stochastic neighbor embedding (t-SNE) visualizations generated from feature embeddings of separate specialized models. In particular, embeddings data <NUM> includes two thousand images from each vertical <NUM>, <NUM>, and <NUM>, where the date is projected down to 2D space for visualization.

<FIG> indicates that the feature embeddings fsj are separated across verticals <NUM>, <NUM>, <NUM> in the feature space. In other words, an embedding model for each vertical fsj (from model Ms) uses only a part of the dimensional space (e.g., <NUM>-d), and therefore one unified model can be learned to combine embeddings outputs for each apparel vertical included in embedding of data <NUM> (e.g., <NUM> total verticals).

<FIG> is an example flow diagram of a process for generating a unified machine learning model for multiple object verticals based on a particular loss function. Process <NUM> corresponds to an improved process for generating unified machine learning models, where the generated models have item recognition accuracy metrics that are at least equal to an accuracy metric of two or more distinct specialized models. Process <NUM> can be implemented using system <NUM> or <NUM> described above, where system <NUM> can perform all described functionality associated with sub-system <NUM>.

Process <NUM> includes block <NUM> where system <NUM> determines respective learning targets for each object vertical in a group of object verticals. In some implementations, a neural network of system <NUM> determines respective learning targets based on two or more embedding outputs of the neural network.

For example, the object verticals can be apparel categories, such as a dresses, shoes, or handbags. Further, each vertical can correspond to an embedding output that is produced when a particular model is trained to identify or recognize apparel or clothing items in a vertical. Example apparel items can include cocktail dresses, basketball sneakers, or brand name monogram handbags.

At block <NUM> of process <NUM>, system <NUM> trains the neural network to identify data associated with each object vertical in the group based on a first loss function (e.g., L2-loss). In some implementations, the neural network is trained using the respective learning targets that were determined for each object vertical. For example, given an image file or image data, system <NUM> can train the neural network to at least: i) identify a dress item in an image based on analysis of pixel data of the image; ii) identify a shoe item in an image based on analysis of pixel data of the image; or iii) identify a handbag item in an image based on analysis of pixel data of the image.

At block <NUM>, system <NUM> generates, a unified machine learning model that is configured to identify items that are included in the data associated with each object vertical of the group of verticals. For example, data processing apparatus of system <NUM> can use the neural network trained based on the first loss function to generate the unified machine learning model that performs one or more of the object recognition functions described herein.

In some instances, determining the respective learning targets includes: i) training the neural network to identify data associated with each of the object verticals, where the neural network is trained based on a second loss function; and ii) generating at least two embedding outputs, where each embedding output indicates a particular learning target of the respective learning targets. In addition to indicating a particular learning target, each embedding output can include a vector of floats (or parameters) that correspond generally to attributes of the image data associated with a particular object vertical.

In some implementations, system <NUM> generates respective machine learning models, where each of the models are generated using the neural network trained based on the second loss function (e.g., triplet loss) that is different than the first loss function. Moreover, each of the models may use a vector of floats for a particular embedding output to identify apparel or clothing items for a particular object vertical. In some instances, generating the embeddings occurs in response to training the neural network.

As discussed above briefly with reference to <FIG>, in some implementations, generating a unified machine learning model can include combining training data associated with different apparel verticals. To calibrate object identification and retrieval performance (e.g., determine learning targets), triplet loss is first used to learn embeddings for each vertical. A goal for vertical combination can be to use fewer numbers of individual specialized models, but without any observed retrieval accuracy degradation.

Table <NUM> shows examples of retrieval accuracy metrics of an (<NUM>) individual model, (<NUM>) dress-top joint, or combined, model, and (<NUM>) dress-top-outerwear joint model, on verticals for "dresses", "tops", and "outerwear. " Compared to the individual model, the dress-top joint model performances are very similarly or slightly better on "dresses" and "tops", however, the dress-top joint model does poorly with regard to retrieval accuracy of apparel items in the "outerwear" vertical category.

Further, a dress-top-outerwear joint model can cause significant accuracy degradation on all three verticals. Accuracy data of Table <NUM> indicates that some verticals can be combined to achieve better accuracy than individually trained models, but only to a certain extent, after which model training difficulties of the triplet loss function causes accuracy degradation (described above).

An example process of system <NUM>, <NUM> includes combining different verticals of training data. In particular, nine apparel verticals can be combined into four groups, where one combined model is trained for each group. In some instances, more or fewer than nine apparel verticals can be combined into a particular number of groups based on user requirements.

The four groups are shown in Table <NUM> and can have comparable performance retrieval accuracy as the individually trained models of each group. In some implementations, "clean triplets" are used to fine-tune each of the four models, where the clean triplets are obtained from "image search" triplets (described below). For example, system <NUM> can be configured to fine-tune model performance using the clean data to accomplish effective improvements in retrieval accuracy for each of the four models.

In general, by combining training data, four embedding models can be obtained for nine distinct apparel verticals. A unified model for all nine verticals can be trained, or learned, and then generated based on the above described teachings. In some implementations, a generated model is deployed for operational use by multiple users. For example, the unified model can receive image data transmitted by a user, where the user seeks to obtain identifying data about a particular object or apparel item included in an image.

In some implementations, the unified model is configured to receive image data for an image, identify or recognize an apparel item in the image, and determine identifying information about the apparel item in the image. In some instances, the unified model is configured to retrieve a related image of the apparel item, and provide, for output to the user via a mobile device, identifying information about the apparel item and/or the related image of the apparel item.

<FIG> shows a diagram <NUM> that includes graphical representations of respective embedding models for object verticals that correspond to a particular apparel category. Moreover, the depictions of diagram <NUM> can correspond to a process, executable by system <NUM>, <NUM>, for extracting one or more feature embeddings. As described above, in the context of apparel recognition, an example two-stage approach can be used for extracting feature embeddings associated with image data of an item or apparel object.

As shown in diagram <NUM>, for a given image/image data <NUM>, at block <NUM> a clothing item can be first detected and localized in the image. At block <NUM>, an embedding (e.g., a vector of floats) is then obtained from a cropped image to represent the clothing item. The embedding is used by the system <NUM>, <NUM> to compare a similarity image for item/object retrieval. In some implementations, the embedding obtained from the cropped image can correspond to a learning target for an apparel vertical to which the clothing item belongs.

As shown by respective arrows <NUM>, each embedding model <NUM>, <NUM>, <NUM>, and <NUM> corresponds to object retrieval functions for identifying and retrieving certain apparel items. The apparel items can correspond to items depicted in the particular cropped portions of image data. In contrast, arrow <NUM> indicates that uniform embedding model <NUM> corresponds to object retrieval functions for identifying and retrieving apparel items for objects depicted in each of the cropped portions of image data in block <NUM>.

Block <NUM> includes respective cropped images that each include a representation of a clothing or apparel item for an object vertical category of the embedding model. For example, a first cropped image data that depicts a handbag corresponds to embedding model <NUM> for identifying and retrieving image data for handbags. Likewise, a second cropped image data that depicts pants corresponds to embedding model <NUM> for identifying and retrieving image data for pants. However, each cropped image data at block <NUM> corresponds to unified embedding model <NUM> for identifying and retrieving image data for various types of apparel items.

<FIG> shows an example diagram <NUM> including computing functions for obtaining image data for training one or more machine learning models. Diagram <NUM> can correspond to computing functions executable by one or more computing modules of system <NUM> or <NUM> described above.

At logic block <NUM>, training data relating to images is first collected from search queries. The search queries can be accessed from an example search system that receives and stores large volumes of image search queries. For example, system <NUM> can be configured to access a query data storage device, such as privately owned query repository that stores thousands (e.g., <NUM>,<NUM>) user queries submitted using Google Image Search. The search queries can include specific product or apparel item names.

At block <NUM>, parsing logic for an example query text parser is executed by system <NUM> to obtain an apparel class label for each text query obtained from the query repository. The text queries can be associated with nine distinct verticals: i) dresses, ii) tops, iii) footwear, iv) handbags, v) eyewear, vi) outerwear, vii) skirts, viii) shorts, and ix) pants. At block <NUM>, system <NUM> selects a particular number of top rated images (e.g., <NUM> images) for each search query, where images are rated based on image pixel quality and an extent to which query text data accurately describes an apparel item of the image.

Image data of the top rated images can be used to form the "image search triplets" described above, where a triplet includes a positive image, a negative image, and an anchor image. System <NUM> can identify at least a subset of triplets (e.g., <NUM>,<NUM> triplets for each object vertical) for system or user rating and verification as to the correctness of each image in the triplet. In some implementations, rating and image verification includes determining whether an anchor image and a positive image of the triplet are from the same product/vertical category. Subsets of triplet images that are rated and verified as correct can be used to form a second set of triplets referred to herein as "clean triplets".

At logic block <NUM>, prior to generating a unified model, base network <NUM> can be initialized from a model pre-trained using one or more types of image data (e.g., ImageNet data). In some implementations, when generating a unified embedding model, the same training data can be used for learning the unified embedding model as was used for triplet feature learning of two or more specialized models. For example, generating a unified embedding learning model may only require vertical label data, which can be obtained via a localizer/classifier, as described above. Hence, unified embedding learning can be generated using the same training images as the training images generated during triplet embedding learning.

Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus.

In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's user device in response to requests received from the web browser.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components.

The computing system can include users and servers. A user and server are generally remote from each other and typically interact through a communication network. The relationship of user and server arises by virtue of computer programs running on the respective computers and having a user-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a user device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the user device). Data generated at the user device (e.g., a result of the user interaction) can be received from the user device at the server.

Claim 1:
A computer-implemented method on a data processing apparatus for generating a unified machine learning computing model for object recognition in digital images or audio signal data using a plurality of neural networks, the method comprising:
determining, by the data processing apparatus and for the plurality of neural networks, each of the plurality of neural networks being a specialized machine learning model, respective learning targets for each of a plurality of object verticals, wherein each object vertical defines a distinct category for an object, and wherein the respective learning targets for each of the plurality of object verticals correspond to first embedding outputs from respective specialized machine learning models taking digital images or audio signal data as input, each machine learning model trained based on a first loss function, wherein the first embedding outputs comprise feature vectors for object recognition in digital images or audio signal data;
characterized in that:
generating, by the data processing apparatus and using the plurality of neural networks, a unified machine learning model configured to generate the second embedding outputs for identifying objects of the plurality of object verticals, comprising
adding one or more of the plurality of object verticals to a group of object verticals:
training (<NUM>), by the data processing apparatus, based on a second loss function and the digital images or audio signal data as inputs, a neural network to generate for the inputs, second embedding outputs similar to the first embedding outputs corresponding to the learning targets for the object verticals in the group; and
while a threshold level of accuracy is maintained or until all object verticals have been processed:
adding further one or more of the plurality of object verticals to the group of object verticals;
repeating the training (<NUM>).