Selecting logo images using machine-learning-logo classifiers

This disclosure relates to methods, non-transitory computer readable media, and systems that can initially train a machine-learning-logo classifier using synthetic training images and incrementally apply the machine-learning-logo classifier to identify logo images to replace the synthetic training images as training data. By incrementally applying the machine-learning-logo classifier to determine one or both of logo scores and positions for logos within candidate logo images, the disclosed systems can select logo images and corresponding annotations indicating positions for ground-truth logos. In some embodiments, the disclosed systems can further augment the iterative training of a machine-learning-logo classifier to include user curation and removal of incorrectly detected logos from candidate images, thereby avoiding the risk of model drift across training iterations.

BACKGROUND

Software developers and engineers have designed logo-detection systems to automatically detect logos within digital images and identify positions of such logos within the digital images. Detected logos may correspond to different logo classes, such as a logo for a specific automobile company and a logo for a specific clothing company. In practice, logo-detection systems have been applied to recognize vehicles for intelligent traffic control, recognize logos to address consumer confusion or perception, or detect use of a trademark or trade dress. To train a machine-learning model to detect logos and corresponding logo positions, some logo-detection systems use synthetic images or weakly labelled real images lacking ground-truth data for logos within images. Despite significant advances in logo detection, computing limitations inhibit existing logo-detection systems from accurately detecting logos in real images, developing real images with corresponding ground-truth data, and scaling a sample of real logo images to train machine-learning models. Some existing logo-detection systems also apply training algorithms that foment model drift during the training of machine-learning models.

In some cases, for example, conventional closed-set-logo-detection systems cannot develop (or train on) sufficient samples of real logo images with corresponding ground truths to accurately detect logos of certain logo classes. Because annotating real-world images that display logos can be time consuming and computationally expensive, some conventional techniques cannot produce, in practice, a critical volume of such images to accurately train a logo-detection system. Rather than rely on real-world images for training, some existing logo-detection systems train on synthetic images and consequently often fail to train the system to consistently classify logos within real-world images with accuracy. Indeed, certain logos lack sufficient corresponding real-world images to train a machine-learning model. Because such annotation is impractically cumbersome and real-world images of certain logos are sparse, some existing logo-detection systems also cannot train a machine-learning model to accurately detect or identify a position of a logo for certain logo classes.

In addition to inaccurate detection, some existing logo-detection systems inaccurately train a machine-learning model by failing to detect or correct model drift. When a machine-learning model incorrectly detects a logo within an input image—by, for example, falsely detecting a logo or failing to detect the presence of the logo—the logo-detection system can adjust or maintain internal parameters of the machine-learning model to reinforce such errors. By propagating errors in logo prediction through training iterations, the machine-learning model can drift to configure parameters to repeat such errors in future logo classifications.

Independent of inaccurate detections or model-drifting training, some logo-detection systems train multi-logo classifiers to detect logos within real images, but with rigid application and without ground-truth data. In some cases, for example, logo-detection systems train a multi-logo classifier to detect a fixed number of logo classes based on weakly labelled real images. Having once trained the multi-logo classifier, such an existing logo-detection system cannot subsequently add logo classes to expand the logo classes detected by the multi-logo classifier.

SUMMARY

This disclosure describes one or more embodiments of methods, non-transitory computer readable media, and systems that solve the foregoing problems in addition to providing other benefits. In particular, the disclosed systems can train a machine-learning-logo classifier using synthetic training images and incrementally apply the machine-learning-logo classifier to identify logo images to replace the synthetic training images as training data. Specifically, the disclosed systems can generate and utilize synthetic training images that portray logos from a logo class to initially train a machine-learning-logo classifier. The disclosed systems can then utilize the machine-learning-logo-classifier to select logo images from real-world images. In addition, the disclosed systems can incrementally replace the synthetic training images with the selected logo images and re-train the machine-learning-logo-classifier with the revised training set. By incrementally re-training and applying the machine-learning-logo classifier, the disclosed systems can select a set of accurate logo images and corresponding annotations indicating logo positions. In some embodiments, the disclosed systems can further augment the iterative training of a machine-learning-logo classifier to include user curation and removal of incorrectly detected logos from candidate images, thereby avoiding the risk of model drift across training iterations.

The following description sets forth additional features and advantages of the disclosed methods, non-transitory computer readable media, and systems.

DETAILED DESCRIPTION

This disclosure describes one or more embodiments of a logo-detection-training system that selects logo images comprising ground-truth logos corresponding to a logo class by initially training a machine-learning-logo classifier using synthetic training images and iteratively applying the machine-learning-logo classifier to identify logo images as replacements for the synthetic training images. To identify logo images and corresponding annotations for ground-truth-logo positions, the logo-detection-training system can apply the machine-learning-logo classifier to determine logo scores indicating predicted logos within candidate logo images and boundary identifiers for predicted logos within the candidate logo images. By augmenting the iterative training of a machine-learning-logo classifier to include user curation and removal of incorrectly detected logos from candidate logo images via a curation user interface, the disclosed logo-detection-training system can further avoid model drift across training iterations.

In some embodiments, for instance, the logo-detection-training system generates a set of synthetic training images portraying logos from a particular logo class and trains a machine-learning-logo classifier based on the synthetic training images. The logo-detection-training system can further apply the machine-learning-logo classifier to candidate logo images to select a subset of logo images. Upon identifying the subset of logo images, the logo-detection-training system can replace a subset of the synthetic training images with the subset of logo images to generate a set of mixed training images. Based on the set of mixed training images, the logo-detection-training system can retrain the machine-learning-logo classifier. By incrementally replacing the synthetic training images with selected candidate logo images, the logo-detection-training system can select logo images comprising ground-truth logos corresponding to the logo class using the retrained machine-learning-logo classifier.

As just mentioned, in some embodiments, the logo-detection-training system generates and utilizes synthetic training images to initially train a machine-learning-logo classifier. For example, the logo-detection-training system can generate synthetic training images by inserting a logo from a logo-class as an overlay to an existing digital image. The logo-detection-training system can select a size, orientation, and location of the logo within the digital image and then apply the logo to generate a synthetic training image. Because the boundary of the logo within the synthetic training image is known, the logo-detection-training system can then train the machine-learning-logo classifier based on the synthetic training images and known boundary identifiers.

Utilizing the machine-learning-logo classifier trained from the synthetic training images, the logo-detection system can identify logo images comprising ground-truth logos from a selected set of candidate logo images. To select candidate logo images, in some embodiments, the logo-detection-training system identifies images from image-search results related to a logo class. For instance, the logo-detection-training system can utilize a search engine to conduct a query for the logo class and utilize the search results as the set of candidate logo images.

As mentioned, upon identifying candidate logo images, the logo-detection system can apply the machine-learning-logo classifier to incrementally determine a set of logo images from the candidate logo images. For example, the logo-detection-training system can apply a machine-learning-logo classifier to candidate logo images to determine logo scores for such candidate images, where a logo score indicates a likelihood that a candidates-ground-truth-logo image portrays a logo corresponding to a logo class. In addition to (or in combination with) such logo scores, in some cases, the machine-learning-logo classifier further determines boundary identifiers indicating positions of candidate logos within the candidate logo images.

Utilizing the identified logo images, the logo-detection-training system can re-train the machine-learning-logo classifier. Specifically, the logo-detection-training system can identify the synthetic images utilized to initially train the machine-learning-logo classifier and replace a subset of the synthetic images with the identified logo images. In replacing a subset of the synthetic images, the logo-detection-training system can generate a mixed training set of synthetic training images and real (non-synthetic) images. The logo-detection-training system can utilize this mixed set of training images to re-train the machine-learning-logo classifier.

The logo-detection-training system can iteratively perform this process to identify an increasing number of logo images. Indeed, the logo-detection-training system can repeatedly train a machine-learning-logo classifier, identify additional logo images, replace synthetic images in the training set with the identified logo images, and re-train the machine-learning-logo classifier. In some embodiments, the logo-detection-training system repeats this process until all of the synthetic training images have been replaced with logo images (e.g., real images portraying logos from the logo class).

As suggested above, the logo-detection-training system can augment or supplement training of a machine-learning-logo classifier to include user curation and removal of incorrectly detected logos from candidate images. For instance, in some embodiments, the logo-detection-training system provides candidate logos from a subset of candidate logo images in a curation user interface for display on a computing device. Such a curation user interface may further include logo-rejection options corresponding to the subset of candidate logo images. Upon receiving an indication of user selection of a logo-rejection option, the logo-detection-training system can remove a corresponding candidate logo image from the subset of candidate logo images.

By removing rejected images from a subset, the logo-detection-training system can employ user curation to select a more accurate subset of candidate logo images for replacing a corresponding subset of synthetic training images in a given training iteration or training epoch. Upon incrementally replacing subsets of synthetic training images with corresponding subsets of candidate logo images, in certain implementations, the logo-detection-training system replaces the synthetic training images with logo images comprising ground-truth logos (and boundary identifiers for such logos) corresponding to a logo class.

The logo-detection-training system can further select and use logo images corresponding to different logo classes to train a multi-logo-classification model. For example, the logo-detection-training system may select a first set of logo images using a first machine-learning-logo classifier and a second set of logo images using a second machine-learning-logo classifier. In one or more embodiments, the logo-detection-training system further trains a multi-logo-classification model to detect logos corresponding to different logo classes based on the first and second sets of logo images. By further training machine-learning-logo classifiers—and selecting corresponding logo images specific to any number of different logo classes—the logo-detection-training system can also retrain the multi-logo-classification model to detect logos corresponding to additional logo classes.

As suggested above, the logo-detection-training system overcomes several technical deficiencies that hinder conventional logo-detection systems. For example, the logo-detection-training system identifies a set of ground-truth training data that previous logo-detection systems could not identify or use for training. Unlike synthetic images or weakly labelled real images, the logo-detection-training system can select logo images with corresponding annotations for one or both of a particular logo and logo positioning. Rather than rely on computationally expensive and time-consuming manual annotation of real images like some conventional logo-detection systems, the logo-detection-training system can iteratively replace synthetic training images with logo images in either an automatic training pipeline or semi-automatic training pipeline with user curation. The logo-detection-training system accordingly can scale logo images to thousands or hundreds of thousands (or more) to accurately train a machine-learning-logo classifier for a particular logo class at a scale that manual annotations cannot replicate.

Based in part on the scalability of the disclosed logo images, the logo-detection-training system improves the accuracy with which logo-detection systems recognize logos corresponding to a particular logo class. Rather than train a machine-learning model on doctored synthetic images or an insufficient (or unrepresentative) collection of real logo images, the logo-detection-training system selects logo images upon which a machine-learning-logo classifier (or multi-logo-classification model) can train to more accurately detect logos of a particular class or logo positions within images. The logo-detection-training system can accordingly facilitate training (or directly train) a machine-learning-logo classifier with logo images to more precisely detect logos in real-world applications than conventional logo-detection systems. As suggested above, such real-word applications range from intelligent traffic control to trademark or trade dress infringement.

In addition to improved logo detection, in some embodiments, the logo-detection-training system also applies a semi-automatic iterative training process that safeguards against model drift. Rather than propagating logo-detection errors through internal parameters of a machine-learning model, the logo-detection-training system can supplement the training process with user curation via a user curation interface. The disclosed user curation interface facilitates removing incorrectly detected candidate logos from candidate logo images. Such a user curation interface introduces flexibility into a training pipeline that conventional systems traditionally applied with rigidity to propound detection errors. As described below, the user-friendly curation interface expeditiously facilitates curation of candidate logo images to avoid false-negative detection and false-positive detection of logos by a machine-learning-logo classifier.

Independent of avoiding model drift, in certain implementations, the logo-detection-training system also increases the flexibility with which a multi-logo-classification model can expand and detect logos corresponding to additional logo classes. Unlike the one-and-done training process of existing multi-logo classifiers, the logo-detection-training system introduces a flexible training process that can incrementally add logo images of particular logo classes and additional logo-class-detection capabilities to a multi-logo-classification model. By separately training and applying machine-learning-logo classifiers specific to respective logo classes, the logo-detection-training system can select logo images specific to such logo classes for training of a multi-logo-classification model.

As indicated by the foregoing description, this disclosure uses a variety of terms to describe features and advantages of the logo-detection-training system. As used in this disclosure, the term “logo” refers to a design, emblem, stylized graphic, symbol, or other graphical representation of a brand, corporation, organization, product, person, service, object, item, or entity. For example, a logo may refer to a specific instance or occurrence of a graphic design or graphic symbol of a brand, corporation, product, service, or entity. To provide but a few examples, a logo may be specific to and represent a company; a brand of beverage, clothing, or eating establishment; store or exchange; or university. Relatedly, the term “logo class” refers to a particular type of logo. A logo class may accordingly refer to a type of logo particular to a company, brand, product, or service. A company or brand can likewise have multiple logos. In some embodiments, a logo class therefore includes multiple copies, instances, or occurrences of a particular logo. A logo class for Adobe Acrobat, for example, may include multiple instances or occurrences of the distinctive logo for Adobe Acrobat. Although many examples provided in this disclosure reflect symbols or brands for different entities, the logo-detection-training system can apply a machine-learning-logo classifier to select logo images comprising ground-truth logos and identify a variety of objects and corresponding object classes portrayed in digital images.

The term “synthetic training image” refers to a digital image altered to include or portray a logo. In particular, in some cases, a synthetic training image refers to a digital image on which a logo is digitally overlaid or superimposed. Accordingly, a synthetic training image may include an instance or occurrence of a logo in a location (or on an object) that would not typically comprise a logo in a real-world application—depending on the image and location of the logo during digital alteration. A synthetic training image may likewise include or correspond to a boundary identifier indicating a position of the logo within the image.

By contrast, the term “candidate logo image” refers to a digital image analyzed to determine if or whether it contains a logo (e.g., a digital image where it is unknown whether the image contains a logo). In particular, a candidate logo image can include an original or real (e.g., non-synthetic) digital image from a repository of digital images that potentially portray a logo. As mentioned above, in some embodiments, a candidate logo image includes a digital image identified as a result of a search for digital images portraying a logo from a particular logo class.

Similarly, the term “logo image” refers to a digital image portraying a logo. In particular, a logo image includes an original or real (e.g., non-synthetic) digital image that portrays a logo and that corresponds to a digital annotation identifying ground-truth data for the logo. As just suggested, a logo image can correspond to ground-truth data that identifies one or both of a logo class and a position of a logo within the digital image (e.g., a boundary identifier). As described in greater detail below, the logo-detection-training system can identify logo images by selecting those candidate logo images that portray a logo corresponding to a logo class.

As used herein, a “machine learning model” refers to a computer representation or algorithm that can be tuned (e.g., trained) based on inputs to approximate unknown functions. In particular, the term “machine-learning model” can include a model that utilizes algorithms to learn from, and make predictions on, known data by analyzing the known data to learn to generate outputs that reflect patterns and attributes of the known data. For instance, a machine-learning model can include but is not limited to a neural network (e.g., a convolutional neural network, recurrent neural network or other deep learning network), a decision tree (e.g., a gradient boosted decision tree), association rule learning, inductive logic programming, support vector learning, Bayesian network, regression-based model (e.g., censored regression), principal component analysis, or a combination thereof.

As mentioned, a machine learning model can include a neural network. A neural network can include a model of interconnected artificial neurons (organized in layers) that communicate and learn to approximate complex functions and generate outputs based on a plurality of inputs provided to the model. In addition, a neural network is an algorithm (or set of algorithms) that implements deep learning techniques that utilize a set of algorithms to model high-level abstractions in data. For example, a neural network can include a convolutional neural network, a recurrent neural network, and/or a graph neural network.

As further suggested above, a “machine-learning-logo classifier” refers to a machine-learning model, such as an artificial neural network, trained to detect logos within images corresponding to a logo class. For instance, in some cases, a machine-learning-logo classifier refers to a machine-learning model trained to determine logo scores indicating a logo class for logos within images and boundary identifiers indicating positions of the logos within images. In some implementations, a machine-learning-logo classifier comprises a convolutional neural network (“CNN”) trained to detect logos, as described above or below.

Additionally, the term “logo score” refers to a score indicating a measure of likelihood that an image includes a logo (e.g., a logo corresponding to a logo class). In some embodiments, for example, a logo score refers to a probability or other quantitative measure that an image (or particular pixels of an image) include a logo corresponding to a logo class. A logo score may include a single score for an image or a series of scores for pixels within an image (e.g., scores in a vector or matrix). Relatedly, a “boundary identifier” refers to an indication of a position of a logo within an image. For instance, a boundary identifier may refer to a bounding box, mask, or data annotation identifying a position of a logo within an image. In some cases, a logo score and a boundary identifier are part of a same output or representation from a machine-learning-logo classifier, such as scores in a vector or matrix indicating a likelihood that particular pixels/locations within a digital image portray a logo.

Turning now toFIG. 1, this figure illustrates a logo-detection-training system100incrementally replacing synthetic training images with logo images using a machine-learning-logo classifier in accordance with one or more embodiments. As depicted inFIG. 1, the logo-detection-training system100initially trains a machine-learning-logo classifier106using synthetic training images and subsequently applies a trained-machine-learning-logo classifier110to generate logo predictions112for candidate logo images. Based on the logo predictions112, the logo-detection-training system100selects a subset of candidate logo images to replace a subset of synthetic training images, thereby generating a set of mixed training images. Based on the set of mixed training images, the logo-detection-training system100retrains the machine-learning-logo classifier106. By iteratively replacing subsets of synthetic training images with subsets of candidate logo images, the logo-detection-training system100selects logo images114corresponding to a logo class.

As suggested above and shown inFIG. 1, the logo-detection-training system100initially trains the machine-learning-logo classifier106using a training image set102comprising subsets of synthetic training images104a-104n. Each synthetic training image from the subsets of synthetic training images104a-104ncomprises a logo corresponding to a logo class. During the initial training, the training image set102includes the subsets of synthetic training images104a-104n, but no candidate logo images.

To initially train the machine-learning-logo classifier106, for example, the logo-detection-training system100applies the machine-learning-logo classifier106to generate logo predictions for each synthetic training image. Such a logo prediction may include a logo score and a boundary identifier for a logo within a synthetic training image. By iteratively generating and comparing logo predictions to ground-truth logos for each synthetic training image to determine a logo-prediction loss, the logo-detection-training system100adjusts the internal parameters of and trains the machine-learning-logo classifier106.

After initially training the machine-learning-logo classifier106, the logo-detection-training system100applies the trained-machine-learning-logo classifier110to candidate logo images to likewise generate the logo predictions112. To identify the candidate logo images, the logo-detection-training system100can identify images from image-search results corresponding to search terms related to a logo class (e.g., a search for a name of a company corresponding to a logo class or a name of a logo for the logo class). For purposes of illustration,FIG. 1depicts both the machine-learning-logo classifier106and the trained-machine-learning-logo classifier110. But the machine-learning-logo classifier106and the trained-machine-learning-logo classifier110represent the same machine-learning-logo classifier at different stages of an iterative process.

As indicated byFIG. 1, the logo-detection-training system100applies the trained-machine-learning-logo classifier110to subsets of candidate logo images108a-108nto generate the logo predictions112. As indicated above, a logo prediction may include, for example, a logo score indicating a likelihood that a candidate logo image includes a logo corresponding to a logo class. A logo prediction for a candidate logo image may likewise include a boundary identifier indicating a position of the logo within the candidate logo image. To generate the logo predictions112in an initial application, the logo-detection-training system100accordingly generates a logo prediction for each candidate logo image from within the subsets of candidate logo images108a-108n.

Based on logo scores from the logo predictions112, the logo-detection-training system100selects a subset of candidate logo images as a subset of logo images to replace a subset of synthetic training images from within the training image set102. For example, the logo-detection-training system100may identify and select candidate logo images corresponding to the highest logo scores (e.g., highest100or1,000logo scores). As indicated inFIG. 1, the logo-detection-training system100selects the subset of candidate logo images108a(as a subset of logo images) after an initial application of the trained-machine-learning-logo classifier110.

Upon selection, the logo-detection-training system utilizes the subset of candidate logo images108aas a subset of logo images for replacing a subset of synthetic training images. Indeed, as shown inFIG. 1, the logo-detection-training system100replaces the subset of synthetic training images104awith the subset of candidate logo images108a. Upon such a replacement, the training image set102constitutes a set of mixed training images—including both synthetic training images and logo images.

As further indicated byFIG. 1, the logo-detection-training system100subsequently retrains the machine-learning-logo classifier106based on the training image set102. The training image set102now comprises the subset of candidate logo images108a, not the subset of synthetic training images104a. In subsequent iterations, the logo-detection-training system100applies the trained-machine-learning-logo classifier110to candidate logo images to generate logo predictions. Based on logo scores from the logo predictions, the logo-detection-training system100selects a new subset of candidate logo images (as a new subset of logo images) and replaces an additional subset of synthetic training images (e.g., the subset of synthetic training images104b) within the training image set102.

AsFIG. 1illustrates, the logo-detection-training system100replaces the subsets of synthetic training images104a-104nwith various subsets of candidate logo images through multiple replacement iterations—until the training image set102includes no synthetic training images. In this manner, the logo-detection-training system100identifies the logo images114.

As suggested above, in certain implementations, the logo-detection-training system100generates synthetic training images as training inputs for a machine-learning-logo classifier. To generate a synthetic training image, the logo-detection-training system100can digitally alter a digital image to include a logo and digitally annotate the digital image to include a boundary identifier indicating a position of the logo (e.g., where metadata corresponding to the digital image includes a logo-class identifier and a boundary identifier).FIG. 2illustrates an example of the logo-detection-training system100generating synthetic training images in accordance with one or more embodiments.

As shown inFIG. 2, the logo-detection-training system100digitally modifies images to include a copy of a logo202corresponding to a logo class. As suggested above, the logo202may correspond to a brand, corporation, product, service, or other entity. In some cases, the logo-detection-training system100overlays a copy of the logo202on a digital image as part of generating a synthetic training image. As shown inFIG. 2, for instance, the logo-detection-training system100overlays copies of the logo206a-206don four respective digital images as part of generating synthetic training images204a-204d.

In addition to overlaying the copies of the logo206a-206d, the logo-detection-training system100also identifies and records a position of each of the copies of the logo206a-206dwithin the synthetic training images204a-204dto generate corresponding boundary identifiers. In some cases, for instance, the logo-detection-training system100records a bounding box for each of the copies of the logo206a-206dto generate a boundary identifier for each of the synthetic training images204a-204d. Additionally, or alternatively, the logo-detection-training system100may identify a location of each pixel within the synthetic training images204a-204dcorresponding to the copies of the logo206a-206d—or each pixel corresponding to a boundary of the copies of the logo206a-206d—to generate boundary identifiers for the synthetic training images204a-204d, respectively. For training purposes, the logo-detection-training system100attaches a logo class corresponding to the logo202and a boundary identifier as a ground-truth logo for each of the synthetic training images204a-204d.

In addition (or in the alternative) to overlaying logos on digital images, the logo-detection-training system100may apply any suitable method of inserting or modifying a digital image to include a logo as part of generating a synthetic training image. For example, in certain implementations, the logo-detection-training system100may apply image-based blending or apply depth and semantic informed positioning of an object model into a real scene to digitally alter a digital image to include a copy of a logo. In some cases, the logo-detection-training system100applies any of the superimposition strategies described by Georgios Georgakis et al., “Synthesizing Training Data for Object Detection in Indoor Scenes,” Robotics: Science and Systems (2017), the entire contents of which are incorporated by reference.

As noted above, the logo-detection-training system100uses synthetic training images to train a machine-learning-logo classifier.FIGS. 3A-3Cdepict the logo-detection-training system100initially training a machine-learning-logo classifier using synthetic training images and incrementally applying the machine-learning-logo classifier to identify logo images to replace the synthetic training images in accordance with one or more embodiments. The following paragraphs describe each ofFIGS. 3A-3Cin turn.

As shown inFIG. 3A, the logo-detection-training system100iteratively applies a machine-learning-logo classifier304to generate logo predictions306a-306ncorresponding to a logo class for synthetic training images302a-302n. The logo-detection-training system100further compares the logo predictions306a-306nto ground-truth logos310a-310ncorresponding to the logo class to determine a logo-prediction loss from a loss function308. Based on the determined logo-prediction loss, the logo-detection-training system100modifies internal parameters of the machine-learning-logo classifier304. By iteratively generating and comparing logo predictions to ground-truth logos for each synthetic training image, the logo-detection-training system100adjusts the internal parameters for and trains the machine-learning-logo classifier304to detect logos within images corresponding to a particular logo class.

As depicted inFIG. 3A, the machine-learning-logo classifier304can comprise an artificial neural network. For example, in some embodiments, the logo-detection-training system100uses a Faster Region-Based CNN (“Faster R-CNN”) or other CNN as the machine-learning-logo classifier304. In particular, the logo-detection-training system100optionally utilizes a Faster R-CNN as described by Shaoqing Ren et al., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,”Advances in Neural Information Processing Systems91-99 (2015) (hereinafter, “Ren”), the entire contents of which are incorporated by reference. Alternatively, the logo-detection-training system100uses a Single Shot MultiBox Detector (“SSD”), a You-Only-Look-Once (“YOLO”) detection system, or a Fast YOLO detection system as the machine-learning-logo classifier304, as described by Wei Liu et al., “SSD: Single Shot MultiBox Detector,”European Conf. on Computer Vision(2016) (hereinafter, “Liu”), the entire contents of which are incorporated by reference. But the logo-detection-training system100may use any suitable machine-learning model as the machine-learning-logo classifier304.

As suggested above, the logo-detection-training system100iteratively inputs the synthetic training images302a-302ninto the machine-learning-logo classifier304as part of initial training. In an initial training iteration shown byFIG. 3A, for example, the logo-detection-training system100feeds the synthetic training image302ato the machine-learning-logo classifier304. Based on the synthetic training image302a, the machine-learning-logo classifier304outputs the logo prediction306a.

As suggested above, the logo prediction306amay include one or both of a logo score and a boundary identifier. For example, in some embodiments, the logo prediction306aincludes a logo score indicating a likelihood that the synthetic training image302aportrays a logo corresponding to a logo class. The logo prediction306amay also include a boundary identifier indicating a position of the logo within the synthetic training image302a. In some cases, the logo prediction306aincludes a box-classification score corresponding to a logo class as a logo score and a bounding box as a bounding box as a boundary identifier as described by Ren.

As indicated above, the machine-learning-logo classifier304may output the logo prediction306ain a variety of formats depending on the layers of the machine-learning-logo classifier304. For instance, in some embodiments, the machine-learning-logo classifier304outputs a matrix or an image map as the logo prediction306a. In such a logo prediction, the matrix or the image map comprises a logo score corresponding to various locations (e.g., a score per pixel or a score per groups of pixels) within the synthetic training image302aand the logo scores collectively indicate a boundary of the logo within the synthetic training images302aas a boundary identifier. Alternatively, the logo prediction306aincludes an image mask indicating a boundary for a logo within the synthetic training image302aand logo scores for portions not covered by the image mask.

After generating the logo prediction306a, the logo-detection-training system100compares the logo prediction306ato the ground-truth logo310a. The ground-truth logo310acorresponds to the synthetic training image302a. As indicated above, in some embodiments, the ground-truth logo310aincludes a ground-truth-logo label with which the logo-detection-training system100compares a logo score (or collection of logo scores) from the logo prediction306a. The ground-truth logo310amay further include a ground-truth-boundary identifier with which the logo-detection-training system100compares a boundary identifier from the logo prediction306a. Alternatively, the ground-truth logo310amay include a ground-truth-logo label and a ground-truth matrix, a ground-truth-image map, or a ground-truth-image mask indicating a position of a logo within the synthetic training image302a.

As further shown inFIG. 3A, in some embodiments, the logo-detection-training system100uses the loss function308to compare the logo prediction306aand the ground-truth logo310aand to determine a logo-prediction loss. To determine a logo-prediction loss, in some embodiments, the logo-detection-training system100determines a classification loss based on a comparison of a logo score and a ground-truth-logo label. In some cases, the logo-detection-training system100further determines a regression loss based on a comparison of a boundary identifier and a ground-truth-boundary identifier. For instance, in some implementations, the logo-detection-training system100implements the loss function308as an overall-loss function by determining (i) a classification loss for a logo score and a ground-truth-logo label using a log-loss function and (ii) a regression loss for a bounding box and a ground-truth-bounding box using a robust-loss function (e.g., Smooth L1) as described by Ren.

Alternatively, in some implementations, the logo-detection-training system100determines (i) a confidence loss based on a comparison of a logo score and a ground-truth-logo label using a softmax-loss function and (ii) a L1loss (e.g., Smooth L1) based on a comparison of a bounding box and a ground-truth-bounding box as described by Liu. But the logo-detection-training system100can use any suitable loss function (or combination of loss functions) as the loss function308. For instance, the logo-detection-training system100may use a binary-cross-entropy-loss function, cross-entropy-loss function, mean-squared-error-loss function, or other suitable loss function as the loss function308. Such loss functions may likewise facilitate a comparison of a ground-truth matrix with a predicted matrix, a ground-truth-image map with a predicted image map, or a ground-truth-image mask with a predicted image mask.

Upon determining a logo-prediction loss from the loss function308, the logo-detection-training system100modifies internal parameters (e.g., weights or values) of the machine-learning-logo classifier304to decrease a logo-prediction loss from the loss function308in a subsequent training iteration (e.g., using back propagation) as indicated by the arrow from the loss function308to the machine-learning-logo classifier304. For example, the logo-detection-training system100may increase or decrease weights or values from some (or all) of a box-regression layer and a box-classification layer within a Faster R-CNN as the machine-learning-logo classifier304to decrease or minimize a loss in a subsequent training iteration. Additionally, or alternatively, the logo-detection-training system100may increase or decrease weights or values from convolutional layers within the machine-learning-logo classifier304to decrease or minimize a loss in a subsequent training iteration.

After modifying internal parameters of the machine-learning-logo classifier304for an initial training iteration, in certain implementations, the logo-detection-training system100performs additional training iterations. In a subsequent training iteration, for instance, the logo-detection-training system100generates a logo prediction306bfor the synthetic training image302b. The logo-detection-training system100further compares the logo prediction306bto a ground-truth logo310busing the loss function308and modifies internal parameters of the machine-learning-logo classifier304based on a determined logo-prediction loss.

By iteratively determining logo-prediction losses from a comparison of logo predictions and ground-truth logos, the logo-detection-training system100trains the machine-learning-logo classifier304to classify logos corresponding to a particular logo class and to identify logo positions within synthetic training images. In some cases, the logo-detection-training system100performs training iterations until having output a logo prediction and determined a corresponding logo-prediction loss for each of the synthetic training images302a-302n. Alternatively, in one or more embodiments, the logo-detection-training system100performs training iterations until the value or weights of the machine-learning-logo classifier304do not change significantly across training iterations or otherwise satisfy a convergence criteria.

As suggested above, after initially training the machine-learning-logo classifier304, the logo-detection-training system100applies the machine-learning-logo classifier304to candidate logo images. In accordance with one or more embodiments,FIG. 3Billustrates an example of the logo-detection-training system100generating logo predictions316a-316nfor candidate logo images314a-314nusing the machine-learning-logo classifier304. Based on logo scores from the logo predictions316a-316n, the logo-detection-training system100subsequently selects a subset of candidate logo images318. Upon optionally curating candidate logo images from the subset of candidate logo images318, the logo-detection-training system100selects a subset of logo images322as training inputs for retraining the machine-learning-logo classifier304.

As shown inFIG. 3B, in some embodiments, the logo-detection-training system100can identify the candidate logo images314a-314nfrom image-search results312related to a logo class. For example, in some embodiments, the logo-detection-training system100uses a search engine to perform an image search based on one or more terms related to a particular logo class. Because the logo class can correspond to a brand, corporation, organization, product, person, service, object, or entity, the search terms for the image search may likewise include a name of the brand, corporation, organization, product, person, service, object or entity.

After receiving the image-search results312, in some implementations, the logo-detection-training system100downloads some or all of the images within the image-search results312(e.g., the initial 200 images or the initial 1,000 images) as the candidate logo images314a-314n. For instance, the logo-detection-training system100may download such images using an application programming interface (“API”) associated with the search engine.

In addition (or in the alternative) to using a search engine to obtain the image-search results312, in certain implementations, the logo-detection-training system100uses a social networking system to perform an image search based on one or more terms related to the particular logo class. In some such cases, the logo-detection-training system100likewise downloads images using an API associated with the social networking system.

Regardless of the method underlying the image-search results312, in some cases, the logo-detection-training system100de-duplicates images from the image-search results312. For instance, the logo-detection-training system100can determine average-pixel values for each image within the image-search results312and identify images as duplicates when the images satisfy an average-pixel-value threshold. But the logo-detection-training system100can apply any suitable de-duplication algorithm, such as Scale-Invariant Feature Extraction (“SIFT”) based de-duplication or Speeded Up Robust Features (“SURF”) based de-duplication. After applying a de-duplication algorithm to the images within the image-search results312, the logo-detection-training system100removes duplicate images from images in the image-search results312as part of identifying the candidate logo images314a-314n.

As suggested above, in certain implementations, the candidate logo images314a-314nderived from the image-search results312constitute weakly labelled data. Such images may be weakly labelled because—before application of the machine-learning-logo classifier304—the images lack object-level-bounding boxes or other boundary identifiers indicating positions of logos. Despite initially lacking such boundary identifiers, the logo-detection-training system100can use such candidate logo images as a basis for generating ground-truth data by, for example, applying the machine-learning-logo classifier304.

As further shown inFIG. 3B, the logo-detection-training system100iteratively applies the machine-learning-logo classifier304to the candidate logo images314a-314nto generate the logo predictions316a-316n. Consistent with the disclosure above, each logo prediction from the logo predictions316a-316nmay include one or both of (i) a logo score indicating a likelihood that a candidate logo image portrays a logo corresponding to a logo class and (ii) a boundary identifier indicating a position of the logo within the candidate logo image. But the logo predictions316a-316nmay take the form of any embodiment described above or below.

After determining the logo predictions316a-316n, the logo-detection-training system100selects the subset of candidate logo images318from among the candidate logo images314a-314nbased on logo scores. For example, in certain implementations, the logo-detection-training system100identifies and selects candidate logo images corresponding to a threshold number of the highest logo scores for the particular logo class from among the candidate logo images314a-314n(e.g., highest200or500logo scores). Alternatively, in one or more embodiments, the logo-detection-training system100arbitrarily identifies a threshold number of candidate logo images corresponding to a logo score for the particular logo class more than (or equal to) a threshold logo score.

As further shown inFIG. 3B, the logo-detection-training system100can augment selection of the subset of candidate logo images318by including user curation320. In some embodiments, for example, the logo-detection-training system100provides a curation user interface within a computing device. Within such a curation user interface, the logo-detection-training system100further provides (i) candidate logos from the subset of candidate logo images318and (ii) a logo-rejection option corresponding to each candidate logo image from the subset of candidate logo images318.FIGS. 5A and 5Binclude examples of a curation user interface. Upon receiving an indication of a user selection of a logo-rejection option, the logo-detection-training system100removes a corresponding candidate logo image from the subset of candidate logo images318.

As suggested above, the logo-detection-training system100can remove any candidate logo image from the subset of candidate logo images318based on detecting a selection of a corresponding logo-rejection option. After providing candidate logos from each candidate logo image within a subset—and removing any rejected candidate logo images—the logo-detection-training system100can generate a modified subset of candidate logo images.

As further shown inFIG. 3B, the logo-detection-training system100selects the subset of logo images322based on the subset of candidate logo images318. In some embodiments, the logo-detection-training system100selects the subset of candidate logo images318(without curation) as the subset of logo images322. Alternatively, in certain implementations, the logo-detection-training system100selects a modified subset of candidate logo images (after the user curation320) as the subset of logo images322.

Regardless of the form of selection, the subset of logo images322can correspond to ground-truth logos. Because each candidate logo image from the subset of candidate logo images318corresponds to a logo prediction, the logo-detection-training system100generates logo predictions for each candidate logo image from the subset of logo images322. In some embodiments, such logo predictions constitute ground-truth logos as digital annotations to the subset of logo images322. As the logo-detection-training system100performs further iterations and retrains the machine-learning-logo classifier304, however, the logo-detection-training system100may modify or update the ground-truth logos corresponding to the subset of logo images322.

As suggested above, the logo-detection-training system100uses a machine-learning-logo classifier to incrementally replace subsets of synthetic training images with subsets of logo images until replacing each synthetic training image with a logo image. In accordance with one or more embodiments,FIG. 3Cillustrates the logo-detection-training system100replacing a subset of synthetic training images with a subset of logo images to generate a set of mixed training images324. After replacing the subset of synthetic training images,FIG. 3Cfurther depicts the logo-detection-training system100retraining the machine-learning-logo classifier304based on the set of mixed training images324.

As shown inFIG. 3C, the logo-detection-training system100replaces a subset of synthetic training images from the synthetic training images302a-302nwith the subset of logo images322. As indicated byFIG. 3C, for example, in some cases, the logo-detection-training system100replaces a subset of synthetic training images comprising, but not limited to, the synthetic training image302awith the subset of logo images322. Such a replaced subset of synthetic training images may include various numbers of synthetic training images, such as one hundred or five hundred synthetic training images. After replacement, the logo-detection-training system100identifies the set of mixed training images comprising both the subset of logo images322and remaining synthetic training images from the synthetic training images302a-302n.

To retrain the machine-learning-logo classifier304, the logo-detection-training system100iteratively inputs images from the set of mixed training images324into the machine-learning-logo classifier304. As suggested byFIG. 3C, the logo-detection-training system100iteratively applies a machine-learning-logo classifier304to generate logo predictions326corresponding to a logo class for images from the set of mixed training images324. The logo-detection-training system100further compares the logo predictions326to ground-truth logos328corresponding to the logo class to determine logo-prediction losses from the loss function308. By iteratively generating and comparing logo predictions to ground-truth logos for each image from the set of mixed training images324, the logo-detection-training system100adjusts the internal parameters for and retrains the machine-learning-logo classifier304to detect logos within images corresponding to the particular logo class.

As suggested above, in some training iterations shown inFIG. 3C, the logo-detection-training system100utilizes a logo image from the set of mixed training images324. In a given training iteration, for instance, the logo-detection-training system100generates a logo prediction for a logo image. The logo-detection-training system100further compares the logo prediction to a ground-truth logo corresponding to the logo image using the loss function308and modifies internal parameters of the machine-learning-logo classifier304based on a determined logo-prediction loss.

In another given training iteration, by contrast, the logo-detection-training system100generates a logo prediction for a synthetic training image from the set of mixed training images324. The logo-detection-training system100further compares the logo prediction to a ground-truth logo corresponding to the synthetic training image using the loss function308and modifies internal parameters of the machine-learning-logo classifier304based on a determined logo-prediction loss.

By iteratively determining logo-prediction losses from a comparison of logo predictions and ground-truth logos, the logo-detection-training system100retrains the machine-learning-logo classifier304to classify logos corresponding to a particular logo class from images (e.g., real images) and to identify logo positions within such images. In some cases, the logo-detection-training system100performs training iterations until having output a logo prediction and determined a corresponding logo-prediction loss for each of the set of mixed training images324. Alternatively, in one or more embodiments, the logo-detection-training system100performs training iterations until the value or weights of the machine-learning-logo classifier304do not change significantly across training iterations or otherwise satisfy a convergence criteria.

As suggested above, in certain implementations, the logo-detection-training system100repeats the process illustrated inFIGS. 3B and 3Cuntil replacing each synthetic training image with a logo image. Through multiple replacement iterations, the logo-detection-training system100replaces subsets from the synthetic training images302a-302nwith subsets of logo images—until no synthetic training images remain. Upon replacing each synthetic training image with a logo image, the logo-detection-training system100identifies a set of logo images. Because the machine-learning-logo classifier304determined a logo prediction for each candidate logo image that became a logo image in at least one of the multiple training iterations, the logo-detection-training system100identifies a ground-truth logo corresponding to each logo image.

As noted above, in some implementations, the logo-detection-training system100can further select and use logo images corresponding to different logo classes to train a multi-logo-classification model.FIG. 4illustrates the logo-detection-training system100training a multi-logo-classification model408based on logo images corresponding to different logo classes in accordance with one or more embodiments.

As shown inFIG. 4, the multi-logo-classification model408can comprise layers or other architecture from the machine-learning models described above. For example, in some embodiments, the logo-detection-training system100uses a Faster R-CNN or other CNN as the multi-logo-classification model408. Alternatively, the logo-detection-training system100uses an SSD, YOLO detection system, or a Fast YOLO detection system as the multi-logo-classification model408. In addition (or in the alternative) to these machine-learning-model examples, the logo-detection-training system100may use any suitable machine-learning model as the multi-logo-classification model408.

As further shown inFIG. 4, the logo-detection-training system100iteratively inputs logo images corresponding to different logo classes into the multi-logo-classification model408—including logo images corresponding to Logo Class A and Logo Class B. Each logo image corresponds to a ground-truth logo. For example, a logo image402from Logo Class A corresponds to a ground-truth logo418, and a logo image404from Logo Class B corresponds to a ground-truth logo420. Logo Classes A and B are merely examples for purposes of illustration. In certain embodiments, the logo-detection-training system100trains the multi-logo-classification model408using logo images corresponding to more logo classes.

In some training iterations shown inFIG. 4, the logo-detection-training system100inputs a logo image corresponding to Logo Class A. For example, in a given training iteration, the logo-detection-training system100generates a logo prediction410for the logo image402. The logo-detection-training system100further (i) compares the logo prediction410to the ground-truth logo418corresponding to the logo image402using a loss function416and (ii) modifies internal parameters of the multi-logo-classification model408based on a determined logo-prediction loss.

In some other training iterations, by contrast, the logo-detection-training system100inputs a logo image corresponding to Logo Class B. For example, in a given training iteration, the logo-detection-training system100generates a logo prediction412for the logo image404. The logo-detection-training system100further (i) compares the logo prediction412to the ground-truth logo420corresponding to the logo image404using the loss function416and (ii) modifies internal parameters of the multi-logo-classification model408based on a determined logo-prediction loss.

As shown inFIG. 4, the loss function416may constitute any loss function described above. To determine a logo-prediction loss for the multi-logo-classification model408, the logo-detection-training system100can, for instance, determine a classification loss based on a comparison of a logo score and a ground-truth-logo label. In some cases, the logo-detection-training system100further determines a regression loss based on a comparison of a boundary identifier and a ground-truth-boundary identifier. As described above, however, the logo-detection-training system100may utilize a classification-loss function, a regression-loss function, a softmax-loss-function, a binary-cross-entropy-loss function, a cross-entropy-loss function, a mean-squared-error-loss function, or other suitable loss function as the loss function416or, when applying an overall-loss function, as part of the loss function416.

As suggested above, in certain implementations, the logo-detection-training system100can retrain the multi-logo-classification model408to detect logos corresponding to additional logo classes—after training the multi-logo-classification model408to detect logos corresponding to Logo Classes A and B. Because the logo-detection-training system100can utilize a machine-learning-logo classifier specific to a logo class to select logo images corresponding to such a logo class, the logo-detection-training system100can subsequently select additional logo images corresponding to an additional logo class using an independently trained machine-learning-logo classifier. Such additional logo images and corresponding ground-truth logos can subsequently provide ground-truth data for retraining a multi-logo-classification model to detect additional logos.

As shown inFIG. 4, for example, the logo-detection-training system100can retrain the multi-logo-classification model408to detect logos corresponding to Logo Class C. When retraining, the logo-detection-training system100may iteratively input logo images corresponding to Logo Class C alone or, alternatively, iteratively input logo images corresponding to Logo Classes A, B, and C. In a given training iteration, for instance, the logo-detection-training system100generates a logo prediction414for a logo image406. The logo-detection-training system100further (i) compares the logo prediction414to a ground-truth logo422corresponding to the logo image404using the loss function416and (ii) modifies internal parameters of the multi-logo-classification model408based on a determined logo-prediction loss.

By iteratively determining logo-prediction losses from a comparison of logo predictions and ground-truth logos, the logo-detection-training system100trains (or retrains) the multi-logo-classification model408to detect logos corresponding to different logo classes from images (e.g., images) and to identify logo positions within such images. In some cases, the logo-detection-training system100performs training iterations until having output a logo prediction and determined a corresponding logo-prediction loss for each image corresponding to Logo Classes A and B during initial training iterations; for each image corresponding to Logo Classes A, B, and C during retraining iterations according to some embodiments; or for each image corresponding to Logo Class C during retraining iterations according to some embodiments. Alternatively, in one or more embodiments, the logo-detection-training system100performs training iterations (or retraining iterations) until the value or weights of the multi-logo-classification model408do not change significantly across training iterations or otherwise satisfy a convergence criteria.

As noted above, the logo-detection-training system100improves the accuracy with which logo-detection systems recognize logos corresponding to a particular logo class. To test the improved accuracy of the logo-detection-training system100, researchers compared (i) a multi-logo-classification model trained without user curation, (ii) a multi-logo-classification model trained with user curation, and (iii) a neural network trained using an existing training pipeline, Scalable Logo Self Training (“SLST”). For both of the multi-logo-classification models in the comparison, the logo-detection-training system100trained a Faster R-CNN comprising a ResNet-101architecture as described by Ren, where the ResNet-101had been pre-trained using a Common Objects in Context (“COCO”) dataset. As part of training both of the different multi-logo-classification models in the comparison, the logo-detection-training system100generated and incrementally replaced two hundred synthetic training images corresponding to different logo classes and set a learning rate of 0.0001. By contrast, the researchers trained the neural network using SLST as described by Hang Su et al., “Weblogo-2m: Scalable Logo Detection by Deep Learning from the Web,”Proceedings of the Institute of Electrical and Electronic Engineers Conf. on Computer Vision and Pattern Recognition270-279 (2017), the entire contents of which are incorporated by reference.

To generate the results shown in Table 1 below, the researchers applied both of the multi-logo-classification models trained by the logo-detection-training system100and the SLST-trained neural network to 3,460 test images independently labelled with ground-truth data. To determine the accuracy of each model in detecting logos from different logo classes in the test images, the researchers determined a mean Average Precision (“mAP”). By convention, the researchers applied a 0.5 IoU rule in which detection is considered correct if the Intersection over Union (“IoU”) between the detected logo and the ground-truth logo exceeds 50%.

TABLE 1ModelmAPMulti-Logo-Classification Model - No User Curation0.74Multi-Logo-Classification Model - User Curation0.81SLST0.69

As shown in Table 1, both of the multi-logo-classification models trained by example embodiments of the logo-detection-training system100demonstrated more accurate logo detection in terms of mAP than the neural network trained by SLST. By adding user curation, the multi-logo-classification model trained by the logo-detection-training system100improved logo detection in terms of mAP by avoiding model drift. The neural network trained by SLST demonstrated less logo-detection accuracy because the SLST's use of synthetic images for training impedes the neural network from accurately detecting logos in real-world images. Further, without user curation, the SLST foments model drift by propagating errors in logo prediction through training iterations into the final neural network.

As noted above, in some embodiments, the logo-detection-training system100provides a curation user interface for display within a computing device to facilitate user curation of candidate logo images. In accordance with one or more embodiments,FIGS. 5A and 5Billustrate a client device500presenting curation user interfaces comprising candidate logos corresponding to candidate logo images selected by the logo-detection-training system100. As described below,FIGS. 5A and 5Bdepict graphical user interfaces from the perspective of the client device500implementing computer-executable instructions from an image-analysis application to perform certain actions for the logo-detection-training system100.

As shown inFIG. 5A, for instance, the client device500presents a curation user interface504aof an image-analysis application within a screen502. The curation user interface504aincludes an image array of candidate logos from candidate logo images. For example, candidate-logo thumbnails506a,506b,506c, and506deach include a candidate logo from a candidate logo image. In some embodiments, such candidate logos correspond to a bounding box from a candidate logo image. While the candidate-logo thumbnails506cand506deach include a candidate logo corresponding to a particular logo class, the candidate-logo thumbnails506aand506binclude candidate logos that do not correspond to the particular logo class.

As indicated above, the logo-detection-training system100selects the candidate logo images corresponding to the candidate logos within the curation user interface504aas part of a subset of candidate logo images. Consistent with the disclosure above, in certain implementations, the logo-detection-training system100selects such a subset of candidate logo images based on logo scores as depicted inFIG. 3Band described above.

In addition to candidate logos, in some embodiments, the curation user interface504aincludes a logo-rejection option corresponding to each candidate logo (and each complementary candidate logo image) shown within the curation user interface504a. In one or more embodiments, a logo-rejection option is represented by a distinct graphical element visible within a curation user interface. Alternatively, a logo-rejection option can be embedded within a candidate-logo thumbnail without a visibly distinct graphical element. As shown inFIG. 5A, for example, each candidate-logo thumbnail includes an embedded logo-rejection option that (upon user selection) causes the logo-detection-training system100to remove a corresponding candidate logo image from a subset of candidate logo images.

As indicated by a comparison ofFIG. 5AandFIG. 5B, the client device500detects user selections of logo-rejection options corresponding to the candidate-logo thumbnails506aand506bby detecting user selections of the candidate-logo thumbnails506aand506b. By contrast, in some embodiments, a client device can detect user selections of logo-rejection options corresponding to candidate-logo thumbnails by detecting user selections of distinct and visible graphical elements representing logo-rejection options. Upon receiving an indication of user selections of the logo-rejection options corresponding to the candidate-logo thumbnails506aand506b, the logo-detection-training system100removes corresponding candidate logo images from a subset of candidate logo images.

As shown inFIG. 5B, for example, the client device500presents a curation user interface504bof the image-analysis application within the screen502—after detecting selections of logo-rejection options. Based on detecting user selections of logo-rejection options embedded within the candidate-logo thumbnails506aand506b, the client device500removes the candidate-logo thumbnails506aand506band sends an indication to the logo-detection-training system100to remove candidate logo images corresponding to the candidate-logo thumbnails506aand506bfrom a subset of candidate logo images. By providing such a curation user interface, the logo-detection-training system100facilitates a flexible and user-friendly interface through which a user can easily review and remove incorrectly detected logos from candidate logo images.

Turning now toFIGS. 6 and 7, these figures provide an overview of an environment in which a logo-detection-training system can operate and an example of an architecture for the logo-detection-training system. In particular,FIG. 6depicts a block diagram illustrating an exemplary system environment (“environment”)600in which a logo-detection-training system606can operate in accordance with one or more embodiments. Specifically,FIG. 6illustrates the environment600comprising server(s)602, third-party server(s)610, a network612, and a client device614. AlthoughFIG. 6illustrates one client device and one user, in alternative embodiments, the environment600can include various numbers of computing devices and associated users. Similarly, althoughFIG. 6illustrates a particular arrangement of the server(s)602, the third-party server(s)610, the network612, and the client device614, various additional arrangements are possible.

As shown inFIG. 6, the server(s)602, the third-party server(s)610, the network612, and the client device614may be communicatively coupled with each other either directly or indirectly, such as coupling through the network612, which is described further below in relation toFIG. 8. The server(s)602and the client device614may include various types of computing devices, including one or more computing devices as discussed further below in relation toFIG. 8.

As depicted inFIG. 6, the server(s)602can generate, store, receive, and/or transmit a variety of types of data, including inputs of synthetic training images, candidate logo images, or logo images into neural networks or user inputs corresponding to a curation user interface. For example, the logo-detection-training system606in conjunction with the server(s)602can receive indications for the client device614of user selections of logo-rejection options via a curation user interface. In some embodiments, the server(s)602comprise a data server, a communication server, or a web-hosting server.

As further shown inFIG. 6, the server(s)602can include an image-analysis system604. In general, the image-analysis system604facilitates the analysis, generation, modification, sharing, accessing, storing, and/or deletion of digital images. For example, the image-analysis system604can use the server(s)602to analyze digital images and detect logos within such images. In certain implementations, the image-analysis system604uses the server(s)602to receive user inputs identifying logos, synthetic training images, candidate logo images, logo images, or other digital images from the client device614or transmit data representing logos, synthetic training images, candidate logo images, logo images, or other digital images to the client device614.

In addition to the image-analysis system604, the server(s)602include the logo-detection-training system606. The logo-detection-training system606is an embodiment (and can perform the functions, methods, and processes) of the logo-detection-training system100described above. In some embodiments, for example, the logo-detection-training system606in conjunction with the server(s)602can train a machine-learning-logo classifier based on synthetic training images comprising logos corresponding to a logo class. Additionally, in some cases, the logo-detection-training system606in conjunction with the server(s)602communicate with the third-party server(s)610to perform image searches and download images to identify candidate logo images.

As suggested above, the logo-detection-training system606in conjunction with the server(s)602may further apply the machine-learning-logo classifier to candidate logo images to select a subset of logo images. By replacing a subset of synthetic training images with the subset of logo images, the logo-detection-training system606in conjunction with the server(s)602can generate a set of mixed training images. By incrementally replacing the synthetic training images with selected candidate logo images, the logo-detection-training system606in conjunction with the server(s)602select logo images comprising ground-truth logos corresponding to the logo class using the retrained machine-learning-logo classifier.

As suggested by previous embodiments, the logo-detection-training system606can be implemented in whole or in part by the individual elements of the environment600. AlthoughFIG. 6illustrates the logo-detection-training system606implemented within the server(s)602, components of the logo-detection-training system606can be implemented in other components of the environment600. For instance, in some embodiments, the client device614comprises the logo-detection-training system108and performs all of the functions, methods, and processes of the logo-detection-training system606described above and below. This disclosure describes example components of the logo-detection-training system606further below (e.g., with regard toFIG. 7).

As further shown inFIG. 6, in some embodiments, the client device614comprises a computing device that allows a user to send and receive digital communications. For example, the client device614can include a desktop computer, laptop computer, smartphone, tablet, or other electronic device. In some embodiments, the client device614further includes one or more software applications (e.g., an image-analysis application616) that allows a user to send and receive digital communications. For example, the image-analysis application616can be a software application installed on the client device614or a software application hosted on the server(s)602. When hosted on the server(s)602, the image-analysis application616may be accessed by the client device614through another application, such as a web browser. In some implementations, the image-analysis application616includes instructions that, when executed by a processor, cause the client device614to present one or more graphical user interfaces, such as curation user interfaces.

As also illustrated inFIG. 6, the image-analysis system604is communicatively coupled to an image-analysis database608. In one or more embodiments, the image-analysis system604accesses and queries data from the image-analysis database608associated with requests from the logo-detection-training system606. For instance, the image-analysis system604may access logos, synthetic training images, candidate logo images, logo images, or other digital images for the logo-detection-training system606. As shown inFIG. 6, the image-analysis database608is separately maintained from the server(s)602. Alternatively, in one or more embodiments, the image-analysis system604and the image-analysis database608comprise a single combined system or subsystem within the server(s)602.

Turning now toFIG. 7, this figure provides additional detail regarding components and features of the logo-detection-training system606in accordance with one or more embodiments. In particular,FIG. 7illustrates a computing device700implementing the image-analysis system604and the logo-detection-training system606. In some embodiments, the computing device700comprises one or more servers (e.g., the server(s)602) and/or one or more client devices (e.g., the client device614).

As shown inFIG. 7, the computing device700includes the image-analysis system604. In some embodiments, the image-analysis system604uses its components to provide tools for analyzing, providing, modifying, storing, and/or deleting digital images. Additionally, in some cases, the image-analysis system604facilitates the analysis and selection of logos corresponding to logo classes and training of machine-learning-logo classifiers or multi-logo-classification models.

As further shown inFIG. 7, the computing device700includes the logo-detection-training system606. The logo-detection-training system606includes, but is not limited to, a digital-image manager702, an image-search manager704, a machine-learning-model trainer706, a machine-learning-model operator708, a curation manager710, and/or a storage manager712. The following paragraphs describe each of these components in turn.

As just mentioned, the logo-detection-training system606includes the digital-image manager702. The digital-image manager702generates, identifies, and receives inputs concerning synthetic training images; and selects and manages candidate logo images or logo images. For example, in some embodiments, the digital-image manager702generates synthetic training images corresponding to a logo class. Additionally, in some embodiments, the digital-image manager702selects candidate logo images based on logo scores and identifies a set of logo images.

As further shown inFIG. 7, the image-search manager704identifies, analyzes, and receives inputs concerning image searches. For example, in some embodiments, the image-search manager704automatically identifies search terms for image searches, receives user inputs identifying search terms for image searches, downloads images from image-search results, and de-duplicates images from image-search results. Additionally, in some embodiments, the image-search manager704interfaces with a search engine or social networking system to perform image searches.

As further shown inFIG. 7, the machine-learning-model operator708applies a trained version of one or both of the machine-learning-logo classifier(s)720and the multi-logo-classification model722. For example, in some embodiments, the machine-learning-model operator708applies the machine-learning-logo classifier(s)720as illustrated inFIG. 1 or 3B, respectively. As another example, in some embodiments, the machine-learning-model operator708applies the multi-logo-classification model722to detect logos corresponding to different logo classes from images and to identify logo positions within such images. In some embodiments, the machine-learning-model operator708further communicates with the storage manager712to apply and/or access the logo images718, the machine-learning-logo classifier(s)720, and/or the multi-logo-classification model722.

In addition to the machine-learning-model operator708, in some embodiments, the logo-detection-training system606further comprises the curation manager710. The curation manager710provides or renders candidate logos from the candidate logo images within a curation user interface, such as the curation user interfaces shown inFIGS. 5A and 5B. In certain implementations, the curation manager710also removes candidate logo image from a subset of candidate logo images based on receiving indications of user selections of logo-rejection options.

In one or more embodiments, each of the components of the logo-detection-training system606are in communication with one another using any suitable communication technologies. Additionally, the components of the logo-detection-training system606can be in communication with one or more other devices including one or more client devices described above. Although the components of the logo-detection-training system606are shown to be separate inFIG. 7, any of the subcomponents may be combined into fewer components, such as into a single component, or divided into more components as may serve a particular implementation. Furthermore, although the components ofFIG. 7are described in connection with the logo-detection-training system606, at least some of the components for performing operations in conjunction with the logo-detection-training system606described herein may be implemented on other devices within the environment600.

Each of the components702-722of the logo-detection-training system606can include software, hardware, or both. For example, the components702-722can include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices, such as a client device or server device. When executed by the one or more processors, the computer-executable instructions of the logo-detection-training system606can cause the computing device(s) to perform the methods described herein. Alternatively, the components702-722can include hardware, such as a special-purpose processing device to perform a certain function or group of functions. Alternatively, the components702-722of the logo-detection-training system606can include a combination of computer-executable instructions and hardware.

Furthermore, the components702-722of the logo-detection-training system606may, for example, be implemented as one or more operating systems, as one or more stand-alone applications, as one or more generators of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components702-722may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components702-722may be implemented as one or more web-based applications hosted on a remote server. The components702-722may also be implemented in a suite of mobile device applications or “apps.” To illustrate, the components702-722may be implemented in a software application, including, but not limited to, ADOBE ILLUSTRATOR, ADOBE EXPERIENCE DESIGN, ADOBE CREATIVE CLOUD, ADOBE PHOTOSHOP, or ADOBE SENSEI. “ADOBE,” “ILLUSTRATOR,” “EXPERIENCE DESIGN,” “CREATIVE CLOUD,” “PHOTOSHOP,” and “SENSEI” are either registered trademarks or trademarks of Adobe Inc. in the United States and/or other countries.

Turning now toFIG. 8, this figure illustrates a flowchart of a series of acts800of training a machine-learning-logo classifier based on synthetic training images and incrementally replacing the synthetic training images with the logo images in accordance with one or more embodiments. WhileFIG. 8illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder, and/or modify any of the acts shown inFIG. 8. The acts ofFIG. 8can be performed as part of a method. Alternatively, a non-transitory computer readable storage medium can comprise instructions that, when executed by one or more processors, cause a computing device to perform the acts depicted inFIG. 8. In still further embodiments, a system comprising one or more memory devices and one or more server devices can perform the acts ofFIG. 8.

As shown inFIG. 8, the acts800include an act810of training a machine-learning-logo classifier based on a set of synthetic training images. In particular, in some embodiments, the act810includes training a machine-learning-logo classifier based on a set of synthetic training images comprising logos corresponding to a logo class. In some cases, the machine-learning-logo classifier comprises a Faster Region-Based Convolutional Neural Network (“Faster R-CNN”).

As further shown inFIG. 8, the acts800include an act820of applying the machine-learning-logo classifier to candidate logo images to select a subset of logo images. In particular, in some embodiments, the act820includes applying the machine-learning-logo classifier to a set of candidate logo images to select a subset of logo images.

For instance, in one or more embodiments, applying the machine-learning-logo classifier to the set of candidate logo images to select the subset of logo images comprises: generating logo scores for the set of candidate logo images utilizing the machine-learning-logo classifier, wherein a logo score indicates a likelihood that a candidate logo image portrays a logo corresponding to the logo class; and selecting the subset of logo images from the set of candidate logo images based in part on the logo scores.

As suggested above, in certain implementations, applying the machine-learning-logo classifier to the set of candidate logo images comprises generating boundary identifiers indicating positions of candidate logos within the set of candidate logo images utilizing the machine-learning-logo classifier.

As further shown inFIG. 8, the acts800include an act830of replacing a subset of synthetic training images with the subset of logo images to generate a set of mixed training images. In particular, in certain implementations, the act830includes replacing a subset of synthetic training images with the subset of logo images to generate a set of mixed training images.

As further shown inFIG. 8, the acts800include an act840of retraining the machine-learning-logo classifier based on the set of mixed training images and an act850of selecting a set of logo images utilizing the retrained machined-learning-logo classifier. In particular, in certain implementations, the act850includes selecting a set of logo images comprising ground-truth logos corresponding to the logo class utilizing the retrained machine-learning-logo classifier. Similarly, in some embodiments, the act850includes determining, from the set of candidate logo images, a set of logo images comprising ground-truth logos corresponding to the logo class utilizing the retrained machine-learning-logo classifier.

In addition to the acts810-850, in certain implementations, the acts800include generating ground-truth-boundary identifiers indicating positions of the ground-truth logos within the set of logo images. Additionally, in some cases, the acts800further include applying the machine-learning-logo classifier to generate logo scores for the set of candidate logo images; and based on the logo scores, providing, for display within a curation user interface, candidate logos from a subset of candidate logo images from the set of candidate logo images and logo-rejection options corresponding to the subset of candidate logo images.

As suggested above, in certain implementations, the acts800further include, after retraining the machine-learning-logo classifier based on the set of mixed training images: applying the machine-learning-logo classifier to an additional set of candidate logo images to select an additional subset of logo images; replacing an additional subset of training images from within the set of synthetic training images with the additional subset of logo images to generate an additional set of mixed training images; and retraining the machine-learning-logo classifier based on the additional set of mixed training images. In some cases, the acts800further include, after retraining the machine-learning-logo classifier based on the set of mixed training images: generating additional logo scores for an additional set of candidate logo images utilizing the machine-learning-logo classifier; selecting an additional subset of logo images from the additional set of candidate logo images based on the additional logo scores; replacing an additional subset of synthetic training images from the set of synthetic training images with the additional subset of logo images to generate an additional set of mixed training images; and retraining the machine-learning-logo classifier based on the additional set of mixed training images.

Relatedly, in some cases, the acts800further include receiving an indication of a user selection of a logo-rejection option corresponding to a candidate logo image from the subset of candidate logo images; based on the indication of the user selection of the logo-rejection option, removing the candidate logo image from the subset of candidate logo images to generate a modified subset of candidate logo images; and selecting the modified subset of candidate logo images as the subset of logo images.

As further suggested above, in some cases, the acts800further include selecting an additional set of logo images comprising additional ground-truth logos corresponding to an additional logo class utilizing an additional machine-learning-logo classifier; and training a multi-logo-classification model based on the set of logo images and the additional set of logo images. Relatedly, in some cases, the acts800further include selecting a new set of logo images comprising new ground-truth logos corresponding to a new logo class utilizing a new machine-learning-logo classifier; and training the multi-logo-classification model based on the set of logo images, the additional set of logo images, and the new set of logo images.

As indicated above, in one or more embodiments, the acts800further include identifying candidate images from among image-search results generated by a search query corresponding to the logo class; and removing duplicate images from the candidate images based on average-pixel values to generate the set of candidate logo images.

In addition (or in the alternative) to the acts described above, in some embodiments, the acts800include performing a step for selecting a set of logo images from the set of candidate logo images utilizing the machine-learning-logo classifier and the set of synthetic training images. For instance, the algorithms and acts described in relation toFIGS. 3B and 3Ccan comprise the corresponding acts for a step for selecting a set of logo images from the set of candidate logo images utilizing the machine-learning-logo classifier and the set of synthetic training images.

A cloud-computing subscription model can be composed of various characteristics such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud-computing subscription model can also expose various service subscription models, such as, for example, Software as a Service (“SaaS”), a web service, Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). A cloud-computing subscription model can also be deployed using different deployment subscription models such as private cloud, community cloud, public cloud, hybrid cloud, and so forth. In this description and in the claims, a “cloud-computing environment” is an environment in which cloud computing is employed.

FIG. 9illustrates a block diagram of exemplary computing device900that may be configured to perform one or more of the processes described above. As shown byFIG. 9, the computing device900can comprise a processor902, a memory904, a storage device906, an I/O interface908, and a communication interface910, which may be communicatively coupled by way of a communication infrastructure912. In certain embodiments, the computing device700can include fewer or more components than those shown inFIG. 9. Components of the computing device900shown inFIG. 9will now be described in additional detail.

In one or more embodiments, the processor902includes hardware for executing instructions, such as those making up a computer program. As an example, and not by way of limitation, to execute instructions for digitizing real-world objects, the processor902may retrieve (or fetch) the instructions from an internal register, an internal cache, the memory904, or the storage device906and decode and execute them. The memory904may be a volatile or non-volatile memory used for storing data, metadata, and programs for execution by the processor(s). The storage device906includes storage, such as a hard disk, flash disk drive, or other digital storage device, for storing data or instructions related to object digitizing processes (e.g., digital scans, digital models).

The communication interface910can include hardware, software, or both. In any event, the communication interface910can provide one or more interfaces for communication (such as, for example, packet-based communication) between the computing device900and one or more other computing devices or networks. As an example and not by way of limitation, the communication interface910may include a network interface controller (“NIC”) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (“WNIC”) or wireless adapter for communicating with a wireless network, such as a WI-FI.

Additionally, the communication interface910may facilitate communications with various types of wired or wireless networks. The communication interface910may also facilitate communications using various communication protocols. The communication infrastructure912may also include hardware, software, or both that couples components of the computing device900to each other. For example, the communication interface910may use one or more networks and/or protocols to enable a plurality of computing devices connected by a particular infrastructure to communicate with each other to perform one or more aspects of the digitizing processes described herein. To illustrate, the image compression process can allow a plurality of devices (e.g., server devices for performing image processing tasks of a large number of images) to exchange information using various communication networks and protocols for exchanging information about a selected workflow and image data for a plurality of images.