Localizing relevant objects in multi-object images

Solutions for localizing relevant objects in multi-object images include receiving a multi-object image; detecting a plurality of detected objects within the multi-object image; generating a primary heatmap for the multi-object image, the primary heatmap having at least one region of interest; determining a relevant detected object corresponding to a region of interest in the primary heatmap; determining an irrelevant detected object not corresponding to a region of interest in the primary heatmap; and indicating the relevant detected object as an output result but not indicating the irrelevant detected object as an output result. Some examples identify a plurality of objects that are visually similar to the relevant object and displaying the visually similar objects to a user, for example as recommendations of alternative catalog items on an e-commerce website. Some examples are able to identify a plurality of relevant objects and display multiple sets of visually similar objects.

BACKGROUND

Localizing relevant objects in images that show multiple items (multi-object images), such as catalog images that may have multiple types of household furnishings, and even humans, is a significant challenge in computer vision (CV) tasks. This is because some or many objects within multi-object images are not relevant to a particular CV task. Current methods that solve localization tasks are data driven and based on machine learning (ML) models that are trained on supervised, labeled data. However, the process of labeling such localization data is often labor-intensive, rendering it expensive. Consequently, it is often not practical to train a detection model for each group of items for which localization is needed.

SUMMARY

The disclosed examples are described in detail below with reference to the accompanying drawing figures listed below. The following summary is provided to illustrate some examples disclosed herein. It is not meant, however, to limit all examples to any particular configuration or sequence of operations.

Solutions for localizing relevant objects in multi-object images include receiving a multi-object image; detecting a plurality of detected objects within the multi-object image; generating a primary heatmap for the multi-object image, the primary heatmap having at least one region of interest; determining a relevant detected object corresponding to a region of interest in the primary heatmap; determining an irrelevant detected object not corresponding to a region of interest in the primary heatmap; and indicating the relevant detected object as an output result but not indicating the irrelevant detected object as an output result. Some examples identify a plurality of objects that are visually similar to the relevant object and displaying the visually similar objects to a user, for example as recommendations of alternative catalog items on an e-commerce website. Some examples are able to identify a plurality of relevant objects and display multiple sets of visually similar objects.

DETAILED DESCRIPTION

The various examples will be described in detail with reference to the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made throughout this disclosure relating to specific examples and implementations are provided solely for illustrative purposes but, unless indicated to the contrary, are not meant to limit all examples.

Solutions for localizing relevant objects in multi-object images include receiving a multi-object image; detecting a plurality of detected objects within the multi-object image; generating a primary heatmap for the multi-object image, the primary heatmap having at least one region of interest; determining a relevant detected object corresponding to a region of interest in the primary heatmap; determining an irrelevant detected object not corresponding to a region of interest in the primary heatmap; and indicating the relevant detected object as an output result but not indicating the irrelevant detected object as an output result. Some examples identify a plurality of objects that are visually similar to the relevant object and displaying the visually similar objects to a user, for example as recommendations of alternative catalog items on an e-commerce website. Some examples are able to identify a plurality of relevant objects and display multiple sets of visually similar objects.

Aspects of the disclosure operate in an unconventional manner by both performing object detection and generating a heatmap, and determining relevant and irrelevant detected objects according to correspondence between detected objects and a region of interest in the heatmap. Aspects of the invention provide practical results by indicating the relevant detected object as an output result but not indicating the irrelevant detected object as an output result. Aspects of the invention provide further practical results by displaying a plurality of visually similar objects (similar to the relevant detected object) to a user, for example on an e-commerce website when the multi-object image comprises a query image.

This permits the intelligent selection of alternative product suggestions when a user is observing a catalog image that contains multiple items (objects). Some catalog images may contain, for example, a plurality of common household objects (e.g., a couch, a chair, a table, a houseplant, an area rug, dishes, draperies, and even human models), when the particular product (or products) being advertised is just a single one (or only a few) of the objects. This may be common when the initial advertiser wishes to display the object for sale in an appealing household context setting. At a later time, an e-commerce website may have access to the catalog images, without being able to control the content, in order to eliminate the superfluous “context” objects. These superfluous objects are irrelevant to the specific item being advertised.

A visitor to an e-commerce website may wish to consider alternative products, and machine learning (ML or artificial intelligence (AI), collectively ML) models may perform object selection based on visual similarity (to the product being viewed), in order to make an automated selection of the alternative products to display to the website visitor. However, the presence of the irrelevant objects in the catalog image may confuse common ML models, risking a scenario in which a user is considering purchasing a piece of furniture, but is instead presented with notebook computers or draperies as alternative product suggestions.

Inputs are images from a catalog, labeled according to the primary product being advertised using each image, and a queried product type. A set of bounding boxes, specifying different regions of an image, is generated for a specified catalog image (e.g., a query image), using an object detection model. Each bounding box contains an object, but may not be the object corresponding to the queried product type (e.g., specified according to a class label). One or more heatmaps is also generated for the specified catalog image, using a pre-trained classification model. The heatmaps have a region of interest, which is a flare region in which pixels have relatively high values.

In some examples, the heatmaps for a given image are generated using a gradient activation map (GAM) process: the image is passed through the classification model to obtain the activation maps of the final N layers of the classification model their corresponding gradients with respect to the top predications scores. The gradients are used to highlight the important areas in each activation map that contribute to the prediction score. To draw the heatmap, the maps of the last layers are combined and upsampled to the size of the input catalog image. The intersection score for each bounding box is calculated to determine relevance. In some examples, an intersection score is calculated using union of the bounding box and the heatmap pixels within the bounding box having sufficiently high values (e.g., the pixel values exceed a threshold value).

FIG.1illustrates an arrangement100for localizing relevant objects in multi-object images. In some examples, the multi-object image comprises a query image, such as a catalog image being viewed by an e-commerce website visitor, from which alternative product suggestions that are visually similar to a specified object are to be identified. Arrangement100includes an object localizer110, that determines which objects in a multi-object image are relevant to a query124, and which are irrelevant, and a localized object processing140that intakes an output result130(from object localizer110) to use for a further practical result, such as providing imagery to a website150for display to a website visitor (e.g., a user) on a user display page152. The multi-object images may be drawn from a catalog102that has a multi-object image104a, a multi-object image104b, and a multi-object image104c. In some examples, catalog102may cover thousands of products, each with one or more images, and each image having up to a dozen or more irrelevant objects (e.g., items used to provide context to the product advertised for sale). Query124identifies an object class for which to search in a multi-object image. In some examples, query124is defined by activity of a visitor in website150(e.g., which products the website visitor or user is examining).

Object localizer110comprises an ML model120which includes a general object detector112and a heatmap generator114(e.g., a pre-trained classification model). In some examples, heatmap generator114uses a GAM process to generate a heatmap for a particular object type, as identified according to an object class identification (object ID). The heatmap identifies the most relevant region (e.g., a region of interest, such as a flare region of high pixel values) in an image that contributes to the image having a particular object ID label. In some examples, object detector112and heatmap generator114use different networks that are independently trained. In some examples, heatmap generator114comprises a deep learning model with a convolutional neural network (CNN). In some examples, the CNN of heatmap generator114comprises 50 layers or more, and may be a residual CNN. In some examples, heatmap generator114produces an object class score122(e.g., an intersection score) that identifies a similarity between two images, such as a portion of a larger image that has been cropped to a region containing an object of interest. In similarity tasks, two images X and Y may be mapped to vectors and assigned a similarity score S(FX, FY). In some examples, a cosine similarity or dot product, or an inner product (or other function) is used for S. In some examples, the representation produced by F is extracted from a hidden layer of heatmap generator114. In some examples, object class score122for each of multiple detected objects are ranked, with the highest score or scores identified as being relevant to a query.

The GAM process combines one or more activation maps of a network with one or more corresponding gradient maps of the network to produce one or more saliency maps. An activation map identifies neurons n a network that are activated and a gradient map indicates the contribution of a pixel to an output score (e.g., a similarity score). Saliency maps are combined (when more than a single one is generated) to produce the output heatmap.

In some examples, heatmap generator114generates a region of interest for a single relevant object at a time, and generating multiple regions of interest for multiple relevant objects requires multiple iterations (for each object ID) and combination of the individual regions of interest (e.g., by superposition) into a composite heatmap. In some examples, heatmap generator114is trained to generate multiple regions of interest for multiple relevant objects in a single operation. Thus, various operational modes are available for heatmap generator114.

A correlator116determines when a detected object, detected by object detector112, corresponds with a region of interest in a heatmap output by heatmap generator114. An object selector118identifies objects with a corresponding region of interest as relevant and objects without a corresponding region of interest as irrelevant, and also controls other logical operation of object localizer110. For example, if a region of interest in a heatmap does not correspond with a detected object (e.g. a missed detection scenario), object selector118controls object localizer110to send at least that (possibly cropped) portion of the received multi-object image back to object detector112to detect the object responsible for the region of interest (e.g., flare region) in the heatmap. This missed detection remediation is illustrated inFIG.4.

Object localizer110outputs output result130, which contains a relevant object, but not an irrelevant object, from one of multi-object images104a-104c(sourced from catalog102). As illustrated, output result130indicates a first relevant detected object134aand its object ID136a, and a second relevant detected object134band its object ID136b. object IDs136aand136bare object identifiers used in query124. In some examples, output result130indicates a single relevant detected object. In some examples, output result130indicates additional relevant detected objects. In some examples, output result130indicates relevant objects as bounding boxes that had been generated by object detector112and selected as relevant by object selector118. In such examples, relevant detected objects134aand134bare identified as the bounding boxes within the received multi-object image (e.g., multi-object image104a) that, upon cropping, show the particular relevant object. In some examples, object IDs are not included in output result130.

In some examples, localized object processing140intakes output result130and crops the multi-object image to bounding boxes around the relevant detected objects (e.g., as identified by relevant detected objects134aand134b). In some examples, localized object processing140uses a similar image selector146to select similar images from a library142. As illustrated, library142holds an image144awith its label146a(for an object within image144a), an image144bwith its label146b, and an image144c, with its label146c. In some examples, similar image selector146identifies visually similar images by matching object ID136aand/or object ID136bwith one or more of labels146a-146c. It should be understood that, although library142is illustrated with three images and corresponding labels, library142may have thousands of images. The selected visually similar images are then provided to website150(e.g., an e-commerce website) for display to user on user display page152. In some examples, localized object processing140provides the cropped images to a trainer160for inclusion in training data162, thus enhancing the training of another ML model168(providing a further practical result).

Trainer160may be used to train ML model120using training data162. As illustrated training data162holds labeled images: image164awith its label166a, image164bwith its label166b, and image164cwith its label166c. It should be understood that, although training data162is illustrated with three images and corresponding labels, training data162may have thousands of images. An example of generating training data is described in relation toFIGS.11and12.

FIGS.2A and2Bshow practical applications for localizing relevant objects in multi-object images. InFIG.2A, a received multi-object image200a(which may be a version of multi-object image104a) shows a table202(a detected object). A plurality of visually similar images212, specifically a visually similar image222a, a visually similar image222b, a visually similar image222c, and a visually similar image222d, each contains an object (another table) that is visually similar to table202. However, multi-object image200aalso shows a notebook computer324(seeFIG.3) and a couch. Both of these objects are irrelevant to the query.

WhereasFIG.2Ashows the detection of a single relevant object,FIG.2Billustrates the detection of three relevant objects in a single multi-object image. A received multi-object image200b(which may be a version of multi-object image104b) shows a drape204(a detected object), a couch206(another detected object), and a table208(yet another detected object). A plurality of visually similar images is provided for each of drape204, couch206, and table208. Specifically, for drape204, a plurality of visually similar images214includes a visually similar image224a, a visually similar image224b, a visually similar image224c, and a visually similar image224d, each contains an object (another drape) that is visually similar. For couch206, a plurality of visually similar images216includes a visually similar image226a, a visually similar image226b, a visually similar image226c, and a visually similar image226d, each contains an object (another couch) that is visually similar. For table208, a plurality of visually similar images218includes a visually similar image228a, a visually similar image228b, a visually similar image228c, and a visually similar image228d, each contains an object (another table) that is visually similar. However, multi-object image200balso shows an area rug that is irrelevant to the query.

FIG.3illustrates the suppression of an irrelevant object, specifically notebook computer324within multi-object image200a, from output result130. Object detector112detects table202and indicates it with a bounding box312, and also detects notebook computer324and indicates it with a bounding box314. Heatmap generator114, cued to detect a table, generates a heatmap300with a region of interest302. As can be seen inFIG.3, bounding box312corresponds with region of interest302, but bounding box312does not correspond with a region of interest. Thus, object selector118selects bounding box312to indicate as output result130(specifically as relevant detected object134a). Notebook computer324is not indicated in output result130.

FIG.4illustrates the correction of a missed detection of a relevant object. Object detector112initially detects couch206and indicates it with a bounding box416, but misses table208. Heatmap generator114, cued to detect a table and a couch, generates a heatmap400with a region of interest406(for couch206) and also a region of interest408(for table208) from multi-object image200b. As can be seen inFIG.4, bounding box416corresponds with region of interest406, but region of interest408does not correspond with a bounding box. Thus, at least the portion of multi-object image200baround region of interest408is resubmitted to object detector112. This this focused portion, object detector112now detects table208and indicates it with a bounding box418. Object selector118selects bounding boxes416and418to indicate as output result130(specifically as relevant detected objects134aand134b).

FIG.5is a flowchart500illustrating exemplary operations involved in localizing relevant objects in multi-object images with arrangement100. In some examples, operations described for flowchart500are performed by computing device1400ofFIG.14. Flowchart500commences with operation502, which includes generating training data162. In some examples, operation502includes generating multi-object training data as a collage of images of multiple objects, as described in relation toFIGS.11and12. In some examples, operation502generates training data for other ML models (e.g., other ML model168), rather than only just for ML model120. This may occur when objects detected and identified as relevant by object localizer110are sent to trainer160by localized object processing140.

ML model120is trained by trainer160, using training data162, at504. In some examples, operation504includes training a multi-object heatmap generator using training data labeled with tags for multiple objects (e.g., heatmap generator114is able to generate regions of interest for multiple different classes of objects at a time). In some examples, operation504includes training a single-object heatmap generator using training data labeled with tags for a single object type (e.g., heatmap generator114generates regions of interest for individual classes of objects and combines them into a final heatmap, as described in relation toFIGS.9and10). Operation506includes receiving a multi-object image (e.g., multi-object image104a). In some examples, the multi-object image comprises a query image (e.g., an image from which an object identified by query124is to be localized).

Operation508includes detecting a plurality of detected objects (e.g., table208or208, drape204, and/or couch206) within the multi-object image. Detecting objects results in generating bounding boxes (e.g., bounding boxes312,314,416, and418) around the objects in the multi-object image. In some examples, in a first pass through operation508, only a single object is detected, and it is the second (or later) pass that detects an additional object(s). Operation510includes generating a primary heatmap for the multi-object image, the primary heatmap having at least one region of interest (e.g., region of interest302,406, or408). In some examples, generating the heatmap comprises generating GAMs. In some examples, operation510includes operations shown in flowchart700ofFIG.7, and illustrated inFIG.6. In some examples, operation510includes operations shown in flowchart900ofFIG.9, and illustrated inFIG.8.

Decision operation512determines whether all relevant regions of interest in the primary heatmap correspond to detected objects. If a region of interest is “orphaned” (e.g., no corresponding detected object), decision operation512includes identifying that a region of interest in the primary heatmap does not correspond to a detected object of the plurality of detected objects. In such a scenario, flowchart500returns to operation508. Operation508then includes performing a second object detection process for a region of the multi-object image corresponding to the first region of interest in the primary heatmap, to detect the first relevant detected object. For example, as illustrated inFIG.4, table208is not detected initially, but is detected after determining that region of interest408does not correspond to a detected object. In some examples, the multi-object image is cropped to a region corresponding to the first region of interest in the primary heatmap, in order to better focus object detector112and improve the likelihood of detecting an object in the focused portion of the multi-object image. When operation508is repeated, in some examples, operation510is skipped after the second (or later) iteration of operation508.

Operation514includes determining a first relevant detected object corresponding to a region of interest in the primary heatmap (e.g., table202corresponds to region of interest302in heatmap300ofFIG.3). In some examples, operation514also includes determining a second relevant detected object corresponding to a second region of interest of the primary heatmap (e.g., when multiple objects of a single object class, or multiple objects are to be localized in the multi-object image). Operation516includes determining an irrelevant detected object not corresponding to a region of interest in the primary heatmap (e.g., notebook computer324does not correspond to a region of interest in heatmap300). In some examples, operations514and516are performed simultaneously, as a scoring and ranking operation, as described in relation to flowchart1000ofFIG.10.

Operation518includes indicating the first relevant detected object as an output result (e.g., as relevant detected object134a, possibly with object ID136a, in output result130), but not indicating the irrelevant detected object as an output result. In some examples, operation518further includes indicating the second relevant detected object as a second output result (e.g., as relevant detected object134b, possibly with object ID136b, in output result130). Operation520includes cropping the multi-object image to a bounding box around the first relevant detected object. In some examples, operation520further includes cropping the multi-object image to a bounding box around the second relevant detected object. In some examples, cropped images around relevant objects are provided, along with their object IDs (for labeling) as training data for other ML models, going back to operation502.

Operation522includes identifying a plurality of visually similar images containing objects that are visually similar to the first relevant detected object. In some examples, the identification of visually similar objects leverages object IDs. In such examples, identifying a plurality of visually similar images containing objects that are visually similar to the first relevant detected object comprises determining an object identification of the first relevant detected object; and searching for objects having a corresponding object identification (e.g., label146a) corresponding to the object identification (e.g., object ID136a) of the first relevant detected object, wherein the objects that are visually similar to the first relevant detected object have the corresponding object identification. In some examples, operation522further includes identifying a second plurality of visually similar images containing objects that are visually similar to the second relevant detected object. Operation524includes displaying the plurality of visually similar images to a user (e.g., on website150). In some examples, operation524further includes displaying the second plurality of visually similar images to the user.

FIG.6illustrates the improvement of heatmaps, as may be used with some examples of operation510of flowchart500.FIG.7is a flowchart700(substituting for or supplementing operation510) illustrating exemplary operations involved in heatmap improvement illustrated inFIG.6. In some examples, operations described for flowchart700are performed by computing device1400ofFIG.14. Flowchart700commences with operation702, which includes generating an initial heatmap600for a multi-object image. A first region of interest602is identified at704. Operation706includes identifying a corresponding region of interest in the multi-object image, for example using a bounding box604around region of interest602. Operation708includes suppressing, in the multi-object image, a suppressed region612corresponding to region of interest602in initial heatmap600.

Operation710includes generating a second heatmap610for the multi-object image with suppressed region612. Operation712includes combining (e.g., superimposing) region of interest602of initial heatmap600with second heatmap610. Operation712produces primary heatmap620that is output from operation510(and flowchart700).

FIG.8illustrates the combination of heatmap hotspots (regions of interest), as may be used with some examples of operation510of flowchart500.FIG.9is a flowchart900(substituting for or supplementing operation510) illustrating exemplary operations involved in heatmap combination illustrated inFIG.9. In some examples, flowcharts700and900may be used together. In some examples, operations described for flowchart900are performed by computing device1400ofFIG.14.

Flowchart900commences with operation902, which includes generating a first heatmap802for a multi-object image800(which may be an example of multi-object image104a) for a first object type. First heatmap802has a first region of interest812. Operation904includes generating a second heatmap804for multi-object image800for a second object type. Second heatmap804has a first region of interest814. Operation906includes combining (e.g., superimposing) region of interest812of first heatmap802with region of interest814of second heatmap804. This produces a primary (final) heatmap806that contains both region of interest812and region of interest814.

Flowchart900is similar to flowchart700in that both involve combining regions of interest form separately generated heatmaps. Flowchart700is used when the flare region of the initial heatmap obscures other possible regions of interest, and so the suppressed region permits detection of the other regions of interest. Flowchart700may be used when heatmap generator114is capable of detecting regions of interest for multiple classes of objects simultaneously. Flowchart900is used when heatmap generator114detects regions of interest for one class of objects at a time. However, the suppressed region of flowchart700may also be used with the operations of flowchart900.

FIG.10is a flowchart1000illustrating exemplary operations involved in scoring detected objects, for example producing one or values of object class score122. In some examples, operations described for flowchart1000are performed by computing device1400ofFIG.14. In some examples, flowchart1000substitutes for operations514and516of flowchart500. Flowchart1000commences with operation1002, which includes determining a class score (e.g., object class score122) for each detected object. This includes determining class scores for each object that will later be determined to be relevant or irrelevant (e.g., determining a class score for each of the first relevant detected object and the irrelevant detected object). In some examples, object class score122is calculated using the union of a bounding box and heatmap pixels within the bounding box having sufficiently high values (e.g., above a threshold).

Operation1004includes ranking the class scores, wherein the class score for the first relevant detected object exceeds the class score for the irrelevant detected object. Decision operation1006either selects scores above a threshold or the top N scores (e.g., the top score or top 2 or 3 scores), and identifies the corresponding object as a relevant object1008or an irrelevant object1010.

FIG.11illustrates the generation of training data162that may be used by some examples of operation502of flowchart500.FIG.12is a flowchart1200(substituting for or supplementing operation502) illustrating exemplary operations involved in the generation of training data illustrated inFIG.11. Specifically, flowchart1200describes generating multi-object training data as a collage of images of multiple objects. In some examples, operations described for flowchart1200are performed by computing device1400ofFIG.14. Flowchart1200commences with operation1202, which includes generating a plurality of labeled images of a plurality of object types. For example,FIG.11shows the plurality of labeled images as a labeled image1102, a labeled image1104, a labeled image1106, a labeled image1108, and a labeled image1110. Operation1204includes generating a labeled collage1100of the plurality of labeled images of the plurality of object types. Flowchart1200may be used when ML model120is to be trained to recognize multiple object types in a multi-object image using labeled collage1100.

FIG.13is a flowchart1300illustrating exemplary operations involved in localizing relevant objects in multi-object images. In some examples, operations described for flowchart1300are performed by computing device1400ofFIG.14. Flowchart1300commences with operation1302, which includes receiving a multi-object image. Operation1304includes detecting a plurality of detected objects within the multi-object image. Operation1306includes generating a primary heatmap for the multi-object image, the primary heatmap having at least one region of interest. Operation1308includes determining a first relevant detected object corresponding to a region of interest in the primary heatmap. Operation1310includes determining an irrelevant detected object not corresponding to a region of interest in the primary heatmap. Operation1312includes indicating the first relevant detected object as an output result but not indicating the irrelevant detected object as an output result.

Additional Examples

An example method of localizing relevant objects in multi-object images comprises: receiving a multi-object image; detecting a plurality of detected objects within the multi-object image; generating a primary heatmap for the multi-object image, the primary heatmap having at least one region of interest; determining a first relevant detected object corresponding to a region of interest in the primary heatmap; determining an irrelevant detected object not corresponding to a region of interest in the primary heatmap; and indicating the first relevant detected object as an output result but not indicating the irrelevant detected object as an output result.

An example system for localizing relevant objects in multi-object images comprises: a processor; and a computer-readable medium storing instructions that are operative upon execution by the processor to: receive a multi-object image; detect a plurality of detected objects within the multi-object image; generate a primary heatmap for the multi-object image, the primary heatmap having at least one region of interest; determine a first relevant detected object corresponding to a region of interest in the primary heatmap; determine an irrelevant detected object not corresponding to a region of interest in the primary heatmap; and indicate the first relevant detected object as an output result but not indicate the irrelevant detected object as an output result.

One or more example computer storage devices has computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising: receiving a multi-object image; detecting a plurality of detected objects within the multi-object image; generating a primary heatmap for the multi-object image, the primary heatmap having at least one region of interest; determining a first relevant detected object corresponding to a region of interest in the primary heatmap; determining an irrelevant detected object not corresponding to a region of interest in the primary heatmap; and indicating the first relevant detected object as an output result but not indicating the irrelevant detected object as an output result.

Alternatively, or in addition to the other examples described herein, examples include any combination of the following:identifying that a first region of interest in the primary heatmap does not correspond to a detected object of the plurality of detected objects;performing a second object detection process for a region of the multi-object image corresponding to the first region of interest in the primary heatmap, to detect the first relevant detected object;cropping the multi-object image to a bounding box around the first relevant detected object;identifying a plurality of visually similar images containing objects that are visually similar to the first relevant detected object;displaying the plurality of visually similar images to a user;determining a second relevant detected object corresponding to a second region of interest of the primary heatmap;indicating the second relevant detected object as a second output result;determining a first relevant detected object corresponding to a region of interest in the primary heatmap and determining an irrelevant detected object not corresponding to a region of interest in the primary heatmap comprises: determining a class score for each of the first relevant detected object and the irrelevant detected object; and ranking the class scores, wherein the class score for the first relevant detected object exceeds the class score for the irrelevant detected object;the class score is calculated using a union of a bounding box and heatmap pixels within the bounding box having sufficiently high values;generating the primary heatmap comprises: generating a first heatmap for the multi-object image for a first object type; generating a second heatmap for the multi-object image for a second object type; and combining a region of interest of the first heatmap with a region of interest of the second heatmap;the multi-object image comprises a query image;identifying a plurality of visually similar images containing objects that are visually similar to the first relevant detected object comprises determining an object identification of the first relevant detected object; and searching for objects having a corresponding object identification corresponding to the object identification of the first relevant detected object, wherein the objects that are visually similar to the first relevant detected object have the corresponding object identification;cropping the multi-object image to a bounding box around the second relevant detected object;identifying a second plurality of visually similar images containing objects that are visually similar to the second relevant detected object;displaying the second plurality of visually similar images to the user;generating the primary heatmap comprises: generating an initial heatmap for the multi-object image; suppressing, in the multi-object image, a suppressed region corresponding to the region of interest in the initial heatmap; generating a second heatmap for the multi-object image with the suppressed region; and combining the region of interest of the initial heatmap with the second heatmap;generating the heatmap comprises generating a GAM;generating multi-object training data as a collage of images of multiple objects;training a multi-object heatmap generator using training data labeled with tags for multiple objects;generating a plurality of labeled images of a plurality of object types;generating a labeled collage of the plurality of labeled images of the plurality of object types; andtraining an ML model to recognize multiple object types in a multi-object image using the labeled collage.

Example Operating Environment

FIG.14is a block diagram of an example computing device1400for implementing aspects disclosed herein, and is designated generally as computing device1400. Computing device1400is but one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the examples disclosed herein. Neither should computing device1400be interpreted as having any dependency or requirement relating to any one or combination of components/modules illustrated. The examples disclosed herein may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program components, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program components including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks, or implement particular abstract data types. The disclosed examples may be practiced in a variety of system configurations, including personal computers, laptops, smart phones, mobile tablets, hand-held devices, consumer electronics, specialty computing devices, etc. The disclosed examples may also be practiced in distributed computing environments when tasks are performed by remote-processing devices that are linked through a communications network.

Computing device1400includes a bus1410that directly or indirectly couples the following devices: computer-storage memory1412, one or more processors1414, one or more presentation components1416, I/O ports1418, I/O components1420, a power supply1422, and a network component1424. While computing device1400is depicted as a seemingly single device, multiple computing devices1400may work together and share the depicted device resources. For example, memory1412may be distributed across multiple devices, and processor(s)1414may be housed with different devices.

Bus1410represents what may be one or more busses (such as an address bus, data bus, or a combination thereof). Although the various blocks ofFIG.14are shown with lines for the sake of clarity, delineating various components may be accomplished with alternative representations. For example, a presentation component such as a display device is an I/O component in some examples, and some examples of processors have their own memory. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “hand-held device,” etc., as all are contemplated within the scope ofFIG.14and the references herein to a “computing device.” Memory1412may take the form of the computer-storage media references below and operatively provide storage of computer-readable instructions, data structures, program modules and other data for the computing device1400. In some examples, memory1412stores one or more of an operating system, a universal application platform, or other program modules and program data. Memory1412is thus able to store and access data1412aand instructions1412bthat are executable by processor1414and configured to carry out the various operations disclosed herein.

In some examples, memory1412includes computer-storage media in the form of volatile and/or nonvolatile memory, removable or non-removable memory, data disks in virtual environments, or a combination thereof. Memory1412may include any quantity of memory associated with or accessible by the computing device1400. Memory1412may be internal to the computing device1400(as shown inFIG.14), external to the computing device1400(not shown), or both (not shown). Examples of memory1412in include, without limitation, random access memory (RAM); read only memory (ROM); electronically erasable programmable read only memory (EEPROM); flash memory or other memory technologies; CD-ROM, digital versatile disks (DVDs) or other optical or holographic media; magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices; memory wired into an analog computing device; or any other medium for encoding desired information and for access by the computing device1400. Additionally, or alternatively, the memory1412may be distributed across multiple computing devices1400, for example, in a virtualized environment in which instruction processing is carried out on multiple devices1400. For the purposes of this disclosure, “computer storage media,” “computer-storage memory,” “memory,” and “memory devices” are synonymous terms for the computer-storage memory1412, and none of these terms include carrier waves or propagating signaling.

Processor(s)1414may include any quantity of processing units that read data from various entities, such as memory1412or I/O components1420. Specifically, processor(s)1414are programmed to execute computer-executable instructions for implementing aspects of the disclosure. The instructions may be performed by the processor, by multiple processors within the computing device1400, or by a processor external to the client computing device1400. In some examples, the processor(s)1414are programmed to execute instructions such as those illustrated in the flow charts discussed below and depicted in the accompanying drawings. Moreover, in some examples, the processor(s)1414represent an implementation of analog techniques to perform the operations described herein. For example, the operations may be performed by an analog client computing device1400and/or a digital client computing device1400. Presentation component(s)1416present data indications to a user or other device. Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc. One skilled in the art will understand and appreciate that computer data may be presented in a number of ways, such as visually in a graphical user interface (GUI), audibly through speakers, wirelessly between computing devices1400, across a wired connection, or in other ways. I/O ports1418allow computing device1400to be logically coupled to other devices including I/O components1420, some of which may be built in. Example I/O components1420include, for example but without limitation, a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, etc.

The computing device1400may operate in a networked environment via the network component1424using logical connections to one or more remote computers. In some examples, the network component1424includes a network interface card and/or computer-executable instructions (e.g., a driver) for operating the network interface card. Communication between the computing device1400and other devices may occur using any protocol or mechanism over any wired or wireless connection. In some examples, network component1424is operable to communicate data over public, private, or hybrid (public and private) using a transfer protocol, between devices wirelessly using short range communication technologies (e.g., near-field communication (NFC), Bluetooth™ branded communications, or the like), or a combination thereof. Network component1424communicates over wireless communication link1426and/or a wired communication link1426ato a cloud resource1428across network1430. Various different examples of communication links1426and1426ainclude a wireless connection, a wired connection, and/or a dedicated link, and in some examples, at least a portion is routed through the internet.

The order of execution or performance of the operations in examples of the disclosure illustrated and described herein is not essential, and may be performed in different sequential manners in various examples. For example, it is contemplated that executing or performing a particular operation before, contemporaneously with, or after another operation is within the scope of aspects of the disclosure. When introducing elements of aspects of the disclosure or the examples thereof, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. The term “exemplary” is intended to mean “an example of” The phrase “one or more of the following: A, B, and C” means “at least one of A and/or at least one of B and/or at least one of C.”