MOTIF-BASED IMAGE CLASSIFICATION

A method for displaying images similar to a selected image includes receiving, from a user, a selection of an anchor image, generating, using a machine learning model, an anchor embeddings set for the anchor image and respective candidate embeddings sets for a plurality of candidate images. The method also includes calculating a distance between the anchor embeddings set and each of the plurality of candidate embeddings sets and displaying at least one of the plurality of candidate images based on the calculated distance.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to image classification techniques for determining visually similar objects in images based on visual motifs.

DETAILED DESCRIPTION

Image classification techniques are typically used to determine the category or categories to which an input image belongs. For example, convolutional neural networks (CNNs) for images include a series of layers that calculate kernel convolutions on pixel data to detect edges, corners, and shapes within an image, which may be used to classify the image. A typical CNN training process involves providing a set of labeled image samples to the CNN, comparing the predicted category for each data sample to its label, and tuning the weights of the CNN to better predict each data sample's label.

While image classifiers such as traditional CNNs are effective in some contexts, known image classifiers are not robust as to subjective qualities, such as “style.” Human observers often categorize certain items, such as paintings, furniture, decorations, and the like, according to subjective stylistic classifications that are not rigidly defined. An improved image classifier can classify images of such objects according to generally subjective stylistic categories, using a machine learning based classifier trained to recognize stylistic motifs.

In some embodiments, an image classifier according to the present disclosure may find use in e-commerce, such as to recognize a style of a product and to recommend similar products, or coordinating products, to the user. For example, a user may select a product on an e-commerce website (an anchor product), and the e-commerce system may determine the style of the anchor product and recommend one or more replacement or coordinating products of a similar style. Such an e-commerce system, however, is merely one example implementation of the present disclosure.

Referring to the drawings, wherein like numerals refer to the same or similar features in the various views,FIG.1is a block diagram illustrating an example classification system100for classifying images and determining an extent to which two images are visually similar. The system100may include a data source102, an image classification system104, a server130, and a user computing device106.

The data source102may include a plurality of documents107, a plurality of images108, and a plurality of partial labels109. The documents107may be documents specific to or otherwise accessible through a particular electronic user interface, such as a website or mobile application. For example, each document107may be an item information page respective of a particular item. The images108may include images items to which similar styles may apply. Each document may have or be associated with one or more images108. For example, the images108may include a plurality of home decor items of different types. The labels109may include a respective one or more classification labels for each of the images108. The labels109may be “partial” in the sense that a given label may not be completely applied across the data set; as a result, the images108and partial labels109may be considered a “noisy” data set. The terminology reflected in the labels109may be non-standardized or inconsistent. For example, a “cream” object in one label may instead be described as “off-white” in another label indicative of the same underlying color. The data source102may include documents107, images108, and labels109of a plurality of categories, in some embodiments, and accordingly machine earning model training may be based on multi-category data.

In addition to receiving data from the data source102, the image classification system104may update the data source102. For example, the image classification system104may push newly-received documents107or images108to be included within the data source102, or the image classification system104may add, edit, or modify labels109that are used by the image classification system104(e.g., to train a machine learning algorithm, etc.)

The image classification system104may include a processor112and a non-transitory, computer readable memory114storing instructions that, when executed by the processor112, cause the processor112(and therefore the system104) to perform one or more processes, methods, algorithms, steps, etc. of this disclosure. For example, the memory114may include functional modules116,118, and120.

The functional modules116,118, and120of the image classification system104may include an attribute module116configured to generate a set of visual attributes for the image classification system104based on the information received from the data source102. In particular, the attribute module116may standardize the labels109, divide the standardized labels into classes, and flatten the resultant labels and classes to visual attributes (or motifs) that correspond to binary values. For example, label109‘ruby red’ may be standardized to ‘red,’ classified as ‘color,’ and flattened to color.red.' Combinations of these visual attributes may be thought of as motifs because they may be indicative of characteristics that are significant and are repeated throughout a body of images, much like how a literary motif is a theme that may be symbolically significant and repeated throughout a work. Each motif may be a combination (e.g., linear combination) and/or a mixture of visual attributes, or may be derived from a combination and/or mixture of visual attributes. By defining a motif (or motifs) for various images, an overall impression of each image may be defined using a combination of visual attributes rather than by isolating a single visual attribute. This may enable images to be compared based on the overall impression, leading to a more robust comparison.

The functional modules116,118, and120of the image classification system104may also include a training module118configured to train a machine learning model. The training module118may use the images108, labels109, and the sets of visual attributes from the attribute module116to train the machine learning model. In some embodiments, the training module118may use a portion of the documents107, images108, and labels109(either raw or standardized by the attribute module116) to generate a set of training data, and may divide the remaining data source102information to generate a set of validation data. Based on the training data examples, the training module118may train the machine learning algorithm to predict one or more similar images, or to predict one or more similar documents based on similar images.

The functional modules116,118, and120of the image classification system104may further include a similarity module120configured to determine one or more images108and/or documents107that are similar to an anchor image or document using the trained machine learning model. In some embodiments, the trained machine learning model may be used to generate respective embeddings sets for a plurality of images, and relative similarity of images may be determined as a distance between respective embeddings. For example, the similarity module120may generate an anchor embeddings set for the anchor image and several candidate embeddings sets for the images that may be candidates for similarity. From there, the similarity module120may determine a distance (e.g., Euclidean, etc.) between the anchor embeddings set and each candidate embeddings set, and then may display or otherwise present the candidate image(s) corresponding to the candidate embeddings set(s) closest to the anchor embeddings.

The system100may further include a server130in electronic communication with the image classification system104and with a plurality of user computing devices106a,106b,and106c(which may be referred to individually as a user computing device106or collectively as the user computing devices106). The server130may provide a website, data for a mobile application, or other electronic user interface through which the users of the user computing devices106may navigate and otherwise interact with the items associated with documents107. In some embodiments, the server130may receive, in response to a user selection, an indication of documents or images similar to the selected document from the image classification system104, and present one or more documents from the similar documents to the user (e.g., through the interface).

FIG.2is a flow chart illustrating an example method200of classifying one or more images and determining the similarity of two images to each other. One or more portions of the method200may be performed by the image classification system104, in some embodiments.

The method200may include, at block202, receiving training data including images of many types of items and partial classification labels respective of the images. For example, the received training data may include information from the data source102.

The method200may further include, at block204, conditioning the training data. Such conditioning may include, for example, standardizing the labels of the training data, such that similar terms intended to convey the same classification are made equivalent; flattening the training data labels, such that labels are in a Boolean or other form on which a machine learning model may be trained; and/or unifying syntax of labels, whereby equivalent terms of different syntax (e.g., a plural and singular of the same concept) are made to have equivalent syntax.

The method200may further include, at block206, training a machine learning model (e.g., the trained machine learning model of the similarity module120) according to the conditioned training data. Block206may be performed by the training module118, and may include training the machine learning model to minimize a loss function, sch as a Binary Cross-Entropy (BCE) loss function according to equations (1) and (2) below:

where σ is a Sigmoid activation function, a is an attribute number (e.g., to identify the visual attribute within the determined set from the attribute module116), s is a predicted label, y is a target label, and p is a modifier based on positive errors. In some embodiments, p may be greater than 1 to penalize false negative predictions more severely than false positives based on an assumption that labels109may be more likely to be absent (e.g., not supplied by the document107creator) than incorrect (e.g., wrongly applied by the document107creator).

The method200may further include, at block208, receiving a user image selection. The user image selection may be or may include, for example, a user selection of a document on an e-commerce website, which document may be associated with one or more images.

The method200may further include, at block210, classifying the image (e.g., anchor image) selected by the user. In some embodiments, block210may include inputting the selected image to the machine learning model trained at block206to classify the selected image. The machine learning model may generate embeddings respective of the selected image. Accordingly, the classification of the selected image may be or may include embeddings respective of the anchor image.

The method200may further include, at block212, classifying a plurality of additional images. The additional images may be respective of items in a same, similar, and/or different categories to the user-selected image. Block212may include, for example inputting each of the plurality of additional images to the machine learning model trained at block206to generate respective embeddings for each of the plurality of images. The plurality of additional images may be stored in the images108, and may be associated with one or more documents107.

The method200may further include, at block214, determining a similarity each additional image to the selected image. Block214may include a mathematical comparison of the embeddings of each of the additional image to the embeddings of the selected image, such as by computing a Euclidean distance, for example.

The method200may further include, at block216, returning the most similar image to the user. The most similar image may be determined as the image corresponding to the embeddings nearest to the embeddings of the selected image. Block216may include recommending a document that is the subject of the most similar image to the user in response to the user's selection at block208.

FIG.3is a flow chart illustrating an example method300of presenting one or more images similar to a user-selected image. One or more portions of the method300may be performed by the image classification system104, in some embodiments.

The method300may include, at block302, receiving a selection of an anchor image from a user (e.g., via user computing device106). The selection may be directly of an image, such that the user, via an interface, selects a particular image. In some embodiments, the user selection may be of a document, and the anchor image may be derived from (e.g., contained in or otherwise associated with) the selected document. For example, as discussed above with reference to documents107and images108, each document107may be associated with one or more images108, such that an anchor image is derived from the images108associated with the selected document. In some embodiments, such as those in which multiple images are associated with the selected document, the anchor image may be determined as the image having an alignment that may be set as the standard alignment. For example, if the selected document is a couch and the associated images include views of the couch from multiple perspectives and angles, the anchor image may be determined to be the image of the couch from the front. This standard alignment may be pre-set for a document or type of document, such that all documents of a particular type (e.g., couch, pillow, etc.) may have a same standard alignment (e.g., all documents classified as couches have the ‘front’ alignment as standard). The alignment may be a pre-set characteristic of the image (e.g., label109), or the alignment may be determined using image recognition techniques. For example, a large-scale ontology tree of image types corresponding to a range of visual concepts that include or cover the documents107herein (e.g., home goods). If multiple images are determined to each have the standard alignment (e.g., multiple pictures of the front of a couch) and the alignments had been determined using image recognition techniques, the anchor image may be determined as the image with the highest confidence score for the respective alignment determination. In this way, the confidence score for the determination of alignment may be a tiebreaker of sorts.

The method300may also include, at block304, generating an anchor embeddings set indicative of the anchor image. The anchor embeddings set may be generated by a trained machine learning model, such as the trained machine learning model of similarity module120. In some embodiments, the machine learning model may be a multi-layer classification model that takes an image as an input and outputs an embeddings vector indicative of one or more visual attributes of the image. The final layer of the multi-layer classification model may be a layer that calculates binary values for each of the visual attributes, which may be the flattened attributes from the attribute module118. By taking the derivation from the penultimate model layer and calculating binary values, the generated anchor embeddings set may itself be indicative of binary values for the visual attributes associated with the anchor image. These binary values, in turn, may be representative of a motif (e.g., combination and/or mixture of visual attributes) of the anchor image.

The method300may also include, at block306, generating candidate embeddings sets indicative of one or more candidate images. The candidate images may be images pulled from the images108that may or may not be similar or related to the anchor image. For example, the candidate images may be all images in the images108that have a same associated document type to the anchor document, or the candidate images may be all images in the images108that have a different associated document type to the anchor document. Similar to the generated embeddings sets at block304, the candidate embeddings sets may be generated by a trained machine learning model, such as the trained machine learning model of similarity module120. As described above, the machine learning model may be a multi-layer classification model that takes an image as an input and outputs an embeddings vector indicative of one or more visual attributes of the image. As such, the generated candidate embeddings sets may similarly be indicative of binary values for the visual attributes associated with each candidate image.

The method300may also include, at block308, calculating a distance between the anchor embeddings set and each candidate embeddings set. The distance may be calculated according to a cosine similarity, Euclidean distance, or other appropriate multi-dimensional distance calculation for vectors.

The method300may also include, at block310, displaying one or more candidate images that may be similar to the anchor image based on the calculated distance from block308. The displayed candidate image(s) may be those image(s) corresponding to the one or more candidate embeddings sets closest to or within a threshold distance of the anchor embeddings set. In some embodiments, the similar candidate image may be displayed, and may include an interactive element that enables a user to navigate to the document associated with the candidate image. In other embodiments, the document associated with the similar candidate image is directly displayed, and may include other interactive elements (e.g., an option to navigate to the document, purchase an item associated with the document, etc.).

FIG.4is a flow chart illustrating an example method400of presenting one or more images similar to a user-selected image. One or more portions of the method400may be performed by the image classification system104, in some embodiments.

The method400may include, at block402, training a machine learning model to receive an image as input and output binary values indicative of a plurality of visual attributes of the received image. The plurality of visual attributes may be derived from the labels109by standardizing the labels109, dividing the standardized labels into classes (e.g., style, color, pattern, etc.), and flattening the classes and standardized labels into the plurality of visual attributes. The visual attributes are generated such that a binary value (e.g., yes or no, 1 or 0, etc.) may be indicative of whether an image possesses a particular visual attribute. For example, as discussed above with regard to the attribute module116, a generated visual attribute may be ‘color.red,’ where a ‘1’ for an image with regard to this generated visual attribute would indicate that the image (or the document corresponding to the image) may be red.

The machine learning model may be configured to initially generate embeddings for the received image with at least a first layer of the model. The embeddings may be generated in response to training the machine learning model using images (e.g., from the images108) and labels (e.g., labels109) associated with the images. As such, the embeddings may be indicative of an image's characteristics relative to the pool of images (e.g., images108) used to train the machine learning model. A final (e.g., second) layer of the machine learning model may be configured to receive the generated embeddings and to output binary values for the visual attributes based on the generated embeddings. As such, the input image may first be contextualized within a set of images and, based on that contextualization, analyzed for various visual attributes.

The machine learning model may be trained using a cross-entropy loss function in which the loss may be based on the binary values output at the final layer of the machine learning model. This cross-entropy loss function may be the BCE loss function described above with reference to block206of method200. By minimizing the loss of the cross-entropy loss function, the machine learning model may be trained to generate predicted binary values that are relatively close to actual binary values generated for each image in a training data set.

The method400may also include, at block404, receiving a selection of an anchor image from a user (e.g., via user device106). As discussed above with reference to block302of method300, the selection may be directly of an image, such that the user is, via an interface, selecting a particular image. The selection may also be of a document, and the anchor image may be derived from the selected document. For example, each document107may be associated with one or more images108, such that an anchor image is derived from the images108associated with the selected document. In some embodiments, such as those in which multiple images are associated with the selected document, the anchor image may be determined as the image having an alignment that may be set as the standard alignment. For example, if the selected document is a couch and the associated images include views of the couch from multiple perspectives and angles, the anchor image may be determined to be the image of the couch from the front.

The method400may also include, at block406, generating an anchor embeddings set for the anchor image using the machine learning model trained at block402. Because the machine learning model may be trained to output binary values indicative of visual attributes, the anchor embeddings set from block406may include these binary values for the anchor image. In generating the anchor embeddings set, the machine learning model may initially generate a set of intermediate embeddings that may be fed into the final layer of the machine learning model, and the final layer may output the anchor embeddings set.

The method may further include, at block408, generating respective candidate embeddings set, using the trained machine learning model from block402, for a set of images that may be candidates for selection and presentation as similar images to the anchor image. These candidate images may be retrieved from the images108, and may include all images in the images108or may include a particular subset of images based on labels (e.g., labels109) corresponding to the images. For example, the candidate images may be all images that correspond to labels ‘area rugs’ or ‘red.’ The candidate embeddings sets may be generated in the same manner as the anchor embeddings sets, such that the machine learning model initially generates an intermediate embeddings in at least a first layer of the model to contextualize the candidate image within a larger set. From there, the model generates the candidate embeddings in a final layer based on the intermediate embeddings. Similar to the anchor embeddings set, the candidate embeddings sets may be indicative of binary values relative to the visual attributes for each candidate image.

The method400may also include, at block410, calculating a distance between the anchor embeddings set and each candidate embeddings set, and, at block412, displaying one or more candidate images that may be similar to the anchor image based on the calculated distance from block410. The distance may be calculated according to a cosine similarity, Euclidean distance, or other appropriate multi-dimensional distance calculation for vectors. The displayed candidate image(s) may be those image(s) corresponding to the one or more candidate embeddings sets closest to or within a threshold distance of the anchor embeddings set. In some embodiments, the similar candidate image may be displayed, and may include an interactive element that enables a user to navigate to the document associated with the candidate image. In other embodiments, the document associated with the similar candidate image is directly displayed, and may include other interactive elements (e.g., an option to purchase the document).

FIG.5is a flow chart illustrating an example method500of presenting one or more images similar to a user-selected image. One or more portions of the method500may be performed by the image classification system104, in some embodiments.

The method500may include, at block502, deriving a set of images from a set of documents. These images may be images108, and these documents may be documents107ofFIG.1. When multiple images are associated with a single document, a single image for use with the method500may be selected by determining an alignment of each of the multiple images and selecting the image (or images) that have an alignment that matches with a standard alignment. For example, if the selected document is a couch and the associated images include views of the couch from multiple perspectives and angles, the selected image may be determined to be the image of the couch from the front. This standard alignment may be pre-set for a document or type of document, such that all documents of a particular type (e.g., couch, pillow, etc.) may have a same standard alignment (e.g., all documents classified as couches have the ‘front’ alignment as standard). The alignment may be a pre-set characteristic of the image (e.g., label109), or the alignment may be determined using image recognition techniques. If multiple images are determined to each have the standard alignment (e.g., multiple pictures of the front of a couch) and the alignments had been determined using image recognition techniques, the selected image may be determined as the image with the highest confidence score for the respective alignment determination. In this way, the confidence score for the determination of alignment may be a tiebreaker of sorts.

The method500may also include, at block504, deriving pre-determined labels from each image derived at block502and, at block506, standardizing the labels. These labels may be labels109ofFIG.1, and deriving the labels may include associating each image with corresponding labels from labels109. The labels may be standardized by consolidating or revising each label based on a root word(s). By focusing on the root words, the syntax for the derived labels may be unified, such that there may be consistency across the set of labels. For example, ‘ruby red’ and ‘reds’ may both be standardized as ‘red.’ This standardization may be performed by a word processing tool, or by a machine learning model trained to recognize root words.

The method500may further include, at block508, generating label classes based on the standardized labels from block506, and, at block510, flattening the label classes with the standardized labels to generate a defined set of visual attributes. The label classes may be overarching categories that capture sets of labels, such that the label classes may be indicative of a universal visual description. For example, ‘red’ and ‘blue’ may be included in a ‘color’ class, while ‘round’ and ‘trapezoid’ may be included in a ‘shape’ class. From there, the label classes may be flattened with the standardized labels to generate, for each standardized label, a single attribute that may carry a binary value indicative of whether or not the attribute is present. For example, ‘red’ and ‘blue’ may be flattened to ‘color.red’ and ‘color.blue’ respectively. Because each visual attribute may carry a binary value, an image can be described by a string of binary values (e.g., 1 or 0) corresponding to the determined visual attributes.

The method500may include, at block512, training a machine learning model to receive an image and output binary values indicative, for the received image, of the visual attributes determined at block510. As described above with reference to block402of method400, the machine learning model may be configured to initially generate embeddings for the received image with at least a first layer of the model. The embeddings may be generated in response to training the machine learning model using images (e.g., from the images108) and labels (e.g., labels109) associated with the images. As such, the embeddings may be indicative of an image's characteristics relative to the pool of images (e.g., images108) used to train the machine learning model. A final (e.g., second) layer of the machine learning model may be configured to receive the generated embeddings and to output binary values for the visual attributes based on the generated embeddings.

The machine learning model may be trained using a cross-entropy loss function in which the loss may be based on the binary values output at the final layer of the machine learning model. This cross-entropy loss function may be the BCE loss function described above with reference to block206of method200. By minimizing the loss of the cross-entropy loss function, the machine learning model may be trained to generate predicted binary values that are relatively close to actual binary values generated for each image in a training data set.

The method500may also include, at block514, receiving a selection of an anchor image from a user (e.g., via user device106). As discussed above with reference to block302of method300and block404of method400, the selection may be directly of an image, such that the user is, via an interface, selecting a particular image. The selection may also be of a document, and the anchor image may be derived from the selected document. For example, each document107may be associated with one or more images108, such that an anchor image is derived from the images108associated with the selected document. In some embodiments, such as those in which multiple images are associated with the selected document, the anchor image may be determined as the image having an alignment that may be set as the standard alignment. For example, if the selected document is a couch and the associated images include views of the couch from multiple perspectives and angles, the anchor image may be determined to be the image of the couch from the front.

The method500may also include, at block516, generating an anchor embeddings set for the anchor image using the machine learning model trained at block512. Because the machine learning model may be trained to output binary values indicative of visual attributes, the anchor embeddings set from block516may be indicative of these binary values for the anchor image. In generating the anchor embeddings set, the machine learning model may initially generate a set of intermediate embeddings that may be fed into the final layer of the machine learning model, and the final layer may output the anchor embeddings set.

The method may further include, at block518, generating respective candidate embeddings set, using the trained machine learning model from block512, for a set of images that may be candidates for selection and presentation as similar images to the anchor image. These candidate images may be retrieved from the images108, and may include all images in the images108or may include a particular subset of images based on labels (e.g., labels109) corresponding to the images. For example, the candidate images may be all images that correspond to labels ‘area rugs’ or ‘red.’ The candidate embeddings sets may be generated in the same manner as the anchor embeddings sets, such that the machine learning model initially generates an intermediate embeddings in at least a first layer of the model to contextualize the candidate image within a larger set. From there, the model generates the candidate embeddings in a final layer based on the intermediate embeddings. Similar to the anchor embeddings set, the candidate embeddings sets may be indicative of binary values relative to the visual attributes for each candidate image.

The method500may also include, at block520, calculating a distance between the anchor embeddings set and each candidate embeddings set, and, at block522, displaying one or more candidate images that may be similar to the anchor image based on the calculated distance from block520. The distance may be calculated according to a cosine similarity, Euclidean distance, or other appropriate multi-dimensional distance calculation for vectors. The displayed candidate image(s) may be those image(s) corresponding to the one or more candidate embeddings sets closest to or within a threshold distance of the anchor embeddings set. In some embodiments, the similar candidate image may be displayed, and may include an interactive element that enables a user to navigate to the document associated with the candidate image. In other embodiments, the document associated with the similar candidate image is directly displayed, and may include other interactive elements (e.g., an option to purchase the document).

The systems and methods described herein may provide many benefits over traditional image processing systems. For example, the methods described herein first contextualize an image with a larger global set of images before applying labels to the image, which provides a more robust and accurate image classification. Additionally, by generating attributes that can correspond to a single binary value rather than plaintext or other word-based labels, an image can be described by a string of digits. Not only does this save memory space and improve processing as a result, but this also allows for better image classification, as it removes much of the subjective nature typically inherent in determining whether two (or more) images are similar. Furthermore, the binary labels enable a machine learning model to be trained to classify images, as traditional plaintext-based image labeling does not work well with machine learning models. By leveraging machine learning models, these methods may provide a more accurate and less human-intensive image classification process. For example, in testing the methods described herein, the machine learning model aspects led to greatly improved image recognition and, subsequently, responses from users.

FIG.6is a diagrammatic view of an example embodiment of a user computing environment that includes a computing system environment600, such as a desktop computer, laptop, smartphone, tablet, or any other such device having the ability to execute instructions, such as those stored within a non-transient, computer-readable medium. Furthermore, while described and illustrated in the context of a single computing system, those skilled in the art will also appreciate that the various tasks described hereinafter may be practiced in a distributed environment having multiple computing systems linked via a local or wide-area network in which the executable instructions may be associated with and/or executed by one or more of multiple computing systems.

In its most basic configuration, computing system environment600typically includes at least one processing unit602and at least one memory604, which may be linked via a bus. Depending on the exact configuration and type of computing system environment, memory604may be volatile (such as RAM610), non-volatile (such as ROM608, flash memory, etc.) or some combination of the two. Computing system environment600may have additional features and/or functionality. For example, computing system environment600may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks, tape drives and/or flash drives. Such additional memory devices may be made accessible to the computing system environment600by means of, for example, a hard disk drive interface612, a magnetic disk drive interface614, and/or an optical disk drive interface616. As will be understood, these devices, which would be linked to the system bus, respectively, allow for reading from and writing to a hard disk618, reading from or writing to a removable magnetic disk620, and/or for reading from or writing to a removable optical disk622, such as a CD/DVD ROM or other optical media. The drive interfaces and their associated computer-readable media allow for the nonvolatile storage of computer readable instructions, data structures, program modules and other data for the computing system environment600. Those skilled in the art will further appreciate that other types of computer readable media that can store data may be used for this same purpose. Examples of such media devices include, but are not limited to, magnetic cassettes, flash memory cards, digital videodisks, Bernoulli cartridges, random access memories, nano-drives, memory sticks, other read/write and/or read-only memories and/or any other method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Any such computer storage media may be part of computing system environment600.

A number of program modules may be stored in one or more of the memory/media devices. For example, a basic input/output system (BIOS)624, containing the basic routines that help to transfer information between elements within the computing system environment600, such as during start-up, may be stored in ROM608. Similarly, RAM610, hard disk618, and/or peripheral memory devices may be used to store computer executable instructions comprising an operating system626, one or more applications programs628(which may include the functionality of the image classification system104ofFIG.1or one or more of its functional modules116,118,120, for example), other program modules630, and/or program data632.

Still further, computer-executable instructions may be downloaded to the computing environment600as needed, for example, via a network connection.

An end-user may enter commands and information into the computing system environment600through input devices such as a keyboard634and/or a pointing device636. While not illustrated, other input devices may include a microphone, a joystick, a game pad, a scanner, etc. These and other input devices would typically be connected to the processing unit602by means of a peripheral interface638which, in turn, would be coupled to bus. Input devices may be directly or indirectly connected to processor602via interfaces such as, for example, a parallel port, game port, firewire, or a universal serial bus (USB). To view information from the computing system environment600, a monitor640or other type of display device may also be connected to bus via an interface, such as via video adapter643. In addition to the monitor640, the computing system environment600may also include other peripheral output devices, not shown, such as speakers and printers.

The computing system environment600may also utilize logical connections to one or more computing system environments. Communications between the computing system environment600and the remote computing system environment may be exchanged via a further processing device, such a network router642, that is responsible for network routing. Communications with the network router642may be performed via a network interface component644. Thus, within such a networked environment, e.g., the Internet, World Wide Web, LAN, or other like type of wired or wireless network, it will be appreciated that program modules depicted relative to the computing system environment600, or portions thereof, may be stored in the memory storage device(s) of the computing system environment600.

The computing system environment600may also include localization hardware646for determining a location of the computing system environment600. In embodiments, the localization hardware646may include, for example only, a GPS antenna, an RFID chip or reader, a WiFi antenna, or other computing hardware that may be used to capture or transmit signals that may be used to determine the location of the computing system environment600.

The computing environment600, or portions thereof, may comprise one or more components of the system100ofFIG.1, in embodiments.

In some embodiments, a method includes receiving, by a server system from a user, a selection of an anchor image, generating, by the server system using a machine learning model, an anchor embeddings set for the anchor image, the anchor embeddings set indicative of binary values associated with the anchor image for a set of visual attributes, generating, by the server system, using the machine learning model, respective candidate embeddings sets for a plurality of candidate images, calculating a distance between the anchor embeddings set and each of the plurality of candidate embeddings sets, and displaying at least one of the plurality of candidate images based on the calculated distance.

In some of these embodiments, receiving the selection of the anchor image includes receiving, by the server system from the user, a selection of an anchor document, and deriving the anchor image from the anchor document by retrieving a set of images associated with the anchor document, determining an alignment of each image in the set of images, receiving an indication of a standard alignment, and selecting, as the anchor image, the image in the set of images having the standard alignment. In some of these embodiments, the indication of the standard alignment is based on a document type of the anchor document. In other of these embodiments, deriving the anchor image further includes determining that two or more images of the set of images have the standard alignment, determining a confidence score associated with the alignment determination for each of the two or more images, and selecting the anchor image as the image of the two or more images with a highest confidence score.

In some of these embodiments, the displayed at least one candidate image corresponds to the candidate embeddings set that is closest to the anchor embeddings set. In some of these embodiments, each candidate embeddings set is indicative of binary values associated with each candidate image for the set of visual attributes. In some of these embodiments, the binary values represent a motif for each candidate image. The motif may be a combination of visual attributes.

In other embodiments, a method includes training a machine learning model to receive an input image and to output binary values for a plurality of visual attributes of the input image, receiving, by a server system from a user, a selection of an anchor image, generating, by the server system using the machine learning model, an anchor embeddings set for the anchor image, generating, by the machine learning model, respective candidate embeddings sets for a plurality of candidate images, each candidate embeddings set associated with a candidate image, calculating a distance between the anchor embeddings set and each of the plurality of candidate embeddings sets, and displaying at least one of the plurality of candidate images based on the calculated distance.

In some of these embodiments, the machine learning model generates embeddings for an image with a first layer of the model that, during training, is input to a second layer that outputs the binary values for the plurality of visual attributes of the image. In other of these embodiments, training the machine learning model includes minimizing a cross-entropy loss function, wherein a loss of the loss function includes a sum of losses respective of the binary values for the plurality of visual attributes.

In some of these embodiments, the plurality of visual attributes are generated by defining a set of images, each image associated with a document, deriving, from each image of the set of images, one or more pre-determined labels, standardizing the one or more pre-determined labels, generating label classes based on the standardized labels, and flattening the label classes with the pre-determined labels to generate the set of visual attributes. In some of these embodiments, flattening includes consolidating the one or more standardized labels and the label classes into the set of visual attributes, wherein each visual attribute includes at least one label class and at least one standardized label, and wherein the binary value for each visual attribute is indicative of an image associated with the binary value having the visual attribute.

In some of these embodiments, receiving the selection of the anchor image includes receiving, by the server system from the user, a selection of an anchor document, and deriving the anchor image from the anchor document by retrieving a set of images associated with the anchor document, determining an alignment of each image in the set of images, receiving an indication of a standard alignment, and selecting, as the anchor image, the image in the set of images having the standard alignment.

In some of these embodiments, the displayed at least one candidate image corresponds to the candidate embeddings set closest to the anchor embeddings set. In other of these embodiments, displaying the at least one candidate image includes displaying a candidate document associated with each of the at least one candidate image.

In some embodiments, a method includes deriving a set of images from a set of documents, deriving, from each image of the set of images, one or more pre-determined labels, standardizing the one or more pre-determined labels, generating label classes based on the standardized labels, and flattening the label classes with the standardized labels to generate a defined set including a plurality of visual attributes, training a machine learning model to receive an input image and to output binary values for the plurality of visual attributes of the input image, receiving, by a server system from a user, a selection of an anchor image, generating, by the server system using a machine learning model, an anchor embeddings set for the anchor image, the anchor embeddings set indicative of binary values associated with the anchor image for the set of visual attributes, generating, by server system using the machine learning model, respective candidate embeddings sets for a plurality of candidate images, each candidate embeddings set indicative of binary values associated with each candidate image for the set of visual attributes, calculating a distance between the anchor embeddings set and each of the plurality of candidate embeddings sets, and displaying at least one of the plurality of candidate images based on the calculated distance.

In some of these embodiments, flattening includes consolidating the one or more standardized labels and the label classes into the set of visual attributes, wherein each visual attribute includes at least one label class and at least one standardized label, and wherein the binary value for each visual attribute is indicative of an image associated with the binary value having the visual attribute.

In some of these embodiments, receiving the selection of the anchor image includes receiving, by the server system from the user, a selection of an anchor document, and deriving the anchor image from the anchor document by retrieving a set of images associated with the anchor document, determining an alignment of each image in the set of images, receiving an indication of a standard alignment, and selecting, as the anchor image, the image in the set of images having the standard alignment. In some of these embodiments, deriving the anchor image further includes determining that two or more images of the set of images have the standard alignment, determining a confidence score associated with the alignment determination for each of the two or more images, and selecting the anchor image as the image of the two or more images with a highest confidence score.

In some of these embodiments, displaying the at least one candidate image includes displaying a candidate document associated with each of the at least one candidate image.

While this disclosure has described certain embodiments, it will be understood that the claims are not intended to be limited to these embodiments except as explicitly recited in the claims. On the contrary, the instant disclosure is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the disclosure. Furthermore, in the detailed description of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be obvious to one of ordinary skill in the art that systems and methods consistent with this disclosure may be practiced without these specific details. In other instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure various aspects of the present disclosure.

Some portions of the detailed descriptions of this disclosure have been presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer or digital system memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, logic block, process, etc., is herein, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these physical manipulations take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system or similar electronic computing device. For reasons of convenience, and with reference to common usage, such data is referred to as bits, values, elements, symbols, characters, terms, numbers, or the like, with reference to various embodiments of the present invention.

It should be borne in mind, however, that these terms are to be interpreted as referencing physical manipulations and quantities and are merely convenient labels that should be interpreted further in view of terms commonly used in the art. Unless specifically stated otherwise, as apparent from the discussion herein, it is understood that throughout discussions of the present embodiment, discussions utilizing terms such as “determining” or “outputting” or “transmitting” or “recording” or “locating” or “storing” or “displaying” or “receiving” or “recognizing” or “utilizing” or “generating” or “providing” or “accessing” or “checking” or “notifying” or “delivering” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data. The data is represented as physical (electronic) quantities within the computer system's registers and memories and is transformed into other data similarly represented as physical quantities within the computer system memories or registers, or other such information storage, transmission, or display devices as described herein or otherwise understood to one of ordinary skill in the art.