Machine learning image processing

A machine learning image processing system performs natural language processing (NLP) and auto-tagging for an image matching process. The system facilitates an interactive process, e.g., through a mobile application, to obtain an image and supplemental user input from a user to execute an image search. The supplemental user input may be provided from a user as speech or text, and NLP is performed on the supplemental user input to determine user intent and additional search attributes for the image search. Using the user intent and the additional search attributes, the system performs image matching on stored images that are tagged with attributes through an auto-tagging process.

PRIORITY

The present application claims foreign priority under 35 USC 119(a)-(d) to European patent application 16290130.0, filed Jul. 6, 2016, and to European patent application 17290038.3, filed Mar. 10, 2017, both of which are incorporated by reference in their entireties.

BACKGROUND

Digital image processing typically involves processing a digital image, for example, from a digital still image or digital video, to ascertain, detect, and/or classify particular features or objects in the image. Pattern recognition may be applied during the image processing to detect a particular object in the image. Digital image processing with pattern recognition has been used in a wide variety of applications, such as facial recognition, detection of land features from aerial photographs, vehicle license plate determination, etc. Different types of conventional machine learning functions may be used for pattern recognition.

DETAILED DESCRIPTION

A machine learning image processing system, according to an example of the present disclosure, builds and trains multiple machine learning classifiers, such as convolutional neural networks (CNNs). The machine learning classifiers may accurately and automatically extract images and perform image processing to detect particular attributes of the extracted images. The attributes may be used for image matching to identify visually similar content.

CNNs include many layers to detect and classify particular features of images relevant for a problem in hand. Furthermore, each layer of the CNN typically has a multitude of parameters associated with it. The specific values of those parameters necessary for a successful and accurate image classification may not be known a priori. The machine learning image processing system, according to an embodiment, provides a method for building and training CNNs to output an accurate classification of an image.

Multiple CNNs may be built and trained by the machine learning image processing system. According to an example of the present disclosure, a first CNN built and trained by the machine learning image processing system may be an image extraction CNN. The image extraction CNN is trained to identify objects from a target image for extraction. The target image is any image that is provided to the image extraction CNN to identify and extract an image of an object therefrom. The extracted image may include a portion of the target image containing an object from the target image. An object in an image is anything that is captured in the image. For example, a target image may include multiple people and/or multiple physical items (e.g., vehicles, street signs, clothes, bags, or any tangible item), and each person or item in the image is an object. The machine learning image processing system may identify objects in the target image, and crop one or more of the objects from the target image to create extracted images of the objects.

Another CNN built and trained by the machine learning image processing system may include an image attribute CNN. The image attribute CNN determines attributes of an image. In an example, an extracted image is created using the image extraction CNN, and the image attribute CNN determines attributes of the extracted image. The attributes of the extracted image may be compared to attributes of stored images to find similar images in an image matching process. The attributes are high-level abstractions represented by vectors of numeric values that may include visual features of an image.

The examples of the present disclosure are generally associated with digital image processing using machine learning. A technical problem associated with pattern recognition and other types of image processing is how to identify an object from an image. The image extraction CNN described in further detail below is able to accurately and automatically extract an image of an object from a target image. Another technical problem associated with image processing is accurate image matching. The image attribute CNN is able to identify particular attributes of the target image which may be used for image matching. Furthermore, both the image extraction CNN and the image attribute CNN can operate in real-time to facilitate accurate image matching of objects from a target image.

With reference toFIG. 1, there is shown a system diagram of a machine learning image processing system100, according to an example of the present disclosure. The system100is referred to as machine learning because it may use machine learning functions to generate classifiers to make predictions on images. Examples of classifiers that may be created and used in the system100are CNNs, as is further discussed below. It should be understood that the system100may include additional components and that one or more of the components described herein may be removed and/or modified without departing from a scope of the system100. The system100may include an image processing server130, an image matching server131and a data repository175.

The system100may receive digital images140, and the digital images140may be stored in the data repository175. The digital images140may be received via communications network119. The digital images140may be provided in files, e.g., JPG, GIF, TIFF, PNG, or BMP files. The digital images140may be provided in digital video and generated from one or more frames of digital video. The digital images140may be provided by any number of sources.

The communications network119may include local area networks (LANs) and wide area networks (WANs), such as the Internet. The communications network119may include signal bearing mediums that may be controlled by software, applications and/or logic. The communications network119may include a combination of network elements to support data communication services. The communications network119may encompass wired and/or wireless network technologies.

Examples of operations performed by the image processing server130are shown at132. For example, the image processing server130may include an image extractor CNN builder115that builds image extractor CNN120. Test data sets and validation sets may be created, for example, from the digital images140or other digital images, and may be used to build and validate the image extractor CNN120. Building and validating the image extractor CNN120is further described with respect toFIGS. 2 and 3A. The image extractor CNN120, for example, is trained to identify classes (i.e., categories) of objects in digital images and locations of the objects in the digital images. The training data for the image extractor CNN120may include digital images, each having one or more objects, a class for each object, and a bounding box (e.g., size and location of bounding box) identifying the location of each object in the digital image.

After building the image extractor CNN120, the image extractor CNN120may be used to identify objects in digital images. Target image150is a digital image provided as input to the image extractor CNN120, and the image extractor CNN120makes a prediction as to whether the target image150contains an object in one of the classes for which it was trained. If the image extractor CNN120generates a prediction that indicates the target image150contains an object in one of the classes, the image extractor CNN120identifies a bounding box in the target image150that surrounds the object. The target image150may be cropped around the bounding box, and the resulting image may be saved as the extracted image151.

An image attribute CNN builder116builds image attribute CNN121. Test data sets and validation sets may be created, for example, from the digital images140or other digital images, and may be used to build and validate the image attribute CNN121. Building and validating the image attribute CNN121is further described with respect toFIGS. 2 and 4A. The image attribute CNN121, for example, is trained to identify classes (i.e., categories) of digital images.

After building the image attribute CNN121, the image attribute CNN121may be used to determine attributes of images classified by the image attribute CNN121. For example, the extracted image151is provided as input to the image attribute CNN121. The image attribute CNN121generates a prediction of whether the extracted image151is in a class for which it was trained. However, instead of using the prediction generated by the image attribute CNN121, the image processing server130determines attributes (shown as extracted image attributes160) of the extracted image151determined by an intermediate layer of the image attribute CNN121during its analysis of the extracted image151to generate the prediction. For example, the image attribute CNN121may be comprised of multiple convolutional layers, fully connected layers, and a binarized sigmoidal layer. A fully connected layer computes the output as Y=FC(X)=W*X+B, where X is the output of the previous layer, a matrix of m*1 values; W is the weight parameter of the layer, a matrix of n*m values; B is the bias parameter of the layer, a matrix of n*1 values; and Y is the output of the fully connected layer, a matrix of n*1 values. This output is the input of the following RELU layer: y=RELU(x)=0 if x<0; y=RELU(x)=x if x>=0, where each element of the matrix X is referred to as an element x. The output of the RELU layer is a matrix of the same dimension as the input, e.g., a matrix of n*1 values in the range of [0,+∞). The output of the RELU layer is the input of the binarized sigmoidal layer, which may generate a value of 0 or 1 for each attribute of multiple attributes of the extracted image. The sigmoidal layer computes the output as sigmoid(x)=1/(1+exp(−x)) for each input element x. The input element x may include values determined from a matrix that is output from a previous layer of the image CNN121, for example the RELU layer explained above. The output value of the sigmoidal layer is a matrix of n*1 values in the range of (0,1). The output matrix is then binarized by applying a threshold: y=0 if x<threshold; y=1 if x>=threshold. The extracted image attributes160may include the values for the attributes determined by the sigmoidal layer of the image attribute CNN121. The extracted image attributes160may be stored in the data repository175and may be used by the image matching server131to identify similar images, as is further described below.

Examples of operations performed by the image matching server131are shown at134. The image matching server131can compare the extracted image151to other images to identify images that are similar to the extracted image151. Attributes of the images may be compared to determine how similar the images are to each other. The images being compared to the extracted image151and/or image data for those images, including their attributes, may be stored in the data repository175. The image matching server131may include an image comparator161that compares the extracted image attributes160to image attributes of other images, for example, stored in the data repository175, to identify similar images. The output of the image comparator161may include similarity values162that represent an amount of similarity between the extracted image attributes160and the attributes of other images being compared to the extracted image151. In an example, to determine similarity between the extracted image attributes160and attributes of another image, a Hamming distance may be calculated. The Hamming distance is an example of a similarity value of the similarity values162. A similarity detector163determines from the similarity values162a set of one or similar images165that are visually similar to the extracted image151. For example, images associated with the “n” smallest Hamming distances are identified as images visually similar to the extracted image151, where “n” is an integer greater than or equal to 1. Determining the similar images165that are the most similar to the extracted image151may be used for a variety of applications, such as for facial recognition, vehicle detection, license plate detection, content delivery, etc. In another example, similar images but not exact images are identified by the image matching server131, as is further discussed below.

Server platform190is an example of hardware that may be used in the image processing server130, image matching server131or other servers described herein. It should be understood that the server platform190may include additional components and that one or more of the components described herein may be removed and/or modified as is known to one of ordinary skill in the art.

The server platform190may include one or more processors191, data storage193, and an input/output (I/O) interface192. The components of the server platform190are shown on a single computer or server as an example and in other examples the components may exist on multiple computers or servers. The server platform190may store data in the data storage193and/or may manage the storage of data stored in a separate computing device, for instance, through the I/O interface192. The data storage193may include physical memory, a hard drive, an optical drive, a flash drive, an array of drives, or any combinations thereof, and may include volatile and/or non-volatile data storage.

The processor191, which may comprise a microprocessor, a micro-controller, an application specific integrated circuit (ASIC), Graphical Processing Unit (GPU) or the like, is to perform various processing functions of the respective server. The processing functions may include classifier building functions, such as performed by image extractor CNN builder115and image attribute CNN builder116, image matching functions, such as performed by the image comparator161and the similarity detector163, and other functions. The processing functions performed by the image processing server130and the image matching server131, and other functions, operations and methods described herein may be embodied as machine readable instructions194that are stored in a non-transitory computer readable medium, such as the data storage193, and executed by a processor, such as processor191. In addition to storing the machine readable instructions194, the data storage193may store data or functions, such as classifiers which may include image extractor CNN120and image attribute CNN121. The image processing server130and the image matching server131are shown as separate servers. However, the functions and operations of these servers may be performed on a single server or multiple servers which may be connected via a network.

The I/O interface192includes a hardware and/or a software interface. The I/O interface192may be a network interface connected to a network through a network device, such as a router. For example, the I/O interface192may be a wireless local area network (WLAN) or a network interface controller (NIC). The WLAN may link to the network device through a radio signal. Similarly, the NIC may link to a network device through a physical connection, such as a cable.

The data repository175may include a database comprised of database tables or another type of data storage system. The data repository175can be implemented as a standalone or distributed repository.

FIG. 2shows an example of operations for generating classifiers255, which may include operations performed by the image extractor CNN builder115and the image attribute CNN builder116to build the image extractor CNN120and the image attribute CNN121. The training sets may include supervised training sets that include labeled data objects, which are used to train the CNNs to generate the classifiers255, such as the image extractor CNN120and the image attribute CNN121. The image processing server130, for example, may receive digital images202from a labeled training set at an image pre-processor205. The image pre-processor may crop and enhance particular content in the images from the training set to input into intermediate CNN builder215. The intermediate CNN builder215may select various architectures and parameters to train an intermediate CNN225. The intermediate CNN225may be then be evaluated on digital images232in a validation set. CNN validator235may determine whether to flag the intermediate CNN225as meeting a designated validation threshold. If the intermediate CNN225does not meet the validation threshold, the intermediate CNN225is not flagged and continues to be trained on the digital images202from the training set by the intermediate CNN builder215. Shared weights of the CNN may be adjusted in an iterative process until the validation threshold is met. When the intermediate CNN225meets the validation threshold, the intermediate CNN225may be selected as the classifier255. The classifier255may be used to classify digital images, such as digital image252, into a class or category at275. The classification may be a prediction of whether the digital image belongs to the class or category. The prediction may be accompanied by a confidence value that indicates accuracy of the classification.

FIG. 3Ashows an example of building and training the image extractor CNN120. The image extractor CNN120may be a Fast Region Based Convolution Network (Fast R-CNN). One or more training sets are provided as input to a Fast R-CNN320to train the Fast R-CNN320to create the image extractor CNN120. The Fast R-CNN320is trained to identify classes of objects in images. For example, the classes of objects to be identified by the image extractor CNN120are determined. One or more training sets are created that include images with objects in the classes. The training sets may include supervised training sets that include labeled data objects. For example, the following classes c1-c7are determined for the training sets: c1=bags, c2=shoes, c3=watches, c4=dresses, c5=shirts, c6=sunglasses, and c7=skirt. Images with objects including bags, shoes, watches, dresses, shirts, sunglasses, and skirts are identified and included in the training sets. Also, locations of the objects in the images are determined. For each image, the classes of the one or more objects in the images, and the location of each object are specified to create the supervised training sets. Images301and302are shown for the supervised training set. An object in image301is a bag, and a location and size of bounding box311is specified to identify the location of the bag in the image301. An object in image302is a watch, and a location and size of bounding box312is specified to identify the location of the watch in the image302. The supervised training set may include images with objects in all the classes to train the Fast R-CNN320to create the image extractor CNN120. Multiple CNNs may be created. For example, a CNN120may be created for each of the classes c1-c7. Accordingly, the image extractor CNN120may comprise multiple CNNs, each trained for a particular class of objects.

Once the image extractor CNN120is created, it may be used to classify objects in images.FIG. 3Bshows an example of the target image150which is provided as input to the image extractor CNN120. The image extractor CNN120identifies two classes of objects, i.e., a bag and a skirt, in the target image150and the locations of the objects, shown by bounding boxes321and322. A confidence value is determined for each identified object. Objects with a confidence value greater than a threshold are extracted. For example, the skirt is extracted as shown, and the image of the skirt is saved in the data repository175. The extracted image of the skirt may be used to identify similar images by the image matching server131.

FIG. 4Ashows an example of training a CNN to create the image attribute CNN121. Each image in the training set for training the CNN includes a class label. To train the CNN, network parameters for the CNN are initialized. Predetermined parameter models may be available that specify initial network parameters. Training data is input into the initialized CNN to calculate predictions of classes for the images of the training set. The difference between the expected classes which are specified for each image and the predicted classes are determined and used to update the network parameters with a back-propagation function. This process is iterated until a satisfactory prediction rate is achieved to create the image attribute CNN121.

As shown inFIG. 4A, the image attribute CNN121for example includes a number of convolutional and subsampling layers followed by fully connected layers. The input to a convolutional layer is a m×m×r image where m is the height and width of the image and r is the number of channels, e.g. an RGB (red, green, blue) image has r=3. The convolutional layer will have k filters (or kernels) of size n×n where n is smaller than the dimension of the image. The size of the filters gives rise to the locally connected structure which are each convolved with the image to produce k feature maps of size m−n+1 (supposing the stride=1 and padding=0). Each map is then subsampled typically with mean or max pooling over p×p contiguous regions where p may range between 2 for small images and N (N>=5) for larger inputs. Either before or after the subsampling layer, an additive bias and rectified linear unit (RELU) layer is applied to each feature map.

Once the CNN is trained to create the image attribute CNN121, the image attribute CNN121may be used to index visual features of the extracted image151as shown inFIG. 4B. For example, the output of the “sigmoidal layer8” is used as the visual index of the extracted image151, and includes a binarized vector, such as [0 1 1 0 . . . 1], representing the visual features (also referred to as the attributes) of the extracted image151. For example, the binarized vector represents an index of visual features xidetermined from a previous layer (e.g., RELU layer7) that are binarized to 0 or 1. For example, for each visual feature x, a value of 0 or 1 is determined based on whether xiis greater than or equal to a threshold (e.g., xi=1) or less than the threshold (e.g., xi=0). The threshold may be selected by maximizing the classification accuracy on training data. Images to be compared to an extracted image are also run through the image attribute CNN121to determine a binarized vector of attributes for each image, and the binarized vectors are stored in the data repository175. The binarized vector of attributes for the extracted image151may be compared to the binarized vectors of attributes for other images to identify similar images. Hamming distance between two binarized vectors of equal length is the number of positions at which the corresponding symbols are different. For example, H([0 11],[1 1 1])=1; and H([0 0 1],[1 1 1])=2. Comparisons that have the smallest hamming distances yield the images that are most similar to the extracted image151.

FIG. 5shows an example of a method500. The method500and other methods described herein may be performed by one or more servers or other types of computers including at least one processor executing machine readable instructions embodying the methods. For example, the image processing server130and/or the image matching server131shown inFIG. 1may store machine readable instructions194embodying the methods, and processor191may execute the machine readable instructions. The method500is described by way of example as being performed by the system100.

At501, the target image150is received for example by the image processing server130. At502, meta data for the target image is determined. The meta data for example identifies a class of an object in the target image150. The meta data may be provided with the file containing the target image150or may be provided in a web site containing the target image150if the target image150is obtained from a web site. In an example, the meta data may be used to identify a particular CNN for determining the extracted image151from the target image. For example, if the CNN120comprises multiple CNNs trained to identify objects in different classes, the meta data may be used to identify a class of objects associated with the target image150, and the class of objects may be used to identify a particular CNN associated with the class. The target image150is applied to the identified CNN to determine the extracted image151. In some situations, meta data may not be available or the meta data may not identify a class or include information that can identify a particular CNN to use. In those situations, the target image150may be applied to multiple CNNs. The output with the highest confidence value may be selected to extract the extracted image151or if none of the outputs has a satisfactory confidence value, such as determined by comparing to a threshold, then no image may be extracted.

At503, the target image150and the meta data is applied to the image extractor CNN120to extract an object from the target image150if the target image150includes an object in a particular class of objects that the image extractor CNN120was trained to identify.FIG. 3A-Bdescribe an example of the image extractor CNN120identifying an object from the target image150and generating the extracted image151which includes the object. At504, the extracted image151is applied to the image attribute CNN121to determine attributes for the extracted image151. For example, a binarized vector of image features are determined by the image attribute CNN121for the extracted image151. At505, the attributes of the extracted image151are compared to attributes of other images. For example, the binarized vector of image features for the extracted image151is compared to binarized vector of image features for other images to find images that are similar to the extracted image151. At506, visually similar images are identified based on the comparisons performed at505. Visual similarity may be based on similarity of features of images visible to the human eye. Similarity may be determined based on a mathematical comparison of attributes (e.g., visual features), such as based on calculated Hamming distances. For example. “similar” images may include images that are a closest match when comparing the attributes of the extracted image151and the attributes of the stored images. If Hamming distance is used for the comparisons, then images with the smallest Hamming distances may be considered similar images. A Hamming distance threshold may be set such that if an image has a Hamming distance greater than the threshold, then the image is not considered a similar image to the extracted image151.

The system100may be used for a variety of applications. One example of an application of the system100is to identify and deliver visually similar content to online visitors of a web site.FIG. 6Ashows an example of an online visual content matching system600that may include the system100. For example, the online visual content matching system600may include the image processing server130, the image matching server131and the data repository175of the system100, which is described with respect toFIG. 1. The online visual content matching system600may also include an online content server601which is further described below, and the network119includes the Internet. The online visual content matching system600can identify content viewed by an online user visiting a website, such as one of websites651a,651bor651n, and identify stored content similar to the web site content viewed by the online user. The similar content may be delivered to the online user via the Internet.

FIG. 6Bshows an example of a method650performed by an online user and the online visual content matching system600. The method650is described with respect to the online visual content matching system600by way of example. At651, online user630visits a web site, such as web site651a.At602, the online content server601collects information about the online user's browsing activity. At603, the online content server601determines, based on the online user's browsing activity, visual content on the web sites visited by the online user630and which may have been clicked on or otherwise viewed by the online user630. The visual content may be the visual content652displayed on the web site651a.A cookie may be used to determine the visual content according to the online user's browsing activity. For example, a cookie is stored in the online user's browser that can track the online user's browsing activity. The cookie may be a third party cookie that can be sent to the online content server601to determine the browsing activity, including browsing history, of the online user630. The cookie may identify visual content652viewed or clicked on by the online user630at the web site651a.Also, the cookie may identify the uniform resource locator (URL) of web sites visited by the online user630. The online content server601may identify the visual content displayed on the web sites, such as by parsing the code, e.g., HyperText Markup Language (HTML) or other types of code, of the web sites. For example, the visual content652displayed on the web site651ais identified from the cookie or by parsing the code of the web site651ato identify images or image files. The visual content associated with the browsing activity of the online user630may be stored in the data repository175.

At604, the image processing server130determines the target image150from the visual content, e.g., visual content652, determined at603. For example, the visual content652may be an image provided in a GIF file or another type of file. The file is read to determine the target image150. At605, the image processing server130creates the extracted image151including an object identified from the target image150. For example, the image extractor CNN120identifies an object from the visual content652, and the object is cropped from the target image150to create the extracted image151. At606, the image attribute CNN121determines extracted image attributes160for the extracted image151.

At607, the image matching server131determines visual content similar to the visual content652. For example, the extracted image attributes160are compared to image attributes of other images to find images similar to the visual content652. For example, visual content653is identified as similar content to the visual content652. The identified similar content is referred to as the visually similar content. The similar content, e.g., images, determined at607may be images that are similar but are not exact matches or are not the same image previously viewed or clicked on by the online user630. If the same image or the same online advertisement including the same image is repeatedly delivered to the online user630, the online user630may ignore it. The image matching server131may identify similar images stored in the data repository175and filter out images that are considered to be the same as the extracted image, e.g., the same as the visual content652. For example, as shown inFIG. 7A.

At608, the online user630visits another web site, such as the web site651b.At609, the online content server601delivers the visually similar content, e.g., visual content653, to the web site651b.The tracking cookie may be used to determine the current web site visited by the online user630. The visual content653may be displayed in the web site651bto the online user630. The visually similar content may also be displayed on the web site651aif the online user630remains at the web site651ainstead of moving to the web site651b,or may be displayed at any web site the online user630visits if the web site has the ability to display visual content delivered by the online content server601.

According to an example, the visual content652may be an image of a product viewed by the online user630at the web site651a.The online visual content matching system600may be used to identify visually similar content to the image of the product viewed by the online user630at the web site651a.For example, as shown inFIG. 7A, images700may include images for candidate online advertisements or for candidate product recommendations that may displayed to online user630. An online advertisement may include a web page including at least one image. The images700are provided as input to the image attribute CNN121to determine the binarized vectors701, such as [0 1 1 0 . . . 1], etc., representing the visual features (also referred to as the attributes) of the images700. The images700and/or the binarized vectors701may be stored in the data repository175.FIG. 7Bshows an extracted image703, which may be an image of a product viewed by the online user630at the web site651a.The extracted image703is provided as input to the image attribute CNN121to determine the binarized vector702, representing the visual features (also referred to as the attributes) of the extracted image703. The binarized vectors701of the images700are compared to binarized vector702of the extracted image703to identify a closest matching image that is not the same image. For example, the comparison may include determining Hamming distances, and the Hamming distances are shown. The closest matching image is the image with a Hamming distance of25. An online advertisement or a product recommendation including the closest matching image (e.g., visually similar content) may be presented to the online user630. The visually similar content may be products that look similar to the product viewed at the web site651abut are different. For example, while the online user630is at the web site651a,the visually similar content may be delivered to the web site651aand presented to the online user630, so the online user630may be able to view products similar to the product displayed in the visual content652. Also, the visually similar content may be delivered to a different web site, such as the web site651b,when the online user630visits the web site651b.

In another example, the visually similar content may be provided in online advertisements. For example, the online user630views visual content652at web site651a.By way of example, the visual content652may be a product viewed on the web site651aor may include visual content associated with an online advertisement displayed and clicked on at the web site651a. The online content server601may include an online advertisement targeting platform that identifies online advertisements to deliver to an online user based on their preferences, needs, habits, etc. The online content server601identifies online advertisements for products that are visually similar to the visual content652. For example, the data repository175stores images of products and/or their attributes, such as determined by the image attribute CNN121. The image matching server131compares the attributes of the visual content652, which may also be determined by the image attribute CNN121, to the stored attributes of images of products to find visually similar products. The online content server601selects online advertisements for the visually similar products. The online content server601may deliver the online advertisements for the visually similar products to the web site651awhile the online user is at the web site651aor while the online user is at a different web site, such as the web site651b.Accordingly, the online user630may view online advertisements that are relevant to the browsing activities of the online user630and which may be relevant to the online user's preferences, needs, habits, etc. This is further illustrated by the example discussed below.

For example, the online user630user views fashion images with colorful dresses at the web site651a.The online visual content matching system600identifies visually similar images to the dresses viewed by the online user630and may present online advertisements or product recommendations for products from the visually similar images. The visually similar images may be for products having different brands, and the brands may be different than the brand of dresses viewed at the web site651a.Thus, the online user630can be presented with a variety of products and brands related to the visual browsing history of the online user630and match the visual preferences of the online user630. This is markedly different from conventional online advertisement retargeting platforms. Currently, when an online user visits different web sites, the online user may be repeatedly presented with online advertisements for the same product. Often, these online advertisements are ignored. The online visual content matching system600facilitates the delivery of online advertisements for different products which are visually related to the online user's browsing activity. Accordingly, the online visual content matching system600provides a technical solution to the technical problem of how to identify diverse visual content related to the online user's browsing activity. The technical solution may include the machine learning classifiers, such as the CNNs120and121, that facilitate the real-time selection and delivery of visually similar content to the online user.

According to an embodiment, a data repository stores image data for images; an image processing server receives a target image; determines meta data for the target image, wherein the meta data describes an object in the target image; applies the target image and the meta data to an image extraction convolutional neural network to identify an object in the target image and extract an image of the object; and applies the extracted image to an image attribute convolutional neural network to determine attributes represented by vectors of numeric values for the extracted image. An image matching compares the attributes for the extracted image with attributes of the images having image data stored in the data repository; and identifies images similar to the extracted image based on the comparison of the attributes for the extracted image with the attributes of the images having image data stored in the data repository. The image extraction convolutional neural network determines a location and a bounding box to extract the image of the object from the target image, and crop the target image around the bounding box to generate the extracted image of the object. The meta data of the target image comprises a class for each of the objects, and the image extraction convolutional neural network determines a location and a size of a bounding box for each object in the target image to extract an image of each object from the target image. The image extraction convolutional neural network determines a confidence value by a softmax layer in the neural network for each class indicating an accuracy of predictions by the image extraction convolutional neural network that the target images contains the objects at the locations. The image processing server is to reject objects associated with predictions having a confidence value that is less than a threshold.

According to another embodiment of the present disclosure, a machine learning image processing system800performs natural language processing (NLP) and auto-tagging for an image matching process. The NLP and auto-tagging may be used to enhance the image similarity determinations performed by the system100, as is further discussed below. The system800facilitates an interactive process, e.g., through a mobile application, to obtain an image and supplemental user input from a user to execute an image search. The supplemental user input may be provided from a user as speech or text, and NLP is performed on the supplemental user input to determine user intent and additional search attributes for the image search. Using the user intent and the additional search attributes, the system800may perform enhanced image matching on stored images that are tagged with attributes through an auto-tagging process. In an embodiment, the stored images may include images of products, and the enhanced image matching may be performed to identify matching product images that may be used as product recommendations for online shopping or other e-commerce. Accordingly, the system800may facilitate an improved and interactive method for providing product recommendations based on enhanced image searching. It will be apparent to one of ordinary skill in the art that the interactive and enhanced image matching performed by the system800may be used for applications and uses other than image-based product recommendation, including substantially any application that may utilize image matching.

With reference toFIG. 8, there is shown a system diagram of the system800. The system800may include the components of the system100but may further include an NLP subsystem801and an auto-tagging subsystem811to provide interactive and enhanced image searching and matching. As discussed above with respect to the system100shown inFIG. 1and other figures discussed above, the image extractor CNN builder115may build the image extractor CNN120, and the image attribute CNN builder116may build the image attribute CNN121. The image extractor CNN120may be used to identify objects in digital images, and the image attribute CNN121may be used to determine attributes of images classified by the image attribute CNN121, which may be used by the image comparator161and the similarity detector163to determine similar images165to a target image, such as target image150.

The system800may include the components of the system100to determine similar images165to a target image. For example, as shown inFIG. 8, the system800may include image processing server130and image matching server131similar to the system100. Also, the system800may include the classifier builder (e.g.,115and116) and the classifiers (e.g.,120and121) and the image matcher (e.g.,161and163) which may be used to determine similar images to a target image. An image matching subsystem of the system800may include the image matcher, the classifiers and other components for determining matching images. The image matching subsystem may determine predictions for tags by applying the classifiers to images, and can further utilize supplemental user input to determine a matching subset of images as is further discussed below, and transmit the matching subset of images to the mobile device841.

The system800includes the NLP subsystem801that may create and store an NLP model802for determining a supplemental image search attribute based on supplemental user input. Also, the auto-tagging subsystem811may include an auto-tagging classifier812to determine attributes of images stored in the data repository175, and the images may be tagged with their attributes determined by the auto-tagging classifier812. Tagging the images with their attributes may include storing the attributes determined for each image, such as in the form of meta data for each image, in the data repository175. The attributes tagged to each image may be used to further search the image repository175for matching images based on the supplemental user input142.

The system800may communicate with a mobile application842hosted on a mobile device841to perform various operations discussed below. An example of a platform890, including hardware and software components for the mobile device841, is shown. The mobile device841includes a processor (e.g., one or more processors) and data storage, including memory895. The processor891for example is an integrated circuit. The processor891may be a chipset with central processing unit and/or custom processing circuits, such as an application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) and/or a graphic processing unit. The processor891may run an operating system (OS)843and applications, including the mobile application842. The OS843and the applications may include machine readable instructions894stored in the memory895or other non-transitory data storage and executed by the processor891. The mobile device841includes input/output (I/O) devices893such as keyboard, display (e.g., touch screen display), speaker, microphone, digital camera, etc. The mobile device841may include I/O interface892(e.g., one or more I/O interfaces) which may be wired and/or wireless, for communicating with other devices. For example, the I/O interface892may include a Wi-Fi interface, a cellular interface, a Bluetooth interface, etc. The mobile device841may include a smart phone, tablet, laptop, or another type of mobile device. Also, instead of a mobile device, another type of device, such as a desktop computer, workstation, etc., may be used. Also, instead of mobile application842, a web application may be used to provide one or more operations of the mobile application842.

Examples of operations performed by the mobile device841and the system800are now described. A user840may launch the mobile application842on the mobile device841and be prompted to capture an image of an object. For example, the mobile application841may be an application that provides product recommendations based on image matching and may facilitate purchase of a product. The user840captures an image of an object that the user may be desired by the user840. The user840may take a picture of the object with a camera of the mobile device841to capture a digital image of the object, e.g., target image850. The mobile application842may transmit the target image850to the system800via the network119. The system800determines matching images143afrom the data repository175that match the target image850. For example, such as described with respect toFIG. 1, the image extractor CNN120may be used to identify an object in a digital image, such as an object in the target image850, and the image attribute CNN121may be used to determine attributes of the image of the object in the target image850. The image comparator161and the similarity detector163may determine images from the data repository175that match the target image850, such as images (e.g., matching images143a) that match an object extracted from the target image850.

The user840may provide supplemental user input142including additional criteria for searching for images similar to the matching images143a. In an embodiment, the system800may transmit the matching images143ato the mobile device841, and the mobile application842displays the matching images143aon the mobile device841. The user840may view the matching images143a,and provide supplemental user input142to further define the criteria for identifying matching images and products. In another embodiment, the user may provide the supplemental user input142without the system800providing the matching images143ato the mobile application842. In an example, the user840may provide the supplemental user input142in the form of speech or text. Also, speech input may be converted to text by a speech-to-text module of the mobile application842. The mobile application842transmits the supplemental user input142to the system800via the network119, and the system800receives the supplemental user input142, and provides the supplemental user input142to the NLP model802. The NLP model802determines one or more supplemental image search attributes from the supplemental user input142. The image matching server131may use the one or more supplemental image search attributes and the attributes tagged to the images stored in the data repository175to determine a new set of matching images, e.g., matching images143b.The system800may transmit the matching images143bto the mobile device841, and the mobile application842may display the matching images143b.The process may be repeated any number of times. For example, the user840may provide additional supplemental user input after viewing the matching images143b,and the system800may determine a new set of matching images from the matching images143band the additional supplemental user input for transmission to the mobile device841. In an example, the mobile application842displays the matching images143b,and the user840may select an image of the matching images143bto determine more information about a product displayed in the matching image and/or to purchase the product.

FIG. 9is a flow diagram showing operations performed by the mobile application842and the system800. For example, the user841takes a picture with the mobile device841to capture an image of an object for a search. The captured image is shown as the target image850, and the object in the image, for example, is a plaid, women's shirt. The mobile application842may be executed on the mobile device841to allow the user to capture the target image850via a digital camera of the mobile device841, and the mobile application842sends the target image850to the system800. The system800performs similarity matching, such as discussed above, to determine the matching images143afrom the data repository175. The tagged attributes of the matching images143aare determined, for example, from the meta data for the matching images143a.The meta data may include the tagged attributes determined through the auto-tagging process. Examples of the tagged attributes of the matching images143aare shown inFIG. 9. For example, the tagged attributes, labeled901for a first matching image910, include Dress (82%) and Red (80%). The numeric values in parentheses represent a confidence value of a prediction made by the auto-tagging classifier812that the image includes an object belonging to particular class. For example, the auto-tagging classifier812predicts that the first matching image910has an 82% probability of belonging to the dress class and an 80% probability of belonging to the red class. The predicted classes may be for categories of tags, also referred to as tag types. For example, the categories of tags may include brand, color, price, size, style, garment type, etc., and under each category of tag, there may be several classes. For example, under color, there may be classes for red, green, blue, etc., and under garment type, there may be classes for dress, skirt, shirt, etc. Other examples of tagged attributes are shown inFIG. 9for other ones of the matching images143a.

An example of the supplemental user input142is also shown inFIG. 9. For example, the user840speaks into a microphone of the mobile device841to provide the supplemental user input142to the mobile application842. The speech may include “I wanna a same style shirt, but a green one”. The speech may be converted to text, and the text is provided to the NLP subsystem801. The NLP subsystem801applies the text to the NLP model802to determine object attribute search criteria from the text. For example, the NLP subsystem801determines that the intent of the user's speech was to search for the same style shirt as shown in the target image850, but in a green plaid instead of a red plaid. Then, the image searching is enhanced to identify green plaid shirts from the matching images143aand/or other images stored in the data repository175to identify the matching images143b.The matching images143bmay be sorted based on relevance, so images that are less likely to be matches may be placed in the bottom row. The matching images143bmay be displayed to the user840via the mobile application842, and the user840may select one of the matching images to execute a purchase of a product shown in selected one of the matching images143b,such as if the mobile application842is an ecommerce mobile application.

FIG. 10shows the auto-tagging process, according to an embodiment. For example, the auto-tagging classifier812makes class predictions on images input to the auto-tagging classifier812. For example, the image1000is provided as input to the auto-tagging classifier812, and the output of the auto-tagging classifier812are predictions1001. The predictions1001include tags1-4. Each tag may be a class, and includes a probability of whether the image1000belongs to the class. For example, tags1-4indicate respectively that the auto-tagging classifier812predicts the image1000has a 95% probability of containing a plaid object; a 90% probability of containing a half sleeve object; an 82% probability of containing a dress, and an 80% probability of containing a red object. All or some of the images in the data repository175may be provided as input to the auto-tagging classifier812to determine tag predictions for each image and the tag predictions, e.g., the attributes, are stored with the images, and may be referred to as tagged attributes.

The auto-tagging classifier812may be a CNN such as shown inFIG. 4B, and the auto-tagging classifier812may be trained such as described above with respect toFIGS. 2 and 3A. For examples, labeled training sets may be used to train the auto-tagging classifier812to classify images into various classes. The types of tags, some of which are shown inFIGS. 9 and 10, may be predefined depending on the objects being classified. InFIGS. 9 and 10, the objects in images that are being classified, for example, are garments, and the predefined types of tags may be related to garments or fashion, such as brand, color, price, size, style, etc. The auto-tagging classifier812may make tag predictions for each type of tag. In an example, each type of tag may include multiple classes, such as different colors for the color tag type or different sizes for the size tag type, and so on. The auto-tagging classifier812may make predictions for each class. An auto-tagging classifier may be created for each tag type and determine predictions for the classes under the tag type. For example, the auto-tagging classifier812may include an auto-tagging classifier for brand (e.g., logo), an auto-tagging classifier for color, an auto-tagging classifier for price, an auto-tagging classifier for size, and an auto-tagging classifier for style. The auto-tagging classifier for color may make predictions for multiple colors. For example, the image1000is provided as input to the auto-tagging classifier for color, and the output may include a probability for a class prediction for each of multiple colors. Similarly, each of the other auto-tagging classifiers may make predictions for each class under the tag type being predicted by the respective auto-tagging classifier. The image may be tagged with the class having the highest probability under each tag type. In an example, the auto-tagging classifier812may include a classifier used for the similarity matching to determine the matching images143a.

FIG. 11shows an example of using the NLP model802. For example, the supplemental user input142may include speech, such as “I wanna a same style shirt, but a green one”. The speech may be converted to text by a speech-to-text program or module, and the text is provided to the NLP subsystem801. The NLP subsystem801applies the text to the NLP model802to determine the object attribute search criteria1101from the text. For example, the NLP subsystem801determines that the intent of the user's speech was to search for the same style shirt as shown in the target image850, but in a green plaid instead of a red plaid. The NLP model802may be trained according to a predetermined dictionary of terms related to fashion or garments. The terms may include the tag types and classes discussed above. Also, terms such as “same” or “different” or “but” or “and” may be used to determine whether attributes of the target image850should be modified or whether additional attributes should be used to search for the matching images143b.In an example, the Natural Language Toolkit (NLTK), which an off-the-shelf suite of libraries and programs for symbolic and statistical natural language processing for English written in the Python® programming language, may be used to build the NLP model802.

FIG. 12shows an example of the system800generating visual recommendations, which may include visual product recommendations comprised of the matching images143b.The client catalogue may include product images, and may be stored in the data repository175. The product images may be tagged by the auto-tagging subsystem811. The input to the system800may be an image captured by the mobile application842, such as the target image850. The input to the system800may also include the supplemental user input142comprised of speech. Similarity matching performed by the system800may determine the matching images143a.The NLP subsystem801may determine the object attribute search criteria1101from the supplemental user input142, and the system800determines a recommendation, such as the matching images143bfrom the matching images143a,the object attribute search criteria1101, and the tagged images in the data repository175. The matching images143bmay be displayed by the mobile application842on the mobile device841.

FIG. 13shows an example of a method1300. The method1300may be performed by the system800or another system to perform image matching. The method1300is described by way of example as being performed by the system800, and may be performed by other systems. The method1300and other methods described herein may be performed by one or more servers or other types of computers including at least one processor executing machine readable instructions embodying the methods. For example, the image processing server130and/or the image matching server131shown inFIG. 8may store machine readable instructions194embodying the methods, and processor191may execute the machine readable instructions. Also, one or more steps of the method1300may be performed according to the steps of the method500shown inFIG. 5. For example, the method500includes steps501-506for identifying visually similar images. These steps may be performed for steps1301-1303to determine the initial set of matching images, such as the matching images143a.Also, one or more of the steps of the method1300and steps of other methods described herein may be performed in a different order than shown or substantially simultaneously.

At step1301, the system800receives the target image850. In an example, the target image850may be captured and sent to the system800by the mobile device841executing the mobile application842, and the system800receives the target image850, via a network interface (e.g., I/O interface192), connecting a computer of the system800to the network119.

At1302, the system800applies the target image850to at least one image attribute machine learning classifier. For example, the target image850is applied to the auto-tagging classifier812to determine attributes of the target image850. The auto-tagging classifier812may determine predictions for multiple tag types (e.g., brand, color, price, size, style, garment type, etc.) and multiple classes for each tag type. The auto-tagging classifier812may comprise multiple classifiers. The target image850may be tagged with a class for each tag type that has the highest probability or confidence value. In an example, the target image850may be applied to the CNNs120and121to determine attributes of the target image850. The CNNs121may include the auto-tagging classifier812or another classifier that can classify for one or more tag types.

At1303, the system800determines an initial set of matching images, such as the matching images143a.The initial set of matching images (e.g., the matching images143a) may be determined by comparing the attributes of the target image850determined at step1302to attributes of images stored in the data repository175. In an example, the matching images143amay be determined according to the similarity matching described with respect to steps505-506of the method500. Also, the matching images143amay be sent to the mobile application842via the network119, which may prompt the user840to provide the supplemental user input142.

At1304, the system800receives the supplemental user input142via the network119. For example, the user840may provide speech or text further explaining search criteria. The search criteria may be based on the target image850and/or an image of the matching images143a.

At1305, the system800applies the supplemental user input142to the NLP model802to determine at least one supplemental image search attribute. For example, the supplemental user input142may include the following speech “I wanna a same style shirt, but a green one”. The speech may be converted to text, and the text is provided to the NLP model802to determine at least one supplemental image search attribute. In this example, the supplemental image search attribute may include green plaid.

In an example, the at least one supplemental image search attribute determined from the output of the NLP model802may be used as object attribute search criteria. Also, the system800may determine whether the at least one supplemental image search attribute is a modification to attributes of the target image850determined at1302or is an additional attribute. For example, the speech “but a green one” is determined to be a modification because the NLP model802recognizes “but” as a modifier to an existing attribute of the target image850. The modified attribute is “green”. The NLP model802may determine that the attributes of the target image850include red plaid, and modifies red plaid to green plaid for the image search. If the speech included “and Gucci brand” then the NLP model802may determine the at least one supplemental image search attribute is an additional attribute for the brand tag type in the Gucci class.

At1306, the system800identifies a matching subset of the images (e.g., the matching images143b) stored in the data repository175that match the target image850and the supplemental user input142. For example, the attributes of the target image850and the at least one supplemental image search attribute are used to search the tags of the images in the matching images143aand/or the images in the data repository175to identify the matching images143b.

At1307, the matching images143bmay be transmitted to the mobile application842via the network119. Then, the user840may view the matching subset of images to the mobile application for display by the mobile application. The system800may transmit the matching images143bto the mobile device841, and the mobile application842may display the matching images143b.The method1300may be repeated any number of times. For example, the user840may provide additional supplemental user input after viewing the matching images143b,and the system800may determine a new set of matching images from the matching images143band the additional supplemental user input for transmission to the mobile device841.