Patent Description:
In recent years, electronic commerce (E-commerce), through which products are sold using the Internet, has been actively carried out, and many EC (Electronic Commerce) sites have been built on the web to carry out such electronic commerce. EC sites are often built using the languages of countries around the world so that users (consumers) in many countries can purchase products. By accessing EC sites from a personal computer (PC) or a mobile terminal such as a smartphone, users can select and purchase desired products without visiting actual stores, regardless of the time of day.

There is a known function to search for, and present, one or more similar images based on an image of the product specified by the user (a product image), including an image of a product similar to the specified product, for the purpose of increasing the user's willingness to purchase products.

For example, Patent Literature Document <NUM> discloses a technique for deleting a background image from a product image to extract a product area, and searching for an image that includes an area similar to the product area.

Such a function can also be used to search for similar products in response to a user's request at a store that sells products dealt with on an EC site, using a terminal (a store terminal) provided at the store.

<CIT> discloses a method and an apparatus for extracting a representative feature of an object. The method includes receiving a query image, generating a saliency map for extracting an inner region of an object corresponding to a specific product included in the query image by applying the query image to a first learning model that is trained on a specific product, applying the saliency map as a weight to a second learning model that is trained for object feature extraction, and extracting feature classification information of the inner region of the object by inputting the query image into the second learning model to which the weight is applied.

In the paper entitled "<NPL>et al. disclose a panoptic segmentation network based on the YOLOv3 real-time object detection network by adding a semantic and instance segmentation branches, that fully parses a scene by assigning each pixel a label and instance id.

According to the technique disclosed in Patent Literature Document <NUM>, similar images are searched for based on a product area extracted from a product image, and at this time, the color of the product area is also taken into consideration when the search is carried out. To extract the color of a product from a product image, it is first necessary to accurately extract the area of the product. However, according to the technique disclosed in Patent Literature Document <NUM>, the foreground image and the background image of an image are simply separated from each other. Therefore, there is the possibility of the areas of the products subjected to the similarity search being not accurately extracted, and the extracted colors not properly representing the colors of the products.

The present invention is made in view of the above problems, and an objective thereof is to provide a technique for extracting a color that is to be used for a search, from an input image, with high accuracy.

The present invention provides an information processing apparatus, in accordance with claim <NUM>.

The present invention also provides an information processing method, in accordance with claim <NUM>.

The present invention also provides an information processing program, in accordance with claim <NUM>.

The present invention also provides an information processing system, in accordance with claim <NUM>.

According to the present invention, it is possible to extract a color that is to be used for a search, from an input image, with high accuracy. In addition, by using information regarding the extracted color, it is possible to carry out a highly accurate similar image search, which improves usability.

A person skilled in the art will be able to understand the above-mentioned objective, aspects, and effects of the present invention and objectives, aspects, and effects of the present invention that are not mentioned above from the following embodiments for carrying out the invention by referencing the accompanying drawings and the recitations in the scope of claims.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. Among the constituent elements disclosed below, those having the same function are designated by the same reference numerals, and the descriptions thereof will be omitted. Note that the embodiments disclosed below are examples of means for realizing the present invention, and should be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and on various conditions. The present invention is not limited to the embodiments below. In addition, not all the combinations of the features described in the embodiments are essential for the solutions according to the present invention.

<FIG> shows a configuration of an information processing system according to the present embodiment. The information processing system includes a user device <NUM> such as a terminal device or a store terminal provided in a store, and an information processing apparatus <NUM>.

The user device <NUM> is a device such as a smartphone or a tablet, and is configured to be able to communicate with the information processing apparatus <NUM> via a public network such as an LTE (Long Term Evolution) network or a wireless communication network such as a wireless LAN (Local Area Network). The user device <NUM> includes a display unit (a display surface) such as a liquid crystal display, and the user can perform various operations, using a GUI (Graphic User Interface) provided on the liquid crystal display. The operations include various operations on contents such as images displayed on the screen, e.g., a tap operation, a slide operation, and a scroll operation that are performed with a finger, a stylus, or the like.

The user device <NUM> may be a device such as a desktop PC (Personal Computer) or a laptop PC. In such a case, the user uses an input device such as a mouse or a keyboard to perform an operation. The user device <NUM> may be provided with a separate display surface.

The user device <NUM> transmits a search query to the information processing apparatus <NUM>, following a user operation. A search query corresponds to a request that is associated with an image (a product image (an object image)) that includes a product (an object), and has been made to carry out a search for similar images that are similar to the product image (images that include a product that is similar to the product). In the following description, a product image subjected to a similar image search may also be referred to as a query image. For example, the user can send a search query by selecting one product image from one or more product images displayed on the display unit of the user device <NUM> as a query image, and thereafter selecting a predetermined search button. The search query can include (can be associated with) information regarding the query image in a format that can be decoded by the information processing apparatus <NUM> or a URL format.

The information processing apparatus <NUM> is a server device that can be used to build an EC site and distribute web contents. In the present embodiment, the information processing apparatus <NUM> is configured to be able to provide a search service. Through the search service, the information processing apparatus <NUM> can generate content (a search result) corresponding to a search query received from the user device <NUM>, and distribute (output) the content to the user device <NUM>.

The information processing apparatus <NUM> according to the present embodiment acquires a product image associated with a search query received from the user device <NUM>, generates a plurality of feature vectors with reference to a plurality of attributes of the product included in the product image, generates a compounded feature vector in which the plurality of feature vectors are concatenated with each other, and searches for similar images that are similar to the product image, using the compounded feature vector.

<FIG> shows an example of a functional configuration of the information processing apparatus <NUM> according to the present embodiment.

The information processing apparatus <NUM> shown in <FIG> includes an acquisition unit <NUM>, a first feature inference unit <NUM>, a second feature inference unit <NUM>, a gender inference unit <NUM>, a color extraction unit <NUM>, a concatenation unit <NUM>, a similarity search unit <NUM>, a training unit <NUM>, an output unit <NUM>, a learning model storage unit <NUM>, and a search database <NUM>. The learning model storage unit <NUM> stores various learning models (a first feature predictive model <NUM>, a second feature predictive model <NUM>, a gender predictive model <NUM>, and a segment extraction model <NUM>) that are to be used by the first feature inference unit <NUM>, the second feature inference unit <NUM>, the gender inference unit <NUM>, and the color extraction unit <NUM>. The various learning models will be described later. The search database <NUM> is a database that stores information related to similar image searches, and may be provided outside the information processing apparatus <NUM>.

The acquisition unit <NUM> acquires a product image (a query image). In the present embodiment, the acquisition unit <NUM> receives a search query transmitted by the user device <NUM> and acquires a product image associated with (included in) the search query.

In a background example not forming part of the invention, the product image may be an image expressed by three colors, namely red (R), green (G), and blue (B). Alternatively, in another background example not forming part of the invention, the product image may be an image expressed by a luminance (Y (Luma)) representing brightness and color components (Cb and Cr (Chroma)) (an image generated from an RGB image (YCbCr image) through a YCbCr-conversion). In an embodiment, the product image is data (a coefficient) generated from a YCbCr image through a DCT (Discrete Cosine Transform) conversion (compression) performed by a coding unit (not shown) included in the information processing apparatus <NUM>. It is also possible to employ a configuration in which the acquisition unit <NUM> acquires data, which is a product image, subjected to (a YCbCr conversion and) a DCT conversion performed by a device other than the information processing apparatus <NUM>.

The acquisition unit <NUM> outputs the acquired product image to the first feature inference unit <NUM>, the second feature inference unit <NUM>, the gender inference unit <NUM>, and the color extraction unit <NUM>.

The first feature inference unit <NUM>, the second feature inference unit <NUM>, the gender inference unit <NUM>, the color extraction unit <NUM>, and the concatenation unit <NUM> will be described with reference to <FIG> as well. <FIG> is a conceptual diagram showing feature vectors and a compounded feature vector.

The first feature inference unit <NUM> applies the product image (corresponding to an input image <NUM> in <FIG>) acquired by the acquisition unit <NUM>, to the first feature predictive model <NUM>, and performs supervised learning to infer (predict) a first feature of the product and generate a first feature vector <NUM> that indicates the first feature. The first feature indicates an upper-level (generalized) classification of the product, and is also referred to as a category. Note that, in the present specification, a feature vector represents a value/information that represents a feature.

The second feature inference unit <NUM> applies the product image acquired by the acquisition unit <NUM>, to the second feature predictive model <NUM>, and performs supervised learning to infer (predict) a second feature of the product and generate a second feature vector <NUM> that indicates the second feature. The second feature indicates a lower-level (subdivided) classification of the product, and is associated with the first feature. The second feature is also referred to as a genre. Note that the second feature inference unit <NUM> may be configured to apply the product image to the first feature predictive model <NUM> to infer the first feature, and infer the second feature from the inferred first feature.

As described above, the first feature indicates an upper-level (generalized) product classification type, and the second feature indicates a lower-level (subdivided) product classification type.

Specific examples of the first feature (category) include product classification types such as men's fashion, ladies' fashion, fashion goods, innerwear, shoes, accessories, and watches.

When the first feature is ladies' fashion, examples of the second feature (genre) include product category types such as pants, a shirt, a blouse, a skirt, and a one-piece dress.

The first feature inference unit <NUM> and the second feature inference unit <NUM> respectively output the generated first feature vector <NUM> and second feature vector <NUM> to the concatenation unit <NUM>.

The gender inference unit <NUM> applies the product image acquired by the acquisition unit <NUM> to the gender predictive model <NUM> and performs supervised learning to infer (predict) the gender targeted by the product and generate a gender feature vector <NUM> indicating the gender. In the present embodiment, the gender inference unit <NUM> can identify not only the gender such as male or female but also other classifications such as kid and unisex.

The gender inference unit <NUM> outputs the generated gender feature vector <NUM> to the concatenation unit <NUM>.

The color extraction unit <NUM> applies the product image acquired by the acquisition unit <NUM> to the segment extraction model <NUM> and performs supervised learning to acquire a product area as a segment, extract colors of the segment (the segmented area) (corresponding to color extraction <NUM> in <FIG>), and generate a color feature vector <NUM> indicating the colors. The processing performed by the color extraction unit <NUM> will be described later.

The color inference unit <NUM> outputs the generated color feature vector <NUM> to the concatenation unit <NUM>.

The concatenation unit <NUM> concatenates the feature vectors output by the first feature inference unit <NUM>, the second feature inference unit <NUM>, the gender inference unit <NUM>, and the color extraction unit <NUM> with each other, embeds these feature vectors in a multidimensional feature space (hereinafter referred to as a feature space), to generate a compounded feature vector <NUM> (corresponding to concatenation <NUM> in <FIG>). That is to say, the concatenation unit <NUM> generates a compounded feature vector <NUM> by concatenating the first feature vector <NUM>, the second feature vector <NUM>, the gender feature vector <NUM>, and the color feature vector <NUM> with each other, and embedding the concatenated feature vector into a single (common) feature space.

As will be described later, the first feature vector <NUM> is expressed in <NUM> dimensions (200D), the second feature vector <NUM> is expressed in <NUM> dimensions (153D), the gender feature vector <NUM> is expressed in four dimensions (4D), and the color feature vector <NUM> is expressed in six dimensions (6D). Therefore, the compounded feature vector <NUM> is expressed in <NUM> dimensions (363D).

In the compounded feature vector <NUM>, as shown in <FIG>, the gender feature vector <NUM>, the second feature vector <NUM>, the color feature vector <NUM>, and the first feature vector <NUM> are concatenated in this order. This order of concatenation is an example, and the order is not limited to this order.

The concatenation unit <NUM> outputs the generated compounded feature vector <NUM> to the similarity search unit <NUM>.

Using the compounded feature vector <NUM> generated by the concatenation unit <NUM> as an input, the similarity search unit <NUM> searches for similar images that are similar to the product image acquired by the acquisition unit <NUM>. In the present embodiment, the similarity search unit <NUM> carries out a similar image search in the feature space. The similarity search unit <NUM> is configured to search for similar images using, for example, a known nearest neighbor search engine. For example, an engine that employs the FAISS (Facebook AI Similarity Search) algorithm is known as a nearest neighbor search engine. Note that the entirety or a part of the configuration of the similarity search unit <NUM> may be provided outside the information processing apparatus <NUM> so as to be associated therewith.

The output unit <NUM> outputs information including one or more images (similar images) corresponding to one or more image IDs that are the results of the search carried out by the similarity search unit <NUM>. For example, the output unit <NUM> may provide such information via a communication I/F <NUM> (<FIG>).

The training unit <NUM> trains the first feature predictive model <NUM>, the second feature predictive model <NUM>, the gender predictive model <NUM>, and the segment extraction model <NUM>, and stores these trained learning models in the learning model storage unit <NUM>.

In the present embodiment, the first feature predictive model <NUM>, the second feature predictive model <NUM>, and the gender predictive model <NUM> are each a learning model for machine learning that employs an image recognition model. <FIG> shows a schematic architecture of the image recognition model. The segment extraction model <NUM> will be described later.

As shown in <FIG>, the image recognition model according to the present embodiment includes intermediate layers that include a plurality of convolution layers, and an output layer used to classify/predict classes, and outputs feature vectors predicted based on an input product image. For example, the EfficientNet model by Google Research is used for intermediate layers. When EfficientNet is used, MBConv (Mobile Inverted Bottleneck Convolution) is used for each convolution layer. The layers are configured such that a feature map is extracted in the intermediate layers, and the ultimate feature vector is generated in the output layer while the number of dimensions of the map is reduced. Note that the number of convolution layers is not limited to a specific number.

The first feature predictive model <NUM>, the second feature predictive model <NUM>, and the gender predictive model <NUM> may each employ the architecture of the image recognition model shown in <FIG>, and respectively output the first feature vector <NUM>, the second feature vector <NUM>, and the gender feature vector <NUM>.

The first feature predictive model <NUM>, the second feature predictive model <NUM>, and the gender predictive model <NUM> are each subjected to training processing using individual training (teacher) data. Here, training processing that these learning models are subjected to will be described.

First feature predictive model <NUM>: A model that predicts the first feature (a category (an upper-level classification of the product)) from the product image and outputs the first feature vector <NUM>. Combinations of a product image (an input image) and the category of the product serving as correct answer data are used as training data. In training data, the categories of products have been set in advance, and it is assumed that there are <NUM> different categories in the present embodiment. Examples of the categories of fittings include men's fashion, ladies' fashion, fashion goods, innerwear, shoes, accessories, and watches, as mentioned above. Categories may also include food, gardening, computers/peripherals, and so on.

In the present embodiment, the first feature predictive model <NUM> is configured to be able to classify <NUM> different categories, and the first feature vector <NUM> is a vector that can express <NUM> dimensions.

Second feature predictive model <NUM>: A model that predicts the second feature (a genre (a lower-level classification of the product)) from the product image and outputs the second feature vector <NUM>. Combinations of a product image (an input image) and the genre of the product serving as correct answer data are used as training data. In training data, the genres of products have been set in advance in association with the categories that are upper-level classifications.

In the present embodiment, the second feature predictive model <NUM> is configured to be able to infer <NUM> different genres for each first feature vector <NUM> (category) generated by the first feature inference unit <NUM>, and the second feature vector <NUM> is a vector that can express <NUM> dimensions.

Alternatively, the second feature predictive model <NUM> may be configured to infer the first feature to generate a first feature vector <NUM>, and infer the second feature to generate a second feature vector <NUM> based on the first feature.

Gender predictive model <NUM>: A model that predicts a gender from the product image and outputs the gender feature vector <NUM>. Combinations of a product image (an input image) and gender information regarding the gender targeted by the product, which serves as correct answer data, are used as training data. As described above, in the present embodiment, examples of genders include not only male and female but also kid and unisex. In training data, gender features corresponding to products have been set in advance.

The gender predictive model <NUM> is configured to be able to infer four different genders (male, female, kid, and unisex), and the gender feature vector <NUM> is a vector that can express four dimensions.

Note that the gender predictive model <NUM> may be configured to predict gender based on the first feature vector <NUM> and/or the second feature vector <NUM> rather than from the image recognition model shown in <FIG>, and generate and output the gender feature vector <NUM>.

Next, the processing performed by the color extraction unit <NUM> will be described together with the segment extraction model <NUM>.

<FIG> is a conceptual diagram showing a flow of the processing performed by the color extraction unit <NUM> according to the present embodiment. In the present embodiment, the color extraction unit <NUM> is configured to use data generated by converting an RGB image to a YCbCr image and further performing DCT-conversion on the YCbCr image, as an input product image, to extract the colors of the product. Such conversion processing may be performed by the acquisition unit <NUM> or a device other than the information processing apparatus <NUM>.

A block <NUM> shown in <FIG> represents processing that is performed from when product images are acquired by the color extraction unit <NUM> to when the images are input to the segment extraction model <NUM>. The input images subjected to the YCbCr conversion and the DCT conversion are denoted as DCT-Y501, DCT-Cb501, and DCT-Cr501. For example, the images DCT-Y501, DCT-Cb501, and DCT-Cr501 respectively contain components [<NUM>,<NUM>,<NUM>], [<NUM>,<NUM>,<NUM>], and [<NUM>,<NUM>,<NUM>], and the dimensionalities of each image represent [the number of channels (n_channels), width, height].

Subsequently, the color extraction unit <NUM> performs upsampling processing on DCT-Cb502 and DCT-Cr503 to generate DCT-Cb504 and DCT-Cr505. Thereafter, the color extraction unit <NUM> concatenates DCT-Y501, DCT-Cb504, and DCT-Cr505 channel-wise to generate concatenated DCT data <NUM>. That is to say, the sizes of the Y elements, the Cb elements, and the Cr elements are adjusted, and the concatenated DCT data <NUM> (image data) is generated. The concatenated DCT data <NUM> is input to the segment extraction model <NUM>.

The segment extraction model <NUM> includes an object detection model <NUM> and a semantic segmentation model <NUM>. The object detection model <NUM> is, for example, a YOLO (You Only Look Once) learning model. The semantic segmentation model <NUM> is a learning model that is constituted by a neural network such as an FCN (Fully Convolutional Network).

The object detection model <NUM> is a learning model that uses the concatenated DCT data <NUM> as input to infer (predict) a bounding box of at least one object contained in the input image (a rectangular area that surrounds the object) and the type (class) of the object. Here, not only the bounding box and type of the object to be searched for (i.e., the product to be searched for) but also the bounding box and type of an object that is not to be searched for may be included. The inferred bounding box and type are input to the semantic segmentation model <NUM>.

The semantic segmentation model <NUM> is a learning model that uses the inferred bounding box and class as inputs to infer (predict) the area (segmentation) of the product to be searched for (the target object). In the present embodiment, the semantic segmentation model <NUM> is configured to output pixel information regarding the segmented area (i.e., the product area). Note that the semantic segmentation model <NUM> may be configured to use only the inferred bounding box as an input to output pixel information regarding the segmented area.

<FIG> shows an example of an architecture of the segment extraction model <NUM>. As described above, the segment extraction model <NUM> includes the object detection model <NUM> and the semantic segmentation model <NUM>.

The object detection model <NUM> includes a backbone section <NUM> for pre-training and a head section <NUM> for inferring the bounding box and the class (type). Here, Conv indicates a convolutional layer, Bottleneck CSP (Cross Stage Partial Network) indicates a layer that formulates a feature, and SPP (Spatial Pyramid Pooling) indicates a pooling layer. In the semantic segmentation model <NUM>, Identity indicates an activation function that performs nothing in particular. The object detection model <NUM> and the semantic segmentation model <NUM> are trained by the training unit <NUM>, using any given product images as training data.

Next, processing through which colors of the segmented area (the product area) are extracted from pixel information regarding the area, generated as described above, to generate a color feature vector <NUM> will be described. <FIG> shows a flow of the color extraction processing.

The color extraction unit <NUM> acquires pixel information (S51), and subsequently extracts colors in a color palette, using a color map, for each pixel (S52). The following steps S52 - S54, when using RGB or YCbCr, do not form part of the invention. Embodiments use data generated from a YCbCr image through a DCT conversion. In S52, for example, the color extraction unit <NUM> converts the segmented area (from RGB or YCbCr) into a Lab color space to express the colors of all the pixels in the area as color values. Note that the colors in the color palette are also converted into those in the Lab color space. The color extraction unit <NUM> can determine the distance (difference) between two color values for each pixel to extract the color of each pixel.

Next, the color extraction unit <NUM> selects the top two colors as the primary color (first color) and the secondary color (second color) from among the colors extracted from all the pixels in the area (S53), and outputs color information vector <NUM> that includes RGB information regarding the two colors (S54). By extracting not only one color but the top two colors of the colors that exclusively occupy the area showing the product, even if the area contains a plurality of colors like a dot pattern or a checkered pattern, it is possible to generate a color information vector <NUM> that more precisely reflects information regarding the colors in the area, to perform a similar image search.

As described above, in the present embodiment, the color information vector <NUM> is a vector that can express six dimensions, i.e., <NUM> (RGB) × <NUM> (the primary color and the secondary color).

Although the top two colors are extracted in the present embodiment, a predetermined number of colors from the top color may be extracted.

<FIG> is a block diagram showing an example of a hardware configuration of the information processing apparatus <NUM> according to the present embodiment.

The information processing apparatus <NUM> according to the present embodiment can be implemented on one or more computers of any type, one or more mobile devices of any type, and one or more processing platforms of any type.

Although <FIG> shows an example in which the information processing apparatus <NUM> is mounted on a single computer, the information processing apparatus <NUM> according to the present embodiment may be mounted on a computer system that includes a plurality of computers. The plurality of computers may be connected to each other via a wired or wireless network so as to be able to communicate with each other.

As shown in <FIG>, the information processing apparatus <NUM> may include a CPU <NUM>, a ROM <NUM>, a RAM <NUM>, an HDD <NUM>, an input unit <NUM>, a display unit <NUM>, a communication I/F <NUM>, and a system bus <NUM>. The information processing apparatus <NUM> may also be provided with an external memory.

The CPU (Central Processing Unit) <NUM> performs overall control on the operation of the information processing apparatus <NUM>, and controls each of the components (<NUM> to <NUM>) via the system bus <NUM>, which is a data transmission line.

The ROM (Read Only Memory) <NUM> is a non-volatile memory that stores a control program or the like required for the CPU <NUM> to perform processing. Note that the program may be stored in a non-volatile memory such as an HDD (Hard Disk Drive) <NUM> or an SSD (Solid State Drive), or an external memory such as a removable storage medium (not shown).

The RAM (Random Access Memory) <NUM> is a volatile memory and functions as a main memory, a work area, or the like of the CPU <NUM>. That is to say, the CPU <NUM> loads a required program or the like from the ROM <NUM> into the RAM <NUM> when performing processing, and executes the program or the like to realize various functional operations.

The HDD <NUM> stores, for example, various kinds of data and various kinds of information required for the CPU <NUM> to perform processing using a program. Also, the HDD <NUM> stores, for example, various kinds of data and various kinds of information obtained as a result of the CPU <NUM> performing processing using a program or the like.

The input unit <NUM> is constituted by a keyboard or a pointing device such as a mouse.

The display unit <NUM> is constituted by a monitor such as a liquid crystal display (LCD). The display unit <NUM> may be configured in combination with the input unit <NUM> to function as a GUI (Graphical User Interface).

The communication I/F <NUM> is an interface that controls communication between the information processing apparatus <NUM> and external devices.

The communication I/F <NUM> provides an interface with the network and communicates with external devices via the network. Various kinds of data, various parameters, and so on are transmitted and received to and from external devices via the communication I/F <NUM>. In the present embodiment, the communication I/F <NUM> may perform communication via a wired LAN (Local Area Network) that conforms to a communication standard such as Ethernet (registered trademark), or a dedicated line. However, the network that can be used in the present embodiment is not limited to these networks, and may be constituted by a wireless network. Examples of this wireless network include wireless PANs (Personal Area Networks) such as Bluetooth (registered trademark), ZigBee (registered trademark), and UWB (Ultra Wide Band). Examples of the wireless network also include wireless LAN (Local Area Networks) such as Wi-Fi (Wireless Fidelity) (registered trademark) and wireless MANs (Metropolitan Area Networks) such as WiMAX (registered trademark). Furthermore, the examples include wireless WANs (Wide Area Networks) such as LTE/<NUM>, <NUM>, and <NUM>. The network need only be able to connect devices so as to be able to communicate with each other, and the communication standard, the scale, and the configuration thereof are not limited to the above examples.

At least some of the functions of the constituent elements of the information processing apparatus <NUM> shown in <FIG> can be realized by the CPU <NUM> executing a program. However, at least some of the functions of the constituent elements of the information processing apparatus <NUM> shown in <FIG> may be configured to operate as dedicated hardware. In this case, the dedicated hardware operates under the control of the CPU <NUM>.

The hardware configuration of the user device <NUM> shown in <FIG> may be the same as that shown in <FIG>. That is to say, the user device <NUM> may include a CPU <NUM>, a ROM <NUM>, a RAM <NUM>, an HDD <NUM>, an input unit <NUM>, a display unit <NUM>, a communication I/F <NUM>, and a system bus <NUM>. The user device <NUM> can display various kinds of information provided by the information processing apparatus <NUM> on the display unit <NUM>, and perform processing corresponding to an input operation received from the user via a GUI (a component constituted by the input unit <NUM> and the display unit <NUM>).

The user device <NUM> may be provided with a camera (not shown), and is configured to perform image capturing processing under the control of the CPU <NUM> according to a user operation.

<FIG> shows a flowchart of processing that is performed by the information processing apparatus <NUM> according to the present embodiment. The processing shown in <FIG> can be realized by the CPU <NUM> of the information processing apparatus <NUM> loading a program stored in the ROM <NUM> or the like into the RAM <NUM> and executing the program.

In S71, the acquisition unit <NUM> acquires a product image that serves as a query image. For example, the acquisition unit <NUM> can acquire the product image by acquiring the image or the URL indicating an image included in a search query transmitted from the user device <NUM>.

S72 to S75 are processing steps performed to generate (infer) feature vectors (the first feature vector <NUM>, the second feature vector <NUM>, the gender feature vector <NUM>, and the color feature vector <NUM>) for the product image acquired in S71. The processing steps S72 to S75 may be performed in an order different from the order shown in <FIG>, or performed in parallel.

In S72, the first feature inference unit <NUM> applies the product image acquired by the acquisition unit <NUM> to the first feature predictive model <NUM> to generate a first feature vector <NUM>. As described above, in the present embodiment, the first feature predictive model <NUM> is configured to be able to infer <NUM> different first features (categories), and the first feature vector <NUM> is a vector that can express <NUM> dimensions.

In S73, the second feature inference unit <NUM> applies the product image acquired by the acquisition unit <NUM> to the second feature predictive model <NUM> to generate a second feature vector <NUM>. As described above, in the present embodiment, the second feature predictive model <NUM> is configured to be able to infer <NUM> different second features (genres) for each first feature (category), and the second feature vector <NUM> is a vector that can express <NUM> dimensions. The second feature vector <NUM> may be configured to have a plurality of levels. For example, if the product category inferred by the first feature inference unit <NUM> is women's fashion, the product genre to be inferred by the second feature inference unit <NUM> may be configured to have two levels, i.e., from the upper level to the lower level, of women's fashion_bottoms/pants.

In S74, the gender inference unit <NUM> applies the product image acquired by the acquisition unit <NUM> to the gender predictive model <NUM> to generate a gender feature vector <NUM>. As described above, in the present embodiment, the gender predictive model <NUM> is configured to be able to infer four different genders (male, female, kid, and unisex), and the gender feature vector <NUM> is a vector that can express four dimensions.

In S75, the color extraction unit <NUM> generates a color feature vector <NUM> from the product image acquired by the acquisition unit <NUM>. The processing performed to generate a color feature vector <NUM> is as described above, and the color feature vector <NUM> is a vector that can express six dimensions.

Upon the inference of each feature vector being complete through S72 to S75, processing proceeds to S76. In S76, the concatenation unit <NUM> concatenates the first feature vector <NUM>, the second feature vector <NUM>, the gender feature vector <NUM>, and the color feature vector <NUM> output in S72 to S75, and embeds the concatenated vector into a feature space to generate a compounded feature vector <NUM>.

In S77, the similarity search unit <NUM> receives the compounded feature vector <NUM> generated by the concatenation unit <NUM> as an input, and searches for images (similar images) that are similar to the product image acquired by the acquisition unit <NUM>. The search processing (neighborhood search processing) can be performed using the FAISS (Facebook AI Similarity Search) algorithm. FAISS is a neighborhood search algorithm that employs LSH (Locality Sensitive Hashing).

Before performing the search processing, the similarity search unit <NUM> generates a compounded feature vector <NUM> for each of the plurality of product images that serve as training data. Here, each product image is provided with an image ID (index/identifier) for identifying the image. It is assumed that the similarity search unit <NUM> stores the compounded feature vector <NUM> in the search database <NUM> while associating (mapping) the compounded feature vector <NUM> with the image ID of the product image indicated by the vector. The format of the image ID is not limited to a specific format, and may be information corresponding to a URL.

The similarity search unit <NUM> calculates the similarity (Euclidean distance) in a single (common) feature space between each of the plurality of compounded feature vectors stored in the search database <NUM> and the compounded feature vector <NUM> generated by the concatenation unit <NUM>, and acquires one or more compounded feature vectors similar to the compounded feature vector <NUM>. Such processing corresponds to nearest neighbor search processing. Subsequently, the similarity search unit <NUM> acquires one or more image IDs corresponding to the acquired one or more similar compounded feature vectors, and outputs similar images corresponding to the image IDs.

If a compounded feature vector <NUM> has been once generated by the concatenation unit <NUM> and the compounded feature vector <NUM> is associated by the similarity search unit <NUM> with the image ID, it is possible to search for similar images without performing the processing to generate four feature vectors.

For example, if there is a compounded feature vector corresponding to the image ID of the product image associated with the search query received from the user device <NUM>, the similarity search unit <NUM> can retrieve the corresponding compounded feature vector based on the image ID from the search database <NUM>, and search for similar images based on the corresponding compounded feature vector.

<FIG> is a conceptual diagram for the similar image search processing in S77 described above. As shown in <FIG>, nearest neighbor search processing is performed based on the compounded feature vector <NUM> generated from the product image or the compounded feature vector <NUM> retrieved based on the image ID of the product image. In the nearest neighbor search processing, compounded feature vectors with a high degree of similarity to the compounded feature vector <NUM> are searched for. In the present embodiment, a vector whose Euclidean distance to the compounded feature vector <NUM> in the feature space is short is determined to have a high degree of similarity. Thereafter, images with the image IDs corresponding to the retrieved compounded feature vectors are searched for in an image ID database (included in the search database <NUM>), and the retrieved images are output as similar images.

The similarity search unit <NUM> may sequentially read feature vectors from the beginning of the compounded feature vector <NUM> and perform a similarity search. For example, as shown in <FIG>, if a gender feature vector <NUM>, a second feature vector <NUM>, a color feature vector <NUM>, and a first feature vector <NUM> are concatenated with each other in this order in a compounded feature vector <NUM>, the similarity search unit <NUM> can first read out the gender feature vector <NUM> to perform search processing, and subsequently read the second feature vector <NUM> to perform search processing.

In S78, the output unit <NUM> outputs (distributes) information that includes images (similar images) corresponding to one or more image IDs that are the results of the search performed by the similarity search unit <NUM>, to the user device <NUM>. That is to say, the acquisition unit <NUM> provides the user device <NUM> with information that includes similar images, as a response (search results) to the search query received from the user device <NUM>.

Next, examples of screens displayed on the user device <NUM> according to the present embodiment will be described with reference to <FIG> and <FIG>. <FIG> and <FIG> show examples of screens displayed on the user device <NUM> according to the present embodiment. A screen <NUM> is an example of a screen displayed on the display unit <NUM> of the user device <NUM>. For example, the user operates the user device <NUM> to access a desired electronic commerce site (website such as an EC site) and input and transmit a desired search word to the information processing apparatus <NUM>. As a result, data of the screen <NUM> is provided and displayed on the display unit <NUM> of the user <NUM>.

Upon the user selecting an area <NUM> on the screen <NUM> (examples of selection operations includes a press operation, a touch operation, and so on; the same applies hereinafter), a product image <NUM> in the area <NUM> and a search button <NUM> for the product image <NUM> are displayed. The search button <NUM> is displayed so as to be selectable. At this time, if the user further selects the search button <NUM>, a search query associated with the product image <NUM> serving as a query image is transmitted to the information processing apparatus <NUM>. The image ID attached to the product image <NUM> may also be included in the search query and transmitted.

The information processing apparatus <NUM> upon receiving the search query generates a first feature vector <NUM>, a second feature vector <NUM>, a gender feature vector <NUM>, and a color feature vector <NUM> from the product image <NUM> associated with the search query. Subsequently, the information processing apparatus <NUM> generates a compounded feature vector <NUM> from the four feature vectors, searches for one or more similar images based on the compounded feature vector <NUM>, and outputs the search results (one or more similar images and various kinds of information related to the images) to the user device <NUM>.

<FIG> shows an example of a screen displayed on the display unit <NUM>, on which the search results received by the user device <NUM> from the information processing apparatus <NUM> are displayed. In this example, it is assumed that four similar images 85A to 85D are retrieved based on the product image <NUM>, and the four similar images 85A to 85D are displayed on the screen <NUM>. Although only the images are shown on the screen <NUM>, various kinds of information such as prices and pieces of attribute information related to the images may also be displayed. If the EC site is a mall-type EC site (an EC site like a shopping mall on the Web) and the products included in the product image <NUM> are dealt with by different sellers, product images <NUM> with different prices and different sellers may be retrieved as similar images. Also, a similar image in which the product included in the product image <NUM> is displayed in a different layout may be retrieved.

As described above, the information processing apparatus <NUM> according to the present embodiment accurately extracts the product area from the product image and extracts the colors of the product area to generate a color feature vector that is more suitable for the product. Also, the information processing apparatus <NUM> searches for similar images based on a compounded feature vector in which a plurality of feature vectors including the color feature vector are concatenated with each other. As a result, it is possible to search for similar images from the viewpoint of every feature of the product, provide similar images with higher accuracy than before, and improve usability.

Although the above embodiment describes an example in which a compounded feature vector <NUM> is generated from four feature vectors, the number of feature vectors that are concatenated with each other is not limited to four. For example, a compounded feature vector <NUM> may be generated from the second feature vector <NUM> and the color feature vector <NUM>, and similar images may be searched for based on the compounded feature vector <NUM>. Also, it is possible to employ a configuration with which similar images are searched for based on the compounded feature vector <NUM> in which another feature vector generated through machine learning is concatenated.

In the first embodiment, the user device <NUM> selects one product image on a website such as an EC site, and the information processing apparatus <NUM> searches for similar images similar to the selected product image and provides the similar images to the user device <NUM>.

Meanwhile, if the user device <NUM> is equipped with a camera (an image capturing means), the user may search for products similar to the product included in the product image captured by the camera as well in addition to searching them from the products dealt with on the accessed EC site, and consider purchasing the product. In addition, the user may select a desired image from images already captured by a camera and stored in the storage unit of the user device <NUM>, or images acquired from an external device, and search for products similar to the product included in the selected image to consider purchasing such products.

Therefore, the present embodiment describes an example in which the user searches for similar images based on an image captured with a camera or an image selected from the storage unit of the user device <NUM>. Note that, in the present embodiment, the descriptions of matters common to those in the first embodiment will be omitted.

The configuration of the information processing apparatus <NUM> according to the present embodiment is the same as that in the first embodiment. Also, the flow of processing performed by the information processing apparatus <NUM> according to the present embodiment is the same as that of the processing shown in <FIG> described in the first embodiment. The product image serving as a query image in the first embodiment corresponds to an image captured by the user device <NUM> or an image selected from the storage unit.

Next, examples of screens displayed on the user device <NUM> according to the present embodiment will be described with reference to <FIG> show examples of screens displayed on the user device <NUM> according to the present embodiment. A screen <NUM> in <FIG> is an example of a screen displayed on the display unit <NUM> of the user device <NUM>. For example, the user operates the user device <NUM> to access a desired electronic commerce site (an EC site) and input and transmit a desired search word to the information processing apparatus <NUM>. As a result, information regarding the screen <NUM> is provided and displayed on the display unit <NUM> of the user device <NUM>.

In addition, the CPU <NUM> of the user device <NUM> performs control so that a camera button <NUM> and a photo library button <NUM> are also displayed on the display unit <NUM> of the user device <NUM> in response to a user operation. In the example shown in <FIG>, control is performed so that the camera button <NUM> and the photo library button <NUM> are displayed on the screen <NUM> provided by the information processing apparatus <NUM>. However, the camera button <NUM> and the photo library button <NUM> need only be displayed on a screen that is associated with the EC site accessed by the user. Also, the camera button <NUM> and the photo library button <NUM> may be configured in other forms, such as being constituted by physical buttons.

The camera button <NUM> is a button used to start up a camera function (a camera application) provided in the user device <NUM>. Upon the camera button <NUM> being selected, the user device <NUM> enters a state (an image capturing mode) in which the user device <NUM> can capture an image of a desired subject.

The photo library button <NUM> is a button used to browse one or more images stored in the storage unit of the user device such as the RAM <NUM>. Upon the photo library button <NUM> being selected, one or more images stored in the storage unit are displayed on the display unit <NUM> of the user device <NUM>.

<FIG> shows an example of a screen when the user selects the camera button <NUM> on the screen <NUM> in <FIG> and captures an image that serves as a query image used to search for similar images. In a screen <NUM> shown in <FIG>, an image <NUM> is the captured image. Also, in the screen <NUM>, a search button <NUM> for the image <NUM> is displayed. The search button <NUM> is displayed so as to be selectable. In this state, if the user selects the search button <NUM>, a search query associated with the image <NUM> serving as a query image is transmitted to the information processing apparatus <NUM>.

The information processing apparatus <NUM> upon receiving the search query generates a first feature vector <NUM>, a second feature vector <NUM>, a gender feature vector <NUM>, and a color feature vector <NUM> from the image <NUM> associated with the search query. Subsequently, the information processing apparatus <NUM> generates a compounded feature vector <NUM> from the four feature vectors, searches for one or more similar images based on the compounded feature vector <NUM>, and outputs the search results (one or more similar images and various kinds of information related to the images) to the user device <NUM>.

<FIG> shows an example of a screen when the user selects the photo library button <NUM> on the screen <NUM> in <FIG>. A screen <NUM> shown in <FIG> displays captured images stored in the storage unit of the user device <NUM> or images acquired from an external device. The user can change the one or more images displayed on the screen <NUM> by swiping the screen <NUM> to the right or left, for example. In the screen <NUM>, an image <NUM> displayed in the center serves as a query image. Also, in the screen <NUM>, a search button <NUM> for the image <NUM> is displayed. The search button <NUM> is displayed so as to be selectable.

If the user selects the search button <NUM> in the state shown in the screen <NUM>, a search query associated with the image <NUM> serving as a query image is transmitted to the information processing apparatus <NUM>. In the example in <FIG>, the image displayed in the center of the screen <NUM> is used as the query image. However, the query image may be selected from one or more images stored in the storage unit of the user device <NUM>.

As described above, according to the present embodiment, the query image is selected from among images captured by the user, images already captured, or images acquired from an external device, instead of from images on the website such as an EC site. As a result, the user can more freely select a query image and search for similar images similar to the query image, which contributes to improvement in usability.

In the first embodiment, the user device <NUM> selects one product image on a website such as an EC site, and the information processing apparatus <NUM> searches for similar images similar to the selected product image and provides the similar images to the user device <NUM>. In the second embodiment, the user device <NUM> selects one image from among images captured by the device and images already acquired, and the information processing apparatus <NUM> searches for similar images similar to the selected image and provides the similar images to the user device <NUM>. The present embodiment describes an example in which the first embodiment and the second embodiment are combined.

Note that, in the present embodiment, the descriptions of matters common to those in the first embodiment and the second embodiment will be omitted.

The configuration of the information processing apparatus <NUM> according to the present embodiment is the same as that in the first embodiment. Also, the flow of processing performed by the information processing apparatus <NUM> according to the present embodiment is the same as that of the processing shown in <FIG> described in the first embodiment.

However, the processing performed by the similarity search unit <NUM> is different from that in the above-described embodiments. The user device <NUM> transmits a search query in which a product image that serves as a query image and an image that contains text information (text image) selected from the product image are associated with each other, and the similarity search unit <NUM> of the information processing apparatus <NUM> searches for similar images, using the product image and the text image.

Examples of screens displayed on the user device <NUM> according to the present embodiment will be described with reference to <FIG> and <FIG>. <FIG> and <FIG> show examples of screens displayed on the user device <NUM> according to the present embodiment. A screen <NUM> in <FIG> is an example of a screen displayed on the display unit <NUM> of the user device <NUM>. For example, the user operates the user device <NUM> to access a desired electronic commerce site (an EC site) and input and transmit a desired search word to the information processing apparatus <NUM>. As a result, information regarding the screen <NUM> is provided and displayed on the display unit <NUM> of the user <NUM>.

In addition, the CPU <NUM> of the user device <NUM> performs control so that a camera button <NUM> and a photo library button <NUM> are also displayed on the display unit <NUM> of the user device <NUM> in response to a user operation. The function of the camera button <NUM> is the same as the camera button <NUM> in <FIG>.

It is assumed that a product image <NUM> is displayed on the screen <NUM> in <FIG> in response to the user's search operation. Here, it is assumed that the user selects the camera button <NUM> to start the image capturing mode and captures an image of an area <NUM>. An image <NUM> displayed on the display unit <NUM> after the image capturing is an image corresponding to the area <NUM>, and is an image that contains text information (a text image). Note that the image <NUM> is not limited to an image obtained through an image capturing operation, and may be an image obtained through any selection operation performed in response to a user operation. Also, with the image <NUM>, a search button <NUM> for the product image <NUM> (or the area <NUM>) is displayed. The search button <NUM> is displayed so as to be selectable.

In this state, if the user selects the search button <NUM>, a search query associated with the product image <NUM> and the image (text image) <NUM> is transmitted to the information processing apparatus <NUM>.

The information processing apparatus <NUM> upon receiving the search query generates a first feature vector <NUM>, a second feature vector <NUM>, a gender feature vector <NUM>, and a color feature vector <NUM> from the image <NUM> associated with the search query. Subsequently, the information processing apparatus <NUM> generates a compounded feature vector <NUM> from the four feature vectors.

If a compounded feature vector <NUM> has already been generated from the image <NUM>, the similarity search unit <NUM> searches for and acquires the compounded feature vector <NUM> based on the image ID.

Next, the similarity search unit <NUM> analyzes the image <NUM> associated with the search query to extract text information. Various known image processing techniques and machine learning can be used to extract the text information. In the present embodiment, the similarity search unit <NUM> is configured to extract text information (for example, at least one of the product name and the brand name) from the image <NUM>, using machine learning. In the case of the image <NUM>, the product name to be extracted is "Mineral Sunscreen" and the brand name to be extracted is "ABC WHITE".

The similarity search unit <NUM> searches for one or more similar images similar to the image <NUM> based on the compounded feature vector <NUM> and the extracted text information, and outputs search results (one or more similar images and various kinds of information related to the image) to the user device <NUM>.

<FIG> shows an example of a screen displayed on the display unit <NUM>, on which the search results received by the user device <NUM> from the information processing apparatus <NUM> are displayed. In this example, it is assumed that two similar images 1008A and 1008B are retrieved based on the image <NUM>, and the two similar images 1008A and 1008B are displayed on a screen <NUM>. Although the screen <NUM> shows only images, various kinds of information such as prices and pieces of attribute information related to the images may also be displayed.

As described above, the information processing apparatus <NUM> according to the present embodiment infers (predicts) a plurality of attributes (features) of a product based on a product image to generate a plurality of feature vectors, and generates a compounded feature vector in which the plurality of feature vectors are concatenated with each other. Furthermore, the information processing apparatus <NUM> extracts text information from the text image in the product image. Thereafter, the information processing apparatus <NUM> searches for similar images based on the compounded feature vector and the text information. As a result, it is possible to provide similar images with higher accuracy than before, and improve usability.

The present embodiment describes the acquisition unit <NUM> as being configured to acquire one product image. However, if a plurality of images are associated with the search query, or a plurality of search queries are received at a time, the information processing apparatus <NUM> may perform a similar image search for each of the images.

Claim 1:
An information processing apparatus (<NUM>) comprising:
an acquisition means (<NUM>) for acquiring an object image that contains one or more objects;
a first inference means (<NUM>) for inferring one or more rectangular areas that respectively surround the one or more objects in the object image, and a type of each of the one or more objects, by applying the object image to a first learning model;
a second inference means (<NUM>) for inferring an area of a target object in the object image by applying the one or more rectangular areas and the type of each of the one or more objects to a second learning model; and
an extraction means (<NUM>) for extracting colors of the target object,
wherein the object image is image data generated from a Y element, a Cb element, and a Cr element of data generated from a YCbCr image through a DCT conversion; and
wherein the image data is data generated by adjusting sizes of Y elements, Cb elements, and Cr elements of pieces of data subjected to DCT conversion, and concatenating the pieces of data with each other.