Patent Description:
<CIT> discloses a video recording apparatus which includes: an object classifier configured to classify an object which is included in an input video and related to a detected event, according to a type of the object; an object color extractor configured to extract a representative color of the object; and a search color storage configured to transform the representative color into a search color and store the search color as color information to be used for searching the object or the detected event from the input video.

"Tag Tagging: Towards More Descriptive Keywords of Image Content" relates to an automatic scheme called tag tagging to supplement semantic image descriptions by associating a group of property tags with each existing tag. In the proposed scheme, a lazy learning approach is first applied to estimate the corresponding image regions of each initial tag, and then a set of property tags are derived for each initial tag.

Particular embodiments are the subject of the dependent claims.

Target object color analysis and tagging apparatuses, methods for target object color analysis and tagging, and non-transitory computer readable media having stored thereon machine readable instructions to provide target object color analysis and tagging are disclosed herein. The apparatuses, methods, and non-transitory computer readable media disclosed herein provide for color tagging of a target object by assigning a color tag of a plurality of color tags associated with the target object. The target object may include an element such as a person, an animal, clothing worn by a person, furniture near a person, etc., and/or a region of an image, such as the sky, a forest, a road, etc. Color tags may be described as specified colors (e.g., beige, black, blue, red, etc.) that are to be analyzed for assignment to the target object.

In the case of images, videos, and other such sources that include objects, color may be extracted. However, it is technically challenging to extract color from a target object, where it may be uncertain as to what the target object is, and where the target object is located. For example, it is technically challenging to extract color from a target object, where it may be uncertain as to what the target is from a plurality of other objects. It is also technically challenging to determine a color of an object, where the color of the object may need to be matched to a limited number of available colors.

In order to address at least these technical challenges with respect to extraction of color from a target object and determination of a color of an object, the target object color analysis and tagging as disclosed herein provide for assignment of a color tag from a plurality of available color tags to an identified target object by ascertaining, for an image (or a video, or generally another source) that is to be analyzed, an attribute of the image. Examples of an attribute of the image may include text associated with the image, sound associated with the image, a shape (e.g., a face) included in the image, etc. Based on the attribute of the image, a target object that is to be identified and color tagged in the image may be determined.

A plurality of objects is extracted from the image based on a learning model. For example, assuming that the target object is a shirt worn by a person, the plurality of objects may include furniture adjacent to the person, other clothes worn by the person, shoes worn by the person, etc..

The target object in the image is identified based on a comparison of the target object that is to be identified and color tagged in the image and the plurality of extracted objects from the image. Assuming that the attribute of the image includes text associated with the image, the target object may be determined by determining, based on an analysis of a repository of available objects based on the text associated with the image, categories of related objects. For example, assuming that the target object is a shirt worn by a person, the categories may include sweaters, t-shirts, jackets, shirts, and other such clothing worn by a person. The categories may further include sub-categories. For example, a sweaters category may include sub-categories that include sleeveless sweaters, sweaters with sleeves, etc..

Color information is extracted from the identified target object, and a plurality of color tags associated with the identified target object is ascertained. For example, assuming that the target object is a shirt worn by a person, the color tags may include colors such as beige, blue, green, red, etc..

A plurality of color distances is determined between the color information and the plurality of color tags. For example, the color distances may represent CIEDE2000 color distances between the color information and the plurality of color tags.

Based on a determination of a minimum color distance from the plurality of color distances, a color tag of the plurality of color tags that is to be assigned to the identified target object is determined. Further, the color tag may be assigned to the identified target object.

For the apparatuses, methods, and non-transitory computer readable media disclosed herein, the elements of the apparatuses, methods, and non-transitory computer readable media disclosed herein may be any combination of hardware and programming to implement the functionalities of the respective elements. In some examples described herein, the combinations of hardware and programming may be implemented in a number of different ways. For example, the programming for the elements may be processor executable instructions stored on a non-transitory machine-readable storage medium and the hardware for the elements may include a processing resource to execute those instructions. In these examples, a computing device implementing such elements may include the machine-readable storage medium storing the instructions and the processing resource to execute the instructions, or the machine-readable storage medium may be separately stored and accessible by the computing device and the processing resource. In some examples, some elements may be implemented in circuitry.

<FIG> illustrates a layout of an example target object color analysis and tagging apparatus (hereinafter also referred to as "apparatus <NUM>").

Referring to <FIG>, the apparatus <NUM> includes an image analyzer <NUM>, which is executed by at least one hardware processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>), to ascertain, for an image <NUM> that is to be analyzed, an attribute <NUM> of the image <NUM>. Further, the image analyzer <NUM> is to determine, based on the attribute <NUM> of the image <NUM>, a target object <NUM> that is to be identified and color tagged in the image <NUM>. According to an example, the attribute <NUM> of the image <NUM> may include audible and/or visible attributes associated with the image <NUM>. According to an example, the target object <NUM> may include an element and/or a region of the image <NUM>.

An object identifier <NUM>, which is executed by the at least one hardware processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>), is to extract, based on a learning model, a plurality of objects <NUM> from the image <NUM>. Further, the object identifier <NUM> is to identify, based on a comparison of the target object <NUM> that is to be identified and color tagged in the image <NUM> and the plurality of extracted objects <NUM> from the image, the target object <NUM> in the image <NUM>.

A color analyzer <NUM>, which is executed by the at least one hardware processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>), is to extract color information <NUM> from the identified target object <NUM>. The color analyzer <NUM> is to ascertain a plurality of color tags <NUM> associated with the identified target object <NUM>. Further, the color analyzer <NUM> is to determine a plurality of color distances <NUM> between the color information <NUM> and the plurality of color tags <NUM>.

A color tagger <NUM>, which is executed by the at least one hardware processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>), is to determine, based on a determination of a minimum color distance from the plurality of color distances <NUM>, a color tag <NUM> of the plurality of color tags <NUM> that is to be assigned to the identified target object <NUM>.

A background analyzer <NUM>, which is executed by the at least one hardware processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>), is to determine whether background color (i.e., color of the image other than the color of objects in the image, or color of the image outside of the boundaries of the target object <NUM>) should be removed from the image <NUM>. In this regard, the analysis by the background analyzer <NUM> may be performed prior to extraction of the color information <NUM> from the identified target object <NUM>. With respect to determining whether background color should be removed from the image <NUM>, the background analyzer <NUM> may determine a histogram of a plurality of color clusters, where the color clusters are determined from the entire image <NUM> (e.g., the extracted color information <NUM> from the identified target object <NUM> and color information from a remaining portion of the image <NUM>). The plurality of color clusters may be determined, for example, by using k-means clustering. The background analyzer <NUM> may sort histogram bins associated with the determined histogram of the plurality of color clusters in descending order. The background analyzer <NUM> may determine a difference between a highest order bin and a subsequent bin of the sorted histogram bins. The difference may represent a difference between background color and a color of an object. The highest order bin may represent a bin which includes a highest count of color occurrences for a particular color in the image <NUM>, the subsequent bin may represent a bin which includes the second highest count of color occurrences for the particular color in the image <NUM>, and so forth. The background analyzer <NUM> may determine whether the difference is greater than a specified threshold. For example, the specified threshold may be <NUM>%. In this regard, in response to a determination that the difference is greater than the specified threshold, the background analyzer <NUM> may remove background color from the image <NUM>. Otherwise, in response to a determination that the difference is less than the specified threshold, the background analyzer <NUM> may not remove background color from the image <NUM>. In this manner, background color interference with respect to extraction of the color information <NUM> from the identified target object <NUM> may be minimized.

The attribute <NUM> of the image <NUM> includes text within the image <NUM>. In this regard, the image analyzer <NUM> is executed by the at least one hardware processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) to determine, based on the text associated with the image, the target object <NUM> that is to be identified and color tagged in the image <NUM> by determining, based on an analysis of a repository <NUM> of available objects <NUM> based on the text associated with the image, categories of related objects. According to an example, the categories of related objects may include the target object <NUM> that is to be identified and color tagged in the image <NUM>. Further, the object identifier <NUM> is executed by the at least one hardware processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) to identify, based on the comparison of the categories of related objects and the plurality of extracted objects <NUM> from the image, the target object <NUM> in the image <NUM>. The tfw categories of related objects may be determined as a function of synonyms determined from the text associated with the image <NUM>.

According to an example, the color analyzer <NUM> is executed by the at least one hardware processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) to extract color information <NUM> from the identified target object <NUM> by applying k-means clustering.

According to an example, the color analyzer <NUM> is executed by the at least one hardware processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) to determine the plurality of color distances <NUM> between the color information <NUM> and the plurality of color tags <NUM> by determining values of L*C*h for each of the plurality of color tags <NUM>. In this regard, L* may represent lightness, C* may represent chroma, and h may represent a hue angle. The color analyzer <NUM> may also determine values of L*C*h for the extracted color information <NUM>. Further, the color analyzer <NUM> may determine, based on the L*C*h values for each of the plurality of color tags <NUM> and the L*C*h values for the extracted color information <NUM>, the plurality of color distances <NUM> between the color information <NUM> and the plurality of color tags <NUM>.

According to an example, the color analyzer <NUM> is executed by the at least one hardware processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) to determine the plurality of color distances <NUM> between the color information <NUM> and the plurality of color tags <NUM> by determining CIEDE2000 color distances <NUM> between the color information <NUM> and the plurality of color tags <NUM>.

<FIG> illustrates an example of color tagging to illustrate operation of the apparatus <NUM> in accordance with an example of the present disclosure.

Referring to <FIG> and <FIG>, as disclosed herein, the image analyzer <NUM> ascertains, for the image <NUM> that is to be analyzed, the attribute <NUM> of the image <NUM>. For example, as shown in <FIG>, the image <NUM> may include the image of a person wearing a shirt. The attribute <NUM> of the image <NUM> may include text such as "men's sky blue shirt".

The image analyzer <NUM> determines, based on the attribute <NUM> of the image <NUM>, the target object <NUM> that is to be identified and color tagged in the image <NUM>. The image analyzer <NUM> determines, based on an analysis of the repository <NUM> of available objects <NUM> based on the text associated with the image, categories of related objects. For the example of <FIG> that includes the image <NUM> of a person wearing a shirt, the repository may include categories of related objects that include sweaters, t-shirts, shirts, and other such clothing worn by a person. The categories may also include sub-categories. For example, a sweaters category may include sub-categories that include sleeveless sweaters, sweaters with sleeves, etc. The image analyzer <NUM> may match the attribute <NUM> (i.e., each component or word of the attribute <NUM>) to the objects in the repository <NUM>, and use data mining techniques such as bag of the words and N-gram to extract features of the attribute <NUM>. In this regard, the extracted features may represent synonyms of the attribute <NUM>, where the synonyms are used to identify the categories present in the repository <NUM>. For the example of <FIG> where the target object includes a shirt, synonyms of the attribute <NUM> may include sweaters, t-shirts, jackets, shirts, and other such clothing worn by a person. Further, for the example of <FIG>, synonyms of "men's" "sky" and "blue" may also be determined to determine associated categories, but these categories may be eliminated by the object identifier <NUM> as they do not match to any of the objects extracted from the image <NUM> as described below.

As disclosed herein, the object identifier <NUM> extracts, based on the learning model, the plurality of objects <NUM> from the image <NUM>. For example, the learning model may include a convolutional neural network (CNN) based deep learning model. The CNN may be trained, for example, by the object identifier <NUM> on product images to provide information in the images. For the example of <FIG>, the CNN model may provide information such as shirt, pants, etc..

As disclosed herein, the object identifier <NUM> identifies, based on a comparison of the target object <NUM> that is to be identified and color tagged in the image <NUM> and the plurality of extracted objects <NUM> from the image, the target object <NUM> in the image <NUM>. For the example of <FIG>, the object identifier <NUM> may implement a lexical database of the English language to refine the aforementioned extracted categories to match with object information extracted by using the learning model. For the example of <FIG>, the object identifier <NUM> may compare the categories of sweaters, t-shirts, and shirts to the extracted objects that include shirt and pants to identify the target object <NUM> as shirt. Further, the object identifier <NUM> may identify aspects such as the location of the target object <NUM> so that this location information may be used to extract color information from the target object <NUM>.

As disclosed herein, the color analyzer <NUM> extracts color information <NUM> from the identified target object <NUM>. For the example of <FIG>, the color analyzer <NUM> may extract color information <NUM> from the identified target object <NUM> that includes a shirt.

As disclosed herein, the color analyzer <NUM> ascertains a plurality of color tags <NUM> associated with the identified target object <NUM>. For the example of <FIG>, the color analyzer <NUM> may ascertain the plurality of color tags <NUM> that include blue, beige, black, red, etc..

Further, as disclosed herein, the color analyzer <NUM> determines a plurality of color distances <NUM> between the color information <NUM> and the plurality of color tags <NUM>. In this regard, for the example of <FIG>, assuming that the color of the shirt is sky blue, and assuming that none of the color tags <NUM> include sky blue, the color distances <NUM> between the color information <NUM> and the plurality of color tags <NUM> may be used to identify the closest color tag to the color information <NUM>. In this regard, the color tagger <NUM> determines, based on a determination of a minimum color distance from the plurality of color distances <NUM>, a color tag <NUM> (e.g., the color tag blue) of the plurality of color tags <NUM> that is to be assigned to the identified target object <NUM>. Thus, for the example of <FIG>, the color tagger <NUM> may determine that the color tag blue is the closest color tag to the color information <NUM> for the shirt.

<FIG> respectively illustrate an example block diagram <NUM>, a flowchart of an example method <NUM>, and a further example block diagram <NUM> for target object color analysis and tagging, according to examples. The block diagram <NUM>, the method <NUM>, and the block diagram <NUM> may be implemented on the apparatus <NUM> described above with reference to <FIG> by way of example and not of limitation. The block diagram <NUM>, the method <NUM>, and the block diagram <NUM> may be practiced in other apparatus. In addition to showing the block diagram <NUM>, <FIG> shows hardware of the apparatus <NUM> that may execute the instructions of the block diagram <NUM>. The hardware may include a processor <NUM>, and a memory <NUM> storing machine readable instructions that when executed by the processor cause the processor to perform the instructions of the block diagram <NUM>. The memory <NUM> may represent a non-transitory computer readable medium. <FIG> may represent an example method for target object color analysis and tagging, and the steps of the method. <FIG> may represent a non-transitory computer readable medium <NUM> having stored thereon machine readable instructions to provide target object color analysis and tagging according to an example. The machine readable instructions, when executed, cause a processor <NUM> to perform the instructions of the block diagram <NUM> also shown in <FIG>.

The processor <NUM> of <FIG> and/or the processor <NUM> of <FIG> may include a single or multiple processors or other hardware processing circuit, to execute the methods, functions and other processes described herein. These methods, functions and other processes may be embodied as machine readable instructions stored on a computer readable medium, which may be non-transitory (e.g., the non-transitory computer readable medium <NUM> of <FIG>), such as hardware storage devices (e.g., RAM (random access memory), ROM (read only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), hard drives, and flash memory). The memory <NUM> may include a RAM, where the machine readable instructions and data for a processor may reside during runtime.

Referring to <FIG>, and particularly to the block diagram <NUM> shown in <FIG>, the memory <NUM> may include instructions <NUM> to ascertain, for the image <NUM> that is to be analyzed, the attribute <NUM> of the image <NUM>.

The processor <NUM> may fetch, decode, and execute the instructions <NUM> to determine, based on the attribute <NUM> of the image, the target object <NUM> that is to be identified and color tagged in the image <NUM>.

The processor <NUM> may fetch, decode, and execute the instructions <NUM> to extract, based on a learning model, the plurality of objects <NUM> from the image <NUM>.

The processor <NUM> may fetch, decode, and execute the instructions <NUM> to identify, based on a comparison of the target object <NUM> that is to be identified and color tagged in the image <NUM> and the plurality of extracted objects <NUM> from the image, the target object <NUM> in the image <NUM>.

The processor <NUM> may fetch, decode, and execute the instructions <NUM> to extract color information <NUM> from the identified target object <NUM>.

The processor <NUM> may fetch, decode, and execute the instructions <NUM> to ascertain the plurality of color tags <NUM> associated with the identified target object <NUM>.

The processor <NUM> may fetch, decode, and execute the instructions <NUM> to determine the plurality of color distances <NUM> between the color information <NUM> and the plurality of color tags <NUM>.

The processor <NUM> may fetch, decode, and execute the instructions <NUM> to determine, based on a determination of a minimum color distance from the plurality of color distances <NUM>, the color tag <NUM> of the plurality of color tags <NUM> that is to be assigned to the identified target object <NUM>.

Referring to <FIG> and <FIG>, and particularly <FIG>, for the method <NUM>, at block <NUM>, the method includes ascertaining (e.g., by the image analyzer <NUM>), for the image <NUM> or a video that is to be analyzed, the attribute <NUM> of the image <NUM> or the video.

At block <NUM>, the method includes determining (e.g., by the image analyzer <NUM>), based on the attribute <NUM> of the image <NUM> or the video, the target object <NUM> that is to be identified and color tagged in the image <NUM> or the video.

At block <NUM>, the method includes extracting (e.g., by the object identifier <NUM>), based on a learning model, the plurality of objects <NUM> from the image <NUM> or the video.

At block <NUM>, the method includes identifying (e.g., by the object identifier <NUM>), based on a comparison of the target object <NUM> that is to be identified and color tagged in the image <NUM> or the video and the plurality of extracted objects <NUM> from the image <NUM> or the video, the target object <NUM> in the image <NUM> or the video.

At block <NUM>, the method includes extracting (e.g., by the color analyzer <NUM>), color information <NUM> from the identified target object <NUM>.

At block <NUM>, the method includes ascertaining (e.g., by the color analyzer <NUM>), a plurality of color tags <NUM> associated with the identified target object <NUM>.

At block <NUM>, the method includes determining (e.g., by the color analyzer <NUM>), the plurality of color distances <NUM> between the color information <NUM> and the plurality of color tags <NUM>.

At block <NUM>, the method includes determining (e.g., by the color tagger <NUM>), based on a determination of a minimum color distance from the plurality of color distances <NUM>, the color tag <NUM> of the plurality of color tags <NUM> that is to be assigned to the identified target object <NUM>.

Referring to <FIG> and <FIG>, and particularly <FIG>, for the block diagram <NUM>, the non-transitory computer readable medium <NUM> includes instructions <NUM> to ascertain (e.g., by the image analyzer <NUM>), for the image <NUM> that is to be analyzed, the attribute <NUM> of the image <NUM>.

The processor <NUM> fetches, decodes, and executes the instructions <NUM> to determine (e.g., by the image analyzer <NUM>), based on the attribute <NUM> of the image, the target object <NUM> that is to be identified and color tagged in the image <NUM>.

The processor <NUM> fetches, decodes, and executes the instructions <NUM> to extract (e.g., by the object identifier <NUM>), based on a learning model, the plurality of objects <NUM> from the image <NUM>.

The processor <NUM> fetches, decodes, and executes the instructions <NUM> to identify (e.g., by the object identifier <NUM>), based on a comparison of the target object <NUM> that is to be identified and color tagged in the image <NUM> and the plurality of extracted objects <NUM> from the image, the target object <NUM> in the image <NUM>.

The processor <NUM> fetches, decodes, and executes the instructions <NUM> to extract (e.g., by the color analyzer <NUM>), color information <NUM> from the identified target object <NUM>.

The processor <NUM> fetches, decodes, and executes the instructions <NUM> to ascertain (e.g., by the color analyzer <NUM>), the plurality of available color tags <NUM>.

The processor <NUM> fetches, decodes, and executes the instructions <NUM> to determine (e.g., by the color analyzer <NUM>), the plurality of color distances <NUM> between the color information <NUM> and the plurality of available color tags <NUM>.

The processor <NUM> fetches, decodes, and executes the instructions <NUM> to determine (e.g., by the color tagger <NUM>), based on a determination of a minimum color distance from the plurality of color distances <NUM>, the color tag <NUM> of the plurality of available color tags <NUM> that represents a closest color tag to the identified target object <NUM>.

Claim 1:
A target object color analysis and tagging apparatus (<NUM>) comprising:
an image analyzer (<NUM>), executed by at least one hardware processor (<NUM>; <NUM>), to
ascertain, for an image (<NUM>) or video that is to be analyzed, an attribute (<NUM>) of the image or video, wherein the attribute of the image or video includes text within the image or video, and
determine, based on the attribute of the image or video, categories of related objects by:
determining, based on an analysis of a repository (<NUM>) of objects based on the text within the image or video, categories of related objects, wherein the categories of related objects are determined as a function of synonyms determined from the text within the image or video, wherein the categories of related objects include a target object that is to be identified;
an object identifier (<NUM>), executed by the at least one hardware processor, to
extract, based on a learning model, a plurality of objects (<NUM>) from the image or video, and
identify, based on a comparison of the categories of related objects and the plurality of extracted objects from the image or video, the target object in the image or video, based on matching a category and an extracted object;
a color analyzer (<NUM>), executed by the at least one hardware processor, to
extract color information (<NUM>) from the identified target object,
ascertain a plurality of color tags (<NUM>) associated with the identified target object, and
determine a plurality of color distances (<NUM>) between the color information and the plurality of color tags; and
a color tagger (<NUM>), executed by the at least one hardware processor, to
determine, based on a determination of a minimum color distance from the plurality of color distances, a color tag (<NUM>) of the plurality of color tags that is to be assigned to the identified target object.