Device and method for detecting specified objects in images using metadata

The purpose of the present invention is to provide an image analysis technique enabling a detection subject to be rapidly detected from image data. This image analysis device generates metadata for a query image containing the detection subject, and using the metadata, narrows down the image data serving as the search subject beforehand and then conducts object detection.

TECHNICAL FIELD

The present invention relates to a technique for detecting specific objects included in image data.

BACKGROUND ART

Along with development in IT infrastructures for persons/in companies, a huge amount of multimedia data (such as documents, videos/images, voices, or various log data) has been stored in large storages. In order to extract information efficiently from vast amount of stored data, various information search techniques for individual media data have been invented and put into practical use.

As an example of information search with respect to multimedia data, a method may be assumed for detecting objects or specific regions included in images. Object detection or region identification in images correspond to morphological analysis in document analysis (means for separating documents into words to determine word classes), which are important in analyzing meanings of images.

As a method for detecting objects in images, the method of Non Patent Literature 1 is commonly known, and is commercialized as face region detecting function in digital cameras or in monitoring systems. In the method of Non Patent Literature 1, vast amount of image samples of the detection target is collected, and multiple of discriminators on the basis of image brightness are generated by machine learning. These discriminators are combined to generate a determinator for partial regions of the image. The object region is identified by thoroughly searching the partial regions in the image.

The detection targets are usually human faces currently. However, if wide ranges of contents stored in storages are the detection targets, it is desired to detect various objects such as cars, animals, buildings, diagrams, or various goods. In addition, in order to process huge size data, it is required to improve efficiency of analysis process.

Regarding improvement in efficiency of analysis process, Patent Literature 1 listed below describes a method for utilizing existence probability of objects, thereby limiting the region to which image processing for detecting object regions is performed. The method of Patent Literature 1 utilizes static information of imaging system such as focal point distance or resolution, thereby determining regions to which image processing is performed. It may be advantageous in environments where imaging environments or imaging devices are limited such as in-vehicle cameras and where structured data is managed.

CITATION LIST

Patent Literature

Patent Literature 1: JP Patent Publication (Kokai) 2010-003254 A

Non Patent Literature

SUMMARY OF INVENTION

Technical Problem

The technique described in Patent Literature 1 assumes that the imaging environment is specified to some degree and target data for image processing is structured. However, the imaging environment or the photographic subject is not always predictable in advance generally. In addition, in environments where the target data for image processing is generated in ad hoc manner, such data is not structured. In such environments, the method described in Patent Literature 1 may not be advantageous for reducing the time to detect objects.

The technique described in Non Patent Literature 1 is effective if the detected target is predefined such as in face detection. However, in such applications where users sequentially specify different detection targets, it is individually necessary to collect samples or to perform machine learning, thus not practical in terms of processing time.

The present invention is made in the light of the above-described technical problems. It is an objective of the present invention to provide image analysis techniques that can rapidly detect the targets within image data.

Solution to Problem

An image analysis device according to the present invention: generates metadata of query images including the detection target; narrows the image data for search targets using the metadata in advance; and performs object detection.

Advantageous Effects of Invention

With the image analysis device according to the present invention, it is possible to rapidly extract images including any object from vast amount of image data.

Technical problems, configurations, and effects other than mentioned above will be understood with reference to the following embodiments.

DESCRIPTION OF EMBODIMENTS

Embodiment 1: System Configuration

FIG. 1is a configuration diagram of an image analysis system100according to an embodiment 1 of the present invention. The image analysis system100is a system which objective is to search images including any object specified by a user among vast amount of images. The image analysis system100includes an image/document storage device101, an input device102, a display device103, a data storage device104, an image database105, and an image analysis device106.

The image/document storage device101is a storage medium storing image data. The image/document storage device101may be configured using storage systems connected to networks, such as NAS (Network Attached Storage) or SAN (Storage Area Network). It is assumed that the size of image data analyzed by the image analysis system100is, for example, several hundreds thousand pieces of image data.

The input device102is an input interface for transferring user operations to the image analysis device106such as mouse, keyboard, or touch device. The display device103is an output interface such as liquid crystal display. The display device103is used for such as displaying image analysis results of the image analysis device106, or for interactive operations with users. The data storage device104is a storage storing analysis results of the image analysis device106. The data storage device104is used for upper applications to utilize the analysis results.

The image database105is a database management system for storing images. The image database105not only temporarily stores analyzed data but also is utilized in the analysis process itself as a dictionary for generating metadata. Details will be described later usingFIG. 2.

The image analysis device106is a device that detects, among image data stored in the image database105, objects included in the query image specified by the user. The image analysis device106includes an image/document input unit107, a metadata generator108, an analysis target determinator109, on object region detector110, an operational information input unit111, and a data output unit112.

The image/document input unit107reads out, from the image/document storage device101, image data to be stored in the image database105and its related bibliographic information. The image/document input unit107associates these information with each other and stores them into the image database105. The image document input unit107also reads out the query image including the detection target object, and passes the query image to the metadata generator108and to the object region detector110.

The metadata generator108automatically generates metadata of the query image by image recognition process using the image database105as a dictionary. The metadata mentioned here is highly abstractive data included in image data. For example, metadata is information such as words describing the image, creation date, or creation location. Hereinafter, for the sake of simplicity, it is assumed that “metadata=word”. However, the metadata generator108may generate various metadata. A reliability value is assigned to metadata. The generated metadata is sent to the analysis target determinator109. The sequence for generating metadata will be described later usingFIG. 5.

The analysis target determinator109searches the bibliographic information stored in the image database105using the metadata generated by the metadata generator108as a search key, thereby acquiring a list of image data having the bibliographic information matching with the search key. The metadata used as the search key may be automatically selected according to its reliability, or may be selected by the user from candidates of metadata. If the user selects the metadata used as the search key, a list of metadata candidates or the hit number of search result are presented to the user through the data output unit112in order to perform an interactive operation between the user and the image analysis device106. In addition, search parameters such as the metadata specified as the search key or thresholds are received from the operational information input unit111. The image list acquired by above is sent to the object region detector110as candidates for analysis target.

The object region detector110identifies, using an image analysis process, coordinates of the region from the image where the specified object is present. The detection target is not fixed and may be specified by the user each time. In addition, objects with various concepts (e.g. human face, car, cat, star mark, etc.) may be the detection target simultaneously. The analysis result is sent to the data output unit112as a coordinate of the rectangle region of the object (e.g. [horizontal coordinate of left-top of the rectangle, vertical coordinate of left-top of the rectangle, horizontal coordinate of right-bottom of the rectangle, vertical coordinate of right-bottom of the rectangle]) and a reliability value indicating “likelihood of object”. At this time, the metadata generated by the metadata generator108may be outputted in association with the analysis result as meaning information of the detected object.

The operational information input unit111receives a user operation from the input device102, and sends its signal to the image analysis device106. The data output device112receives the image list for image analysis or image analysis result, and outputs them to the display device103and to the data storage device104.

FIG. 2is a diagram showing a configuration and a data example of the image database105. A configuration example with table format is shown here. However, any data format is allowed. The image database105is a database that stores image features and bibliographic information associated with each other. The image database105includes an image ID field1051, an image data field1052, an image feature field1053, and a bibliographic information field1054.

The image ID field1051stores identifiers of each image data. The image data field1052is a field that stores image data in binary format, and is used when the user checks the analysis result. The image feature field1053stores numerical vector data with fixed length by which features of the image itself such as color or shape are quantified. The bibliographic information field1054stores bibliographic information (such as sentence, category, date and time, location) associated with the image. The bibliographic information field may be separated into multiple fields if necessary.

Embodiment 1: Operations of Each Units

Overall configurations of the image analysis system100have been described so far. Hereinafter, the operational principle of the image analysis system100will be summarized, and then detailed operations of each functional units will be described.

The image analysis system100searches, from the database105using image recognition process, image data in which the object included in the query image specified by the user is included. In a simplified manner, an object detection process may be performed with respect to all images in the image database105. However, the processing speed of object detection is usually slow. Thus it is not practical to perform object detection processes with respect to all of vast amount of all image sets.

For example, it is assumed that 0.5 second of image recognition process is required for each image. Then 140 hours is necessary to analyze one million images. If the detected object is limited to “front face of human”, an analyze process is performed only once when building the database and the analysis result may be reused for subsequent analysis processes to reduce processing time. However, if the detected object is not fixed and any specified object is to be detected, it is necessary to perform the analysis process after the user specifies the object to be detected. Thus the response time may be problematic.

Thus the image analysis system100automatically generate the metadata of the detection target object, and attempts to reduce the processing time by narrowing, using the metadata, the image data to which the object detection process is performed.

FIG. 3is a diagram showing a dataflow explaining a sequence to generate metadata of the query image specified by the user and to narrow the object detection target using the metadata. As shown inFIG. 3, it is assumed that image data and bibliographic information are already stored in the image database105.

The query image301is the query image inputted by the user through the image/document input device107. It is assumed here that only one object (star shape) is present in the query image301.

The metadata generator108generates metadata302of the query image301(S301). The metadata302is outputted in a list with scores (=reliabilities of metadata). Details of generation of the metadata302will be described later usingFIG. 5.

The analysis target determinator109searches, using the metadata generated by the metadata generator108as a search key, the bibliographic information matching with the search key from the image database105, thereby acquiring a group303of image data matching with the search condition (S302). An example is shown where an OR search of three words “star”, “pentagram”, “astral body” is performed. However, AND search may be combined if necessary. In the image having metadata similar to the search target image, it is highly likely that there is the object therewithin.

The object region detector110identifies the region of each image in the image group303where objects similar to the object included in the query image301are present (S303). The processing time of this process is increased according to the number of images included in the image group303. Details of object detection will be described later usingFIG. 7.

The detection result304is described, for each of images in the image group303, along with the number of detected objects, with the location of the object (dotted rectangle in the detection result304), and with the reliability of “likelihood of object” (percentage in the detection result304), for example. As the meaning information of each detected objects, the metadata generated in step S301may be associated with the detection result304. The data output unit112displays the detection result304on the display device103, or outputs the detection result304as data into the data storage device104.

As shown inFIG. 3, the image analysis system100performs object detection after limiting the images that are highly likely to include the detection target object using the metadata of the query image301. Thus it is possible to reduce the processing time.

On the other hand, the “appearance” of the image does not always match with the bibliographic information in terms of meaning. In the example of image305, the bibliographic information matches with the search key but the “appearance” of the object is different. In the example of image306, although the image includes an object which “appearance” is similar, the bibliographic information does not include words matching with the condition. The former example increases the processing time because of redundant image analysis process. The latter example causes detection failure. A method for decreasing detection failure will be described in an embodiment 2.

FIG. 4is a flowchart showing a process by the image analysis system100to identify the object region in the image. Hereinafter, each step inFIG. 4will be described.

The image/document input unit107stores the received image data and the bibliographic information into the image database105. The image database105extracts the image feature from the image data, and stores the image feature in association with the bibliographic information. The image/document input unit107may perform the process for extracting the image feature. This step may be performed before performing step S402and subsequent steps. It is not necessary to perform this step every time when performing this flowchart.

The image/document input unit107acquires the query image including the detection target object (S402). The metadata generator108generates the metadata of the query image (S403). Details will be described later usingFIG. 5.

The analysis target determinator109determines, among the metadata generated by the metadata generator108in step S403, the metadata that is used for narrowing the image data to which the object detection is performed. Specifically, it may be determined mechanically according to the reliability of the metadata (e.g. automatically selecting from those with higher reliabilities within a predetermined range). Alternatively, the metadata may be presented to the user through the data output unit111and the user may select anyone of the presented metadata.

The analysis target determinator109searches the bibliographic information stored in the image database105using the metadata selected in step S404as a search key, thereby acquiring a group of image data matching with the search key. This image group is the target for the object detection process.

The image analysis device106performs step S407with respect to each of the image data included in the image group acquired in step S405. In step S407, the object region detector110extracts, among the images included in the image group acquired in step S405, the region that is similar to the object included in the query image. The method for extracting the object region will be described later usingFIG. 7.

The data output unit112outputs the detection result of object regions detected by the object region detector110. The detection result may be outputted in the processed order. Alternatively, the detection result may be outputted after sorting it on the basis of the number of detected objects or on the basis of the reliability. Further, as shown in the detection result304inFIG. 3, additional information such as the number of detected objects, detection reliability, or the rectangle indicating the detected object region may be outputted along with the detection result. Yet further, the detection result may be outputted on the display through the display device103, or may be outputted in the form of data describing the detection result and the above-mentioned additional information.

If there is not more objects to be detected (if there is no more instruction from the user), this flowchart terminates. If there are other objects in the query image, or other objects are to be detected such as when the user newly specifies another query image, the flowchart returns to step S402and the same processes will be performed.

FIG. 5is a diagram showing a sequence by the metadata generator108to generate metadata of the query image. Hereinafter, each step inFIG. 5will be described.

The metadata generator108searches, using the query image301as a search key, images similar to the search key from the image database105. Similar image search is a method for searching similar images by extracting information such as colors or shapes of the image as high order vector information, and by evaluating the similarity between images on the basis of the distance between the vectors. As a result, a group501of images which “appearance” are similar to that of the query image301is acquired. In addition, the image database105stores images and bibliographic information associated with each other, thus a group502of bibliographic information is acquired from the group502of similar images.

The metadata generator108extracts characteristic words included in the bibliographic information group. It is desirable if organized data such as category codes of images is attached as the bibliographic information. Even if documents such as descriptive texts are attached, such documents are highly likely to include characteristic words showing the meaning of the image. Then in this step, the metadata generator108separates each of the bibliographic information into atomic data (minimum unit) (e.g. separates from documents to words), and assumes that the minimum unit is the metadata. Then it is possible to generate the metadata of the query image301.

The metadata generator108counts the frequency by which the metadata generated in the sequence 1 appears in the bibliographic information. The metadata generator108calculates, using the appearance frequency, a score of each of the metadata generated in the sequence 1. The appearance frequency may be simply sorted, as the score of the metadata, in descending order of the score. Alternatively, an evaluation indicator in which a weight is attached to the appearance frequency may be used as the score.

(FIG. 5: Step S502: Example of Calculating the Score No. 1)

TF-IDF (Term Frequency-Inverse Document Frequency) may be used as the score of the metadata. TF-IDF is an evaluation indicator in which a frequency tf(t) of the metadata t is multiplied by an inverse document frequency idf(t). The inverse document frequency idf(t) is calculated by Equation 1 below, where N is the number of records in the database, and df(t) is the frequency of the bibliographic information including the metadata t among whole of the database.

A stochastic evaluation indicator may be used as the score of the metadata. For example, when evaluating a metadata t, the scale kl(t) of the difference of probability distribution between p(t) and q(t) shown in Equations 2-4 below may be used as the score of the metadata, where: q(t) is a probability by which the metadata t is included in the bibliographic information when randomly acquiring images among whole of the database; p(t) is a probability by which the metadata t is included in the bibliographic information when randomly acquiring images among the image group of the similar image search result.

df′ (t): frequency of bibliographic information including metadata t in the similar image search result

M: number of images in the similar image search result

FIG. 6is s flowchart showing a process sequence by the metadata generator108to generate the metadata of the query image301. Hereinafter, each step inFIG. 6will be described.

The metadata generator108calculates the image feature of the query image301(S601). The metadata generator108performs a similar image search using the image feature extracted in step S601as a search key (S602). The smaller the distance between the feature vectors of each of the images is, the higher the similarity between the images is. The search result is sorted according to the distance, and then is outputted.

The metadata generator108performs steps S604-S606with respect to the similar images acquired in step S602.

The metadata generator108reads out, from the image database105, the bibliographic information associated with the similar image acquired in step S602(S604). The metadata generator108separates the bibliographic information acquired in step S604into atomic data, and uses it as the metadata (S605). For example, if the bibliographic information is a document, a morphology analysis is performed to separate the bibliographic information into words. The process for separating the bibliographic information may be performed in advance when storing the document into the image database105, for the sake of efficiency.

The metadata generator108counts the frequency by which the metadata generated in step S605appears in the bibliographic information that is read out in step S604. The metadata generator108calculates an accumulated frequency for each of the metadata throughout steps S603-S607. At this time, in order to reflect the image similarity into the frequency of the metadata, the similarity may be weighted and then may be added to the accumulated frequency.

The metadata generator108calculates the score of the metadata using the accumulated frequency for each of the metadata calculated in steps S603-S607. The method for calculating the score is as described inFIG. 5.

The metadata generator108sorts the metadata in the order of the score calculated in step S608. The metadata generator108excludes the metadata below or equal to a threshold, and outputs the sorted result.

FIG. 7is a diagram showing a detection method for object regions in step S407inFIG. 4. This method is for detecting regions where the object is present in the image by using the image of the object to be detected as a template, and by detecting regions matching with the template.

Firstly, an image feature of a typical image (template) of the object to be detected is extracted, and then is stored in a template database704. The template image mentioned here corresponds to the query image301. If it is desired to detect multiple objects, for example, the template database704may store multiple templates (images of the detection targets) corresponding to each of the objects. The templates stored in the template database704are reset every time when the object to be detected changes.

When an input image701(an image in the image database105) that is a target for object detection is given, the object region detector110varies location or size of a scan window702to extract a candidate region703of the object. Next, the object region detector110searches templates in the template database704which feature vector is the closest to that of the candidate region703, for all of the candidate regions703. If the distance between the feature vector of the found template and that of the candidate region703is at or below a predetermined threshold, it is determined that the candidate region703includes the object of the template, and the candidate region703is added to the detection result. At this time, the distance between the feature vector of the closest template and that of the candidate region703may be used as a reliability of the detection result.

FIG. 8is a flowchart showing a process for the object region detector110to detect the object. Hereinafter, each step inFIG. 8will be described.

The object region detector110calculates the feature of the template, and stores it into the template database. If there are multiple of the input images701as the object detection target, and if the same template is used to perform the detection process, this step may be performed only once at the first iteration.

The object region detector110extracts the candidate region703in the input image701. The candidate region703is mechanically extracted by moving the scan window for each step or by changing the size of the scan window.

The object region detector110performs steps S802-S806for all of the candidate regions703.

The object region detector110calculates the reliability of the candidate region703. For example, as described inFIG. 7, the distance between the feature of the template and the feature of the candidate region703may be used for calculating the reliability.

If the reliability of the candidate region703calculated in step S803is at or below a predetermined threshold, the flowchart proceeds to step S805, otherwise the flowchart skips step S805(S804). The object region determinator110adds the candidate region703with the reliability above the predetermined threshold to the detection result list (S805).

The object region detector110outputs the detection result list, and this process flow terminates. The detection result is outputted as a set of coordinate information in the input image701(e.g. [horizontal coordinate of left-top of the rectangle, vertical coordinate of left-top of the rectangle, horizontal coordinate of right-bottom of the rectangle, vertical coordinate of right-bottom of the rectangle]) and the reliability.

FIG. 9is a diagram showing a process sequence between each functional units in a process where the image analysis system100identifies object regions in images. Hereinafter, each step inFIG. 9will be described.

The user inputs, through the input device102, images to be stored in the image database105and documents associated with the images (S901). The group of images and documents is sent to the image database105through the image analysis device106. The image database105extracts features from the images received from the image analysis device106, and stores the features in association with the bibliographic information acquired from the documents (S902). Steps S901-S902correspond to step S401inFIG. 4.

The user inputs an image (query image) of the object to be detected (S903). The image analysis device106requests the image database105to search similar images using the query image as a search key (S904). The image database105extracts image features from the query image, searches images similar to the query image using the image features, and responds the similar image and its bibliographic information to the image analysis device106(S905). The image analysis device106generates the metadata of the query image using the bibliographic information received from the image database105, and calculates the score of the metadata (S906).

The image analysis device106presents the metadata generated in step S906and its score to the user through the display device103or through the data storage device104(S907). The user selects the metadata that is used for narrowing the images to be searched with reference to the metadata and its score (S908). It is possible for the image analysis device106to automatically select the metadata such as by omitting step S908and by selecting the metadata from higher scores sequentially.

The image analysis device106requests the image database105to search, using the metadata selected by the user in step S908as a search key, images which bibliographic information matches with the search key (S909). The image database105searches the bibliographic information corresponding to the search query, and responds the image associated with the bibliographic information to the image analysis device106(S910).

The image analysis device106detects objects included in the query image for each of the images acquired in step S910, thereby identifying the regions similar to the query image. The detection result is described with coordinates of the rectangle of the object in the image (e.g. [horizontal coordinate of left-top of the rectangle, vertical coordinate of left-top of the rectangle, horizontal coordinate of right-bottom of the rectangle, vertical coordinate of right-bottom of the rectangle]) and the reliability indicating “likelihood of object”. The detection result is outputted through the data output unit112.

FIG. 10is a diagram showing a configuration example of an operational screen that is used for acquiring images including specified objects from the image database105. This screen may be provided on the display device103. The user sends operational information to the operational information input unit111by operating a cursor1006displayed in the screen using the input device102.

The operational screen inFIG. 10has a query image input area1001, a similar image search button1002, a metadata generation button1003, a similar image display area1004, a metadata display area1007, a detection target number display area1008, a predicted processing time display area1009, a detection start/stop button1010, and a detection result display area1011.

The user firstly inputs a query image stored in the image/document storage device101into the query image input area1001. A dialog specifying file paths in the file system may be used, for example. Alternatively, intuitive operations may be used such as drag & drop.

When the user clicks the similar image search button1002, the image analysis device106acquires the image similar to the query image from the image database105, and displays it in the similar image display area1004. The image analysis device106generates the metadata of the query image using the bibliographic information of the similar image displayed in the similar image display area1004. The metadata may be generated using all of the similar images. Alternatively, the user checks the similar images and specifies the similar image to be used. The user uses a checkbox1005to specify the similar image, for example. In the example shown inFIG. 10, the checkbox of the rightmost similar image is disabled so that the similar image will not be used when generating the metadata.

The user selects the metadata that is used for narrowing the images to which the object detection is performed, considering such as the metadata and its score or the number of images. The user selects the metadata using a checkbox1012, for example. The image analysis device106searches the bibliographic information every time when the checkbox1012is clicked, and displays the number of images in which the selected metadata is included in the bibliographic information in the detection target number display area1008. In addition, the predicted processing time for performing object detection with respect to that number of images is displayed in the predicted processing time display area1009. The processing time may be approximately calculated according to the number of images to which object detection is performed. This enables the user to efficiently select the metadata.

When the detection start/stop button1010is clicked, the analysis target determinator109acquires, using the metadata selected through the above-described operations, a group of images to which object detection is performed. The object region detector110performs object detection with respect to the image group. The detection processes performed by the object region detector110are independent for each of the images. Thus the processed images may be displayed in the detection result display area1011sequentially in the processed order, or the process may be started/stopped every time when the detection start/stop button1010is clicked.

As discussed thus far, the image analysis system100according to the embodiment 1 performs object detection only for image data including the metadata of the query image as the bibliographic information. This enables efficiently narrowing the target images for object detection among vast amount of images, thereby rapidly searching the images including the object specified by the user.

The image analysis system100according to the embodiment 1, for example in searches or examinations of picture trademarks, may be used when determining whether figures that are to be newly registered are used in registered picture trademarks. In this case, category codes or descriptive texts may be utilized as the bibliographic information of the image required for generating the metadata.

The image analysis system100according to the embodiment 1 may be applied to auction sites or shopping sites. This enables rapidly searching products including patterns or marks specified by the user. In this case, titles of the products or comments from the sellers may be utilized as the bibliographic information of the image.

The image analysis system100according to the embodiment 1 may be applied to video contents. This enables checking scenes in which celebrities or landmarks are present and checking the location of such scenes in the frame image. In this case, closed captions or texted voices may be utilized as the bibliographic information of the image.

In the image analysis system100described in the embodiment 1, the analysis target determinator109narrows, by bibliographic information search, the images that are target of object detection. Therefore, even if the object specified by the user is actually included, images without sufficient bibliographic information are not searched and thus are not included in the analysis result. Hereinafter, a method for decreasing detection failure by extending the bibliographic information will be described. Other configurations are approximately the same as those of the embodiment 1. Thus hereinafter differences will be mainly described.

FIG. 11is a diagram showing an example extending the bibliographic information. For the sake of comparison,FIG. 11(a)shows a conceptual diagram of search without extending the bibliographic information andFIG. 11(b)shows a conceptual diagram of search extending the bibliographic information in the embodiment 2.

As shown inFIG. 11(a), the image analysis system100described in the embodiment 1 searches the bibliographic information using metadata “star” as the search condition in order to search images including the object included in the query image301. As a result, object detection will be performed to an image if the bibliographic information includes “star” as for the image1101. On the other hand, object detection is not performed to images if the bibliographic information does not include “star” as for the image1102. However, the image1102actually includes regions similar to the query image301. Thus the object detection fails for the image1102.

In the embodiment 2, as shown inFIG. 11(b), the metadata is also generated for images stored in the image database105. The method for generating the metadata may be the same as that for generating the metadata of the query image301. The newly generated metadata is stored in the image database105as additional bibliographic information. The image analysis device106also searches the additional bibliographic information when narrowing the images to which object detection is performed. This enables searching images that do not include “star” as the bibliographic information as for the image1103.

Comparing with images where a single object is present, images where multiple objects are present generally have more variations of “appearance” due to change in layout of objects. Thus it is unlikely that such images are found as images similar to the query image. On the other hand, if an image is found where the similarity to the query image is high and where multiple objects are present, the informational quantity will not be significantly degraded even if the bibliographic information of the similar image is reused.

FIG. 12is a flowchart showing a sequence of the process extending the bibliographic information. This flowchart is performed by the metadata generator108with respect to all images in the image database105, where steps S1201-S1204are repeated. This flowchart may be performed when the system load is small, for example. Alternatively, this flowchart may be performed immediately after images are stored into the image database105initially. Hereinafter, each step inFIG. 12will be described.

The metadata generator108generates the metadata of images in the image database105using the existing bibliographic information stored in the image database105. The method for generating the metadata is the same as that shown inFIG. 6. However, the threshold of similarity may be stricter than that ofFIG. 6, or image feature may be used that are not changed even if the object layouts vary.

The metadata generator108stores the metadata generated in step S1202into the image database105as additional bibliographic information.

FIG. 13is a Venn diagram showing analyzed targets for explaining a process in which detection failure is decreased by extending the bibliographic information.FIG. 13(a)is a Venn diagram using the existing bibliographic information only.FIG. 13(b)is a Venn diagram using the extended bibliographic information.

InFIG. 13(a), the group1301is a group of all images stored in the image database105. If the object specified by the user is searched without using the image analysis system100, the group1301is the target for image analysis process.

The group1302is a group of images including regions of “star-shaped figure” specified by the user. It is desirable if the image analysis system100outputs this group.

The group1303is an image group acquired by the image analysis system100by performing bibliographic information search using an automatically generated metadata “star” is used as a query key. The image analysis system100performs object detection to this group.

The group1304is an image group to which object detection is not performed because the images include “star-shaped figures” but do not include “star” in the bibliographic information, thus not handled as detection targets.

The group1305is an image group that is the target of object detection and that the object is detectable because the images include “star-shaped figure”. However, it depends on the identification performance of the object detector whether the object will actually be detected. The method for improving the performance of the object detector will be described later usingFIG. 15in an embodiment 3.

The group1306is an image group which does not require object detection because the images include “star” in the bibliographic information but do not include regions similar to “star-shaped figure” specified by the user.

As shown inFIG. 13(b), if the bibliographic information is extended, the group including “star” in the bibliographic information will be enlarged. At this time, since the group is expanded according to the result of similar image search, it is highly likely that the expanded region includes “star-shaped figure”. As a result, although the processing time for detecting objects will be increased, detection failure will be decreased.

FIG. 14is a chart showing a relationship between a processing time of image analysis and coverage. The horizontal axis indicates the processing time and the vertical axis indicates the coverage. Coverage means the percentage indicating the processed proportion of the group1302inFIG. 13. In the horizontal axis, the processing time100is the time when all images to be searched, i.e. the group1302inFIG. 13, are analyzed.

InFIG. 14, it is assumed that the proportion occupied by the group1305among the group1302is 60%, and that the proportion occupied by the group1304among the group1302is 40%. By extending the bibliographic information, it is assumed that the proportion occupied by the group1305among the group1302will be 80%, and that the proportion occupied by the group1304among the group1302will be 20%. In addition, it is assumed that the processing time for performing object detection with respect to the group1305is one-tenth of the processing time for performing object detection with respect to the group of all images.

The line1401represents for the transition of the coverage by which the coverage changes when all images (the group1301) are analyzed. The coverage linearly increases if images are randomly picked up from the image database105.

The polygonal line1402represents for the transition of the coverage by which the coverage changes if the analyzed targets are narrowed using the metadata. The line from the start to the point1404indicates the detection process with respect to the narrowed images. The line after the point1404indicates the detection process with respect to images including other images. At the point1404, it is understood that 60% of coverage is achieved by the processing time 1/10 of the line1401.

The polygonal line1403represents for the transition of the coverage by which the coverage changes if the analyzed targets are narrowed using the extended bibliographic information according to the method of the embodiment 2. The line from the start to the point1405indicates the detection process with respect to the narrowed images. The line after the point1405indicates the detection process with respect to images including other images. Although the processing time until the point1405is increased because of the increased detection targets, the coverage is improved.

As shown inFIG. 14, the processing time and the coverage are in the trade-off relationship. Thus it is necessary to determine whether the bibliographic information should be extended, depending on the application. When examining picture trademarks, it is sufficient if one similar image is found, thus it is preferred to perform detection process after sufficiently narrowing the processed targets to obtain highly responsive system. If it is desired to improve the coverage, the original bibliographic information may be used first and then additional bibliographic information may be used if necessary.

As discussed thus far, the image analysis system100according to the embodiment 2 generates the metadata of images stored in the image database105, adds the generated metadata into the image database105as new bibliographic information, and performs the same process as that of the embodiment 1. This enables covering images that are not detected by using the existing bibliographic information only.

In an embodiment 3 of the present invention, a method for improving accuracy of object detection by utilizing intermediate data during the process of the image analysis system100. This method uses multiple templates described inFIG. 7for detecting objects. Other configurations are approximately the same as those of the embodiments 1-2. Thus hereinafter similar image search using multiple templates when generating the metadata of the query image will be mainly described.

FIG. 15is a diagram showing a method for increasing accuracy of object detection by expanding templates that are used in searching images similar to the query image. Hereinafter, expanding templates in the embodiment 3 will be described usingFIG. 15.

In the object detection method described inFIG. 7, the object region is identified by checking portions of the image, the templates, and the similarity. Thus if only the query image301is used as the template, it is impossible to detect star shapes with significantly different appearances such as the image1505. In addition, “solar shape” such as the image1506or “planet shape” such as the image1507cannot be detected even if those images have the same concept of “star”.

The image analysis device106according to the embodiment 3 uses, as additional templates, the intermediate data generated during the process. Specifically, the image group1501acquired as images similar to the query image301during generating the metadata of the query image301is used as the template for object detection. In other words, not only the object specified by the user but also objects similar to the specified object are targets for object detection. Thus in the embodiment 3, the template of the target object is expanded using similar images acquired by similar image search performed for generating the metadata.

The image analysis device106searches images similar to the query image301according to the method described inFIG. 6(S601-S602). The appearances of the image group1501do not completely match with the query image301. However, since those images are close to the object specified by the user, those images may be appropriate as templates when performing object detection later. Thus those similar images are stored in a template database1504. The template database1504is a database that temporarily stores templates that are used when performing object detection. The template database1504is reset every time when the query image301is changed.

The image analysis device106generates the metadata of the query image301and of the image group1501according to the method described inFIG. 6. Now it is assumed that metadata “star” is generated, for example.

The image analysis device106searches the bibliographic information matching with the metadata “star”. As a result, the image groups1505-1507including the image1503corresponding to the concept of “star” are acquired. The result of bibliographic information search may be images including multiple objects or may include many noises as described inFIG. 13. Templates may be selected through interactive operations with the user using operational screens such asFIG. 10. For example, the images acquired by bibliographic information search may be displayed on the operational screen, and then the user may select, among the displayed images, images that will be used as templates in object detection.

The image analysis device106performs object detection with respect to the image groups1505-1507using multiple templates stored in the template database1504. For example, the image group1501is used as the template in addition to the query image301. If the user specifies the image1503as the template on the operational screen, it is further used as the template. This enables detecting star-shaped regions (e.g. solar shape or Saturn shape such as the image1503) which appearances are not similar to that of the query image301.

As discussed thus far, the image analysis device106according to the embodiment 3 uses, as expanded templates for object detection, similar images acquired when generating the metadata of the query image301or images acquired when searching the bibliographic information. This enables detecting objects which concept is the same but which “appearance” is different.

In an embodiment 4 of the present invention, a configuration example will be described where the image analysis system100is embedded into a content cloud system. Hereinafter, a summary of the content cloud system will be described first. Then a method for embedding the image analysis system100into the content cloud system as an analysis module will be described. The configuration of the image analysis system100is the same as that of the embodiments 1-3.

FIG. 16is a schematic diagram of a content cloud system1600according to the embodiment 4. The content cloud system1600includes an Extract Transform Load (ETL) module1603, a content storage1604, a search engine1605, a metadata server1606, and a multimedia server1607. The content cloud system works on common computers comprising one or more CPUs, memories, and storage devices. The system itself comprises various modules. Each of the modules may be executed on independent computers. In such cases, each of storages and modules are connected by networks. The system is implemented by distributed process performing data communications through the network.

The application program1608sends a request to the content cloud system1600through the network. The content cloud system1600sends, to the application1608, information according to the request.

The content cloud system1600receives, as inputs, any form of data1601such as video data, image data, document data, or voice data. The data1601is, for example, picture trademark and its gazette document, website images and HTML documents, or video data with closed captions or with voices. The data1601may be structured data or non-structured data. The data inputted to the content cloud system1600is temporarily stored in the storage1602.

The ETL1603monitors the storage1602. When the data1601is stored into the storage1602, the ETL1603activates the information extraction module16031. The extracted information (metadata) is archived in the content storage1604.

The information extraction module16031includes, for example, a text index module and an image identification module. Examples of metadata may be such as time, N-gram index, image recognition result (object name, region coordinate in image), image feature and its related word, or voice recognition result. Any program that extracts some information (metadata) may be used as the information extraction module16031. Since commonly known techniques may be employed, details of the information extraction module16031are omitted here. If necessary, the metadata may be compressed using data compression algorithm. After the ETL1603extracts the information, information such as data filename, data registration date, original data type, or metadata text information may be stored in Relational Data Base (RDB).

The content storage1604stores the information extracted by the ETL1603and the data1601before processed that is temporarily stored in the storage1602.

When requested from the application program1608, the search engine1605performs text search according to the index created by the ETL1603in the case of text search, and sends the search result to the application program1608. Commonly known techniques may be applied as the algorithm of the search engine1605. The search engine1605may include not only modules for searching texts but also modules searching data such as images or voices.

The metadata server1606manages the metadata stored in the RDB. For example, it is assumed that data filenames, data registration dates, original data types, or metadata text information extracted by the ETL1603are stored in the RDB. When requested from the application1608, the metadata server1606sends the information in the RDB according to the request.

The multimedia server1607associates the metadata information extracted by the ETL1603with each other, and stores the meta information in structured manner of graph format. An example of association is: with respect to a voice recognition result of “apple” stored in the storage1604, a relationship between original voice file, image data, or related words may be represented in network format. When requested from the application1608, the multimedia server1607sends corresponding meta information to the application1608. For example, if a request of “apple” is received, the metadata server1607sends, according to the built graph structure, meta information related in the network graph such as images including apple, average price, or song name of artist.

In the content cloud system1600, the image analysis system100works as the information extraction module16031in the ETL1603. The image/document storage device101and the data storage device104inFIG. 1correspond to the storage1602and the content storage1604inFIG. 16respectively. The image analysis device106corresponds to the information extraction module16031. If multiple of the information extraction module16031is embedded into the ETL1603, resources of one computer may be shared or independent computers may be used for each of the modules. The image database105inFIG. 1corresponds to the dictionary data16032that is required by the ETL1603to extract information.

As discussed thus far, the image analysis system100according to the present invention may be applied to components of the content cloud system1600. The content cloud system1600may integrate information across media by generating metadata that is commonly usable among each media data. This enables providing highly valued information to users.

The present invention is not limited to the embodiments, and various modified examples are included. The embodiments are described in detail to describe the present invention in an easily understood manner, and the embodiments are not necessarily limited to the embodiments that include all configurations described above. Part of the configuration of an embodiment can be replaced by the configuration of another embodiment. The configuration of an embodiment can be added to the configuration of another embodiment. Addition, deletion, and replacement of other configurations are also possible for part of the configurations of the embodiments.

The configurations, the functions, the processing units, the processing means, etc., may be realized by hardware such as by designing part or all of the components by an integrated circuit. A processor may interpret and execute programs for realizing the functions to realize the configurations, the functions, etc., by software. Information, such as programs, tables, and files, for realizing the functions can be stored in a recording device, such as a memory, a hard disk, and an SSD (Solid State Drive), or in a recording medium, such as an IC card, an SD card, and a DVD.

REFERENCE SIGNS LIST