Patent Abstract:
There is disclosed an image retrieving method of an image retrieving apparatus which has an image memory to store image data of an image photographed by a photographing device and photographic information of the image, and retrieves desired image data from the image memory, includes selecting a plurality of image data from the image memory, classifying the photographic information corresponding to the selected image data in accordance with a predetermined classification standard, classifying the image data stored in the image memory based on the photographic information classified by the classification standard, adding keywords to the classified image data, and retrieving desired image data based on the added keywords.

Full Description:
CROSS-REFERENCE TO RELATED APPLICATIONS  
       [0001]     This application is based upon and claims the benefit of priority from prior Japanese Patent Applications No. 2004-121824, filed Apr. 16, 2004; and No. 2004-179567, filed Jun. 17, 2004, the entire contents of both of which are incorporated herein by reference.  
       BACKGROUND OF THE INVENTION  
       [0002]     1. Field of the Invention  
         [0003]     The present invention relates to an image retrieving technology, and more particularly to a technology of retrieving desired images including an image to which no keyword is added.  
         [0004]     2. Description of the Related Art  
         [0005]     As a method of designating a retrieval key in conventional image retrieval, there has been known a method of adding keywords to images and extracting an image as a result of text-retrieving based on the keywords.  
         [0006]     In the case of performing image retrieval by using a plurality of keywords, a technology has been disclosed which classifies keywords into groups by meanings and sets priorities to make image retrieval processing efficient (Jpn. Pat. Appln. KOKAI Publication No. 2000-29901).  
         [0007]     According to a technology disclosed in Jpn. Pat. Appln. KOKAI. Publication No. 8-16651, a first keyword is created by using at least one of an input date and a name of image data. Further, a second keyword is generated by referring to information regarding the first keyword. By using these first and second keywords, it is possible to add keywords easily understood by an operator.  
         [0008]     Additionally, there has been proposed a technology of using a directory structured to enable quick retrieval (Jpn. Pat. Appln. KOKAI Publication No. 2000-276484).  
       BRIEF SUMMARY OF THE INVENTION  
       [0009]     A first aspect of the present invention is directed to an image retrieving apparatus includes 
        an image memory which stores image data of an image photographed by a photographing device and photographic information of the image, an image selection unit which selects image data from the image memory, a photographic information classification unit which classifies the photographic information corresponding to the image data selected by the image selection unit in accordance with a predetermined classification standard, an image classification unit which classifies the image data stored in the image memory based on the photographic information classified by the photographic information classification unit, a keyword addition unit which adds keyword to the image data classified by the image classification unit, and a retrieval unit which retrieves desired image data based on the keywords added by the keyword addition unit.        
 
         [0011]     A second aspect of the present invention is directed to an image retrieving method of an image retrieving apparatus which has an image memory to store image data of an image photographed by a photographing device and photographic information of the image, and retrieves desired image data from the image memory, includes selecting a plurality of image data from the image memory, classifying the photographic information corresponding to the selected image data in accordance with a predetermined classification standard, classifying the image data stored in the image memory based on the photographic information classified by the classification standard, adding keywords to the classified image data, and retrieving desired image data based on the added keywords.  
         [0012]     A third aspect of the present invention is directed to a program executed in an image retrieving apparatus which has an image memory to store image data of an image photographed by a photographing device and photographic information of the image data, and retrieves desired image data from the image memory, causing a computer to execute, an image selection step of selecting a plurality of image data from the image memory, a photographic information classification step of classifying the photographic information corresponding to the image data selected in the image selection step in accordance with a predetermined classification standard, an image classification step of classifying the image data stored in the image memory based on the photographic information classified in the photographic information classification step, a keyword addition step of adding keywords to the image data classified in the image classification step, and a retrieval step of retrieving the image data based on the keywords added in the keyword addition step.  
         [0013]     Advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out hereinafter. 
     
    
     BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING  
       [0014]     The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention, and together with the general description given above and the detailed description of the embodiments given below, serve to explain the principles of the invention.  
         [0015]      FIG. 1  is a diagram showing a structure of an image retrieval screen by an image retrieving method according to a first embodiment of the present invention;  
         [0016]      FIG. 2  is a diagram showing an image retrieval screen when a calendar tab is selected;  
         [0017]      FIG. 3  is a diagram showing an image retrieval screen when an Exif tag tab is selected;  
         [0018]      FIG. 4  is a diagram showing a configuration of an image retrieving apparatus;  
         [0019]      FIG. 5  is a flowchart showing a rough image retrieving process;  
         [0020]      FIG. 6  is a flowchart showing a process of registering an image in an image management DB;  
         [0021]      FIG. 7  is a diagram showing a structure of a registration record;  
         [0022]      FIG. 8  is a table showing accessory information of an image file of an Exif form;  
         [0023]      FIG. 9A  is a database showing contents of a representative Exif tag;  
         [0024]      FIG. 9B  is a database showing contents of a representative Exif tag;  
         [0025]      FIG. 10  is a flowchart showing a folder displaying process;  
         [0026]      FIG. 11  is a flowchart showing a folder displaying process;  
         [0027]      FIG. 12  is a flowchart showing a keyword retrieval displaying process;  
         [0028]      FIG. 13  is a flowchart showing a keyword deleting process;  
         [0029]      FIG. 14  is a flowchart showing a calendar displaying process;  
         [0030]      FIG. 15  is a flowchart showing a calendar displaying process;  
         [0031]      FIG. 16  is a flowchart showing an Exif tag displaying process;  
         [0032]      FIG. 17  is a flowchart showing an Exif tag displaying process;  
         [0033]      FIG. 18  is a flowchart showing an Exif tag contents displaying process;  
         [0034]      FIG. 19  is a diagram showing an image retrieval screen when a folder tab is selected;  
         [0035]      FIG. 20  is a diagram showing an image retrieval screen when a calendar tab is selected;  
         [0036]      FIG. 21  is a diagram showing an image retrieval screen when an Exif tag tab is selected;  
         [0037]      FIG. 22  is a graph explaining a principle of image classification and retrieval;  
         [0038]      FIG. 23  is a diagram showing a configuration of an image retrieving apparatus to which the image retrieving method of the embodiment of the invention is applied;  
         [0039]      FIG. 24  is a flowchart showing a rough registration process in a keyword storing unit;  
         [0040]      FIG. 25  is a flowchart showing a rough similarity analyzing process;  
         [0041]      FIG. 26  is a graph showing a distribution range of one variable;  
         [0042]      FIG. 27  is a diagram showing a structure of the keyword storing unit;  
         [0043]      FIG. 28  is a flowchart showing a rough registration process in a management data storing unit;  
         [0044]      FIG. 29  is a diagram showing a structure of the management data storing unit;  
         [0045]      FIG. 30  is a flowchart showing a rough additional registration process in the management data storing unit;  
         [0046]      FIG. 31  is a flowchart showing a rough image retrieving process based on a keyword; and  
         [0047]      FIG. 32  is a flowchart showing a rough image retrieving process based on an image. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
     First Embodiment  
       [0048]     According to an image retrieving method of an embodiment of the present invention, a user executes management and retrieval of images based on an image retrieval screen displayed on a display unit of an image retrieving apparatus.  
         [0049]     A structure of an image retrieval screen of an image retrieving method according to a first embodiment of the present invention will be described by referring to  FIG. 1 .  
         [0050]     An image retrieval screen  1  comprises a switching tab  2 , a browse area  3 , a keyword area  4 , and a thumbnail area  5 . In the switching tab  2 , tabs (folder tab  2   a , calendar tab  2   b , and Exif tag tab  2   c ) are disposed to switch images displayed on the image retrieval screen  1 . The browse area  3  displays classification information corresponding to a switched tab. The keyword area  4  comprises a keyword character string  4   a  and a keyword deletion button  4   b . The thumbnail area  5  displays a thumbnail image.  
         [0051]     Next, an outline of the image retrieving method will be given by referring to FIGS.  1  to  3 .  
         [0052]      FIG. 1  shows the image retrieval screen  1  when the folder tab  2   a  is selected. When the user selects the folder tab  2   a , a folder tree is displayed in the browse area  3 . A structure of this folder tree is displayed in a known form, and thus detailed description thereof will be omitted. When the user selects a desired folder from this tree, an image file present below the folder is retrieved from an image management DB  34 . Then, a thumbnail image that is an index image of the image file is displayed in the thumbnail area  5 .  
         [0053]     Next, when the user clicks the character string  4   a  of a specific keyword in the keyword area  4 , a thumbnail image of an image to which a keyword has been added is displayed in the thumbnail area  5 . Designation and displaying of the specific keyword is referred to as “narrowed-down displaying”. On the other hand, when the user clicks the character string  4   a  of “all displaying” in the keyword area  4 , a thumbnail image of an image to which any one of keywords displayed in the keyword area  4  has been added is displayed in the thumbnail area  5 . Designation and displaying of all the keywords are referred to as “all displaying”.  
         [0054]     Subsequently, when the user drags and drops the thumbnail image of the thumbnail area  5  on a specific keyword character string  4   a  of the keyword area  4 , the specific character is added as a keyword to the image. It is to be noted that even when the user drags and drops the thumbnail image of the thumbnail area  5  on the “all displaying” character string  4   a  of the keyword area  4 , a keyword is not added. On the other hand, when the user drags and drops the thumbnail image of the thumbnail area  5  on the keyword deletion button  4   b  of the keyword area  4 , a specific keyword or all keywords are deleted from the image.  
         [0055]     When the user drags and drops the folder of the browse area  3  on the specific character string  4   a  or the keyword deletion button  4   b  of the keyword area  4 , keywords are added or deleted to/from all the image files below the folder in the aforementioned manner.  
         [0056]      FIG. 2  shows the image retrieval screen  1  when the calendar tab  2   b  is selected. When the user selects the calendar tab  2   b , a calendar is displayed in the browse area  3 . Accordingly, when the user selects a calendar year/month and a desired date from the calendar, a thumbnail image of an image file photographed or the like on that date is displayed in the thumbnail area  5 .  
         [0057]     Next, when the user clicks the character string  4   a  of a specific keyword in the keyword area  4 , a thumbnail image of an image having the keyword is displayed in the thumbnail area  5 . On the other hand, when the user clicks the character string  4   a  of “all displaying” in the keyword area  4 , a thumbnail image of an image having any one of keywords displayed in the keyword area  4  is displayed in the thumbnail area  5 .  
         [0058]     Subsequently, when the user drags and drops the thumbnail image of the thumbnail area  5  on a specific keyword character string  4   a  of the keyword area  4 , the specific character is added as a keyword to the image. Even when the user drags and drops the thumbnail image of the thumbnail area  5  on the “all displaying” character string  4   a  of the keyword area  4 , a keyword is not added. On the other hand, when the user drags and drops the thumbnail image of the thumbnail area  5  on the keyword deletion button  4   b  of the keyword area  4 , a specific keyword or all keywords are deleted from the image.  
         [0059]     When the user drags and drops the date of the browse area  3  on the specific character string  4   a  or the keyword deletion button  4   b  of the keyword area  4 , keywords are added or deleted to/from all the image files having the date in the aforementioned manner.  
         [0060]      FIG. 3  shows the image retrieval screen  1  when the Exif tag tab  2   c  is selected. When the user selects the Exif tag tab  2   c , an Exif tag selection box  3   a  is displayed in the browse area  3 . When the user selects a desired Exif tag, a thumbnail image of an image file to which the Exif tag has been added is displayed in the thumbnail area  5 . Then, the number of image files is displayed as a graph for every contents  3   b  in the Exif tag.  
         [0061]     Next, when the user clicks the character string  4   a  of a specific keyword in the keyword area  4 , a thumbnail image of an image having the keyword is displayed in the thumbnail area  5 . On the other hand, when the user clicks the character string  4   a  of “all displaying” in the keyword area  4 , a thumbnail image of an image having any one of keywords displayed in the keyword area  4  is displayed in the thumbnail area  5 .  
         [0062]     Subsequently, when the user drags and drops the thumbnail image of the thumbnail area  5  on a specific keyword character string  4   a  of the keyword area  4 , the specific character is added as a keyword to the image. When the user drags and drops the thumbnail image of the thumbnail area  5  on the “all displaying” character string  4   a  of the keyword area  4 , all keywords are added to the image. When the user drags and drops the thumbnail image of the thumbnail area  5  on the keyword deletion button  4   b  of the keyword area  4 , a specific keyword or all keywords are deleted from the image.  
         [0063]     When the user drags and drops the contents  3   b  of the browse area  3  on the specific character string  4   a  or the keyword deletion button  4   b  of the keyword area  4 , keywords are added or deleted to/from all the image files having the tag contents in the aforementioned manner.  
         [0064]     Next, a configuration of an image retrieving apparatus for realizing the aforementioned image retrieving method, and its main process will be described.  
         [0065]      FIG. 4  shows a configuration of an image retrieving apparatus  30 . The image retrieving apparatus  30  comprises a display unit  31 , an operation input unit  32 , a communication unit  33 , an image management DB  34 , an information memory  35 , a program memory  36 , and a processing unit  37 .  
         [0066]     The display unit  31  is a CRT, a liquid crystal display, a plasma display or a projector for displaying the image retrieval screen  1 . The operation input unit  32  is an input device such as a keyboard or a mouse for receiving an operation instruction input from the user. The communication unit  33  is an interface for transferring information such as an image file with an external device (not shown) such as a digital camera.  
         [0067]     The image management DB  34  saves various pieces of information for managing images, such as a keyword, folder path information, and Exif tag information. The information memory  35  stores an image file. For example, an image file received from the external device such as a digital camera is saved through the communication unit  33  in the information memory  35 . The program memory  36  saves a program for controlling each function of the image retrieving apparatus  30 . The processing unit  37  is in overall control of an operation of the image retrieving apparatus  30 .  
         [0068]     Next, a rough image retrieving process will be described by referring to FIGS.  5  to  18 . It is to be noted that the described process concerns main ones of image retrieving functions. Accordingly, even in the case of functions not described below, those described with reference to FIGS.  1  to  4  are within the image retrieving functions.  
         [0069]     In a step S 01  of  FIG. 5 , the processing unit  37  executes registration of images in the image management DB  34  ( FIG. 6 ).  
         [0070]     In steps T 01  to T 02  of  FIG. 6 , when the user designates a folder path having an image to be processed from the operation input unit  32 , the processing unit  37  retrieves images from the folder. Then, in steps T 03  to T 04 , for all the retrieved images, determination is made as to whether management information regarding the images has been registered or not in the image management DB  34 . In the case of Yes in the step T 03 , i.e., the management has not been registered, one record of the management information regarding the images is created and stored in the information management DB  34 . Then, all the images in the folder are registered, and the process returns.  
         [0071]      FIG. 7  shows a structure of a registration record registered in the information management DB. A registration record  10  comprises a registration ID area  10   a , a folder path area  10   b , a file name area  10   c , a thumbnail information area  10   d , an Exif tag information area  10   e , and a keyword area  10   f.    
         [0072]     In the registration ID area  10   a , a unique symbol is added to each record. In the folder path area  10   b , path information to a folder having an image saved there is written. In the file name area  10   c , a name is written to identify an image file. In the thumbnail information area  10   d , thumbnail image data is saved. In the Exif tag information area  10   e , contents of an Exif tag are written. In the keyword area  10   f , a keyword is saved to be added to the image.  
         [0073]     It is to be noted that not only the image data but also photographic information regarding an image are recorded in the image file. Representative photographic information is Exif information recorded in a header of the image file.  
         [0074]      FIG. 8  is a table showing an example of a representative structure of accessory information (Exif tag) of an image file of an Exif form which is a kind of photographic information. Such information is recorded in the image file together with the image data as described above.  
         [0075]      FIG. 9A  shows representative contents of a numerical value Exif tag  12 , and  FIG. 9B  representative contents of a character string Exif tag  13 .  
         [0076]     Representative items of the numerical value Exif tag  12  are “photographing date”, “diaphragm value”, “shutter speed”, “focal distance”, and the like. Contents of each item of the numerical value Exif tag  12  are represented by numerals. Representative items of the character string Exif tag  13  are “flash”, “macro”, “photographic mode”, and the like. Contents of each item of the character string Exif tag  13  are character strings which are predetermined.  
         [0077]     Referring back to  FIG. 5 , when the user selects any one of the switching tabs  2 , in a step S 02 , the processing unit  37  displays the image retrieval screen  1  corresponding to the selected switching tab  2 , and waits for user&#39;s next operation. Then, when the user selects the folder tag  2   a  and executes an operation for image retrieval, folder display processing of a step S 03  ( FIGS. 10, 11 ) is executed.  
         [0078]     In the case of Yes in a step T 10  of  FIG. 10 , i.e., when the user clicks the folder of the browse area  3 , in steps T 11  to T 12 , the processing unit  37  retrieves the registration record  10  from the image management DB  34 , and specifies one having a relevant folder name written in the folder path area  10   b . Then, thumbnail information stored in the thumbnail information area  10   d  of the specified registration record  10  is extracted and displayed in the thumbnail area  5 .  
         [0079]     Next, in the case of Yes in both of steps T 13  and T 14 , i.e., when the folder is dropped on the keyword character string  4   a  of the keyword area  4 , in steps T 15  to T 16 , the registration record  10  is retrieved from the image management DB  34 , and one having a name of the dropped folder written in the folder path area  10   b  is specified. Then, a corresponding keyword is registered in the keyword area  10   f  of the specified registration record  10 , and the process returns.  
         [0080]     On the other hand, in the case of Yes in a step T 17 , i.e., when the thumbnail image is dropped on the keyword character string  4   a  of the keyword area  4 , in a step T 18 , the registration record  10  is retrieved from the image management DB  34 , and one having a name of an image file written in the file name area  10   c , the image file corresponding to the dropped thumbnail image, is specified. Then, a corresponding keyword is registered in the keyword area  10   f  of the specified registration record  10 , and the process returns.  
         [0081]     In the case of Yes in a step T 21  of  FIG. 11 , i.e., when the keyword character string  4   a  of the keyword area  4  is clicked, in a step T 22 , keyword retrieval displaying is executed ( FIG. 12 ), and the process returns.  
         [0082]     In the case of Yes in a step P 01  of  FIG. 12 , i.e., when the “all displaying” character string  4   a  of the keyword area  4  is clicked, in a step P 02 , a limit by a keyword is released to display all thumbnail images in the thumbnail area  5 . In the case of No in the step P 01  of  FIG. 12 , i.e., when a specific character string  4   a  of the keyword area  4  is clicked, in steps P 03  to P 04 , among currently displayed images, an image having its specific character string written as a keyword in the keyword area  10   f  of the corresponding registration record  10  is retrieved, and its thumbnail image is displayed in the thumbnail area  5 .  
         [0083]     In the case of Yes in both of steps T 25  and T 26  of  FIG. 11 , i.e., when the folder is dropped on the keyword deletion button  4   b  of the keyword area  4 , in steps T 27  to T 28 , the registration record  10  is retrieved from the image management DB  34 , and one having a name of a relevant folder written in the folder path area  10   b  is specified. Then, keyword deletion processing of a step T 28  ( FIG. 13 ) is executed, and the process returns.  
         [0084]     In the case of Yes in a step P 10  of  FIG. 13 , i.e., when the thumbnail image is displayed in the thumbnail area  5  in a manner of “narrowed-down displaying”, in a step P 11 , for all the image files below the folder, a keyword being displayed in a narrowed-down manner alone is deleted from the keyword area  10   f  of the registration record  10 . On the other hand, in the case of No in the step P 10 , i.e., when the thumbnail image is displayed in the thumbnail area  5  in a manner of “all displaying”, in a step P 12 , for all the image files below the folder, all keywords are deleted from the keyword area  10   f  of the registration record  10 .  
         [0085]     On the other hand, in the case of Yes in the step T 25 , No in the step T 26 , and Yes in the step T 29 , i.e., when the thumbnail image is dropped on the keyword deletion button  4   b  of the keyword area  4 , keyword deletion processing of a step T 30  ( FIG. 13 ) is executed, and the process returns.  
         [0086]     In the case of Yes in the step P 10  of  FIG. 13 , i.e., when the thumbnail image displayed in the thumbnail area  5  in the manner of “narrowed-down displaying”, in the step P 11 , for the thumbnail image, the keyword being displayed in the narrowed-down manner alone is deleted from the keyword area  10   f  of the registration record  10 . On the other hand, in the case of No in the step P 10 , i.e., when the thumbnail image is displayed in the thumbnail area  5  in the manner of “all displaying”, in the step P 12 , for the thumbnail image, all the keywords are deleted from the keyword area  10   f  of the registration record  10 .  
         [0087]     Referring back to  FIG. 5 , when the user selects the calendar tab  2   b  and executes an operation of image retrieval, calendar display processing shown in a step S 04  is carried out ( FIGS. 14, 15 ).  
         [0088]     In the case of Yes in a step T 35  of  FIG. 14 , i.e., when the user clicks the date of the calendar of the browse area  3 , in steps T 36  to T 37 , the processing unit  37  retrieves the registration record  10  from the image management DB  34 , and specifies one in which “photographing day” in the numerical value Exif tag  12  of the Exif tag information area  10   e  matches the calendar date. Then, thumbnail information stored in the thumbnail information area  10   d  of the specified registration record  10  is extracted and displayed in the thumbnail area  5 .  
         [0089]     Next, in the case of Yes in both of steps T 38  and T 39 , i.e., when the folder is dropped on the keyword character string  4   a  of the keyword area  4 , in steps T 40  to T 41 , the registration record  10  is retrieved from the image management DB  34 , and one in which “photographing day” in the numerical value Exif tag  12  of the Exif tag information area  10   e  matches the calendar date is specified. Then, a corresponding keyword is registered in the keyword area  10   f  of the specified registration record  10 , and the process returns.  
         [0090]     On the other hand, in the case of Yes in a step T 42 , i.e., when the thumbnail image is dropped on the keyword character string  4   a  of the keyword area  4 , in a step T 43 , the registration record  10  is retrieved from the image management DB  34 , and one having a name of an image file written in the file name area  10   c , the image file corresponding to the dropped thumbnail image, is specified. Then, a corresponding keyword is registered in the keyword area  10   f  of the specified registration record  10 , and the process returns.  
         [0091]     In the case of Yes in a step T 46  of  FIG. 15 , i.e., when the keyword character string  4   a  of the keyword area  4  is clicked, in a step T 47 , keyword retrieval displaying is executed ( FIG. 12 ), and the process returns. This keyword retrieval displaying is similar to the foregoing, and thus detailed description thereof will be omitted.  
         [0092]     In the case of Yes in both of steps T 50  and T 51  of  FIG. 15 , i.e., when the date is dropped on the keyword deletion button  4   b  of the keyword area  4 , in steps T 52  to T 53 , the registration record  10  is retrieved from the image management DB  34 , and one in which “photographing day” in the numerical value Exif tag  12  of the Exif tag information area  10   e  matches the calendar rate is specified. Then, keyword deletion processing of a step T 53  ( FIG. 13 ) is executed, and the process returns. This keyword deletion processing is similar to the foregoing, and thus detailed description thereof will be omitted.  
         [0093]     On the other hand, in the case of Yes in the step T 50 , No in the step T 51 , and Yes in the step T 54 , i.e., when the thumbnail image is dropped on the keyword deletion button  4   b  of the keyword area  4 , keyword deletion processing of a step T 55  ( FIG. 13 ) is executed, and the process returns. This keyword deletion processing is similar to the foregoing, and thus detailed description thereof will be omitted.  
         [0094]     Referring back to  FIG. 5 , when the user selects the Exif tag tab  2   c  and executes an operation of image retrieval, Exif tag display processing shown in a step S 05  is carried out ( FIGS. 16, 17 ).  
         [0095]     In a step T 60  of  FIG. 16 , the processing unit  37  registers all kinds of Exif tags ( FIG. 9 ) registered in the image management DB  34  in the Exif tag selection box  3   a . In a step S 61 , Exif tag contents display processing ( FIG. 18 ) is executed.  
         [0096]     In the case of Yes in a step P 15  of  FIG. 18 , i.e., when an Exif tag selected by the user is a numerical value type, in steps P 16  to P 19 , the Exif tag information area  10   e  is checked to retrieve maximum and minimum values of the designated Exif tag. Then, a range of the maximum and minimum values is divided into a predesignated number of areas. Next, the numbers of images included in the divided ranges are retrieved, and the range in which the number of retrieved images is largest is displayed in a graph, and the process returns.  
         [0097]     In the case of No in the step P 15  of  FIG. 18 , i.e., when an Exif tag selected by the user is a character string type, in steps P 20  to P 21 , a kind of the Exif tag to be displayed from the Exif tag information area  10   e  and a name of the Exif tag are checked from the Exif tag database  13  to read contents  3   b  of the Exif tag displayed in the browse area  3 . Next, the number of images included in the read contents  3   b  of the Exif tag is retrieved. Then, in a step P 19 , an item whose retrieving number of times is largest in the contents  3   b  of the Exif tag is displayed as a maximum value in a graph, and the process returns.  
         [0098]     In the case of Yes in a step T 62  of  FIG. 16 , i.e., when the user clicks the graph of the browse area  3 , in steps T 63  to T 64 , the processing unit  37  searches in the Exif tag information area of the registration record  10  to specify one which an Exif tag value is within a range of the graph. Then, a thumbnail image is extracted from the specified registration record  10 , and displayed in the thumbnail area  5 .  
         [0099]     Next, in the case of Yes in both of steps T 67  and T 68 , i.e., when the graph is dropped on the keyword character string  4   a  of the keyword area  4 , in steps T 69  to T 70 , search is carried out in the Exif tag information area  10   e  of the registration record  20  to specify one in which an Exif tag value is within the range of the graph. Then, a corresponding keyword is registered in the specified registration record  10 , and the process returns.  
         [0100]     On the other hand, in the case of Yes in a step T 71 , i.e., when the thumbnail image is dropped on the keyword character string  4   a  of the keyword area  4 , in a step T 72 , the registration record  10  is retrieved from the image management DB  34 , and one having a name of a relevant image file written in the file name area  10   c  is specified. Then, a corresponding keyword is registered in the keyword area  10   f  of the specified registration record  10 , and the process returns.  
         [0101]     In the case of Yes in a step T 75  of  FIG. 17 , i.e., when the keyword character string  4   a  of the keyword area  4  is clicked, in a step T 76 , keyword retrieval displaying is executed ( FIG. 12 ), and the process returns. This keyword retrieval displaying is similar to the foregoing, and thus detailed description thereof will be omitted.  
         [0102]     Next, in the case of Yes in both of steps T 77  and T 78  of  FIG. 17 , i.e., when the graph is dropped on the keyword deletion button  4   b  of the keyword area  4 , in steps T 79  to T 80 , search is carried out in the Exif tag information area  10   e  of the registration record  10  to specify one in which an Exif tag value is within the range of the graph. Then, keyword deletion processing of a step T 80  ( FIG. 13 ) is executed, and the process returns. This keyword deletion processing is similar to the foregoing, and thus detailed description thereof will be omitted.  
         [0103]     On the other hand, in the case of Yes in the step T 77 , No in the step T 78 , and Yes in the step T 81 , i.e., when the thumbnail image is dropped on the keyword deletion button  4   b  of the keyword area  4 , keyword deletion processing of a step T 82  ( FIG. 13 ) is executed, and the process returns. This keyword deletion processing is similar to the foregoing, and thus detailed description thereof will be omitted.  
         [0104]     According to the embodiment of the invention, the keyword and the image are correlated by dragging and dropping. However, the invention is not limited to this system. An operation may be performed which selects a keyword and an image and correlates them.  
       Second Embodiment  
       [0105]     Next, an image retrieving method according to a second embodiment of the present invention will be described. The second embodiment is different from the first only in a structure of an image retrieval screen. Portions similar to those of the first embodiment are denoted by similar reference numerals, and detailed description thereof will be omitted.  
         [0106]      FIG. 19  shows an image retrieval screen  1  when a folder tab  2   a  is selected. Different from the first embodiment, a folder is displayed together with a thumbnail image in a thumbnail area  5 .  
         [0107]      FIG. 20  shows the image retrieval screen  1  when a calendar tab  2   b  is selected. Different from the first embodiment, a calendar is displayed in the thumbnail area  5 .  
         [0108]      FIG. 21  shows the image retrieval screen  1  when an Exif tag tab  2   c  is selected. Different from the first embodiment, an Exif tag selection button  3   a  and a graph are displayed in the thumbnail area  5 .  
         [0109]     According to the first and second embodiments described above, the operations regarding the keyword registration are unified. That is, the followings can be carried out by the same operation: (1) addition of a keyword to an index image relevant to specific photographic information of designated year, month and day, (2) addition or deletion of a keyword to/from an index image relevant to specified year, month and day correlated with a selected keyword, and (3) addition or deletion of a keyword to/from an index image relevant to specific photographic information correlated with a selected keyword. Thus, it is possible to simplify an operation regarding complicated keyword registration.  
       Third Embodiment  
       [0110]     Next, an image retrieving method according to a third embodiment of the present invention will be described. According to the image retrieving method of the third embodiment, a keyword is correlated with an image based on an Exif tag which is one piece of photographic information given when the image is photographed.  
         [0111]      FIG. 22  illustrates a principle of image classification and retrieval. To simplify explanation, a two-dimensional coordinate is taken as an example.  
         [0112]     X and Y axes of  FIG. 22  represent pieces of photographic information, e.g., a shutter speed and a photographing date. Dots (shown) indicate photographed images. It can be imagined from the drawing that the photographed images can be classified into groups based on the pieces of photographic information. Thus, a desired image can be retrieved by determining a group to which an image to be retrieved belongs and extracting an image belonging to the relevant group.  
         [0113]     In an electronic camera, an Exif form is used as an image recording file form as described above. In the image file of the Exif form, photographic image is given as Exif tag information during photographing.  FIG. 8  shows an example of an Exif tag. In other words, it is supposed that such information is implicitly added as a keyword to the image without any awareness of the user.  
         [0114]     According to the present invention, based on the aforementioned consideration, image retrieval processing is carried out by treating image information as an implicit keyword and executing mathematical processing.  
         [0115]      FIG. 23  shows a configuration of an image retrieving apparatus  110  to which the image retracing method of the embodiment of the invention is applied.  
         [0116]     The image retrieving apparatus  110  comprises a central processing unit  111 , a display unit  112 , and an input/output unit  113 . The central processing unit  111  executes image classification/management/retrieval. The display unit  112  is a CRT or a TFT liquid crystal display for displaying a screen or the like regarding image retrieval. The input/output unit  113  comprises an input device such as a keyboard or a mouse for receiving an operation instruction input from a user, and a communication interface for transferring information such as image data with an external device.  
         [0117]     The central processing unit  111  comprises a processing unit  115  for executing image retrieval or the like, and a storing unit  116  for storing information regarding the image retrieval.  
         [0118]     The processing unit  115  comprises a photographic information extraction unit  120 , a distribution/range deciding unit  121 , an image retrieval unit  122 , a keyword selection unit  123 , and a keyword retrieval unit  124 . The photographic information extraction unit  120  extracts image information from a specified image file. The distribution/range deciding unit  121  generates image distribution information by using photographic information as a parameter, and decides a range belonging to a specific group from the distribution information. The keyword selection unit  123  selects a specific keyword from a list of keywords. The keyword retrieval unit  124  specifies an image file having the specific keyword. The image retrieval unit  123  retrieves a specific image file.  
         [0119]     The storing unit  116  comprises an image storing unit  127 , a keyword storing unit  128 , and a management data storing unit  129 . The image storing unit  127  stores an image file. The keyword storing unit  128  stores a keyword and a distribution range by correlating them. The management data storing unit  129  stores a keyword and photographic information by correlating them.  
         [0120]     Next, a process of image retrieval according to the embodiment of the invention will be described.  
         [0121]     The user executes a registration process in the keyword storing unit  128  as a preparation for image retrieval. In this process, a keyword representing a situation properly set by the user (referred to as “situation keyword”, hereinafter) and information regarding a relevant image file are stored in the keyword storing unit  128 .  
         [0122]      FIG. 24  is a flowchart showing a rough registration process in the keyword string unit  128 .  
         [0123]     In a step S 101 , the user inputs a situation keyword by operating the input/output unit  113 . The situation keyword is a word representing a situation, for example, when an image is photographed, and it may be “indoor”, “evening glow”, “night view” or the like.  
         [0124]     In a step S 102 , the user selects an image suitable for addition of the situation keyword among image files saved in the image storing unit  127 . Hereinafter, the selected image will be referred to as a sample image. In the case of Yes in a step S 103 , i.e., when sample images are dually selected, in a step S 104 , an error message is output to the display unit  112 , and the process waits for user&#39;s next selection of an image.  
         [0125]     In the case of No in the steps S 103 , i.e., when sample images are not dually selected, in a step S 105 , the input situation keyword and the sample image are correlated. Then, in the case of No in a step S 106 , i.e., when another sample image is selected, the process from the step S 102  is repeated. It is to be noted that the proper number of sample images may be selected by user&#39;s intention.  
         [0126]     In the case of Yes in the step S 106 , i.e., when the selection is finished, the photographic information extraction unit  120  extracts Exif information from the image file of the sample image. Then, the distribution/range deciding unit  21  executes similarity analysis processing ( FIG. 25 ) shown in a step S 108 .  
         [0127]     The similarity analysis processing calculates Mahalanobis&#39; distance based on the Exif information of the sample image, and decides a range of similarities to the sample image. Specifically, when white dots in the center of  FIG. 22  represent sample images, a range containing data of the white dots (“sample data”, hereinafter), i.e., an elliptical range in  FIG. 22 , is decided by an arithmetic operation. Hereinafter, Mahalanobis&#39; distance will be described.  
         [0128]     In view of a general case, there are assumed to be p variables considered as causes. In this case, Mahalanobis&#39; generalized distance D 2  is represented by the following equation (1): 
 
 D   2 =( x− μ)′Σ −1 ( x− μ)  (1) 
 
         [0129]     In the equation (1), (x−μ) indicates a difference between data x 1 , x 2 , . . . of variables 1, 2, . . . and population mean values μ 1 , μ 2 , . . . of the variables in a matrix, and represented by the following equation (2) when expressed in factors:  
               (     x   -   μ     )     =     (             x   1     -     μ   1                   x   2     -     μ   2               …               x   P     -     μ   P             )             (   2   )             
 
         [0130]     In the equation (1), (x−μ)′ is represented by a matrix obtained by replacing a row and a column of (x−μ) with each other, i.e., a transposed matrix, and represented by the following equation (3): 
 
( x− μ)=( x   1 −μ 1    x   2 −μ 2    . . . x   P −μ P )  (3) 
 
 In the equation (1), Σ is a variance-covariance matrix, and represented by the following equation (4)  
             ∑     =     (           σ   1   2           σ   12         …         σ     1   ⁢   p                 σ   21           σ   2   2         …         σ     2   ⁢   p               …       …       …       …             σ   p1           σ   p2         …         σ   p   2           )               (   4   )             
 
         [0131]     Here, σ i   2  (I=1, 2, . . . p) which is a diagonal factor of the variance-covariance matrix is a variance of a variable i, and σ ij (I, j)=1, 2, . . . , i and j being different) indicates a covariance between the variables i and j. Σ −1  is an inverse matrix of Σ.  
         [0132]     A similarity analysis process shown in  FIG. 25  is for realizing the aforementioned arithmetic operation.  
         [0133]     In a step T 101  of  FIG. 25 , an average value of sample data, i.e., a population mean value (μ), is calculated. In a step T 102 , a variance (σ i   2 ) of the sample data is calculated. In a step T 103 , a covariance (σ ij ) of the sample data is calculated. In a step T 104 , a variance-covariance matrix (Σ) of the sample data is calculated. In a step T 105 , an inverse matrix (Σ −1 ) of the variance-covariance matrix of the sample data is calculated. Then, in a step T 106 , Mahalanobis&#39; distance (D 2 ) is calculated based on the average value and the inverse matrix of the variance-covariance matrix, and the process returns.  
         [0134]     Referring back to  FIG. 24 , in a step S 109 , a distribution range is obtained. Here, a distribution range of one variable is a range of an X axis which is one axis corresponding to an elliptical range shown in  FIG. 26 . Then, this distribution range is represented by making “center point” and “width” a set. For the “center point”, a population mean value μ of the variable is used. For the “width”, a width calculated for the variable based on the Mahalanobis&#39; distance (D 2 ) is used.  
         [0135]     Then, the obtained distribution range and the situation keyword are registered in the keyword storing unit  128 .  FIG. 27  shows a structure of the keyword storing unit  128 . In the keyword storing unit  128 , a distribution range of each photographic information (each Exif information) is recorded for each situation keyword input by the user. According to data of the keyword storing unit  128 , for example, it can be understood that Exif information X 2  of an image having a situation keyword “evening glow” is distributed in a range of a center point  170  and a width  64 . This distribution range indicates a similarity range of the situation keyword.  
         [0136]     After the user finishes the registration of the situation keyword in the keyword storing unit  128 , a registration process in the management data storing unit  129  is executed based on this registration information. This process allocates situation keywords to all the images stored in the image storing unit  127 .  
         [0137]      FIG. 28  is a flowchart showing a rough registration process in the management data storing unit  129 .  
         [0138]     In a step S 111 , a loop of situation keywords is started. That is, the following processing is repeated for each of all the situation keywords. In a step S 112 , a loop of all images is started. That is, the following processing is repeated for each of all the images.  
         [0139]     In a step S 113 , search is executed in the keyword storing unit  128  to take out a similarity range of photographic information of one selected situation keyword. Next, in a step S 114 , photographic image saved in an image file of an image to be processed is taken out.  
         [0140]     In a step S 115 , checking is made as to whether the image to be processed is within the similarity range of the situation keyword or not. In other words, checking is made as to whether each photographic information of the image to be processed belongs to the distribution range (similarity range) of the photographic information stored in the keyword storing unit  128  or not. In the case of Yes in a step S 115 , i.e., when each photographic information of the image to be processed belongs to the similarity range of all pieces of target photographic information, the image and the situation keyword are correlated to be temporarily stored.  
         [0141]     In a step S 117 , if the similarity determination processing is repeated for all the images, the loop of all the images is finished. Then, in a step S 118 , the image and the situation keyword that have temporarily been stored are stored in the management data storing unit  129 .  FIG. 29  shows a structure of the management data storing unit  129 . In the management data storing unit  129 , “image unique ID” which is information for specifying the image, “image reference address” indicating a position in the image storing unit  127  in which the image has been stored, “index image” which is a thumbnail image of the image, and at least one situation keyword are stored.  
         [0142]     Then, when the aforementioned processing is executed for all the situation keywords stored in the keyword storing unit  128 , in a step S 119 , the loop of the situation keywords is finished.  
         [0143]     The registration process in the management data storing unit  129  can be executed for a newly input image. For example, through the input/output unit  113 , an image read from the camera or the like can be additionally registered in the management data storing unit  129 .  
         [0144]      FIG. 30  is a flowchart showing a rough additional registration process in the management data storing unit  129 .  
         [0145]     In a step S 121 , an image input loop is started. That is, the following processing is executed for each image read through the input/output unit  113 . In a step S 122 , a loop of situation keywords is started. That is, the following processing is repeated for each of all the situation keywords.  
         [0146]     In a step S 123 , photographic information saved in an image file of an input image to be processed is taken out. Next, in a step S 124 , search is executed in the keyword storing unit  128  to take out a similarity range of photographic information corresponding to one selected situation keyword.  
         [0147]     In a step S 125 , checking is made as to whether the image to be processed is within the similarity range of the situation keyword or not. In other words, checking is made as to whether each photographic information of the image to be processed belongs to the distribution range (similarity range) of the photographic information stored in the keyword storing unit  128  or not. In the case of Yes in a step S 125 , i.e., when each photographic information of the image to be processed belongs to the similarity range of all pieces of target photographic information, in a step S 126 , the situation keyword is stored for the input image in the management data storing unit  129 .  
         [0148]     Then, when the aforementioned processing is executed for all the situation keywords stored in the keyword storing unit  128 , in a step S 127 , the loop of the situation keywords is finished. In a step S 128 , when the similarity determination processing is executed for all the read images, the image input loop is finished.  
         [0149]     After the end of the registration operations in the keyword storing unit  128  and the management data storing unit  129 , an image can be retrieved. According to the image retrieving method of the embodiment, a method of retrieving an image by designating a keyword, and a method of designating an image and retrieving an image similar to the image can be used.  
         [0150]      FIG. 31  is a flowchart showing a rough image retrieving process by keywords.  
         [0151]     When the user executes an image retrieving operation by keywords through the input/output unit  113 , in a step S 131 , a list of registered situation keywords is displayed in the display unit  112 . When the user selects the displayed keywords, in a step S 132 , search is executed in the management data storing unit  129  to extract all images having the selected situation keywords. Then, in a step S 133 , thumbnail images of the extracted images are displayed as a retrieving result list.  
         [0152]      FIG. 32  is a flowchart showing a rough image retrieving process based on images.  
         [0153]     In a step S 136 , when the user selects an image file from the image storing unit  127 , and instructs retrieval of images similar to the image, in a step S 137 , situation keywords correlated with the selected image are all extracted from the management data storing unit  129 .  
         [0154]     Next, search is executed in the management data storing unit  129  to extract all images having the selected situation keywords. Then, in a step S 139 , thumbnail images of the extracted images are displayed as a retrieving result list.  
         [0155]     According to the image retrieving method of the third embodiment, each image is displayed in a feature space based on image information of a photographing time which accompanies the image, e.g., Exif information, and a distribution range of a preselected sample image is decided by using Mahalanobis&#39; distance. Then, the distribution range of the preselected sample image is registered together with its situation keyword in the keyword storing unit. Next, determination is made as to whether each Exif information of all the images to be managed is within this distribution range or not. Then, the situation keyword is registered in the management data storing unit of the image determined to be within the range.  
         [0156]     The list of the situation keywords registered in the keyword storing unit is displayed. Search is executed in the management data storing unit by using the selected situation keywords as keys, and the list of retrieving results is displayed.  
         [0157]     An image file is selected from the image storing unit, situation keywords registered in the management data storing unit of the selected image are extracted. By using these situation keys as keys, search is executed in the management data storing unit, and the list of retrieving results is displayed.  
         [0158]     Furthermore, when Exif information input at the time of inputting image data from the camera or the like is extracted, and determination is made as to whether it is within the distribution range of each keyword of the keyword storing unit or not, the situation keyword is registered in the management data storing unit.  
         [0159]     With the foregoing configuration, images accurately photographed in a photographing situation desired by a retriever including an image to which no keyword is added or with which no keyword is correlated based on human&#39;s subjective view of image data itself can be extracted as retrieving results.  
         [0160]     A situation keyword indicating a photographing situation of each image is accurately added in the management data storing unit of the image input into the apparatus by the image input means without any addition by a human. Thus, it is not necessary for the human to add any new keywords.  
         [0161]     Accordingly, it is possible to carry out retrieval more easily understood by the retriever than retrieval of a value of photographic information contained in image data directly as retrieving conditions.  
         [0162]     According to the embodiment, the distribution range of the preselected sample image is calculated by using the Mahalanobis&#39; distance. However, the method of calculating a distribution range is not limited to the Mahalanobis&#39; distance. A distribution range may be decided based on each index defined based on a statistical method, e.g., an average value, a variance, maximum and minimum values or the like.  
         [0163]     Each function of the embodiment may be configured by using hardware, or realized by causing a computer to read a program which describes each function by using software. Additionally, each function may be configured by properly selecting software or hardware.  
         [0164]     Furthermore, each function can be realized by causing a computer to read a program stored in a recording medium (not shown). Here, the recording medium of the embodiment can employ any recording form as long as a program can be recorded and read by the computer.  
         [0165]     Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general invention concept as defined by the appended claims and their equivalents.

Technology Classification (CPC): 6