Abstract:
A system, apparatus and method for automatically annotating digital images. An electronic capture device captures a digital representation of a scene. An apparatus automatically annotates the digital images with the physical and cultural features that may be included in the image based upon the location of the electronic capture device, and its orientation.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    1. Field of the Invention  
           [0002]    This invention relates to the field of photograph annotation, and more specifically, to a system, method, and apparatus for automatically annotating digital photographs with the physical and cultural features that may be included in a image based upon the location of the camera, its orientation, as well as parameters internal to the camera.  
           [0003]    2. Discussion of the Related Art  
           [0004]    There are digital cameras and image-capturing personal digital assistants (“PDAs”) that allow a user to add annotations to photographs. For example, some digital cameras allow a user to type in an annotation/caption for a photograph immediately after the photograph has been taken. However, when a user takes many pictures, it is burdensome for the user to manually write/type a caption for each photograph.  
           [0005]    Some digital cameras utilize a global positioning system (“GPS”). GPS can be used to determine the latitude and longitude coordinates of the location where a photograph was taken. Some cameras can annotate photographs with GPS coordinates. Additionally, it is possible to associate a photograph&#39;s time stamp with that of a GPS log to deduce the location where a given image was taken even if the camera lacks a built-in GPS device. However, merely annotating an image with the photographer&#39;s location is not sufficient to determine what the image might contain. Furthermore, photographs captured at the same location, may contain significantly different subjects if the camera is oriented in different directions. What is needed is an automatic mechanism for annotating digital images with the very features recorded within the camera&#39;s field of view (FOV), for example, the names of physical (e.g., mountains, rivers, etc.) and cultural (e.g., buildings, bridges) features that may be contained in the images.  
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0006]    [0006]FIG. 1 illustrates a method of calculating the field of view according to an embodiment of the invention;  
         [0007]    [0007]FIG. 2 illustrates a user taking a photograph of a scenic site according to an embodiment of the invention;  
         [0008]    [0008]FIG. 3 illustrates a process of a user taking photographs at a scenic location and then saving photos, GPS data, and digital compass data in a camera memory according to an embodiment of the invention;  
         [0009]    [0009]FIG. 4 illustrates a general overview of digital photographs being transferred from a camera and acquiring annotations according to an embodiment of the invention;  
         [0010]    [0010]FIG. 5 illustrates a method by which digital photographs receive annotations according to an embodiment of the invention; and  
         [0011]    [0011]FIG. 6 illustrates a digital photograph to which annotations have been added according to an embodiment of the invention.  
     
    
     DETAILED DESCRIPTION  
       [0012]    An embodiment of the present invention describes a system for automatically annotating photographs taken by an electronic capture device such as a digital camera. Further, the present invention concerns a method for automatically annotating digital images with the names of physical and cultural features that may be included in the image. Embodiments of the present invention relate to a digital camera (still or dual mode) that annotates each image with photographic information, such as the lens manufacturer, the focal length of the lens, and the focal distance to the subject. This information is encompassed by the EXIF digital imaging standard (Exchangeable image file format for Digital Still Cameras: Version 2.1 Jun. 12, 1998 Japan Electronic Industry Development Association (JEIDA)) and supported by a majority of emerging middle to high-end digital cameras. The EXIF picture format also provides a way to store the latitude and longitude global position coordinates of the location where a picture was taken. The camera&#39;s orientation is also included in the EXIF specification, therefore, cameras with built-in compasses may have that information written into the EXIF JPEG (Joint Photographic Experts Group: ISO/IEC JTC1 SC29 Working Group 1) files the camera creates.  
         [0013]    [0013]FIG. 1A illustrates a digital camera  110  and lens  130  that may be augmented with both, a GPS unit  120  for determining the global position, and a digital compass  121  for detecting the orientation or bearing of the camera  110  at the time a photo is taken. The digital camera  110  may include an operating system with programmable software allowing the camera  110  to be scripted to send commands to the GPS unit  120  and digital compass  121  through the digital camera&#39;s  110  serial port and to embed received data into JPEG images. In this way, the camera&#39;s origin and orientation are recorded when pictures are taken.  
         [0014]    The images, stored in JPEG format, that includes the EXIF standard, may be uploaded to a computer such as a PC, Macintosh, or Unix workstation in a variety of ways, e.g., through the computer&#39;s universal serial bus (USB) port. Once uploaded to a computer, application software in the computer can determine each image&#39;s field of view and further annotate the image file with that computed information.  
         [0015]    Referring to FIG. 1B, the field of view (FOV) of an image can be determined given the lens focal length  133  and lens  130  manufacturer, and the focal distance  140  to the subject, using application software running on the computer. Both of these pieces of information, the lens focal length  133  and lens  130  manufacturer, as well as the focal distance  140  to the subject, are part of the EXIF standard and are supported by various models of digital cameras.  
         [0016]    Mathematical models that map between the 2-dimensional retinal plane of a digital camera and the 3-dimensional physical world are well known in the art. Such models are used by embodiments of the present invention to calculate the location of the edges of a frustum defined by the field of view of the image (line segments AB, AC, and BC below) given (1) the location of the camera  100  within a global coordinate system (e.g., latitude and longitude), and (2) the orientation/bearing of the camera (0-360 degrees).  
         [0017]    The field of view, often referred to as the angle of view, may be calculated using the following equation (referring to FIG. 1B):  
         field of view:  W= 2 tan −1   Y′/ 2  f    
         [0018]    Embodiments of the present invention determine the geographical coordinates and the image&#39;s field of view, which may then be used to query a geographical database such as the Geographic Names Information System (GNIF), via the Internet, for the names of physical and cultural features contained within that view.  
         [0019]    The GNIF, developed by the United States Geological Survey (USGS) in cooperation with the U.S. Board on Geographic Names (BGN), contains information about almost 2 million physical and cultural features in the United States. The database contains the federally recognized name of each included feature as well as the feature&#39;s location by state, county, and geographic coordinates. The GNIS is the nation&#39;s official repository of domestic geographic names information. Similar repositories exist for other countries.  
         [0020]    [0020]FIG. 2 illustrates a user  200  taking a photograph of a scenic place  220  with an electronic capture device such as a digital camera  110  according to an embodiment of the present invention. In the preferred embodiment, the GPS unit  120  and digital compass  121  are attached to the digital camera  110 . In other embodiments, the GPS unit  120 , digital compass  121 , and the camera  110  may be separate devices. In other embodiments, a digital camera  110  for taking still photographs need not be used. For example, a video camera, or any other suitable device may be used.  
         [0021]    [0021]FIG. 3 illustrates a process of a user  200  taking photographs according to an embodiment of the present invention. First, the user  200  brings  300  a camera  110  to a scenic place  220 . Next, the user  200  takes a photograph  305  with the camera  110 . The camera  110  saves  310  the photograph in a memory, e.g., RAM, flash, HD, CD, DVD, etc. The user  200  may then take  315  another photograph, if desired. The user  200  may continue taking photographs until the user  200  desires to move to a new location. If additional photographs are taken, the same process is repeated, with respect to the taking and saving of photographs.  
         [0022]    When the user  200  has finished taking photographs of the scene  220 , the camera  110  saves  320  the GPS  120  and digital compass  121  data measured at the time that the photographs were taken in a memory in the camera  110 .  
         [0023]    [0023]FIG. 4 illustrates a general overview of digital photographs being transferred from the camera  110  and acquiring annotations according to an embodiment of the present invention. As shown, the camera  110  is connected to a computing and communication device such as a computer  400 . A program executed by the computer  400  may be used to determine which photographs were taken near which scenic area.  
         [0024]    The computer  400  may contact a geographical database such as the Geographic Names Information System (GNIS), via the Internet. In an embodiment of the present invention, the information stored (GPS and compass data) in the digital camera  110  is used to determine the four endpoints of the field of view of each photograph in terms of a global coordinate system, such as latitude-longitude. That information, i.e., the four latitude-longitude pairs, is then used to query the GNIS for all physical and cultural features found within those coordinates. The GNIS  405  then returns all names of physical and cultural features located within that region for a given photo. The application program can then record within the image file the names of the returned features using fields defined in the EXIF standard.  
         [0025]    [0025]FIG. 5 illustrates a process by which digital photographs receive annotations according to an embodiment of the present invention. First, the user  200  uploads  500  the photograph information to the computer  400  (as discussed above with respect to FIG. 4). Next, the program associates  505  photographs with geographical locations. The program, or another program, contacts  510  the GNIS database. The database GNIS supplies to the computer  515  names of physical and cultural features located within the region for a given photo. The program then annotates  520  the photographs with names of physical and cultural features located within the region for a given photo.  
         [0026]    [0026]FIG. 6 illustrates a digital photograph to which annotations have been added according to an embodiment of the invention. First, the user  200  takes a photograph  600  of a house, for example. The communication device, i.e. computer  400 , contacts the GNIS and receives annotation information. The computer annotates the photograph.  
         [0027]    The annotated photograph  610  may read:  
         [0028]    John Smith&#39;s colonial style house. 1234 West Kissel Boulevard Springfield, Mass. 12345  
         [0029]    While the description above refers to particular embodiments of the present invention, it will be understood that many modifications may be made without departing from the spirit thereof. The accompanying claims are intended to cover such modifications as would fall within the true scope and spirit of the present invention. The presently disclosed embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims, rather than the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.