Patent Publication Number: US-9412035-B2

Title: Place-based image organization

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of U.S. patent application Ser. No. 12/963,527, filed on Dec. 8, 2010, and titled “PLACE-BASED IMAGE ORGANIZATION” the entire disclosure of which is hereby incorporated herein by reference for all purposes. 
    
    
     BACKGROUND 
     The modern consumer is able to capture images with an increasingly broad array of devices. As a result the consumer carries out a very time-consuming and labor-intensive process of organizing images captured by different devices into a digital image library. Geo-tagging facilitates image organization by matching an image to a geographic position at the time of image creation, but few devices offer this technology. Moreover, digital photography has been in use prior to the maturity geo-tagging technology, meaning a multitude of existing images cannot leverage geo-tagging or other location services. 
     SUMMARY 
     Systems and methods for image organization are provided. A computing device in accordance with embodiments of the present disclosure includes a processor configured to receive an image set comprising a plurality of images, compare a portion of each image in the image set with a verified cue library including a plurality of visual cues that correspond to different geographic place tags, and display a recommendation panel including a recommended place tag corresponding to a visual cue that matches the portion. 
     This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a schematic diagram of an embodiment of a computing system of the present disclosure. 
         FIG. 2  is view of an embodiment of an image organization graphical user interface of the present disclosure. 
         FIG. 3  is another view of the image organization graphical user interface of  FIG. 2 . 
         FIG. 4  is another view of the image organization graphical user interface of  FIG. 2 . 
         FIG. 5  is a flowchart of an embodiment of an image organization method performed at user computing device. 
         FIG. 6  is a flowchart of an embodiment of an image organization method performed at server computing device. 
     
    
    
     DETAILED DESCRIPTION 
     The present description is directed to systems and methods for recommending and assigning place tags to images in order to facilitate organization of the images through indexing by place. More particularly, the present description is directed to recommending place tags for images based on a comparison of the images with different verified cues that correspond to the place tags. By recommending place tags for images based on matching different verified cues, images may be assigned place tags even after image creation. Accordingly, legacy images that were not assigned a place tag at creation may be assigned a place tag, so that the legacy images may be indexed by place. Moreover, images may be place-tagged without utilizing global position system (GPS) enabled cameras and the like. 
     Furthermore, the recommendation function may be applied to a single image or an image set including a plurality of images so that each image in the image set does not have to be individually place-tagged. By recommending place tags for an entire set of images, manual tagging of single images can be reduced or eliminated. Accordingly, image organization via place indexing may be performed quickly and easily in an automated or semi-automated manner, as desired. 
       FIG. 1  schematically shows a nonlimiting embodiment of a computing system  100  in which methods or processes for recommending and assigning place tags to images may be carried out. The computing system  100  is shown in simplified form. It is to be understood that virtually any computer architecture may be used without departing from the scope of this disclosure. The computing system  100  may include a user computing device  102  that communicates with a server computing device or computing device cloud  146  through a network  144 . 
     In different embodiments, the user computing device  102  may take the form of a desktop computer, laptop computer, tablet computer, home entertainment computer, network computing device, mobile computing device, mobile communication device, gaming device, etc. Additionally, the computing device cloud  146  may include a plurality of different sources that communication with each other through network  144 . The different sources of the computing device cloud  146  may take the form of a variety of different computing devices including a mainframe computer, server computer, desktop computer, laptop computer, tablet computer, home entertainment computer, network computing device, mobile computing device, mobile communication device, gaming device, etc. 
     In different embodiments, the network  144  may take the form of a local area network (LAN), wide area network (WAN), wired network, wireless network, personal area network, or a combination thereof including the Internet. 
     The user computing device  102  includes a logic subsystem  104  and a data-holding subsystem  106 , and a display subsystem  140  and/or other components not shown in  FIG. 1 , such as user input devices including keyboards, mice, game controllers, cameras, microphones, and/or touch screens, for example. 
     The logic subsystem  104  may include one or more physical devices configured to execute one or more instructions. For example, the logic subsystem  104  may be configured to execute one or more instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more devices, or otherwise arrive at a desired result. 
     The logic subsystem  104  may include one or more processors that are configured to execute software instructions. Additionally or alternatively, the logic subsystem  104  may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic subsystem  104  may be single core or multicore, and the programs executed thereon may be configured for parallel or distributed processing. The logic subsystem  104  may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and/or configured for coordinated processing. One or more aspects of the logic subsystem  104  may be virtualized and executed by remotely accessible networked computing devices such as a logic subsystem  148  included in computing device cloud  146 . 
     The data-holding subsystem  106  may include one or more physical, non-transitory, devices configured to hold data and/or instructions executable by the logic subsystem to implement the herein described methods and processes. When such methods and processes are implemented, the state of data-holding subsystem  106  may be transformed (e.g., to hold different data). 
     The data-holding subsystem  106  may include removable media and/or built-in devices. The data-holding subsystem  106  may include optical memory devices (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory devices (e.g., RAM, EPROM, EEPROM, etc.) and/or magnetic memory devices (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), among others. The data-holding subsystem  106  may include removable computer-readable storage media, which may be used to store and/or transfer data and/or instructions executable to implement the herein described methods and processes. The removable computer-readable storage media may take the form of CDs, DVDs, HD-DVDs, Blu-Ray Discs, EEPROMs, and/or floppy disks, among others. The data-holding subsystem  106  may include devices with one or more of the following characteristics: volatile, nonvolatile, dynamic, static, read/write, read-only, random access, sequential access, location addressable, file addressable, and content addressable. In some embodiments, the logic subsystem  104  and data-holding subsystem  106  may be integrated into one or more common devices, such as an application specific integrated circuit or a system on a chip. 
     It is to be appreciated that data-holding subsystem  106  includes one or more physical, non-transitory devices. In contrast, in some embodiments aspects of the instructions described herein may be propagated in a transitory fashion by a pure signal (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for at least a finite duration. Furthermore, data and/or other forms of information pertaining to the present disclosure may be propagated by a pure signal. 
     The terms “module,” “program,” and “engine” may be used to describe an aspect of the computing system  100  that is implemented to perform one or more particular functions. In some cases, such a module, program, or engine may be instantiated by the user computing device  102  via logic subsystem  104  executing instructions held by data-holding subsystem  106 . In some cases, such a module, program, or engine may be instantiated by the server computing device/cloud  146  via logic subsystem  148  executing instructions held by data-store  150 . It is to be understood that different modules, programs, and/or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” are meant to encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc. 
     Note components of the user computing device  102  that may be substantially the same as those of the server computing device/cloud  146  are described no further. 
     A plurality of images may be collected to form an image library  108 . In some embodiments, the image library  108  may be stored in the data-holding subsystem  106  of the user computing device  102 . In some embodiments, the image library  108  may be stored in a data store  150  of the server computing device/cloud  146 . In some embodiments, the image library  108  may include a local component that includes images  110  stored in the data-holding subsystem  106  of the user computing device  102  and a remote component that includes images  114  stored in the data store  150  or other data-holding devices dispersed throughout the computing device cloud  146 . 
     An image organization program  118  may be configured to organize the image library  108  according to different indices. More particularly, the image organization program  118  may be configured to recommend place tags to be assigned to images in the image library  108  so that the image library  108  may be indexed by place. A place tag may include metadata that identifies a geographic location, such as a place name and/or latitude and longitude coordinates. In some cases, an image in the image library may be assigned more than one place tag. For example, an image of the Eiffel Tower may be assigned an “Eiffel Tower” place tag, a “Paris” place tag and a “France” place tag. 
     In some embodiments, the data-holding subsystem  106  may hold instructions constituting the image organization program  118  that are executed locally on the user computing device  102  by logic subsystem  104 . Correspondingly, in some cases, place tags  112  existing locally on the user computing device may be tagged to the local images  110  by the image organization program  118 . In some embodiments, the data-store  152  may hold instructions constituting the image organization program  118  that are executed remotely in the computing device cloud  146  by the logic subsystem  148 . Correspondingly, in some cases, place tags  116  aggregated from different sources in the computing device cloud  146  may be tagged to remote images  114  by the image organization program  118 . In different embodiments, the image organization program  118  may take the form of, or be incorporated into, an executable application, browser plug-in, HTML5 file, mobile application, desktop application, etc. 
     In a local implementation of the image organization program  118  that is executed on the user computing device  102 , the image organization program  118  may include a place tag module  120  that may be configured to recommend place tags to be assigned to a selected set of local images  110  in the image library  108 . The place tag module  120  may include a verified cue library  122  of cues that correspond to different place tags  112 . The verified cue library  122  may be populated with local existing cues identified from local images  110  stored in data-holding subsystem  106 . In one example, the verified cue library  122  takes the form of a relational database in which different cues are linked to different place tags through different relationships. 
     The verified cue library  122  may include visual cues  128 , place name cues  126 , and supplemental cues  124 . The visual cues  128  may include visual patterns or signatures that define a geographic place that corresponds to a place tag. Nonlimiting examples of visual cues  128  may include light temperature, shadows, object shape, language, etc. 
     The place name cues  126  may include various geographic locations that may be compared to a file name of the image  110  in order to recommend a corresponding place tag. Nonlimiting examples of place name cues  126  may include landmarks, neighborhoods, cities, states countries, etc. 
     The supplemental cues  124  may include information extracted from other programs  138  executed on the user computing device  102  that may link metadata of an image to a geographic place tag. As one example, a supplemental cue may include information extracted from an entry into a schedule or calendar program that provides a location of the user computing device  102  at a particular time that corresponds to a geographic place tag. As another example, a supplemental cue may include information extracted from a network connection program. In particular, an internet protocol (IP) address may be used to identify a location of the user computing device  102  at a particular time that corresponds to a geographic place tag. 
     As yet another example, in an embodiment where the user computing device  102  is a phone, a supplemental cue may include information extracted from placing a phone call. In particular, the position of the phone may be provided by the location of a cellular tower that relays the call. In the above examples, the place tag module  120  may compare a time stamp of an image with a time provided by the supplemental cue and may recommend a geographic place tag that corresponds to the supplemental cue. It will be appreciated that supplemental cues linked to image metadata other than time stamps may be extracted from the other programs  138  executed by the user computing device  102 . 
     The image organization program  118  may include a visual cue analyzer  130  configured to analyze images  110  to extract portions of each image to compare to the visual cues  128  in the verified cue library  122 . The place tag module  120  may recommend place tags  112  that correspond to visual cues  128  in the verified cue library  122  that match portions of the images  110  provided by the visual cue analyzer  130 . 
     The image organization program  118  may include a file analyzer  132  configured to extract metadata including file names from images  110  to compare to the place name cues  126  in the verified cue library  122 . The place tag module  120  may recommend place tags  112  that correspond to place name cues  126  in the verified cue library  122  that match the metadata including the file name provided by the file analyzer  132 . 
     The image organization program  118  may include a device analyzer  134  configured to extract information from other programs  138  executed by the user computing device  102  that may be used to match image metadata with geographic place tags. The place tag module  120  may recommend place tags  112  that correspond to supplemental cues  124  in the verified cue library  122  that match the information provided by the device analyzer  134 . 
     In some cases, a user may assign a user-provided place tag  142  to a selected image in the image library  108 . The place tag module  120  may be configured to compare cues in the verified cue library  122  that correspond to the user-provided place tag  142  with images in the image library  108 . The place tag module  120  may provide a recommendation that the user-provided place tag  142  be assigned to one or more matching images in the image library  108 . 
     The image organization program  118  may be configured to display an image organization graphical user interface (GUI)  136  on display subsystem  140 . The image organization GUI  136  may be configured to display recommended place tags to a user. Further, the image organization GUI  136  may be configured to display images organized by different indices to a user. The image organization GUI  136  will be discussed in further detail below with reference to  FIGS. 2-4 . As the herein described methods and processes change the data held by the data-holding subsystem  106 , and thus transform the state of the data-holding subsystem, the state of display subsystem  140  may likewise be transformed to visually represent changes in the underlying data. Display subsystem  140  may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic subsystem  104  and/or data-holding subsystem  106  in a shared enclosure, or such display devices may be peripheral display devices. 
     Continuing with  FIG. 1 , in a global implementation of the image organization program  118  that is executed by the computing device cloud  146 , the image organization program  118  may include a place tag module  154  that may include a verified cue library  156  of cues that correspond to different place tags  116 . The verified cue library  156  may be populated with verified cues identified from images  114  aggregated from different sources (e.g., computing devices) of the computing device cloud and stored in data-store  150 . In one example, the verified cue library  156  takes the form of a relational database in which different cues are linked to different place tags through different relationships. The verified cue library  156  may include visual cues  160  and place name cues  158 . 
     The verified cue library  156  may be repeatedly updated with different verified cues that correspond to place tags that are aggregated from different sources of the computing device cloud  146 . For example, the image organization program  118  may receive a new cue  166  from a source of the computing device cloud  146  that is added to the verified cue library  156 . When the new cue  166  is added the verified cue library aggregated information may be increased to make the verified cue library  156  more comprehensive. In particular, since there are more cues, place tags, and linking relationships in the global implementation of the verified cue library  156 , recommendations may be more accurate and the likelihood of matching a place tag to an image may be increased. 
     The image organization program  118  may include a visual cue analyzer  162 , a file analyzer  164 , and an image organization GUI  168 . These components may be substantially the same as those of the local implementation and are described no further. 
     In some embodiments, the global image organization program  118  may recommend place tags for images that are stored globally in the data store  150 . In some embodiments, the global image organization program  118  may recommend place tags for images that are stored locally in the data-holding subsystem  106  of the user computing device  102 . In other words, a selected set of images may be received by the global implementation of the image organization program from a user computing device. The global implementation of the image organization program may analyze the images in the image set, and may send recommended place tags corresponding to cues that match images in the selected image set to the user computing device. 
       FIGS. 2-4  are different views of an embodiment of an image organization GUI  200  of the present disclosure. In one example, the image organization GUI may be displayed on display subsystem  140  by the image organization program  118  as shown in  FIG. 1 . 
     The image organization GUI  200  may include an image display region  202  including a plurality of images  204 . The plurality of images  204  may form part or all of an image library. Each of the plurality of images  204  in the image display region  202  may include metadata that may be used to organize or filter images by different indices. 
     The image organization GUI  200  may include an image organization control region  206  including a plurality of category selectors  208  that may be selectable to organize the plurality of images  204  by different indices. In the illustrated embodiment, the plurality of category selectors  208  includes a people category selector, an albums category selector, and a places category selector as examples; however, it will be appreciated that any suitable category selector may be included in the image organization control region  206 . 
     Each image that includes metadata including an assigned organizational tag, such as a place tag, displayed in the image display region  202  may include a tag icon  212 . The tag icon  212  may indicate to a user that the image is tagged and can be indexed by a selected one of the plurality of category selectors  210 . Each image without an assigned organizational tag, such as a place tag, displayed in the image display region may include a recommendation icon  214  indicating that no place tag is assigned to that image. The recommendation icon  214  may aid a user in identifying and selecting sets of images without place tags, so that place tags can be recommended for and assigned to the entire selected image set. 
     The image organization GUI  200  may include a recommendation panel  216  including one or more place tag selectors  218 . The place tag selectors  218  may be selectable to assign a recommended place tag to selected images in the image display region  202  that match a verified cue that corresponds to the recommended place tag. In the case of visual cues, one or more place tag selectors may be selectable to assign a recommended place tag to selected images that include a portion that matches a visual cue that corresponds to the recommended place tag. In the case of place name cues, one or more place tag selectors may be selectable to assign a recommended place tag to selected images that include a file name that matches a place name cue that corresponds to the recommended place tag. 
     In some embodiments, the recommendation panel  216  may include a ranked list  220  of recommended place tags that correspond to the place tag selectors  218 . The ranked list  220  may be organized by the image organization program  118  based a relationship or correspondence strength between the image and a matching verified cue. 
     The image organization GUI  200  may include a user-provided place tag region  222  configured to receive a user-provided place tag to be assigned to one or more selected images in the image display region  202 . As shown in  FIG. 3 , the image organization program  118  may be configured to, in response to receiving a user-provided place tag  224  entered in the user-provided place tag region  222 , indicate one or more images in the display region  202  that matches a verified cue that corresponds to the user-provided place tag  224 . The images matching cues corresponding to the user-provided place tag  224  are indicated by an indicator  228  (e.g., dashed lines). The indicator  228  may also indicate images in an image set that are selected for place tag recommendations. The indicator  228  may aid a user in verifying which images match a recommended place tag. 
     The image organization program  118  may be configured to, in response to receiving a selection of a recommended place tag selector  218 , display a verification window  226  in the image organization graphical user interface  200 . The verification window  226  may include a verification selector  230  configured to accept a user verification and assign the recommended place tag of selector  218  to the selected images as indentified by indicator  228 . The verification window  226  may include a rejection selector  232  configured to accept a user rejection and deselect the selected images as identified by indicator  228 . 
       FIG. 4  shows another view of the image organization graphical user interface  200 . In this view, the plurality of images  204  in the image display region  202  has been assigned the same place tag as indicated by tag icons  212 . The place category selector  210  in the image organization control region  206  has been selected to organize the plurality of images  204  displayed in the image display region  202  by assigned place tags, and the place tag selector  218  in the recommendation panel  216  has been selected. In this example, by selecting the place tag selector  218  only images that have been assigned the corresponding place tag are displayed in the image display region as indicated by the tag icon  212  shared by each image, and other images are filtered out of the image display region  202 . 
       FIG. 5  is a flowchart of an embodiment of an image organization method  500 . The method  500  may be performed at a user computing device, such as the user computing device  102  shown in  FIG. 1 . At  502 , the method may include receiving an image set comprising a plurality of images. In some cases, one or more of the plurality of images in the image set may not have an assigned place tag. In some cases, one or more of the plurality of images in the image set may have one or more assigned place tags. 
     In some cases, at  504 , the method may include receiving a user-provided place tag for a selected image of the image set. The user-provided place tag may act as a guide to facilitate place-tagging of other matching images in the image set. If a user-provided place tag is not provided  504  may be omitted from the method  500 . 
     At  506 , the method may include comparing each image in the image set with a verified cue library. The verified cue library may include a plurality of visual cues that correspond to different geographic place tags. At  508 , the method may include comparing a portion of each image in the image set with the plurality of visual cues. 
     The verified cue library may include a plurality of place name cues that correspond to different geographic place tags. At  510 , the method may include comparing a file name of each image in the image set with the place name cues. The verified cue library may include a plurality of supplemental cues provided from other application executable by the user computing device that correspond to different geographic place tags. At  512 , the method may include comparing a time stamp of each image in the image set with the supplemental cues. If a user-provided place tag for a selected image set is received, at  514 , the method may include comparing a portion of each other image in the image set with visual cues that correspond to the user-provided place tag. If a user-provided place tag is not provided  514  may be omitted from the method  500 . 
     At  516 , the method may include displaying a recommendation panel including a recommended place tag corresponding to a cue that matches an image. For visual cues, a recommended place tag that corresponds to a visual cue that matches a portion of an image may be displayed in the recommendation panel. For place name cues, a recommended place tag that corresponds to a place name cue that matches a file name of an image may be displayed in the recommendation panel. For supplemental cues, a recommended place tag that corresponds to a supplemental cue that matches a time stamp of an image may be displayed in the recommendation panel. 
     At  518 , the method may include displaying a recommendation window including a recommendation that the user-provided place tag be assigned to images having portions that match the visual cues that correspond to the user-provided place tag. If a user-provided place tag is not provided  518  may be omitted from the method  500 . 
     At  520 , the method may include receiving a verification that the recommended place tag matches one or more images. The verification may be provided by a user through selection of a verification selector. 
     At  522 , the method may include assigning the recommended place tag to one or more images of the image set in response to receiving the verification that the recommended place tag matches the one or more images. Assigning the recommended place tag may include writing the place tag to metadata of an image file of one or more images of the image set. 
     By recommending place tags that correspond to cues that match images, images may be assigned place tags without leveraging global position system (GPS) or other device location tracking technology. Accordingly, a user&#39;s image collection can be organized by place, even if the images were not geo-tagged at image creation. 
       FIG. 6  is a flowchart of an embodiment of an image organization method  600 . The method  600  performed at a server computing device, such as the server computing device or cloud  146  shown in  FIG. 1 . At  602 , the method may include receiving, from a user computing device, an image set comprising a plurality of images. In some cases, one or more of the plurality of images in the image set may not have an assigned place tag. In some cases, one or more of the plurality of images in the image set may have one or more assigned place tags. 
     At  604 , the method may include comparing each image in the image set with a verified cue library including a plurality of cues that correspond to different geographic place tags. The plurality of cues may be aggregated from a plurality of different sources in a computing device cloud. The verified cue library may include a plurality of visual cues that correspond to different geographic place tags. At  606 , the method may include comparing a portion of each image in the image set with the plurality of visual cues. The verified cue library may include a plurality of place name cues that correspond to different geographic place tags. At  608 , the method may include comparing a file name of each image in the image set with the place name cues. 
     At  610 , the method may include sending, to the user computing device, a recommended place tag corresponding to a cue that matches an image. For visual cues, a recommended place tag that corresponds to a visual cue that matches a portion of an image may be sent to the user computing device. For place name cues, a recommended place tag that corresponds to a place name cue that matches a file name of an image may be sent to the user computing device. 
     At  612 , the method may include updating the verified cue library with a new cue. The new cue may correspond to a geographic place tag. For example, the cue may include a relational pointer that points to the geographic place tag in the verified cue library. The new cue may take the form of a visual cue or a place name cue. In one example, the new cue may be received from one of the plurality of different sources in the computing device cloud. In another example, the new cue may be received from a user via the user computing device. 
     At  614 , the method may include comparing each image in the image set to the new cue. At  616 , the method may include sending, to the user computing device, the geographic place tag that corresponds to the new cue that matches the images. If the new cue is a visual cue, a geographic place tag that corresponds to the new visual cue that matches a portion of an image may be sent to the user computing device. If the new cue is a place name cue, a geographic place tag that corresponds to the new place name cue that matches a file name of an image may be sent to the user computing device. 
     By updating the verified cue library with cues aggregated from a plurality of different sources, the cue library may become more comprehensive and may provide more accurate place tag recommendations. Moreover, by sending additional place tag recommendations that are updated with the addition of new cues, the likelihood of images being assigned place tags may be increased. 
     In some embodiments, the above described methods and processes may be tied to a computing system including one or more computers. In particular, the methods and processes described herein may be implemented as a computer application, computer service, computer API, computer library, and/or other computer program product. 
     It is to be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated may be performed in the sequence illustrated, in other sequences, in parallel, or in some cases omitted. Likewise, the order of the above-described processes may be changed. 
     The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.