Source: https://patents.google.com/patent/US10303975B2/en
Timestamp: 2020-01-21 17:15:10
Document Index: 720277335

Matched Legal Cases: ['Application No. 2', 'Application No. 2', 'Application No. 10', 'Application No. 10724937', 'Application No. 2012', 'Application No. 10', 'Application No. 201080030849', 'Application No. 201080030849', 'Application No. 201410455635']

US10303975B2 - Landmarks from digital photo collections - Google Patents
US10303975B2
US10303975B2 US15/663,796 US201715663796A US10303975B2 US 10303975 B2 US10303975 B2 US 10303975B2 US 201715663796 A US201715663796 A US 201715663796A US 10303975 B2 US10303975 B2 US 10303975B2
US15/663,796
US20180211134A1 (en
2013-02-05 Priority to US13/759,916 priority patent/US9020247B2/en
2015-04-10 Priority to US14/683,643 priority patent/US9721188B2/en
2017-07-30 Application filed by Google LLC filed Critical Google LLC
2017-07-30 Priority to US15/663,796 priority patent/US10303975B2/en
2017-09-29 Assigned to GOOGLE INC. reassignment GOOGLE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ADAM, HARTWIG, ZHANG, LI
2018-07-26 Publication of US20180211134A1 publication Critical patent/US20180211134A1/en
2019-05-28 Publication of US10303975B2 publication Critical patent/US10303975B2/en
230000013016 learning Effects 0 abstract claims 4
This application is a continuation of, and claims priority to, U.S. patent application Ser. No. 14/683,643, filed Apr. 10, 2015, which is a continuation of U.S. patent application Ser. No. 13/759,916, filed Feb. 5, 2013, now U.S. Pat. No. 9,020,247, Issued Apr. 28, 2015, which is a continuation of U.S. patent application Ser. No. 12/466,880, filed May 15, 2009, now U.S. Pat. No. 8,396,287, Issued Mar. 12, 2013, the contents of which are incorporated herein by reference in their entirety.
Further features and advantages of the present invention, as well as the structure and operation of various embodiments thereof are described in detail below with reference to the accompanying drawings. It is noted that the invention is not limited to the specific embodiments described herein. Such embodiments are presented herein for illustrative purposes only. Additional embodiments will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein.
FIG. 2 shows more details of unsupervised image annotator module 127, according to an embodiment of the present invention. In this embodiment, unsupervised image annotator module 127 includes three processing modules: a landmark identifier 201, an appearance model generator 202, and an image annotator 203. Modules 201, 202, and 203 can be implemented in software, firmware, hardware, or a combination thereof. In one embodiment, modules 201-203 are implemented in software using the C++ programming language. In one embodiment, a computer program product may have logic including the computer program logic of modules 201-203 recorded on a computer readable medium such as a hard disk, flash disk, or other form of storage.
FIG. 5 shows processing steps 501-504 in the generation of the set of n-grams according to step 401 described above. In step 501, one or more text/image sources 105 is accessed, for example, by landmark identifier module 201. Accessing of text/image sources 105 can include connecting to such sources either over a local network, or a wide area network such as the Internet. The text/image sources 105 that are selected to be processed can be identified based on various methods such as input from users or operators, automatic identification and classification of web sited by program components (e.g., identification of photo repository web sites by web bets), or a list of websites or other repositories that are monitored for content. Methods of connecting to sources such as text/image sources 105 are well known. Where necessary, an implementation of the present invention should also consider aspects of copyrights, privacy, etc., that may be involved in the use of images owned by various parties.
Predetermined rules can be applied to determine a narrowed and/or filtered set of tags that refer to landmarks from the potentially large number of available tags. For example, in step 503, one or more filtering rules or criteria can be applied to the set of n-grams of potential landmark descriptors collected in step 502. One filter that can be applied to the list of potential landmark descriptor n-grams is a bad words filter. The bad words filter includes a list of n-grams and phrases that are predetermined as bad and/or unhelpful to discriminate among landmarks. Another filter that is applied can be a stop word list. The stop word list can include n-grams that are expected to occur so frequently in tags and/or descriptors that they are unlikely to be helpful as landmark descriptors. Words such as “of,” “the,” and “and” are example n-grams that can be included in a stop word list. Another filter that can be applied is a minimum reliability measure, such as a minimum number of authors filter. The minimum number of authors filter can be used to remove any n-grams from the list of potential landmark descriptor n-grams that have less than a predetermined number of unique authors using those n-grams in their tags. For example, it may be predetermined that for any n-gram to be included in n-gram collection 108, the n-gram should be detected in the tags used by three or more unique authors.
S ⁡ ( k ) = ∑ for ⁢ ⁢ all ⁢ ⁢ image ⁢ ⁢ pairs ⁢ ⁢ ( i , j ) ⁢ W ⁡ ( i ) * L ⁡ ( i , k ) * M ⁡ ( i , j ) * L ⁡ ( j , k ) * W ⁡ ( j ) ∑ for ⁢ ⁢ all ⁢ ⁢ image ⁢ ⁢ pairs ⁢ ⁢ ( i , j ) ⁢ W ⁡ ( i ) * L ⁡ ( i , k ) * M ⁡ ( i , j ) * ( 1 - L ⁡ ( j , k ) ) * W ⁡ ( j )
The processing functionality of module 127 and/or modules 201-203, can be achieved in software, hardware, or a combination thereof. For example, modules 201 and 203 may be implemented entirely as software modules, or some of the functionality of the appearance model generator module 202 may be implemented using hardware such as a field programmable gate array (FPGA). It will be understood by a person of skill in the art that unsupervised image annotator module 127 and or computer 101 may include additional components and modules that facilitate the functions of the present invention.
1. A method for detecting and annotating landmarks in digital images:
electronically accessing a plurality of digital images in an image collection;
retrieving an n-gram set generated from one or more texts associated with one or more images in the plurality of digital images;
choosing, from the n-gram set, one or more scored n-grams having at least a minimum reliability measure that is based on a number of unique authors of the one or more texts;
automatically assigning, to at least one of the one or more images in the image collection, a tag descriptive of a landmark, to generate a set of landmark-tagged images, wherein images in the set of landmark-tagged images are algorithmically determined to include the landmark, wherein the tag is based upon the one or more chosen scored n-grams from the one or more texts associated with the at least one of the one or more images, wherein the landmark identifies a geographic point or geographic area, and prioritizing images of popular landmark locations within the image collection according to a landmark popularity measure for each landmark location represented within the image collection, wherein the landmark popularity measure is based on a number of images of respective landmark locations uploaded to a photo sharing website that stores the image collection;
learning an appearance model for the landmark from the set of landmark-tagged images; and
detecting the landmark in a new image that is not part of the image collection using the appearance model, wherein the method is performed by at least one processor.
2. The method of claim 1, further comprising annotating the new image with the tag descriptive of the landmark.
assigning correlation-weights to the plurality of digital images, wherein each image in the plurality of digital images is assigned a correlation-weight based on correlation of metadata of each image with other images in the plurality of digital images;
generating a matching-images graph from the plurality of digital images, wherein each edge in the matching-images graph is assigned a match score representing a level of match between two images connected by each edge; and
linking said scored n-grams to images in the plurality of digital images based upon said correlation-weights and the matching-images graph to generate links between scored n-grams and images in the plurality of digital images.
estimating a geo-reliability score for each image of the plurality of digital images using the matching-images graph, wherein the geo-reliability score is an estimation of accuracy of geo-location information of each image based on a comparison of visual consistency of other images with geo-location coordinates within a predetermined distance to each image.
computing a variance of geo-location for a scored n-gram of said n-gram set, wherein the variance is based on geo-locations of images linked to said scored n-gram in said matching-images graph; and
removing from said n-gram set any scored n-grams having a variance of geo-location exceeding a predetermined threshold.
6. The method of claim 3, wherein an n-gram score of each scored n-gram in the n-gram set is based on the matching-images graph.
7. The method of claim 6, wherein the n-gram score of each scored n-gram in the n-gram set is computed as a ratio of strength of internal edges of said matching-images graph and strength of external edges of said matching-images graph, wherein an internal edge exists between images having at least one common scored n-gram, and wherein an external edge exists between images not having at least one common scored n-gram.
8. The method of claim 3, further comprising merging two or more scored n-grams in said n-gram set.
9. The method of claim 8, wherein the merging is based at least on one of,
a similarity between respective scores of the two or more scored n-grams, and
an overlap of images having the two or more scored n-grams in linked scored n-grams.
10. The method of claim 3, wherein the metadata includes information relating to at least one of:
a geo-location, or
11. The method of claim 3, wherein each link in the matching-images graph represents matching feature descriptors between two images of the plurality of digital images.
12. The method of claim 1, wherein the appearance model includes visual information and non-visual information, and wherein the detecting the landmark includes using the visual information and the non-visual information to identify the landmark.
13. The method of claim 1, wherein an n-gram score of each scored n-gram in the n-gram set is based upon a level of match between images associated with each scored n-gram and a level of correlation between each image associated with the landmark with other images associated with the landmark, and wherein each scored n-gram is associated with one or more images having associated text from which each scored n-gram was obtained.
14. A method for detecting and annotating landmarks in digital images:
automatically assigning, to one or more images in an image collection that includes a plurality of digital images, a tag descriptive of a landmark, to generate a set of landmark-tagged images, wherein images in the set of landmark-tagged images are algorithmically determined to include the landmark, wherein the tag is based upon one or more scored n-grams from one or more texts associated with the one or more images, wherein the landmark identifies a geographic point or geographic area, and prioritizing images of popular landmark locations within the image collection according to a landmark popularity measure for each landmark location represented within the image collection, wherein the landmark popularity measure is based on a number of images of respective landmark locations uploaded to a photo sharing website that stores the image collection;
learning an appearance model for the landmark from the set of landmark-tagged images wherein the appearance model includes visual information and non-visual information; and
detecting the landmark in a new image that is not part of the image collection using the appearance model, wherein the detecting the landmark includes using the visual information and the non-visual information to identify the landmark, and wherein the method is performed by at least one processor.
15. A method for detecting and annotating landmarks in digital images:
automatically assigning, to one or more images in an image collection that includes a plurality of digital images, a tag descriptive of a landmark, to generate a set of landmark-tagged images, wherein images in the set of landmark-tagged images are algorithmically determined to include the landmark,
wherein the tag is based upon one or more scored n-grams from one or more texts associated with the one or more images, wherein an n-gram score of each of the one or more scored n-grams is based upon a level of match between images associated with each scored n-gram and a level of correlation between each image associated with the landmark with other images associated with the landmark, and wherein each scored n-gram is associated with one or more images having associated text from which each scored n-gram was obtained,
wherein the landmark identifies a geographic point or geographic area, and prioritizing images of popular landmark locations within the image collection according to a landmark popularity measure for each landmark location represented within the image collection, wherein the landmark popularity measure is based on a number of images of respective landmark locations uploaded to a photo sharing website that stores the image collection;
US15/663,796 2009-05-15 2017-07-30 Landmarks from digital photo collections Active US10303975B2 (en)
US14/683,643 Continuation US9721188B2 (en) 2009-05-15 2015-04-10 Landmarks from digital photo collections
US20180211134A1 US20180211134A1 (en) 2018-07-26
US10303975B2 true US10303975B2 (en) 2019-05-28
CN103853797B (en) * 2012-12-07 2017-10-17 中兴通讯股份有限公司 A kind of picture retrieval method and system based on n member picture indices structures
CN103853792B (en) * 2012-12-07 2018-06-15 中兴通讯股份有限公司 A kind of picture semantic automatic marking method and system
US10474714B2 (en) * 2013-05-01 2019-11-12 Kble Ltd Method and component for classifying resources of a database
CN103218460B (en) * 2013-05-14 2016-08-10 清华大学 Image tag complementing method based on the sparse reconstruct of optimum linearity
CN104794171B (en) * 2015-03-31 2018-06-05 百度在线网络技术（北京）有限公司 Mark the method and device of picture geographical location information
CN106776658A (en) * 2015-11-25 2017-05-31 宏碁股份有限公司 Method and its electronic installation that photo is arranged
US20050036712A1 (en) 2003-05-08 2005-02-17 Toshiaki Wada Image retrieving apparatus and image retrieving program
KR20060026924A (en) 2006-02-14 2006-03-24 (주)올라웍스 Tagging method and system for digital data
US20070110316A1 (en) 2005-11-14 2007-05-17 Fujifilm Corporation Landmark search system for digital camera, map data, and method of sorting image data
WO2008045704A1 (en) 2006-10-10 2008-04-17 Microsoft Corporation Identifying sight for a location
US20080140644A1 (en) 2006-11-08 2008-06-12 Seeqpod, Inc. Matching and recommending relevant videos and media to individual search engine results
CN101228785A (en) 2005-07-26 2008-07-23 松下电器产业株式会社 Image data management device and image data management method
US20080292186A1 (en) 2007-03-14 2008-11-27 Kabushiki Kaisha Toshiba Word recognition method and word recognition program
US20090049408A1 (en) 2007-08-13 2009-02-19 Yahoo! Inc. Location-based visualization of geo-referenced context
US20090216794A1 (en) 2008-02-27 2009-08-27 General Electric Company Method and system for accessing a group of objects in an electronic document
JP4671235B2 (en) 2006-01-26 2011-04-13 田岡化学工業株式会社 Method for producing fluorene derivative
JP2007316876A (en) * 2006-05-25 2007-12-06 Hitachi Ltd Document retrieval program
WO2008152805A1 (en) * 2007-06-14 2008-12-18 Panasonic Corporation Image recognizing apparatus and image recognizing method
2010-05-14 EP EP20100724937 patent/EP2430572A1/en not_active Ceased
US9020247B2 (en) * 2009-05-15 2015-04-28 Google Inc. Landmarks from digital photo collections
"Canadian Office Action", CA Application No. 2,762,090, dated May 2, 2016.
"Examination Report for CA Application No. 2,762,090", dated Apr. 10, 2017, 4 Pages.
"KIPO Notice of Preliminary Rejection", Korean Application No. 10-2011-7029949, dated Dec. 18, 2015.
"Notice of Acceptance", in Australian Office Action No. 2010248862, dated May 30, 2016, 2 pages.
Batur et al, Adaptive Active Appearance Models Nov. 2005, IEEE Transactions on Image Processing vol. 14, No. 11, pp. 1707-1721. *
Batur, , "Adaptive active appearance models", IEEE transactions on image processing, vol. 14, No. 11, Nov. 2005, pp. 1707-1721.
EPO, "Office Action in European Application No. 10724937.7", dated Nov. 9, 2017, 5 Pages.
JPO Notice of Allowance mailed in Japanese Application No. 2012-511045, dated Dec. 12, 2014.
Kennedy, et al., "Generating diverse and representative image search results for landmarks", Proceeding of the 17th international conference on World Wide Web, Apr. 21-25, 2008, pp. 297-306.
KIPO Office Action mailed in KR Patent Application No. 10-2010-7027837, dated Feb. 27, 2015.
Li, et al., "Modeling and Recognition of Landmark hnage Collections Using Iconic Scene Graphs", Proceedings of ECCV 2008, Lecture Notes in Computer Science, Springer, Oct. 12, 2008, pp. 427-440.
Lowe, "Object Recognition from Local Scale-Invariant Features", Proc. of the International Conference on Computer Vision, 1999, pp. 1150-1157.
SIPO Office Action mailed in Chinese Application No. 201080030849.3, dated Jan. 17, 2014.
SIPO Office Action mailed in Chinese Patent Application No. 201080030849.3, dated Jan. 19, 2015.
SIPO, Notification for Patent Registration Formalities and Notification on the Grant of Patent Right for Invention (with English translations) for Chinese Patent Application No. 201410455635.0, dated Jan. 17, 2018, 4 pages.
Takeuchi, , "Evaluation of Image-Based Landmark Recognition Techniques", Technical Report CMU-RI-TR-98-20, Carnegie Mellon University, Jul. 1, 1998, 16 pages.
Toyama et al "Geographic location tags on digital images", Nov. 2003 ACN, pp. 156-166. *
Toyama, et al., "Geographic Location Tags on Digital Images", ACM, Nov. 2003, pp. 156-166.
Tsai, et al., "Extent: Inferring image metadata from context and content", Proc. IEEE International Conference on Multimedia and Expo, 2005, pp. 1154-1157.
USPTO, "Final Rejection in U.S. Appl. No. 15/284,075", dated Aug. 6, 2018, 15.
USPTO, "Non-Final Office Action received in U.S. Appl. No. 15/284,074", dated Mar. 8, 2018, 15 Pages.
US8396287B2 (en) 2013-03-12
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ADAM, HARTWIG;ZHANG, LI;REEL/FRAME:043746/0276