Source: http://patents.com/us-8700641.html
Timestamp: 2018-12-16 11:39:59
Document Index: 356840315

Matched Legal Cases: ['art 1', 'art 2', 'Application No. 06838486', 'Application No. 06838488', 'Application No. 08', 'Application No. 200680051559', 'Application No. 2008', 'Application No. 60', 'Application No. 60', 'Application No. 60']

US Patent # 8,700,641. Detecting repeating content in broadcast media - Patents.com
United States Patent 8,700,641
Covell , et al. April 15, 2014
Covell; Michele (Palo Alto, CA), Baluja; Shumeet (Leesburg, VA), Fink; Michael (Jerusalem, IL)
Covell; Michele
Baluja; Shumeet
Fink; Michael
13/195,330
US 20120059845 A1 Mar 8, 2012
11563665 Nov 27, 2006 7991770
60740760 Nov 29, 2005
60823881 Aug 29, 2006
Current U.S. Class: 707/749 ; 707/722; 707/758
Field of Search: ;707/748,749,758,722,751,600,603
4811399 March 1989 Landell et al.
6023693 February 2000 Masuoka et al.
6494720 December 2002 Meyrowitsch
6563909 May 2003 Schmitz
6704920 March 2004 Brill et al.
6751601 June 2004 Zegers
6763339 July 2004 Fu et al.
6773266 August 2004 Dornbush et al.
6782186 August 2004 Covell et al.
6879967 April 2005 Stork
6892191 May 2005 Schaffer
6895514 May 2005 Kermani
6944632 September 2005 Stern
7103801 September 2006 Marilly et al.
7107207 September 2006 Goodman
7266492 September 2007 Goodman
7281219 October 2007 Hamilton et al.
7386479 June 2008 Mizuno
7472096 December 2008 Burges et al.
7617164 November 2009 Burges et al.
7831531 November 2010 Baluja et al.
2002/0133499 September 2002 Ward et al.
2003/0101144 May 2003 Moreno
2004/0025174 February 2004 Cerrato et al.
2004/0030688 February 2004 Aridor et al.
2005/0066352 March 2005 Herley
2005/0086682 April 2005 Burges et al.
2005/0096920 May 2005 Matz et al.
2005/0193016 September 2005 Seet et al.
2005/0283792 December 2005 Swix et al.
2006/0174348 August 2006 Rhoads et al.
2008/0090551 April 2008 Gidron et al.
2008/0263041 October 2008 Cheung
1524857 Apr 2005 EP
2004049438 Feb 2004 JP
Author: Koo et al.; Title: Speech Recognition and Utterance Verification Based on a Generalized Confidence Score; Date: 2001; pp. 821-832. cited by examiner .
U.S. Appl. No. 11/468,265, filed Aug. 29, 2006, Covell et al. cited by applicant .
"Community" definition. Oxford English Dictionary. Accessed Apr. 27, 2009. http://dictionary.oed.com/cgi/entry/50045241?single=1&query.sub.--type=wo- rd&queryword=community, 17 pages. cited by applicant .
"Compression" definition. Oxford English Dictionary. Accessed Apr. 27, 2009. http://dictionary.oed.com/cgi/entry/50045890?single=1&query.sub.--t- ype=word&queryword=compression, 4 pages. cited by applicant .
"Database" definition. Oxford English Dictionary. Accessed Apr. 27, 2009. http://dictionary.oed.com/cgi/entry/50057772?single=1&query.sub.--type=wo- rd&queryword=database, 2 pages. cited by applicant .
"Encrypt" definition. Oxford English Dictionary. Accessed Apr. 27, 2009. http://dictionary.oed.com/cgi/entry/00292459?single=1&query.sub.--type=wo- rd&queryword=encrypt, 1 page. cited by applicant .
"Shazam Experience Music" [online]. [retrieved on May 30, 2007]. Retrieved from the Internet: <URL: www.shazam.com/music/portal/sp/s/media-type/html/user/anon/page/default/t- emplate/Myhome/music.html>, 2 pages. cited by applicant .
Burges et al., "Duplicate Detection and Audio Thumbnails with Audio Fingerprinting" [online]. 2004, [retrieved on Nov. 21, 2006]. <URL: www.research.microsoft.com/.about.cburges/tech.sub.--reports/tr-2004-19.p- df>, 5 pages. cited by applicant .
Burges et al., "Using Audio Fingerprinting for Duplicate Detection and Thumbnail Generation," Mar. 2005, 4 pages. cited by applicant .
Cano et al., "A Review of Algorithms for Audio Fingerprinting" [online]. 2002, [retrieved on Nov. 21, 2006]. Retrieved from the Internet: <URL: www.iua.upf.es/mtg/publications/MMSP-2002-pcano.pdf>, 5 pages. cited by applicant .
Cohen et al., "Finding Interesting Associations without Support Pruning," 2001, Retrieved from the Internet: <URL: www.dbis.informatik.huberlin.de/dbisold/lehre/WS0405/kDD/paper/CDFG.sub.-- -00.pdf>, 12 pages. cited by applicant .
Ding, et al. "Robust Technologies towards Automatic Speech Recognition in Car Noise Environments," Signal Processing, 2006 8th International Conference, 2006, vol. 1, 3 pages. cited by applicant .
Gauch et al., "Identification of New Commercials Using Repeated Video Sequence Detection," Image Processing, 2005, ICIP 2005, IEEE International Conference, 2005, pp. 1252-1255. cited by applicant .
Gorin et al., "On adaptive acquisition of language," Acoustics, Speech, and Signal Processing, 1990, 1:601-604. cited by applicant .
Haitsma et al, "A Highly Robust Audio Fingerprinting System" [online]. 2002,[retrieved on Nov. 16, 2006]. Retrieved from the Internet: <URL: www.ismir2002.ismir.net/proceedings/02-FP04-2.pdf>, 9 pages. cited by applicant .
Jacobs et al., "Fast Multiresolution Image Querying" [online]. 1995, [retrieved on Nov. 21, 2006]. Retrieved from the Internet: <URL: www.grail.cs.washington.edu/projects/query/mrquery.pdf>, 10 pages. cited by applicant .
Jeong et al., "An information theoretic approach to adaptive system training using unlabeled data," Proceedings of International Joint Conference on Neural Networks, 2005, 1:191-195. cited by applicant .
Kang et al., "A Multi-layer Adaptive Function Neural Network (MADFUNN) for Analytical Function Recognition," International Joint Conference on Neural Networks, 2006, pp. 1784-1789. cited by applicant .
Kang et al., "An adaptive function neural network ADFUNN) for phrase recognition," Proceedings 2005 IEEE International Joint Conference on Neural Networks, 2005, 1:593-597. cited by applicant .
Ke et al., "Computer Vision for Music Identification" [online]. 2005, [retrieved on Nov. 21, 2006]. Retrieved from the Internet: <URL: www.cs.cmu.edu/.about.yke/musicretrieval/cvpr2005-mr.pdf>, 8 pages. cited by applicant .
Lin et al., "Input Data Representation for Self-Organizing Map in Software Classification," 2009 Second International Symposium on Knowledge Acquisition and Modeling, 2009,2:350-353. cited by applicant .
Miyazawa, "An all-phoneme ergodic HMM for unsupervised speaker adaptation," IEEE International Conference on Acoustics, Speech, and Signal Processing, 1993, 2:574-577. cited by applicant .
Sadlier et al., "Automatic TV Advertisement Detection from MPEG Bitstream," Pattern Recognition, 2001, 35(12):2719-2726. cited by applicant .
Shazam, "Shazam Entertainment Brings Music Recognition to Windows Mobile 5.0 Powered Smartphones" [online]. 2006, [retrieved on Nov. 16, 2006]. Retrieved from the Internet: <URL: www..shazam.com/music/portal/sp/s/media-type/html/user/anon/page/default/- template/pages/p/company.sub.--release30.html>, 1 page. cited by applicant .
Stanford, "C5276 Information Retrieval and Web Mining" [online]. 2005, [retrieved on Nov. 16, 2006]. Retrieved from the Internet: <URL: www.stanford.edu/class/cs276/handouts/lecture19.pdf>, 8 pages. cited by applicant .
Stanford, "Data Mining: Associations" [online]. 2002, [retrieved on Nov. 16, 2006]. Retrieved from the Internet: <URL: www.stanford.edu/class/cs206/cs206-2.pdf>, 11 pages. cited by applicant .
Stollnitz et al., "Wavelets for Computer Graphics: A Primer, Part 1," IEEE Computer Graphics and Applications, 1995, 15(3), 8 pages. cited by applicant .
Stollnitz et al., "Wavelets for Computer Graphics: Applications, A Primer, Part 2," IEEE Computer Graphics and Applications, 1995, 15(4), 9 pages. cited by applicant .
Viola et al., "Robust Real-Time Object Detection," Int. J. Computer Vision, 2002, 25 pages. cited by applicant .
Wang et al., "Training neural networks with additive noise in the desired signal," IEE Transactions on Neural Networks, 1999, 10(6):1511-1517. cited by applicant .
Wang et al., "Training neural networks with additive noise in the desired signal," Neurla Networks Proceedings, 1998. IEEE World Congress on Computational Intelligence, 1998, 2:1084-1089. cited by applicant .
Wang, "The Shazam Music Recognition Service," Communications of the ACM, 2006, 49(8):44-48. cited by applicant .
Weixin et al. "Learning to Rank Using Semantic Features in Document Retrieval," Global Congress on Intelligent Systems, 2009, 3:500-504. cited by applicant .
Yang, "MACS: Music Audio Characteristic Sequence Indexing for Similarity Retrieval," Oct. 21-24, 2001, New Paltz, New York. pp. 123-126. cited by applicant .
Zhao, "Connectionist training of non-linear hidden Markov models for speech recognition," Neural Networks, 1991 IEEE International Joint Conference, 1991, 2:1647-1652. cited by applicant .
Authorized Officerc. Gabriel. Supplemental EP Search Report for EP Application No. 06838486.6 dated Feb. 16, 2010, 8 pages. cited by applicant .
Authorized Officerc. Gabriel. Supplemental EP Search Report for EP Application No. 06838488.2 dated Feb. 16, 2010, 8 pages. cited by applicant .
Authorized Officer B. Copenheaver. PCT International Search Report for PCT/US2006/045549 dated Oct. 9, 2007, 10 pages. cited by applicant .
Authorized Officer S. Beliveau. PCT International Search Report for PCT/US2006/045551 dated Jul. 21, 2008, 20 pages. cited by applicant .
International Preliminary Report on Patentability, Application No. PCT/US06/45549 mailed Jun. 12, 2008, 7 pages. cited by applicant .
Authorized Officerc. Gabriel, European Search Report, EP Application No. 08 15 3719 mailed Sep. 26, 2008, 8 pages. cited by applicant .
International Preliminary Report on Patentability, Application No. PCT/US06/45551 mailed Apr. 2, 2009, 11 pages. cited by applicant .
Chinese Patent Office Action for Application No. 200680051559.0 dated Jan. 22, 2010, 14 pages. cited by applicant .
Notice of Reasons for Rejection for Japanese Patent Application No. 2008-543391, mailed Dec. 13, 2011, 3 pages. cited by applicant.
This application claims the benefit of priority from U.S. Provisional Patent Application No. 60/740,760, for "Environment-Based Referrals," filed Nov. 29, 2005, which application is incorporated by reference herein its entirety.
This application claims the benefit of priority from U.S. Provisional Patent Application No. 60/823,881, for "Audio Identification Based on Signatures," filed Aug. 29, 2006, which application is incorporated by reference herein its entirety.
This application is related to U.S. patent application Ser. No. 11/563,653, for "Determining Popularity Ratings Using Social and Interactive Applications For Mass Media," filed Nov. 27, 2006, and U.S. patent application Ser. No. 11/563,661, for "Social and Interactive Applications For Mass Media," filed Nov. 27, 2006. Each of these patent applications is incorporated by reference herein in its entirety.
1. A method, comprising: determining, by a system including a processor, a first match M.sub.h between a first audio descriptor representing a first recording at a first time step in an environment and a first reference descriptor, the first match M.sub.h having-a first confidence score C.sub.h indicative of a confidence of the first match, the first time step having a time step length l; determining, by the system, a second match M.sub.0 between a second audio descriptor representing a second recording at a second time step in the environment and a second reference descriptor, the second match M.sub.0 having a second confidence score C.sub.0 indicative of a confidence of the second match, the second time step having the time step length l, wherein the first time step is prior to the second time step, and the first match M.sub.h and the second match M.sub.0 are non-identity matches determined using a direct or locality sensitive hashing function and a validation process to select a most accurate match out of a plurality of candidate matches, wherein the first confidence score C.sub.h and the second confidence score C.sub.0 are based upon a log-likelihood function given by an audio fingerprinting process; discounting, by the system, the first confidence score C.sub.h by a discount value l/L to generate a discounted first confidence score C.sub.h-l/L, where L is an expected dwell time between a channel change; in response to the discounted first confidence score C.sub.h-l/L being greater than the second confidence score C.sub.0, employing, by the system, the first reference descriptor associated with the first match M.sub.h for selecting related content; and in response to the discounted first confidence score C.sub.h-l/L not being greater than the second confidence score C.sub.0, employing, by the system, the second reference descriptor associated with the second match M.sub.0 for selecting the related content.
2. The method of claim 1, further comprising: in response to the discounted first confidence score C.sub.h-l/L being greater than the second confidence score C.sub.0, designating, by the system, the first reference descriptor as a best match to the second audio descriptor of the second time step.
3. The method of claim 1, further comprising: in response to the discounted first confidence score C.sub.h-l/L not being greater than the second confidence score C.sub.0, designating, by the system, the second reference descriptor as a best match to the second audio descriptor of the second time step.
6. A non-transitory computer-readable medium having instructions stored thereon that, in response to execution, cause a system including a processor to perform operations comprising: determining a first match M.sub.h between a first audio descriptor representing a first recording at a first time step in an environment and a first reference descriptor, the first match M.sub.h having-a first confidence score C.sub.h indicative of a confidence of the first match, the first time step having a time step length l; determining a second match M.sub.0 between a second audio descriptor representing a second recording at a second time step in the environment and a second reference descriptor, the second match M.sub.0 having a second confidence score C.sub.0 indicative of a confidence of the second match, the second time step having the time step length l, wherein the first time step is temporally prior to the second time step, and the first match M.sub.h and the second match M.sub.0 are non-identity matches determined using a direct or locality sensitive hashing function and a validation process to select a most accurate match out of a plurality of candidate matches, wherein the first confidence score C.sub.h and the second confidence score C.sub.0 are based upon a log-likelihood function given by an audio fingerprinting process; discounting the first confidence score C.sub.h by a discount value l/L to generate a discounted first confidence score C.sub.h-l/L, where L is an expected dwell time between a channel change; in response to the discounted first confidence score C.sub.h-l/L being greater than the second confidence score C.sub.0, employing the first reference descriptor associated with the first match M.sub.h for selecting related content; and in response to the discounted first confidence score C.sub.h-l/L not being greater than the second confidence score C.sub.0, employing the second reference descriptor associated with the second match M.sub.0 for selecting the related content.
7. The non-transitory computer-readable medium of claim 6, the operations further comprising: in response to the discounted first confidence score C.sub.h-l/L being greater than the second confidence score C.sub.0, designating the first reference descriptor as a best match to the second audio descriptor of the second time step.
8. The non-transitory computer-readable medium of claim 6, the operations further comprising: in response to the discounted first confidence score C.sub.h-l/L not being greater than the second confidence score C.sub.0, designating the second reference descriptor as a best match of the second time step.
11. A system, comprising: a processor, communicatively coupled to a memory that stores computer-executable instructions, that executes or facilitates execution of the computer-executable instructions to perform operations comprising: a social application server: determine a first match M.sub.h between a first audio descriptor representing a first recording at a first time step in an environment and a first reference descriptor, the first match M.sub.h having-a first confidence score C.sub.h indicative of a confidence of the first match, the first time step having a time step length l; determine a second match M.sub.0 between a second audio descriptor representing a second recording at a second time step in the environment and a second reference descriptor, the second match M.sub.0 having a second confidence score C.sub.0 indicative of a confidence of the second match, the second time step having the time step length l, wherein the first time step is temporally prior to the second time step, and the first match M.sub.h and the second match M.sub.0 are non-identity matches determined using a direct or locality sensitive hashing function and a validation process to select a most accurate match out of a plurality of candidate matches, wherein the first confidence score C.sub.h and the second confidence score C.sub.0 are based upon a log-likelihood function given by an audio fingerprinting process; discount the first confidence score C.sub.h by a discount value l/L to generate a discounted first confidence score C.sub.h-l/L, where L is an expected dwell time between a channel change; in response to the discounted first confidence score C.sub.h-l/L being greater than the second confidence score C.sub.0, employ the first reference descriptor associated with the first match M.sub.h for selecting related content; and in response to the discounted first confidence score C.sub.h-l/L not being greater than the second confidence score C.sub.0, employ the second reference descriptor associated with the second match M.sub.0 for selecting the related content.
12. The system of claim 11, wherein the social application server further determine: in response to the discounted first confidence score C.sub.h-l/L being greater than the second confidence score C.sub.0, designate the first reference descriptor as a best match to the second audio descriptor of the second time step.
13. The system of claim 11, wherein the social application server further determine: in response to the discounted first confidence score C.sub.h-l/L not being greater than the second confidence score C.sub.0, designate the second reference descriptor as a best match to the second audio descriptor of the second time step.
Conventional television and interactive television systems lack the ability to detect rebroadcasts of advertising embedded in television programming. Conventional recording devices allow users to store television programs (including commercials) for rebroadcast at a later date or time. A common complaint among broadcasters is their inability to profit from these rebroadcasts, which from the broadcasters' perspective amounts to "free" advertising for the advertisers who bought space on the show's original airing.
In some implementations, the client-interface 102 includes an ambient audio detector (e.g., a microphone) for monitoring and recording the ambient audio of a mass media broadcast in a broadcast environment (e.g., a user's living room). One or more ambient audio segments or "snippets" are converted into distinctive and robust statistical summaries, referred to as "audio fingerprints" or "descriptors." In some implementations, the descriptors are compressed files containing one or more audio signature components that can be compared with a database of previously generated reference descriptors or statistics associated with the mass media broadcast.
A technique for generating audio fingerprints for music identification is described in Ke, Y., Hoiem, D., Sukthankar, R. (2005), Computer Vision for Music Identification, In Proc. Computer Vision and Pattern Recognition, which is incorporated herein by reference in its entirety. In some implementations, the music identification approach proposed by (hereinafter "Ke et al.") is adapted to generate descriptors for television audio data and queries, as described with respect to FIG. 4.
A technique for generating audio descriptors using wavelets is described in U.S. Provisional Patent Application No. 60/823,881, for "Audio Identification Based on Signatures." That application describes a technique that uses a combination of computer-vision techniques and large-scale-data-stream processing algorithms to create compact descriptors/fingerprints of audio snippets that can be efficiently matched. The technique uses wavelets, which is a known mathematical tool for hierarchically decomposing functions.
In "Audio Identification Based on Signatures," an implementation of a retrieval process includes the following steps: 1) given the audio spectra of an audio snippet, extract spectral images of, for example, 11.6*w ms duration, with random spacing averaging d-ms apart. For each spectral image: 2) compute wavelets on the spectral image; 3) extract the top-t wavelets; 4) create a binary representation of the top-t wavelets; 5) use min-hash to create a sub-fingerprint of the top-t wavelets; 6) use LSH with b bins and 1 hash tables to find sub-fingerprint segments that are close matches; 7) discard sub-fingerprints with less than v matches; 8) compute a Hamming distance from the remaining candidate sub-fingerprints to the query sub-fingerprint; and 9) use dynamic programming to combined the matches across time.
In some implementations, the descriptors and an associated user identifier ("user id") for identifying the client-side interface 102 are sent to the audio database server 104 via network 108. The audio database server 104 compares the descriptor to a plurality of reference descriptors, which were previously determined and stored in an audio database 110 coupled to the audio database server 104. In some implementations, the audio database server 104 continuously updates the reference descriptors stored in the audio database 110 from recent mass media broadcasts.
In some implementations, the social application server 106 serves a web page to the client-side interface 102, which is received and displayed by a web browser (e.g., Microsoft Internet Explorer.TM.) running at the client-side interface 102. The social application server 106 also receives the user id from the client-side interface 102 and/or audio database server 104 to assist in aggregating personalized content and serving web pages to the client-side interface 102.
In some implementations, client software running at the client-side interface 102 continually monitors and records n-second (e.g., 5 second) audio files ("snippets") of ambient audio. The snippets are then converted into m-frames (e.g., 415 frames) of k-bit encoded descriptors (e.g., 32-bit), according to a process described with respect to FIG. 4. In some implementations, the monitoring and recording is event based. For example, the monitoring and recording can be automatically initiated on a specified date and at a specified time (e.g., Monday, 8:00 P.M.) and for a specified time duration (e.g., between 8:00-9:00 P.M.). Alternatively, the monitoring and recording can be initiated in response to user input (e.g., a mouse click, function key or key combination) from a control device (e.g., a remote control, etc.). In some implementations, the ambient audio is encoded using a streaming variation of the 32-bit/frame discriminative features described in Ke et al.
In some implementations, the client software runs as a "side bar" or other user interface element. That way, when the client-side interface 102 is booted up, the ambient audio sampling can start immediately and run in the "background" with results (optionally) being displayed in the side bar without invoking a full web-browser session.
In some implementations, the descriptors are sent to the audio database server 104 as a query submission (also referred to as a query descriptor) in response to a trigger event detected by the monitoring process at the client-side interface 102. For example, a trigger event could be the opening theme of a television program (e.g., opening tune of "Seinfeld") or dialogue spoken by the actors. In some implementations, the query descriptors can be sent to the audio database server 104 as part of a continuous streaming process. In some implementations, the query descriptors can be transmitted to the audio database server 104 in response to user input (e.g., via remote control, mouse clicks, etc.).
The process 300 begins when a client-side interface (e.g., client-side interface 102) monitors and records snippets of ambient audio of a mass media broadcast in a broadcast environment (302). The recorded ambient audio snippets are encoded into descriptors (e.g., compressed statistical summaries), which can be sent to an audio database server (304) as queries. The audio database server compares the queries against a database of reference descriptors computed from mass media broadcast statistics to determine candidate descriptors that best match the query (308). The candidate descriptors are sent to a social application server or other network resource, which uses the candidate descriptors to aggregate personalized information for the user (310). For example, if the user is watching the television show "Seinfeld," then query descriptors generated from the show's ambient audio will be matched with reference descriptors derived from previous "Seinfeld" broadcasts. Thus, the best matching candidate descriptors are used to aggregate personalized information relating to "Seinfeld" (e.g., news stories, discussion groups, links to ad hoc social peer communities or chat rooms, advertisements, etc.). In some implementations, the matching procedure is efficiently performed using hashing techniques (e.g., direct hashing or locality sensitive hashing (LSH)) to achieve a short list of candidate descriptors, as described with respect to FIG. 4. The candidate descriptors are then processed in a validation procedure, such as described in Ke et al.
The statistics used to generate popularity ratings can be generated using a counter for each broadcast channel being monitored. In some implementations, the counters can be intersected with demographic group data or geographic group data. The popularity ratings can be used by viewers to "see what's hot" while the broadcast is ongoing (e.g., by noticing an increased rating during the 2004 Super Bowl half-time performance). Advertisers and content providers can also use popularity ratings to dynamically adjust the material shown in response to ratings. This is especially true for advertisements, since the short unit length and numerous versions of advertisements generated by advertising campaigns are easily exchanged to adjust to viewer rating levels. Other examples of statistics include but are not limited to: popularity of a television broadcast versus a radio broadcast by demographics or time, the popularity of times of day, i.e., peak watching/listening times, the number of households in a given area, the amount of channel surfing during particular shows (genre of shows, particular times of day), the volume of the broadcast, etc.
The personalized information is sent to the client-side interface (312). The popularity ratings can also be stored in a database for use by other processes (318), such as the dynamic adjustment of advertisements described above. The personalized information is received at the client-side interface (314) where it is formatted and presented in a user interface (316). The personalized information can be associated with a commenting medium (e.g., text messages in a chat room) that is presented to the user in a user interface. In some implementations, a chat room can include one or more subgroups. For example, a discussion group for "Seinfeld" might include a subgroup called "Seinfeld Experts," or a subgroup may be associated with a particular demographic, such as women between the ages of 20-30 who watch "Seinfeld," etc.
Ke et al. uses computer vision techniques to find highly discriminative, compact statistics for audio. Their procedure trained on labeled pairs of positive examples (where x and x' are noisy versions of the same audio) and negative examples (where x and x' are from different audio). During this training phase, machine-learning technique based on boosting uses the labeled pairs to select a combination of 32 filters and thresholds that jointly create a highly discriminative statistic. The filters localize changes in the spectrogram magnitude, using first and second order differences across time and frequency. One benefit of using these simple difference filters is that they can be calculated efficiently using a integral image technique described in Viola, P. and Jones, M. (2002), Robust Real-Time Object Detection, International Journal of Computer Vision, which is incorporated by reference herein in its entirety.
In some implementations, validation is achieved by viewing each database hit as support for a match at a specific query-database offset. For example, if the eight descriptor (q.sub.8) in a 5-second, 415-frame-long "Seinfeld" query snippet, q, hits the 1008.sup.th database descriptor (x.sub.1008), this supports a candidate match between the 5-second query and frames 1001 through 1415 in the audio database. Other matches between q.sub.n and x.sub.1000+n (1.ltoreq.n.ltoreq.415) would support this same candidate match.
In addition to temporal consistency, we need to account for frames when conversations temporarily drown out the ambient audio. This can be modeled as an exclusive switch between ambient audio and interfering sounds. For each query frame i, there is a hidden variable, y.sub.i: if y.sub.i=0, the i.sup.th frame of the query is modeled as interference only; if y.sub.i=1, the i.sup.th frame is modeled as from clean ambient audio. Taking an extreme view (pure ambient or pure interference) is justified by the extremely low precision with which each audio frame is represented (32 bits) and softened by providing additional bit-flop probabilities for each of the 32 positions of the frame vector under each of the two hypotheses (y.sub.i=0 and y.sub.i=1). Finally, we model the between-frame transitions between ambient-only and interference-only states as a hidden first-order Markov process, with transition probabilities derived from training data. For example, we can re-use the 66-parameter probability model given by Ke et al., CVPR 2005.
The final model of the match probability between a query vector, q, and an ambient-database vector at an offset of N frames, x.sub.N, is:
.function..times..times..times..times..function..times..times..times..fun- ction..times..times. ##EQU00001## where <q.sub.n,x.sub.m> denotes the bit differences between the 32-bit frame vectors q.sub.n and x.sub.m. This model incorporates both the temporal consistency constraint and the ambient/interference hidden Markov model.
In some implementations, post-match filtering is used to handle these intermittent low-confidence mismatches. For example, we can use a continuous-time hidden Markov model of channel switching with an expected dwell time (i.e., time between channel changes) of L seconds. The social application server 106 indicates the highest-confidence match within the recent past (along with its "discounted" confidence) as part of state information associated with each client session. Using this information, the server 106 selects either the content-index match from the recent past or the current index match, base on whichever has the higher confidence.
We use M.sub.h and C.sub.h to refer to the best match for the previous time step (5 seconds ago) and its log-likelihood confidence score. If we simply apply the Markov model to this previous best match, without taking another observation, then our expectation is that the best match for the current time is that same program sequence, just 5 seconds further along, and our confidence in this expectation is C.sub.h-l/L, where l=5 seconds is the query time step. This discount of l/L in the log-likelihood corresponds to the Markov model probability, e.sup.-l/L, of not switching channels during the l-length time step.
.times..times.> ##EQU00002##
, where M.sub.0 is the match that is used by the social application server 106 for selecting related content and M.sub.0 and C.sub.0 are carried forward on the next time step as M.sub.h and C.sub.h.
FIG. 5 is a flow diagram of one embodiment of a user interface 208 for interacting with mass personalization applications. The user interface 208 includes a personalized layer display area 502, a commenting medium display area 504, a sponsored links display area 506 and a content display area 508. The personalized layer display area 502 provides complementary information and/or images related to the video content shown in the content display area 508. The personalized layers can be navigated using a navigation bar 510 and an input device (e.g., a mouse or remote control). Each layer has an associated label in the navigation bar 510. For example, if the user selects the "Fashion" label, then the fashion layer, which includes fashion related content associated with "Seinfeld," will be presented in the display area 502.
In some implementations, the client-side interface 102 receives personalized information from the social application server 106. This information can include a web page, email, a message board, links, instant message, a chat room, or an invitation to join an ongoing discussion group, eRoom, video conference or netmeeting, voice call (e.g., Skype.RTM.), etc. In some implementations, the user interface 208 provides access to comments and/or links to comments from previously seen broadcasts or movies. For example, if user is currently watching a DVD of "Shrek" he may want to see what people said about the movie in the past.
In some implementations, the display area 502 includes a rating region 512, which is used to display popularity ratings related to a broadcast. For example, the display area 512 may show how many viewers are currently watching "Seinfeld" compared to another television show that is broadcast at the same time.
The sponsored links display area 506 includes information, images and/or links related to advertising that is associated with the broadcast. For example, one of the links 518 may take the user to a web site that is selling "Seinfeld" merchandise.
In some implementations, a button 522 is included in the content display area that can be used to bookmark video. For example, by clicking the button 522, the "Seinfeld" episode shown in the display area 508 is added to the user's favorites video library, which can then be viewed on-demand through a web-based streaming application or other access methods. According to the policy set by the content owner, this streaming service can provide free single-viewing playback, collect payments as the agent for the content owners, or insert advertisements that would provide payment to the content owners.
In some implementations, the system 600 includes one or more processors 602 (e.g., CPU), optionally one or more display devices 604 (e.g., CRT, LCD, etc.), a microphone interface 606, one or more network interfaces 608 (e.g., USB, Ethernet, FireWire.RTM. ports, etc.), optionally one or more input devices 610 (e.g., mouse, keyboard, etc.) and one or more computer-readable mediums 612. Each of these components is operatively coupled to one or more buses 614 (e.g., EISA, PCI, USB, FireWire.RTM., NuBus, PDS, etc.).
The term "computer-readable medium" refers to any medium that participates in providing instructions to a processor 602 for execution, including without limitation, non-volatile media (e.g., optical or magnetic disks), volatile media (e.g., memory) and transmission media. Transmission media includes, without limitation, coaxial cables, copper wire and fiber optics. Transmission media can also take the form of acoustic, light or radio frequency waves.
The computer-readable medium(s) 612 further includes an operating system 616 (e.g., Mac OS.RTM., Windows.RTM., Unix, Linux, etc.), a network communications module 618, client software 620 and one or more applications 622. The operating system 616 can be multi-user, multiprocessing, multitasking, multithreading, real-time and the like. The operating system 616 performs basic tasks, including but not limited to: recognizing input from input devices 610; sending output to display devices 604; keeping track of files and directories on storage devices 612; controlling peripheral devices (e.g., disk drives, printers, image capture device, etc.); and managing traffic on the one or more buses 614.
The network communications module 618 includes various components for establishing and maintaining network connections (e.g., software for implementing communication protocols, such as TCP/IP, HTTP, Ethernet, USB, FireWire.RTM., etc.).
If a "summary" (i.e., shorter version) of non-repeated program material is needed, one way to get that is to remove the advertisements (as detected by repeated material) and to take segments from the material just preceding and just following the advertisement location. On broadcast television, these positions in the program typically contain "teasers" (before the ads) and "recaps" (just after the ads). If a summary is to be made of a news program that includes a mix of non-repeated and repeated non-advertisement material, typically the repeated non-advertisement material corresponds to a sound bite. These segments generally contribute less information than the anchorperson's narration of the news story and are good candidates for removal. If a summary is to be made of a narrative program (e.g. a movie or a serial installment), repeated audio tracks typically correspond to theme sounds, mood music, or silence. Again, these are typically good segments to remove from a summary video. The process 700 described below provides a way of detecting these repeated audio tracks so they can be removed from the summary video.
The non-identity matches that are strongest are "grown" forwards and backwards in time, to find the beginning and ending points of the repeated material (708). In some implementations, this can be done using known dynamic programming techniques (e.g., Viterbi decoding). In extending the match forward in time, the last time slice in the strong "seed" match is set as "matching" and the last time slice of the first below-believable-strength match for the same database offset between the query and the match is set as "not matching." In some implementations, match scores for individual frames in between these two fixed points are used as observations, and a first-order Markov model allowing within state transitions, plus a single transition from "matching" to "not-matching" states, is used. The transition probability from matching to not matching to 1/L can be set somewhat arbitrarily, where L is the number of frames between these two fixed points, corresponding to the least knowledge of the transition location within the allowed range. Another possibility for selecting transition probabilities would use the match strength profiles to bias this estimate to an earlier or later transition. But this would increase the complexity of the dynamic programming model and is not likely to improve the results, since the match strengths are already used as observations within this period. The same process is used to grow the segment matches backwards in time (e.g., just switch past/future and run the same algorithm).
In some implementations, advertisers can participate in auctions related to the presence of ambient audio that is related to the product or service that the advertiser want to sell. For example, multiple advertisers could bid in an auction for the right to associate its products or services with an audio snippet or descriptor associated with "Seinfeld." The winner of the auction could then put some related information in front of the viewer (e.g., the sponsored links) whenever the subject ambient audio is present. In some implementations, advertisers could bid on ambient audio snippets having a meta-level description. For example, advertisers could bid on audio that is associated with a television ad (e.g., this is the audio associated with a Ford Explorer TV ad), on closed captioning (e.g., the captioning says "Yankees baseball"), on program segment location (e.g., this audio will occur 15 min into the "Seinfeld" and will occur 3 minutes after the previous commercial break and 1 min before the next commercial break), or on low-level acoustic or visual properties (e.g., "background music," "conversational voices," "explosive-like", etc.)
Previous Patent US 8,700,640 | Next Patent US 8,700,642