Source: http://patents.com/us-10200744.html
Timestamp: 2019-06-19 02:44:32
Document Index: 550034154

Matched Legal Cases: ['Application No. 09820936', 'Application No. 11833486', 'Application No. 12163712', 'Application No. 13168509', 'art 1', 'Application No. 61']

US Patent # 1,020,0744. Overlay rendering of user interface onto source video - Patents.com
United States Patent 10,200,744
Brockmann , et al. February 5, 2019
A method of generating a blended output including an interactive user interface and one or more supplemental images. At a client device, a video stream containing an interactive user interface is received from a server using a first data communications channel configured to communicate video content and a command is transmitted to the server that relates to a user input received through the interactive user interface. In response to the transmitting, an updated user interface is received using the first data communications channel, and one or more supplemental images are received using a second data communications channel. Each supplemental image is associated with a corresponding transparency coefficient. The updated user interface and the one or more supplemental images are blended according to the transparency coefficient for each supplemental image to generate a blended output and the blended output is transmitted toward the display device for display thereon.
Brockmann; Ronald A. (Utrecht, NL), Gorter; Onne (Hilversum, NL), Dev; Anuj (Amsterdam, NL), Hiddink; Gerritt (Amersfoot, NL)
Family ID: 1000003804236
15/139,166
US 20170055023 A1 Feb 23, 2017
14298796 Jun 6, 2014 9326047
61832069 Jun 6, 2013
Current CPC Class: H04N 21/4316 (20130101); H04N 21/26283 (20130101); H04N 21/4307 (20130101); H04N 21/4348 (20130101); H04N 21/440263 (20130101); H04N 21/440281 (20130101); H04N 21/482 (20130101); H04N 21/4821 (20130101); H04N 21/64322 (20130101); H04N 21/8126 (20130101); H04N 21/8545 (20130101); H04N 21/4622 (20130101); H04L 67/02 (20130101)
Current International Class: H04N 21/431 (20110101); H04N 21/643 (20110101); H04N 21/43 (20110101); H04N 21/434 (20110101); H04N 21/482 (20110101); H04N 21/8545 (20110101); H04N 21/462 (20110101); H04N 21/4402 (20110101); H04N 21/81 (20110101); H04N 21/262 (20110101); H04L 29/08 (20060101)
5642498 June 1997 Kutner
5995146 November 1999 Rasmussen
5995488 November 1999 Kalhunte et al.
6115076 September 2000 Linzer
6268864 July 2001 Chen et al.
6310601 October 2001 Moore et al.
6385771 May 2002 Gordon
6438140 August 2002 Jungers et al.
6477182 November 2002 Calderone
6480210 November 2002 Martino et al.
6560496 May 2003 Michnener
6721956 April 2004 Wsilewski
6813690 November 2004 Lango et al.
6850490 February 2005 Woo et al.
7031385 April 2006 Inoue et al.
7093028 August 2006 Shao et al.
7095402 August 2006 Kunil et al.
7151782 December 2006 Oz et al.
7945616 May 2011 Zeng et al.
8074248 December 2011 Sigmon et al.
8656430 February 2014 Doyle
8781240 July 2014 Srinivasan et al.
8839317 September 2014 Rieger et al.
9204113 December 2015 Kwok et al.
2001/0005360 June 2001 Lee
2001/0027563 October 2001 White et al.
2001/0043215 November 2001 Middleton, III et al.
2002/0178278 November 2002 Ducharme
2002/0178447 November 2002 Ptotnick et al.
2003/0005452 January 2003 Rodriguez
2003/0018796 January 2003 Chou et al.
2003/0039398 February 2003 McIntyre
2003/0066093 April 2003 Cruz-Rivera et al.
2003/0107443 June 2003 Clancy
2003/0126608 July 2003 Safadi
2003/0126611 July 2003 Kuczynski-Brown
2003/0217360 November 2003 Gordon et al.
2004/0055007 March 2004 Allport
2004/0216045 October 2004 Martin et al.
2005/0155063 July 2005 Bayrakeri
2006/0174021 August 2006 Osborne et al.
2006/0174026 August 2006 Robinson
2006/0184614 August 2006 Baratto et al.
2006/0203913 September 2006 Kim et al.
2006/0267995 November 2006 Radloff
2007/0009042 January 2007 Craig
2007/0162953 July 2007 Bollinger et al.
2007/0178243 August 2007 Dong et al.
2007/0192798 August 2007 Morgan
2007/0234220 October 2007 Khan et al.
2008/0046373 February 2008 Kim
2008/0052742 February 2008 Kopf
2008/0086747 April 2008 Rasanen et al.
2008/0097953 April 2008 Levy et al.
2008/0144711 June 2008 Chui et al.
2008/0181221 July 2008 Kampmann et al.
2008/0184120 July 2008 O-Brien-Strain et al.
2008/0253685 October 2008 Kuranov et al.
2008/0271080 October 2008 Grossweiler et al.
2009/0204920 August 2009 Beverly et al.
2009/0316709 December 2009 Polcha et al.
2010/0009623 January 2010 Hennenhoefer et al.
2010/0054268 March 2010 Divivier
2010/0073371 March 2010 Ernst et al.
2010/0325668 December 2010 Young et al.
2011/0072474 March 2011 Springer et al.
2011/0099594 April 2011 Chen et al.
2011/0110433 May 2011 Bjontegaard
2011/0161517 June 2011 Ferguson
2011/0173590 July 2011 Yanes
2011/0211591 September 2011 Traub et al.
2011/0231878 September 2011 Hunter et al.
2011/0261889 October 2011 Francisco
2011/0283304 November 2011 Roberts
2012/0023250 January 2012 Chen et al.
2012/0030706 February 2012 Hulse et al.
2012/0216232 August 2012 Chen et al.
2012/0221853 August 2012 Wingert et al.
2012/0284753 November 2012 Roberts et al.
2013/0047074 February 2013 Vestergaard
2013/0276015 October 2013 Rothschild
2013/0297887 November 2013 Woodward et al.
2013/0305051 November 2013 Fu et al.
2014/0081954 March 2014 Elizarov
2014/0157298 June 2014 Murphy
2014/0168515 June 2014 Sagliocco et al.
2014/0223307 August 2014 McIntosh et al.
2014/0223482 August 2014 McIntosh et al.
2014/0269930 September 2014 Robinson et al.
2014/0289627 September 2014 Brockmann et al.
2014/0317532 October 2014 Ma et al.
2014/0344861 November 2014 Berner et al.
2015/0023372 January 2015 Boatright
2015/0037011 February 2015 Hubner et al.
2015/0135209 May 2015 LaBosco et al.
103647980 Mar 2014 CN
0477786 Apr 1992 EP
0 881 808 Dec 1998 EP
1051039 Nov 2000 EP
1877150 Jan 2008 EP
2357555 Aug 2011 EP
2529739 Jan 1984 FR
2479164 Oct 2011 GB
60-054324 Mar 1985 JP
63-033988 Feb 1988 JP
7-160292 Jun 1995 JP
8-265704 Oct 1996 JP
10-228437 Aug 1998 JP
2001-145112 May 2001 JP
2001-245291 Sep 2001 JP
2002-300556 Oct 2002 JP
2003-087673 Mar 2003 JP
2004-501445 Jan 2004 JP
2004-312283 Nov 2004 JP
8-095599 Apr 2005 JP
2005-123981 May 2005 JP
2005-260289 Sep 2005 JP
2006-246358 Sep 2006 JP
2007-129296 May 2007 JP
2007-522727 Aug 2007 JP
2007-264440 Oct 2007 JP
2009-159188 Jul 2009 JP
10-2005-0001362 Jan 2005 KR
10-2005-0085827 Aug 2005 KR
10-2006-0095821 Sep 2006 KR
WO 1982002303 Jul 1982 WO
WO 1989008967 Sep 1989 WO
WO 1994016534 Jul 1994 WO
WO 1994019910 Sep 1994 WO
WO 1994021079 Sep 1994 WO
WO 1995032587 Nov 1995 WO
WO 1995033342 Dec 1995 WO
WO 1996014712 May 1996 WO
WO 1996027843 Sep 1996 WO
WO 1996031826 Oct 1996 WO
WO 1996037074 Nov 1996 WO
WO 1996042168 Dec 1996 WO
WO 1997016925 May 1997 WO
WO 1997033434 Sep 1997 WO
WO 1997039583 Oct 1997 WO
WO 1998026595 Jun 1998 WO
WO 1999000735 Jan 1999 WO
WO 1999030496 Jun 1999 WO
WO 1999030497 Jun 1999 WO
WO 1999030500 Jun 1999 WO
WO 1999030501 Jun 1999 WO
WO 1999035840 Jul 1999 WO
WO 1999041911 Aug 1999 WO
WO 1999056468 Nov 1999 WO
WO 99/65232 Dec 1999 WO
WO 1999066732 Dec 1999 WO
WO 2000002303 Jan 2000 WO
WO 00/38430 Jun 2000 WO
WO01/56293 Aug 2001 WO
WO 03/091832 Nov 2003 WO
WO2004/057609 Jul 2004 WO
WO2005/076575 Aug 2005 WO
WO 05/120067 Dec 2005 WO
WO2008/044916 Apr 2008 WO
WO2009/038596 Mar 2009 WO
WO2013/184604 Dec 2013 WO
ActiveVideo Networks, Inc., International Preliminary Report on Patentablity, PCT/US2013/036182, dated Oct. 14, 2014, 9 pgs. cited by applicant .
ActiveVideo Networks Inc., Communication Pursuant to Rule 94(3), EP08713106-6, dated Jun. 25, 2014, 5 pgs. cited by applicant .
ActiveVideo Networks Inc., Communication Pursuant to Rules 161(2) & 162 EPC, EP13775121.0, dated Jan. 20, 2015, 3 pgs. cited by applicant .
ActiveVideo Networks Inc., Examination Report No. 1, AU2011258972, dated Jul. 21, 2014, 3 pgs. cited by applicant .
ActiveVideo Networks Inc., Certificate of Patent JP5675765, dated Jan. 9, 2015, 3 pgs. cited by applicant .
Brockmann, Notice of Allowance, U.S. Appl. No. 13/445,104, dated Dec. 14, 2014, 14 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/668,004, dated Feb. 26, 2015, 17 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/686,548, dated Jan. 5, 2015, 12 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/911,948, dated Dec. 26, 2014, 12 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/911,948, dated Jan. 29, 2015, 11 pgs. cited by applicant .
Dahlby, Office Action, U.S. Appl. No. 12/651,203, dated Dec. 3, 2014, 19 pgs. cited by applicant .
Gordon, Notice of Allowance, U.S. Appl. No. 12/008,697, dated Dec. 8, 2014, 10 pgs. cited by applicant .
Gordon, Office Action, U.S. Appl. No. 12/008,722, dated Nov. 28, 2014, 18 pgs. cited by applicant .
Regis, Notice of Allowance, U.S. Appl. No. 13/273,803, dated Nov. 18, 2014, 9 pgs. cited by applicant .
Regis, Notice of Allowance, U.S. Appl. No. 13/273,803, dated Mar. 2, 2015, 8 pgs. cited by applicant .
Sigmon, Notice of Allowance, U.S. Appl. No. 13/311,203, dated Dec. 19, 2014, 5 pgs. cited by applicant .
TAG Networks Inc, Decision to Grant a Patent, JP 2008-506474, dated Oct. 4, 2013, 5 pgs. cited by applicant .
ActiveVideo Networks Inc., Decision to refuse a European patent application EPC, EP09820936.4, dated Feb. 20, 2015, 4 pgs. cited by applicant .
ActiveVideo Networks Inc., Communication Pursuant to Article 94(3) EPC, 10754084.1, dated Feb. 10, 2015, 12 pgs. cited by applicant .
ActiveVideo Networks Inc., Communication under Rule 71(3) EPC, Intention to Grant, EP08713106.6, dated Feb. 19, 2015, 12 pgs. cited by applicant .
ActiveVideo Networks Inc., Notice of Reasons for Rejection, JP2014-100460, dated Jan. 15, 2015, 6 pgs. cited by applicant .
ActiveVideo Networks Inc., Notice of Reasons for Rejection, JP2013-509016, dated Dec. 24, 2014 (dated Jan. 14, 2015), 11 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/737,097, dated Mar. 16, 2015, 18 pgs. cited by applicant .
Craig, Decision on Appeal -Reversed-, U.S. Appl. No. 11/178,177, dated Feb. 25, 2015, 7 pgs. cited by applicant .
Craig, Notice of Allowance, U.S. Appl. No. 11/178,177, dated Mar. 5, 2015, 7 pgs. cited by applicant .
Craig, Notice of Allowance, U.S. Appl. No. 11/178,181, dated Feb. 13, 2015, 8 pgs. cited by applicant .
ActiveVideo Networks, Inc., Certificate of Grant, EP08713106.6-1908, dated Aug. 5, 2015, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Certificate of Grant, AU2011258972, dated Nov. 19, 2015, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Certificate of Grant, AU2011315950, dated Dec. 17, 2015, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Certificate of Grant, AU2011249132, dated Jan. 7, 2016, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Certificate of Grant EP13168509.11908, dated Sep. 30, 2015, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Certificate of Patent, JP2013534034, dated Jan. 8, 2016, 4 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Rules 161(1) and 162 EPC, EP14722897.7, dated Oct. 28, 2015, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Rules 161(1) and 162 EPC, EP14740004.8, dated Jan. 26, 2016, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Rules 161(1) and 162 EPC, EP14736535.7, dated Jan. 26, 2016, 2 pgs. cited by applicant .
ActiveVideo, Communication Pursuant to Article-94(3) EPC, EP12767642.7, dated Sep. 4, 2015, 4 pgs. cited by applicant .
AcriveVideo, Communication Pursuant to Article 94(3) EPC, EP10841764.3, dated Dec. 18, 2015, 6 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Rules 70(2) abd 70a(2) EP13735906.3, dated Nov. 27, 2015, 1 pg. cited by applicant .
ActiveVideo Networks, Inc., Decision to Grant, EP08713106.6-1908, dated Jul. 9, 2015, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Decision to Grant, EP13168509.1-1908, dated Sep. 3, 2015, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Decision to Grant, JP2014100460, dated Jul. 24, 2015, 5 pgs. cited by applicant .
ActiveVideo Networks, Inc., Decision to Refuse a European Patent Application, EP08705578.6, dated Nov. 26, 2015, 10 pgs. cited by applicant .
ActiveVideo Networks Inc., Examination Report No. 2, AU2011249132, dated May 29, 2015, 4 pgs. cited by applicant .
ActiveVideo Networks Inc., Examination Report No. 2, AU2011315950, dated Jun. 25, 2015, 3 pgs. cited by applicant .
ActiveVideo Networks Inc., Extended European Search Report, EP13735906.3, dated Nov. 11, 2015, 10 pgs. cited by applicant .
ActiveVideo, International Search Report and Written Opinion, PCT/US2015/027803, dated Jun. 24, 2015, 18 pgs. cited by applicant .
ActiveVideo, International Search Report and Written Opinion, PCT/US2015/027804, dated Jun. 25, 2015, 10 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Search Report and Written Opinion, PCT-US2015028072, dated Aug. 7, 2015, 9 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Preliminary Report on Patentability, PCT-US2014030773, dated Sep. 15, 2015, 6 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Preliminary Report on Patentability, PCT/US2014041430, dated Dec. 8, 2015, 6 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Preliminary Report on Patentability, PCT-US2014041416, dated Dec. 8, 2015, 6 pgs. cited by applicant .
ActiveVideo Networks, Inc., K1PO'S Notice of Preliminary Rejection, KR10-2010-7019512, dated Jul. 15, 2015, 15 pgs. cited by applicant .
ActiveVideo Networks, Inc., KIPO'S Notice of Preliminary Rejection, KR10-20107021116, dated Jul. 13, 2015, 19 pgs. cited by applicant .
ActiveVideo Networks, Inc., KIPO'S Notice of Preliminary Rejection, KR10-2011-7024417, dated Feb. 18, 2016, 15 pgs. cited by applicant .
ActiveVideo Networks, Inc., KIPO'S Second Notice of Preliminary Rejection, KR10-2010-7019512, dated Feb. 12, 2016, 5 pgs. cited by applicant .
ActiveVideo, Notice of Reasons for Rejection, JP2013-509016, dated Dec. 3, 2015, 7 pgs. cited by applicant .
ActiveVideo, Notice of German Patent, EP602008040474-9, dated Jan. 6, 2016, 4 pgs. cited by applicant .
ActiveVideo Networks B.V., Office Action, IL222830, dated Jun. 28, 2015, 7 pgs. cited by applicant .
ActiveVideo Networks, Inc., Office Action, JP2013534034, dated Jun. 16, 2015, 6 pgs. cited by applicant .
Avinity-Systems-BV, PreTrial-Reexam-Report-JP2009530298, dated Apr. 24, 2015, 6 pgs. cited by applicant .
Brockmann, Notice of Allowance, U.S. Appl. No. 13/911,948, dated Jul. 10, 2015, 5 pgs. cited by applicant .
Brockmann, Notice of Allowance, U.S. Appl. No. 13/438,617, dated May 22, 2015, 18 pgs. cited by applicant .
Brockmann, Notice of Allowance, U.S. Appl. No. 13/445,104, dated Apr. 23, 2015, 8 pgs. cited by applicant .
Brockmann, Notice of Allowance, U.S. Appl. No. 14/262,674, dated Sep. 30, 2015, 7 pgs. cited by applicant .
Brockmann, Notice of Allowance, U.S. Appl. No. 13/911,948, dated Aug. 21, 2015, 6 pgs. cited by applicant .
Brockmann, Notice of Allowance, U.S. Appl. No. 13/911,948, dated Aug. 5, 2015, 5 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 12/443,571, dated Jul. 9, 2015, 28 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 13/668,004, dated Aug. 3, 2015, 18 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 13/686,548, dated Aug. 12, 2015, 13 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/686,548, dated Feb. 8, 2016, 13 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 13/737,097, dated Aug. 14, 2015, 17 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 14/262,674, dated May 21, 2015, 7 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 12/443,571, dated Dec. 4, 2015, 30 pgs. cited by applicant .
Dahlby, Office Action U.S. Appl. No. 12/651,203, dated Jul. 2, 2015, 25 pgs. cited by applicant .
Dahlby, Final Office Action, U.S. Appl. No. 12/651,203, dated Dec. 11, 2015, 25 pgs. cited by applicant .
Gecsei, J., "Adaptation in Distributed Multimedia Systems," IEEE Multimedia, IEEE Service Center, New York, NY, vol. 4, No. 2, Apr. 1, 1997, 10 pgs. cited by applicant .
Gordon, Notice of Allowance, U.S. Appl. No. 12/008,697, dated Apr. 1, 2015, 10 pgs. cited by applicant .
Gordon, Final Office Action, U.S. Appl. No. 12/008,722, dated Jul. 2, 2015, 20 pgs. cited by applicant .
Gordon, Notice of Allowance, U.S. Appl. No. 12/008,722, dated Feb. 17, 2016, 10 pgs. cited by applicant .
Jacob, Bruce, "Memory Systems: Cache, DRAM, Disk," The Cache Layer, Chapter 22, p. 739. cited by applicant .
McElhatten, Office Action, U.S. Appl. No. 14/698,633, dated Feb. 22, 2016, 14 pgs. cited by applicant .
Ohta, K., et al., "Selective Multimedia Access Protocol for Wireless Multimedia Communication," Communications, Computers and Signal Processing, 1997, IEEE Pacific Rim Conference NCE Victoria, BC, Canada, Aug. 1997, vol. 1, 4 pgs. cited by applicant .
Sigmon, Notice of Allowance, U.S. Appl. No. 13/311,203, dated Apr. 14, 2015, 5 pgs. cited by applicant .
Wei, S., "QoS Tradeoffs Using an Application-Oriented Transport Protocol (AOTP) for Multimedia Applications Over IP." Sep. 23-26, 1999, Proceedings of the Third International Conference on Computational Intelligence and Multimedia Applications, New Delhi, India, 5 pgs. cited by applicant .
ActiveVideo Networks, Inc., Certificate of Grant, HK10102800.4, dated Jun. 10, 2016, 3 pgs. cited by applicant .
ActiveVideo Networks, Inc., Certificate of Patent, IL215133, dated Mar. 31, 2016, 1 pg. cited by applicant .
ActiveVideo Networks, Inc., Certificate of Grant, HK14101604, dated Sep. 8, 2016, 4 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Article 94(3) EPC, EP14722897.7, dated Jun. 29, 2016, 6 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Article 94(3) EPC, EP11738835.5, dated Jun. 10, 2016, 3 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Rules 161(1) and 162 EPC, EP15785776.4, dated Dec. 8, 2016, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Rules 161(1) and 162 EPC, EP15721482.6, dated Dec. 13, 2016, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Rules 161(1) and 162 EPC, EP15721483.4, dated Dec. 15, 2016, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Decision to Refuse an EP Patent Application, EP 10754084.1, dated Nov. 3, 2016, 4 pgs. cited by applicant .
ActiveVideo Networks, Inc. Notice of Reasons for Rejection, JP2015-159309, dated Aug. 26, 2016. 11 pgs. cited by applicant .
ActiveVideo Networks, Inc. Denial of Entry of Amendment, JP2013-509016, dated Aug. 30, 2016, 7 pgs. cited by applicant .
ActiveVideo Networks, Inc. Notice of Final Rejection, JP2013-509016, dated Aug. 30, 2016, 3 pgs. cited by applicant .
ActiveVideo Networks, Inc., Partial Supplementary Extended European Search Report, EP13775121.0, dated Jun. 14, 2016, 7 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Preliminary Report on Patentability, PCT-US2015028072, dated Nov. 1, 2016, 7 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Preliminary Report on Patentability, PCT-US2015/027803, dated Oct. 25, 2016, 8 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Preliminary Report on Patentability, PCT-US2015/027804, dated Oct. 25, 2016, 6 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Search Report and Written Opinion, PCT/US2015/000502, dated May 6, 2016, 8 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Search Report and Written Opinion, PCT/US2016/040547, dated Sep. 19, 2016, 6 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Search Report and Written Opinion, PCT/US2016/051283, dated Nov. 29, 2016, 10 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Article 94(3), EP13735906.3, dated Jul. 18, 2016, 5 pgs. cited by applicant .
ActiveVideo, Intent to Grant, EP12767642.7, dated Jan. 2, 2017, 15 pgs. cited by applicant .
Avinity Systems B.V., Notice of Grant-JP2009530298, dated Apr. 12, 2016, 3 pgs. cited by applicant .
Avinity Systems B.V., Decision to Refuse an EP Patent Application, EP07834561.8, dated Oct. 10, 2016, 17 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/668,004, dated Mar. 25, 2016, 17 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 13/668,004, dated Nov. 20, 2016, 20 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/737,097, dated May 16, 2016, 23 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 13/737,097, dated Oct. 20, 2016, 22 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 14/298,796, dated Sep. 11, 2015, 11 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 12/443,571, dated Aug. 1, 2016, 32 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 14/217,108, dated Apr. 13, 2016, 8 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 14/696,462, dated Feb. 8, 2017, 6 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 14/217,108, dated Dec. 1, 2016, 9 pgs. cited by applicant .
Dahlby, Advisory Action, U.S. Appl. No. 12/651,203, dated Nov. 21, 2016, 5 pgs. cited by applicant .
Hoeben, Office Action, U.S. Appl. No. 14/757,935, dated Sep. 23, 2016, 28 pgs. cited by applicant .
McElhatten, Final Office Action, U.S. Appl. No. 14/698,633, dated Aug. 18, 2016, 16 pgs. cited by applicant .
ActiveVideo Networks BV, International Preliminary Report on Patentability, PCT/NL2011/050308, dated Sep. 6, 2011, 8 pgs. cited by applicant .
ActiveVideo Networks BV, International Search Report and Written Opinion, PCT/NL2011/050308, dated Sep. 6, 2011, 8 pgs. cited by applicant .
Activevideo Networks Inc., International Preliminary Report on Patentability, PCT/US2011/056355, dated Apr. 16, 2013, 4 pgs. cited by applicant .
ActiveVideo Networks Inc., International Preliminary Report on Patentability, PCT/US2012/032010, dated Oct. 8, 2013, 4 pgs. cited by applicant .
ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2011/056355, dated Apr. 13, 2012, 6 pgs. cited by applicant .
ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2012/032010, dated Oct. 10, 2012, 6 pgs. cited by applicant .
ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2013/020769, dated May 9, 2013, 9 pgs. cited by applicant .
ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2013/036182, dated Jul. 29, 2013, 12 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Search Report and Written Opinion, PCT/US2009/032457, dated Jul. 22, 2009, 7 pgs. cited by applicant .
ActiveVideo Networks Inc., Extended EP Search Rpt, Application No. 09820936-4, 11 pgs. cited by applicant .
ActiveVideo Networks Inc., Extended EP Search Rpt, Application No. 11833486.1, 6 pgs. cited by applicant .
AcitveVideo Networks Inc., Korean Intellectual Property Office, International Search Report; PCT/US2009/032457, dated Jul. 22, 2009, 7 pgs. cited by applicant .
Avinity Systems B.V., Extended European Search Report, Application No. 12163712-8, 10 pgs. cited by applicant .
Craig, Notice of Allowance, U.S. Appl. No. 11/178,176, dated Dec. 23, 2010, 8 pgs. cited by applicant .
Craig, Notice of Allowance, U.S. Appl. No. 11/178,183, dated Jan. 12, 2012, 7 pgs. cited by applicant .
Craig, Notice of Allowance, U.S. Appl. No. 11/178,183, dated Jul. 19, 2012, 8 pgs. cited by applicant .
Craig, Notice of Allowance, U.S. Appl. No. 11/178,189, dated Oct. 12, 2011, 7 pgs. cited by applicant .
Craig, Notice of Allowance, U.S. Appl. No. 11/178,176, dated Mar. 23, 2011, 8 pgs. cited by applicant .
Craig, Notice of Allowance, U.S. Appl. No. 13/609,183, dated Aug. 26, 2013, 8 pgs. cited by applicant .
Craig, Final Office Action, U.S. Appl. No. 11/103,838, dated Feb. 5, 2009, 30 pgs. cited by applicant .
Craig, Final Office Action, U.S. Appl. No. 11/178,181, dated Aug. 25, 2010, 17 pgs. cited by applicant .
Craig, Final Office Action, U.S. Appl. No. 11/103,838, dated Jul. 6, 2010, 35 pgs. cited by applicant .
Craig, Final Office Action, U.S. Appl. No. 11/178,176, dated Oct. 1, 2010, 8 pgs. cited by applicant .
Craig, Final Office Action, U.S. Appl. No. 11/178,183, dated Apr. 13, 2011, 16 pgs. cited by applicant .
Craig, Final Office Action, U.S. Appl. No. 11/178,177, dated Oct. 26, 2010, 12 pgs. cited by applicant .
Craig, Final Office Action, U.S. Appl. No. 11/178,181, dated Jun. 20, 2011, 21 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/103,838, dated May 12, 2009, 32 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/103,838, dated Aug. 19, 2008, 17 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/103,838, dated Nov. 19, 2009, 34 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,176, dated May 6, 2010, 7 pgs. cited by applicant .
Craig, Office-Action U.S. Appl. No. 11/178,177, dated Mar. 29, 2011, 15 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,177, dated Aug. 3, 2011, 26 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,177, dated Mar. 29, 2010, 11 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,181, dated Feb. 11, 2011, 19 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,181, dated Mar. 29, 2010, 10 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,182, dated Feb. 23, 2010, 15 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,183, dated Dec. 6, 2010, 12 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,183, dated Sep. 15, 2011, 12 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,183, dated Feb. 19, 2010, 17 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,183, dated Jul. 20, 2010, 13 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,189, dated Nov. 9, 2010, 13 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,189, dated Mar. 15, 2010, 11 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,189, dated Jul. 23, 2009, 10 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 11/178,189, dated May 26, 2011, 14 pgs. cited by applicant .
Craig, Office Action, U.S. Appl. No. 13/609,183, dated May 9, 2013, 7 pgs. cited by applicant .
Pavlovskaia, Office Action, JP 2011-516499, dated Feb. 14, 2014, 19 pgs. cited by applicant .
European Patent Office, Extended European Search Report for International Application No. PCT/US2010/027724, dated Jul. 24, 2012, 11 pages. cited by applicant .
FFMPEG, http://www.ffmpeg.org, downloaded Apr. 8, 2010, 8 pgs. cited by applicant .
Herr, Notice of Allowance, U.S. Appl. No. 11/620,593, dated May 23, 2012, 5 pgs. cited by applicant .
Herr, Notice of Allowance, U.S. Appl. No. 12/534,016, dated Feb. 7, 2012, 5 pgs. cited by applicant .
Herr, Notice of Allowance, U.S. Appl. No. 12/534,016, dated Sep. 28, 2011, 15 pgs. cited by applicant .
Herr, Final Office Action, U.S. Appl. No. 11/620,593, dated Sep. 15, 2011, 104 pgs. cited by applicant .
Herr, Office Action, U.S. Appl. No. 11/620,593, dated Mar. 19, 2010, 58 pgs. cited by applicant .
Herr, Office Action, U.S. Appl. No. 11/620,593, dated Apr. 21, 2009 27 pgs. cited by applicant .
Herr, Office Action, U.S. Appl. No. 11/620,593, dated Dec. 23, 2009, 58 pgs. cited by applicant .
Herr, Office Action, U.S. Appl. No. 11/620,593, dated Jan. 24, 2011, 96 pgs. cited by applicant .
Herr, Office Action, U.S. Appl. No. 11/620,593, dated Aug. 27, 2010, 41 pgs. cited by applicant .
Hoarty, The Smart Headend--A Novel Approach to Interactive Television, Montreux Int'l TV Symposium, Jun. 9, 1995, 21 pgs. cited by applicant .
ICTV, Inc., International Preliminary Report on Patentability, PCT/US2006/022585, dated Jan. 29, 2008, 9 pgs. cited by applicant .
ICTV, Inc., International Search Report / Written Opinion, PCT/US2006/022585, dated Oct. 12, 2007, 15 pgs. cited by applicant .
ICTV, Inc., International Search Report / Written Opinion, PCT/US2008/000419, dated May 15, 2009, 20 pgs. cited by applicant .
ICTV, Inc., International Search Report / Written Opinion; PCT/US2006/022533, dated Nov. 20, 2006; 8 pgs. cited by applicant .
Ozer, Video Compositing 101. available from http://www.emedialive.com, Jun. 2, 2004, 5pgs. cited by applicant .
Porter, Compositing Digital Images, 18 Computer Graphics (No. 3), Jul. 1984, pp. 253-259. cited by applicant .
RSS Advisory Board, "RSS 2.0 Specification", published Oct. 15, 2007. Not Found. cited by applicant .
Sigmon, Final Office Action, U.S. Appl. No. 11/258,602, dated Feb. 23, 2009, 15 pgs. cited by applicant .
Sigmon, Office Action, U.S. Appl. No. 11/258,602, dated Sep. 2, 2008, 12 pgs. cited by applicant .
TAG Networks, Inc., Communication pursuant to Article 94(3) EPC, European Patent Application, 06773714.8, dated May 6, 2009, 3 pgs. cited by applicant .
TAG Networks Inc, Decision to Grant a Patent, JP 209-544985, dated Jun. 28, 2013, 1 pg. cited by applicant .
TAG Networks Inc., IPRP, PCT/US2006/010080, dated Oct. 16, 2007, 6 pgs. cited by applicant .
TAG Networks Inc., IPRP, PCT/US2006/024194, dated Jan. 10, 2008, 7 pgs. cited by applicant .
TAG Networks Inc., IPRP, PCT/US2006/024195, dated Apr. 1, 2009, 11 pgs. cited by applicant .
TAG Networks Inc., IPRP, PCT/US2006/024196, dated Jan. 10, 2008, 6 pgs. cited by applicant .
TAG Networks Inc., International Search Report, PCT/US2008/050221, dated Jun. 12, 2008, 9 pgs. cited by applicant .
TAG Networks Inc., Office Action, CN 200680017662.3, dated Apr. 26, 2010, 4 pgs. cited by applicant .
TAG Networks Inc., Office Action, EP 06739032.8, dated Aug. 14, 2009, 4 pgs. cited by applicant .
TAG Networks Inc., Office Action, EP 06773714.8, dated May 6, 2009, 3 pgs. cited by applicant .
TAG Networks Inc., Office Action, EP 06773714.8, dated Jan. 12, 2010, 4 pgs. cited by applicant .
TAG Networks Inc., Office Action, JP 2008-506474, dated Oct. 1, 2012, 5 pgs. cited by applicant .
TAG Networks Inc., Office Action, JP 2008-506474, dated Aug. 8, 2011, 5 pgs. cited by applicant .
TAG Networks Inc., Office Action, JP 2008-520254, dated Oct. 20, 2011, 2 pgs. cited by applicant .
TAG Networks, IPRP, PCT/US2008/050221, dated Jul. 7, 2009, 6 pgs. cited by applicant .
TAG Networks, International Search Report, PCT/US2010/041133, dated Oct. 19, 2010, 13 pgs. cited by applicant .
TAG Networks, Office Action, CN 200880001325.4, dated Jun. 22, 2011, 4 pgs. cited by applicant .
TAG Networks, Office Action, JP 2009-544985, dated Feb. 25, 2013, 3 pgs. cited by applicant .
Talley, A general framework for continuous media transmission control, Oct. 13-15, 1997, 10 pgs. cited by applicant .
The Toolame Project, Psych nl.c, 1999, 1 pg. cited by applicant .
TVHEAD, Inc., First Examination Report, in 1744/MUMNP/2007, dated Dec. 30, 2013, 6 pgs. cited by applicant .
TVHEAD, Inc., International Search Report, PCT/US2006/010080, dated Jun. 20, 2006, 3 pgs. cited by applicant .
TVHEAD, Inc., International Search Report, PCT/US2006/024194, dated Dec. 15, 2006, 4 pgs. cited by applicant .
TVHEAD, Inc., International Search Report, PCT/US2006/024195, dated Nov. 29, 2006, 9 pgs. cited by applicant .
TVHEAD, Inc., International Search Report, PCT/US2006/024196, dated Dec. 11, 2006, 4 pgs. cited by applicant .
TVHEAD, Inc., International Search Report, PCT/US2006/024197, dated Nov. 28, 2006, 9 pgs. cited by applicant .
Wang, A compressed domain beat detector using MP3 audio bitstream, Sep. 30-Oct. 5, 2001, 9 pgs. cited by applicant .
Wang, A multichannel audio coding algorithm for inter-channel redundancy removal, May 12-15, 2001, 6 pgs. cited by applicant .
Wang, schemes for re-compressing mp3 audio bitstreams,Nov. 30-Dec. 3, 2001, 5 pgs. cited by applicant .
Wang, The impact of the relationship between MDCT and DFT on audio compression, Dec. 13-15, 2000, 9 pgs. cited by applicant .
ActiveVideo Networks Inc., International Preliminary Report on Patentability, PCT/US2013/020769, dated Jul. 24, 2014, 6 pgs. cited by applicant .
ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2014/030773, dated Jul. 25, 2014, 8 pgs. cited by applicant .
ActiveVideo Networks Inc., International Search Report and Written Opinion, PCT/US2014/041416, dated Aug. 27, 2014, 8 pgs. cited by applicant .
ActiveVideo Networks Inc., Extended EP Search Rpt, Application No. 13168509.1, 10 pgs. cited by applicant .
ActiveVideo Networks Inc., Communication Pursuant to Rules 70(2) and 70a(2), EP10841764.3, dated Jun. 6, 2014, 1 pg. cited by applicant .
ActiveVideo Networks Inc., Communication Pursuant to Rules 70(2) and 70a(2), EP11833486.1, dated Apr. 24, 2014, 1 pg. cited by applicant .
ActiveVideo Networks Inc., Communication Pursuant to Article 94(3) EPC, EP08713106.6-1908, dated Jun. 26. 2014, 5 pgs. cited by applicant .
ActiveVideo Networks Inc., Communication Pursuant to Article 94(3) EPC, EP09713486.0, dated Apr. 14, 2014, 6 pgS. cited by applicant .
ActiveVideo Networks Inc., Examination Report No. 1, AU2011258972, dated Apr. 4, 2013, 5 pgs. cited by applicant .
ActiveVideo Networks Inc., Examination Report No. 1, AU2010339376, dated Apr. 30, 2014, 4 pgs. cited by applicant .
ActiveVideo Networks Inc., Summons to attend oral-proceeding, Application No. EP09820936-4, dated Aug. 19, 2014, 4 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Search Report and Written Opinion, PCT/US2014/041430, dated Oct. 9, 2014, 9 pgs. cited by applicant .
Active Video Networks, Notice of Reasons for Rejection, JP2012-547318, dated Sep. 26, 2014, 7 pgs. cited by applicant .
Adams, Jerry, NTZ Nachrichtechnische Zeitschrift. vol. 40, No. 7, Jul. 1987, Berlin DE pp. 534-536; Jerry Adams: `Glasfasernetz fur Breitbanddienste in London`, 5 pgs. No English Translation Found. cited by applicant .
Avinity Systems B.V., Communication pursuant to Article 94(3) EPC, EP 07834561.8, dated Jan. 31, 2014, 10 pgs. cited by applicant .
Avinity Systems B.V., Communication pursuant to Article 94(3) EPC, EP 07834561.8, dated Apr. 8, 2010, 5 pgs. cited by applicant .
Avinity Systems B.V., International Preliminary Report on Patentability, PCT/NL2007/000245, dated Mar. 31, 2009, 12 pgs. cited by applicant .
Avinity Systems B.V., International Search Report and Written Opinion, PCT/NL2007/000245, dated Feb. 19, 2009, 18 pgs. cited by applicant .
Avinity Systems B.V., Notice of Grounds of Rejection for Patent, JP 2009-530298, dated Sep. 30, 2013, 4 pgs. cited by applicant .
Avinity Systems B.V., Notice of Grounds of Rejection for Patent, JP 2009-530298, dated Sep. 25, 2012, 6 pgs. cited by applicant .
Avinity Systems B. V., Final Office Action, JP-2009-530298, dated Oct. 7, 2014, 8 pgs. cited by applicant .
Bird et al., "Customer Access to Broadband Services," ISSLS 86--The International Symposium on Subrscriber Loops and Services Sep. 29, 1986, Tokyo,JP 6 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 12/443,571, dated Mar. 7, 2014, 21 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 13/668,004, dated Jul. 16, 2014, 20 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 13/686,548, dated Sep. 24, 2014, 13 pgs. cited by applicant .
Brockmann, Final Office Action, U.S. Appl. No. 13/438,617, dated Oct. 30, 2014, 19 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/686,548, dated Mar. 10, 2014, 11 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/668,004, dated Dec. 23, 2013, 9 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/438,617, dated May 12, 2014, 17 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 12/443,571, dated Jun. 5, 2013, 18 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 12/443,571, dated Nov. 5, 2014, 26 pgs. cited by applicant .
Chang, Shih-Fu, et al., "Manipulation and Compositing of MC-DOT Compressed Video," IEEE Journal on Selected Areas of Communications, Jan. 1995, vol. 13, No. 1, 11 pgs. cited by applicant .
Dahlby, Office Action, U.S. Appl. No. 12/651,203, dated Jun. 5, 2014, 18 pgs. cited by applicant .
Dahlby, Final Office Action, U.S. Appl. No. 12/651,203, dated Feb. 4, 2013, 18 pgs. cited by applicant .
Dahlby, Office Action, U.S. Appl. No. 12/651,203, dated Aug. 16. 2012, 18 pgs. cited by applicant .
Dukes, Stephen D., "Photonics for cable television system design, Migrating to regional hubs and passive networks," Communications Engineering and Design, May 1992, 4 pgs. cited by applicant .
European Patent Office, Supplementary European Search Report, Application No. EP 09 70 8211, dated Jan. 5, 2011, 6 pgs. cited by applicant .
Frezza, W., "The Broadband Solution-Metropolitan CATV Networks," Proceedings of Videotex '84, Apr. 1984, 15 pgs. cited by applicant .
Gordon, Notice of Allowance, U.S. Appl. No. 12/008,697, dated Mar. 20, 2014, 10 pgs. cited by applicant .
Gordon, Final Office Action, U.S. Appl. No. 12/008,722, dated Mar. 30, 2012, 16 pgs. cited by applicant .
Gordon, Final Office Action, U.S. Appl. No. 12/035,236, dated Jun. 11, 2014, 14 pgs. cited by applicant .
Gordon, Final Office Action, U.S. Appl. No. 12/035,236, dated Jul. 22, 2013, 7 pgs. cited by applicant .
Gordon, Final Office Action, U.S. Appl. No. 12/035,236, dated Sep. 20, 2011, 8 pgs. cited by applicant .
Gordon, Final Office Action, U.S. Appl. No. 12/035,236, dated Sep. 21, 2012, 9 pgs. cited by applicant .
Gordon, Final Office Action, U.S. Appl. No. 12/008,697, dated Mar. 6, 2012, 48 pgs. cited by applicant .
Gordon, Office Action, U.S. Appl. No. 12/035,236, dated Mar. 12, 2013, 9 pgs. cited by applicant .
Gordon, Office Action, U.S. Appl. No. 12/035,236, dated Mar. 22, 2011, 8 pgs. cited by applicant .
Gordon, Office Action, U.S. Appl. No. 12/035,236, dated Mar. 28, 2012, 8 pgs. cited by applicant .
Gordon, Office Action, U.S. Appl. No. 12/035,236, dated Dec. 16, 2013, 11 pgs. cited by applicant .
Gordon, Office Action, U.S. Appl. No. 12/008,697, dated Aug. 1, 2013, 43 pgs. cited by applicant .
Gordon, Office Action, U.S. Appl. No. 12/008,697, dated Aug. 4, 2011, 39 pgs. cited by applicant .
Gordon, Office Action, U.S. Appl. No. 12/008,722, dated Oct. 11, 2011, 16 pgs. cited by applicant .
Handley et al, "TCP Congestion Window Validation," RFC 2861, Jun. 2000, Network Working Group, 22 pgs. cited by applicant .
Henry et al. "Multidimensional Icons" ACM Transactions on Graphics, vol. 9, No. 1 Jan. 1990, 5 pgs. cited by applicant .
Isensee et al., "Focus Highlight for World Wide Web Frames," Nov. 1, 1997, IBM Technical Disclosure Bulletin, vol. 40, No. 11, pp. 89-90. cited by applicant .
ICTV, Inc., International Search Report / Written Opinion, PCT/US2008/000400, dated Jul. 14, 2009, 10 pgs. cited by applicant .
ICTV, Inc., International Search Report / Written Opinion, PCT/US2008/000450, dated Jan. 26, 2009, 9 pgs. cited by applicant .
Kato, Y., et al., "A Coding Control algorithm for Motion Picture Coding Accomplishing Optimal Assignment of Coding Distortion to Time and Space Domains," Electronics and Communications in Japan, Part 1, vol. 72, No. 9, 1989, 11 pgs. cited by applicant .
Koenen, Rob,"MPEG-4 Overview--Overview of the MPEG-4 Standard" Internet Citation, Mar. 2001 (Mar. 2001), http://mpeg.telecomitalialab.com/standards/mpeq-4/mpeg-4.htm, May 9, 2002, 74 pgs. cited by applicant .
Langenberg, E, et al., "Integrating Entertainment and Voice on the Cable Network," SCTE , Conference on Emerging Technologies, Jan. 6-7, 1993, New Orleans, Louisiana, 9 pgs. cited by applicant .
Large, D., "Tapped Fiber vs. Fiber-Reinforced Coaxial CATV Systems", IEEE LCS Magazine, Feb. 1990, 7 pgs. cited by applicant .
"MSDL Specification Version 1.1" International Organisation for Standardisation Organisation Internationale EE Normalisation, ISO/IEC JTC1/SC29/WG11 Coding of Moving Pictures and Autdio, N1246, MPEG96/Mar. 1996, 101 pgs. cited by applicant .
Regis, Notice of Allowance U.S. Appl. No. 13/273,803, dated Sep. 2, 2014, 8 pgs. cited by applicant .
Regis, Notice of Allowance U.S. Appl. No. 13/273,803, dated May 14, 2014, 8 pgs. cited by applicant .
Regis, Final Office Action U.S. Appl. No. 13/273,803, dated Oct. 11, 2013, 23 pgs. cited by applicant .
Regis, Office Action U.S. Appl. No. 13/273,803, dated Mar 27, 2013, 32 pgs. cited by applicant .
Richardson, Ian E.G., "H.264 and MPEG-4 Video Compression, Video Coding for Next-Genertion Multimedia," Johm Wiley & Sons, US, 2003, ISBN: 0-470-84837-5, pp. 103-105, 149-152, and 164. cited by applicant .
Rose, K., "Design of a Switched Broad-Band Communications Network for Interactive Services," IEEE Transactions on Communications, vol. com-23, No. 1, Jan. 1975, 7 pgs. cited by applicant .
Sigmon, Notice of Allowance, U.S. Appl. No. 13/311,203, dated Sep. 22, 2014, 5 pgs. cited by applicant .
Sigmon, Notice of Allowance, U.S. Appl. No. 13/311,203, dated Feb. 27, 2014, 14 pgs. cited by applicant .
Sigmon, Final Office Action, U.S. Appl. No. 13/311,203, dated Sep. 13, 2013, 20 pgs. cited by applicant .
Sigmon, Office Action, U.S. Appl. No. 13/311,203, dated May 10, 2013, 21 pgs. cited by applicant .
Smith, J. et al., "Transcoding Internet Content for Heterogeneous Client Devices" Circuits and Systems, 1998. ISCAS '98. Proceedings of the 1998 IEEE International Symposium on Monterey, CA, USA May 31-Jun. 3, 1998, New York, NY, USA,IEEE, US, May 31, 1998 (May 31, 1998), 4 pgs. cited by applicant .
Stoll, G. et al., "GMF4iTV: Neue Wege zur-Interaktivitaet Mit Bewegten Objekten Beim Digitalen Fernsehen," Fkt Fernseh Und Kinotechnik, Fachverlag Schiele & Schon GmbH, Berlin, DE, vol. 60, No. 4, Jan. 1, 2006, ISSN: 1430-9947, 9 pgs. No English Translation Found. cited by applicant .
Toms, N., "An Integrated Network Using Fiber Optics (Info) for the Distribution of Video, Data, and Telephone in Rural Areas," IEEE Transactions on Communication, vol. Com-26, No. 7, Jul. 1978, 9 pgs. cited by applicant .
Jurgen_Two-way applications for cable television systems in the '70s, IEEE Spectrum, Nov. 1971, 16 pgs. cited by applicant .
Welzenbach et al., "The Application of Optical Systems for Cable TV," AEG-Telefunken, Backnang, Federal Republic of Germany, ISSLS Sep. 15-19, 1980, Proceedings IEEE Cat. No. 80 CH1565-1, 7 pgs. cited by applicant .
Yum, TS P., "Hierarchical Distribution of Video with Dynamic Port Allocation," IEEE Transactions on Communications, vol. 39, No. 8, Aug. 1, 1991, XP000264287, 7 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 14/217,108, dated Aug. 10, 2017, 14 pgs. cited by applicant .
ActiveVideo Networks, Inc., Extended European Search Report, EP15785776.4, dated Aug. 18, 2017, 8 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Search Report and Written Opinion, PCT/US2016/064972, dated Feb. 17, 2017, 9 pgs. cited by applicant .
ActiveVideo Networks, Inc., Decision to grant an European Patent, EP06772771.9, dated Oct. 26, 2017, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Intention to grant an European Patent, EP06772771.9, dated Jun. 12, 2017, 5 pgs. cited by applicant .
ActiveVideo Networks, Inc., Certficate of Grant, EP06772771.9, dated Nov. 22, 2017, 1 pg. cited by applicant .
ActiveVideo Networks, Inc., Decision to grant an European Patent, EP11833486.1, dated Oct. 26, 2017, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Certificate of Grant, EP11833486.1, dated Nov. 22, 2017, 1 pg. cited by applicant .
ActiveVideo Networks, Inc., Certificate of Grant, EP12767642-7, dated Jun. 7, 2017, 1 pg. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Article 94(3), EP14740004.8, dated Aug. 24, 2017, 7 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Article 94(3), EP15721482.6, dated Nov. 20, 2017, 7 pgs. cited by applicant .
ActiveVideo Networks, Inc., Notification of German Patent, DE602012033235.2, dated Jun. 13, 2017, 3 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Rules 161(2) and 162,EP16818840.7, dated Feb. 20, 2018, 3 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Preliminary Report on Patentability, PCT/US2016/040547, dated Jan. 2, 2018, 5 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 15/199,503, dated Feb. 7, 2018, 12 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 15/261,791, dated Feb. 21, 2018, 26 pgs. cited by applicant .
Hoeben, Final Office Action, U.S. Appl. No. 14/757,935, dated Feb. 28, 2018, 33 pgs. cited by applicant .
Visscher, Office Action, U.S. Appl. No. 15/368,527, dated Feb. 23, 2018, 23 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Under Rule 71(3), Intention to Grant, EP11833486.1, dated Apr. 21, 2017, 7 pgs. cited by applicant .
ActiveVideo Networks, Inc., KIPO'S Notice of Preliminary Rejection, KR10-2012-7031648, dated Mar. 27, 2017, 4 pgs. cited by applicant .
ActiveVideo Networks, Inc., Decision to grant an European Patent, EP12767642.7, dated May 11, 2017, 2 pgs. cited by applicant .
ActiveVideo Networks, Inc., Transmission of Certificate of Grant, EP12767642-7, dated Jun. 7, 2017, 1 pg. cited by applicant .
ActiveVideo Networks, Inc., Intention to Grant, EP06772771.9, dated Jun. 12, 2017, 5 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Article 94(3), EP14722897.7, dated Jul. 19, 2017, 7 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 13/668,004, dated Mar. 31, 2017, 21 pgs. cited by applicant .
Brockmann, Notice of Allowance, U.S. Appl. No. 14/696,462, dated Jul. 21, 2017, 6 pgs. cited by applicant .
Brockmann, Notice of Allowance, U.S. Appl. No. 14/298,796, dated Mar. 17, 2016, 9 pgs. cited by applicant .
Brockmann, Office Action, U.S. Appl. No. 12/443,571, dated May 31, 2017, 36 pgs. cited by applicant .
Hoeben, Final Office Action, U.S. Appl. No. 14/757,935, dated Apr. 12, 2017, 29 pgs. cited by applicant .
McElhatten, Office Action, U.S. Appl. No. 14/698,633, dated Feb. 10, 2017, 15 pgs. cited by applicant .
ActiveVideo Networks, Inc., Extended European Search Report, EP15873840.1, dated May 18, 2018, 9 pgs. cited by applicant .
ActiveVideo Networks, Inc., Communication Pursuant to Rules 70(2) and 70a(2), EP15873840.1, dated Jun. 6, 2018, 1 pg. cited by applicant .
ActiveVideo Networks, Inc., International Preliminary Report on Patentability, PCT/US2016/064972, dated Jun. 14, 2018, 7 pgs. cited by applicant .
ActiveVideo Networks, Inc., International Search Report and Written Opinion, PCT/US2017/068293, dated Mar. 19, 2018, 7 pgs. cited by applicant.
This application is a continuation of U.S. application Ser. No. 14/298,796, filed Jun. 6, 2014, which claims priority to U.S. Provisional Patent Application No. 61/832,069, entitled "Overlay Rendering of User Interface onto Source Video," filed Jun. 6, 2013, which are hereby incorporated by reference in their entireties.
1. A method of providing an interactive user interface for generating a blended output that includes the interactive user interface and one or more supplemental images, the method comprising: at a server remote from a client device: transmitting a video stream that includes an interactive user interface towards the client device using a first data communications channel configured to communicate video content; receiving a command that relates to a user input received through the interactive user interface; generating an updated interactive user interface in accordance with the command; in accordance with a determination that the client device is not capable of overlaying the updated interactive user interface and one or more supplemental images: determining, at the server, a state of a decoder at the client device by retrieving one or more frames buffered on the server to obtain a state that corresponds to a frame corresponding to the command related to the user input; and blending the updated interactive user interface and the one or more supplemental images determined based on the state of the decoder at the client device to generate a blended output frame with the one or more supplemental images over the updated interactive user interface, wherein each supplemental image of the one or more supplemental images is associated with a corresponding transparency coefficient and the blending is performed according to the transparency coefficient for each supplemental image of the one or more supplemental images; and transmitting the blended output frame toward the client device for display on a display device.
2. The method according to claim 1, further comprising: switching between transmitting the blended output frame and transmitting the updated interactive user interface; and transmitting the updated interactive user interface toward the client device for display on the display device.
3. The method according to claim 1, wherein an encoding specification for the video stream is an MPEG specification, an AVS specification, or a VC-1 specification.
4. The method according to claim 1, wherein the first data communications channel comprises at least one of: quadrature amplitude modulation (QAM) using a cable network infrastructure, user datagram protocol over internet protocol (UDP/IP) using an internet protocol television (IPTV) infrastructure, or hypertext transfer protocol (HTTP) using a public or private internet infrastructure, and wherein an image format of the one or more supplemental images is a bitmap (BMP) file format, a portable network graphics (PNG) file format, a joint photographic experts group (JPEG) file format, or a graphics interchange format (GIF) file format.
5. The method according to claim 1, further comprising, prior to determining, at the server, the state of the decoder at the client device that corresponds to the frame corresponding to the command related to the user input, determining that the client device is not capable of overlaying the updated interactive user interface and the one or more supplemental images.
The present invention relates to interactive video distribution systems, and more particularly to blending a source video with an interactive user interface to generate a single image, where the source video and interactive user interface are separately provided.
It is known in the prior art to provide interactive user interfaces for television programs. Such interactive user interfaces include, for example, electronic program guides (EPG) that may be manipulated to search for broadcast programs or schedule recordings. Interactive user interfaces also include simple video games, menuing systems to access video on demand, and other similar such mechanisms.
Interactive user interfaces may be combined with source video, such as video on a broadcast or cable channel. There are two broad ways to combine such interfaces with source video: scale down the source video and fill the rest of the screen with the interactive user interface, or keep the source video full-screen but overlay the user interface onto the screen. As an example of the first combination, modern EPGs often show dynamically-generated channel information with a small preview window that shows video for a current channel. As an example of the second combination, television sets often provide volume controls as elements that overlay an area of the screen, typically near the bottom or along one side, while continuing to display the underlying source video content full-screen.
The latter method to combine user interfaces with source video can itself be broken into two different categories: opaque user interfaces and translucent, or partially transparent, user interfaces. Different techniques can be used for these different categories. For example, if it is known in advance that a user interface will be opaque, then the pixels of the underlying source video content may be discarded at the beginning of the overlay process. This ability to discard pixels simplifies processing of the overlays and permits compositing of the user interface directly into the source image. For certain block-based encoding schemes, compositing can be accomplished at a block level. However, for partially transparent user interfaces, the underlying pixels must be retained and blended with the user interface.
It is also known in the art to overlay images using blending. For purposes of the present disclosure, "blending" refers to a process of alpha compositing; that is, the process of combining two colors using a transparency coefficient, a. Using this technique, each pixel of each image may be viewed as being associated with four values: three color values and one alpha value, each between 0.0 and 1.0, either by storing these values per pixel or in a lookup table such as for example a palette. If the color values are red-green-blue, for example, then these four values are denoted RGBA. Alpha blending takes as input the RGBA values of a foreground pixel and a background pixel, and produces as output a pixel having RGBA values color(output)=.alpha.(f)*color(f)+(1-.alpha.(f)*color(b) and a(output)=.alpha.(f)+.alpha.(b)*(1-.alpha.(f), where .alpha.(f) and .alpha.(b) are the transparency coefficients of the foreground and background pixels, respectively. In other words, the colors and transparency coefficients of the output are a weighted average of the foreground and background pixel, using ".alpha." as the weight. Thus, if .alpha.=0.0 in the foreground pixel, then the colors in the output pixel are the same as that of the background (that is, the foreground pixel is not visible). If .alpha. is increased from 0.0 toward 1.0, more of the foreground pixel becomes visible, until when .alpha.=1.0 the color of the output pixel is the same as that of the foreground pixel (that is, the background pixel is completely overlaid by the foreground pixel).
However, it is generally disadvantageous to blend user interfaces at the server (e.g., at a cable headend), for a number of reasons. First, a typical television provider will have hundreds of thousands or millions of subscribers, a significant portion of whom will, at any given time, require interactive user interfaces. Each subscriber may be watching a different source video, and blending all of these source videos with any number of user interfaces is a problem that does not scale well. Second, blending a user interface with a source video requires access to the pixels of the source video, but the source video that is broadcast is typically ingested from a content provider, encoded according to a transmission encoding that exceeds available computational power. Third, a significant latency may be caused by the blending process, creating an unacceptable `sluggishness` in the response of the user interface.
Various embodiments of the invention overcome the disadvantages of blending, at the server, interactive user interfaces with underlying source video in two distinct ways. First, many client devices, such as set top boxes or smart televisions, have the ability to perform alpha blending. Thus, it is possible to transmit the user interface from a remote server, such as one found at a cable headend, to the client device, on demand and out-of-band using a separate protocol, such as a modified RFB or XRT protocol. Second, even if a client device does not have the ability to perform alpha blending locally, such blending can be accelerated at the remote server through a combination of image caching and reconstruction of the client device decoder state to the point where blending becomes a scalable operation.
Some implementations include a method of providing, at a client device, an interactive user interface for generating an output, for a display, that includes a source video and an interactive user interface. The method includes receiving, at a client device remote from a server, the source video from the server using a first data communications channel configured to communicate video content, wherein the first data communications channel comprises a quadrature amplitude modulation (QAM) protocol. Furthermore, the method includes transmitting to the server a command related to an interactive user interface, and receiving, in response to the transmitting, one or more images of the interactive user interface using a second data communications channel different from the first data communications channel, wherein the second data communications channel comprises a transmission control protocol over internet protocol (TCP/IP) protocol. The source video is blended with the received one or more images to generate an output, and the output is transmitted toward a display device for display thereon.
In some embodiments, the interactive user interface comprises a menu.
In some embodiments, the received video content is encoded using an MPEG specification, an AVS specification, or a VC-1 specification. Furthermore, in some embodiments, the one or more images of the interactive user interface are encoded using a bitmap (BMP) file format, a portable network graphics (PNG) file format, a joint photographic experts group (JPEG) file format, or a graphics interchange format (GIF) file format.
In some embodiments, each image of the one or more images is associated with a corresponding transparency coefficient, and wherein blending the source video with the received one or more images comprises blending according to the transparency coefficient.
In some embodiments, wherein the blending comprises blending in a spatial domain.
In another aspect, a method includes providing, at a server, an interactive user interface for generating a output, for a display, that includes the interactive user interface and a source video. The method includes transmitting frames of a source video toward a client device, remote from the server, using a data communications channel configured to communicate video content, while simultaneously buffering in a memory of the server a plurality of encoded frames from the source video for subsequent transmission to the client device. The buffered frames include a first frame that is intra-encoded and one or more additional frames that are inter-encoded based on the first frame. Responsive to receiving from the client device a command that relates to the interactive user interface, the method includes determining a buffered frame in the plurality of buffered frames that corresponds to a time associated with the command, and blending the determined frame with one or more images of the interactive user interface to generate an output. Using the data communications channel, the output is transmitted toward the client device for display on the display device.
In some embodiments, transmitting the frames of the source video and transmitting the output frame each comprise transmitting according to a screen resolution or a screen dimension of the display device.
In some embodiments, the encoding specification is an MPEG specification, an AVS specification, or a VC-1 specification. Furthermore, in some embodiments, the one or more images of the interactive user interface are encoded using a bitmap (BMP) file format, a portable network graphics (PNG) file format, a joint photographic experts group (JPEG) file format, or a graphics interchange format (GIF) file format.
In some embodiments, the data communications channel comprises at least one of: quadrature amplitude modulation (QAM) using a cable network infrastructure, user datagram protocol over internet protocol (UDP/IP) using an internet protocol television (IPTV) infrastructure, or hypertext transfer protocol (HTTP) using a public or private internet infrastructure.
In some embodiments, each image of the one or more images is associated with a corresponding transparency coefficient, and wherein blending the determined frame with the one or more images comprises blending according to the transparency coefficient.
In some embodiments, blending the determined frame with one or more images includes (i) decoding the determined frame according to the encoding specification to generate a decoded frame; (ii) blending the decoded frame with the one or more images in a spatial domain to generate a blended frame; and (iii) encoding the blended frame according to the encoding specification to generate the output frame. Furthermore, in some implementations, encoding the blended frame comprises searching for motion vectors.
In some embodiments, the output frame is encoded according to the encoding specification.
In yet another aspect, a method includes combining, at a client device, an interactive user interface for generating a blended output, for a display, that includes the interactive user interface and one or more supplemental images. The method includes receiving, at a client device remote from a server, an interactive user interface from the server using a first data communications channel configured to communicate video content. Furthermore, the method includes transmitting to the server a command that relates to an interactive user interface, and receiving, in response to the transmitting, an updated user interface from the server using the first data communications channel, and the one or more supplemental images for supplementing the interactive user interface using a second data communications channel different from the first data communications channel. The updated user interface and the one or more supplemental images are blended to generate a blended output, and the blended output is transmitted toward the display device for display thereon.
In some embodiments, the interactive user interface comprises a source video stitched with user interface content.
In some embodiments, the encoding specification is an MPEG specification, an AVS specification, or a VC-1 specification.
In some embodiments, the first data communications channel comprises at least one of: quadrature amplitude modulation (QAM) using a cable network infrastructure, user datagram protocol over internet protocol (UDP/IP) using an internet protocol television (IPTV) infrastructure, or hypertext transfer protocol (HTTP) using a public or private internet infrastructure.
In some embodiments, the one or more supplemental images are encoded using a bitmap (BMP) file format, a portable network graphics (PNG) file format, a joint photographic experts group (JPEG) file format, or a graphics interchange format (GIF) file format.
In some embodiments, the second data communications channel comprises at least one of transmission control protocol over internet protocol (TCP/IP), remote frame buffer (RFB) protocol, and extended remoting technology (XRT) protocol.
In some embodiments, each supplemental image of the one or more supplemental images is associated with a corresponding transparency coefficient, and wherein blending the updated user interface with the one or more supplemental images comprises blending according to the transparency coefficient.
In some embodiments, blending comprises blending in a spatial domain.
In some embodiments, the command is a request for secure content, wherein the one or more supplemental images are received from a third party server, and the second data communications channel uses a secure transport protocol.
In yet another aspect, the method includes providing, at a server, an interactive user interface for generating a blended output, for a display, that includes the interactive user interface and one or more supplemental images. The method includes transmitting, at a server remote from a client device, the interactive user interface from a server using a first data communications channel configured to communicate video content, and receiving a command that relates to the interactive user interface. Furthermore, the method includes generating an updated interactive user interface, blending the updated user interface and the one or more supplemental images to generate a blended output frame, and transmitting the blended output frame toward a client device for display on a display device thereon.
In some embodiments, the method further includes transmitting the updated interactive user interface toward the client device for display on the display device thereon, and switching between transmitting the blended output frame and transmitting the updated interactive user interface.
In some embodiments, the image format of the one or more supplemental images is a bitmap (BMP) file format, a portable network graphics (PNG) file format, a joint photographic experts group (JPEG) file format, or a graphics interchange format (GIF) file format.
In some embodiments, the method includes first determining that the client device is not capable of overlaying.
FIG. 1 schematically shows a system in accordance with one embodiment of the invention;
FIG. 2 is a flowchart showing operation of a client device in the system of FIG. 1;
FIG. 3 schematically shows a sequence of frames of video in relation to an interactivity command in accordance with a second embodiment of the invention;
FIG. 4 schematically shows a system in accordance with a second embodiment of the invention;
FIG. 5 is a flowchart showing operation of a server in the system of FIG. 4;
FIG. 6A schematically shows a system in accordance with a third embodiment of the invention;
FIG. 6B is a flowchart showing operation of a client device in the system of FIG. 6A;
FIG. 7A schematically shows a system in accordance with a fourth embodiment of the invention;
FIG. 7B is a flowchart showing operation of a server in the system of FIG. 7A;
FIG. 8A schematically shows a system in accordance with a fifth embodiment of the invention; and
FIG. 8B is a flowchart showing operation of a client device in the system of FIG. 8A.
"Video" means both silent moving images and moving images accompanied by sound, except where otherwise indicated.
An "encoding specification" is a specification according to which video data are encoded by a transmitting electronic device and decoded by a receiving electronic device. Examples of encoding specifications are MPEG-2, MPEG-4, AVS, and VC-1.
A "client device" is an electronic device capable of receiving and decoding data according to an encoding specification for display on a display device. Examples of client devices include cable and satellite set top boxes, some video game consoles, and some televisions.
FIG. 1 schematically shows a system in accordance with one embodiment of the invention. This embodiment includes a client device 10 that provides an output display signal to a display device 11. The client device 10 generally receives signals, such as linear broadcast television signals, from one or more servers 12, by way of a first data communications network 131. The client device 10 also receives images that form an interactive user interface, such as electronic program guide signals, by way of a second data communications network 132. The client device 10 then combines these signals to generate the output display signal. The aforementioned elements are now described in more detail.
The client device 10 may be implemented as a set top box, a video game console, a television, or other electronic device known in the art. The client device 10 includes an overlay module 101 that is capable of overlaying an image on an input video signal to generate an output video signal as a sequence of composite images. The operation of the overlay module 101 is described in more detail in connection with FIG. 2. The client device 10 also includes a video decoder 102, which is capable of decoding audiovisual data that was encoded according to an encoding specification. Such video decoders are well known in the art, and may be implemented as an integrated circuit. Audiovisual data typically are encoded to reduce size for transmission through the data communications network 131.
The client device 10 also has several input/output (I/O) ports 103. One I/O port 103 is used to receive audiovisual data from the data communications network 131, while another is used to receive images from the data communications network 132. Another I/O port may be used in some embodiments to receive images that comprise an interactive user interface. In other embodiments, the same I/O port is used to receive both the audiovisual data and the interactive user interface images. Another I/O port is used to accept user input in the form of commands. Some commands may instruct the client device 10 to tune to a different channel (i.e., to receive different audiovisual data from the data communications network 131 or from another data network such as the Internet). Other commands may instruct the client device to record audiovisual data, either as it arrives at the client device 10 or at a future time and on a specified channel. Some commands will cause the display of an interactive user interface, while other commands will not. Various embodiments of the present invention are directed toward processing of commands that cause the display of such a user interface. The I/O ports 103 may be implemented using hardware known in the art, such as an IR receiver to interface with a remote control, a coaxial jack to interface with a cable television distribution network, a wired or wireless Ethernet port to interface with an Ethernet network, a video jack to provide the output display signal to the display device 11, and so on. The display device 11 itself may be implemented as a standard CRT, LCD, LED, or plasma monitor as is known in the art, or other similar device.
The one or more servers 12 may be implemented using computer equipment known in the art; however their functions are novel when operated in accordance with various embodiments of the present invention. In accordance with some embodiments of the invention, a large number of servers 12 may be present, and cooperate to provide the functions described below. However, for convenience and clarity, the remainder of the detailed description will assume that only one server 12 is present.
The server 12 includes a number of audio, video, and/or audiovisual data sources 121, an application execution environment 122, and an encoder 123. Note that other components may be used in an implementation of the server 12, although these have been omitted for clarity. These components are now described in more detail.
The audio/video data sources 121 may be, for example, non-linear multimedia data stored on a non-volatile storage device in the form of a movie, television program, television commercial, game graphics and sounds, user interface sounds, or other such form. The data sources 121 also may include linear multimedia data sources, such as a television broadcast stream received live by antenna or private network.
An application execution environment 122 executes an interactive application on behalf of a user. The application may be, for example, a menuing system, a video game system, or other interactive application. The environment 122 responds to input interactive commands by providing images to the client device 10 using data communications network 132. The environment 122 includes at least application logic 1221, a source of images 1222, and an image cache 1223. Application logic 1221 may be implemented as an executable file or a script that provides a state machine for operating an interactive user interface. Any format of application file may be used as application logic 1221; for example, a hypertext markup language (HTML) file that includes JavaScript may be used, or a compiled binary file may be used.
The application logic 1221 may dynamically generate one or more images 1222 that comprise the interactive user interface. The images 1222 often persist in a volatile memory of the server 12 for speed of access, for example in an image cache 1223. The images 1222 may be generated by the application execution environment logic 1221 according to a screen resolution or a screen dimension of the display device 11, which may be statically configured or may be determined dynamically when the client device 10 first establishes a communications session with the server 12. Typically, for efficiency purposes, the application logic 1221 will transmit images from the image cache 1223 if possible, and dynamically create images 1222 for transmission only if they are not already in the image cache 1223. The use of a cache 1223 advantageously permits interactive user interface images to be reused by the server 12 (or by other servers) between different requests for the user interface, even if those requests come from different end users or at different times. Images in the image cache 1223 typically are indexed using a hashing function defined by the environment 122. The use of the hashing function permits many images to be quickly retrieved from the image cache 1223, advantageously providing increased scalability. Additionally and/or alternatively, in some embodiments, server(s) 12 will transmit references to the images (such as Uniform Resource Locators or URLs), as opposed to the images themselves, so that the client can retrieve them on demand (e.g., by means of HTTP). Such embodiments would be advantageous, as an intermediate network cache (not shown), accessible through second data communications channel 132, may be used to store reusable images closer to the client device.
The encoder 123 encodes the source audiovisual data according to an encoding specification, such as MPEG, AVS, or VC-1. The encoder 123 and the decoder 102 use the same encoding specification, so that the encoded audiovisual data may be decoded once it passes through the data communications network 131. In the case that the source audiovisual data are already encoded, the encoder acts as a simple pass-through. However, in the case that the source audiovisual data are not in a format decodable by the decoder 102, the encoder 123 transcodes the data into a decodable format.
As can be seen from FIG. 1, the encoded audiovisual data (from the encoder 123) and the user interface images (either from images 1222 or the cache 1223) travel to the client device along two different data channels. The first data channel through the first data communications network 131 is designed specifically to communicate video content. Thus, for example, the network 131 may include a cable network infrastructure that deploys quadrature amplitude modulation (QAM), as is known in the art. Alternately, the network 131 may have an internet protocol television (IPTV) infrastructure that uses user datagram protocol over internet protocol (UDP/IP) to communicate encoded video. In yet another implementation, the network 131 may be part of a public or private internet infrastructure, and use hypertext transfer protocol (HTTP) tunneling to communicate the encoded video.
By contrast, the second data communications network 132 may be designed to communicate images, rather than video. In particular, this means that the second network 132 may operate on a much lower bandwidth or a higher reliability than the first network 131. Thus, for example, the second network 132 may support data channels using the transmission control protocol over internet protocol (TCP/IP), the remote frame buffer (RFB) protocol, or the extended remoting technology (XRT) protocol. Images that are transmitted on the second network 132 may be encoded, for example, using a bitmap (BMP) file format, a portable network graphics (PNG) file format, a joint photographic experts group (JPEG) file format, or a graphics interchange format (GIF) file format. The use of PNG is particularly advantageous, as each pixel is stored with a corresponding transparency coefficient (a value).
FIG. 2 is a flowchart showing operation of a client device 10 in the system of FIG. 1 in accordance with an embodiment of the invention. In particular, FIG. 2 illustrates a method of providing, in the client device 10, an interactive user interface for simultaneous display with a source video on a display device 11. The method begins with a process 21 in which the client device 10 receives source video using a first data communications channel 131. In a typical embodiment, the client device 10 will display this source video as it arrives on the display device 11, as is known in the art. Next, in process 22 the client device 10 transmits a command related to an interactive user interface to a server 12. This command may be transmitted, for example, in response to the client device 10 receiving on an I/O port 103 a signal that a button or buttons on a remote control has been pressed. The button or buttons may be provided on the remote control to call up an interactive program guide, a video game, or other interactive application.
In process 23, the client device 10 responsively receives one or more images of the interactive user interface, using a second data communications channel 132. For example, the images might include a number of buttons, switches, or dials for collective simultaneous display as a user interface. Alternately, the images might be designed to be displayed sequentially, as in the case of a "trick play" interface that includes a video timeline and a mark indicating a current time stamp. Seeking through the video may be performed by pressing a fast-forward or rewind button on the remote control, and movement of the timing mark along the timeline typically may be sped up by repeated button presses. The images may come from the images 1222 or the image cache 1223 of the application execution environment 122.
Next, in a process 24, the client device 10, and in particular the overlay module 101, alpha blends the source video with the received images to generate an output frame of pixels. In accordance with various embodiments of the invention, the received interactive user interface images are considered to be partially transparent foreground images (0.0<.alpha.<1.0), and frames of the source video are considered to be opaque background images (.alpha.=1.0). The choice of a for the user interface images advantageously may be made to be approximately 0.5, so that the interactive user interface appears evenly blended with the background source video. Or, the value of a may be varied on a per-pixel basis (i.e., per pixel alpha blending) within each image; for example, providing a downward a-gradient at the edges of a user interface image will produce an effect of the image `fading into the background` at its edges. Global alpha blending and per pixel alpha blending may be combined by multiplying each per pixel alpha blending value with the global alpha blending value before the blending process is applied. The blending process 24 is performed using an appropriate received user interface image or images with respect to each frame of the source video for as long as the interactive user interface should be displayed on the screen, thereby providing a continuously-displayed interactive user interface.
Finally, in process 25, the client device 10 transmits each output frame toward the display device 11 for display. An I/O port 103 may be used in processes 21, 22, 23, 25 to receive or transmit data. A computing processor may be used in process 24 to perform the required blending.
The above embodiments are preferred because the image cache 1223 may be used to increase scalability of the content delivery platform provided by the server 12 (or a server cloud). This is true because it is feasible to cache individual user interface images separately from their underlying source videos, while it is generally infeasible to cache a vast number of pre-blended images due to limited storage space. The separate caching of user interface images, in turn, is a result of the ability of the client device 10 to receive these images using an I/O port 103 and perform blending in the overlay module 101.
In some situations, it may be impossible to use these embodiments, because a client device 10 may not have the necessary I/O ports 103 or an overlay module 101. In these situations, it is instead necessary to perform blending at the server 12, rather than the client 10, and such blending has its own challenges.
One such challenge is that the user interface images must be blended by the server 12, but can be sent to the client device 10 only as encoded audiovisual data. Therefore, it is necessary to decode the source video into a spatial domain (i.e., as a frame of pixels), blend the user interface images with the source video in the spatial domain, then re-encode the blended image according to the encoding specification. These processes require server computational capacity, and do not scale well.
Another challenge is that there is noticeable latency between the time at which the interface command occurs and when the user interface can be displayed. This challenge is illustrated by consideration of FIG. 3, which schematically shows a time sequence 31 of frames of video in relation to an interactivity command 32. In this figure, a sequence 31 of frames includes a number of individual video frames 311-317. The frames are labeled by a frame type, which may be either intra-encoded or inter-encoded. An intra-encoded frame encodes video data according to data found only in the frame, while an inter-encoded frame encodes video data according to data found in the given frame and in surrounding frames. For purposes of clarity, MPEG frame types are used in the figures and detailed description to provide an example implementation, but any encoding specification may be used in accordance with an embodiment of the invention.
The sequence 31 of frames includes two types of frames: I-frames that are intra-encoded and P-frames that are inter-encoded. I-frames are encoded using image information found only in themselves. Thus, I-frames encode a full-screen image, which is useful to indicate a `scene change` or to eliminate display artifacts. Two frames 311, 317 are I-frames. P-frames are encoded using information found in the previous image by estimating movement of pixels using two-dimensional "motion vectors". Thus, P-frames are useful for predicting movement fixed or slow-moving `camera pan` images where most of the image content of the previous frame is present in the next frame. This relationship between P-frames and their predecessor frames is indicated by the backwards-facing arrows in FIG. 3. Frames 312-316 are P-frames. MPEG also defines a B-frame, not shown in FIG. 3, which interpolates both forward and backward between other frames.
Suppose an interface command 32 arrives at the server 12 when a P-frame 316 is being displayed on the display device 11. Because it is inter-encoded, the information in this P-frame is insufficient by itself to reconstruct the complete image being displayed (i.e., to reconstruct the decoder state). In fact, the information necessary is found in a combination of the frames 311-316. One could introduce a latency 33 between the time of the command 32 and the next I-frame 317, at which time the overlay image is blended 34. However, if the group of pictures 31 contains two seconds worth of source video, the average wait time from the command 32 to the next I-frame 317 (and the appearance of the user interface) is one second, which is unacceptably unresponsive. Therefore, in various embodiments of the invention, all of the data in each group of pictures 31 (that is, from one intra-encoded frame until the next one) are buffered in the server 12 before being transmitted to permit blending of the interactive user interface images with the currently-displayed image from the source video.
The server 12 uses buffered frames to simulate, for blending, the state of the decoder 102 in the client device 10. This process is illustrated by the sequence 35, in which an encoder in the server 12 constructs the state of the decoder 102. The server 12 retrieves the first frame 311 of the buffered frames, and uses it as an initial simulated state 351. The server 12 then retrieves the second frame 312 of the buffered frames, and applies its data to the initial simulated state 351 to obtain a second simulated state 352. The server 12 retrieves the third frame 313 of the buffered frames, and applies its data to the second simulated state 352 to obtain a third simulated state 353. This process continues until the simulation reaches a state 356 that corresponds to a frame 316 corresponding to a time associated with the command 32. Once the server 12 has recovered the state of the decoder 102, it may perform blending as described above in connection with element 24 of FIG. 2.
FIG. 4 schematically shows a system in accordance with an embodiment of the invention in which the server 12 performs blending. The disclosure of FIG. 4 overlaps to large extent with that of FIG. 1, so only the changes will be remarked upon here. As noted above, in the scenario under consideration, the client device 10 in FIG. 4 lacks an overlay module 101 found in FIG. 1. Therefore, the server 12 includes, in addition to the encoder 123 of FIG. 1, a decoder/blender 124 for decoding and blending source video with an interactive user interface. Note that while the functions of decoding and blending are combined in decoder/blender 124 for purposes of this disclosure, these functions may be implemented in separate hardware or software. Also as described above, the server 12 further includes a buffer memory 125 for buffering frames of source video data. During ordinary operation of the system of FIG. 4, most frames of source video data buffered in the buffer memory 125 are discarded without being blended, and the decoder/blender 124 acts as a simple pass-through. However, when a user provides an interactive command to the application execution environment 122, the environment 122 provides images to the blender 124 (either preferably statically from its cache 1223, or dynamically from the image generator 1222) for blending with the buffered video. The decoder/blender 124 decodes the source video data and simulates the state of the decoder 102 as described with respect to element 35 of FIG. 3. The decoder/blender 124 then blends the interactive user interface images into the source video, one frame at a time. The decoder/blender 124 provides an output to the encoder 123, which encodes the data according to the appropriate encoding specification for transmission to the client device 10.
FIG. 5 is a flowchart showing operation of a server in the system of FIG. 4. In particular, FIG. 5 shows a method of providing, in a server 12, an interactive user interface for simultaneous display with a source video on a display device 11. In a first process 51, the server 12 transmits frames of the source video toward the client device 10 for display. Simultaneously, in a second process 52, the server 12 buffers frames from the source video for subsequent transmission. In process 53, the server 12 receives from the client device 10 a command 32 that relates to the interactive user interface. In process 54, the decoder/blender 124 determines a buffered frame 316 in the buffer memory 125 that corresponds to a time associated with the command 32. In process 55, the decoder/blender 124 blends the determined frame with one or more images of the interactive user interface received from the application execution environment 122 to generate an output frame that is subsequently encoded by the encoder 123. Then, in process 56, the server 12 transmits the output frame toward the client device 10 for display on the display device 11.
Note that the encoder 123 may be required to do a motion vector search after blending. There are several optimizations that can be performed to speed up this process. In a first optimization, the encoder 123 could make use of motion information found in the original video frame when it was decoded by the decoder/blender 124. However, the encoder 123 must verify whether the same motion is still present in the blended image due to the presence of the interactive user interface. In a second optimization, the source video images could be divided into rectangular areas, and motion vectors for each area are encoded separately. In this case, motion vectors for rectangles that do not intersect the user interface are unaffected by the blending, and no additional motion vector search is required for these rectangles.
FIG. 6A schematically shows a system in accordance with an embodiment of the invention in which overlay images are used to supplement a streamed interactive user interface. In U.S. application Ser. No. 12/443,571 ("Method for Streaming Parallel User Sessions, System and Computer Software"), the contents of which are hereby incorporated by reference in its entirety, a system is disclosed where an interactive user interface is streamed to a client device over a first data communications channel. The streamed interactive user interface is realized by stitching a plurality of fragments and streams into a single compliant audiovisual stream. It has been identified that for a number of reasons it is beneficial to overlay images over an encoded stream instead of encoding them in the stream, which also holds for cases in which the audiovisual stream is an interactive user interface. For example, it is beneficial to overlay images in cases involving a sprite-like user interface element (e.g., a cursor). Such a user interface element is generally arbitrarily placed on the screen and it may be more efficient to decouple the element from the interactive user interface by overlaying images. In particular, if the user interface element was instead encoded (e.g., by the fragment encoder) and subsequently stored in cache, the cache would quickly reach capacity because a sprite-like user interface element, unlike some other user interface elements (e.g., a menu), does not have a predefined position. Another example may be that the interactive user interface has a partial screen video element over which another user interface element is supposed to be rendered. In this case it is more efficient from a scalability point of view to render only the new interface element as overlay image(s).
The system disclosed in FIG. 6A is fundamentally the same as that shown in FIG. 1. Here, the client device 60 receives an interactive user interface via a first data communications channel 63 from a server 62. In some embodiments, server 62 runs an application in the application execution engine 621 that generates fragments by means of a fragment encoder 630; caches these fragments in a cache 632; and combines these (cached) fragments by means of a stitcher 622 (otherwise known as an assembler) to generate, and subsequently stream, an interactive user interface via the first data communications channel 63 to the client device 60 (as described in, U.S. application Ser. No. 12/443,571 ("Method for Streaming Parallel User Sessions, System and Computer Software")). Optionally, in some embodiments, the interactive user interface is directly encoded by an encoder of server 62 (not shown in FIG. 6A) from pixel data. The interactive user interface may be supplemented by the generation of images 634 that are to be overlain by the client device 60. These images may also be stored in a cache 632 for reuse across sessions in the same way as fragments are reused across sessions. For example, in some implementations, the interactive user interface includes a source video with images from cache 632 overlaid. The images may be sent via a second data communications channel 64 to the I/O ports 601 of client device 60. Additionally and/or alternatively, in some embodiments, server(s) 62 will transmit references to the images (such as Uniform Resource Locators or URLs), as opposed to the images themselves, so that the client can retrieve them on demand (e.g., by means of HTTP). Such embodiments are advantageous, as an intermediate network cache 641, accessible through second data communications channel 64, can be used to store reusable images closer to the client device. The stream received from server 62 is decoded in the decoder 602 and combined with the images received or retrieved from server 62 in the overlay module 603 for display on 61 as described in the embodiment described in relation to FIG. 1. In some implementations, client device 60 switches between 1) receiving the interactive user interface from the stitcher, and 2) blending the the interactive user interface from the stitcher with overlay images.
FIG. 6B is a flowchart showing operations of a client device in the system of FIG. 6A. The flow chart is very similar to the operations described in the flow chart in FIG. 2. However, instead of receiving a source video using the first data communications channel, the client device receives (6000) the interactive user interface via the first data communications channel. In some embodiments, the interactive user interface is a video stream, such as an MPEG video stream. Next, a command related to that interactive user interface is transmitted (6010) to the server. The client may subsequently receive (6020) updates to the interactive user interface via the first data communications channel and/or supplemental images from the same server to supplement the interactive user interface. The remaining processes 24 and 25 are the same as those described with respect to FIG. 1.
In some embodiments, since the first data communications channel and the second data communications channel are completely independent channels, the graphical information transmitted over both data channels is likely to be related. Therefore, special care must be taken when the images are combined with the video stream representing the interactive user interface. A loosely coupled synchronization mechanism, such as for example a presentation timestamp and timeout for each image, may be used to synchronize the display of images with the streamed interactive user interface.
FIG. 7A schematically shows an alternative embodiment of the system described in FIG. 6A. The system disclosed in FIG. 7A is similar to the systems depicted by FIGS. 4 and 6A. Here, the server (specifically, overlay module 724 of server 72), and not the client device, overlays images over the encoded stream. In other words, the blending occurs at the server, as described in relation to FIG. 4.
As illustrated, in some implementations, client device 703 does not include an overlay module. Moreover, as shown, the system of FIG. 7A does not utilize a second data communications channel.
As in the system of FIG. 6A, stitcher 722 generates an interactive user interface by combining fragments, generated by fragment encoder 730, and stored in cache 732. Overlay module 724 overlays images 734 over the resulting interactive user interface received from stitcher 722. As illustrated, client device 70 then receives the encoded stream, which includes interactive user interface and overlay images 734, via a first data communications channel 73 from server 72. In optional implementations, server 72 (or, alternatively, overlay module 724) is configured to switch between transmitting (i) the encoded stream including the interactive user interface and overlay images 734, and (ii) only the interactive user interface. Alternatively, in some implementations, overlay module 724 and stitcher 722 exist and operate as a single component of server 72.
FIG. 7B is a flowchart showing operations of a client device in the system of FIG. 7A. The flow chart is very similar to the operations described in the flow chart in FIG. 6B, but written with respect to a server (e.g., server 72) that is configured (e.g., overlay module 724) to overlay images. In process 7000, the server transmits the interactive user interface via a first data communications channel. Next, in process 7010, the server receives a command related to the interactive user interface. In process 7020, the server generates an updated interactive user interface. Further, in process 7030, the server blends the updated interactive user interface with supplemental images to generate a blended output frame which, in process 7040, is transmitted towards the client device. As described above, in optional implementations, the server switches between transmitting (i) the blended output frame including the interactive user interface and overlay images, and (i) the interactive user interface.
FIG. 8A schematically shows an alternative embodiment, similar to the system described in FIG. 6A, in which the supplemental overlay images are sourced from a third party server. The embodiment provides a strict separation between an interactive user interface and information from a third party, by conveying the interactive user interface and information from a third party over separate data communications channels. An example of a system requiring such a separation is a banking application where the interactive user interface is the same for every user, except for account related information that is sent directly to the end user as supplemental images (e.g., supplemental images sent by third party server 85) over a secure data communications channel (e.g., second data communications channel 84).
The system disclosed in FIG. 8A is very similar to the system depicted by FIG. 6A. The main difference being that one or more images originate from a third party server 85, and are sent as supplemental images to client device 80 over second data communications channel 84. In some embodiments, second data communications channel 84 is a secure channel (e.g., a secure transport protocol is used for the images, such as HTTPS). The application may use application logic 834 to liaise with application logic 840 of an application 844 on a third party server 85 via a communication channel 87 to generate one or more images 842 that supplement the interactive user interface with third party information.
FIG. 8B is a flowchart showing operations of a client device in the system of FIG. 8A. The flow chart is similar to the flow chart in FIG. 6B. Here, the device transmits (8020) a request for secure content, and supplemental images are received (8030) from a third party server over a second data communications channel, where, in some embodiments, the second data communications channel uses a secure transport protocol.
The embodiments of the invention described above are intended to be merely exemplary; numerous variations and modifications will be apparent to those skilled in the art. All such variations and modifications are intended to be within the scope of the present invention as defined in any appended claims. For those skilled in the art it will also be evident that it may be beneficial for systems to switch between the embodiments of the invention on demand.
Previous Patent US 10,200,743 | Next Patent US 10,200,745