Abstract:
A method of building an interactive display related to TV programs in an internet protocol television (IPTV) environment includes parsing a plurality of TV programs provided in an IPTV environment for display to a user. The step of parsing includes capturing an image in each TV program, identifying video content in each captured image, interpreting the video content, capturing audio data, identifying an audio content in each captured audio data, and interpreting the audio content for each captured audio data. The method further includes the steps of identifying a selected TV program by the user; determining a correlation level between the interpreted video and audio content of the selected TV program and the interpreted video and audio content of each of the other TV programs; determining TV programs other than the selected TV program; and building and presenting a mosaic display on a TV display including the selected TV program.

Description:
CROSS REFERENCE TO RELATED APPLICATION 
     This application is related to commonly-owned, co-pending European Patent Application No. EP08305760 filed on Oct. 31, 2008, the contents and disclosure of which is expressly incorporated by reference in its entirety. 
     FIELD OF THE INVENTION 
     The present invention relates to a method, and system for implementing said method, for creating a dynamic TV display, and more specifically, a dynamic and personalized TV display in an internet protocol television (IPTV) environment. 
     BACKGROUND OF THE INVENTION 
     Television (TV) broadcasts are currently designed to deliver a consistent and static product for all users. Typically, a user watching television selects from viewing a multiplicity of prescheduled broadcasts. One shortcoming of current TV broadcasts is that there is little control by a user pertaining to how channels are presented. A further disadvantage of current TV broadcasts is that there is no relation between the presentation of channels and a users channel preference. 
     The term IPTV widely used in this document stand for Internet Protocol Television and is an umbrella term describing television (TV) and video delivered using internet technology instead of normal linear radio frequency (RF) broadcast. An IPTV solution is based on three groups of components as described below. Referring to  FIG. 1  an illustrative IPTV architecture  10  includes a triple play environment is one in which a plurality of services is provided to a user over an Internet Protocol (IP) transmission medium or IP network or backbone  14 . The plurality of services provided to an end user  18  over such a service includes, but are not limited to, TV live, Video, Video on demand (VOD), mail, games, voice over IP (VoIP), notification, chat, video conferencing, etc. The end user  18  may have a personal computer (PC)  19   b , a set-top box  19   c  for a TV set  19   e  in their home  19   d , all connected to a modem  19   a  for accessing the IP backbone  14  such as the Internet. 
     The head end component  22  includes a content provider and specific equipment needed to transmit and receive the digital assets (for example, satellite receiver  24 , and encoders  26 ). Such equipment ensures the transport of digital assets onto the IP network  14 . The head end component  22  provides a platform for the live content available and can also manage the dynamic information sent by a broadcaster  32  for an EPG (Electronic Programming Guide). Thus, the broadcaster  32  is the main link between the head end component  22  and a middleware component  42 . 
     The middleware component  42  includes infrastructure ensuring the delivery of the digital assets to the end users, including, for example, IP network infrastructure, portal servers, data storage, and video on demand servers. The customer premises as the end user may include equipment or components installed at a user&#39;s home for consuming of the digital assets by the end users. Such devices are heterogeneous as the list of services provided to the user are numerous, and may include, for example, phones, personal digital assistant (PDA), set-top boxes, personal computers (PCs), digital subscriber line (DSL) modems, and/or wireless access point. 
     The IPTV architecture  10  shown in  FIG. 1  provides every viewer with a personal relationship with a broadcaster through a transactional request/response mechanism, rather than picking up a broadcast sent to everyone. IPTV networks generally are based on one of two models, a closed model using a private IP network (local area network (LAN) or DSL, for example, cable TV), and an open model, using the public internet. Typically, when a user subscribes to an IPTV offer they receive a modem and a box which sits on or near the TV  19   e , or a set-top box  19   c . The IPTV service provides, for example, mail, liveTV, a program guide, personal video recording and video on demand provided by the service provider (for example, Verizon™, Telefonica®, France Telecom®). Live TV channels in an IPTV environment are transmitted to consumer set-top boxes  19   c  using IP multicasting technology. Each channel uses a different multicast group and the internet group management protocol (IGMP) protocol is used to leave or join a given group based on the TV channel selected. 
     IP multicast provides a method for one-to-many delivery of information packets (i.e., formatted block of data). Unicast is sending information packets to a single destination. Comparatively, broadcasting refers to transmitting a packet that will be received by every device on a network, limited to a broadcast domain. To achieve one-to-many delivery using IP unicast traffic, each information packet or datagram (i.e., from a service which does not notify the user if delivery fails) is sent to all nodes in a group, and thus is sent multiple times. Thus, a single datagram is sent, but all nodes process it, even those that are not interested. Broadcast delivery service is unsuitable for internetworks, as routers are designed to prevent the spread of broadcast traffic. With IP multicast, a single datagram is sent and forwarded across routers only to the network segments containing nodes that are interested in receiving it. 
     IP multicast traffic may be used for audio and video teleconferencing, distance learning, and data transfer to a large number of hosts. Mosaic displayed channels are specific TV channels gathered into a matrix, or mosaic display, of X by Z thumbnails of live channels available on the network on a single video channel. A user may then navigate through each thumbnail in the mosaic display to select a channel for viewing. The specific channels are encoded at the head end part of the IPTV infrastructure using dedicated hardware components mixing X by Z channels into a single video stream with X by Z audio channels. On existing digital video broadcasting (DVB) distribution channels (DVB-T, DVB-S, etc. . . . ) all the channels presented in the mosaic are statics, that is, the list of channels per mosaic are based on the head end configuration. The mosaic may be built by the head end component  22  which collects all the TV channels. The head end component  22  reencodes the channels in one stream with a smaller format. Thus, the user typically enters a selection provided on the TV screen allowing scrolling through pages of mosaic TV channels and allows a selection of a channel. One disadvantage of a static mosaic is that the user does not have control of the mosaic and/or is unable to selectively change the mosaic. Another disadvantage of the static mosaic is the cumbersome manner that a user has to scroll though many pages of channel. Further, the channel description must include textual description as the image may not adequately indicate the channel content. 
     It would therefore be desirable to provide a method, and system implementing said method, for providing a TV display which reflects the user&#39;s preferences. It would further be desirable for the display to dynamically reflect the user&#39;s preferences in TV viewing. 
     SUMMARY OF THE INVENTION 
     In an aspect of the invention, a method of building an interactive display related to TV programs in an internet protocol television (IPTV) environment comprises: parsing a plurality of TV programs provided in an IPTV environment for display to a user, wherein the step of parsing includes: capturing an image in each TV program; identifying video content in each captured image wherein the video content comprises at least one object; interpreting the video content for each captured image; capturing audio data for each of the TV programs; 
     identifying an audio content in each captured audio data wherein the audio content comprises text and/or sound; and interpreting the audio content for each captured audio data; 
     identifying a selected TV program by the user; determining a correlation level using predefined rules between the interpreted video and audio content of the selected TV program and the interpreted video and audio content of each of the other TV programs; determining TV programs other than the selected TV program using the correlation level of the TV programs; and building and presenting a mosaic display on a TV display with the selected TV program and the plurality of other TV programs. 
     In a related aspect, the steps may be repeated and/or the method may further includes a first step of: requesting the subsequent steps by a user. In another related aspect, the display may be a mosaic having a plurality of interactive areas. The method may further include: selecting the interactive areas using a head end device which provides a plurality of TV programs in an IPTV environment. 
     In another aspect of the invention, a system for building an interactive display related to TV programs in an internet protocol television (IPTV) environment includes a computer having a computer program stored on a computer readable medium for parsing a plurality of TV programs provided in an IPTV environment for display to a user. A metadata aggregator captures an image in each TV program. The meta data aggregator: identifies video content in each captured image wherein the video content comprises at least one object, interprets the video content for each captured image; capturing audio data for each of the TV programs, identifies an audio content in each captured audio data wherein the audio content comprises text and/or sound, and interprets the audio content for each captured audio data. The computer program further identifies a selected TV program by the user, and determines a correlation level using predefined rules between the interpreted video and audio content of the selected TV program and the interpreted video and audio content of each of the other TV programs; determining TV programs other than the selected TV program using the correlation level of the TV programs. A mosaic builder electrically communicates with a TV of the user and the metadata aggregator. A TV display of the user&#39;s TV presents a mosaic display using the mosaic builder. The mosaic display includes the selected TV program and the plurality of other TV programs. 
     In a related aspect, the display is a mosaic having a plurality of interactive areas. The system may further include: a head end device providing selection of the interactive areas which provides a plurality of TV programs in an IPTV environment. The metadata may be gathered from a TV channel provider using an electronic program guide. The system may also include: a thumbnail generator for decoding several IPTV video streams simultaneously. Further, metadata for non live TV channels may be generated from video demand catalogs and/or other video server providers and used by the metadata aggregator. 
     In another aspect of the invention, a computer program product comprises a computer readable medium having recorded thereon a computer program for enabling a processor in a computer system to build an interactive display related to TV programs in an internet protocol television environment, the computer program performs the steps of: parsing a plurality of TV programs provided in an IPTV environment for display to a user, wherein the step of parsing includes: capturing an image in each TV program; identifying video content in each captured image wherein the video content comprises at least one object; interpreting the video content for each captured image; capturing audio data for each of the TV programs; identifying an audio content in each captured audio data wherein the audio content comprises text and/or sound; and interpreting the audio content for each captured audio data; identifying a selected TV program by the user; determining a correlation level using predefined rules between the interpreted video and audio content of the selected TV program and the interpreted video and audio content of each of the other TV programs; determining TV programs other than the selected TV program using the correlation level of the TV programs; and building and presenting a mosaic display on a TV display with the selected TV program and the plurality of other TV programs. 
     In a related aspect, the steps may be repeated and/or the program further includes a first step of: requesting the subsequent steps by a user. Further, the display may be a mosaic having a plurality of interactive areas. The computer program product further including the step of: selecting the interactive areas using a head end device which provides a plurality of TV programs in an IPTV environment. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof which is to be read in connection with the accompanying drawings, in which: 
         FIG. 1  is a perspective view illustrating a prior art communications system; 
         FIG. 2  is a front elevational view of a mosaic TV display; 
         FIG. 3  is a front elevational view of a mosaic TV display according to one embodiment of the invention; 
         FIG. 4  is a block diagram of a system according to one embodiment of the invention for providing the mosaic shown in  FIG. 3 ; 
         FIG. 5  is a flow chart of an embodiment of the present invention employing the system shown in  FIG. 4 ; and 
         FIG. 6  is the embodiment of the mosaic shown in  FIG. 3  with a title menu and exemplary pictures in the mosaic. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     The system and method of the present invention creates meta data (i.e., information about data) on TV channels received by a user on their TV. The audio and the video streams are parsed to create a database of the meta data. The meta data may be defined according to the audio part of the TV channels or based on objects included in the TV channels. The most pertinent TV channels are proposed to the user according to what the user is watching. In one embodiment of the present invention, a mosaic format as shown in  FIG. 2  is presented to the user. The mosaic display dynamically presents the most pertinent TV channels based on their current watching and interests. In parallel, the TV channels proposed are recorded on a network personal video recorder (NPVR). Thus, the system of the present invention presents the user with channel selections, for example, on the TV screen which allows the user to select a channel by clicking on the TV screen, e.g., an interactive link or thumbnail. Thereby, the user&#39;s TV will go to the channel selected. 
     Referring to  FIG. 3 , another embodiment of the system of the present invention includes a dynamic mosaic presented on a graphical user interface (GUI)  150  of a video channel composed of one main TV channel  154  and related TV channels  162  based on user preferences. The dynamic mosaic can be used like a video browser where a current video channel is referenced as the main video channel  154  in the center of the mosaic and all the others channels aggregated on the mosaic are related to one or more attributes of the main TV video channel. When a user navigates to another TV video channel other than the main video channel  154 , the mosaic  150  is dynamically rebuilt based on the new main TV video channel  154  attributes and metadata together with a user profile. 
     As shown on  FIG. 3 , the mosaic GUI  150  includes the main video channel  154  in the center of the mosaic  150  which is currently the preferred video channel. An area  158  of the mosaic  150  includes a list of categories of TV channels available through the mosaic application  294  stored in a middleware component  290  on a computer readable medium  292  and communicating with a processor  296  (shown in  FIG. 4 ) in a computer system  289 . For example, TV channels available may include movies, shows, sports, and soap operas. Area  158  may be a hierarchical organization with categories and sub-categories, for example, sports, including tennis, soccer, and skiing. Each category and sub-category may have a new mosaic generated based on the user selection. A plurality of TV channels  162  are related to the main TV channel  100  and positioned around a perimeter of the main TV channel  100 . Keywords are determined from attributes of the main channel  100 , to generate a keyword list of related TV channels which is displayed on the mosaic application  292  based the keywords pertinence to the main TC channel  100 . A textual information area  166  of the mosaic  150  provides textual information related to the main TV channel  154 , for example, program name, timeframe, summary, and the like. 
     Referring to  FIG. 4 , system architecture  200  according to an embodiment of the invention includes components to implement the dynamic mosaic application  294  of the present invention. A metadata aggregator  250  gathers each TV channel and a list of related materials composed of other TV channels with relevant similarity with the current one and ordered by importance. The list of related TV channels can be made of other live feeds available on the IPTV infrastructure, but can also be gathered from other sources like video on demand catalogs, network personal video recording (NPVR) providers or any other accessible video server like internet video provider. The metadata for live TV channels is retrieved, in one instance, by using metadata embedded inside the live feeds, where for each channel some basic information can be retrieved, for example, title, genre, summary, timeframe. Alternatively, metadata can be gathered from a professional TV channel provider using an electronic program guide EPG  210  which provides additional information on current and forecast programs including pictures, resume, actors, etc. Additional mechanism for generating metadata information include audio recognition onto live feeds, generating dynamic keywords, and image recognition on broadcasted video to generate metadata. 
     Metadata for non live TV channels is generated from video on demand catalogs  220  or other video server provider  230  accessible from the same network, for example, internet video service providers. For example, if one live channel is broadcasting an interview of a celebrity, the user can be interested in watching a movie where the celebrity is part of the cast, or seeing the celebrity on a video on demand provider such as YouTube®. 
     Another component of the system architecture  200  is a mosaic builder  260  which links the metadata aggregator  250  and the user premises equipment, e.g., set top box  270 . For example, when a user requests the mosaic application  292  while watching a TV channel or a movie using a middleware IPTV application a request is sent to the mosaic builder  260 . The request is retrieved from the metadata aggregator  250  and all the related video (live &amp; non live) which is similar with the current watched program is displayed on the user TV screen, for example, using the screen template  150  shown on  FIG. 3 . 
     The mosaic builder  260  provides the mosaic interface  150  ( FIG. 3 ). The mosaic builder  260  includes generating the mosaic using parameters including, for example, a user identification (ID), a current channel ID, and generating categories, or sub-categories. The user ID parameter identifies each end user and allows the mosaic builder  260  to retrieve the associated user profile template &amp; preferences. The current channel ID identifies the main video channel  154  in  FIG. 3  and allows the mosaic builder  260  to retrieve related video from the metadata aggregator  250 . A category tree of a selected menu item may include categories, and subcategories. 
     An example of the mosaic builder&#39;s  260  steps described above for providing the mosaic interface  150  includes a GetMosaic method including the following steps: 
     1. GetMosaic (for example, user ID, current channel ID, category, sub-category); 
     2. Identify user ID parameter for each end user and allow the mosaic builder to retrieve the associated user profile template &amp; preferences; 
     3. Identify the current channel ID identify the main video channel  154  ( FIG. 3 ) and allow the mosaic builder to retrieve related video to it; and 
     4. Determine category, subcategory, etc., in the category tree of the selected menu item, if any, (in this case the current channel ID is set to null) otherwise this parameter is not used. 
     A component user profile  280  is used by the mosaic builder  260 , in two ways, firstly, if there is not enough related video to fulfill the mosaic video elements  162 , then general user preferences are used to complement the mosaic with the preferred user channels, at least one per category (for example, sport, music, etc.). 
     To build the mosaic main menu  158  ( FIG. 3 ) with the user template preferences model, the model can be implemented as an XML document or file, stored in the computer readable medium  290 , describing the entire user preferred categories and subcategories tree description. An embodiment of user template preferences (XML tags), including a mosaic XML main menu data representation, includes: 
     
       
         
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 &lt;Main categories&gt; 
               
               
                   
                   &lt;Category name=”News” priority=1 /&gt; 
               
               
                   
                   &lt;Category name=”Movie” priority=2 /&gt; 
               
               
                   
                     &lt;sub-categories&gt; 
               
               
                   
                       &lt;Category name=”SciFi” priority=1 /&gt; 
               
               
                   
                       &lt;Category name=”Romance” priority=2 /&gt; 
               
               
                   
                     &lt;/sub-categories&gt; 
               
               
                   
                 &lt;/Main categories&gt; 
               
               
                   
                   
               
             
          
         
       
     
     The user template preferences XML file is used by the mosaic builder  260  to build the mosaic main menu  158  allowing the user to navigate through the list of channels available and organized in categories and sub-categories. The user set-top box  270  ( FIG. 4 ) receives from the mosaic builder  260  all the required information to build the mosaic GUT  150  ( FIG. 3 ). 
     The following embodiment of XML syntax (data representation) is used to represent the mosaic display  100  ( FIG. 2 ) information: 
     
       
         
               
             
           
               
                   
               
             
             
               
                 &lt;Mosaic&gt; 
               
               
                  &lt;main channel&gt; 
               
               
                   &lt;channel ID=BBC URL=igmp://xxx.yyy.zzz.vvv&gt; 
               
               
                    &lt;title ProgramID=1234&gt;Breaking News&lt;/title&gt; 
               
               
                    &lt;info&gt;textual infos&lt;/info&gt; 
               
               
                    ...... 
               
               
                   &lt;/channel&gt; 
               
               
                  &lt;/main channel&gt; 
               
               
                  &lt;related channel&gt; 
               
               
                   &lt;channel ID=BBC2 URL=igmp://xxx.yyy.zzz.vvv&gt; 
               
               
                    &lt;title ProgramID=4321&gt;Evening News&lt;/title&gt; 
               
               
                    &lt;Thumbnail&gt;http://xxx.yyy.zzz.vvv/channel?BBC2&lt;thumbnail&gt; 
               
               
                    &lt;info&gt;additional textual info&lt;/info&gt; 
               
               
                    ...... 
               
               
                   &lt;/channel&gt; 
               
               
                   &lt;channel ID=VOD-A URL=rtsp://xxx.yyy.zzz.vvv/movieA&gt; 
               
               
                    &lt;title&gt;The movie&lt;/title&gt; 
               
               
                    &lt;Thumbnail&gt;http://xxx.yyy.zzz.vvv/movieA.jpg&lt;thumbnail&gt; 
               
               
                    &lt;info&gt;this movie talk about...&lt;/info&gt; 
               
               
                    ...... 
               
               
                   &lt;/channel&gt; 
               
               
                  &lt;/related channel&gt; 
               
               
                 &lt;/Mosaic&gt; 
               
               
                   
               
             
          
         
       
     
     Another component in the system  200  is a thumbnail TV-Generator  240  which is used when the user set-top box is not able to decode several IPTV video streams simultaneously (for example, unicast or multicast based). The thumbnail TV-Generator is dynamically invoked to generate screenshots of the requested live TV channel, the resulting image is then used to fulfill the video elements  162  of the mosaic. Based on a timeout parameter, a request is sent by the IPTV middleware application  290  ( FIG. 4 ) to refresh the screenshot of a video element channel thumbnail to produce an animated image placed into the video element of the mosaic  162  ( FIG. 3 ). 
     For each TV Channel or video asset  162  available as a related video element to the main channel  154  ( FIG. 3 ) the metadata aggregator  250  maintains a table, for example, as follows: 
     
       
         
               
               
               
               
             
           
               
                 TABLE 
               
               
                   
               
               
                 Channel ID 
                 Program ID 
                 Keywords 
                 Category hierarchy 
               
               
                   
               
             
             
               
                 BBC 
                 1234 
                 News, Breaking 
                 News/international/ 
               
               
                   
                   
                 news, irak, 
                 national/UK 
               
               
                   
                   
                 Pakistan, . . . 
               
               
                 VOD-A 
                 xxx 
                 Movie, action, 
                 Movie/SciFi/2007 
               
               
                   
               
             
          
         
       
     
     In the table, the channel ID identifies the channel or video asset. The program ID uniquely identifies the program itself and allows retrieval of contextual information from external sources like EPG providers. The keywords are a collection of keywords associated with the program and are used to retrieve similarities with the main video. The category hierarchy is used to organize channels and programs into hierarchy trees facilitating the user selection. 
     Referring to  FIGS. 4 and 5 , a flowchart  300  according to an embodiment of the invention illustrates the logic (e.g., a computer program) as an embodiment of the mosaic application  294  ( FIG. 4 ) used to generate the mosaic GUI  150  ( FIG. 3 ) and a sample mosaic GUI  400  in  FIG. 6 . An IPTV user watches regular TV channels, step  304 , the user can access the mosaic application  294  in step  308  by selecting a dedicated button of its remote control or any other device able to send command to the set-top box  19   b  ( FIG. 1 ). Accessing the mosaic application  294  ( FIG. 4 ) starts with a request, in step  312 , sent to an HTTP network (for example, wireless IP transmission) from the mosaic builder  260  ( FIG. 4 ) using the GetMosaic method (described above) with the user ID and the current channel ID parameter filled. If the user selection came from the main menu, then only the category, sub-category parameter and the user ID parameter are used. 
     In step  316 , the mosaic builder  260  ( FIG. 4 ) retrieves the user profile with all its preferences, e.g., the user&#39;s center of interest, favorite channels, etc. The mosaic builder  260  ( FIG. 4 ) builds a search request to the metadata aggregator  250  which retrieves all the live and non live video assets filling the search request ordered by similarity importance. In step  320 , the mosaic builder  260  uses the search report to create the XML syntax data (described above) response to the mosaic application  294  ( FIG. 4 ). In step  324 , the mosaic application  294  used this XML response to build the GUI interface of the mosaic application. If the thumbnail TV generator  240  ( FIG. 4 ) is used, the mosaic application  294  starts a timer for a specified time period, for example a specified number of milliseconds, for each video element  162  ( FIG. 3 ) of a mosaic GUT  400  ( FIG. 6 ), in step  328 . The mosaic GUT  400  is an example of a mosaic display wherein like elements have the same reference numerals as the GUT interface shown in  FIG. 3 . When a timeout occurs a request is sent to the thumbnail TV generator  240  component to retrieve a new thumbnail image of the live channel to be displayed inside the mosaic GUT  400  ( FIG. 6 ). Then, the timer is restarted for the specified time period. In step  332 , the application  294  waits for the user to change the channel, and if the channel is changed the program returns to before step  308  to re-generate the mosaic GUT  400  ( FIG. 6 ) as shown in  FIG. 6 . 
     Therefore, an advantage of the present invention is the dynamic mosaic dynamically proposing channels related to the main channel being viewed by the user. A further advantage of the present invention is the ability for the user to select the number of TV channels to add, and thereby view simultaneously, to the dynamic mosaic. 
     While the present invention has been particularly shown and described with respect to preferred embodiments thereof, it will be understood by those skilled in the art that changes in forms and details may be made without departing from the spirit and scope of the present application. It is therefore intended that the present invention not be limited to the exact forms and details described and illustrated herein, but falls within the scope of the appended claims.