Patent Description:
In recent years, set top boxes and other television receivers have been enhanced to provide additional features beyond mere presentation of broadcast television content. For example, commercials in broadcast programs can now be augmented with additional features to allow viewers to obtain additional information, or to take other actions relating to the content shown on the television screen. Such features may be provided through user interaction with the main television display, and/or with a "second screen" device such as a tablet or other computer, a mobile phone or the like. Several content enhancement implementations are described in <CIT>, <CIT> and <CIT>.

Current content enhancement techniques, however, have been found to be inadequate for modern rich content applications. Legacy systems, for example, are often limited in the amounts and types of content that can be supported. Systems based upon conventional closed captioning standards, for example, can be limited in their support for video content, software applications, second screen integration and/or the like.

It is therefore desirable to provide systems, devices and/or processes that improve the user experience through enhanced content. These and other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background section. <CIT> describes a system that receives and stores pre-set data that describers a particular format of timed text that is associated with a program, the timed text being interpreted via the pre-set data when output to a display during presentation. <CIT> is concerned with an enhanced TV system delivering additional data to a client beyond audio, video, teletext and subtitle data.

A discussion of timed text and visual overlays in timed text are given in:.

The present invention is defined by appended independent claims <NUM> and <NUM> with preferred embodiments being defined by the claims dependent thereon.

In embodiments, various systems, devices and processes activate auxiliary or enhanced features associated with broadcast television programs by using carrier-based active text enhancement (CATE) signals that are embedded within timed text (TT) associated with the broadcast program. The active text enhancements can be interpreted by the viewer's set top box (STB) or other receiver to activate software applications, video clips, imagery, interactive interface features and/or the like on a primary and/or on a secondary display. In contrast to prior implementations, timed text enhancements can flexibly reference different types of content to provide a richer and more powerful viewer experience.

Exemplary embodiments will hereinafter be described in conjunction with the following drawing figures, wherein like numerals denote like elements, and.

The following detailed description of the invention is merely exemplary in nature and is not intended to limit the invention or the application and uses of the invention. Furthermore, there is no intention to be bound by any theory presented in the preceding background or the following detailed description.

Carrier-based active text enhancement (CATE) signals are embedded in a timed text (TT) stream that is delivered in conjunction the broadcast television signal. A SMPTE TT stream, for example, can be encoded with text enhancements and provided in conjunction with an MPEG or similar multiplex that encodes the broadcast programming. The text enhancements are interpreted by the television receiver device to activate auxiliary content or other supplemental events, as appropriate. Data that accompanies the text enhancement (e.g., video, image, text, software application/applet content, URLs and/or other data) can also be provided within the timed text stream, if desired, although other embodiments may obtain the data through other channels, as described more fully below.

CATE signals may be received at any time prior to the processing of the auxiliary content. Generally speaking, it will be desirable for the receiver to obtain any auxiliary content prior to the broadcast of the text enhancement so that the auxiliary content can be activated at the proper time in synchronization with the broadcast audio-video streams. Auxiliary data may be provided according to a batch scheme (e.g., as part of an electronic program guide or other out-of-band feature in which data for multiple programs is received simultaneously), for example. Other embodiments will use URLs or other links within the CATE signal to access needed data from a web server or other source in real time, near real time, just-in-time, and/or according to any other timing scheme, as appropriate.

In prior implementations, supplemental content triggers were embedded within closed captioning signals or the like. Closed captioning generally refers to receiving, formatting, and displaying text on a television or other display that allows viewers to access additional or interpretive information about programming content. Federal law mandates that all broadcast television programming in the United States must include closed caption information. For conventional digital (e.g., ATSC) television broadcasts, caption streams are typically encoded in a well- known format such as EIA-<NUM> format, EIA-<NUM> format, and/or the like. Most set top boxes (STBs) and other television receivers are therefore designed to interpret EIA-<NUM>, EIA-<NUM> and/or similar formats in received broadcasts. Other closed captioning formats or standards may be implemented in the United States or elsewhere.

Closed captions often provide textual transcriptions of the audio portions of programs as the programs progress, thereby allowing the hearing impaired and others to follow the program content without listening to the audio track typically associated with the programming. Motion pictures, video disk (e.g., DVD) content, streamed audio/video, video games and the like may similarly incorporate closed captions using any number of standard or non-standard techniques.

More recently, some television systems have used the closed caption stream that accompanies broadcast programming to encapsulate prompts for some forms of supplemental content. Most conventional closed caption formats, including EIA- <NUM> and EIA-<NUM>, however, can be severely limited in the amount of space that is available. EIA-<NUM>, for example, generally provides only enough space for a relatively small set of textual data. EIA-<NUM> allows somewhat more data space for limited graphical capabilities, but even this expanded space is severely restricted for most purposes. As a result, enhanced content available from conventional closed caption systems can be severely limited.

As more and more consumers receive their television programming from video on demand (VOD) and similar non-broadcast sources of streaming video, the Society of Motion Picture and Television Engineers (SMPTE) and others have recommended a transition from the older closed caption standards (e.g., EIA <NUM>/<NUM>) to more advanced "timed text" formats such as SMPTE-TT for streaming media, video on demand, and similar network-based delivery systems. Newer timed text formats commonly incorporates syntax, structures and tags similar to the extensible markup language (XML) to provide more advanced features and to improve flexibility in comparison to the more traditional legacy formats. The SMPTE-TT format is described in SMPTE Standard ST <NUM>-<NUM>-<NUM>, which was approved on December <NUM>, <NUM>. The SMPTE standards are updated from time to time, and the various later versions should be considered to be equivalents of the initial <NUM> standard. Equivalent timed text standards include the W3C Timed Text Markup Language (TTML) and Web Video Text Tracks (WebVTT), among others.

Timed text standards are generally intended for use with streaming video services and web content rather than terrestrial, satellite or cable broadcast television. As a result, the timed text standards by themselves generally lack the ability to function effectively in a broadcast television environment. At the very least, conventional timed text formats lack the ability to activate enhanced or auxiliary content in a broadcast setting.

Various embodiments, however, are able to expand the functionality of conventional timed text formats by providing carrier action text enhancement (CATE) signals within the timed text that can activate auxiliary content. The enhanced capabilities of timed text can be used to allow richer feature sets, to provide more data with the activation signal, to capitalize upon additional applications and features, and/or to provide other benefits as desired. To that end, set top boxes and other television receivers intended for consumer use can be designed to interpret timed text (e.g., SMPTE timed text) to identify CATE signals within broadcast television programming. Further, CATE signals may be useful within video streaming, second screen, place shifting, time shifting and/or any other television viewing environments.

The extensible features of timed text can flexibly allow for representation of new data types that could not be represented within closed caption-based systems. Unlike the previous closed caption formats that were limited to text or rudimentary graphics, the XML-like features of many timed text formats can be readily adapted to contain motion video content, URLs, software code (e.g., code in executable, compilable, interpretable and/or other form) or the like.

Auxiliary content could expand the user experience in any number of ways. To provide just a few examples, auxiliary content could represent video content that is presented to a viewer on the television or other primary display. Such content may be activated by the user input, as desired. Other implementations could use the auxiliary content to deliver a software application executable by the receiver device to play a game or provide other interactive experiences with the viewer. Other implementations could use auxiliary content to provide web content, such as sports statistics, weather, news, information about the program being viewed (e.g., actor/actress information, electronic program guide information and/or the like), or any other information as desired.

In many cases, the auxiliary content is related to the programming that is currently shown. An advertisement, for example, could be enhanced with additional video or other content that is shown while playback of the main program is paused, or the viewer could be presented with an application or web page that provides more information about the product, or that processes an order for the product advertised.

The CATE signals initiate the synchronized (or non-synchronized) presentation of supplemental applications or data on a second screen. In various embodiments, an application executing on a tablet or other computer, a mobile phone or another device could be activated in response to the CATE signal to thereby direct the application to present web content, to accept an order for a product, and/or to take any other action. Many other types of auxiliary content could be provided in any number of alternate but equivalent embodiments.

Turning now to the drawing figures and with initial reference to <FIG>, an example system <NUM> for activating auxiliary content via timed text includes a content source <NUM> that includes a production system <NUM>, a distribution system <NUM>, and/or a transmission system <NUM> as appropriate. Content with formatted timed text information <NUM>, once created, is provided over any sort of broadcast or other distribution channel <NUM> for rendering at a media receiver <NUM> and eventual playing on any sort of television display <NUM>, second screen <NUM>, and/or other display as appropriate. As shown in <FIG>, timed text information <NUM> may be inserted into the programming at any stage of production <NUM>, distribution <NUM> and/or transmission <NUM>.

In various embodiments, the timed text content <NUM> may include closed caption data, as well as any CATE signals or the like that are interpretable by the receiving device to activate auxiliary content. The stream of timed text <NUM> may itself also include some or all of the auxiliary content data <NUM> for presenting graphics, video, interactive features, web content and/or any other auxiliary content as desired, although other embodiments may deliver the auxiliary content data <NUM> via other mechanisms and paths, as described below.

Timed text <NUM> may be formatted in any manner. In various embodiments, timed text <NUM> includes conventional header information (as described by the SMPTE-TT or another specification) or other information that allows receiver <NUM> to properly interpret the timed text <NUM> that accompanies video programming rendered by the receiver <NUM>. Such information may include the sizes and locations of data fields, for example, or other information relating to the structure, size, arrangement or other formatting of the timed text <NUM>. In various embodiments, timed text <NUM> also carries information related to the presentation of caption or other data, such as location of presentation on the screen, scrolling speed or time, pop on/off data, timing data, and/or the like. Formatting information may be defined within the data itself <NUM> as desired (e.g., using XML or similar constructs), or defined externally by a standard, or by a schema sent via an electronic programming guide (EPG) feature or other out-of-band channel accessible to the receiver device <NUM>.

Receiver device <NUM> suitably receives broadcast content via direct broadcast satellite (DBS), terrestrial broadcast, cable broadcast or the like. Generally, the received content will be provided as an MPEG or similar multiplex with separate streams for video, audio, timed text and the like. Streams are synchronized for playback using a presentation time stamp (PTS) or the like. When the receiver no encounters a CATE signal in the timed text stream, this signal is interpreted to activate auxiliary content such as a graphical or video presentation, an interaction with a purchasing or web application, and/or the like. Typically, the CATE signal in timed text <NUM> will indicate the timing of the auxiliary content and the particular data that will be used for the auxiliary content <NUM>. Auxiliary content data <NUM> may be delivered in any number of different ways: a comparative example of <FIG> may embed auxiliary content data <NUM> within the timed text stream itself, whereas embodiments may separately fetch or otherwise deliver auxiliary content data <NUM> via a network server <NUM>, via a separate out-of-band signal contained in broadcast <NUM>, and/or in any other way.

In the exemplary embodiment illustrated in <FIG>, production system <NUM>, distribution system <NUM> and transmission system <NUM> represent any systems, devices and/or organizations capable of producing, distributing and/or transmitting program content, respectively. As noted in <FIG>, timed text content <NUM> may be inserted into the programming content in any manner at any stage of production, distribution and/or transmission. In various embodiments, timed text <NUM> is formatted and associated with the programming at the production stage <NUM> by an author, editor, producer or other party. Certain producers, distributors and/or broadcasters may have preferred formats for timed text <NUM>, and each may produce, distribute and/or broadcast content using their own formats as desired. Format data may change from program to program, or may be consistent for each producer, distributor, broadcaster or other entity. All of the programs broadcast on a particular channel, for example, may have common format data that may nevertheless differ from the format data used by another network.

Timed text <NUM> may be encoded into any sort of programming in any manner, such as during a conventional "post production" phase or the like. Insertion of timed text <NUM> during or just after production may be appropriate in any sort of programming, including cinema programming, television programming or any other programming that is delivered in any sort of standard format, such as any form of MPEG format that can be rendered by a player/receiver device <NUM>. In other embodiments, however, timed text <NUM> may be added to programming during intermediate distribution <NUM> (e.g., encoding or transcoding of programming that is stored onto DVDs or other portable media, or otherwise distributed to network affiliates, broadcasters and/or any other parties). In still other embodiments, timed text <NUM> may be inserted into the program stream just prior to broadcast or other transmission <NUM>, much as current caption data is encoded in many current television broadcasts. Timed text <NUM> may also be inserted prior to place shifting or other streaming of a network video stream, or in any other setting as desired. Typically, timed text <NUM> is formatted and automatically inserted into the program stream by computerized machinery to so that the timed text content is digitally represented, formatted and stored on any appropriate media prior to transmission or other distribution.

Distribution channel <NUM> represents any sort of data link, broadcast or other wireless connection, physical media, and/or other avenue for delivering programming content. Examples of distribution channels could include, without limitation, broadcast television, very small aperture terminal (VSAT) satellite (e.g., for movie distribution), direct broadcast satellite (DBS), cable television, cellular or other wireless telephone networks, Internet or other data communications networks, and/or the like. As noted above, timed text <NUM> including CATE signals and/or auxiliary content <NUM> may be inserted into or otherwise encoded with programming content <NUM> in any manner, and at any phase of production, post- production, distribution and/or delivery of the programming content. Indeed, timed text <NUM> and/or auxiliary content data <NUM> that is associated with a broadcast program <NUM> need not be provided with the video content itself. In some implementations, timed text <NUM> and/or auxiliary content data <NUM> that is associated with a broadcast program <NUM> may be retrieved from a separate server (e.g., server <NUM>) and/or other source of information, as desired. This timed text data <NUM> may supplement or be provided in addition to conventional closed caption data that may be provided within the broadcast, as desired.

Receiver <NUM> is any device, component, circuitry or logic capable of receiving and processing video programming content. As shown in <FIG>, receiver <NUM> includes a receiver interface <NUM>, a controller <NUM> with appropriate processor <NUM> and memory <NUM> resources, and a display interface <NUM> as appropriate. Some implementations may also include a network interface <NUM> for communicating on network <NUM>, as appropriate. In various embodiments, receiver <NUM> is a conventional television receiver (e.g., a set top box or other broadcast, satellite and/or cable television receiver) capable of receiving signals via distribution channel <NUM> and providing an output signal <NUM> that can be displayed to the viewer. In various embodiments, display <NUM> is any sort of television or other monitor that is capable of receiving a program signal <NUM> from a set -top box, decoder or other external receiver <NUM> as desired.

Receiver interface <NUM> is any hardware, firmware and/or software interface capable of receiving programming content. In various embodiments, receiver interface implements a demodulator/decoder feature for receiving and demodulating digital television programming over a terrestrial broadcast, satellite, and/or cable programming medium.

The example television receiver <NUM> illustrated in <FIG> includes a storage medium <NUM>, as appropriate. Storage medium <NUM> may be implemented as additional memory, as a disk drive, or in any other manner. Many embodiments may provide a digital video recorder (DVR) or other recording feature that allows content to be stored in storage medium <NUM> for later viewing. Such content may be stored with timed text data <NUM> and/or auxiliary content data <NUM> to allow for decoding and viewing of auxiliary content at the time that the programming stored on the recorder is rendered. Although <FIG> shows auxiliary content data <NUM> stored within storage medium <NUM>, equivalent embodiments could store this data in memory <NUM>, and/or in any other memory, disk or other storage medium available to receiver <NUM>, including any sort of cloud storage or other remote storage that is accessible over network <NUM> or the like.

Controller <NUM> is any sort of control logic or the like that interacts with receiver interface <NUM> and display interface <NUM> to output imagery to the viewer on display <NUM>. Controller <NUM> also controls the reception, storage and processing of video content via receiver interface <NUM>. To that end, controller <NUM> suitably directs the reception and storage of pre-set data <NUM>, as well as the interpretation of timed text information <NUM> that is provided within received programming. Additional detail about such processing is provided below with respect to <FIG>.

In various embodiments, controller <NUM> may be implemented using any sort of microprocessor, microcontroller, digital signal processor or other processor <NUM> capable of directing the actions and processes of receiver <NUM>. Typically, processor <NUM> will be associated with any sort of memory <NUM>, such as any sort of static, dynamic, flash or other memory capable of storing programming instructions and/or data for processing by processor <NUM>. In various embodiments, receiver <NUM> is based upon a "system on chip" (SoC) implementation that incorporates a hybrid microcontroller <NUM> with memory <NUM>, input/output and/or other features to perform the various signal processing and other actions of receiver <NUM>. Various SoC and other integrated hardware implementations are available from Texas Instruments, Conexant Systems, Broadcom Inc. , NXP Semiconductors Inc. and many other suppliers as appropriate. Other embodiments may implement processor <NUM> and/or memory <NUM> using any sort of application specific integrated circuit (ASIC) or the like. Still other embodiments may implement processor <NUM> and/or the other features of receiver <NUM> with any number of discrete and/or integrated processing components (e.g., any sort of microprocessor or microcontroller), memories <NUM>, input/output features and/or other features as desired.

Display interface <NUM> is any physical and/or logical interface to display <NUM>. As noted above, in some implementations receiver <NUM> and display <NUM> are provided in an integrated product (e.g., a conventional television). In other embodiments wherein receiver <NUM> provides video output signals <NUM> to an external display <NUM>, such signals <NUM> may be provided in any compatible format. In embodiments wherein display <NUM> is a conventional television, for example, display interface <NUM> may provide video output signals <NUM> in any conventional format, such as component video, composite video, S-video, High- Definition Multimedia Interface (HDMI, e.g., any version of the CEA-<NUM> standards), Digital Visual Interface (DVI), IEEE <NUM>, universal serial bus (USB) and/or any other formats as desired.

Display <NUM> is any sort of television, monitor and/or other display capable of presenting video imagery <NUM> to a viewer. In various embodiments, display <NUM> operates in conjunction with receiver <NUM> to generate video imagery <NUM> in any manner. Other embodiments may combine the functions and components of receiver <NUM> and display <NUM> into a common housing, as in a conventional television with a built-in tuner or other receiver. In the example illustrated in <FIG>, imagery <NUM> is generated with auxiliary content <NUM> that is activated within timed text <NUM>.

Network interface <NUM> operates using any implementation of protocols or other features to support communication by device <NUM> on network <NUM>. In various embodiments, network interface <NUM> supports conventional LAN, WAN or other protocols (e.g., the TCP/IP or UDP/IP suite of protocols widely used on the Internet) to allow device <NUM> to communicate on network <NUM> as desired. Network interface <NUM> typically interfaces with network <NUM> using any sort of LAN adapter hardware, such as a conventional network interface card (NIC) or the like provided within device <NUM>. Network interface <NUM> could be implemented with a conventional ETHERNET controller chip that operates with a conventional electrical transformer to communicate over a conventional RJ45 jack in at least one embodiment, although other embodiments may provide different features, including any sort of WI-FI or other wireless interface as desired.

The television receiver <NUM> operates in conjunction with a second screen device <NUM> such as a tablet computer, personal computer, mobile telephone, video game playing device, or the like. Second screen device typically includes conventional computing hardware such as a processor, memory and interfaces to network <NUM> or the like. The device <NUM> executes a media player or other application <NUM> that communicates with the receiver <NUM> to obtain media streams, auxiliary content data <NUM> and/or instructions to process certain features, as desired. In the example illustrated in <FIG>, second screen device is a tablet or similar device that presents auxiliary content related to the broadcast content that is being rendered on display <NUM>. The auxiliary content may be synchronized to the content on display <NUM> using PTS information and/or instructions transmitted from the receiver device <NUM> to device <NUM>, as desired. The device <NUM> responds to certain CATE signals, for example, by instructing the second screen device to present a user prompt for more information (as shown in <FIG>), to present web content obtained from a URL contained in the CATE signal that is forwarded by the receiver <NUM>, to execute an applet or other software module, or the like. Media content, URLs, software modules and auxiliary content data <NUM> may be transferred from the receiver <NUM> to device <NUM> in some embodiments, and/or device <NUM> may be able to directly download the appropriate auxiliary content data <NUM> from web services <NUM> or the like, as desired, using URLs contained within CATE signals and forwarded to device <NUM> by receiver device <NUM>. Processing auxiliary content on the second screen device <NUM> allows viewers to obtain additional information or to place orders without disturbing other viewers of display <NUM> or holding up playback of the broadcast program <NUM>, as desired. Many equivalent embodiments could be created to make further use of the second screen device <NUM>, as desired.

In practice, then, receiver <NUM> suitably receives television-type programming in any format or medium. The programming may be received at receiver interface <NUM> from any broadcast or other transmission source, from any sort of streaming or portable media source, and/or any other distribution channel <NUM>. The programming is provided with (or at least associated with) timed text information <NUM> that includes caption or other information. This information <NUM> may be extracted from the programming stream, obtained from a network service <NUM>, and/or otherwise obtained from any other source as described herein. The timed text <NUM> is interpreted by the viewer's television receiver <NUM>, and any CATE signals occurring within the timed text <NUM> are processed to activate auxiliary content, as described herein. Timed text and program content may be further processed in conjunction with any user preferences or other information to generate suitable imagery on display <NUM> and/or on secondary device <NUM> to thereby present the program and the timed text information to the viewer as desired.

Video, graphical, application or other data <NUM> used to implement the auxiliary content activated by CATE signals in timed text <NUM> may be delivered to the receiver <NUM> in any manner. In various embodiments, auxiliary content data <NUM> is provided through the same distribution channel <NUM> as the program itself. Data for auxiliary content <NUM> may be delivered as part of the out-of-band data generally associated with an electronic program guide (EPG) or the like. Such data may be delivered to receiver device <NUM> via an out-of-band signal that is separate from television programming signals, but that uses the same DBS, cable or other distribution medium that is used to deliver the programming signals, as desired.

Timed text data <NUM> and/or auxiliary content data <NUM> are delivered to receiver device <NUM> from a separate source via a backchannel such as the Internet, a telephone link, and/or another data communications network. Such embodiments may provide auxiliary content data <NUM> from a server <NUM> operating on the network <NUM> that delivers data <NUM> in response to requests placed by receiver <NUM>, as desired. Such requests may be posited on a batch or as-needed basis, as described more fully below.

Receiver device <NUM> may support other functions and features in addition to those described above. The device <NUM> may include a digital video recorder (DVR) feature that stores broadcast programs to storage <NUM> or the like for time shifted viewing. The device <NUM> supports live or delayed encoding/transcoding of received programs for place shifting to a remote device <NUM>, such as a notebook, tablet or other computer, or a mobile phone, video game player or other media device. In such arrangements, timed text <NUM> may be stored with the DVR and place shifted along with the audio and video content so that that time shifted and place shifted video stream is nevertheless able to support the auxiliary features <NUM> activated by the timed text <NUM> during playback at later times and other locations. Put another way, the timed text <NUM> and associated auxiliary content data <NUM> may be retrieved and/or interpreted at any time and location of playback, thereby allowing auxiliary features <NUM> whenever and wherever the media stream is played back. If the timed text data <NUM> and auxiliary content data <NUM> is available to the playback device, the device may be able to provide the auxiliary content just as though the viewer was watching a live broadcast at his or her own television receiver. A place shifting client processes CATE signals and auxiliary content delivered via a place shifted media stream, as desired.

In the example illustrated in <FIG>, the timed text <NUM> accompanying a broadcast television program includes one or more CATE signals that activate enhanced features <NUM> at certain times during playback of the broadcast program. <FIG>, for example, provides an overlay during playback of a commercial occurring during broadcast <NUM> on display <NUM>. The overlay content <NUM> in this instance allows the viewer to push a button for more information and/or to complete an order for the advertised product. This overlay may be active for the duration of the commercial (e.g., as described by PTS data in the CATE signal), and/or for some period of time after the commercial is complete. If the user does select one of the buttons for further content, then video, graphical or other auxiliary content <NUM> may be presented, an application provided as auxiliary content data <NUM> could be executed on receiver device <NUM>, and/or web content referenced by a URL in the CATE signal could be provided, as desired. Other embodiments could forward information from the CATE signal to a second screen device <NUM> so that the auxiliary content is delivered via application <NUM> executing on device <NUM>. Equivalent embodiments may expand, alter or otherwise modify these features in any way, and/or may provide other types of auxiliary content, as desired.

<FIG> shows one example of a timed text stream <NUM> that includes data 107A-E for auxiliary content <NUM>, as well as CATE signals <NUM>, <NUM>, <NUM>, <NUM> and <NUM> that activate various events. In this example, data <NUM> could include graphics, video, text, source or binary code, applications/applets and/or other data as desired. In this example, data <NUM> is delivered in the stream prior to the CATE signal <NUM>-<NUM> that activates the use of that data <NUM>. Data 107A, for example, may include graphics, imagery, code or other data that would be useful to receiver <NUM> in responding to an event activated by CATE signal <NUM>.

Timed text stream <NUM> as shown in <FIG> may be delivered as part of an MPEG multiplex or other package that also includes one or more video streams <NUM>, audio streams, and/or the like. The timed text stream <NUM> could augment a conventional closed caption stream in some embodiments, or the closed caption stream could be combined with the timed text stream <NUM> as desired. Generally, the various streams within a package <NUM> would share a common time reference, such as an MPEG presentation time stamp (PTS). This time reference allows activated content to be synchronized with events occurring during the video stream <NUM>, as desired. That is, auxiliary content can be activated for the viewer at the particular times indicated within the CATE signal, as appropriate.

In some embodiments, it may be desirable for the CATE signal to be delivered prior to the time of the video event so that the receiver <NUM> has time to obtain, interpret and/or otherwise process the auxiliary content for display at the appropriate time during the video presentation <NUM>. Signals <NUM> and <NUM> in <FIG>, for example, would include a PTS that identifies the time that the auxiliary content is activated even though the signal itself is received prior to that time in the presentation. This lead time allows the receiver <NUM> to retrieve data from server <NUM>, to store the retrieved auxiliary content data <NUM> into compile, interpret or otherwise process executable code contained in data stream <NUM>, or to take other actions as needed. Other embodiments may provide features in the timed text <NUM> that allow the receiving device <NUM> to obtain any needed auxiliary content data <NUM> before it is needed. Such auxiliary content data <NUM> will typically be identified in a manner that is referenced by the CATE signal so that the auxiliary content can be readily and quickly activated when the CATE signal is received and processed.

The CATE signals <NUM>-<NUM> themselves contain sufficient information to activate the desired auxiliary content at the appropriate times during playback of the video stream <NUM>. In various embodiments, CATE signals are described by an XML or similar schema that allows the signal to specify the times that the auxiliary contents are active (e.g., a beginning and ending PTS), the type of auxiliary content (e.g., video, graphical, textual, interactive, URL, etc.) and/or the location of data <NUM> needed to execute the auxiliary content.

Data for auxiliary content <NUM> may be formatted in any manner. In various embodiments, data <NUM> that is delivered via stream <NUM> may be described in an XML schema or the like that describes the various types and locations of specific data types within a larger data structure. As noted above, embodiments download the data <NUM> from an external service <NUM>, or deliver the data using an EPG or similar out-of- band transmission over broadcast medium <NUM>.

The timed text stream <NUM> may be stored in a DVR <NUM>. The timed text stream <NUM> is forwarded to a second screen or other display device <NUM> via place shifting. Whenever and wherever the video program is decoded, the timed text stream <NUM> may be similarly interpreted so that auxiliary content "follows" the video stream <NUM>. For example, if a user stores a broadcast program on a DVR <NUM> for later viewing, the timed text <NUM> is still interpreted at the time of playback so that auxiliary content data <NUM> can be retrieved and/or processed as desired. Similarly, other media players <NUM> that obtain place shifted streams or encoded media files from receiver <NUM> are similarly programmed to interpret the timed text <NUM> in the same manner as receiver <NUM>. That is, second screen device <NUM> interprets timed text <NUM> to process CATE signals <NUM>-<NUM>, to retrieve auxiliary content data <NUM> from any appropriate source, and/or to process and provide auxiliary content to the viewer on the display associated with the device <NUM>, as desired.

With reference now to <FIG>, an example method <NUM> for processing timed text <NUM> at a video receiver such as receiver <NUM> suitably includes the broad functions of receiving the data <NUM> associated with the auxiliary content (function <NUM>), interpreting the timed text associated with a received broadcast to identify CATE signals that activate auxiliary events (functions <NUM>-<NUM>), and then responding to the occurrence of a CATE signal to process the auxiliary content at the desired time(s). Method <NUM> may be repeated as needed for any number of different programs (function <NUM>), as desired.

Data <NUM> used to implement the auxiliary content may be received at any time, and according to any temporal basis (function <NUM>). Auxiliary content data <NUM> may be obtained in batches, or in any sort of "as-needed" or "on-demand" basis, as appropriate. Batch processing could represent any technique for obtaining auxiliary content data <NUM> for multiple broadcasts at the same time, such as providing the data <NUM> as part of a program guide or other regular download to receiver <NUM> from a broadcaster <NUM> or other source. To that end, data <NUM> may be received via an out- of-band channel separate from the programming content, but still part of a DBS, cable or other distribution channel <NUM>. Other implementations may allow receiver <NUM> to request the data <NUM> from a service <NUM> available via network <NUM> or another back channel, as desired. Such data could be requested from a URL contained in a CATE signal broadcast prior to the start of the auxiliary content, such as signals <NUM> and <NUM> described above. Still other embodiments could receive data <NUM> as part of the timed text stream <NUM>, as described with reference to <FIG> above. Data <NUM> may therefore be provided automatically (e.g., as part of a regular batch download), in response to requests placed by the receiver <NUM>, as part of a broadcast or on demand stream, and/or in any other manner as desired. In some implementations, the receiver <NUM> may assume certain default values for URLs or other types of data <NUM> if no explicit data <NUM> is received. Such default values may be associated with the content provider, the programming channel, the program itself, the manufacturer of the receiver <NUM>, the geographic region of the receiver <NUM>, and/or any other factors as desired. Data <NUM> is stored within receiver <NUM> until it is needed. As noted above, data <NUM> may be stored within memory <NUM>, within storage medium <NUM>, and/or in any other storage medium available to receiver <NUM> as desired.

Program content is received and decoded in any suitable manner (function <NUM>). In various embodiments, broadcast content may be received by controlling a tuner within receiver <NUM> to select a channel containing programming desired by the viewer; this programming is received and decoded as appropriate. In various embodiments, content is received as an MPEG or similar multiplex that includes multiple component streams each containing video, audio, timed text and/or other information. In such embodiments, timed text <NUM> may be provided within a data stream, as described above, or in any other appropriate manner.

As noted previously, the timed text <NUM> associated with the particular program is interpreted to identify carrier-based active text enhancement signals that activate auxiliary content or events (function <NUM>). CATE signals may be identified through headers or codes used within the stream, or by other features in accordance with the timed text format that is in use. As noted above, CATE signals will typically specify a start and end time (e.g., by PTS), a type of auxiliary content to be presented, and/or the location of the data <NUM> that is associated with the activated event. Other embodiments could provide additional or alternate content, as desired.

If a CATE signal is identified (function <NUM>), then the information in the signal is processed to render the auxiliary content (function <NUM>). As noted above, the signal is interpreted by the processing device to locate associated data <NUM> and to render the auxiliary content at the appropriate times. Many different types of auxiliary content could be provided in a wide range of alternate embodiments. Some embodiments may allow the viewer to view additional textual, graphical and/or video content, for example. Other embodiments could connect the viewer to a URL or other service via network <NUM> to complete purchase of an indicated product, to obtain additional information about displayed content, to play games associated with the displayed content, to obtain auxiliary information (e.g., sports statistics, information about actors and actresses appearing in a displayed feature, etc.) and/or any other features as desired.

As noted above, various embodiments may use CATE signals to initiate functions or features on a tablet or other second screen device <NUM>. In some implementations, the receiver device <NUM> maintains a data connection with the second screen device <NUM> so that instructions may be sent from device <NUM> to a media player or other application executing on device <NUM>. Upon receipt of an instruction from device <NUM>, the application executing on device <NUM> suitably obtains and/or renders auxiliary content <NUM> on the display of the second screen device <NUM>, as desired. Such embodiments may maintain timing between the application executing on device <NUM> and the playback of media content by device <NUM> so that the second screen presentation remains in synch with the content, thereby allowing the viewer to conveniently obtain additional information or to take other actions (e.g., to place an order) relating to objects or other content currently on the display.

Within the place shifting application, second screen device <NUM> also acts as a receiver device <NUM>. That is, the second screen device <NUM> receives broadcast content indirectly via a place shifting device, a video service and/or any other source (e.g., device <NUM>, a remote service digital video recorder (RSDVR) or video on demand service on network <NUM>, or the like). Even if the broadcast television content is received indirectly, some embodiments could allow the second screen device <NUM> to process timed text <NUM>, to obtain auxiliary content data <NUM>, to identify CATE signals relating to the received content, and to render the auxiliary content to the viewer as part of the viewing experience, as desired.

Generally speaking, the various functions and features of method <NUM> may be carried out with any sort of hardware, software and/or firmware logic that is stored and/or executed on any platform. Some or all of method <NUM> may be carried out, for example, by logic executing within receiver <NUM> in <FIG>. In one embodiment, processor <NUM> executes software logic that performs each of the various functions shown in <FIG>. Such logic may be stored in memory <NUM> or in any other storage available to processor <NUM> as desired. Hence, the particular logic and hardware that implements any of the various functions shown in <FIG> may vary from context to context, implementation to implementation, and embodiment to embodiment in accordance with the various features, scenarios and structures set forth herein. The particular means used to implement each of the various functions shown in <FIG>, then, could be any sort of processing structures that are capable of executing conventional software logic in any format. Such processing hardware may include processor <NUM> or other components of receiver <NUM> in <FIG>, as well as any other processors or other components associated with any conventional television, receiver, media player and/or the like.

Claim 1:
A system (<NUM>) to process auxiliary content associated with broadcast television programming for presentation of the broadcast television programming on a second screen device (<NUM>), the system comprising:
a television receiver interface (<NUM>) configured to receive live broadcast television programming (<NUM>) and the auxiliary content (<NUM>) associated with the broadcast television programming, the television programming comprising a video stream and timed text in a timed text stream that includes caption data for presentation with the video stream, wherein the auxiliary content is received separately from the timed text via a broadcast out-of-band channel or via a network server (<NUM>), and the auxiliary content may be received at any time and according to any temporal basis;
a second screen device (<NUM>) configured to provide imagery for presentation to a viewer;
a data storage (<NUM>) configured to store the broadcast television programming as a recorded television program together with data associated with the auxiliary content; and
a processor (<NUM>) configured to provide a place shifted video stream (<NUM>) of the stored television program to the second screen device, the timed text stream comprising an auxiliary content activation signal (<NUM>-<NUM>) embedded in the timed text (<NUM>) that identifies the auxiliary content, a location of the data associated with the auxiliary content and/or a type of the auxiliary content, and a time to present the auxiliary content,
the processor further configured to interpret the timed text associated with the received broadcast television programming and to thereby recognize the auxiliary content activation signal,
wherein the auxiliary content activation signal initiates processing of the data associated with the auxiliary content that is stored in the data storage to thereby render the auxiliary content to the viewer for viewing via the second screen device at a time after the live broadcast in time synchronization with the presentation of the recorded television program.