Patent ID: 12225263

In accordance with common practice the various features illustrated in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, the drawings may not depict all of the components of a given system, method or device; because, known methods, components, and circuits have not been described in exhaustive detail. Finally, like reference numerals may be used to denote like features throughout the specification and figures.

DETAILED DESCRIPTION

The various implementations described herein include systems, methods and/or devices that are enabled to determine what a particular consumer is watching on television in order to deliver matched functionality, content and/or services, as a part of an interactive television suite of technologies. In particular, in some implementations, the process of determining what a particular consumer is watching is enabled so that communication between the primary set-top box and a supplemental set-top box and/or television-integrated device is reduced, thereby avoiding the need for one or more provider specific APIs. For example, one aspect of the disclosure is a method of identifying a media program by capturing display data from the media program, and extracting text from the display data in response to determining that the display data includes the text overlay, wherein the extracted text is associated with the media program. In some cases, a media program is identified by (1) generating one or more content fingerprints from a portion (e.g., audio or video) of the media content program, and (2) comparing the one or more content fingerprints with a set of fingerprints corresponding to (e.g., extracted from) a known media program.

Additionally and/or alternatively, in some implementations, a supplemental set-top box (or television integrated device) is configured to operate a primary set-top box, without a provider-specific API, in order to determine the channel line-up available to a particular subscriber through the primary set-top box. In addition, in some implementations, similar methods and features can be used to identify channel lineups available to viewers of non-subscription media content, such as over the air broadcasts that are received by a TV receiver, such as a TV receiver included, without limitation, in a computer, a media player, such as a DVD or Blu-ray player, or a digital video recorder (DVR). In some implementations, it is useful to make control of the menus of a second device from a first device more reliable, even if the second device does not provide access to TV program. These methods are advantageous, because they make universal remote controls more reliable.

A common factor in each of these situations is that the supplemental set-top box does not have direct knowledge of the TV channels that are available to the primary set-top box or other TV receiver. Consequently, the supplemental set top box needs to implement indirect strategies to identify a TV channel that is being tuned by, or entire channel lineup available to, the primary set-top box or other TV receiver.

Numerous details are described herein in order to provide a thorough understanding of the example implementations illustrated in the accompanying drawings. However, the invention may be practiced without these specific details. And, well-known methods, procedures, components, and circuits have not been described in exhaustive detail so as not to unnecessarily obscure more pertinent aspects of the example implementations.

FIG.1Ais a diagram of an implementation of a client-server environment100. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, the client-server environment100includes a set-top box103, a television (TV)110, a client device102(e.g., a supplemental set-top box), a second screen client device120(e.g. a laptop, a tablet computer, a smartphone, etc.), a remote control device112, a communication network104, a media server130, a broadcast system140, and a content provider150. The second screen client device120, the client device102, the media server130, the broadcast system140, and the content provider150are capable of being connected to the communication network104in order to exchange information with one another and/or other devices and systems. The client-server environment100is merely an example provided to discuss more pertinent features of the present disclosure. Those skilled in the art will appreciate from the present disclosure that any number of such devices and/or systems may be provided in a client-server environment, and particular devices may be altogether absent.

In some implementations, the media server130is implemented as a single server system, while in other implementations it is implemented as a distributed system of multiple servers. Solely for convenience of explanation, the media server130is described below as being implemented on a single server system. Similarly, in some implementations, the broadcast system140is implemented as a single server system, while in other implementations it is implemented as a distributed system of multiple servers. Solely, for convenience of explanation, the broadcast system140is described below as being implemented on a single server system. Similarly, in some implementations, the content provider150is implemented as a single server system, while in other implementations it is implemented as a distributed system of multiple servers. Solely, for convenience of explanation, the content provider150is described below as being implemented on a single server system. Moreover, the functionality of the broadcast system140and the content provider150can be combined into a single server system. Additionally and/or alternatively, while only one broadcast system and only one content provider is illustrated inFIG.1Afor the sake of brevity, those skilled in the art will appreciate from the present disclosure that fewer or more of each may be present in an implementation of a client-server environment.

In some implementations, the second screen client device120(e.g. a tablet computer or laptop, etc.), the set-top box103, the TV110, the client device102, and a remote control112are provided in combination within, for example, a home or commercial establishment. The set-top box103is configured to receive and decode signals including media content from the broadcast system140or the like. In operation, the set-top box103decodes the signal received from the broadcast system140or the like and provides audio and video content to the TV110for display. While a TV has been used in the illustrated example, those skilled in the art will appreciate from the present disclosure that any number of displays devices, including computers, laptop computers, tablet computers, smart-phones and the like, can be used to display a video stream and play the associated audio stream. Additionally and/or alternatively, in some implementations the functions of the set-top box103and the TV110are combined into a single device.

In some implementations, the remote control112is used to control the operation of one or more of the set-top box103, the TV110, and the client device102. In one mode of operation, the client device102monitors the display118of the TV110to determine when a program information overlay is provided. In another mode of operation, the client device102emulates the function of the remote control device112(e.g., by sending IR commands via an IR blaster125(FIG.1B)) to prompt the set-top box103to display the program information overlay. And in yet another mode of operation, the client device102receives IR transmissions from the remote control112that are directed to the set-top box103. In turn, the client device102may then communicate with any one of the media monitoring server130, the broadcast system140, and the content provider150through the communication network104. More specific details pertaining to how the client device102acquires real time program information related to what a particular user is watching is described below with reference toFIGS.3-5.

In some implementations, the set-top box103is any computer device capable of connecting to the communication network104, receiving video streams, extracting information from video streams and presenting video streams for the display using the TV110(or another display device). In some implementations, the set-top box103is configured as a receiver for cable TV and/or satellite TV, a digital video recorder (DVR), a digital media receiver, a TV tuner, a computer, and/or any other device that outputs TV signals. In some implementations, the TV110is a conventional TV display that is not connectable to the Internet and that displays digital and/or analog TV content received via over the air broadcasts or a satellite or cable connection.

Similarly, in some implementations the client device102is any computer device capable of connecting to the communication network104and the TV110. As described below with reference toFIG.5, in some implementations, the client device102is capable of receiving infrared (IR) transmissions directed to the set-top box103from the remote control device112. Additionally and/or alternatively, in some implementations, the client device102is capable of wired and/or wireless communication with the client device120. In some implementations, the client device102includes one or more processors121, non-volatile memory122such as a hard disk drive, and a local database123. In some implementations, the memory122includes application instructions122aand associated data122b. In some implementations, the local database123and/or the associated data122bstores information related to local channel lineups, including one or more local channel lineups180-idiscovered through operations described herein. In some implementations, the operations are performed by the application instructions122a.

As discussed in greater detail below, the processor121executes the one or more applications in accordance with a set of instructions received from the media monitoring server130. The client device102may also have input devices such as a keyboard, a mouse and/or track-pad (not shown). In some implementations, the client device102includes a touch screen display, a digital camera and/or any number of supplemental devices to add functionality.

As is typical of televisions, the TV110includes a display118and speakers. Additionally and/or alternatively, the TV110can be replaced with another type of display device for presenting video content to a user, such as for example, a computer, a tablet device, a mobile telephone, a projector, or other type of video display system. The display device can be coupled to the set-top box103via a wireless or wired connection.

As used herein, a TV signal is an electrical, optical, or other type of data transmitting medium that includes audio and/or video components corresponding to a TV channel. In some implementations, the TV signal is a terrestrial over-the-air TV broadcast signal or a sign distributed/broadcast on a cable-system or a satellite system. In some implementations, the TV signal is transmitted as data over a network connection. For example, the set-top box103can receive video streams from an Internet connection. Audio and video components of a TV signal are sometimes referred to herein as audio signals and video signals. In some implementations, a TV signal corresponds to a TV channel that is being displayed on the TV110.

FIG.1Bis a diagram of a portion of the client-server environment100including an example schematic screen display on the TV110. Again, the TV110includes the display118on which video of a media program is displayed. Further, while the video is being displayed, the set-top box103may provide a text overlay119that includes channel and title information. For example, the text overlay119typically includes the channel the media program is being presented on, as well as the title of the media program. The text overlay119also often includes information about actors, characters, and/or a synopsis of the media program. In some implementations, as shown inFIG.1B, the client102is connected to the set-box103via a connection131(e.g., an HDMI, connection, a cable connection, or a wireless connection) by which the client102can receive video signals171afrom the set top box103the same as or comparable to the video signals171bprovided by the set top box103to the television110via the connection135(e.g., an HDMI connection). This allows the client102to analyze and perform optical character recognition (OCR) on the text overlay119provided by the set top box103along with the video and/or audio content and/or metadata associated with the show being watched (all provided as part of the video signals171). In some implementations, the connections131,135between the set top box103and the TV110and/or the client102are substantially or entirely one way. That is, while it is possible for the set top box103to provide signals to the TV110and the client102, it is not possible for the TV110and the client102to control the set top box103via commands sent back to the set top box103via the connections131and135. In some implementations, the client102is connected with the television110via the connection137(e.g., a wireless connection, such as a WIFI, BLUETOOTH, RF, or IF connection).

With further reference toFIG.1A, the second screen client device120may be any computer device that is capable of connecting to the communication network104, such as a computer, a laptop computer, a tablet device, a netbook, an Internet kiosk, a personal digital assistant, a mobile phone, a gaming device, or any other device that is capable of communicating with the media server130.

The communication network104may be any combination of wired and wireless local area network (LAN) and/or wide area network (WAN), such as an intranet, an extranet, including a portion of the Internet, or a wireless network provided by a wireless carrier. It is sufficient that the communication network104provides communication capability between the second screen client device120and the media monitoring server130. In some implementations, the communication network104uses the HyperText Transport Protocol (HTTP) to transport information using the Transmission Control Protocol/Internet Protocol (TCP/IP). HTTP permits the client device102to access various resources available via the communication network104. However, the various implementations described herein are not limited to the use of any particular protocol.

In some implementations, the media monitoring server130includes a front end server134that facilitates communication between the media monitoring server130and the communication network104. The front end server134receives content information from the client device102. As described in greater detail below, in some implementations, the content information includes program information, such as the program title, actor names, character names, plot summaries, etc. In some implementations, the front end server134is configured to send a set of instructions to the client device102. In some implementations, the front end server134is configured to send content files, links to content files and/or metadata associated with the content files. The term “content file” includes any document or content of any format including, but not limited to, a video file, an image file, a music file, a web page, an email message, an SMS message, a content feed, an advertisement, a coupon, a playlist or an XML document. In some implementations, the front end server134is configured to send or receive one or more video streams. In some implementations, the front end server134is configured to receive content directly from the broadcast system140and/or the content provider150over the communication network104.

In some implementations, the media monitoring server130includes a content database136. In some implementations, the content database136includes advertisements, videos, images, music, web pages, email messages, SMS messages, content feeds, advertisements, coupons, playlists, XML documents, and ratings associated with various media content or any combination thereof. In some implementations, the content database136includes links to advertisements, videos, images, music, web pages, email messages, SMS messages, content feeds, advertisements, coupons, playlists, XML documents and ratings associated with various media content. In some implementations, the content database136is a distributed database.

In some implementations, the broadcast system140includes media content such as TV programs. In some implementations, the broadcast system140also includes metadata associated with one or more of the TV programs, and application program interface instructions. In some implementations, the application program interface instructions include instructions that can be provided to a client device and/or may be executed by the broadcast system140under a client-server model or the like in order to link a particular TV program with the associated metadata and/or link the TV program and the associated metadata with external data and/or services, such as a social networking application. Similarly, in some implementations, the content provider150additionally and/or alternatively includes media content such as TV programs, as well as metadata associated with one or more of the TV programs.

FIG.2Ais a diagram of an example implementation of the client device102, discussed above with reference toFIG.1A, and which may be operable as a supplemental set-top box and/or a television-integrated device. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, the client device102includes one or more processing units (CPU's)202, one or more network or other communications interfaces208, memory206, a TV tuner232, a video I/O230, and one or more communication buses204, for interconnecting these and various other components. The communication buses204optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components. The client device102may also include a user interface212comprising a display device213and a keyboard and/or mouse (or other pointing device)214and/or a remote control (etc.). Memory206includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory206may optionally include one or more storage devices remotely located from the CPU(s)202. Memory206, or alternatively the non-volatile memory device(s) within memory206, comprises a non-transitory computer readable storage medium. In some implementations, memory206or the computer readable storage medium of memory206store the following programs, modules and data structures, or a subset thereof, an operating system216, a network communication module218, an OCR module226, software modules228, and data220.

The operating system216includes procedures for handling various basic system services and for performing hardware dependent tasks.

The network communication module218facilitates communication with other devices via the one or more communication network interfaces204(wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on.

In some implementations, the OCR module226enables the client device102to process screen shots (e.g. images from the TV110) to obtain the program information included in the program information overlay provided by the primary set-top box103. To that end, the OCR module226includes OCR instructions324and metadata325. In some implementations, the OCR instructions324include instructions that perform OCR operations to recognize program information from TV images and content and the metadata325includes data employed by the OCR module to support OCR operations. For example, in some implementations the metadata includes information related to common combinations of words associated with TV channels and programs, such as channel names, broadcaster names, and words commonly used in program titles. In some implementations, e.g., when the client102is equipped with appropriate processing power, screen shots are processed locally on the client102, e.g., so as to avoid delay caused by transmitting the screen shots to a remote sever.

In other implementations, processing screen shots (e.g., one or more of the OCR operations) take place on a server computer connected with the client102, e.g., via the communication network104. In some implementations, the client102transmits unprocessed (e.g., raw) or partially processed screenshots to a server for processing. In some implementation, the client102compresses (e.g., applying one or more compression algorithms, lossy or lossless) or adjusts (e.g., transmitting black and white images) the screen shots, e.g., so as to consume less network bandwidth.

In still other implementations, where load balancing techniques are used, screen shot processing (e.g., one or more of the OCR operations) take place partially on the client102and partially on a server computer. For example, in a set of 10 screen shots, high resolution screen shots processed locally (e.g., so as to avoid a potentially lengthy transmission delay), and low resolution screen shots are transmitted to a server for batch processing, which are usually quicker. For another example, in a set of 1000 screen shots, the first 500 screen shots are processed locally, while the second 500 screen shots are transmitted to a server for parallel or concurrent processing, e.g., so as to reduce overall processing time.

In still further implementations, some stages of screen shot processing take place on a client, while other stages take place on a server. For example, after a screen shot is taken, a client applies OCR techniques to the screen shot, and resulting text (produced by the OCR techniques), and/or associated annotations, are sent to a remote server, which then decides what instructions execute on a set top box (e.g., a channel switch, recording a user preference, and displaying a channel listing).

In some implementations, the software modules include a display/user module234, a channel lineup manager module236, a channel review module238, and a channel selection module240. In some implementations, the display/user module234displays information provided by other software modules. The displayed information can include a channel lineup280discovered by some combination of the OCR module226, channel lineup manager module436, channel review module238, channel selection module240, and a command sending module242.

In some implementations, for some functions such as changing channels (e.g., to extract channel lineup), a process sometimes called (audio or video) fingerprinting is used in lieu of OCR techniques. In these implementations, the client102includes a shared capture module (or infrastructure) that captures audio or video content on all live channels. The captured fingerprints are then extracted (by the client102or by a server computer). For example, when a first electronic device (e.g., the set-top box103or the television110) tunes to a particular channel, a second electronic device (e.g., the client102or the media server130) computes a fingerprint from a portion of audio or video content coming from the first electronic device. The fingerprint is sent to a server, which compares the fingerprint from the first device with the captured fingerprint. If there is no match, this is an indication that the channel is not available or that the tuning failed (e.g., a bad IR code). If there is a match, the specific channel can be added to the lineup.

In some implementations, the channel lineup manager module236maps the discovered channel numbers to actual over the air channels as applicable and to cable channel names (e.g., by using the channel map information282). The channel review module238enables a user to review, correct and remove channel information. In some implementations, the channel selection module440detects tuned signal strength and identifies whether a tuned channel is presenting non-superfluous content—i.e., anything that is not noise. In some implementations, the channel selection module240also allows a user to tune channels on the TV using simple channel up/down commands, or by entering an actual channel no., and causes appropriate channel information to be displayed on the TV. In some implementations, the channel selection module440employs an IR blaster125(FIG.1B) to send appropriate channel tuning/selection and “info” commands to the STB (or alternatively, the TV110). In some implementations, the channel selection module240and other modules or operations described herein that interact with the IR blaster125to send commands to the TV110or the STB125do so via a command sending module242. Thus, using the recognized channel lineup information, a user is able to control the TV110via the client device102(which only possesses a one-way connection to TV and the STB) as if the client device102possessed a two-way connection to the TV, similar to that of the STB103.

In some implementations, since channel information is not provided to the client device102via the connection with the set top box103, presenting channel information requires the client102or an interconnected sever to perform fingerprinting (e.g., of audio or video content snippets from media content played on the television110) to confirm that the correct channel has been selected. This also means that the client device102needs to be able to issue an appropriate number of channel “Up” or “Down” commands (e.g., IR commands sent using the IR blaster12(FIG.1B)) to the cable/set top box103(FIG.1B) to navigate to the desired channel and then confirm that the correct channel has been arrived at. In some implementations, this is done by: (1) sending an “Info” IR command (similar to a user hitting the “Info” button on the TV remote control) and doing a quick OCR check of the resulting displayed on screen information—or (2) by fingerprinting the channel content confirm that it matches what the requested channel selection should be showing.

In some implementations, the display/user module234displays information provided by other software modules. The displayed information can include a channel lineup180discovered by some combination of the OCR module226, channel lineup manager module236, channel review module238and a channel selection module240. In some implementations, the data220includes OCR data261and a local database123of program information. In some implementations, the program information is received from a broadcast system and/or content provider. In some implementations, the OCR data261includes text segments261-1, images261-2and user ID information261-3. In some implementations, the local database123includes one or more discovered channel lineups280-iand channel map information282that stores information that maps channel numbers to actual over the air channels as applicable and to cable channel names.

In some implementations, each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and each of the modules or programs corresponds to a set of instructions for performing a function described above. The set of instructions can be executed by one or more processors (e.g., the CPUs202). The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures or modules, and thus various subsets of these modules may be combined or otherwise re-arranged in various implementations. In some implementations, memory206stores a subset of the modules and data structures identified above. Furthermore, memory206may store additional modules and data structures not described above.

FIG.2Bshows information associated in some implementations with individual channels458saved in a channel lineup280-1stored in the local database123. This information, or subsets or supersets thereof, can be associated with any number of one or more of the channel lineups280-i. In the illustrated implementation(s), a representative first channel lineup280-1includes, for an individual channel458-1in the lineup, one or more of: input source information460-1, displayed channel number468-1, displayed channel name470-1, meaningful channel number472-1, meaningful channel name474-1, signal strength476-1and broadcast location478-1. In some implementations, the input sources460-1indicate the source of a respective particular channel using Yes/No or similar binary values. In some implementations the input sources460-1indicate whether the tuned channel is from a cable provider462-1, an over the air source464-1(such as a terrestrial broadcast TV channel) or from a satellite source468-1. Alternatively, the input sources460-1can be represented with a single field that holds a value that indicates the source for the respective channel (e.g. “OTA,” “Cable,” or “Satellite-Dish”). The displayed channel number468-1and name470-1are the channel number and name displayed on the TV display110(FIG.1B) and captured via OCR operations. The displayed channel number468-1and name470-1might not be meaningful to a user (e.g., the channel number468-1could be a three digit value assigned by a cable company with no relationship to a known channel number). As a result, the channel information includes in some implementations, for each tuned channel, a meaningful channel number472-1(e.g., a recognizable over the air channel number) and channel name474-1(e.g., a recognizable network or channel name). The channel information for a representative channel458-1also includes in some implementations information related to channel reception and transmission, including signal strength476-1and broadcast location478-1.

FIG.3is a flowchart representation of an implementation of a method300of determining what a consumer is currently watching. In some implementations, the method is performed by a device, such as a supplemental set-top box and/or a television integrated module, in order to enable determining what a consumer is currently watching. For example, with reference toFIG.1A, in some implementations, the method300is performed by the client device102(i.e., the supplemental set-top box). As noted above, when a user changes the channel or requests program information by, for example, using one or more channel selection options or pressing an “INFO” button, the primary set-top box displays an overlay including program information, such as the current channel, the title of the playing program and information about the program. Accordingly, the method includes identifying instances during which the primary set-top box provides the overlay by monitoring the display output provided by the primary set-top box, and parsing the overlay for the desired information. Additionally and/or alternatively, in some implementations, the method includes detecting when channel identifiers are displayed without being prompted by user interaction (i.e., a user pressing a button)

To that end, the method includes performing a screen or display data capture from the display device that is displaying the television program selected by the consumer through a primary set-top box or the like in order acquire an image of what is being displayed (301). In some implementations, the screen capture data is obtained from the video signal provided by the primary set-top box. Additionally and/or alternatively, in some implementations, the screen capture data is obtained from a display (i.e., video) card included in the set-top box and/or the operating system of the set-top box. Additionally and/or alternatively, the screen capture data is obtained from a TV app running on at least one of the primary set-top box, the supplemental set-top box, the TV, and a second screen device. Additionally and/or alternatively, in some implementations, the supplemental set-top box (or the like) includes a camera operable to capture an image of the display of a television or the like. The method includes analyzing the image to identify whether the image includes indicators that an overlay is being displayed by the primary set-top box (302). For example, in some implementations, the indicators include the color of the overlay, the percentage of display area the overlay typically covers when visible, the arrangement of text with the overlay, contrast between a portion of the display likely to be displaying the program and another portion of the display likely to be displaying the overlay.

The method includes determining whether one or more of the indicators are present as indicated by the image analysis (303). If none of the indicators are present for a particular screen capture (“No” path from303), the method includes waiting a pseudo-random amount of time or a preset delay time (304) before capturing another screen shot or the like. On the other hand, if at least one indicator is present or another threshold number of the indicators is present (“Yes” path from303), the method includes applying an optical character recognition technique to the image in order to identify and extract the displayed information (305). Having extracted the title and/or program information, the method includes cross-referencing the extracted information with a local and/or remote database to ensure the validity of the information (306). For example, with further reference toFIG.1A, the client device102may communicate with the media server130in order to check the validity of the extracted display information. More specifically, the client device102may transmit all or a portion of the extracted information to the media server130. In turn, the media server130may then check the extracted information against a content database (e.g., the content database133,FIG.1) to ensure that the extracted data is correct and/or provide corrected information to the client device102when the extracted data is incorrect but can be matched to database information.

FIG.4is a flowchart representation of an implementation of a method400of determining what a consumer is currently watching. In some implementations, the method is performed by a device, such as a supplemental set-top box and/or a television integrated module, in order to enable determining what a consumer is currently watching by, in part, emulating the operation of a remote control device associated with the primary set-top box. To that end, the method includes transmitting a first infrared (IR) code (or the like) to the primary set-top box in order to initiate the display of the program information overlay (401). For example, with further reference toFIG.1A, one of the client device102and the client device120may transmit an IR code to the set-top box103.

The method then includes performing a screen capture to acquire a screen shot (i.e. image) including the program information overlay (402). Having acquired the screen shot, the method includes transmitting a second IR code (or the like) to the primary set-top box in order to initiate the removal of the program information overlay on the playing television program (403). The method includes analyzing the image or screen shot to obtain the program information presented in the overlay (404). Having extracted the title and/or program information, the method includes cross-referencing the extracted information with a local and/or remote database133to ensure the validity of the information (406).

FIG.5is a flowchart representation of an implementation of a method500of determining what a consumer is currently watching. In some implementations, the method is performed by a device, such as a supplemental set-top box and/or a television integrated module, in order to enable determining what a consumer is currently watching, by detecting IR transmission from a remote control device associated with the primary set-top box. For example, with further reference toFIG.1A, the client device102may detect IR transmissions directed to the primary set-top box103from the remote control device120.

To that end, the method includes detecting an IR transmission or equivalent from a remote control device associated with a primary set-top box (501). In turn, the method includes parsing the transmission in order to identify the command that is represented by the IR transmission (502). Having identified the command, the method includes determining whether or not the identified command causes the primary set-top box to provide the program information overlay in the video stream provided to the display device. If the detected command is not likely to cause the primary set-top box to provide the program information overlay (“No” path from503), the method includes circling back to detect another IR transmission. On the other hand, if the detected command is likely to cause the primary set-top box to provide the program information overlay (“Yes” path from503), the method includes performing a screen or display data capture from the display device that is displaying the television program selected by the consumer through a primary set-top box or the like in order acquire an image of what is being displayed (504). The method includes analyzing the image or screen shot to obtain the program information presented in the overlay (505). And, the method includes cross-referencing the extracted information with a local and/or remote database to ensure the validity of the information (506).

FIG.6is a flowchart representation of an implementation of a method600of determining channel line-up access available through a primary set-top box. In some implementations, the method is performed by a device, such as a supplemental set-top box and/or a television integrated module, in order to enable determining channel line-up access available through a primary set-top box by using the primary set-top box channel control commands. For example, with further reference toFIG.1A, the client device102may transmit IR transmissions to the primary set-top box103or provide equivalent signaling.

To that end, the method600includes selecting an initial candidate channel to probe using the primary set-top box channel control commands (601). In some implementations, a candidate channel is any one of the suspected channel numbers that may be available through the primary set-top box. Having selected a candidate channel, the method600includes transmitting a control command to the primary set-top box to tune the primary set-top box to the candidate channel (602). For example, in some implementations, transmitting a control command includes transmitting an IR code (602a). In some implementations, transmitting a control command includes transmitting a command packet over a data connection (602b).

Subsequent to transmitting the control command, the method600includes performing a screen data capture (603). In some implementations, performing a screen data capture includes reading a display pixel buffer and/or reading from a display pixel bus in order to obtain at least a single frame of screen display information (603a). The method600then includes performing an image analysis to determine if the candidate channel is available as a part of a subscription associated with the primary set-top box, and thus delivering non-superfluous television content (604). In some implementations, non-superfluous television content includes television programs, sporting events, news broadcasts, movies, etc. In some implementations, superfluous television content includes data such as pay-per-view title screens, subscription information (e.g. a contact number or website address), looping advertisements and/or content trailers indicating the type of content that would be available to a subscriber of the channel that is currently not available because the primary set-top box is not associated with such a subscriber. In some implementations, performing image analysis includes performing and/or applying OCR to identify text content in the screen display data (604a). In some implementations, performing image analysis includes fingerprinting the at least one frame of screen display data to identify the program content (604b). In some implementations, performing image analysis includes performing a video analysis to determine whether or not the candidate channel is being used to deliver non-superfluous television content (604c). In some implementations, in addition to, or as a substitute for visual analysis604c, the channel selection model440(FIG.2A) or equivalent functionality determines (604d) whether a particular channel is carrying non-superfluous television content by analyzing signal strength of channels.

Based on the image analysis, the method600includes deciding whether the candidate channel is being used to deliver non-superfluous television content (605). If the candidate channel is being used to deliver non-superfluous television content (“Yes” path from605), the method600includes marking the candidate channel as available or a member of the subscription line-up available through the primary set-top box (606). If the candidate channel is not being used to deliver non-superfluous television content (“No” path from605), the method600includes marking the candidate channel as unavailable or not a member of the subscription line-up available through the primary set-top box (607). Information for active channels is then saved in one of the channel lineups180(611). Information saved for the active channels includes information recognized from the displayed channel information (such as channel number and name) and information obtained from local or external sources, such as a program guide stored on the client device102or a server that is coupled to the client device102via a network104(e.g., the media server130,FIG.1A).

In some implementations, when there are multiple input sources, such as cable TV, over the air TV, and satellite TV, channels from a particular input source are saved and stored in a respective channel lineup. For example,FIG.2Aillustrates two saved channel lineups180-1and180-2. Also, in some implementations, to make the channel lineups more user-friendly and/or to identify channels that are duplicated across multiple lineups, the displayed and OCR'd channel information is mapped to meaningful names/channel numbers, such as local over the air (OTA) channel numbers or channel names, such as network names (612). These meaningful channel names and numbers are then saved in the channel lineups along with the recognized displayed channel information. The information (e.g., channel map information282) used to map the OCRed channel information to meaningful channel names and numbers can be previously stored on the client device102and/or downloaded from the server. Alternatively, the client device102can issue one or more requests to the server to map the channel information based on information available to the server for channel lineups available in the area of the client device102for different input sources/content providers.

Subsequent to deciding upon the status of the candidate channel, the method600includes determining whether or not to continue based on the suspected existence of additional channels (608). In some implementations, at least one addition candidate channel is suspected to exist when each of the candidate channels considered thus far has only been considered once and/or an equal number of times without overlap. If at least one more channel is not suspected to exist continue (“No” path from608), the method600ends (610). If at least one more channel is suspected to exist continue (“No” path from608), the method600includes selecting a new candidate channel and reiterating the portion of the method starting at block602(609). In some implementations, selecting a new candidate channel includes using the “channel up” (or “channel down”) primary set-top box control command (609a). In some implementations, selecting a new candidate channel includes selecting a new channel from a list or the like (609b). In addition, the method described with reference toFIG.6(and the related methods described with reference toFIGS.4and5) can be used to identify channel lineups available to viewers of non-subscription media content, such as over the air broadcasts that are received by a TV receiver, such as a TV receiver included, without limitation, in a computer, a media player, such as a DVD or Blu-ray player, or a digital video recorder (DVR).

In some implementations, in addition to TV content, other types of media content and services can be provided through the primary set-top box103. The content or other services can be provided via a direct (wired or wireless) connection to the set-top box103, as built-in features of the set-top box103, or via a network connection. For example, a digital video recorder (DVR) built into a set-top box can be used to record TV content or playback recorded TV content; the set-top box103can be used to access video-on-demand (VOD) content available from the cable or satellite provider associated with the set-top box; or the set-top box can be used to monitor or control a home security system. Using capabilities similar to those described above with reference toFIGS.1-6, in some implementations a client device102is configured to receive display information from services/features provided via a primary STB103and to interact with (e.g., control and access) those services without using custom APIs for a particular primary set-top box.

For example, in some implementations, in a trial and error process the client device102sends via a one-way wireless connection provided by the set top box (such as an IR or RF interface) a sequence of one or more commands to cause the set-top box103to exercise certain features/services and/or access certain STB content (e.g., DVR services or VOD content). In some implementations, consistent with techniques described above, the client device102then analyzes display characteristics (e.g., position, size, graphics and content) of text overlays and/or other images/information associated with the resulting images displayed by the television110in response to the one or more commands and determines whether those text overlays are as expected given the one or more commands that were sent. In some implementations, the client device102performs this determination by comparing the identified display characteristics of the resulting text overlays to a database of expected overlay display characteristics generated by particular makes and models of set top box in response to particular commands associated with particular set-top box feature sets. If the identified overlay display characteristics are consistent with the expected overlay display characteristics, the client device102stores information identifying the set top box's make and model, as well as correct sequence of one or more commands used to access particular features. In this way, the client device102can identify useful commands and features provided by the set-top box103and subsequently control those features using the identified command sequences.

In some implementations, once the client device102has identified the make, model and feature set of the set top box103, the client device102can download from another device (e.g., a network server) command sequences for controlling the set-top box103.

In any of these contexts, the command sequences can be conventional remote control commands supported by the STB130, which the client device102can send using a compatible IR or RF transmitter, or network commands (e.g., WiFi or Bluetooth) supported by the STB103, which the client device102can send via its own network connections. In this way, the client device102is able to control the feature set provided by the set top box103—even though the only connections provided by the set top box103are one way connections (e.g., output audio and video signals with no control capabilities or remote control connections). Of course, the methods described herein are applicable to any communication modalities that are supported by a STB to enable control of its feature set and access to video and audio signals that reflect its outputs. Similarly, the methods described herein are applicable to identifying command sequences and controlling feature sets of any electronic device via comparable one way connections.

As a further illustration,FIG.7is a diagram of a portion of the client-server environment100including an example schematic screen display on the TV110and showing additional features provided by the set-top box103and the client device102ofFIG.1. The TV110includes the display118on which video of a media program is displayed. In addition to receiving and displaying TV content on the TV110(or other display), the set-top box103is configured with a variety of services/features, including, without limitation, video on demand content703, digital video recorder (DVR) features705and home security system features707. These services/features can be provisioned to/by the set-top box103in a variety of ways. For example, the VOD content703is provided by the content provider associated with the STB103(i.e., the STB provider), the DVR services/features705can be provided by computing and storage components internal to the STB103or remotely by the STB provider, and the home security system services/features707can be accessed via a home network or in some cases by direct connection to the STB103.

In some implementations, representative VOD features703include allowing a user to view and select video content from a list of available VOD programs721; representative DVR features705include allowing a user to view and select from a list of recorded programs723and view and/or define a recording schedule725; and representative home security features707that can be accessed through the STB include allowing a user to view images captured from security cameras727, view and reset security system state729and view and update and security system configuration731. In some implementations, state information associated with these features, which is relevant to controlling the features, might not be directly displayed by the STB103in response to user commands (including commands sent by the client device102), but these states can be ascertained by the client device102by sending particular command sequences and observing the resulting video and audio signals171a,773aprovided by the STB103.

Similarly, the STB103has its own features and settings that can be identified or controlled by a user or a client device102with the appropriate command sequences. These features and settings include, without limitation, allowing a user to view information pertaining to: STB provider and channel/feature lineup741, available channels743, and on screen menus745to allow a user to interact with those and other features (such as a top level menu to allow the user to access VOD, DVR and home security services/features). The STB103also has state information that is relevant to controlling the STB103and its associated services/features. In some implementations this state information includes menu state747(i.e., the currently displayed menu), on/off state749(e.g., whether the STB is on or off) and the STB configuration751(e.g., without limitation, its provisioned services and features and the types of its available and/or active connections—such as the fact that an example STB103it supports WiFi commands, has IR and RF command receivers, is DLNA compatible, has 2 HDMI outputs, including one HDMI 2.0 output, and has component video and S-video outputs).

As shown inFIG.7, and consistent withFIG.1B, in some implementations the client device102is any computer device capable of connecting to the communication network104and the TV110, receiving audio and video signals171a,773afrom the set box103, and issuing commands777to the set-top box103. In some implementations, the client102is connected to the set-box103via a connection131by which the client102can receive video signals171aand audio signals773afrom the set top box103the same as or comparable to the video signals171band audio signals773bprovided by the set top box103to the television110via the connection135. This allows the client102to analyze and perform optical character recognition (OCR) on the text overlay119provided by the set top box103along with the video and/or audio content and/or metadata associated with the show being watched (all provided as part of the video signals171). In some implementations, the communications131,135between the set top box103and the TV110and/or the client102are substantially or entirely one way. That is, while it is possible for the set top box103to provide signals to the TV110and the client102, it is not possible for the TV110and the client102to control the set top box103via commands sent back to the set top box103via the connections131and135. Consistent with descriptions ofFIG.1B, in some implementations the client is configured to issue commands777to the set-top box103by a connection775that can be one or more of a network connection (such as a WiFi or Bluetooth network connection) or RF or IR connections). As described with reference toFIG.5, in some implementations, the client device102is capable of receiving infrared (IR) transmissions directed to the set-top box103from the remote control device112. Additionally and/or alternatively, in some implementations, the client device102is capable of wired and/or wireless communication with the client device120, the set top box103and the TV110.

In some implementations, the client device102includes one or more processors121, non-volatile memory122such as a hard disk drive, and a local database123. In some implementations, the memory122includes application instructions122aand associated data122b. In some implementations, as described above, the local database123and/or the associated data122bstores information related to local channel lineups, including one or more local channel lineups180-idiscovered through operations described above. In some implementations, the local database123and/or the associated data122balso store data and commands related to accessing and controlling services and features provided by the STB103. In some implementations, the operations for controlling the STB103, including sending commands to the STB103, are performed by the application instructions122a. In some implementations, the client device102includes IR and/or RF blasters/transmitters to send RF or RF commands777to the set top box103, the TV110or other compatible devices. In some implementations, the client device includes a microphone to receive audio signals773provided by the STB103in order to assess the current state and/or programming and/or feature set of the STB103.

Further, while the video is being displayed, or in response to particular commands sent to the set-top box103, the set-top box103may provide a variety of text overlays119,751,753that include information related to an associated STB service or displayed content type. For example,FIG.7shows example text overlays119,751,753associated respectively with TV program content, features and content associated with a DVR705, and on demand programs available from a video on demand provider703. The set-top box103can also provide access to other services and features, such as services/features of a home security system707controlled through the set-top box103. Each of these features provides via the set-top box103has its own associated text overlays for display with the on screen content755presented on the TV110. The purpose of these overlays and other information is to enable user interaction with the associated services and features. As described with reference toFIGS.1-6, the overlay information can be stimulated, received and analyzed by the client device102in order to provide interactive, two-way control of the STB103— even though the STB is not natively configured with connections to permit the client device102such level of control. Also, as described with reference toFIGS.1-6, in some implementations, the overlay information can be passively received by the client device102(e.g., when generated in response to a user command to the STB103) and then analyzed by the client device102in order to identify a make and model of the STB as well as a current state of the STB103.

In different implementations, the one or more overlays associated with respective STB features and services are typically presented with associated screen positions, bounding box size (e.g., the size of the overlay either as x and y pixel dimensions or a percentage of screen occupied by the overlay), content, content styles and formatting, and graphics. The positions, bounding box sizes and content of the overlays118,751,753shown inFIG.7are simply examples of different styles of overlays119,751and753associated respectfully with TV content, DVR features and VOD features. As described with respect toFIG.1B, the text overlay119typically includes the channel the media program is being presented on, as well as the title of the media program. The text overlay119also often includes information about actors, characters, and/or a synopsis of the media program. The content of the DVR overlay751is based on information provided by features of a digital video recorder705provided by the set top box103, including, for example, a list of recorded programs and a recording schedule displayed in response to corresponding commands777sent by the client device102. The content of the VOD overlay753is based on information provided by VOD features provided by the set top box103, including, for example a list of available VOD programs displayed in response to corresponding commands777sent by the client device102. As noted above, these overlays are presented in a characteristic manner based on the associated make, model and feature set of the STB. Accordingly, these typical display characteristics can be used by the client device102to identify the associated make, model and feature set of the STB and to determine whether the client device102is sending the correct command sequences for controlling the STB and/or particular services and features provide by the STB.

FIG.8Ais a diagram of an example implementation of the client device102, discussed above with reference toFIG.1A, and which may be operable as a supplemental set-top box and/or a television-integrated device. In some implementations, the client device102includes all or a subset of the functionality described with respect toFIG.2Aand all or a subset of the features described with reference toFIG.8A. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, the client device102includes one or more processing units (CPU's)202, one or more network or other communications interfaces208, memory206, a TV tuner232, a video I/O230, and one or more communication buses204, for interconnecting these and various other components. The communication buses204optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components. The client device102may also include a user interface212comprising a display device213and a keyboard and/or mouse (or other pointing device)214and/or a remote control (etc.). Memory206includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory206may optionally include one or more storage devices remotely located from the CPU(s)202. Memory206, or alternatively the non-volatile memory device(s) within memory206, comprises a non-transitory computer readable storage medium. In some implementations, memory206or the computer readable storage medium of memory206stores the following programs, modules and data structures, or a subset thereof, including an operating system216, a network communication module218, an OCR module226, software modules228, and data220.

The operating system216includes procedures for handling various basic system services and for performing hardware dependent tasks.

The network communication module218facilitates communication with other devices via the one or more communication network interfaces204(wired or wireless) and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on.

In some implementations, the OCR module226enables the client device102to process screen shots (e.g. images from the TV110) to obtain the program information included in the program information overlay and information associated with services and features provided by the primary set-top box103. To that end, the OCR module226includes OCR instructions324and metadata325. As described with reference toFIG.2A, in some implementations, the OCR instructions324include instructions that perform OCR operations to recognize program information from TV images and content and the metadata325includes data employed by the OCR module to support OCR operations. For example, in some implementations the metadata includes information related to common combinations of words associated with TV channels and programs, such as channel names, broadcaster names, and words commonly used in program titles. In some implementations, the OCR instructions324include instructions that perform OCR operations to recognize information and graphical characteristics associated with other services and features provided by the STB103, such as DVR, VOD and home security services, and the metadata325includes data employed by the OCR module to support OCR operations. For example, in some implementations the metadata includes information related to common combinations of words and symbols associated with information overlays generated by different set-top box makes and models for different feature sets.

In some implementations, the software modules228include a display/user module234, a channel lineup manager module236, a channel review module238, a channel selection module240, a command sending module242, an STB command/response module814, a UI display comparison module and an STB control module. In some implementations, the display/user module234displays information provided by other software modules. The displayed information can include a channel lineup180discovered by some combination of the OCR module226, channel lineup manager module236, channel review module238, and channel selection module240. In some implementations, the channel lineup manager module236maps the discovered channel numbers to actual over the air channels as applicable and to cable channel names (e.g., by using the channel map information282). The channel review module238enables a user to review, correct and remove channel information. In some implementations, the channel selection module240employs an IR blaster125(FIGS.1B and7) to send appropriate channel tuning/selection and “info” commands to the STB (or alternatively, the TV110). In some implementations, the channel selection module240and other modules or operations described herein that interact with the IR blaster125or other command interfaces to send commands to the TV110or the STB103do so via a command sending module242. Thus, using the recognized channel lineup information, a user is able to control the TV110via the client device102(which only possesses a one-way connection to the STB and no prior knowledge of the applicable channel lineup) as if the client device102possessed a two-way connection to the STB103and had prior knowledge of the channel lineup. Similarly, a user is able to control and otherwise interact with services and features provided by the STB103(such as DVR and VOD services) via the client device102as if the client device102possessed a two-way connection to the STB103.

Related to identifying and controlling other features of the STB, the STB command and response module814includes instructions for sending commands to the STB103for identifying command sequences that can be used to control particular STB services and features (e.g., DVR and VOD services) and identifying responses, if any, to those commands based on review and analysis (supported by the other software modules as required) of the corresponding audio and/or video signals171a,773aprovided by the STB (collectively and/or individually referred to herein as display response data, response data or display responses). The STB command and response module814in some implementations also manages the trial and error process described herein that includes sending trial command sequences834to the STB103in order to identify correct command sequences836for a particular make and model of set top box103and saving locally the correct command sequences836for re-use (e.g., to permit the user to control the associated services and features via the client device102). In some implementations, the UI display comparison module816performs operations similar to those performed by the OCR426to analyze the response data and compare that data to known/expected response data (e.g., Expected Display Responses832) that would be generated by a range of different set top box makes and models for a wide range of services and features in response to specific commands and command sequences. The STB control module in turn is configured to allow the user to access the services and features provided by the STB103by providing access to the correct command sequences836for controlling those various services and features as if the user were doing so via an interface provided directly by the STB103.

The data220includes OCR data161described above (also as extended for use by the software modules described in reference toFIGS.8A-8Cand methods describe in reference toFIGS.9A-9B) and command response data that includes expected display responses832, trial command sequences834and correct command sequences836, already described. The data220also includes local database information123that includes, but is not limited to, data related to channel lineups280(described in reference toFIG.2A), channel map information282, STB state284, STB configuration286, DVR data2880, and VOD data290. These local database items280to290are described in greater detail in reference toFIG.8B.

FIG.8Bis a diagram showing additional details of the data220, which includes OCR data161, command/response data820, and local database data123. The local database data123further includes data regarding channel lineups280, STB state284, STB configuration286, DVR data288and VOD data290. The OCR data161is described above and the command/response data is described below, in reference toFIG.8C. Similarly, the channel lineups data280, which includes for the purposes of this discussion an identity840of the cable or satellite service provider, is described above in reference toFIGS.1-6. The channel lineups280, STB state284, STB configuration286, DVR data288and VOD data290include information that is relevant to and supports control by the client device102of the STB103. Much of this information has been described with reference toFIG.7in relation to features of the STB103. In some implementations, the STB state280-3includes a menu state844of the STB103, which includes, without limitation, an identification of the currently active STB menu and a representation of the menu structure supported by the STB103. The STB state284also includes an on/off data item846representing the current on/off state of the STB103. The STB configuration286includes information related to the hardware and software configuration of the STB103, such whether the user prefers high definition (HD) content848, and which input850to the STB103is preferred by the user. The client device102can use this information to set the configuration of the STB103when it is powered-up, or, conversely, to identify the default configuration of the STB103when it is powered-up. Actual STB configuration and STB state information286and284saved by the client device102can include more or different data items than those described herein; in fact, it is intended that these data items and other data items220described herein are configurable as required in respective implementations to save and provide information (e.g., STB state and configuration data) that supports control of the STB103by the client device102and/or use of the client device102for such purpose by a user. In some implementations, the DVR data288includes information related to recorded programs852available for playback from the DVR705and one or more recording schedules854for the DVR705. In some implementations, the VOD data290includes information related to one or more video on demand programs862requested by the user. By storing such information (in the cloud or locally) the client device102is able to support user interaction with the STB103with minimal involvement of the STB (e.g., the client device102can provide user access to recorded DVR programs without the need to access a list of such programs from the STB103).

FIG.8Cis a diagram showing additional details of the Command/Response Data820ofFIG.8B. This figure shows example data stored by the client device102in one implementation for a set of expected display responses832, trial command sequences834and correct command sequences. The trial commands sequences834includes a set of command sequences for a particular STB device type830-i, where a device type is typically a combination of STB make and model. For each STB device type830-i, the trial command sequences include a specification of a command sequence of one or more commands to access and control a particular feature type. For example,FIG.7includes trial command sequence information for three example device types (Cable Boxes1,2and3), where the sequence information is provided for a number of features. For example, for Cable Box1, the saved command sequences830-1includes sequences for at least three features, Feature Types1,2and3, each with different associated commands. For example, the command sequence for Feature Type1832-1includes one command Cmd1.1836-1for checking the STB On/Off state of appropriate to the make and model of Cable Box1. Other commands sequences836-2and836-3stored for Cable Box1device types include a sequence of commands836-2for Checking STB Provider (Feature Type832-2) and a sequence of commands836-3for Playback DVR Recordings (Feature Type832-3).

As described above, and further below in reference toFIGS.9A-9B, in some implementations the client device102identifies correct command sequences by issuing trial command sequences to the STB103and then comparing the resulting display from the STB103response to a database of expected responses832to see whether there is a match between the resulting and expected display responses. Data stored for the expected display responses represent video and audio (where applicable) characteristics of text overlays and other information provided by a particular type of STB103in response to particular commands (e.g., one of the trial command sequences836). Accordingly, in some implementations, the expected display responses834include, for a particular device type850-iand feature type852, associated display response characteristics, such as the bounding box size coordinates852and associated information content856generated in response to the associated command sequence836. In some implementations, the content856represents the combination of predefined and real time content—such as the actual channel number and identifier displayed in response to an INFO command a set of features—and any associated graphics—associated with a display response to a particular command sequence). In some implementations, the bounding box coordinate information854corresponds to overlay information such as that shown inFIG.7. For example, for the overlays119,751and753, the corresponding bounding box coordinates854would respectively represent position1, position2and position3.

The client device stores correct command sequences836identified through a trial and error process as described herein—or downloaded from a server based on a determined identification of the STB103type and feature set. The information stored for the correct command sequences836is comparable to the information stored for the trial command sequences834. For example,FIG.7shows that information for a Device Type2870-2has been saved as a correct command sequence. In some implementations, this information would be correspond to the trial command data for Device Type2830-2. Note that, in some implementations, the expected display responses832and trial command sequences834are stored in the cloud (e.g., at a server provided by the company that sold the client device or the associated software modules). Correct command sequences836can also be stored in the cloud, but it is generally helpful (e.g., to allow use of the command sequences with low latency in all circumstances) to store the correct sequences locally—at the client device. Methods for employing the data and modules described herein to allow a client device102to control a set-top box are now described with reference toFIGS.9A and9B.

FIGS.9A and9Bare flowchart representations of respective implementations of a method of controlling a primary set top box based on display information provided by a one way connection from the primary set top box.

Referring toFIG.9A, in some implementations, the method900is implemented at a first electronic device.

The method includes sending (902) a first command sequence to a second electronic device via a wireless connection different from a first output connection. The first command sequence is not known to be correct (904).

The method further includes, in response to (906) the sending, capturing (908) display data provided by the second electronic device via the first output connection; and analyzing (910) the display data to determine whether the first command sequence was performed correctly by the second electronic device.

When the first command sequence was not performed correctly, sending (906) a second command sequence to the second electronic device via the wireless connection. In some implementations, the second command sequence is not known to be correct.

In some implementations, the method optionally includes analyzing (914) the display data to determine whether the second command sequence was performed correctly by the second electronic device; and when the second command sequence was performed correctly, updating (906) a first database to identify the second command sequence as a correct command sequence for the second electronic device.

In some implementations, the first output connection of the second device is a one-way connection from the second electronic device to the first electronic device.

In some implementations, the first output connection provides video and audio data corresponding to features provided by the second electronic device.

In some implementations, the second electronic device provides a plurality of features that cannot be controlled by the first electronic device via the first output connection, the one or more programs further comprise instructions for: when the first and second command sequences are intended to control a first feature provided by the second electronic device, selecting the first command sequence and the second command sequence from a database of command sequences based on a device type of the first electronic device and a feature type of the first feature.

In some implementations, the second electronic device is a set top box and the plurality of features includes: video on demand services, digital video recorder services, EPG services, and set top box services.

In some implementations, the video on demand services include: selecting a video on demand selection for viewing.

In some implementations, the digital video recorder services include one or more of: (a) scheduling a recording of a content item provided via the set top box, (b) selecting for viewing a previously recording content item, (c) retrieving the list of programmed recording for determining user preferences, (d) modifying scheduling of a recording, and (e) deleting a recording of a content item.

In some implementations, the set-top box services include one or more of: (a) checking an on/off state of the set top box, (b) checking a menu state of the set top box, (c) checking a provider associated with the set top box, and (d) setting or determining a configuration of the set top box.

In some implementations, the method optionally includes, displaying a user interface on the display providing selectable user options for interacting with a plurality of features provided by the second electronic device; responsive to user selection of an option associated with the first feature, selecting the second command sequence from the first database; and issuing the second command sequence to the second electronic device.

In some implementations, analyzing the display data to determine whether the first command sequence was performed correctly by the second electronic device includes: based on a device type of the second electronic device and a feature type of a first feature intended to be controlled by the first command sequence: accessing a database of expected display responses, wherein the database of expected display responses indicates two or more of: (a) position, (b) bounding box locations, and (c) content of display information provided by the second electronic device for respective feature types; determining from the database of expected display responses a first expected display response associated with the first feature; comparing the captured display data to the first expected display response; and determining based on an outcome of the comparison whether the first command sequence was performed correctly by the second electronic device.

In some implementations, the method optionally includes: in response to the sending, capturing audio data provided by the second electronic device via the first output connection; and analyzing the audio data in combination with the display data to determine whether the first command sequence was performed correctly by the second electronic device.

In some implementations, the method optionally includes: when a particular device type of the second electronic device is unknown: analyzing the display data to determine whether the second command sequence was performed correctly by the second electronic device; and when the second command sequence was performed correctly by the second electronic device; updating a third database to indicate that the second electronic device is the particular device type.

In some implementations, analyzing the display data is performed at least in part at a server with which the first electronic device is in communication.

Referring toFIG.9B, in some implementations, the method950of controlling a primary set top box based on display information provided by a one way connection from the primary set top box is implemented at a first electronic device.

In some implementations, the first electronic device includes one or more processors and memory and is coupled to a display and a first output connection of a second electronic device.

In some implementations, the method950includes, sending (952) a first command sequence to the second electronic device via a wireless connection different from the first output connection. The first command sequence (954) is selected to determine a first state of the second electronic device. The method further includes, in response to (956) the sending, capturing (958) display data provided by the second electronic device via the first output connection; analyzing (960) the display data to determine the first state of the second electronic device; and based on the first state and a device type of the second electronic device, sending (962) a second command sequence to the second electronic device so as to control a feature provided by the second electronic device.

In some implementations, the second electronic device is a set-top box and the plurality of features includes: video on demand services, digital video recorder services, set top box services, and EPG services.

In some implementations, the video on demand services include: selecting a video on demand selection for viewing.

In some implementations, the digital video recorder services include one or more of: (a) scheduling a recording of a content item provided via the set top box, and (b) selecting for viewing a previously recording content item.

In some implementations, the set top box services include one or more of: (a) checking an on/off state of the set top box, (b) checking a menu state of the set top box, (c) checking a provider associated with the set top box, and (c) setting a configuration of the set top box.

It will also be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first electronic device could be termed a second electronic device, and, similarly, a second contact could be termed a first electronic device, which changing the meaning of the description, so long as all occurrences of the “first electronic device” are renamed consistently and all occurrences of the “second electronic device” are renamed consistently. The first electronic device and the second electronic device are both electronic devices, but they are not the same electronic devices.

The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the claims. As used in the description of the implementations and the appended claims, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting,” that a stated condition precedent is true, depending on the context. Similarly, the phrase “if it is determined [that a stated condition precedent is true]” or “if [a stated condition precedent is true]” or “when [a stated condition precedent is true]” may be construed to mean “upon determining” or “in response to determining” or “in accordance with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.

The foregoing description, for purpose of explanation, has been described with reference to specific implementations. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The implementations were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various implementations with various modifications as are suited to the particular use contemplated.