Patent Description:
However, these types of interaction devices and receivers do not allow for the storage of the interactions with the interaction device. Further, the interactions to control broadcast content may only be set at a local level, meaning interactions may control broadcast content differently depending on the interaction device and receiver. Thus, these types of interaction devices and receivers do not allow for control of broadcast content at a server level.

<CIT> discloses a method for controlling an image display apparatus that includes obtaining, by the image display apparatus, emotion information associated with a user of the image display apparatus, determining a contents to be recommended for the user among a plurality of different contents, based on the obtained emotion information associated with the user, and selectively or automatically reproducing one of the recommended contents at the image display apparatus.

<CIT> discloses a system for wireless control of home media sources that utilizes a home digital network to transport packetized IR transmitter data to a remote location. The system is described as allowing for flexible control of multiple devices that accept IR commands without requiring additional wiring, and is described as being independent of any particular IR protocol.

<CIT> discloses a content playback device that has a processor, a memory, a data acquisition portion that acquires streaming data for content from a data transmission device, and a storage portion that stores the acquired streaming data in a storage medium. The content playback device also has a playback portion that plays back the content based on the stored streaming data, a playback control portion that causes the playback portion to halt playback of the content in response to a halt playback request, and an acquisition control portion. The acquisition control portion causes the data acquisition portion to stop acquisition of the streaming data in response to the halt playback request, and causes the data acquisition portion to restart acquisition of the streaming data in response to a resume playback request. The storage portion stores, in the storage medium, the streaming data acquired in response to the resume playback request, starting from a position at which acquisition of the streaming data was stopped.

<CIT> discloses methods and systems for estimating user interest in media content. The method is performed on a server system having one or more processors. The server system receives a volume change event from a client system wherein the client system presents a media content item on a display in the vicinity of a user of the client system. The server system determines identification of a presented media content item currently being presented at the client system. The server system analyzes the received volume change event to determine a level of user interest in the determined media content item.

<CIT> discloses a method (<FIG>) to gauge a user's degree or level of interest in a portion of a media item. Time-stamped indicators over a period of time as a user interacts with a media items via a media device are monitored and analyzed. In particular, one or more time periods between the time-stamped indicators and the number of time-stamped indicators within a period of time are measured. A degree of user interest in the period of the media item based on at leaset one time period between the time-stamped indicators, the number of time-stamped indicators within the period of time and at least one type of user action is determined.

Provided herein are system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for interaction recognition of a television interaction device.

According to a first aspect of the present invention there is provided a method as specified in claim <NUM>. The method may optionally be as specified in claim <NUM> or <NUM>.

According to a second aspect of the present invention there is provided a system as specified in claim <NUM>. The system may optionally be as specified in claim <NUM>.

According to a third aspect of the present invention there is provided a tangible, non-transitory computer-readable device as specified in claim <NUM>.

Provided herein are system, apparatus, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for sensory detection of interactions during broadcast content.

<FIG> is a block diagram of a system for sensory detection of interactions during broadcast content, according to some embodiments. According to an embodiment, a system for sensory detections may comprise content server <NUM>, receiver <NUM>, display <NUM>, and interaction device <NUM>. In an embodiment, content server <NUM> may comprise a plurality of content <NUM> (comprising content A 106A, content B 106B, and content C 106C), interaction storage <NUM>, interaction engine <NUM>, encoder <NUM>, transceiver <NUM>, and memory <NUM>.

According to an embodiment, content server <NUM> may broadcast plurality of content <NUM>. In an embodiment, plurality of content <NUM> may comprise image sources, audio sources, television programs, movies, music, pictures, advertisements, streamable content, internet television, live content, or any combination thereof-to name a few examples. According to an embodiment, plurality of content <NUM> may be stored in content storage <NUM>. Content storage <NUM> may comprise random access memory (RAM), read-only memory (ROM), electronically erasable programmable random access memory (EEPROM), hard disk drive (HDD), solid state drive (SSD), or any combination thereof-to name a few examples.

In an embodiment, content server may broadcast plurality of content <NUM> via transceiver <NUM>. For example, content server <NUM> may broadcast content A 106A via transceiver <NUM>. According to an embodiment, transceiver <NUM> may broadcast plurality of content <NUM> as analog television signals, digital television signals, satellite television signals, analog radio signals, digital radio signals, satellite radio signals, internet video streaming (including dial-up, broadband, fiber, DSL, wireless, mobile network, and satellite internet), internet radio streaming, or any combination thereof-to name a few examples.

According to an embodiment, content server <NUM> may encode and compress content via encoder <NUM> before it is broadcast. For example, encoder <NUM> may encode content 106A before it is broadcast by transceiver <NUM>. In an embodiment, encoder <NUM> may encode plurality of content <NUM> by a plurality of encoding and codec systems, such as NTSC, PAL, SECAM, FM, AM, DAB+, DMB, DTMB, ATSC, ISDB, DVB-S, DVB-S2, DVB-C, DVB-T, DTT, MMDS, MVDS, Flash, MPEG-<NUM>, MPEG-<NUM>, WMV, VP6, Real Video, Real Audio, FLAC, ALAC, AMR, EVRC, are any combination thereof-to name a few examples.

In an embodiment, plurality of content <NUM> broadcast via transceiver <NUM> may be received by receiver <NUM>. Receiver <NUM> may comprise a radio, cable box, a television antenna, a television, a smart phone, a tablet, a streaming device, a gaming console, or any combination thereof-to name a few examples. According to an embodiment, receiver <NUM> may comprise transceiver <NUM> that may receive plurality of content <NUM> broadcast by transceiver <NUM>.

According to an embodiment, receiver <NUM> may display plurality of content <NUM> received from transceiver <NUM> on display <NUM>. Display <NUM> may comprise a television, a monitor, a phone screen, a tablet screen, a projector, or any combination thereof-to name a few examples. For example, content 106A may comprise a television program. Receiver <NUM> may receive content 106A from transceiver <NUM> and the television program on display <NUM>.

In an embodiment, interaction device <NUM> may comprise a device to send signals to receiver <NUM> to manipulate the content displayed on display <NUM>. For example, interaction device <NUM> may send signals to receiver <NUM> to control the speed, volume, channel, scene, screen, color, playback, or any combination thereof of the content displayed on display <NUM>-to name a few examples. As a few non-limiting examples, interaction device <NUM> may comprise a remote control, a smart phone touchscreen, a tablet touchscreen, or any combination thereof. According to an embodiment, interaction device <NUM> may send signals to receiver <NUM> via infrared communications, fiber communications, wired communications, WiFi, wireless communications, electromagnetic communications, or any combination thereof-to name a few examples.

According to another embodiment, interaction device <NUM> may comprise a plurality of sensors that detect interactions with interaction device <NUM>. The plurality of sensors that detect interactions with interaction device <NUM> may comprise interactive buttons, a microphone, a camera, an angle sensor, a motion sensor, or any combination thereof-to name a few examples. As a non-limiting example, interaction device <NUM> may comprise a remote control that detects interactions with buttons of the remote control and the angle of the remote control.

In another embodiment, interaction device <NUM> may comprise a plurality of sensors that monitor the environment and area surrounding interaction device <NUM>. The plurality of sensors that monitor the environment and area surrounding interaction device <NUM> may comprise a proximity sensor, a light sensor, a camera, a microphone, a proximity sensor, a heat sensor, a motion detector, or any combination thereof-to name a few examples. As a non-limiting example, interaction device <NUM> may comprise a camera that detects objects in front of interaction device <NUM>.

According to an embodiment, interaction device <NUM> may send sensor data from its sensors to receiver <NUM>. Interaction device <NUM> may send data from its sensors from its sensors to receiver <NUM> via infrared communications, fiber communications, wired communications, WiFi, wireless communications, electromagnetic communications, or any combination thereof-to name a few examples. For example, interaction <NUM> may comprise a camera and may send data from its camera to receiver <NUM>.

In an embodiment, receiver <NUM> may store sensor data from interaction device's <NUM> sensors in memory <NUM>. Memory <NUM> may comprise RAM, ROM, EEPROM, HDD, SSD, or any combination thereof-to name a few examples. For example, receiver <NUM> may receive camera sensor data from interaction device <NUM>. Once the data is received, receiver <NUM> may store the sensor data in memory <NUM>.

According to an embodiment, receiver <NUM> may send sensor data received from interaction device <NUM> to content server <NUM>. Receiver <NUM> may send the data via transceiver <NUM> to content server <NUM>. Transceiver <NUM> may transmit the sensor data to content server <NUM> via fiber communications, wired communications, WiFi, mobile network communications, wireless communications, or any combination thereof-to name a few.

In an embodiment, receiver <NUM> may send information regarding the content displayed on displayed on display <NUM>. The information regarding the content displayed may comprise the speed, volume, color, resolution, hue, or any combination thereof-to name a few examples-of the content displayed on display <NUM>.

In an embodiment, content server <NUM> may store received sensor data from receiver <NUM> in interaction storage <NUM>. Interaction storage <NUM> may comprise RAM, ROM, EEPROM, HDD, SSD, or any combination thereof-to name a few examples. According to an embodiment, content server <NUM> may classify sensor data received from receiver <NUM>. The sensor data may be classified by type, timestamp, set thresholds, a grading system, type of interaction device, type of receiver, or any combination thereof-to name a few examples. According to an embodiment, content server <NUM> may store received information regarding the content displayed in interaction storage <NUM>.

<FIG> is a block diagram of interaction device system <NUM>, according to some embodiments. According to an embodiment, interaction device system <NUM> may comprise display <NUM>, receiver <NUM>, and interaction device <NUM>. In an embodiment, receiver <NUM> may display received content <NUM> on display <NUM> as discussed in <FIG>.

In an embodiment, interaction device <NUM> may comprise a plurality of buttons <NUM> (comprising button A 208A, button B 208B, button C 208C, button D 208D, button E 208E, and button F 208F), microphone <NUM>, camera <NUM>, plurality of interaction and environmental sensors <NUM>, and microcontroller <NUM>.

According to an embodiment, buttons <NUM> may comprise interactive buttons. Interactive buttons may comprise push buttons, pressure sensitive buttons, physical buttons, virtual buttons, capacitive buttons, resistive buttons, or any combination thereof-to name a few examples. In an embodiment, buttons <NUM> may receive interaction inputs as interactions with buttons <NUM>. These interactions may comprise button presses, button taps, button holds, button touches, button swipes, or any combination thereof-to name a few examples. The received interaction input may comprise the type of interaction. For example, button A may receive a hold interaction, button b may receive a tap interaction, and button C may receive a tap interaction.

In an embodiment, each button of buttons <NUM> may perform a function to manipulate content <NUM> displayed on display <NUM>. These functions may comprise pause, skip, fast forward, rewind, skip back, stop, volume up, volume down, play, mute, channel up, channel down, or any combination thereof-to name a few examples. For example, button A 208A may pause content <NUM> when interacted with, button B 208B may play content <NUM> when interacted with, button C 208C may lower the volume of content <NUM> when interacted with, and button D 208D may raise the volume of content <NUM> when interacted with.

According to an embodiment, each button of buttons <NUM> may perform a function to navigate a user interface displayed on display <NUM>. These functions may comprise selecting text, selecting an image, selecting a menu, enter, scroll up, scroll down, scroll left, scroll right, or any combination thereof-to name a few examples. In an embodiment, the user interface may comprise a navigating the user interface may comprise selecting a movie scene, a song, a feature, a menu, start time for streamed content, or any combination thereof-to name a few.

In an embodiment, microphone <NUM> may comprise a plurality of microphone types. The plurality of microphone types may comprise a front microphone, a rear microphone, electret microphones, dynamic microphones, condenser microphones, directional microphones, bidirectional microphones, omnidirectional microphones, unidirectional microphones, or any combination thereof-to name a few examples. According to an embodiment, microphone <NUM> may detect the ambient noise around interaction device <NUM>. Ambient noise may comprise, for example, background noise, voices, music, or any combination thereof-to name a few examples. For example, microphone <NUM> may detect any voices around interaction device <NUM>.

According to an embodiment, camera <NUM> may comprise a plurality of camera types. The plurality of camera types may comprise forward-facing cameras, a rear-facing cameras, digital cameras, infrared cameras, CMOS cameras, CCD cameras, wireless cameras, or any combination thereof-to name a few examples. In an embodiment, camera <NUM> may detect whether an object is in view of camera <NUM>. The object detected may be a person, an obstacle (such as a furniture, doors, walls, ceilings, floors, or any combination thereof-to name some examples), display <NUM>, or any combination thereof-to name a few examples.

In an embodiment, camera <NUM> may detect an object through background subtraction, Gaussian subtraction, frame differencing, mean filter, or any combination-to name a few.

According to an embodiment, camera <NUM> may detect the movement of detected objects. Camera <NUM> may detect the movement of detected object by frame comparisons, trajectory estimations, vector analysis, or any combination thereof-to name a few examples. As a non-limiting example, camera <NUM> may detect a user's hand in front of camera <NUM>. Camera <NUM> may further detect that the user's hand is oscillating between two positions, such as, for example, in a "waving" gesture.

In an embodiment, sensors <NUM> may comprise interaction and detection sensors that detect information as detailed in <FIG>. For example, sensors <NUM> may comprise an angle sensor that detects the angle of interaction device <NUM> from the ground.

According to an embodiment interaction data from buttons <NUM>, microphone <NUM>, camera <NUM>, and plurality of interaction and environmental sensors <NUM> may be sent to microcontroller <NUM>. Microcontroller <NUM> may comprise a processor, a CPU, a microprocessor, an FPGA, a PIC, an AVR, or any combination thereof-to name a few examples. In an embodiment, when microcontroller <NUM> receives interaction data, a timestamp is created and associated with the data. The timestamp may comprise the time, date, content streamed, duration, or any combination thereof-to name a few examples. For example, microcontroller <NUM> may receive interaction data comprising a button press from buttons <NUM>. Microcontroller <NUM> may then create a timestamp comprising the time the button press was received, and associate that timestamp with the button press.

<FIG> is a flowchart illustrating a process for storing received interactions, according to some embodiments.

According to an embodiment, at <NUM>, content server <NUM> may receive interaction data and associated timestamps from receiver <NUM>. Interaction data may comprise interaction with buttons, data from a camera, data from a microphone, data from environmental sensor, data from interaction sensors, data from any other sensor, or any combination thereof. Interaction data and associated timestamps may be sent from interaction device <NUM> to receiver <NUM> as detailed in the discussion of <FIG>.

In an embodiment, at <NUM>, content server <NUM> may receive userID information from receiver <NUM>. UserID information may comprise a serial number, location, IP address, MAC address, registration data, or any combination thereof-to name a few examples-of receiver <NUM>.

According to an embodiment, at <NUM>, interaction engine <NUM> may assign interaction data and associated timestamps received from receiver <NUM> with the userID information of receiver <NUM> and store the interaction data, associated timestamps, and userID information in interaction storage <NUM>. Associating the data together may comprise storing the data in the same table, storing the data in the same column, creating a pointer, or any combination thereof-to name a few examples. For example, receiver <NUM> may receive interaction data comprising a number of button interactions from interaction device <NUM> along with an associated timestamp. Receiver <NUM> may send the interaction data and timestamp to content server <NUM> along with userID information comprising the IP address of receiver <NUM>. Interaction engine <NUM> may store the IP address of receiver <NUM> and the interaction data in the same table within interaction storage <NUM>.

According to an embodiment, at <NUM>, interaction <NUM> may determine whether or not content server <NUM> has received a stored interaction request. The stored interaction request may comprise a request for particular interaction data, particular userID information, timestamps, or any combination thereof -to name a few examples. In an embodiment, the stored interaction request may be received by content server <NUM> as a query, keyboard strokes, mouse clicks, mouse drags, or any combination thereof-to name a few examples. If content server <NUM> has received a stored interaction request, then <NUM> is performed, otherwise <NUM> is repeated.

In an embodiment, at <NUM>, interaction engine <NUM> determines whether if the stored interaction request matches any of the interaction data stored in interaction storage <NUM>. For example, content server <NUM> may receive a stored interaction request comprising data requesting interaction data associated with a specific timestamp. Interaction engine <NUM> may then determine whether any interaction data associated with the specific timestamp is stored in interaction storage <NUM>. If there is no data stored in interaction storage <NUM>, interaction engine <NUM> may repeat <NUM>, otherwise interaction engine <NUM> may perform <NUM>.

According to an embodiment, at <NUM>, interaction <NUM> may return a result. The result may comprise the data stored in interaction storage <NUM> that matches the stored interaction request.

<FIG> is a flowchart illustrating a process for modifying broadcast content based upon received interactions, according to some embodiments.

At <NUM>, content server <NUM> may receive interaction data and associated timestamps as well as information regarding the content displayed on display <NUM> from receiver <NUM>. Interaction data may comprise interaction with buttons, data from a camera, data from a microphone, data from environmental sensor, data from interaction sensors, or any combination thereof. Interaction data and associated timestamps may be sent from interaction device <NUM> to receiver <NUM> as detailed in the discussion of <FIG>.

At <NUM>, interaction engine <NUM> may determine if the received interaction data matches any interaction data from a stored list of modifying interactions. The stored list of modifying interactions may comprise tables, columns, lists, vectors, rows, or any combination thereof-to name a few examples-of interactions associated with content <NUM> being broadcast by content server <NUM>. For example, the stored list of modifications may comprise button interactions, button interaction rhythms, button functions, user interface navigations, songs, music, gestures, facial expressions, background objects, environmental sensor data, interaction sensor data, background data, or any combination thereof-to name a few examples-associated with content <NUM> being broadcast. Interaction engine <NUM> may determine that received interaction data matches a modifying interaction from the stored list of modifying interactions, in which case the system moves on to <NUM>, otherwise <NUM> is repeated. For example, content server <NUM> may receive interaction data comprising an interaction with a button to perform the volume down function while content A 106A is being broadcast. Interaction engine <NUM> may then compare the received interaction data to the stored list of modifying interactions associated with content A 106A, which may comprise an interaction with a button to perform the volume down function. When interaction engine <NUM> matches the received interaction data to the stored list of modifying interactions associated with content A 106A, the system may perform <NUM>.

According to an embodiment, content server <NUM> may receive interaction data that comprises button interactions and timestamps. For example, content server <NUM> may receive interaction data comprising button A 208A pressed at timestamp A, button B 208B pressed at timestamp B, button A 208A pressed at timestamp C, and button D 208D pressed at timestamp D. In an embodiment, interaction engine <NUM> may determine a rhythm the buttons were interacted with through use of the associated timestamps. Interaction <NUM> may further match the determined rhythm to a rhythm from the stored list of modifying interactions.

According to an embodiment, content server <NUM> may receive interaction data that comprises timestamps and button interactions comprising user interface navigations. For example, content server <NUM> may receive interaction data comprising an interaction with button A 208A, comprising selecting a scene, at timestamp A, an interaction with button B 208B comprising starting a scene, at timestamp B, and an interaction with button A <NUM>, comprising selecting the scene, at timestamp C. In an embodiment, interaction engine <NUM> may determine a history of scene selection through use of the associated timestamps. Interaction engine <NUM> may further match the determined history of scene selection to a history of scene selections from the stored list of modifying interactions.

In an embodiment, content server <NUM> may receive interaction data that comprises ambient noise received by microphone <NUM> and a timestamp. For example, content server <NUM> may receive interaction data comprising ambient noise received by microphone <NUM> at timestamp A. According to an embodiment, interaction engine <NUM> may determine whether the ambient noise matches a specific song or piece of music. Interaction engine <NUM> may make this determination by comparing the ambient noise to a music database, using a music recognition software, sending the ambient noise to a music recognition engine, comparing the ambient noise to stored sound files, or any combination thereof-to name a few examples. In an embodiment, interaction engine <NUM> may further match the determined song or music from the ambient noise to a song or piece of music from the stored list of modifying interactions.

According to another embodiment, interaction engine <NUM> may determine whether the ambient noise matches a specific voice line or phrase. Interaction engine <NUM> may make this determination by comparing the ambient noise to data from a database, sending the ambient noise to a speech recognition engine, using a speech recognition software, using a speech-to-text software, or any combination thereof-to name a few examples. In an embodiment, interaction engine <NUM> may further match the determined voice line or phrase to a voice line or phrase from the stored list of modifying interactions.

According to an embodiment, content server <NUM> may receive interaction data that comprises an image or video received from camera <NUM> and a timestamp. For example, content server <NUM> may receive interaction data comprising an image from camera <NUM>. In an embodiment, interaction engine <NUM> may determine whether the object in an image comprises a face. Interaction engine <NUM> may make this determination through the use of background filtering, facial recognition software, a facial recognition engine, principal component analysis, linear discriminate analysis, elastic bunch graph matching, or any combination thereof-to name a few examples. In an embodiment, interaction engine <NUM> may further match the presence of a face in the image to the requirement of a face in front of an interaction device from the stored list of modifying interactions.

In another embodiment, interaction engine <NUM> may determine facial expressions of a face present in a received image or video from camera <NUM>. Interaction engine <NUM> may make this determination through the use of background filtering, facial recognition software, a facial recognition engine, principal component analysis, linear discriminate analysis, elastic bunch graph matching, or any combination thereof-to name a few examples. In an embodiment, interaction engine <NUM> may further match the determined facial expression from the received image or video to facial expressions from the stored list of modifying interactions.

According to an embodiment, content server <NUM> may receive interaction data that comprises an orientation of the interaction device and a timestamp. For example, content server <NUM> may receive interaction data comprising a first angle the interaction device is oriented at timestamp A, a second angle at timestamp B, and a third angle at timestamp C. In an embodiment, interaction engine <NUM> may determine a gesture being made with the interaction device through the angles and associated timestamps. For example, based on the timing of the timestamps and the change in the angles, interaction engine <NUM> may determine that a waving gesture is being made with the interaction device. Interaction <NUM> may further match the determined gesture to a gesture from the stored list of modifying interactions.

At <NUM>, interaction engine <NUM> may modify the content streamed by content sever <NUM> based upon which interaction from the stored list of modifying interactions was matched. Modifying the content may comprise rewinding the content, speeding up the content, changing the resolution of the content, changing the volume of the content, stopping the content, pausing the content, playing the content, skipping to another portion of the content, selecting new content to play, or any combination thereof-to name a few examples. In an embodiment, each interaction from the stored list of modifying interactions may comprise an associated content modification.

For example, content server <NUM> may receive interaction data that comprises button interactions and timestamps. From these button interactions and timestamps, interaction engine <NUM> may determine a rhythm that matches a rhythm from the stored list of modifying interactions. The matched rhythm from the stored list of modifying interactions may be associated with a content modification comprising stopping the content.

In an embodiment, modifying the content may comprise broadcasting new content. For example, content server <NUM> may be broadcasting content A 106A to receiver <NUM> when content server <NUM> receives interaction data comprising ambient noise received by microphone <NUM>. Interaction engine <NUM> may then match the received ambient noise to a song from the stored list of modifying interactions. Further, the song from the stored list of modifying interactions may be associated with broadcasting new content. Based on this match, interaction engine <NUM> may end the broadcast of content A 106A to receiver <NUM>, and begin broadcasting content B 106B to receiver <NUM>.

<FIG> is a flowchart illustrating a process for modifying broadcast content based upon received content data, according to some embodiments.

At <NUM>, content server <NUM> may receive content data regarding the content displayed on display <NUM> from receiver <NUM>. The content data may comprise the speed, volume, color, resolution, hue, or any combination thereof-to name a few examples-of the content displayed on display <NUM>. Content server <NUM> may receive the content data from receiver <NUM> as detailed in the discussion of <FIG>.

At <NUM>, interaction engine <NUM> may compare the received content data to thresholds from a stored list of content thresholds. The stored list of content thresholds may comprise upper and lower threshold value regarding volume, resolution, color, hue, or any combination thereof-to name a few examples. If values from the content data are greater than the respective upper threshold or lower than the respective lower threshold, then the system moves on to <NUM>. Otherwise, <NUM> is repeated.

As a non-limiting example, content server <NUM> may receive content data comprising the volume, speed, and resolution of content displayed on display <NUM>. Interaction engine <NUM> may then compare the volume, speed, and resolution of the content displayed to the upper and lower thresholds for volume, speed, and resolution from the stored list of content thresholds. Interaction engine <NUM> may then determine that the volume of the content displayed falls below the lower threshold for volume from the stored list of content thresholds.

At <NUM>, interaction engine <NUM> may modify the content streamed by content sever <NUM> based upon which threshold from the stored list of content thresholds was exceeded. Modifying the content may comprise rewinding the content, speeding up the content, changing the resolution of the content, changing the volume of the content, stopping the content, pausing the content, playing the content, or any combination thereof-to name a few examples. In an embodiment, each threshold from the stored list of content thresholds may comprise an associated content modification.

For example, content server <NUM> may receive content data that comprises a volume. Interaction engine <NUM> may then that the received volume falls below the lower threshold for volume from the stored list of content thresholds. The lower threshold for volume from the stored list content thresholds may be associated with a content modification comprising pausing the content.

<FIG> is a flowchart illustrating a process for detecting an interaction class, according to some embodiments.

At <NUM>, interaction engine <NUM> may match the received interaction data to the stored list of modifying interactions as detailed in <FIG>. If the received interaction data matches interactions from the list of modifying interactions then the system moves to <NUM>, otherwise <NUM> is repeated. At <NUM>, interaction engine <NUM> may classify the received interaction data. Classifying the received interaction data may comprise assigning the interaction data a type, assigning a grade to the interaction, sorting the interaction into content played, or any combination thereof-to name a few examples.

In an embodiment, interaction engine <NUM> may assign the received interaction data a type. The type may comprise a button interaction, a song, an object in an image, a gesture, a phrase, a functionality of a button, or any combination thereof-to name a few examples. For example, the received interaction data may comprise a plurality of button interactions. Interaction engine <NUM> may then assign the type "button interactions" to the received interaction data.

According to an embodiment, a grade may be assigned to the received interaction data based on the matched interaction from the stored list of modifying interactions. In an embodiment, each interaction from the stored list of modifiable may comprise a grade. The grade may be based on the frequency the interaction is received by content server <NUM>, the type of interaction, the complexity of the interaction, or any combination thereof-to name a few examples.

In an embodiment, the received interaction data may be assigned a class based on the content being broadcast while the interaction was received. For example, the received interaction data may be received while content A 106A is being broadcast and comprise a plurality of button interactions. Interaction engine <NUM> may then assign the a class to the plurality of button interactions based on the data being received while content A 106A was being broadcast.

At <NUM>, interaction engine <NUM> determines if the class assigned to the received interaction data matches a class from the stored list of modifying classes. The stored list of modifying classes may comprise types, grades, content groups, or any combination thereof, with each comprising a respective content modification. The respective content modification may comprise rewinding, speeding up, changing the resolution, changing the volume stopping, pausing, playing, or any combination thereof-to name a few examples-of the content streamed. If the assigned class matches a class from the stored list of modifying classes then the system moves to <NUM>, otherwise <NUM> is repeated.

At <NUM>, the received interaction data and its assigned class is stored in interaction storage <NUM>. At <NUM>, the content streamed by content server <NUM> is modified according to the respective content modification of the assigned class. For example, content server <NUM> may receive interaction data that is assigned a class. This class may comprise a content modification of stopping the content. Interaction engine <NUM> may then stop the content being broadcast by content server <NUM>.

<FIG> is an exemplary case study of an interaction recognition of a television content interaction device, according to an embodiment.

According to an embodiment, content server <NUM> may comprise a content server such as, for example, content server <NUM>, and may stream content to interaction device <NUM> over the internet. The content streamed to interaction device <NUM> may comprise a video, a television show, a movie, an advertisement, a song, a live stream, an audio file, or any combination thereof-to name a few examples. In an embodiment, interaction device <NUM> may comprise a smart phone comprising receiver <NUM>, display <NUM>, and a plurality of buttons <NUM> (comprising button 710A, button 710B, button 710C, and button 710D.

In an embodiment, receiver <NUM> may comprise, for example, receiver <NUM> and display <NUM> may comprise, for example, display <NUM>. According to an embodiment, buttons <NUM> may comprise interactive buttons, resistive buttons, capacitive buttons, physical buttons, virtual buttons, or any combination thereof-to name a few examples.

According to an embodiment, the content streamed to interaction device <NUM> may comprise a prompt for a button rhythm. The button rhythm may comprise a desired order and timing in which buttons <NUM> are to be pressed. For example, the content streamed to interaction device <NUM> may comprise a prompt to press button 710B, two seconds later to press button 710A, and one second later to press button 710C. In an embodiment, a user may interact with buttons <NUM> according to the button prompt. For example, a user may press button 710B, two seconds later press button 710A, and one second later press button 710C. According to an embodiment, when each of these buttons are pressed by the user, interaction data comprising the buttons that were interacted with and the timestamps of each interaction are sent from interaction device <NUM>, via receiver <NUM>, to content server <NUM> as detailed in the descriptions of <FIG> and <FIG>.

In an embodiment, once the interaction data is received by content server <NUM>, content server <NUM> may determine a button rhythm from the interaction data and timestamps as detailed in <FIG>. According to an embodiment, content server <NUM> may then compare the button rhythm to a stored list of modifying interactions and may further modify the content streamed to interaction device <NUM> according to the comparison, as detailed in <FIG>.

For example, content server <NUM> may stream, over the internet, a television show starring a specific actor to interaction device <NUM>, a smartphone. The content streamed to interaction device <NUM> may further comprise a button rhythm prompt, that, when followed, will stream a different television show starring the same actor. The user may then interact with buttons <NUM> according to the button prompt. Interaction device <NUM> may then send data comprising the button interactions with buttons <NUM> to content server <NUM>. Content server <NUM> may then compare the data comprising the buttons interactions to a stored list of modifying interactions and change the stream to a different television show starring the same actor.

According to an embodiment, content server <NUM> may comprise a content server such as, for example, content server <NUM>, and may stream content to interaction device <NUM> over the internet. The content streamed to interaction device <NUM> may comprise a video, a television show, a movie, an advertisement, a song, a live stream, an audio file, or any combination thereof-to name a few examples. In an embodiment, interaction device <NUM> may comprise a gaming console comprising receiver <NUM> and camera <NUM>. According to an embodiment, interaction device <NUM> may displayed the streamed content on display <NUM>.

In an embodiment, receiver <NUM> may comprise, for example, receiver <NUM> and display <NUM> may comprise, for example, display <NUM>. According to an embodiment, camera <NUM> may comprise a forward-facing camera, a rear-facing camera, digital camera, infrared camera, CMOS camera, CCD camera, wireless camera, or any combination thereof-to name a few examples.

According to an embodiment, camera <NUM> may capture an image while content is being streamed from content server <NUM>. Camera <NUM> may further, through background subtraction, Gaussian subtraction, frame differencing, mean filter, or any combination thereof, detect an object within the image captured, such as, for example, the face of a user. In an embodiment, interaction device <NUM> may send the captured image with a detected object to content server <NUM>. Content Server <NUM>, using background filtering, facial recognition software, a facial recognition engine, principal component analysis, linear discriminate analysis, elastic bunch graph matching, or any combination thereof, may determine a facial expression from the captured image.

In an embodiment, content server <NUM> may compare the determined facial expression to a stored list of modifying interactions and may further modify the content streamed to interaction device <NUM> according to the comparison, as detailed in <FIG>.

For example, content server <NUM> may stream, over the internet, a movie with a Motion Picture Association of America (MPAA) R rating to interaction device <NUM>, a gaming console. While the content is being streamed, camera <NUM> of interaction device <NUM> may capture an image which comprises a user making a facial expression indicating displeasure. Interaction device <NUM> may then send the captured image to content server <NUM>, and content server <NUM> may determine, from the captured image, that a user is making a facial expression of displeasure. Content server <NUM> may then compare the determined facial expression to a stored list of modifying interactions and change the stream to a different movie with a lower MPAA rating.

Various embodiments can be implemented, for example, using one or more well-known computer systems, such as computer system <NUM> shown in <FIG>. For example, computer system <NUM> can compare received interaction data to a stored list of modifying interactions. Computer system <NUM> can further modify content that is broadcast based on received interaction data. Computer system <NUM> can be any computer capable of performing the functions described herein.

One or more processors <NUM> may each be a graphics processing unit (GPU). In an embodiment, a GPU is a processor that is a specialized electronic circuit designed to process mathematically intensive applications. The GPU may have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, videos, etc..

Computer system <NUM> also includes a main or primary memory <NUM>, such as random access memory (RAM). Main memory <NUM> may include one or more levels of cache. Main memory <NUM> has stored therein control logic (i.e., computer software) and/or data.

According to an exemplary embodiment, secondary memory <NUM> may include other means, instrumentalities or other approaches for allowing computer programs and/or other instructions and/or data to be accessed by computer system <NUM>. Such means, instrumentalities or other approaches may include, for example, a removable storage unit <NUM> and an interface <NUM>. Examples of the removable storage unit <NUM> and the interface <NUM> may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and/or any other removable storage unit and associated interface.

In an embodiment, a tangible apparatus or article of manufacture comprising a tangible computer useable or readable medium having control logic (software) stored thereon is also referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system <NUM>, main memory <NUM>, secondary memory <NUM>, and removable storage units <NUM> and <NUM>, as well as tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system <NUM>), causes such data processing devices to operate as described herein.

Based on the teachings contained in this disclosure, it will be apparent to persons skilled in the relevant art(s) how to make and use embodiments of the invention using data processing devices, computer systems and/or computer architectures other than that shown in <FIG>. In particular, embodiments may operate with software, hardware, and/or operating system implementations other than those described herein.

It is to be appreciated that the Detailed Description section, and not the Summary and Abstract sections (if any), is intended to be used to interpret the claims. The Summary and Abstract sections (if any) may set forth one or more but not all exemplary embodiments of the invention as contemplated by the inventor(s), and thus, are not intended to limit the invention or the appended claims in any way.

While the invention has been described herein with reference to exemplary embodiments for exemplary fields and applications, it should be understood that the invention is not limited thereto. Other embodiments and modifications thereto are possible, and are within the scope of the invention. For example, and without limiting the generality of this paragraph, embodiments are not limited to the software, hardware, firmware, and/or entities illustrated in the figures and/or described herein. Further, embodiments (whether or not explicitly described herein) have significant utility to fields and applications beyond the examples described herein.

Also, alternative embodiments may perform functional blocks, blocks, operations, methods, etc. using orderings different than those described herein.

References herein to "one embodiment," "an embodiment," "an example embodiment," or similar phrases, indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic.

Claim 1:
A method, comprising:
receiving interaction data at a content server (<NUM>, <NUM>, <NUM>) from a content receiver (<NUM>, <NUM>, <NUM>, <NUM>) regarding a content (<NUM>) displayed on a display (<NUM>) of the content receiver (<NUM>, <NUM>, <NUM>,<NUM>), wherein the content server (<NUM>, <NUM>, <NUM>) comprises at least one processor;
determining, by the at least one processor, a modifying interaction when the interaction data matches any interaction data from a stored list of modifying interactions associated with the content, wherein the interaction data comprises a plurality of button interactions with a plurality of different buttons (<NUM>, <NUM>) and a plurality of respective timestamps, wherein the determining, by the at least one processor, the interaction from the interaction data further comprises determining, by the at least one processor, a rhythm from the plurality of button interactions and the plurality of respective timestamps, and wherein the rhythm comprises an order and timing in which the plurality of different buttons are pressed;
storing, by the at least one processor, the determined modifying interaction;
selecting, by the at least one processor, a content modification based upon the determined modifying interaction; and
modifying, by the at least one processor, content streamed from the content server (<NUM>, <NUM>, <NUM>) to the content receiver according to the selected content modification, wherein modifying the content comprises rewinding the content, speeding up the content, changing the resolution of the content, changing the volume of the content, stopping the content, pausing the content, playing the content, skipping to another portion of the content, selecting new content to play, or any combination thereof.