Patent ID: 12250421

DETAILED DESCRIPTION

The accompanying drawings, which form a part hereof, show examples of the disclosure. It is to be understood that the examples shown in the drawings and/or discussed herein are non-exclusive and that there are other examples of how the disclosure may be practiced.

FIG.1shows an example communication network100in which features described herein may be implemented. The communication network100may comprise one or more information distribution networks of any type, such as, without limitation, a telephone network, a wireless network (e.g., an LTE network, a 5G network, a WiFi IEEE 802.11 network, a WiMAX network, a satellite network, and/or any other network for wireless communication), an optical fiber network, a coaxial cable network, and/or a hybrid fiber/coax distribution network. The communication network100may use a series of interconnected communication links101(e.g., coaxial cables, optical fibers, wireless links, etc.) to connect multiple premises102(e.g., businesses, homes, consumer dwellings, train stations, airports, etc.) to a local office103(e.g., a headend). The local office103may send downstream information signals and receive upstream information signals via the communication links101. Each of the premises102may comprise devices, described below, to receive, send, and/or otherwise process those signals and information contained therein.

The communication links101may originate from the local office103and may comprise components not shown, such as splitters, filters, amplifiers, etc., to help convey signals clearly. The communication links101may be coupled to one or more wireless access points127configured to communicate with one or more mobile devices125via one or more wireless networks. The mobile devices125may comprise smart phones, tablets or laptop computers with wireless transceivers, tablets or laptop computers communicatively coupled to other devices with wireless transceivers, and/or any other type of device configured to communicate via a wireless network.

The local office103may comprise an interface104, such as a termination system (TS). The interface104may comprise a cable modem termination system (CMTS) and/or other computing device(s) configured to send information downstream to, and to receive information upstream from, devices communicating with the local office103via the communications links101. The interface104may be configured to manage communications among those devices, to manage communications between those devices and backend devices such as servers105-107, and/or to manage communications between those devices and one or more external networks109. The local office103may comprise one or more network interfaces108that comprise circuitry needed to communicate via the external networks109. The external networks109may comprise networks of Internet devices, telephone networks, wireless networks, wireless networks, fiber optic networks, and/or any other desired network. The local office103may also or alternatively communicate with the mobile devices125via the interface108and one or more of the external networks109, for example, via one or more of the wireless access points127.

The push notification server105may be configured to generate push notifications to deliver information to devices in the premises102and/or to the mobile devices125. The content server106may be configured to provide content to devices in the premises102and/or to the mobile devices125. This content may comprise, for example, video, audio, text, web pages, images, files, etc. The content server106(or, alternatively, an authentication server) may comprise software to validate user identities and entitlements, to locate and retrieve requested content, and/or to initiate delivery (e.g., streaming) of the content. The application server107may be configured to offer any desired service. For example, an application server may be responsible for collecting, and generating a download of, information for electronic program guide listings. Another application server may be responsible for monitoring user viewing habits and collecting information from that monitoring for use in selecting advertisements. Yet another application server may be responsible for formatting and inserting advertisements in a video stream being transmitted to devices in the premises102and/or to the mobile devices125. The local office103may comprise additional servers, additional push, content, and/or application servers, and/or other types of servers. Although shown separately, the push server105, the content server106, the application server107, and/or other server(s) may be combined. The servers105,106,107, and/or other servers, may be computing devices and may comprise memory storing data and also storing computer executable instructions that, when executed by one or more processors, cause the server(s) to perform steps described herein.

An example premises102amay comprise an interface120. The interface120may comprise circuitry used to communicate via the communication links101. The interface120may comprise a modem110, which may comprise transmitters and receivers used to communicate via the communication links101with the local office103. The modem110may comprise, for example, a coaxial cable modem (for coaxial cable lines of the communication links101), a fiber interface node (for fiber optic lines of the communication links101), twisted-pair telephone modem, a wireless transceiver, and/or any other desired modem device. One modem is shown inFIG.1, but a plurality of modems operating in parallel may be implemented within the interface120. The interface120may comprise a gateway111. The modem110may be connected to, or be a part of, the gateway111. The gateway111may be a computing device that communicates with the modem(s)110to allow one or more other devices in the premises102ato communicate with the local office103and/or with other devices beyond the local office103(e.g., via the local office103and the external network(s)109). The gateway111may comprise a set-top box (STB), digital video recorder (DVR), a digital transport adapter (DTA), a computer server, and/or any other desired computing device.

The gateway111may also comprise one or more local network interfaces to communicate, via one or more local networks, with devices in the premises102a.Such devices may comprise, for example, display devices112(e.g., televisions), STBs or DVRs113, personal computers114, laptop computers115, wireless devices116(e.g., wireless routers, wireless laptops, notebooks, tablets and netbooks, cordless phones (e.g., Digital Enhanced Cordless Telephone—DECT phones), mobile phones, mobile televisions, personal digital assistants (PDA)), landline phones117(e.g., Voice over Internet Protocol—VoIP phones), and any other desired devices. Example types of local networks comprise Multimedia Over Coax Alliance (MoCA) networks, Ethernet networks, networks communicating via Universal Serial Bus (USB) interfaces, wireless networks (e.g., IEEE 802.11, IEEE 802.15, Bluetooth), networks communicating via in-premises power lines, and others. The lines connecting the interface120with the other devices in the premises102amay represent wired or wireless connections, as may be appropriate for the type of local network used. One or more of the devices at the premises102amay be configured to provide wireless communications channels (e.g., IEEE 802.11 channels) to communicate with one or more of the mobile devices125, which may be on- or off-premises.

The mobile devices125, one or more of the devices in the premises102a,and/or other devices may receive, store, output, and/or otherwise use assets. An asset may comprise a video, a game, one or more images, software, audio, text, webpage(s), and/or other content.

FIG.2shows hardware elements of a computing device200that may be used to implement any of the computing devices shown inFIG.1(e.g., the mobile devices125, any of the devices shown in the premises102a,any of the devices shown in the local office103, any of the wireless access points127, any devices with the external network109) and any other computing devices discussed herein (e.g., home entertainment controllers, network managers, audiovisual data processors, primary user devices, secondary user devices). The computing device200may comprise one or more processors201, which may execute instructions of a computer program to perform any of the functions described herein. The instructions may be stored in a non-rewritable memory202such as a read-only memory (ROM), a rewritable memory203such as random access memory (RAM) and/or flash memory, removable media204(e.g., a USB drive, a compact disk (CD), a digital versatile disk (DVD)), and/or in any other type of computer-readable medium or memory. Instructions may also be stored in an attached (or internal) hard drive205or other types of storage media. The computing device200may comprise one or more output devices, such as a display device206(e.g., an external television and/or other external or internal display device) and a speaker214, and may comprise one or more output device controllers207, such as a video processor or a controller for an infra-red or BLUETOOTH transceiver. One or more user input devices208may comprise a remote control, a keyboard, a mouse, a touch screen (which may be integrated with the display device206), a microphone, etc. The computing device200may also comprise one or more network interfaces, such as a network input/output (I/O) interface210(e.g., a network card) to communicate with an external network209. The network I/O interface210may be a wired interface (e.g., electrical, RF (via coax), optical (via fiber)), a wireless interface, or a combination of the two. The network I/O interface210may comprise a modem configured to communicate via the external network209. The external network209may comprise the communication links101discussed above, the external network109, an in-home network, a network provider's wireless, coaxial, fiber, or hybrid fiber/coaxial distribution system (e.g., a DOCSIS network), or any other desired network. The computing device200may comprise a location-detecting device, such as a global positioning system (GPS) microprocessor211, which may be configured to receive and process global positioning signals and determine, with possible assistance from an external server and antenna, a geographic position of the computing device200.

The computing device200may comprise and/or communicate with a lighting device220, as shown inFIG.2. The lighting device220may comprise one or more Light-Emitting Diodes (LEDs), one or more Liquid-Crystal Displays (LCDs), and/or one or more other types of lighting devices. The computing device200, with the lighting device220, may comprise a smart light bulb, a smart lighting system, an automotive lighting system, and/or another type of lighting system. The lighting device220may be controlled by the one or more processors201or one or more other components in the computing device200. For example, the lighting device220may receive, from the one or more processors201, digital values representing color codes. The digital values may be obtained (e.g., by the at least one processor201) from or based on manifest files. The lighting device220may output, based on the digital values, colored light and/or visual effects associated with content. The lighting device220may comprise various digital circuitry, such as amplifiers, power source/converters, processors, network interfaces, etc., to communicate with the one or more processors201or one or more other components of the computing device200.

AlthoughFIG.2shows an example hardware configuration, one or more of the elements of the computing device200may be implemented as software or a combination of hardware and software. Modifications may be made to add, remove, combine, divide, etc. components of the computing device200. Additionally, the elements shown inFIG.2may be implemented using basic computing devices and components that have been configured to perform operations such as are described herein. For example, a memory of the computing device200may store computer-executable instructions that, when executed by the processor201and/or one or more other processors of the computing device200, cause the computing device200to perform one, some, or all of the operations described herein. Such memory and processor(s) may also or alternatively be implemented through one or more Integrated Circuits (ICs). An IC may be, for example, a microprocessor that accesses programming instructions or other data stored in a ROM and/or hardwired into the IC. For example, an IC may comprise an Application Specific Integrated Circuit (ASIC) having gates and/or other logic dedicated to the calculations and other operations described herein. An IC may perform some operations based on execution of programming instructions read from ROM or RAM, with other operations hardwired into gates or other logic. Further, an IC may be configured to output image data to a display buffer.

FIG.3shows an example configuration of a home entertainment system300. An audiovisual data processor302in the system300may receive a segment of content from a content provider301. The content segment may comprise, for example, text materials, photographs or other still images, audio files, video files, Graphic Interchange Format files (GIFs) and/or other types of animation, interactive applications (and/or portions thereof), and/or other types of content. The content segment may also or alternatively comprise discrete portions of a content item such as a video on demand (VOD) content item, a linear video content item, a live content item, a web streaming content item, a video conferencing content item, etc. The content segment may be displayed or otherwise presented to users via one or more primary user devices309comprising, for example, the display device112(e.g., a smart TV, a smart display, an HDTV, etc.), the personal computer114, the laptop computer115, the mobile devices125, etc. One or more secondary user devices310may comprise a variety of lighting or display devices (e.g., smart light bulbs, internet-capable lighting devices, Light-emitting Diode (LED) display panels, etc.) and/or other IoT products with lighting and/or other display features. If a user command is received by the system300to turn on an output synchronization feature of the system300, the user devices may be controlled to output the content segment (e.g., by causing display of video and/or audio components of the content segment via the primary user device(s)309) in synchronization with output of supplemental lighting and/or other lighting and/or visual effects (e.g., by controlling lighting element(s) of the secondary user device(s)310).

Each component of the system300and other devices (e.g., one or more computing devices of the content provider301, the primary user device(s)309and the secondary user device(s)310) may comprise a local clock. These local clocks may be synchronized via a centralized clock for a home entertainment cloud (e.g., the home entertainment system300). These local clocks may be synchronized using, for example, one or more of Network Time Protocol (NTP), Simple Network Time Protocol (SNTP) or Precision Time Protocol (PTP). NTP may be designed to synchronize the local clocks across the Internet or Local Area Networks (LANs) and provide accurate and synchronized time for each component of the system300and other devices. For example, all network components or devices of a home entertainment cloud may connect to a main server (e.g., a home entertainment controller305). This main server may call to another server, such as a public time server (e.g., in external network109or external network209) to keep track of Coordinated Universal Time (UTC) via the Internet. Through mechanisms such as NTP, SNTP or PTP, each component of the system300or other devices may maintain accurate current time, calibrated to fractions of a second. Such precision may enable network components, devices and/or applications to communicate with each other effectively. For example, the system300and the user devices (e.g., the primary user device(s)309and the secondary user device(s)310) may keep communicating with one another to maintain accurate current time via the Internet and/or one or more LANs. The synchronized local clocks in the system300, the primary user device(s)309, and/or the secondary user device(s)310may facilitate adjusting for differences of times for receipt and/or processing of content segments and manifest files.

The user devices may output audio and/or video components of a content segment and may also output other visual effects. The output of the visual effects may be synchronized with the output of the content segment based on one or more manifest files. A manifest file may comprise feature data that indicates one or more features associated with the content segment, as well as time stamps corresponding to those features (e.g., time stamps indicating a time in the content segment during which a feature occurs). The time stamps may be determined and/or adjusted based on local clocks associated with the system300, the primary user device(s)309and/or the secondary user device(s)310. The time stamps may be used to keep track of a sequence of events associated with the content segment, the manifest file, the primary user device(s)309, the secondary user device(s)310, etc. The time stamps may be used to generate and/or process a file (e.g., manifest file), a log, or a notification that records an event in which data is added, removed, modified or transmitted. The time stamps may indicate current time of a recorded event. The time stamps may be used to record current time in relation to a starting point (e.g., if a user command to synchronize output by the user devices is received) or an end point (e.g., if a user command to stop synchronizing the output by the user devices is received). The controller305, a network manager306and/or a gateway308may be configured to adjust the time stamps to offset based on an expected network delay caused by transmission and/or processing of various network communications, such as, for example, sending or receiving digital signals, user identification, user commands, manifest files, etc.

The audiovisual data processor302may process a content segment, received from the content provider301, to determine audio and video components of the content segment. A plurality of time stamps may be associated with those audio and/or video components. The audiovisual data processor302may analyze the audio component to determine audio features (e.g., beats) using beat detection algorithms, computer software or computer hardware (to be described below). Based on the audio beats, the secondary user device(s)310may change, for example, color or other visual features associated with the content segment at a plurality of times of audio beats. The audiovisual data processor302may extract the video component comprising a plurality of frames comprising color or visual features (Red-Green-Blue (RGB), Hue-Saturation-Brightness (HSB), Cyan-Magenta-Yellow-Key (CMYK) colors, etc.) along with a plurality of time stamps. One or more frames associated with the times of audio beats may be sampled by the audiovisual data processor302. The sampled frames may be analyzed to determine video features associated with those frames (e.g., average color). A manifest file may be generated and may comprise data indicating one or more of those video features and corresponding time stamps. The manifest file, which may be modified by the gateway308prior to transmission to the secondary user device(s)310, may be used by the secondary user device(s)310to determine when and/or how to provide output of visual effects based on the visual features (e.g., by activating and/or deactivating a light source, by controlling a color of the light source, etc.) associated with the content segment, for example, at the times of audio beats. The time stamps may be adjusted by a network administrator or network carriers.

The audiovisual data processor302, using one or more beat detection algorithms implemented using computer software (e.g., TarsosDSP, a Java library for audio processing), firmware and/or hardware, may be used to determine, for the audio component of a content segment, primary frequencies and/or audio beats. TarsosDSP and/or other audio analysis software, firmware, and/or hardware may provide an interface to an audio beat processing algorithm implemented in Java and without any other external dependencies. The audio component may comprise a plurality of audio files (e.g., waveform audio (.wav) files, MPEG-1 audio layer 3 (.mp3) files, Windows Media audio (.wma) files, etc.). Based on TarsosDSP (and/or other software, firmware, and/or hardware), audio beats (e.g., impulses of sound occurring at regular or irregular intervals) may be detected from the audio component by detecting impulses of sound in the audio component and determining time stamps at which these impulses appear during playback of the content segment. Indications of the audio beats may be included in a manifest file to control the secondary user device(s)310, which may turn a lighting device on or off (or otherwise control a lighting device) at occurrences of the audio beats associated with the audio component. Also, indications of other distinct audio points, such as acoustical strokes, accents, and/or tempos in the audio component, may be included in the manifest file. For example, the secondary user device(s)310may pulse a lighting device to fast tempo or slow tempo music associated with the content segment. Based on the manifest file comprising the time stamps, the secondary user device(s)310may pulse colored light in time with distinct audio points in the audio component. The audio beats (or other distinct audio points) may occur in time with changes of color or color intensity of the secondary user device(s)310or turning on or off the secondary user device(s)310.

The controller305may receive a user request to deliver content comprising the content segment to the primary user device(s)309(e.g., smart TVs, smart displays, HDTVs etc.), via the gateway308and the network manager306. The controller305may also receive a request to synchronize output by the user devices (e.g., the primary user device(s)309and the secondary user device(s)310) of the content segment and additional visual effects. The controller305may comprise, for example, an IP set-top box (or other computing device) which provides two way communications on an IP network, decodes video streaming media (e.g., VOD content, web-streaming content, live streaming content, etc.), and/or enables the user devices to link to an IP network. The controller305may also comprise the one or more processors201ofFIG.2. The controller305may alert the network manager306of a user command to pause and/or unpause output of content that comprises the content segment. The network manager306may alert the gateway308to pause or unpause output of that content. If playback of that content segment is paused and unpaused, the gateway308(and/or one or more other computing devices in the system300) may adjust a manifest file associated with the content segment to proceed from times of pausing/unpausing. The adjusted manifest file may be sent from the gateway308to the secondary user device(s)310. Each of the various functional components in the system300(e.g., the controller305, the network manager306, the gateway308, etc.) may comprise a computing device200, which may be configured (e.g., based on stored executable instructions) to provide one or more functions to facilitate synchronized output, by the user devices, of the content segment and additional visual effects. One or more of the components of the system300may be combined in a single computing device, and/or may comprise multiple computing devices. Some or all of the computing device(s) implementing the functional components of the system300may be co-located with the primary user device(s)309and/or the secondary user device(s)310(e.g., in a premises102). Some or all of the computing device(s) implementing the functional components of the system300may be located remotely from the primary user device(s)309and/or the secondary user device(s)310(e.g., in one or more local offices103and/or other locations).

The network manager306may manage and/or connect to various functional components in the system300. The network manager306may comprise, for example, a WiFi network router (e.g., wireless access points127) which manages the system's WiFi network and connected home environment. For example, the network manager306may retrieve from the profile database307user personalization data such as, for example, user identifications, user account information, user locations, user favorites, user comments, user login credentials, user preferred genres, user lighting or color preferences, user specific themes, user commercials/sponsor themes, user pause/unpause settings, user customized room atmospheres, etc. The user personalization data may be used to modify and/or process a manifest file associated with the segment. For example, users may choose to increase color intensity associated with the secondary user device(s)310to represent suspenseful crime scenes that the users are watching on the primary user device(s)309. A user may configure a user profile, stored in the profile database307, to indicate that one or more color code values may be added to (or removed from) a manifest file. A user may choose to tone down colors output by the secondary user device(s)310when the users are watching educational TV programs with their children. Indications of various color characteristics (e.g., hue, saturation, brightness, etc.) and other visual parameters may be included in a manifest file associated with a content segment, via the user personalization data stored by the profile database307. The network manager306may deliver the content segment and/or its manifest file to the gateway308.

The gateway308may send content segments to the primary user device(s)309and may send manifest files to the secondary user device(s)310. Communications from the gateway308to the user devices, as well as communications from the user devices to the gateway308, may comprise other signals, instructions, messages, etc. The gateway308may support an agent or software that determines an expected network delay(s) between itself and each of the primary user device(s)309and/or the secondary user device(s)310and may record the expected network delay in a latency table. For example, the gateway308may cause the agent or software to ping the primary user device(s)309and the secondary user device(s)310and determine data trip times. The gateway308may cause the agent or software to check current path latency, view a list of components, routers or devices along the path, and/or record measurements for total latency by Time to First Byte (TTBF) or Round Trip Time (RTT) in the latency table. The latency table may be used to offset or otherwise adjust a plurality of time stamps in the manifest files. The latency table may be adjusted based on a sum of the expected network delay, for example, from the primary user device(s)309to the controller305and from the controller305to the secondary user device(s)310(to be described later). Network delays caused by processing and/or communications delays associated with other network components (e.g., Ethernet hubs, switches, bridges, routers, gateways, wireless access points, modems, etc.) in the system300may be considered when the latency table is generated and/or updated. The gateway308may read and/or adjust one or more manifest files in view of network delays between the user devices and the home entertainment system300.

A manifest file may also include or otherwise indicate metadata, for example, genre and commercial or sponsor themes, for content. The manifest file may include data that configures display of commercial-related colors or themes during playback of a commercial to enhance an advertising effect. For example, a commercial for a professional sports team may comprise images that display team colors. A manifest file may include codes that cause the secondary user device(s) to output similar colors in synchronization with the images of the commercial. For example, to create such an effect for a team color that comprises midnight green, the following color code may be used—Pantone Matching System (PANTONE) 316 C; Hexadecimal (HEX) #004C54; RGB (0, 76, 84); or CMYK (100, 0, 30, 70). Also, a manifest file may include a color from a designated set of color codes, based on an instruction from the content providers301. For example, the content providers301may cause the manifest file to include a color of with one team (e.g., midnight green) and not to include a color of an opposing team (e.g., blue) during a sports game or its commercial. The instruction from the content providers301may comprise a limited palette of colors that the manifest file may choose from. Other color codes and/or codes for optical, graphic, and/or special effects may be added to the manifest file by the network administrator or commercial sponsors to enhance efficacy of their advertisements. Such enhancement is not limited to commercial or sponsor themes, but may be applied to movies, live broadcasts, web conferencing, video clips, etc. The adjusted manifest file may be sent to each of the secondary user device(s)310to cause output of visual effects.

The system300and the user devices (e.g., the primary user device(s)309and the secondary user device(s)310) may experience network delay during synchronization with a content comprising a content segment. This network delay may range from just a few milliseconds to several hundred milliseconds. The network delay may comprise, for example, time for a bit (or other quantity) of data (e.g., electrical or optical signals, manifest files, time stamps, user commands, user profile or preferences, etc.) to travel across the system300or between the system300and the user devices. Example sources of network delays may comprise, for example, processing delay, transmission delay, propagation delay, etc.

For example, network delays may be caused during synchronization of the user devices for output of a content segment and additional visual effects. On Tuesday at 9:00 pm, a user may be watching a movie (e.g., a VOD movie) on the primary user device(s)309and push a remote control input that indicates the user wants the secondary user device(s)310to provide output to enhance viewing of the movie via the primary user device(s)309. A first delay may be caused by a user command, for example, traveling from the primary user device(s)309to the controller305, via the gateway308and the network manager306. A second delay may be caused by a manifest file associated with a segment of the movie, for example, traveling from the content manifest storage303to the gateway308, via the controller305and the network manager306. A third example delay may be caused by a traveling time, for example, between the gateway308and each of the secondary user device(s)310. These delays may be determined or added by the gateway308or any other component in the system300.

Based on a sum of network delays or any other delays (e.g., processing, transmission and propagation delays), a plurality of time stamps in the manifest file may be adjusted. For example, a net time delay may be 400 milliseconds, measured and/or otherwise determined by the gateway308. The net time delay may represent, for example, a time difference between (i) a time that the primary user device(s)309receive (and/or begin to process) a content segment sent by the gateway308and (ii) a time that the secondary user device(s)310receive (and/or begin to process) a manifest file corresponding to that content segment. The net time delay may be recorded in a latency table stored by the gateway308and/or any other component in the system300. The time stamps may be adjusted to compensate for the net time delay (e.g., offset times). For example, net time delay (e.g.,400milliseconds) may be subtracted from the time stamps to compensate for those net time delay. This subtraction of the net time delay from the time stamps may enable the secondary user device(s)310to synchronize output of additional visual effects with output by the primary user device(s)309of audio and/or visual components of a content segment. For example, the secondary user device(s)310may output colored light (and/or other effects) for video features, indicated by a manifest file, at or near the time that the primary user device(s)309display video frame(s) and/or beat(s) from which those video features were determined. Other time-offsetting schemes may be used to optimize the time stamps within the manifest file. The time stamps may be adjusted based on the latency table managed by the gateway308. The adjusted manifest file may be sent to each of the secondary user device(s)310. If the adjusted manifest file is executed, a user may enjoy an immersive experience while the secondary user device(s)310provide output based on, and synchronized with, video and audio features of the movie that the user is watching.

If a user is watching live content, playback of a segment of the live content may be briefly paused (e.g., less than one second) during a live content synchronization procedure, such as, for example, receiving user instructions from the primary user device(s)309, processing manifest files within the system300, sending the manifest files to the secondary user device(s)310, etc. In this way, the secondary user device(s)310may be in sync with the primary user device(s)309, and the user devices may start displaying synchronized output for the segment. The playback of the segment may be unpaused when the secondary user device(s)310are ready to execute a manifest file received from the gateway308. This brief pause may enable the user devices to prepare for synchronization with the segment without adjusting, for example, a plurality of time stamps within the manifest file. As previously described, if a user instruction to pause and/or unpause playback of content comprising the segment is received, the manifest file may be adjusted based on timings of pausing/unpausing, by the gateway308or one or more other computing devices in the system300. The adjusted manifest file may be sent from the gateway308to the secondary user device(s)310. Also or alternatively, the manifest file may be adjusted based on net time delay as previously described.

FIG.4shows an example timeline400for determining a plurality of time stamps associated with a content segment. The time stamps may be determined by the audiovisual data processor302, as previously described in connection withFIG.3. InFIG.4, a plurality of audio beats (e.g., audio beats401and402) may be detected from an audio component of the content segment. A plurality of frames (e.g., frames1to48) may be detected from a video component of the content segment. The audio beats and the frames may correspond with a plurality of time stamps (e.g., TS1to TS5), respectively. The frames (e.g., frames1,16,24,35and48) and their corresponding time stamps (e.g., TS1to TS5) may be sampled and/or stored by the audiovisual data processor302, to map audio and/or video features of the content segment into functionalities of the secondary user device(s)310. The audiovisual data processor302may sample or determine, for example, a first group of frames (e.g., frames16and35) associated with times of audio beats of the segment (e.g., audio beats401and402) and a second group of frames (e.g., frames1,24and48) associated with a regular sampling scheme (e.g., by sampling one frame from each 24 frames of the video component). A combination of these frame groups and/or other frames determined by one or more other sampling schemes may be used to generate a manifest file associated with the segment.

InFIG.4, the first group of frames16and35(indicated by vertical line patterns) may be determined based on, for example, a plurality of times of audio beats occurring at time stamps TS2and TS4. The audio beats401and402may be determined using an audio beat detection algorithm, for example, TarsosDSP, as previously described. The audio beats401and402may correspond with the frames16and35, as indicated by a set of arrows between the audio beats401and402and the frames16and35. The frames16and35may correspond with the time stamps TS2(e.g., 0.67 sec) and TS4(e.g., 1.46 sec), as indicated by a set of arrows between the frames16and35and the time stamps TS2and TS4. The time stamps TS2and TS4,which may be stored by the audiovisual data processor302, may also correspond with the audio beats401and402.

InFIG.4, a second group of frames (indicated by horizontal line patterns) may be determined based on a regular sampling scheme (e.g., sampling every 24th frame of the video component of the content segment). The video component may comprise a video frame rate of, for example, 24 frames/sec. The video component may comprise any other frame rates such as, for example, 30 frames/sec or 48 frames/sec. If the video frame rate is 24 frames/sec, one frame per each 24 frames may be sampled (e.g., frames1,24and48), as shown inFIG.4. The frames1,24and48may correspond with TS1(e.g., 0 sec), TS3(e.g., 1 sec), TS5(e.g., 2 sec), respectively. Different sampling rates may be used to sample a plurality of frames from the video component. The time stamps TS1, TS3and TS5may be stored by the audiovisual data processor302.

A manifest file may be produced based on information determined from analysis of the first group of frames (frames16and35) and/or the second group of frames (frames1,24, and48). The manifest file may be produced based on a first set of times of audio beats associated with the segment (e.g., the time stamps TS2and TS4corresponding to the first group of frames) and/or a second set of times associated with the regular sampling scheme (e.g., the time stamps TS1, TS3, and TS5corresponding to the second group of frames). The manifest file comprising one or more of these times may be stored by the audiovisual data processor302. In addition to these times, the manifest file may contain corresponding values for colors associated with the times indicated in the manifest file. For example, a manifest file may contain, for times corresponding to beats (e.g., TS2and TS4), values indicating an average of colors in regularly-sampled frames immediately preceding and immediately following the frame corresponding to a beat. In the example ofFIG.4, the manifest file may contain, as a color value corresponding to TS2, one or more values determined based on averages of colors from the frames1and24, and may further contain, as a color value corresponding to TS4, one or more values determined based on averages of colors from the frames24and48. Color values, corresponding to time values in a manifest file, may also or alternatively be determined in other ways. A manifest file may also or alternatively contain, for each of time stamps (e.g., TS1, TS2, TS3, TS4, or TS5), a value indicating a most dominant color of each frame associated with each of the time stamps. For example, the most prominent color in each frame may be determined by determining a baseline of RGB color values and finding a difference in the RGB color values, for example, by using standard deviation.

The secondary user device(s)310receiving a manifest file may determine, based on the time values and corresponding color values in the manifest file, times to output specific colors. Continuing the example from above, the secondary user devices may determine, based on the manifest file, to output at time TS2a color that is based on averages of colors from the frames1and24, and to output at time TS4a color that is based on averages of colors from the frames24and48. By outputting those colors based on the times indicated in the manifest file, the output of the secondary user device(s)310may be synchronized to output of the content segment by the primary user device(s)309.

A content segment, analyzed to determine information for inclusion in a manifest file, may, for example, comprise a segment of live content or VOD content. Although VOD and live content are used in examples that follow, the methods described herein may also or alternatively be performed in connection with segments comprising content of other types and/or formats. Live content may comprise, for example, linear content and/or content being streamed in real time or near real time. As also explained below, live content may be processed differently from other content to accommodate rapid delivery of the content via the gateway308that is subscribed or registered to the manifest message queue304. In the example system shown inFIG.3, for example, it may be advantageous to send live content from the manifest message queue304to the gateway308, instead of sending live content from the manifest message queue304to the gateway308via the network manager306. As also explained below, live content may be received via one or more network elements (e.g., the manifest message queue304) different from elements via which other types of content (e.g. VOD content) are received.

If a content segment is a segment of live content, the content segment and its manifest file may be processed by the audiovisual data processor302and may be stored in a manifest message queue304, as shown inFIG.3. The manifest message queue304may comprise a data structure, for example, a content topic, associated with one or more manifest files. The content topic may comprise a listing or channel of certain types of data (e.g., manifest files) to which one or more devices may subscribe (e.g., via gateway308). The manifest message queue304may publish one or more manifest files that are sent to all subscribing devices. This subscribing function may facilitate distribution of one or more manifest files in real time. As previously described, a user command to synchronize output of the user devices may be received by the controller305. Based on the user commend, playback of content comprising the content segment may be paused (e.g., less than a second). The controller305may send the user command to the network manager306. The network manager306may retrieve an indication of the content topic associated with the live content (or its manifest files) from the manifest message queue304. The network manager306may send an indication of the content topic to the gateway308. Based on the received indication of the content topic, the gateway308may subscribe to the content topic via the manifest message queue304, and may receive the manifest file from the manifest message queue304based on that subscribing. The content segment may be sent to the primary user device(s)309, the manifest file may be sent to the secondary user device(s)310, and the playback of the content comprising the content segment may be unpaused.

If a content segment is a segment of VOD content, the content segment and its manifest file may be processed by the audiovisual data processor303and stored by the content manifest storage303. Playback of the VOD content comprising the content segment may be paused during synchronization of the user devices, and network delays may be avoided or minimized. In this case, the playback may be resumed after the secondary user device(s)310are ready to execute a manifest file for the VOD segment. Alternatively, if the playback is not paused and continues, a net time delay or offset time (e.g., resulting from network delays) may be measured or calculated by the gateway308or one or more other computing devices in the system300. For example, the gateway308may modify or adjust time values in the manifest file (e.g., time stamps in the manifest file) to compensate for the net time delay or offset time, as previously described in connection withFIG.3. The modified manifest file may be sent to the secondary user device(s)310by the gateway308or one or more other computing devices in the system300. The gateway308may also measure network latencies and/or update the latency table based on various network latencies (e.g., processing, transmission and propagation delays), as previously described in connection withFIG.3.

FIGS.5A to5Cshow a flow chart for an example method for synchronizing user devices outputting content segments and one or more visual effects associated with those content segments. This example method may be explained with reference toFIGS.3and4. Although steps of the method are described below as performed by certain functional components fromFIG.3, some or all of the below-described steps could also or alternatively be performed by one or more computing devices corresponding to one or more other functional components fromFIG.3and/or by one or more other computing devices. One or more steps shown inFIGS.5A to5Cmay be rearranged, omitted, or modified, and/or other steps added.

InFIG.5A, at step501, the audiovisual data processor302may receive a content segment (e.g., a segment of VOD content, live content, web-streaming content, etc.) from one or more of the content providers301. The content segment may comprise a plurality of time stamps associated with the segment, as previously described. At step502, audio and video components of the content segment may be determined. The audio component may comprise data for an audio track. The video component may comprise data for one or more video frames. The audio and video components may be stored by the audiovisual data processor302and/or a server (e.g., content server106, app server107, external network209, etc.).

At step503, a plurality of time values, for a manifest file associated with the content segment, may be determined by the audiovisual data processor302. The audio component may be read via a beat tracking algorithm (e.g., a Java class named “BeatRootOnsetEventHandler”) or beat detectors in music visualization software plugins (e.g., Windows Media Player, QuickTime Player, iTunes, etc.). The following scripts and/or codes (which are by way of example only, and are not limiting) may be used to read the audio component to return times of audio beats associated with the content segment:import be.tarsos.dsp.beatroot.BeatRootOnsetEventHandler;public BeatRootOnsetEventHandler 0;public void handleOnset (double time, double salience);public void trackB eats (OnsetHandler beatHandler);

The “BeatRootOnsetEventHandler” class may be used to return times of audio beats from the audio component. A beat tracking algorithm may be configured, using the above example scripts, to analyze the audio component and determine the times of audio beats associated with the content segment. The times of audio beats, which may be the time stamps associated with portions of the content segment during which the beats occur (e.g., the time stamps TS2and TS4inFIG.4), may be stored by the audiovisual data processor302in a manifest file generated for the content segment.

Time values for the manifest file may also or alternatively be determined based on a regular sampling of the content segment. For example, a time stamp corresponding to every Nth frame (e.g., every 24th frame, as in the example ofFIG.4) may be determined, and those time stamps (the time stamps TS1, TS3, and TS5inFIG.4) may be included in the manifest file. Such time stamps based on periodic sampling, together with corresponding color values, may be used to cause the secondary user device(s)310to continuously output color that corresponds to color being output for the content segment. Such continuously-output color may be provided as an alternative to or in conjunction with color output in synchronization with beats or other audio features.

At step504, the audiovisual data processor302may determine, based on the time stamps determined in step503, a plurality of corresponding frames from the video component (e.g., the frames1,16,24,35and48in the example ofFIG.4). The frames determined in step504associated with the matched time stamps may be stored by the audiovisual data processor302.

At step505, the audiovisual data processor302may determine primary color information and/or other features for some or all of the frames determined in step504. For example, average colors (e.g., RGB, HSB, HEX or CMYK colors) associated with the determined frames may be determined. For some or all of the determined frames, an average color associated with whole pixels within a single frame may be determined. Alternatively, an average color associated with a portion of a single frame may be determined (e.g., if that portion of the frame may be focused or emphasized). The information determined for each frame may be added to the manifest as corresponding color values for the time values, and/or may be further processed to obtain color values for inclusion in the manifest file. For example, to determine a color value corresponding to a time value for a beat (e.g., the time value TS2in the example ofFIG.4), the average color values of frames determined by regular sampling (e.g., the frames1and24in the example ofFIG.4) may be further averaged. Values based on that further average may be added to the manifest file as the color values corresponding to a time value for the beat. Color values corresponding to some or all of the time values in the manifest file may be determined and added to the manifest file. Other types of visual information such as, for example, logos, diagrams, icons, signs, texts, symbols or patterns, may also be indicated via the manifest file. Shown below is an example manifest file comprising, for example, time values (in 10−8seconds) and corresponding HSB color values:

[ . . .{“timeLength”:8.7333333E7,“hsb”:{“hue”:7411,“saturation”:144,“brightness”:148}}, {“timeLength”:8.8708333E7,“hsb”:{“hue”:7801,“saturation”:123,“brightness”:145}},{“ti meLength”:9.0083333E7,“hsb”:{“hue”:14563,“saturation”:16, “brightness”:91}},{“timeL ength”:9.1333333E7,“hsb”:{“hue”:15291,“saturation”:14,“brightness”:91}} . . . ]

At step506, the audiovisual data processor302may determine whether the content segment is a segment of live content. As previously described in connection withFIG.3, a content segment of live content may be sent to the manifest message queue304by the audiovisual data processor302, and a content segment of live content may be sent to the content manifest storage303by the audiovisual data processor302. To facilitate sending of the live content segment and/or its manifest file(s) to the gateway308, the live content segment may be stored by the manifest message queue304, the gateway308may be subscribed to the manifest message queue304, and the gateway308may continuously receive the live content segment and/or its manifest file(s) from the manifest message queue304. At step506, the audiovisual data processor302may send an indication for the content segment of live content to the controller305. Then the controller305may receive the indication from the audiovisual data processor302and, after step506, manage, send or direct the content segment and/or its manifest file to one or more computing devices (e.g., the content manifest storage303, the network manager306, the gateway308, the primary user device(s)309, the secondary user device(s)310, etc.). Alternatively, the audiovisual data processor302may send the content segment and its manifest file to the controller305and the controller305may determine whether the content segment is a segment of live content. If the content segment is a segment of live content (step506: yes), the controller305may proceed to step507. At step507, the manifest file for the content segment, as previously described in connection withFIG.3, may be stored by the manifest message queue304(e.g., the controller305may cause the manifest file to be stored in the manifest message queue). The manifest message queue304may comprise a lightweight buffer which temporarily stores the manifest file and may allow other physical components in the system300to connect to the manifest message queue304in order to send and/or receive the live content manifest file. The manifest file may be stored by the manifest message queue304until another component (e.g., the network manager306or the gateway308) requests to subscribe to the manifest message queue304and/or otherwise retrieves the manifest file. The controller305may proceed to step509ofFIG.5B.

At step509, the controller305may determine whether an instruction (e.g., a prior user instruction) to synchronize output by user devices has previously been processed and is still in effect. If such an instruction is not in effect (step509: no), the controller305may proceed to step510.

At step510, the controller305may determine whether an instruction to synchronize user devices has been received. If an instruction has not been received (step510: no), the controller305may proceed to step523(described below). If an instruction has been received (step510: yes), the controller305may proceed to step511.

At step511, playback of the content comprising the content segment (e.g., via the primary user device(s)309) may be paused to prepare for synchronization of the user devices. To pause the playback, the gateway308may, for example, continue to send the primary user device(s)309a frame (e.g., a frame preceding the content segment in the live content) with an overlay indicating playback is paused. The controller305may send a request to pause playback of the content to the gateway308, via the network manager306. The playback of the content may be paused until additional steps (e.g., steps512to517described below) have been performed in preparation for synchronized output, based on the manifest file and subsequent manifest files, of visual effects and the content segment and subsequent content segments.

At step512, the controller305may send the indication for the content segment of live content (e.g., the content's title, index, identification, etc.) to the network manager306. The network manager306may send to the gateway308an indication of a content topic associated with the content segment manifest file (and manifest files for subsequent content segments of the same content). The gateway308, based on the content topic, may subscribe to the manifest message queue304. The gateway308may subsequently receive, based on that subscription and from the manifest message queue304, the manifest file for the content segment and subsequent manifest files for subsequent content segments.

At step513, user profile and/or user preference data, for example, as previously described in connection withFIG.3, may be retrieved from the profile database307and sent to the gateway308by the network manager306or one or more other computing devices in the system300. The gateway308may store that data and use that data to modify manifest files for the content associated with the pending instruction to synchronize. At step514, the gateway308may receive the content segment. At step515, the gateway308may receive the manifest file generated for that segment (e.g., as described in connection with steps501through505). In step516, the gateway308may modify the manifest file based on the data received and stored in step513. For example, a user may be a fan of a sports team and select a set of team colors (e.g., midnight green corresponding to PANTONE 316 C; HEX #004C54; RGB (0, 76, 84); or CMYK (100, 0, 30, 70), as previously described in connection withFIG.3) for the user profile and/or user preference data. The gateway308may add or change the set of team colors in the manifest file at times of team commercials, intermissions, or cheering moments broadcast by the primary user device(s)309. The set of team colors may also be turned on and off by the secondary user device(s)310, based on time stamps in the adjusted manifest file. Other user-selected optical, graphic, and/or special effects may be added to the manifest file by the gateway308or one or more other computing devices in the system300. For example, a user may configure the user profile and/or user preference data to cause a flashing light effect when the sports team succeeds a touchdown (e.g., notified by one or more of the content providers301). Based on the user profile and/or user preference data, the gateway308may adjust color codes and time stamps within the manifest file to visualize a flashing light effect (e.g., a rapid change of colors or turning on/off one or more lights via the secondary user device(s)310). Step516may be omitted, e.g., if no modifications based on the user profile and/or user preference data is desired.

At step517, the gateway308may unpause playback of the content, may send the content segment to the primary user device(s)309, and may send the manifest file to the secondary user device(s)310. Also or alternatively, a request to unpause the playback of the segment of live content on the primary user device(s)309may be sent from the gateway308to the controller305, via the network manager306. The controller305may receive the request to unpause the playback. The controller305may cause the gateway308to unpause the playback, send the segment of live content to the primary user device(s)309, and send the manifest file to the secondary user device(s)310. The primary user device(s)309, after receiving the content segment, may output the audio and video components of the content segment. The secondary user device(s)310, after receiving the manifest file, may control lighting element(s) and/or other elements to produce visual effects based on time values in the manifest file and on color values (and/or other values) corresponding to those time values. Users may thereby experience an immersive experience of synchronized output of the user devices. For example, a primary user device309may show the live content (e.g., a Super Bowl halftime show) and a secondary user device(s)310may show average colors of what the smart TV is displaying and/or pulses to audio beats associated with the Super Bowl halftime show.

At step523, the controller305may determine (e.g., based on data in the content segment, based on program guide or other data, and/or based on other information) whether one or more additional content segments are available for the content. If there are more segments (step523: yes), the controller305may proceed to step501ofFIG.5A. At step501, the home entertainment system300may receive the next content segment, generate a manifest file for that next content segment, etc. If there are no more content segments (step523: no), the controller305may end the method.

Referring back to step509, if an instruction to synchronize output by user devices is still in effect (step509: yes), the controller305may proceed to step518. At step518, the gateway308may receive a next content segment (e.g., one or more additional content segments associated with the live content). At step519, the gateway308may send the next content segment to the primary user device(s)309. At step520, the gateway308may receive the manifest file generated for that next segment (e.g., as described in connection with steps501through505).

At step521, the gateway308may modify the manifest file based on the user profile and/or user preference data previously received and stored in step513of a previous loop, where the controller305proceeded to step510through517(step509: no). The gateway308may also or alternatively include time value adjustment based on net time delay and/or other delay (e.g., network delays if content segments are received from a source in the system300different from the manifest message queue304and a time difference between receipt of the content segments and receipt of associated manifest files is detected). Step521may be omitted if no modification based on the user profile and/or user preference data is desired. At step522, the gateway308or one or more other computing devices in the system300may send the manifest file to the secondary user device(s)310. The controller305may proceed to step523.

Referring back toFIG.5A, if content segment is not a segment of live content (step506: no), the controller305may proceed to step508. If the content segment is not a segment of live content, the content segment may be a segment of VOD content (or other type of content). The manifest file for the content segment of VOD content may be stored by the content manifest storage303. The content segment and its manifest file may be retrieved by the controller305and from the content manifest storage303, and the controller305may proceed to step524ofFIG.5C.

InFIG.5C, at step524, the controller305may determine whether an instruction (e.g., a prior user instruction) to synchronize output by user devices has previously been processed and is still in effect. If such an instruction is not in effect (step524: no), the controller305may proceed to step525.

At step525, the controller305may determine whether an instruction to synchronize user devices has been received. If an instruction has not been received (step525: no), the controller305may proceed to step537(described below). If an instruction has been received (step525: yes), the controller305may cause, for example, the network manager306to perform step526.

At step526, user profile and/or user preference data, for example, as previously described in connection withFIG.3and step513ofFIG.5B, may be retrieved from the profile database307and sent to the gateway308by the network manager306or one or more other computing devices in the system300. The gateway308may store that data and use that data to modify manifest files for the content segment. At step527, the gateway308may receive the content segment. At step528, the gateway308may receive the manifest file generated for that segment (e.g., as described in connection with steps501through505).

At step529, the gateway308may modify the manifest file based on net time delay (and/or other delays), as previously described in connection withFIG.3. The gateway308or one or more computing devices in the system300may employ the latency table recording various network delays to adjust time values (e.g., time stamps) within the manifest file. For example, the gateway308or the network manager306may offset time stamps in the manifest file based on one or more net time delays caused by, for example, processing delay, transmission delay, propagation delay, etc. The latency table may be updated based on network delays associated with the system300and the user devices (e.g., primary user device(s)309and secondary user device(s)310). If a user instruction to pause, fast-forward, or rewind the content segment is received, the latency table may be updated and the gateway308, based on the updated latency table, may adjust the manifest file for that content segment. The adjusted manifest file may be sent to the secondary user device(s)310by the gateway308.

At step530, the gateway308may modify the manifest file based on the data received and stored in step526. Step530may be omitted, e.g., if no modifications based on the user profile and/or user preference data is desired. At step531, the gateway308may send the content segment to the primary user device(s)309and send the manifest file to the secondary user device(s)310.

At step537, the controller305may determine (e.g., based on data in the content segment, based on program guide or other data, and/or based on other information) whether one or more additional content segments are available for the content. If there are more segments (step537: yes), the controller305may proceed to step501ofFIG.5A. At step501, the home entertainment system300may receive the next content segment, generate a manifest file for that next content segment, etc. If there are no more content segments (step537: no), the controller305may end the method.

Referring back to step524, if an instruction to synchronize output by user devices is still in effect (step524: yes), the controller305may cause, for example, the gateway308to perform step532. At step532, the gateway308may receive a next content segment (e.g., one or more additional content segments associated with the VOD content). At step533, the gateway308may send the next content segment to the primary user device(s)309. At step534, the gateway308may receive the manifest file generated for that next segment (e.g., as described in connection with steps501through505).

At step535, the gateway308may modify the manifest file based on the user profile and/or user preference data previously received and stored in step526of a previous loop, where the controller305proceeded to step525through531(step524: no). The gateway308may also or alternatively modify the manifest file based on net time delay (and/or other delays), as previously described in connection with step529ofFIG.5C. At step536, the gateway308or one or more other computing devices in the system300may send the manifest file to the secondary user device(s)310. The controller305may proceed to step537.

The gateway308may also or alternatively send content segments and associated manifest files to the primary user device(s)309, instead of sending the manifest files to the secondary user device(s)310, for example, at step517ofFIG.5Band step531ofFIG.5C. The primary user device(s)309may receive the content segments and the manifest files from the gateway308, process and/or adjust the manifest files, and forward the manifest files to the secondary user device(s)310. The primary user device(s)309may manage or control the secondary user device(s)310via one or more manifest files associated with a content segment of live or VOD content.

The gateway308may also or alternatively send content segments and associated manifest files to the secondary user device(s)310, instead of sending the content segments to the primary user device(s)309, for example, at step517ofFIG.5Band step531ofFIG.5C. The secondary user device(s)310may receive the content segments and the manifest files from the gateway308, process and/or adjust the manifest files, and forward the content segments to the primary user device(s)309.

The gateway308may also or alternatively send content segments and associated manifest files to one or more other computing devices, different from the primary user device(s)309and the secondary user device(s)310. The one or more other computing devices may receive the content segments and the manifest files from the gateway308, process and/or adjust the manifest files, and forward the content segments to the primary user device(s)309and manifest files to the secondary user device(s)310.

The audiovisual data processor302may also or alternatively generate a plurality of manifest files for a single content segment, with each of the manifest files comprising different time values/adjustments. For example, the manifest files may be separately modified or adjusted by the gateway308if each of the secondary user device(s)310receiving the manifest files incurs a different network latency (e.g., if one of the secondary user devices310has more latency than another secondary user device).

The gateway308may also or alternatively modify or adjust the manifest files based on individual user preferences (e.g., one user watching content in one room and another user watching same content in a different room). The gateway308may receive the manifest file for a content segment of that content, make necessary number of copies of the manifest file, modify each of the copies based on the individual user preferences, and send the modified copies to different secondary user device(s)310(e.g., located in different rooms).

Although examples are described above, features and/or steps of those examples may be combined, divided, omitted, rearranged, revised, and/or augmented in any desired manner. Various alterations, modifications, and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be part of this description, though not expressly stated herein, and are intended to be within the spirit and scope of the disclosure. Accordingly, the foregoing description is by way of example only, and is not limiting.