Patent Description:
The invention further relates to a method of controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content, said audiovisual content comprising an audio portion and a video portion, said audio portion comprising multiple audio channels.

The experience of content, visual or auditory, can benefit immensely from a dynamic lighting system. An entertainment lighting system like e.g., Hue Sync, can dramatically alter a user's viewing experience by rendering light colors that are extracted from a scene in real-time, or scripted offline. In addition to this, the accompanying audio of the content could be taken into account, where e.g., the intensity of the audio can be used to modulate the rendered light effects.

For example, <CIT> discloses that a scene accompanied by high intensity audio may be rendered with higher intensity light effects than the same scene accompanied by low intensity audio. <CIT> further discloses that video-based ambient lighting characteristics intended for presentation on a left side of a display may be combined with audio-based ambient lighting data relating to a left channel while a video-based ambient lighting characteristics intended for presentation on a right side of the display may be combined with audio-based ambient lighting data relating to a right-channel. <CIT> discloses a system for visually displaying positional audio information using multiple light sources based on multi-channel audio information.

It is a drawback of <CIT> that the disclosed system does not take advantage of newer audio formats to create enhanced ambient lighting effects which are based both on a video portion and an audio portion of the audiovisual content.

It is a first aspect of the invention to provide a system according to claim <NUM>, which is able to create enhanced ambient lighting effects which are based both on a video portion and an audio portion of the audiovisual content.

It is a second aspect of the invention to provide a method according to claim <NUM>, which can be used to create enhanced ambient lighting effects which are based both on a video portion and an audio portion of the audiovisual content.

In the first aspect of the invention, a system for controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content, said audiovisual content comprising an audio portion and a video portion, said audio portion comprising multiple audio channels, comprises at least one input interface, at least one transmitter, and at least one processor configured to obtain said audiovisual content via said at least one input interface, determine a first characteristic of a first audio channel of said multiple audio channels or of an audio object comprised in said audio portion, said first characteristic being indicative of an audio source position, and associate, based on said first characteristic, said first audio channel or said audio object with a first lighting device of said plurality of lighting devices, wherein said associating is based on said audio source position relative to a position of said first lighting device.

The at least one processor is further configured to determine a second characteristic of a second audio channel of said multiple audio channels, associate, based on said second characteristic, said second audio channel with said first lighting device and with a second lighting device of said plurality of lighting devices, said first audio channel not being associated with said second lighting device, determine whether second audio content in said second audio channel meets one or more predetermined criteria, determine at least a chromaticity based on said video portion of said audiovisual content, determine a first light effect based on said determined chromaticity, wherein if said one or more predetermined criteria are not met, the light intensity of said first light effect is based on first audio content in said first audio channel or in said audio object, and if said one or more predetermined criteria are met, the light intensity of said first light effect is based on said second audio content in said second audio channel, and control, via said at least one transmitter, said first lighting device to render said first light effect.

More and more people are enjoying surround sound configurations at home and most TV programs and movies have surround sound audio nowadays. By taking the semantic properties of the audio channels and optionally audio objects into account when determining the light effects, it becomes possible to determine light effects which reflect what is happening in the audio channels and optionally audio objects and thereby realize a more immersive entertainment experience.

This is achieved by not simply using the intensity of audio content to modulate the light effects but by using the semantics of the audio channels and optionally audio objects to modulate the light effects. For example, the low frequency effects (subwoofer) channel may influence all connected lights such that bass-heavy effects like a loud explosion are not only heard but also seen throughout the entire entertainment area. This can be contrasted to audio effects on left/right audio channels (also referred to as side channels) which may be used to modulate light effects on only the left/right positioned lighting devices.

With the above-described system, a first audio channel or audio object may be mapped to a first lighting device based on an audio source position associated with the first audio channel relative to a position of the first lighting device and a second audio channel may be mapped to multiple lighting devices including the first lighting device. A sound effect reproduced on the first audio channel or audio object can be located more precisely by the user than a sound effect reproduced on the second audio channel, e.g., because the second audio channel is a low frequency effects (subwoofer) channel.

By mapping the first audio channel or audio object to fewer lighting devices, e.g., a single lighting device, than the second audio channel, the light effects determined based on first audio content on the first audio channel or audio object reflect that the corresponding sound effect has a more specific location while the light effects determined based on second audio content on the second audio channel reflect that the corresponding sound effect has a less specific location.

By determining the chromaticity (and optionally the entire color) of the light effects based on at least the video portion of the audiovisual content and determining the light intensity of the light effects based on at least the audio portion of the audiovisual content, wherein the light intensity is based on the second audio content in the second audio channel at certain moments, e.g., in case of loud events, the best light experience may be obtained. Optionally, the lightness of the color is also determined based on the audio portion of the audiovisual content.

Said second characteristic might not be indicative of an audio source position. For example, said second characteristic may indicate whether said second audio channel is a low frequency effect channel. Alternatively, said first characteristic may be determined of said audio object and said second characteristic may be indicative of a desired speaker position for said second audio channel, for example.

Said at least one processor may be configured to determine the light intensity of said first light effect further based on said first audio content in said first audio channel or in said audio object if said one or more predetermined criteria are met. By always determining the light intensity based on the first audio content in the first audio channel, a less intense light experience may be obtained, which is preferred by certain users. For example, the highest light intensity may only be achieved if there is a loud event in both the first audio channel and the second audio channel. By always determining the light intensity based on the first audio content in the audio object, the audio objects are emphasized in the light effects.

Said at least one processor may be configured to determine the light intensity of said first light effect further based on said video portion of said audiovisual content. This may be used to ensure that the light intensity not only matches the audio portion but also the video portion. The user may be able to configure whether the light intensity of the light effects should be determined based on the video portion of the audiovisual content.

Said at least one processor may be configured to determine whether said second audio content in said second audio channel meets said one or more predetermined criteria by determining whether an audio intensity of said second audio content exceeds a threshold. Thus, the light intensity may be determined based on the second content in the second audio channel if there is a loud event in the second audio channel, e.g., a loud explosion.

Said at least one processor may be configured to select a spatial region in a current frame of said video portion in dependence on whether said one or more predetermined criteria are met and determine at least said chromaticity from only said selected spatial region. Although the chromaticity (or entire color) of the light effects is preferably determined based on the video portion of the audiovisual content, the audio portion may still have some influence on the chromaticity (or entire color). For example, the color of the light effect for a lighting device positioned on the left may be extracted from a center region of a video frame if a loud event is detected in the low frequency effects channel and from a left region of the video frame otherwise.

Said first characteristic may be determined of said first audio channel and said at least one processor may be configured to determine whether an audio intensity of said first audio content exceeds a threshold, select a spatial region in a current frame of said video portion in dependence on whether said audio intensity of said first audio content exceeds said threshold, and determine at least said chromaticity from only said selected spatial region. From which spatial region of the video portion the chromaticity (or entire color) is extracted may not only depend on the second audio content in the second audio channel but also on the first audio content in the first audio channel. For example, when a loud event is detected in the second audio channel, the light intensity of the light effects for all lighting devices may be determined based on the second audio content in the second audio channel and the color of the light effect for a lighting device positioned on the left may be extracted from a center region of a video frame if the loud event is also detected in the first audio channel and from a left region of the video frame if not.

Said at least one processor may be configured to determine one or more speaker signals for a loudspeaker based on said audio portion of said audiovisual content. Said at least one processor may be configured to determine the light intensity of said first light effect based on said one or more speaker signals. Instead of determining the light intensity of the first light effect directly based on the audio portion of the audiovisual content, the light intensity may be determined based on the one or more speaker signals. This may be beneficial if the user's audio system is not able to recreate the audio source positions specified in the content close enough or if the user's audio system enhances the audio effects specified in the audiovisual content.

As an example of the latter, audio upmixing algorithms exist that create pseudo channels for traditional content that does not comprise those channels (e.g., Dolby Surround, which does not contain height channels). An example of such an upmixing algorithm is DTS Virtual: X. Other audio analysis steps, e.g. determining the first and second characteristics and/or determining whether the second audio content in the second audio channel meets the one or more predetermined criteria, may also be performed based on the one or more speaker signals.

Alternatively, said at least one processor may be configured to determine the light intensity of said first light effect further based on information on available speakers and/or information on used three-dimensional audio virtualization. This may be beneficial if the user's audio system is not able to recreate the audio source positions specified in the content close enough.

In the second aspect of the invention, a method of controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content, said audiovisual content comprising an audio portion and a video portion, said audio portion comprising multiple audio channels, comprises obtaining said audiovisual content, determining a first characteristic of a first audio channel of said multiple audio channels or of an audio object comprised in said audio portion, said first characteristic being indicative of an audio source position, and associating, based on said first characteristic, said first audio channel or said audio object with a first lighting device of said plurality of lighting devices, wherein said associating is based on said audio source position relative to a position of said first lighting device.

Said method further comprises determining a second characteristic of a second audio channel of said multiple audio channels, associating, based on said second characteristic, said second audio channel with said first lighting device and with a second lighting device of said plurality of lighting devices, said first audio channel not being associated with said second lighting device, determining whether second audio content in said second audio channel meets one or more predetermined criteria, determining at least a chromaticity based on said video portion of said audiovisual content, determining a first light effect based on said determined chromaticity, wherein if said one or more predetermined criteria are not met, the light intensity of said first light effect is based on first audio content in said first audio channel, and if said one or more predetermined criteria are met, the light intensity of said first light effect is based on said second audio content in said second audio channel, and controlling said first lighting device to render said first light effect. Said method may be performed by software running on a programmable device. This software may be provided as a computer program product.

A non-transitory computer-readable storage medium stores at least one software code portion, the software code portion, when executed or processed by a computer, being configured to perform executable operations for controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content, said audiovisual content comprising an audio portion and a video portion, said audio portion comprising multiple audio channels.

The executable operations comprise obtaining said audiovisual content, determining a first characteristic of a first audio channel of said multiple audio channels or of an audio object comprised in said audio portion, said first characteristic being indicative of an audio source position, and associating, based on said first characteristic, said first audio channel or said audio object with a first lighting device of said plurality of lighting devices, wherein said associating is based on said audio source position relative to a position of said first lighting device.

The executable operations further comprise determining a second characteristic of a second audio channel of said multiple audio channels, associating, based on said second characteristic, said second audio channel with said first lighting device and with a second lighting device of said plurality of lighting devices, said first audio channel not being associated with said second lighting device, determining whether second audio content in said second audio channel meets one or more predetermined criteria, determining at least a chromaticity based on said video portion of said audiovisual content, determining a first light effect based on said determined chromaticity, wherein if said one or more predetermined criteria are not met, the light intensity of said first light effect is based on first audio content in said first audio channel, and if said one or more predetermined criteria are met, the light intensity of said first light effect is based on said second audio content in said second audio channel, and controlling said first lighting device to render said first light effect.

<FIG> shows a first embodiment of the system for controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content: an HDMI module <NUM>. The audiovisual content comprises an audio portion and a video portion. The audio portion comprising multiple audio channels. The HDMI module <NUM> may be a Hue Play HDMI Sync Box, for example. In the example of <FIG>, the HDMI module <NUM> controls five lighting devices <NUM>-<NUM>.

In the example of <FIG>, The HDMI module <NUM> can control lighting devices <NUM>-<NUM> via a bridge <NUM>. The bridge <NUM> may be a Hue bridge, for example. The bridge <NUM> communicates with lighting devices <NUM>-<NUM>, e.g., using Zigbee technology. The HDMI module <NUM> is connected to a wireless LAN access point <NUM>, e.g., via Wi-Fi. The bridge <NUM> is also connected to the wireless LAN access point <NUM>, e.g., via Wi-Fi or Ethernet.

Alternatively or additionally, the HDMI module <NUM> may be able to communicate directly with the bridge <NUM>, e.g. using Zigbee technology, and/or may be able to communicate with the bridge <NUM> via the Internet/cloud. Alternatively or additionally, the HDMI module <NUM> may be able to control lighting devices <NUM>-<NUM> without a bridge, e.g. directly via Wi-Fi, Bluetooth or Zigbee or via the Internet/cloud.

The wireless LAN access point <NUM> is connected to the Internet <NUM>. A media server <NUM> is also connected to the Internet <NUM>. Media server <NUM> may be a server of a video-on-demand service such as Netflix, Amazon Prime Video, Hulu, HBO Max, Paramount+, Peacock, Disney+ or Apple TV+, for example. The HDMI module <NUM> is connected to a display device <NUM> and local media receivers <NUM> and <NUM> via HDMI. The local media receivers <NUM> and <NUM> may comprise one or more streaming or content generation devices, e.g., an Apple TV, Microsoft Xbox and/or Sony PlayStation, and/or one or more cable or satellite TV receivers. The display device <NUM> is connected to an audio system <NUM>, e.g., via HDMI ARC. The audio system <NUM> is connected to speakers <NUM>.

In an alternative embodiment, the system for controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content is a display device. In this alternative embodiment, HDMI module logic may be built-in in the display device. Media receivers <NUM> and <NUM> may then also be comprised in the display device, e.g., a smart TV.

The HDMI module <NUM> comprises a receiver <NUM>, a transmitter <NUM>, a processor <NUM>, and memory <NUM>. The processor <NUM> is configured to obtain the audiovisual content via receiver <NUM> from media receiver <NUM> or <NUM>, determine a first characteristic of a first audio channel of the multiple audio channels or of an audio object comprised in the audio portion, and associate, based on the first characteristic, the first audio channel or the audio object with a first lighting device of the lighting devices <NUM>-<NUM>. The first characteristic is indicative of an audio source position and the associating is based on the audio source position relative to a position of the first lighting device.

The processor <NUM> is further configured to determine a second characteristic of a second audio channel of the multiple audio channels and associate, based on the second characteristic, the second audio channel with the first lighting device and with a second lighting device of the lighting devices <NUM>-<NUM>. The first audio channel is not associated with the second lighting device.

The processor <NUM> is further configured to determine whether second audio content in the second audio channel meets one or more predetermined criteria, determine at least a chromaticity based on the video portion of the audiovisual content, determine a first light effect based on the determined chromaticity, and control, via the transmitter <NUM>, the first lighting device to render the first light effect.

If the one or more predetermined criteria are not met, the light intensity of the first light effect is based on first audio content in the first audio channel or in the audio object, and if the one or more predetermined criteria are met, the light intensity of the first light effect is based on the second audio content in the second audio channel.

In the embodiment of the HDMI module <NUM> shown in <FIG>, the HDMI module <NUM> comprises one processor <NUM>. In an alternative embodiment, the HDMI module <NUM> comprises multiple processors. The processor <NUM> of the HDMI module <NUM> may be a general-purpose processor, e.g., ARM-based, or an application-specific processor. The processor <NUM> of the HDMI module <NUM> may run a Unix-based operating system for example. The memory <NUM> may comprise one or more memory units. The memory <NUM> may comprise solid-state memory, for example.

The receiver <NUM> and the transmitter <NUM> may use one or more wired or wireless communication technologies such as Zigbee to communicate with the bridge <NUM> and HDMI to communicate with the display device <NUM> and with local media receivers <NUM> and <NUM>, for example. In an alternative embodiment, multiple receivers and/or multiple transmitters are used instead of a single receiver and a single transmitter. In the embodiment shown in <FIG>, a separate receiver and a separate transmitter are used. In an alternative embodiment, the receiver <NUM> and the transmitter <NUM> are combined into a transceiver. The HDMI module <NUM> may comprise other components typical for a network device such as a power connector. The invention may be implemented using a computer program running on one or more processors.

In the embodiment of <FIG>, the system of the invention is an HDMI module. In an alternative embodiment, the system may be another device, e.g., a mobile device, laptop, personal computer, a bridge, a media rendering device, a streaming device, or an Internet server. In the embodiment of <FIG>, the system of the invention comprises a single device. In an alternative embodiment, the system comprises multiple devices.

<FIG> shows a second embodiment of the system for controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content: a mobile device <NUM>. The mobile device <NUM> may be a smart phone or a tablet, for example. The lighting devices <NUM>-<NUM> can be controlled by the mobile device <NUM> via the bridge <NUM>. The mobile device <NUM> is connected to the wireless LAN access point <NUM>, e.g., via Wi-Fi.

The mobile device <NUM> comprises a receiver <NUM> a transmitter <NUM>, a processor <NUM>, a memory <NUM>, and a display <NUM>. The video portion is preferably displayed on the display device <NUM> but could also be displayed on display <NUM> of the mobile device <NUM>. In the former case, the audio portion may be rendered on the display device <NUM> or on an audio system (not shown in <FIG>) connected to the display device <NUM>, for example.

The processor <NUM> is configured to obtain the audiovisual content via receiver <NUM>, determine a first characteristic of a first audio channel of the multiple audio channels or of an audio object comprised in the audio portion, and associate, based on the first characteristic, the first audio channel or the audio object with a first lighting device of the lighting devices <NUM>-<NUM>. The first characteristic is indicative of an audio source position and the associating is based on the audio source position relative to a position of the first lighting device.

In the embodiment of the mobile device <NUM> shown in <FIG>, the mobile device <NUM> comprises one processor <NUM>. In an alternative embodiment, the mobile device <NUM> comprises multiple processors. The processor <NUM> of the mobile device <NUM> may be a general-purpose processor, e.g., from ARM or Qualcomm or an application-specific processor. The processor <NUM> of the mobile device <NUM> may run an Android or iOS operating system for example. The display <NUM> may be a touchscreen display, for example. The display <NUM> may comprise an LCD or OLED display panel, for example. The memory <NUM> may comprise one or more memory units. The memory <NUM> may comprise solid state memory, for example.

The receiver <NUM> and the transmitter <NUM> may use one or more wireless communication technologies such as Wi-Fi (IEEE <NUM>) to communicate with the wireless LAN access point <NUM>, for example. In an alternative embodiment, multiple receivers and/or multiple transmitters are used instead of a single receiver and a single transmitter. In the embodiment shown in <FIG>, a separate receiver and a separate transmitter are used. In an alternative embodiment, the receiver <NUM> and the transmitter <NUM> are combined into a transceiver. The mobile device <NUM> may further comprise a camera (not shown). This camera may comprise a CMOS or CCD sensor, for example. The mobile device <NUM> may comprise other components typical for a mobile device such as a battery and a power connector. The invention may be implemented using a computer program running on one or more processors.

In the embodiment of <FIG>, lighting devices <NUM>-<NUM> are controlled via the bridge <NUM>. In an alternative embodiment, one or more of lighting devices <NUM>-<NUM> are controlled without a bridge, e.g., directly via Bluetooth. If lighting devices <NUM>-<NUM> are controlled without a bridge, use of wireless LAN access point <NUM> may not be necessary. Mobile device may be connected to the Internet <NUM> via a mobile communication network, e.g., <NUM>, instead of via the wireless LAN access point <NUM>.

A first embodiment of the method of controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content is shown in <FIG>. The audiovisual content comprises an audio portion and a video portion. The audio portion comprises multiple audio channels. The method may be performed by the HDMI module <NUM> of <FIG> or the mobile device <NUM> of <FIG>, for example.

A step <NUM> comprises obtaining audiovisual content. A step <NUM> and a step <NUM> are performed after step <NUM>. Step <NUM> comprises determining a first characteristic of a first audio channel of the multiple audio channels or of an audio object comprised in the audio portion. The first characteristic is indicative of an audio source position. In many audio formats, most of the audio channels are associated with desired speaker position in the room, e.g., front left, front right, center. Some audio formats like Dolby Atmos and DTS:X support the use of audio objects. An audio object is normally associated with a position of the audio object in a virtual 3D space.

A step <NUM> comprises obtaining the position of the first lighting device, e.g., an x/y/z position. This may be done manually, but may also be automated, e.g., via RF-sensing. Step <NUM> further comprises associating, based on the first characteristic determined in step <NUM>, the first audio channel or the audio object with a first lighting device of the plurality of lighting devices, wherein the associating is based on the audio source position relative to the position of the first lighting device.

Step <NUM> comprises determining a second characteristic of a second audio channel of the multiple audio channels. A step <NUM> comprises associating, based on the second characteristic determined in step <NUM>, the second audio channel with the first lighting device and with a second lighting device of the plurality of lighting devices. The first audio channel is not associated with the second lighting device. For example, a low frequency effects (abbreviated as LFE) channel may be associated with all lighting devices in a room or a left audio channel (at listener level or at height level) may be associated with multiple lighting devices on the left side of the room.

In the embodiment of <FIG>, steps <NUM> and <NUM> are performed at least partly in parallel with steps <NUM> and <NUM>. In an alternative embodiment, step <NUM> is performed after step <NUM> or step <NUM> is performed after step <NUM>.

A step <NUM> is performed after steps <NUM> and <NUM> have been completed. Step <NUM> comprises determining whether second audio content in the second audio channel meets one or more predetermined criteria. For example, step <NUM> may comprise determining whether an audio intensity of the second audio content exceeds a threshold. Next, a step <NUM> comprises determining at least a chromaticity (and optionally the entire color) based on the video portion of the audiovisual content.

A step <NUM> comprises determining a first light effect based on the determined chromaticity and based on a light intensity. If the one or more predetermined criteria are not met, the light intensity of the first light effect is based on first audio content in the first audio channel. If the one or more predetermined criteria are met, the light intensity of the first light effect is based on the second audio content in the second audio channel.

Additionally, the light intensity of the first light effect may depend on the distance between a speaker (or a position of an audio object, e.g., rendered using multiple speakers) and the lighting device that renders the first light effect. In this case, if two lighting devices are located on the left, for example, but one is farther away from the left channel speaker(s), the adjustment for the lighting device farther away may be less than for the one that is closer. A step <NUM> comprises controlling the first lighting device to render the first light effect determined in step <NUM>.

A second embodiment of the method of controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content is shown in <FIG>. The audiovisual content comprises an audio portion and a video portion. The audio portion comprises multiple audio channels. The method may be performed by the HDMI module <NUM> of <FIG> or the mobile device <NUM> of <FIG>, for example.

Step <NUM> comprises obtaining audiovisual content. In a step <NUM>, a mapping from audio channel to lighting device is determined. First, a characteristic of each audio channel is determined. Certain audio channels are associated with an audio source position and in this case, the determined characteristic is indicative of this audio source position. For example, a front left channel in a Dolby Digital-encoded audio portion is associated with a desired front left speaker position. However, not all audio channels are associated with an audio source position. An example is the LFE (subwoofer) channel.

Step <NUM> comprises determining the positions, e.g., x/y/z positions, of all lighting devices of the plurality of lighting devices. This may be done manually, but may also be automated e.g., via RF-sensing. The characteristic determined for the LFE audio channel (also referred to in this embodiment as the second audio channel) indicates that it is an LFE channel and is not indicative of an audio source position, because humans are not able to locate the source of low frequency sounds. The LFE audio channel is therefore associated with all lighting devices of the plurality of lighting devices in step <NUM>.

The other audio channels (also referred to in this embodiment as the first audio channels) are associated with lighting devices based on the audio source position associated with the respective audio channel and the position of the respective lighting device. For example, the front left audio channel may be associated with a front left lighting device. The type and capability of a lighting device may influence how the mapping between audio channel and lighting device is made. Furthermore, the type and capability of a lighting device may also influence how the brightness and chromaticity are determined for this lighting device in step <NUM>. For example, a point light source may be treated differently from a linear light source like a light strip.

Step <NUM> comprises determining whether second audio content in the second audio channel, i.e., the LFE audio channel, meets one or more predetermined criteria. In the embodiment of <FIG>, step <NUM> comprises determining whether an audio intensity of the second audio content exceeds a threshold.

Next, the light effects are determined for the plurality of lighting devices in step <NUM>. A chromaticity is determined for each of the light effects based on the video portion of the audiovisual content. Moreover, a light intensity is determined for each of the light effects. The chromaticity is extracted from a certain spatial region of the video frames of the video portion. In this embodiment, this spatial region depends on the position of the lighting device. For example, a chromaticity for a light effect to be rendered by a lighting device on the left is extracted from a region on the left side of the video frames and a chromaticity for a light effect to be rendered by a lighting device on the right is extracted from a region on the right side of the video frames.

In the embodiment of <FIG>, the light intensity of the light effects is based only on the audio portion of the audiovisual content. In an alternative embodiment, the light intensity of the light effects is also based on the video portion of the audiovisual content. For example, an intensity may be extracted from the same spatial region from which the chromaticity is extracted, and this intensity may then be adjusted based on the audio portion. The adjusted intensity is then used as the light effect's light intensity.

If it was determined in step <NUM> that the audio intensity in the second audio channel, i.e., the LFE audio channel, did not exceed the threshold, the light intensity of a light effect for a certain lighting device is determined based on the first audio content in the first audio channel associated with that lighting device. For example, the light intensity for a front left lighting device is then determined based the audio content in the front left audio channel.

If it was determined in step <NUM> that the audio intensity in the second audio channel, i.e., the LFE audio channel, exceeded the threshold, the light intensity of each light effect of each lighting device is determined based on the second audio content in the second audio channel. In the embodiment of <FIG>, the light intensity is only based on the second audio content in this case. In an alternative embodiment, the light intensity of a light effect for a certain lighting device is determined based on both the first audio content in the first audio channel associated with that lighting device and the second audio content in the second audio channel, i.e., the LFE audio channel.

Next, step <NUM> comprises controlling the lighting devices to render the light effects determined in step <NUM>. Step <NUM> is repeated after step <NUM>, and the method then proceeds as shown in <FIG>. Since the characteristics of the audio channels normally do not change during the audiovisual content, step <NUM> is not repeated (for the same audiovisual content) in this embodiment. In the embodiment of <FIG>, the audiovisual content is entirely obtained before the light effects are determined. In an alternative embodiment, the audiovisual content may be streamed and thus obtained in parts. In this alternative embodiment, step <NUM> may be performed after the first part of the audiovisual content has been obtained, for example.

A third embodiment of the method of controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content is shown in <FIG>. The audiovisual content comprises an audio portion and a video portion. The audio portion comprises multiple audio channels. The method may be performed by the HDMI module <NUM> of <FIG> or the mobile device <NUM> of <FIG>, for example.

Step <NUM> comprises obtaining audiovisual content. In a step <NUM>, a mapping from audio channel to lighting device is determined. Step <NUM> of <FIG> is similar to step <NUM> of <FIG> except that associating the LFE audio channel with all lighting devices of the plurality of lighting devices is optional. For example, the LFE audio channel may not be associated with any of the lighting devices. If the LFE audio channel is not associated with all lighting devices of the plurality of lighting devices, the LFE audio channel is not treated as a second audio channel.

If the LFE audio channel is not associated with all lighting devices of the plurality of lighting devices, then one or more other audio channels are associated with multiple lighting devices. These one or more other audio channels are then treated as second audio channels. In this case, one or more second characteristics indicative of respective desired speaker positions are determined for the one or more second audio channels.

For example, a front left audio channel may be associated with two lighting devices on the left of the room. When the audio portion comprises both a front left audio channel and a surround left audio channel, the front left audio channel may be mapped to a front left lighting device and the surround left audio channel may be mapped to a front rear lighting device or both audio channels may be mapped to both lighting devices. The same principle may be used for right audio channels and applies when the audio portion comprises rear audio channels and/or height audio channels. A left audio channel and a right audio channel are preferably not mapped to the same lighting device.

Optionally, both the LFE audio channel and the above-mentioned one or more other audio channels may be treated as second audio channels if the LFE audio channel is associated with all lighting devices of the plurality of lighting devices.

In a step <NUM>, a mapping from audio object to lighting device is determined. For example, an audio object may represent a plane that flies from left to right and may be mapped to different lighting devices depending on its position. A first characteristic indicative of a current audio source position is determined of the audio object.

Step <NUM> comprises determining whether second audio content in the second audio channel meets one or more predetermined criteria. If there is more than one second audio channel, this may be done for each second audio channel. In the embodiment of <FIG>, step <NUM> comprises determining whether an audio intensity of the second audio content exceeds a threshold.

Next, the light effects are determined for the plurality of lighting devices in a step <NUM>. A chromaticity is determined for each of the light effects based on the video portion of the audiovisual content, as described in relation to step <NUM> of <FIG>. Moreover, a light intensity is determined for each of the light effects.

In the embodiment of <FIG>, the light intensity of the light effects is determined (in step <NUM>) based on both the audio portion and the video portion of the audiovisual content. The intensity is extracted from the same spatial region from which the chromaticity is extracted, and this intensity is then adjusted based on the audio portion. The adjusted intensity is then used as the light effect's light intensity.

If the second audio channel is not (just) the LFE audio channel, then in step <NUM>, it is determined for each respective lighting device which respective second audio channel has been associated with the respective lighting device, if any. If a lighting device was not associated with a second audio channel in step <NUM> and an audio object was not associated with the lighting device in step <NUM>, then the light intensity is not adjusted. If a lighting device was not associated with a second audio channel in step <NUM> and an audio object was associated with the lighting device in step <NUM>, then the light intensity is adjusted based only on the first audio content in the audio object.

If a lighting device was associated with a second audio channel in step <NUM> and it was determined in step <NUM> that the audio intensity in the second audio channel did not exceed the threshold, then the light intensity of a light effect for the lighting device is not adjusted based on the second audio content in this second audio channel. If the lighting device was associated with an audio object in step <NUM>, then the light intensity is adjusted based on the first audio content in the audio object.

If a lighting device was associated with a second audio channel in step <NUM> and it was determined in step <NUM> that the audio intensity in the second audio channel exceeded the threshold, then the light intensity of a light effect for the lighting device is adjusted based on the second audio content in this second audio channel. In this case, if the lighting device has been associated with an audio object, then the light intensity is further adjusted based on the first audio content in the audio object in the embodiment of <FIG>. In an alternative embodiment, the light intensity is adjusted based only on the second audio content in this second audio channel in this case.

Optionally, step <NUM> comprises determining the light intensities of the light effects further based on information on available speakers and/or information on used three-dimensional audio virtualization. For example, if a user only has front speakers and a center speaker and his audio system does not support three-dimensional audio virtualization, it may be better not to adjust the light intensity of a light effect rendered on a lighting device in the rear of a room based on audio content of a first audio channel or audio object with an audio source position in the rear of the room, as this would create a contradiction between the rendered light effects and the rendered audio.

Step <NUM>, described in relation to <FIG>, is performed after step <NUM>. Step <NUM> is repeated after step <NUM>, after which the method proceeds as shown in <FIG>. Since the characteristics of the audio channels normally do not change during the audiovisual content, unlike the characteristics of the audio objects, step <NUM> is not repeated in this embodiment. In the embodiment of <FIG>, the audiovisual content is entirely obtained before the light effects are determined. In an alternative embodiment, the audiovisual content may be streamed and thus obtained in parts. In this alternative embodiment, step <NUM> may be performed after the first part of the audiovisual content has been obtained, for example.

<FIG> shows an example of a room <NUM> in which five entertainment lighting devices <NUM>-<NUM> have been installed. Lighting device <NUM> has been installed behind display device <NUM>. Lighting device <NUM> has been installed left of display device <NUM>. Lighting device <NUM> has been installed right of display device <NUM>. Lighting devices <NUM>-<NUM> have been installed at the front of the room. Lighting devices <NUM>-<NUM> have been installed at the rear of the room. Lighting device <NUM> has been installed left of a couch <NUM>. Lighting device <NUM> has been installed right of the couch <NUM>.

Video content <NUM> comprises a video portion <NUM> and an audio portion <NUM>. In this example, the audio portion <NUM> comprises six audio channels (<NUM> audio channels to be precise): a surround left channel, a front left channel <NUM>, a center channel, a front right channel, a surround right channel, and a low frequency effects channel <NUM>. In an alternative example, the audio portion <NUM> may comprise more or less than six audio channels. The audio portion further comprises two audio objects: a first audio object <NUM> and a second audio object <NUM>. In practice, an audio portion which comprises audio objects will comprise more than two audio objects.

In a first usage example, the method of <FIG> is used, and the surround left channel, front left channel <NUM>, center channel, front right channel, and the surround right channel are mapped to lighting devices <NUM>,<NUM>,<NUM>, <NUM>, and <NUM>, respectively. These audio channels are treated as first audio channels. Additionally, the low frequency effects channel <NUM> is associated with all the lighting devices <NUM>-<NUM>. The low frequency effects channel <NUM> is treated as second audio channel. When there is a loud effect on the low frequency effects channel <NUM>, the light intensity of the light effects rendered on lighting devices <NUM>-<NUM> is relatively high.

In a second usage example, the method of <FIG> is used, and the audio object <NUM> is rendered at a virtual source position <NUM>. Like in the first usage example, the low frequency effects channel <NUM> is associated with all the lighting devices <NUM>-<NUM>. The low frequency effects channel <NUM> is treated as a second audio channel. When there is a loud effect on the low frequency effects channel <NUM>, the light intensity of the light effects rendered on lighting devices <NUM>-<NUM> is relatively high.

In this case, the light effect rendered by the lighting device nearest to the virtual source position <NUM>, i.e., lighting device <NUM>, may be even higher than the light effect rendered by the other lighting devices. When there is no loud effect on the low frequency effects channel <NUM>, only the light intensity of the light effect rendered by the lighting device nearest to the virtual source position <NUM> is relatively high, and not the light intensities of the light effects rendered by the other lighting devices.

In a third usage example, the method of <FIG> is used, and the audio object <NUM> is rendered at a virtual source position <NUM>. The surround left channel and the front left channel <NUM> are combined and the combined left channel is associated with both lighting device <NUM> and lighting device <NUM>. Furthermore, the surround right channel and the front right channel are combined, and the combined right channel is associated with both lighting device <NUM> and lighting device <NUM>. These combined audio channels are treated as second audio channels. Alternatively, the surround channels may be absent, and the front left channel and front right channel are then treated as second audio channels.

When there is a loud effect on the combined left audio channel, the light intensity of the light effects rendered on lighting devices <NUM> and <NUM> is relatively high. In this case, the light effect rendered by the lighting device nearest to the virtual source position <NUM>, i.e., lighting device <NUM>, may be even higher than the light effect rendered by the other lighting device, i.e., lighting device <NUM>. When there is no loud effect on the combined left audio channel, only the light intensity of the light effect rendered by the lighting device nearest to the virtual source position <NUM>, i.e., lighting device <NUM>, is relatively high, and not the light intensity of the light effect rendered by lighting device <NUM>.

A fourth embodiment of the method of controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content is shown in <FIG>. The fourth embodiment is an extension of the first embodiment of <FIG>. In the embodiment of <FIG>, step <NUM> of <FIG> is implemented by a step <NUM> and a step <NUM> is performed between steps <NUM> and <NUM>.

Step <NUM> comprises selecting a spatial region in a current frame of the video portion in dependence on whether the one or more predetermined criteria are met, as determined in step <NUM>. Step <NUM> comprises extracting the chromaticity from (only) the spatial region selected in step <NUM>. If an intensity is also extracted from the video portion, as described for example in relation to <FIG>, then this intensity is also extracted only from (only) the selected spatial region.

As an example, when the loudness of the LFE channel does not exceed a threshold, a spatial region on the left of the video frames is selected for a front left lighting device. When the loudness of the LFE channel exceeds the threshold, a spatial region in the center of the video frames is selected for the front left lighting device.

A fifth embodiment of the method of controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content is shown in <FIG>. The fifth embodiment is an extension of the first embodiment of <FIG>. In the embodiment of <FIG>, step <NUM> of <FIG> is implemented by step <NUM>, like in the embodiment of <FIG>. Furthermore, in the embodiment of <FIG>, step <NUM> is implemented by a step <NUM> and steps <NUM> and <NUM> are performed between steps <NUM> and <NUM>.

Step <NUM> comprises determining whether an audio intensity of the second audio content in the second audio channel exceeds a threshold. Step <NUM> comprises determining whether an audio intensity of the first audio content in the first audio channel exceeds a further threshold, which may be the same as the threshold.

Steps <NUM> comprises selecting a spatial region in a current frame of the video portion in dependence on whether the audio intensity of the first audio content exceeds the further threshold, as determined in step <NUM>, and optionally also in dependence on whether the audio intensity of the second audio content exceeds the threshold, as determined in step <NUM>. Step <NUM> comprises extracting the chromaticity from (only) the spatial region selected in step <NUM>.

With the method of <FIG>, the light intensity of the light effects (of all lighting devices) is relatively high if there is a loud event on the LFE audio channel and the chromaticity of a light effect rendered on a certain lighting device depends on whether there is loud event on the first audio channel associated with this lighting device. For example, the loudness of the first audio channel may control if the chromaticity for light effects is taken from the part of the screen assigned to it (e.g., left) or from the screen center. This is shown in <FIG>. Optionally, the louder the sound effect, the more color may be taken from the screen center.

<FIG> shows an example of lighting devices being controlled with the method of <FIG> when the second audio channel is loud and the first audio channel(s) are not loud. In this case, the light intensity of the light effects rendered by the lighting devices <NUM>-<NUM> is relatively high and the chromaticity to be used for the light effects rendered by lighting devices <NUM>,<NUM>, and <NUM> is extracted from spatial regions <NUM>, <NUM>, and <NUM>, respectively.

<FIG> shows an example of lighting devices being controlled with the method of <FIG> when both the second audio channel and the first audio channel(s) are loud. In this case, the light intensity of the light effects rendered by the lighting devices <NUM>-<NUM> is also relatively high. However, the chromaticity to be used for the light effects rendered by lighting devices <NUM>,<NUM>, and <NUM> is extracted only from spatial region <NUM>.

A sixth embodiment of the method of controlling a plurality of lighting devices to render light effects accompanying a rendering of audiovisual content is shown in <FIG>. The sixth embodiment is an extension of the first embodiment of <FIG>. In the embodiment of <FIG>, step <NUM> of <FIG> is implemented by a step <NUM>, step <NUM> of <FIG> is implemented by a step <NUM>, and a step <NUM> is performed after step <NUM> and before steps <NUM> and <NUM>. Furthermore, step <NUM> is implemented by a step <NUM> and step <NUM> is implemented by a step <NUM>.

Step <NUM> comprises determining one or more speaker signals for one or more loudspeakers based on the audio portion of the audiovisual content obtained in step <NUM>. In steps <NUM>, the first characteristic of the first audio channel or the audio object is determined based on the one or more speaker signals determined in step <NUM>. In this case, the first characteristic is indicative of a speaker position associated with the first audio channel or the audio object.

In many cases, the audio source position specified by the audio portion is the same as the rendered audio source position. However, there are a few exceptions, including:.

In the embodiment of <FIG>, for the sake of consistency, the second characteristic is also determined based on the one or more speaker signals in step <NUM>, and it is also determined in step <NUM> whether the audio content in the second audio channels meets the one or more predetermined criteria based on the one or more speaker signals. In step <NUM>, the light intensity of the first light effect is determined based on the one or more speaker signals.

The embodiments of <FIG> and <FIG> have been described as an extension of <FIG>. The embodiments of <FIG> and <FIG> may be extended in a similar manner.

<FIG> depicts a block diagram illustrating an exemplary data processing system that may perform the method as described with reference to <FIG>, <FIG>, and <FIG>.

The processing system <NUM> may also be able to use memory elements of another processing system, e.g., if the processing system <NUM> is part of a cloud-computing platform.

Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, a microphone (e.g., for voice and/or speech recognition), or the like.

Claim 1:
A system (<NUM>,<NUM>) for controlling a plurality of lighting devices (<NUM>-<NUM>) to render light effects accompanying a rendering of audiovisual content (<NUM>), said audiovisual content (<NUM>) comprising an audio portion (<NUM>) and a video portion (<NUM>), said audio portion (<NUM>) comprising multiple audio channels, said system (<NUM>,<NUM>) comprising:
at least one input interface (<NUM>,<NUM>);
at least one transmitter (<NUM>,<NUM>); and
at least one processor (<NUM>,<NUM>) configured to:
- obtain said audiovisual content (<NUM>) via said at least one input interface (<NUM>,<NUM>),
- determine a first characteristic of a first audio channel (<NUM>) of said multiple audio channels or of an audio object comprised in said audio portion (<NUM>), said first characteristic being indicative of an audio source position,
- associate, based on said first characteristic, said first audio channel (<NUM>) or said audio object with a first lighting device (<NUM>) of said plurality of lighting devices (<NUM>-<NUM>), wherein said associating is based on said audio source position relative to a position of said first lighting device (<NUM>),
- determine a second characteristic of a second audio channel (<NUM>) of said multiple audio channels,
- associate, based on said second characteristic, said second audio channel (<NUM>) with said first lighting device (<NUM>) and with a second lighting device (<NUM>) of said plurality of lighting devices (<NUM>-<NUM>), said first audio channel (<NUM>) not being associated with said second lighting device (<NUM>),
- determine whether second audio content in said second audio channel (<NUM>) meets one or more predetermined criteria,
- determine at least a chromaticity based on said video portion (<NUM>) of said audiovisual content (<NUM>),
- determine a first light effect based on said determined chromaticity, wherein if said one or more predetermined criteria are not met, the light intensity of said first light effect is based on first audio content in said first audio channel (<NUM>) or in said audio object, and if said one or more predetermined criteria are met, the light intensity of said first light effect is based on said second audio content in said second audio channel (<NUM>), and
- control, via said at least one transmitter (<NUM>,<NUM>), said first lighting device (<NUM>) to render said first light effect.