Patent Description:
The invention further relates to a method of controlling one or more light sources to render light effects determined based on characteristics of audio content while said audio content is being rendered on an audio rendering device.

A dynamic lighting system can dramatically influence the experience and impression of audio-visual material, e.g., when the colors sent to the lights match what would be seen in the composed environment around the screen. However, a dynamic lighting system cannot only be used to enhance screen content, but also to enhance the experience of listening to music, e.g., by using a software algorithm to analyze an audio stream in real-time and create light effects based on certain audio characteristics such as intensity and frequency bands.

An alternative approach is to preprocess music and extract relevant meta data and translate this to a light script specifying light effects. Some of the streaming services offer such metadata. For example, Spotify has a meta data for each song, that includes different audio properties and can be accessed via the Spotify API. The advantage of using metadata for light effects creation is that it does not require access to the audio stream and allows analysis of the data of the complete song instead of relying on the real-time data.

When light effects are used to enhance audio on connected luminaires, it is important that the light effects are in sync with the audio. Especially when e.g., Bluetooth speakers are used, latencies larger than <NUM> may be introduced. Unfortunately, a difference of <NUM> can be enough to be noticeable and may negatively impact the experience. For example, light effects may be intended to be rendered at the same time as auditory effects in an audio segment and depending on the overall latency of the audio segment, it becomes ambiguous whether an auditory effect 'belongs' to a first or a second light effect.

<CIT> discloses a lighting control device for controlling a lighting device depending on music playback, with a lighting device information storage unit which stores lighting device information, including light emission response time of the lighting device, a lighting device selection unit which selects a lighting device to be controlled, and a light emission timing adjustment unit which uses the light emission response time of the lighting device selected by the lighting device selection unit to adjust light emission timing of the selected lighting device.

Ideally, the system would sync light and audio to provide an optimal user experience. However, this is not always possible. <CIT> describes a solution in case this is not possible. <CIT> describes a device and method that improve the light experience when a variation in delay of the audio segment would affect the light experience. The device and method achieve this by selecting light effects based on information indicating or affecting a variation in delay. This makes it possible to skip light effects that are sensitive to variations in delay. However, skipping light effects may also degrade the user experience somewhat.

It is a first object of the invention to provide a system, which can be used to reduce the impact of a delay between light and audio rendering with no or minimal skipping of light effects.

It is a second object of the invention to provide a method, which can be used to reduce the impact of a delay between light and audio rendering with no or minimal skipping of light effects.

In a first aspect of the invention, a system for controlling one or more light sources to render light effects determined based on characteristics of audio content while said audio content is being rendered on an audio rendering device comprises at least one input interface, at least one output interface, and at least one processor configured to determine, based on input received via said at least one input interface, whether a latency between said one or more light sources rendering said light effects and said audio rendering device rendering a corresponding portion of said audio content will likely exceed a threshold, determine a degree of smoothing based on whether said latency will likely exceed said threshold, said degree of smoothing being higher if said latency will likely exceed said threshold than if said latency will likely not exceed said threshold, determine said light effects based on said characteristics of said audio content while applying smoothing according to said determined degree of smoothing, and control, via said at least one output interface, said one or more light sources to render said light effects.

By increasing the degree of smoothing if the latency between the light and audio rendering will likely (i.e., is expected/estimated to) exceed a certain threshold, this latency, i.e., the light effects being out of sync, may be 'masked' with no or minimal skipping of light effects. Increased smoothing will result in a more 'smeared-out' effect on the light source(s), where the precise on- and offset of a light event is not as clearly distinguishable anymore. Thus, increased smoothing will serve to 'mask' the effects of latency.

The latency between the light and audio rendering may be determined to likely exceed a certain threshold when there is a certain amount of uncertainty in the latency. This could be when the amount of latency cannot be determined automatically or a user does not give an indication of the latency (e.g., a user does not want to fiddle with a latency slider and just wants the system to solve it), for example. Said at least one processor may be configured to determine whether said latency will likely exceed said threshold based on a type of said audio rendering device and/or a user specified latency and/or characteristics of an audio system, for example. Said audio system comprises said audio rendering device.

Said at least one processor may be configured determine an estimate of said latency. The estimate may be determined based on the above-mentioned type of the audio rendering device and/or user specified latency and/or characteristics of the audio system, for example. Alternatively, the at least one processor may be configured to determine whether the latency will likely exceed the threshold without first determining an estimate of the latency, e.g., directly based on system characteristics. As an example of the former, streaming over Bluetooth may be associated with an estimated latency of <NUM> milliseconds. As an example of the latter, streaming over Bluetooth may be associated with the latency likely exceeding the threshold.

Said at least one processor may be configured to determine said degree of smoothing according to a smoothing function which uses said estimate of said latency as input. This allows more smoothing to be applied if the threshold is exceeded by a larger amount (preferably, up to a maximum).

Said at least one processor may be configured to apply said smoothing according to said determined degree of smoothing by determining a fade-in duration and/or a fade-out duration for said light effects based on said determined degree of smoothing. This is a beneficial way of realizing smoothing.

Said at least one processor may be configured to determine said fade-in duration of a light effect further based on a distance between a color and/or intensity of said light effect and a color and/or intensity of the preceding light effect and/or determine said fade-out duration of a light effect based on a distance between said color and/or intensity of said light effect and a color and/or intensity of the succeeding light effect. For example, when the light is already on (e.g., <NUM>% light intensity) and a light effect needs to be rendered for an event at <NUM>% light intensity, it would be beneficial to use a different smoothing profile than when the light is off and a light effect needs to be rendered for an event at <NUM>% light intensity. In the former case, less smoothing would be beneficial. In the latter case, more smoothing would be beneficial.

Said at least one processor may be configured to determine, for a period of a plurality of consecutive periods of said audio content, a quantity of light effects to be rendered during said period, said consecutive periods having a predefined duration, and determine said degree of smoothing for said light effects to be rendered during said period based on whether said latency will likely exceed said threshold and further based on said quantity of light effects determined for said period. When the amount of events exceeds the given threshold, it normally does not make sense to increase smoothing, since in this case the audiovisual mismatch will not be apparent. Examples of such a threshold are <NUM> or <NUM> events per second.

Said at least one processor may be configured to determine said light effects to be rendered during said period in dependence on a user-selected dynamicity level. A higher user-selected dynamicity level typically results in more light effects being rendered. A user may be able to select a dynamicity preset of subtle, medium, high, or intense, for example. When the dynamic preset is intense, smoothing has less benefit. In this case, the number of events is relatively high and the above-mentioned threshold will be exceeded relatively quickly.

Said at least one processor may be configured to determine said degree of smoothing further based on whether said latency will likely not exceed a maximum, said degree of smoothing being higher if said latency will likely exceed said threshold and will likely not exceed said maximum than if said latency will likely exceed said maximum. If the latency is too high, then it is normally not possible to counteract the effects of the latency by using additional smoothing. The maximum may be <NUM> milliseconds, for example.

Said at least one processor may be configured to determine for at least one of said light effects whether said at least one light effect relates to a key event in said audio content and increase an intensity of said at least one light effect. This ensures that although smoothing is increased, the key event will still 'pop' with respect to the rest of the audio content.

Said one or more light sources may comprise a plurality of light sources and said at least one processor may be configured to control said plurality of light sources to alternately render said light effects such that said light effects are distributed over said plurality of light sources. Thus, light events may be distributed over the light sources as well as being smoothed. For example, for a part of a song containing four events per second, the light events may be 'split' and rendered alternating between two connected lamps. Not only does this mask potential out-of-sync issues, but it also provides more room for smoothing.

Said at least one processor may be configured to determine that said latency will likely exceed said threshold when a user specifies a latency larger than a further threshold. If the user has specified a latency which exceeds a realistic threshold (e.g., <NUM> seconds), the specified latency may be considered inaccurate, but it may further be considered that the user is negatively impacted by latency and additional smoothing is therefore needed.

Said at least one processor may be configured to determine whether a user-specified latency value exceeds said threshold and determine said degree of smoothing further based on whether said user-specified latency value is larger than a further threshold. For example, when the user-specified latency value is larger than the further threshold, a degree of smoothing may be determined that is larger than just proportional to the user-specified latency value. The rationale behind this is that large latencies are difficult to detect, so when the user has to manually indicate the latency, there is a bigger chance of user error.

In a second aspect of the invention, a method of controlling one or more light sources to render light effects determined based on characteristics of audio content while said audio content is being rendered on an audio rendering device comprises determining, based on received input, whether a latency between said one or more light sources rendering said light effects and said audio rendering device rendering a corresponding portion of said audio content will likely exceed a threshold, determining a degree of smoothing based on whether said latency will likely exceed said threshold, said degree of smoothing being higher if said latency will likely exceed said threshold than if said latency will likely not exceed said threshold, determining said light effects based on said characteristics of said audio content while applying smoothing according to said determined degree of smoothing, and controlling said one or more light sources to render said light effects. Said method may be performed by software running on a programmable device. This software may be provided as a computer program product.

A non-transitory computer-readable storage medium stores at least one software code portion, the software code portion, when executed or processed by a computer, being configured to perform executable operations for controlling one or more light sources to render light effects determined based on characteristics of audio content while said audio content is being rendered on an audio rendering device.

The executable operations comprise determining, based on received input, whether a latency between said one or more light sources rendering said light effects and said audio rendering device rendering a corresponding portion of said audio content will likely exceed a threshold, determining a degree of smoothing based on whether said latency will likely exceed said threshold, said degree of smoothing being higher if said latency will likely exceed said threshold than if said latency will likely not exceed said threshold, determining said light effects based on said characteristics of said audio content while applying smoothing according to said determined degree of smoothing, and controlling said one or more light sources to render said light effects.

<FIG> shows a first embodiment of the system for controlling one or more light sources to render light effects determined based on characteristics of audio content while the audio content is being rendered on an audio rendering device. In this first embodiment, the system is a mobile device <NUM>. Mobile device <NUM> is able to control lighting devices <NUM>-<NUM> via a wireless LAN access point <NUM> and a bridge <NUM>, and optionally via an Internet server <NUM>, e.g., of the manufacturer of the lighting devices <NUM>-<NUM>. The lighting devices <NUM>-<NUM> may be Philips Hue lamps, for example. The lighting devices <NUM>-<NUM> may each comprise one or more LED elements, for example. The lighting devices <NUM>-<NUM> communicate with the bridge <NUM>, e.g., using Zigbee technology. The bridge <NUM> may be a Philips Hue bridge, for example. The bridge <NUM> is connected to the wireless LAN access point <NUM>, e.g., via Wi-Fi or Ethernet.

Mobile device <NUM> is able to control playback of audio content, e.g., songs, via an Internet server <NUM>, e.g., of a music streaming service such as Spotify. Mobile device <NUM> is able to start and stop playback of audio content available in the music library of the music streaming service. In the example of <FIG>, music is streamed to an audio rendering device <NUM>, e.g., a smart and/or Wi-Fi speaker system or an A/V receiver. The audio rendering device <NUM> is connected to the wireless LAN access point <NUM> and may stream music directly from the Internet server <NUM> via Wi-Fi. Alternatively, music may be streamed from a music app running on the mobile device <NUM> to the audio rendering device <NUM> via Bluetooth. The wireless LAN access point <NUM> is connected to the Internet <NUM>. The Internet servers <NUM> and <NUM> are also connected to the Internet <NUM>. Instead of single Internet servers, clusters of Internet servers may be used. These clusters may be part of one or more clouds.

The mobile device <NUM> comprises a transceiver <NUM>, a transmitter <NUM>, a processor <NUM>, memory <NUM>, and a touchscreen display <NUM>. The processor <NUM> is configured to determine, based on input received e.g., via the receiver <NUM> or the touchscreen display <NUM>, whether a latency between the lighting devices <NUM>-<NUM> rendering the light effects and the audio rendering device <NUM> rendering a corresponding portion of the audio content will likely exceed a threshold. This determination may be made based on system characteristics (type of connected speakers etc.) or user input (e.g., with a slider indicating the approximate latency), for example.

In the example of <FIG>, if music is streamed via Bluetooth, it is difficult to determine the amount of latency and compensate for latency. In this case, the latency may be considered to likely exceed the threshold. If music is streamed directly from the Internet server <NUM> via Wi-Fi, the latency is normally, for most users, below <NUM> and may be considered to likely not exceed the threshold.

The processor <NUM> is further configured to determine a degree of smoothing based on whether the latency will likely exceed the threshold. The degree of smoothing is higher if the latency will likely exceed the threshold than if the latency will likely not exceed the threshold. The processor <NUM> is further configured to determine the light effects based on the characteristics of the audio content while applying smoothing according to the determined degree of smoothing, and control, via the transmitter <NUM>, the lighting devices <NUM>-<NUM> to render the light effects.

In the embodiment of the mobile device <NUM> shown in <FIG>, the mobile device <NUM> comprises one processor <NUM>. In an alternative embodiment, the mobile device <NUM> comprises multiple processors. The processor <NUM> of the mobile device <NUM> may be a general-purpose processor, e.g., from ARM or Qualcomm or an application-specific processor. The processor <NUM> of the mobile device <NUM> may run an Android or iOS operating system for example. The display <NUM> may comprise an LCD or OLED display panel, for example. The processor <NUM> may use touch screen display <NUM> to provide a user interface, for example. The memory <NUM> may comprise one or more memory units. The memory <NUM> may comprise solid state memory, for example.

The receiver <NUM> and the transmitter <NUM> may use one or more wireless communication technologies, e.g., Wi-Fi (IEEE <NUM>) for communicating with the wireless LAN access point <NUM>, for example. In an alternative embodiment, multiple receivers and/or multiple transmitters are used instead of a single receiver and a single transmitter. In the embodiment shown in <FIG>, a separate receiver and a separate transmitter are used. In an alternative embodiment, the receiver <NUM> and the transmitter <NUM> are combined into a transceiver. The mobile device <NUM> may comprise other components typical for a mobile device such as a battery and a power connector. The invention may be implemented using a computer program running on one or more processors.

In the embodiment of <FIG>, the lighting devices <NUM>-<NUM> are controlled by the mobile device <NUM> via the bridge <NUM>. In an alternative embodiment, one or more of the lighting devices <NUM>-<NUM> are controlled by the mobile device <NUM> without a bridge, e.g. directly via Bluetooth or via the wireless LAN access point <NUM>. Optionally, the lighting devices <NUM>-<NUM> are controlled via the cloud, e.g., via Internet server <NUM>. The lighting devices <NUM>-<NUM> may be capable of receiving and transmitting Wi-Fi signals, for example.

<FIG> shows a second embodiment of the system for controlling one or more light sources to render light effects determined based on characteristics of audio content while the audio content is being rendered on an audio rendering device. In this second embodiment, the system is a computer <NUM>. The computer <NUM> is connected to the Internet <NUM> and acts as a server. The computer <NUM> may be operated by a lighting company, for example. In the embodiment of <FIG>, the computer <NUM> is able to control the lighting devices <NUM>-<NUM> via the wireless LAN access point <NUM> and the bridge <NUM> and able to communicate with the Internet server <NUM> of a music streaming service.

The computer <NUM> comprises a receiver <NUM>, a transmitter <NUM>, a processor <NUM>, and storage means <NUM>. The processor <NUM> is configured to determine, based on input received via the receiver <NUM>, whether a latency between the lighting devices <NUM>-<NUM> rendering the light effects and the audio rendering device <NUM> rendering a corresponding portion of the audio content will likely exceed a threshold. This determination may be made based on system characteristics (type of connected speakers etc.) or user input (e.g., with a slider indicating the approximate latency), for example.

In the embodiment of the computer <NUM> shown in <FIG>, the computer <NUM> comprises one processor <NUM>. In an alternative embodiment, the computer <NUM> comprises multiple processors. The processor <NUM> of the computer <NUM> may be a general-purpose processor, e.g., from Intel or AMD, or an application-specific processor. The processor <NUM> of the computer <NUM> may run a Windows or Unix-based operating system for example. The storage means <NUM> may comprise one or more memory units. The storage means <NUM> may comprise one or more hard disks and/or solid-state memory, for example. The storage means <NUM> may be used to store an operating system, applications and application data, for example.

The receiver <NUM> and the transmitter <NUM> may use one or more wired and/or wireless communication technologies such as Ethernet and/or Wi-Fi (IEEE <NUM>) to communicate with the wireless LAN access point <NUM>, for example. In an alternative embodiment, multiple receivers and/or multiple transmitters are used instead of a single receiver and a single transmitter. In the embodiment shown in <FIG>, a separate receiver and a separate transmitter are used. In an alternative embodiment, the receiver <NUM> and the transmitter <NUM> are combined into a transceiver. The computer <NUM> may comprise other components typical for a computer such as a power connector. The invention may be implemented using a computer program running on one or more processors.

In the embodiment of <FIG>, the computer <NUM> receives data from and transmits data to the lighting devices <NUM>-<NUM> via the bridge <NUM>. In an alternative embodiment, the computer <NUM> receives data from and transmits data to one or more of the lighting devices <NUM>-<NUM> without a bridge.

In the embodiments of <FIG>, the system of the invention comprises a mobile device or a computer (e.g., cloud computer). In an alternative embodiment, the system of the invention is a different device, e.g., a bridge. In the embodiments of <FIG>, the system of the invention comprises a single device. In an alternative embodiment, the system of the invention comprises a plurality of devices.

A first embodiment of the method of controlling one or more light sources to render light effects determined based on characteristics of audio content while the audio content is being rendered on an audio rendering device is shown in <FIG>. The method may be performed by the mobile device <NUM> of <FIG> or the computer <NUM> of <FIG>, for example.

A step <NUM> comprises determining, based on received input, whether a latency between the one or more light sources rendering the light effects and the audio rendering device rendering a corresponding portion of the audio content will likely exceed a threshold. This determination may be made based on system characteristics (type of connected speakers etc.) or user input (e.g., with a slider indicating the approximate latency), for example. The latency may be estimated, but alternatively, this determination is made without first estimating the latency, e.g., based directly on system characteristics.

A step <NUM> comprises determining a degree of smoothing based on whether the latency will likely exceed the threshold. The degree of smoothing is higher if the latency will likely exceed the threshold than if the latency will likely not exceed the threshold. A step <NUM> comprises determining the light effects based on the characteristics of the audio content while applying smoothing according to the determined degree of smoothing. The (additional) smoothing is applied to counteract the effects of the latency. A light effect may be determined for each event in the audio content, for example. Events may be data points which have an audio intensity higher than a threshold, for example. These data points may be included in metadata provided by the music streaming service, e.g., Spotify. A step <NUM> comprises controlling the one or more light sources to render the light effects.

A second embodiment of the method of controlling one or more light sources to render light effects determined based on characteristics of audio content while the audio content is being rendered on an audio rendering device is shown in <FIG>. The second embodiment of <FIG> is an extension of the first embodiment of <FIG>. In the embodiment of <FIG>, step <NUM> of <FIG> is implemented by steps <NUM> and <NUM> and step <NUM> of <FIG> is implemented by a step <NUM>.

Step <NUM> comprises determining an estimate of the latency. Step <NUM> comprises determining whether the estimate of the latency determined in step <NUM> exceeds the threshold. If so, then it is considered that the actual latency will likely exceed the threshold. Step <NUM> comprises determining a degree of smoothing based on whether the latency will likely exceed the threshold. If the estimated latency exceeds the threshold, the degree of smoothing is determined according to a smoothing function which uses the estimate of the latency, determined in step <NUM>, as input.

A third embodiment of the method of controlling one or more light sources to render light effects determined based on characteristics of audio content while the audio content is being rendered on an audio rendering device is shown in <FIG>. The third embodiment of <FIG> is an extension of the first embodiment of <FIG>. In the embodiment of <FIG>, step <NUM> of <FIG> is implemented by steps <NUM>,<NUM>,<NUM>,<NUM>, and <NUM>. The different parameters of the light effects are determined in steps <NUM>, <NUM>, and <NUM>.

Step <NUM> comprises determining the color and intensity of a plurality of light effects based on one or more characteristics of the audio content. Step <NUM> comprises determining, for each light effect, a distance between a color and/or intensity of the light effect and a color and/or intensity of the preceding light effect, based on the results of step <NUM>. Step <NUM> comprises determining a fade-in duration for the light effects based on the determined degree of smoothing, as determined in step <NUM>, and based on the distance determined in step <NUM>.

Step <NUM> comprises determining, for each light effect, a distance between the color and/or intensity of the light effect and a color and/or intensity of the succeeding light effect, based on the results of step <NUM>. Step <NUM> comprises determining a fade-out duration for the light effects based on the determined degree of smoothing, as determined in step <NUM>, and based on the distance determined in step <NUM>.

For example, when the light is already on (e.g., <NUM>% light intensity) and a light effect needs to be rendered for an event at <NUM>% light intensity, it would be beneficial to use a different smoothing profile than when the light is off and a light effects needs to be rendered for an event at <NUM>% light intensity. In the former case, less smoothing would be beneficial. In the latter case, more smoothing would be beneficial. In an alternative embodiment, steps <NUM> and <NUM> have been omitted and the fade-in and fade-out durations are not determined based on these distances in steps <NUM> and <NUM>.

The degree of smoothing is preferably determined such that with higher latencies, the smoothing is gradual enough to make the light intensity peak of the light effect not stand out, in order to mask the latency. A maximum fade-in duration, e.g., <NUM> seconds, and/or a maximum fade-out duration, e.g., <NUM> seconds, may be defined.

A fourth embodiment of the method of controlling one or more light sources to render light effects determined based on characteristics of audio content while the audio content is being rendered on an audio rendering device is shown in <FIG>. The method may be performed by the mobile device <NUM> of <FIG> or the computer <NUM> of <FIG>, for example.

Step <NUM> comprises determining, based on received input, whether a latency between the one or more light sources rendering the light effects and the audio rendering device rendering a corresponding portion of the audio content will likely exceed a threshold. In the embodiment of <FIG>, the audio content is divided into consecutive periods having a predefined duration. For example, the audio content may be divided in chunks of <NUM> seconds or <NUM> seconds. In the first iteration of a step <NUM>, the first period of the audio content is selected.

A step <NUM> comprises determining a quantity of light effects to be rendered during the period selected in step <NUM>. A step <NUM> comprises determining a degree of smoothing for the light effects to be rendered during the period selected in step <NUM> based on whether the latency will likely exceed the threshold and further based on the quantity of light effects determined in step <NUM> for this period.

Preferably, step <NUM> comprises checking whether a particular threshold in number of events is crossed or not and applying additional smoothing only when the amount of events is smaller than a given thershold. When the amount of events exceeds the given threshold, it normally does not make sense to increase smoothing, since in this case the audiovisual mismatch will not be apparent. Examples of such a threshold are <NUM> or <NUM> events per second.

Step <NUM> comprises determining the light effects based on the characteristics of the audio content while applying smoothing according to the degree of smoothing, as determined in step <NUM>. In the embodiment of <FIG>, the light effects are determined in step <NUM> in dependence on a user-selected dynamicity level. A user may be able to select a dynamicity preset of subtle, medium, high, or intense, for example. In the embodiment of <FIG>, the number of events for which a light effect is rendered depends on the characteristics of the audio content (e.g., as specified in metadata) and the user-selected dynamicity level. A higher user-selected dynamicity level results in more light effects being rendered. When the dynamic preset is intense, smoothing has less benefit. In this case, the number of events is relatively high and the above-mentioned threshold will be exceeded relatively quickly. In an alternative embodiment, the light effects are not determined in dependence on a user-selected dynamicity level.

A step <NUM> comprises determining whether a period exists in the audio content that is consecutive to the period last selected in step <NUM>. If so, then this period is selected in the next iteration of step <NUM>, after which the method proceeds as shown in <FIG>. If not, then step <NUM> is performed next. Step <NUM> comprises controlling the one or more light sources to render the light effects determined in the multiple iterations of step <NUM>.

<FIG> shows examples of different degrees of smoothing. Graphs <NUM> and <NUM> represent light intensity over time. The light effects <NUM>-<NUM> shown in graphs <NUM> and <NUM> are determined based on the same audio characteristics and for the same events. In the examples of <FIG>, the light is never completely turned off, but alternatively, the light may be completely turned off at certain moments between light effects.

Graph <NUM> represents a situation when the estimated latency is <NUM> milliseconds, as indicated by indicator <NUM>, and the degree of smoothing, in this case fading, is determined to be normal, as indicated by indicator <NUM>. Graph <NUM> represents a situation when the estimated latency is <NUM> milliseconds, as indicated by indicator <NUM>, and the fading is determined to be twice as long as the default fading, as indicated by indication <NUM>. The fade-in duration(s) and the fade-out duration(s) may be determined with the method of <FIG>, for example.

<FIG> shows that when the fade-in duration(s) and the fade-out duration(s) are increased, the light effects <NUM>-<NUM> transition/blend more smoothly and the peaks stand out less. This is done to mask the latency. With an even higher estimated latency, the effect may be more pronounced, although the benefit may only be present up to a certain maximum latency.

In <FIG>, three periods <NUM>-<NUM> are distinguished. The periods may have a length of <NUM> seconds or <NUM> seconds, for example. The fade-in durations and the fade-out durations may depend on the number of events per period, as described in relation to the method of <FIG>.

A fifth embodiment of the method of controlling one or more light sources to render light effects determined based on characteristics of audio content while the audio content is being rendered on an audio rendering device is shown in <FIG>. The fifth embodiment of <FIG> is an extension of the first embodiment of <FIG>. In the embodiment of <FIG>, step <NUM> of <FIG> is preceded by a step <NUM> and implemented by a step <NUM> and step <NUM> of <FIG> is implemented by steps <NUM>, <NUM>, and <NUM>.

Step <NUM> comprises determining a type of the audio rendering device on which the audio content is rendered and/or a user specified latency and/or characteristics of an audio system that comprises the audio rendering device. Step <NUM> comprises determining whether the latency will likely exceed the threshold based on the type of the audio rendering device and/or the user specified latency and/or the characteristics of the audio system, as determined in step <NUM>.

Step <NUM> comprises determining whether a user-specified latency value exceeds the threshold, e.g., <NUM> milliseconds. If so, step <NUM> is performed. If not step, <NUM> is performed. If the user has specified a latency which exceeds a realistic threshold (e.g., <NUM> seconds), the specified latency may be considered inaccurate, but it may further be considered that the user is negatively impacted by latency and additional smoothing is therefore needed.

Steps <NUM> and <NUM> comprise determining a degree of smoothing based on whether the latency will likely exceed the threshold, as determined in step <NUM>. Step <NUM> comprises determining the degree of smoothing further based on whether the user-specified latency value is larger than a further threshold (e.g., <NUM> milliseconds). In step <NUM>, it is determined that an increased smoothing should be used if a (realistic) latency larger than the further threshold has been specified by the user. The rationale behind this is that large latencies are difficult to detect, so when the user has to manually indicate the latency, there is a bigger chance of user error.

In steps <NUM> and <NUM>, the degree of smoothing is higher if the latency will likely exceed the threshold than if the latency will likely not exceed the threshold. Moreover, in the embodiment of <FIG>, in steps <NUM> and <NUM>, the degree of smoothing is further determined based on whether the latency will likely not exceed a maximum, e.g., <NUM> milliseconds. The degree of smoothing is higher if the latency will likely exceed the threshold and will likely not exceed the maximum than if the latency will likely exceed the maximum. If the latency is too high, then it is normally not possible to counteract the effects of the latency by using additional smoothing.

A sixth embodiment of the method of controlling one or more light sources to render light effects determined based on characteristics of audio content while the audio content is being rendered on an audio rendering device is shown in <FIG>. The sixth embodiment of <FIG> is an extension of the first embodiment of <FIG>. In the embodiment of <FIG>, the one or more light sources comprises a plurality of light sources. Furthermore, step <NUM> of <FIG> is implemented by steps <NUM>, <NUM>, <NUM>, and <NUM> and step <NUM> of <FIG> is implemented by a step <NUM>.

In the first iteration of step <NUM>, step <NUM> comprises determining a first light effects based on one or more characteristics of the audio content while applying smoothing according to the determined degree of smoothing, as determined in step <NUM>. Step <NUM> comprises determining a color and an intensity of the light effect and optionally a fade-in duration and/or a fade-out duration.

Step <NUM> comprises determining whether the light effect determined in step <NUM> relates to a key event. A key event corresponds to a moment where being out of sync is the most noticeable. If the light effect determined in step <NUM> relates to a key event, step <NUM> is performed. If not, step <NUM> is skipped and step <NUM> is performed. Step <NUM> comprises increasing the intensity of the light effect determined in step <NUM>. This ensures that although smoothing is increased, the key event will still 'pop' with respect to the rest of the audio content. Step <NUM> is performed after step <NUM>.

Step <NUM> comprises determining whether all light effects have been determined, e.g., whether there are events for which no light effect has been determined yet. If so, then the next light effect is determined in the next iteration of step <NUM>, and the method proceeds as shown in <FIG>. If not, step <NUM> is performed next.

Step <NUM> comprises controlling the plurality of light sources to alternately render the light effects such that the light effects are distributed over the plurality of light sources. Thus, light events may be distributed over the lamps as well as being smoothed. For example, for a part of a song containing four events per second, the light events may be 'split' and rendered alternating between two connected lamps. Not only does this mask potential out-of-sync issues, but it also provides more room for smoothing.

The embodiments of <FIG> and <FIG> differ from each other in multiple aspects, i.e., multiple steps have been added or replaced. In variations on these embodiments, only a subset of these steps is added or replaced and/or one or more steps is omitted. As a first example, step <NUM> may be omitted from the embodiment of <FIG>. As a second example. Steps <NUM> and <NUM> of <FIG> may be added to the embodiments of <FIG>, <FIG>, <FIG>, <FIG>. As a third example, step <NUM> may be omitted from <FIG> and/or added to one or more of the embodiments of <FIG> and <FIG>. One or more of the embodiments of <FIG> and <FIG> may be combined.

<FIG> depicts a block diagram illustrating an exemplary data processing system that may perform the method as described with reference to <FIG> and <FIG>.

The data processing system may be an Internet/cloud server, for example.

Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, a microphone (e.g., for voice and/or speech recognition), or the like.

In such an embodiment, input to the device may be provided by a movement of a physical object, such as e.g., a stylus or a finger of a user, on or near the touch screen display.

Claim 1:
A system (<NUM>,<NUM>) for controlling one or more light sources (<NUM>-<NUM>) to render light effects determined based on characteristics of audio content while said audio content is being rendered on an audio rendering device (<NUM>), said system (<NUM>,<NUM>) comprising:
at least one input interface (<NUM>,<NUM>);
at least one output interface (<NUM>,<NUM>); and
at least one processor (<NUM>,<NUM>) configured to:
- control, via said at least one output interface (<NUM>, <NUM>), said one or more light sources to render said light effects;
characterised in that said at least one processor (<NUM>, <NUM>) is further configured to:
- determine, based on input received via said at least one input interface (<NUM>,<NUM>), whether a latency between said one or more light sources (<NUM>-<NUM>) rendering said light effects and said audio rendering device (<NUM>) rendering a corresponding portion of said audio content will likely exceed a threshold,
- determine a degree of smoothing based on whether said latency will likely exceed said threshold, said degree of smoothing being higher if said latency will likely exceed said threshold than if said latency will likely not exceed said threshold,
- determine said light effects based on said characteristics of said audio content while applying smoothing according to said determined degree of smoothing, and
wherein said at least one processor (<NUM>,<NUM>) is configured to apply said smoothing according to said determined degree of smoothing by determining a fade-in duration and/or a fade-out duration for said light effects based on said determined degree of smoothing.