Patent Description:
The invention further relates to a method of controlling a lighting device to render light effects while an audio rendering device plays a song.

To create a more immersive experience for a user who is listening to a song being played by an audio rendering device, a lighting device can be controlled to render light effects while the audio rendering device plays the song. In this way, the user can create an experience at home which somewhat resembles the experience of a club or concert, at least in terms of lighting. An example of such a light control system is disclosed in <CIT>.

To create an immersive light experience, the accompanying light effects should match the music in terms of e.g. color, intensity, and/or dynamics. The light effects may be synchronized to the bars and/or beats of the music or even to the rhythm of the music, for example.

To improve the lighting experience, the system disclosed in <CIT> detects whether a section of a song is a note-fractioned section, a level-varying section, or a fill-in section and selects light effects for the section based on the detected type of the section, e.g. light blinking to sixteenth-note striking, light brightness increasing with an increase in sweep sound, or light brightness gradually dropping. However, this system is suboptimal in creating an immersive light experience.

<CIT> discloses an electronic device configured to determine information indicating or affecting a variation in delay, e.g. variations in one or more delay components, between a content frame being rendered, e.g. by a mobile device, and a light effect synchronized to the content frame starting to be rendered. The electronic device is further configured to select one or more light effects to be rendered simultaneously with a content item based on the determined information and cause one or more light sources to render the selected one or more light effects simultaneously with the content item.

It is a first object of the invention to provide a system, which controls a lighting device to render immersive light effects related to a song while an audio rendering device plays the song.

It is a second object of the invention to provide a method, which can be used to control a lighting device to render immersive light effects related to a song while an audio rendering device plays the song.

In a first aspect of the invention, a system for controlling a lighting device to render light effects while an audio rendering device plays a song comprises at least one receiver, at least one transmitter, and at least one processor configured to receive information from an audio streaming service via said at least one receiver, said information being indicative of a median or an average of an audio characteristic for each section of a plurality of sections of said song, determine, from said information, a first median or average of an audio characteristic in a first section of said plurality of sections and a second median or average of said audio characteristic in a second consecutive section of said plurality of sections, determine whether a difference between said first and second medians or averages exceeds a threshold, determine said light effects based on said information, gradually reduce a light intensity and/or color saturation of said light effects during a period before the start of said second section in dependence on said difference exceeding said threshold, and control, via said at least one transmitter, said lighting device to render said light effects.

Thus, when the difference between the first and second medians or averages exceeds the threshold, the system creates an anticipatory fading effect that builds up anticipation until the moment the second section starts, which is considered to be a key moment, and thereby creates an immersive light experience. Said audio characteristic may comprise audio intensity (sometimes referred to as loudness) or frequency, for example. Said period may start between <NUM> and <NUM> seconds before the start of said second section, for example. Said threshold maybe <NUM> dB for audio intensity, for example. Compared to averages, medians are not skewed so much by a small proportion of extremely large or small values (outliers).

By not determining lighting effects for a section based only on an analysis of the section itself, as disclosed in <CIT>, but based on differences between audio characteristics of different sections, immersive light effects may be created that would otherwise not be created.

Said information received from said audio streaming service may comprise data points for said plurality of sections of said song and said at least one processor may be configured to determine said first and second medians or averages from said data points. The system may thus be able to use data provided by an audio streaming service, e.g. Spotify, and not need to analyze the song itself. The system can therefore be relatively simple compared to the system of <CIT>.

The information received from the audio streaming service is indicative of a median or an average of an audio characteristic for each section of a plurality of sections of the song but does not necessarily distinguish between different sections. For example, when the information comprises data points, these data points (e.g. loudness points) may not only be used determine the medians or averages, but may also be used to identify the start and end of each section before determining these medians or averages.

Said at least one processor may be configured to select a subset of said data points, an audio intensity of said selected data points exceeding a further threshold, and determine said light effects based on said selected data points. In other words, not only the differences between the first and second medians or averages, but also the light effects themselves, may be determined from data points obtained from the audio streaming service. Alternatively, the system may analyze the song and determine the light effects based on this analysis or determine the light effects from a received light script.

Said at least one processor may be configured to obtain lighting control limitations set by a user and comply with said lighting control limitations when performing said reduction of said light intensity and/or color saturation. Alternatively, said at least one processor may be configured to obtain lighting control limitations set by a user and ignore said lighting control limitations when performing said reduction of said light intensity and/or color saturation. The lighting control limitations may be specified by user preferences like color palette and/or minimum light intensity (sometimes referred to as brightness).

Said at least one processor may be configured to increase said light intensity and/or color saturation of said light effects at the start of said second section in dependence on said difference exceeding said threshold. The start of the second section is a key moment that is emphasized by the anticipatory fading, but this key moment may be further emphasized by increasing the light intensity and/or color saturation of the light effects at the start of the second section.

Said at least one processor may be configured to increase said light intensity and/or color saturation of said light effects during a further period immediately after the start of said second section in dependence on said difference exceeding said threshold and gradually reduce said increase during said further period. By increasing the light intensity and/or color saturation also for light effects in the further period but gradually reducing the increase until the light effects are rendered at a normal level of light intensity and/or color saturation, the key moment is further emphasized.

Said at least one processor may be configured to obtain lighting control limitations set by a user and ignore said lighting control limitations when performing said increase of said light intensity and/or color saturation. The lighting control limitations may be specified by user preferences like color palette and/or maximum light intensity.

Said at least one processor may be configured to control said lighting device to render a special light effect at the start of said second section in dependence on said difference exceeding said threshold. In this way, the start of the second section is further emphasized following the anticipatory fading. The special light effect may be a flash, for example.

Said at least one processor may be configured to control a plurality of lighting devices to render said light effects at the start of said second section and to control a proper subset of said plurality of lighting devices to render said light effects during said period before the start of said second section in dependence on said difference exceeding said threshold. By using less lighting devices to render the lighting effects during the period, i.e. during the anticipatory fading, the effect of the anticipatory fading is increased. This may be especially beneficial when lighting control limitations need to be complied with and the allowable reduction of the light intensity is relatively small.

Said at least one processor may be configured to determine a level of said reduction of said light intensity and/or color saturation based on said difference between said first and second medians or averages. For example, the difference between the first and second medians or averages may be considered to represent the build-up in the music that leads to the start of the second section and the level of the reduction thus depends on this build-up in the music. Alternatively, the build-up in the music may be determined in a different manner, e.g. by analyzing only a part of the current section and/or based on the approximate duration of the build-up.

In a second aspect of the invention, a method of controlling a lighting device to render light effects while an audio rendering device plays a song comprises receiving information from an audio streaming service, said information being indicative of a median or an average of an audio characteristic for each section of a plurality of sections of said song, determining, from said information, a first median or average of an audio characteristic in a first section of said plurality of sections and a second median or average of said audio characteristic in a second consecutive section of said plurality of sections, determining whether a difference between said first and second medians or averages exceeds a threshold, determining said light effects based on said information, gradually reducing a light intensity and/or color saturation of said light effects during a period before the start of said second section in dependence on said difference exceeding said threshold; and controlling said lighting device to render said light effects. Said method may be performed by software running on a programmable device. This software may be provided as a computer program product.

A non-transitory computer-readable storage medium stores at least one software code portion, the software code portion, when executed or processed by a computer, being configured to perform executable operations for determining a suitability of an input modality for controlling a lighting device to render light effects while an audio rendering device plays a song.

The executable operations comprise receiving information from an audio streaming service, said information being indicative of a median or an average of an audio characteristic for each section of a plurality of sections of said song, determining, from said information, a first median or average of an audio characteristic in a first section of said plurality of sections and a second median or average of said audio characteristic in a second consecutive section of said plurality of sections, determining whether a difference between said first and second medians or averages exceeds a threshold, determining said light effects based on said information, gradually reducing a light intensity and/or color saturation of said light effects during a period before the start of said second section in dependence on said difference exceeding said threshold; and controlling said lighting device to render said light effects.

In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Intemet using an Internet Service Provider).

<FIG> shows a first embodiment of the system for controlling a lighting device to render light effects while an audio rendering device plays a song. In this first embodiment, the system is a mobile device <NUM>. A lighting system comprises a bridge <NUM> and lighting devices <NUM>-<NUM>. Lighting devices <NUM>-<NUM> can be controlled via bridge <NUM>, e.g. using Zigbee technology. Lighting devices <NUM>-<NUM> may be Philips Hue lamps, for example. The bridge <NUM> may be a Philips Hue bridge, for example.

The bridge <NUM> is connected to a wireless LAN access point <NUM>, e.g. via Wi-Fi or Ethernet. The wireless LAN access point <NUM> is connected to the Internet <NUM>. Mobile device <NUM> is able to control lighting devices <NUM>-<NUM> via the wireless LAN access point <NUM> and the bridge <NUM>. Internet server <NUM> is a server of an audio streaming service, e.g. Spotify. The Intemet server <NUM> is also connected to the Internet <NUM>. Instead of a single Internet server, a cluster of Internet servers may be used. This cluster may be part of one or more clouds.

The mobile device <NUM> comprises a transceiver <NUM>, a transmitter <NUM>, a processor <NUM>, memory <NUM>, and a touchscreen display <NUM>. The processor <NUM> is configured to receive, from the Internet server <NUM>, via the receiver <NUM>, information which is indicative of a median or an average of an audio characteristic for each section of a plurality of sections of a song and determine, from the information, a first median or average of an audio characteristic in a first section of the plurality of sections and a second median or average of the audio characteristic in a second consecutive section of the plurality of sections, e.g. for each two consecutive sections of the song. The audio characteristic may comprise audio intensity, for example.

The processor <NUM> is further configured to determine whether a difference between the first and second medians or averages exceeds a threshold, determine the light effects based on the information, gradually reduce a light intensity and/or color saturation of the light effects during a period before the start of the second section in dependence on the difference exceeding the threshold, and control, via the transmitter <NUM>, one or more of the lighting devices <NUM>-<NUM> to render the light effects. The period may start between <NUM> and <NUM> seconds before the start of the second section, for example.

For instance, a user may be able to select a given song for playback using mobile device <NUM>, which will then be accompanied by light effects on one or more of the lighting devices <NUM>-<NUM>. The song may be output on speakers of the mobile device <NUM> (not shown), on headphones/earphones connected to the mobile device <NUM> (now shown), or on a smart speaker system <NUM>, for example.

The light effects may be determined based on the metadata provided by Spotify, consisting of highly detailed audio descriptors. Based on this metadata, and the setup of the user, a light script can be created and then streamed to the lighting device(s). The generated light effects are 'in sync' with the song being played. 'In sync' means that the accompanying light effects match the music in terms of e.g. color, intensity, and/or dynamics, and may involve a temporal sync where key elements in the music are rendered on the lighting devices with a non-noticeable temporal difference. Especially for key moments in the music like a section change, the most immersive experience will be created when the light effects highlight these key moments.

The information received from the audio streaming service may specify the median or average of the audio characteristic per section and/or may comprise data points, e.g. per event. The information may comprise data points for the bars, beats, sections, and/or segments of the song, for example. The processor <NUM> may be configured to determine first and second medians or averages from the data points. An example of an audio streaming service providing such information/metadata is Spotify. Spotify indicates the start and duration of each section of the song. A section may be a verse or chorus, for example. Sections are not labelled and may have different durations. Spotify indicates a loudness per section and indicates an onset loudness, peak loudness, and offset loudness per segment. Segments may correspond to piano notes, for example.

In the embodiment of <FIG>, the information comprises data points and the processor <NUM> is further configured to select a subset of the data points of which the audio intensity (normally expressed in dB) exceeds an audio intensity threshold. This audio intensity threshold may be dynamic. The number of selected data points may be reduced to increase smoothness. If the processor <NUM> determines the first and second medians or averages from the data points, then it determines the first and second medians or averages from all data points in the section and not just from the subset of data points.

The processor <NUM> is further configured to determine the light effects based on the selected data points. In a first implementation, only the light intensity of the light effects depends on the audio intensity specified in the metadata. In this first implementation, to determine a color for the light effect, a random number is picked to determine whether the color should be changed within the color palette or even whether a specific color should be selected within the color palette. The color palette may be chosen by the manufacturer and/or selected based on the genre of the song.

During the anticipatory fading, preferably both light intensity and color saturation are faded out, but it is also possible to fade out only lighting intensity or only color saturation. At the start of the second section, at least the original light intensity and/or color saturation are rendered. At that moment, there is no longer any reduction due to fading.

In the embodiment of the mobile device <NUM> shown in <FIG>, the mobile device <NUM> comprises one processor <NUM>. In an alternative embodiment, the mobile device <NUM> comprises multiple processors. The processor <NUM> of the mobile device <NUM> may be a general-purpose processor, e.g. from ARM or Qualcomm or an application-specific processor. The processor <NUM> of the mobile device <NUM> may run an Android or iOS operating system for example. The display <NUM> may comprise an LCD or OLED display panel, for example. The processor <NUM> may use touch screen display <NUM> to provide a user interface, for example. The memory <NUM> may comprise one or more memory units. The memory <NUM> may comprise solid state memory, for example.

The receiver <NUM> and the transmitter <NUM> may use one or more wireless communication technologies, e.g. Wi-Fi (IEEE <NUM>) for communicating with the wireless LAN access point <NUM>, for example. In an alternative embodiment, multiple receivers and/or multiple transmitters are used instead of a single receiver and a single transmitter. In the embodiment shown in <FIG>, a separate receiver and a separate transmitter are used. In an alternative embodiment, the receiver <NUM> and the transmitter <NUM> are combined into a transceiver. The mobile device <NUM> may comprise other components typical for a mobile device such as a battery and a power connector. The invention may be implemented using a computer program running on one or more processors.

In the embodiment of <FIG>, the lighting devices <NUM>-<NUM> are controlled by the mobile device <NUM> via the bridge <NUM>. In an alternative embodiment, one or more of the lighting devices <NUM>-<NUM> are controlled by the mobile device <NUM> without a bridge, e.g. via the Internet server <NUM> and the wireless LAN access point <NUM> or directly via Bluetooth. The lighting devices <NUM>-<NUM> may be capable of receiving and transmitting Wi-Fi signals, for example.

<FIG> shows a second embodiment of the system for controlling a lighting device to render light effects while an audio rendering device plays a song. In this second embodiment, the system is a computer <NUM>. The computer <NUM> is connected to the Internet <NUM> and acts as a server. The computer <NUM> may be operated by a lighting company, for example. In the embodiment of <FIG>, the computer <NUM> is able to control the lighting devices <NUM>-<NUM> via the wireless LAN access point <NUM> and the bridge <NUM>.

The computer <NUM> comprises a receiver <NUM>, a transmitter <NUM>, a processor <NUM>, and storage means <NUM>. The processor <NUM> is configured to receive, from the Internet server <NUM>, via the receiver <NUM>, information which is indicative of a median or an average of an audio characteristic for each section of a plurality of sections of a song and determine, from the information, a first median or average of an audio characteristic in a first section of the plurality of sections and a second median or average of the audio characteristic in a second consecutive section of the plurality of sections, e.g. for each two consecutive sections of the song.

The processor <NUM> is further configured to determine whether a difference between the first and second medians or averages exceeds a threshold, determine the light effects based on the information, gradually reduce a light intensity and/or color saturation of the light effects during a period before the start of the second section in dependence on the difference exceeding the threshold, and control, via the transmitter <NUM>, one or more of the lighting devices <NUM>-<NUM> to render the light effects.

In the embodiment of the computer <NUM> shown in <FIG>, the computer <NUM> comprises one processor <NUM>. In an alternative embodiment, the computer <NUM> comprises multiple processors. The processor <NUM> of the computer <NUM> may be a general-purpose processor, e.g. from Intel or AMD, or an application-specific processor. The processor <NUM> of the computer <NUM> may run a Windows or Unix-based operating system for example. The storage means <NUM> may comprise one or more memory units. The storage means <NUM> may comprise one or more hard disks and/or solid-state memory, for example. The storage means <NUM> may be used to store an operating system, applications and application data, for example.

The receiver <NUM> and the transmitter <NUM> may use one or more wired and/or wireless communication technologies such as Ethernet and/or Wi-Fi (IEEE <NUM>) to communicate with the wireless LAN access point <NUM>, for example. In an alternative embodiment, multiple receivers and/or multiple transmitters are used instead of a single receiver and a single transmitter. In the embodiment shown in <FIG>, a separate receiver and a separate transmitter are used. In an alternative embodiment, the receiver <NUM> and the transmitter <NUM> are combined into a transceiver. The computer <NUM> may comprise other components typical for a computer such as a power connector. The invention may be implemented using a computer program running on one or more processors.

In the embodiment of <FIG>, the computer <NUM> transmits data to the lighting devices <NUM>-<NUM> via the bridge <NUM>. In an alternative embodiment, the computer <NUM> transmits data to the lighting devices <NUM>-<NUM> without a bridge.

<FIG> shows a graph depicting changes to light intensity over time. <FIG> shows two sections <NUM> and <NUM> of a song and illustrates how the start of section <NUM> is a key moment that is highlighted by dimming down the lights prior to this key moment and returning to the original light intensity when the key moment occurs. The start of section <NUM> has been identified as key moment, because the difference between the first median or average of the audio intensity in section <NUM> and a second median or average of the audio intensity in section <NUM> exceeds a threshold, e.g. <NUM> dB. Changes to audio intensity over time are not shown in <FIG>. In an implementation, the start of section <NUM> is only identified as key moment if the second median or average is higher than the first median or average.

In <FIG>, the line <NUM> indicates the light intensity signal. Segment <NUM> of line <NUM> indicates the original intensity signal. As can be seen, the light intensity peaks at the moment section <NUM> starts, i.e. at moment <NUM>. Even though this peak clearly stands out, it may be accentuated even further by reducing the light intensity in anticipation of this peak. This is illustrated with alternative segment <NUM> of the light intensity signal, which is identical to the original signal, but introduces the anticipatory fading to maximize contrast with the peak event during a period <NUM>.

A first embodiment of the method of controlling a lighting device to render light effects while an audio rendering device plays a song is shown in <FIG>. The method may be performed by the mobile device <NUM> of <FIG> or the cloud computer <NUM> of <FIG>, for example.

A step <NUM> comprises receiving information from an audio streaming service. The information is indicative of a median or an average of an audio characteristic for each section of a plurality of sections of the song. In the embodiment of <FIG>, information indicative of all sections of the song is received in a single iteration of step <NUM>. Therefore, step <NUM> only needs to be performed once per song. In an alternative embodiment, the information is streamed, multiple parts of the information are received during the playback of the song, and step <NUM> is performed multiple times.

A step <NUM> comprises determining light effects based on the information received in step <NUM>. In the embodiment of <FIG>, the light effects for the entire song are determined in a single iteration of step <NUM> and step <NUM> therefore only needs to be performed once per song. In the alternative embodiment mentioned above, step <NUM> may be performed multiple times per song, e.g. after receipt of each part of the information. In a further altemative embodiment, step <NUM> is only performed once per song, but step <NUM> is performed multiple times per song.

A step <NUM> comprises determining, from the information received in step <NUM>, a first median or average of an audio characteristic (e.g. audio intensity) in a first section and a second median or average of the audio characteristic in a second consecutive section. For example, in the first iteration of step <NUM>, this determination is made for the first two sections of the song. In the embodiment of <FIG>, step <NUM> is performed multiple times, once for each section that has a consecutive section (i.e. once for each section except the last one). In an alternative embodiment, the medians or averages of all sections are determined in a single iteration of step <NUM>. The start and end of each section may be indicated in the information received in step <NUM> or may be determined from data points comprised in the information received in step <NUM>, for example.

A step <NUM> comprises determining whether a difference between the first and second medians or averages, as determined in step <NUM>, exceeds a threshold. In the embodiment of <FIG>, step <NUM> is performed per light effect for each of the light effects determined in step <NUM>. In each iteration, step <NUM> is performed for the light effect to be rendered next at a certain moment in the current section of the song. The difference evaluated in step <NUM> represents a difference between the current section (the first section of the consecutive sections) and the next section (the second section of the consecutive sections). If it is determined in step <NUM> that this difference (as determined in the last iteration of step <NUM>) does not exceed the threshold, a step <NUM> is performed directly after step <NUM>.

If it is determined in step <NUM> that this difference exceeds the threshold, a step <NUM> is performed. Before step <NUM> is performed, it may be determined in step <NUM> exactly in which period before the start of the next section the anticipatory fading should be applied. This may involve analyzing the build-up in the music that leads to the start of the next section. This analysis may be performed in step <NUM>, for example. Alternatively, the start of the period may be determined to occur a fixed time before the start of the next section, e.g. between <NUM> and <NUM> seconds before the start of the next section.

The threshold may be the same for each pair of consecutive sections but may also be different for a first pair of consecutive sections than for a second pair of consecutive sections. For example, if the sections have been classified, the threshold may be influenced by the types of the sections. For instance, for transitions from "introduction" to first "verse" and for transitions to and from "bridge", a lower threshold may be used, while for transitions from "chorus" to "verse", a higher threshold may be used.

Step <NUM> comprises determining in which period of the current section the next light effect will be rendered. If it is determined that the next light effect will be rendered during the above-mentioned period before the start of the next section, a step <NUM> is performed next. Otherwise, step <NUM> is performed directly after step <NUM>. Step <NUM> comprises gradually reducing a light intensity and/or color saturation of the light effects rendered during the above-mentioned period before the start of the next section.

Step <NUM> comprises controlling a lighting device to render the next light effect, including adjustment if an adjustment was made in step <NUM>. A step <NUM> is performed after step <NUM>. Step <NUM> comprises determining whether the light effect to be rendered after the light effect that has just been rendered, belongs to a new section. If so, step <NUM> is repeated, and the method proceeds as shown in <FIG>. If it is determined in step <NUM> that this light effect belongs to the same section, step <NUM> is repeated, and the method proceeds as shown in <FIG>.

A second embodiment of the method of controlling a lighting device to render light effects while an audio rendering device plays a song is shown in <FIG>. The method may be performed by the mobile device <NUM> of <FIG> or the cloud computer <NUM> of <FIG>, for example. The embodiment of <FIG> is a variation on the embodiment of <FIG>.

In the embodiment of <FIG>, the medians or averages of all sections are determined in a single iteration of step <NUM>. In step <NUM>, a light effect is determined that will be rendered at a certain moment in a certain section of the song. In the first iteration of step <NUM>, the first light effect is determined. If it is determined in step <NUM> that the difference between the median or average of the audio characteristic in this certain section and the median or average in the next section succeeding this certain section exceeds the threshold and it is determined in step <NUM> that the certain moment in the certain section falls in a period before the start of the next section, step <NUM> is performed next.

This period may start between <NUM> and <NUM> seconds before the start of the next section (and thus have a duration of between <NUM> and <NUM> seconds) but could also start much later. For example, if a highest light intensity setting is used to create the light effects, the duration of many light effects is shorter than one second and the period could then also have a duration of less than one second. The duration of the period may additionally or alternatively be determined based on the duration of the first section.

The anticipatory fading is implemented in step <NUM>. The fading could be done on intensity, but also on e.g. color saturation (desaturating the signal prior to the key event), or colorfulness (e.g. number of different colors in the scene) or a combination thereof. In a step <NUM>, a light script that includes the anticipatory fading is created.

Next, a step <NUM> comprises determining whether there is a next light effect to be determined. If so, step <NUM> is repeated for the next light effect and the method proceeds as shown in <FIG>. If it is determined in step <NUM> that all light effects have been created and the light script is ready, step <NUM> is performed. Step <NUM> comprises controlling one or more lighting devices to render the determined light effects. For example, the light script may be streamed to the connected luminaires.

In the embodiment of <FIG>, the light effects still need to be created, e.g. based on data points received from the audio streaming service or based on a local analysis of the song. In an alternative embodiment, a predefined content-based light script is obtained. In this alternative embodiment, step <NUM> may be performed between steps <NUM> and <NUM> and may comprise extracting the light effect to be adjusted from the light script. The light effect adjusted in step <NUM> may then be stored in the light script in step <NUM>. Thus, in this alternative embodiment, the existing light script would be adjusted in step <NUM>. This may be an offline process. For instance, a system may retrieve light scripts for the audio or video content that the user has stored or that the system expects the user to play shortly and adjust the light scripts to include the anticipatory fading.

A third embodiment of the method of controlling a lighting device to render light effects while an audio rendering device plays a song is shown in <FIG>. The embodiment of <FIG> is an implementation of the embodiment of <FIG>. In the embodiment of <FIG>, a step <NUM> is performed before step <NUM>, step <NUM> is implemented by a step <NUM>, step <NUM> is implemented by a step <NUM>, a step <NUM> is performed before step <NUM>, step <NUM> is implemented by a step <NUM>, and step <NUM> is implemented by a step <NUM>.

Step <NUM> comprises obtaining lighting control limitations set by a user. In the embodiment of <FIG>, step <NUM> comprises determining the current user settings/preferences, e.g. the dynamics settings (e.g. low, medium or intense), the current color palette, and/or a lighting system setup.

Step <NUM> comprises receiving information from the audio streaming service which comprises data points for the plurality of sections of the song. Step <NUM> comprises determining the first and second medians or averages from the data points received in step <NUM>.

Step <NUM> comprises selecting a subset of the received data points of which the audio intensity exceeds a further threshold. Step <NUM> comprises determining the light effects based on the data points selected in step <NUM> such that they comply with the lighting control limitations obtained in step <NUM>. Step <NUM> comprises gradually reducing a light intensity and/or color saturation of the light effects but such that the reduced intensity and/or color saturation still complies with the lighting control limitations obtained in step <NUM>.

In step <NUM>, the user settings/preferences light effects determined in step <NUM> are taken into account while adjusting the light effects to include the anticipatory fading. As a first example, the reduced light intensity is not reduced below a user specified minimum light intensity. As a second example, a dynamics setting of 'low' may result in a more subtle anticipatory fading than a dynamics setting of e.g. 'intense'. Other user preferences could offer users the possibility to indicate whether they want to emphasize identified content key moments with special light effects, and to what extent (e.g. mild - medium - boost).

A fourth embodiment of the method of controlling a lighting device to render light effects while an audio rendering device plays a song is shown in <FIG>. The embodiment of <FIG> is an implementation of the embodiment of <FIG>. In the embodiment of <FIG>, a step <NUM> is performed before step <NUM>, step <NUM> is implemented by a step <NUM>, a step <NUM> is performed before step <NUM>, step <NUM> is implemented by a step <NUM>, and step <NUM> is implemented by a step <NUM>.

Step <NUM> comprises obtaining lighting control limitations set by a user, as described in relation to <FIG>. Step <NUM> comprises receiving information from the audio streaming service which comprises the information indicative of a median or an average of an audio characteristic for each section of a plurality of the sections of the song in addition to the song itself. The information may indicate where each section begins, for example. Step <NUM> comprises analyzing the song. Step <NUM> comprises determining the light effects based on the analysis performed in step <NUM>. In step <NUM>, the light effects are determined such that they comply with the lighting control limitations obtained in step <NUM>.

Step <NUM> comprises gradually reducing a light intensity and/or color saturation of the light effects but without requiring that the reduced intensity and/or color saturation complies with the lighting control limitations obtained in step <NUM>. In other words, the lighting control limitations obtained in step <NUM> are ignored when performing the reduction of the light intensity and/or color saturation.

A fifth embodiment of the method of controlling a lighting device to render light effects while an audio rendering device plays a song is shown in <FIG>. The embodiment of <FIG> is an extension of the embodiment of <FIG>. Besides adjusting light effects (just) before the key moment, light effects during and after the key moment may also be adjusted. For instance, the light intensity and color saturation of the light may be amplified at and after the key moment. This may depend on user preferences.

In the embodiment of <FIG>, steps <NUM> and <NUM> of <FIG> has been replaced with steps <NUM> and <NUM> and new steps <NUM> and <NUM> have been added. Step <NUM> comprises determining whether a difference between the first and second medians or averages determined in the last iteration of step <NUM> exceeds a threshold and whether a difference between the first and second medians or averages determined in the second last iteration of step <NUM> exceeds the threshold. If it is determined in step <NUM> that at least one of the differences exceeds the threshold, step <NUM> is performed. If not, step <NUM> is performed directly after step <NUM>.

Step <NUM> comprises determining in which part of the current section the next light effect will be rendered and determining which step should be performed next in dependence on the determined part.

If it was determined in step <NUM> that the difference determined in the last iteration of step <NUM> exceeds the threshold and it is determined in step <NUM> that the next light effect is to be rendered during a period (e.g. of between <NUM> and <NUM> seconds) before the start of the next section, step <NUM> is performed next. Step <NUM> comprises gradually reducing a light intensity and/or color saturation of the light effects rendering during this period.

If it was determined in step <NUM> that the difference determined in the second last iteration of step <NUM> exceeds the threshold and it is determined in step <NUM> that the next light effect to be rendered is the first light effect or one of the first light effects of the current section, step <NUM> is performed. Thus, in this case, the next light effect is rendered at the start of the second section of the two consecutive sections which have been compared in the second last iteration of step <NUM>.

Step <NUM> comprises increasing the light intensity and/or color saturation of the next light effect. If lighting control limitations set by a user have been obtained, as described in relation to step <NUM> of <FIG> and <FIG>, these lighting control limitations may be ignored when performing the increase of the light intensity and/or color saturation in step <NUM>.

If it was determined in step <NUM> that the difference determined in the second last iteration of step <NUM> exceeds the threshold and it is determined in step <NUM> that the next light effect is to be rendered during a further period immediately after the start of the current section, step <NUM> is performed. Thus, in this case, the next light effect is the light effect or one of the light effects that will be rendered immediately after the start of the second section of the two consecutive sections which have been compared in the second last iteration of step <NUM>. Step <NUM> comprises gradually reducing the increase, realized previously in step <NUM>, during the further period.

If it is determined in step <NUM> that none of the conditions for performing step <NUM>, step <NUM>, or step <NUM> are met, step <NUM> is performed directly after step <NUM>. In the intermediate period between the afore-mentioned period and the afore-mentioned further period, step <NUM> is always performed directly after step <NUM>. If the difference determined in the second last iteration of step <NUM> did not exceed the threshold, step <NUM> is also performed directly after step <NUM> at the start of the current section and in the further period immediately after the start of the current section. If the difference determined in the last iteration of step <NUM> did not exceed the threshold, step <NUM> is also performed directly after step <NUM> in the period before the start of the next section.

A sixth embodiment of the method of controlling a lighting device to render light effects while an audio rendering device plays a song is shown in <FIG>. The embodiment of <FIG> is a variation on the embodiment of <FIG>. In the embodiment of <FIG>, step <NUM> has been replaced with a step <NUM>, step <NUM> has been omitted, step <NUM> is implemented by a step <NUM>, and steps <NUM>, <NUM>, and <NUM> have been added.

Furthermore, since step <NUM> of <FIG> has been replaced with step <NUM> and step <NUM> of <FIG> has been omitted, step <NUM> of <FIG> has been replaced with a step <NUM>. If it is determined in step <NUM> that none of the conditions for performing step <NUM> or step <NUM> are met, step <NUM> is performed directly after step <NUM>, i.e. before step <NUM> is performed.

If it was determined in step <NUM> that the difference determined in the second last iteration of step <NUM> exceeds the threshold and it is determined in step <NUM> that the next light effect to be rendered is the first light effect or one of the first light effects of the current section, step <NUM> is performed. Step <NUM> comprises controlling the lighting device to render a special light effect at the start of the current section. Step <NUM> is performed after step <NUM>. Step <NUM> comprises selecting all lighting devices of a plurality of lighting devices.

If it was determined in step <NUM> that the difference determined in the last iteration of step <NUM> exceeds the threshold and it is determined in step <NUM> that the next light effect is to be rendered during a period before the start of the next section, step <NUM> is performed next, as described in relation to <FIG>.

Step <NUM> is performed after step <NUM>. Step <NUM> comprises determine a level of the reduction of the light intensity and/or color saturation based on the difference between the first and second medians or averages, as determined in the last iteration of step <NUM>. In other words, step <NUM> comprises determining how deep the deep the fade out goes. In an alternative embodiment, step <NUM> is combined with step <NUM>. Before step <NUM> is performed, it may be determined in step <NUM> in which period before the start of the next section anticipatory fading should be applied and how deep the fade out goes.

In the embodiment of <FIG>, the difference between the first and second medians or averages is considered to represent the build-up in the music that leads to the start of the next section and the level of the reduction thus depends on this build-up in the music. Alternatively, the build-up in the music may be determined in a different manner, e.g. by analyzing only a part of the current section and/or based on the approximate duration of the build-up. If the build-up is very audible, starting to fade earlier and deeper could be seen as a very natural effect, while if build up is (almost) absent, the fade out should preferably also be done faster, otherwise it might confuse the user of why lights are changing without an audible change in the music, and might even ruin the effect of the key moment by hinting the listener that it is coming up.

As mentioned in relation to step <NUM> of <FIG>, the fade out period may also be done faster if a highest light intensity setting is used to create the light effects. Moreover, the depth of the fade out and/or the depth of the light intensity increase at the start of the next section may also be determined based on the used light intensity setting.

Step <NUM> is performed after step <NUM>. Step <NUM> comprises selecting a proper subset of the plurality of lighting devices. Besides intensity and saturation, it is also possible to use spatial dynamics in order to emphasize the key moment. For instance, just before an explosion in a video, only a subset of the lighting devices (the ones close to the TV) may be rendering the content-based light effects, whereas at moment of the explosion all lighting devices in the room (and possible other lighting devices generating an effect in the user's field of view, e.g. the garden) may be included. Knowing the position of individual lighting devices (relative to each other and/or to a media rendering device or user's field of view) may help to create smart spatial effects. For instance, in anticipation of the key moment, the lighting devices in the peripheral view of the user may gradually fade out and become active again at the key moment.

In a similar way, if trumpets will start to play at the start of the next section and this start is determined to be a key moment, the light sources which are selected to render the trumpets may very gradually fade out (e.g. the violin-associated colors) in anticipation of the key moment. At the start of the next section, these light sources may become active again in a different (trumpet-associated) color. Such light sources may either be lighting devices or segments or pixels of a pixelated lighting device.

Step <NUM> is performed after step <NUM> or step <NUM> has been performed. Step <NUM> comprises controlling the lighting device(s) selected in step <NUM> or step <NUM> to render the (possibly adjusted) light effect(s). The same light effects or different light effects may be rendered by multiple lighting devices if multiple lighting devices have been selected.

The embodiments of <FIG> differ from each other in multiple aspects, i.e. multiple steps have been added or replaced. In variations on these embodiments, only a subset of these steps is added or replaced and/or one or more steps is omitted. As a first example, step <NUM> of <FIG> may be replaced with step <NUM> of <FIG> and/or step <NUM> of <FIG> may be replaced with step <NUM> of <FIG>. As a second example, steps <NUM>, <NUM>, and <NUM> of the embodiment of <FIG> may be added to the embodiment of <FIG> and/or omitted from the embodiment of <FIG>. As a third example, the embodiments of <FIG> and <FIG> may be combined. As a fourth example, the steps added to the embodiment of <FIG> to obtain the embodiments of <FIG> may be added to the embodiment of <FIG>.

<FIG> depicts a block diagram illustrating an exemplary data processing system that may perform the method as described with reference to <FIG>.

The data processing system may be an Intemet/cloud server, for example.

Claim 1:
A system (<NUM>,<NUM>) for controlling a lighting device (<NUM>,<NUM>,<NUM>) to render light effects while an audio rendering device (<NUM>,<NUM>,<NUM>) plays a song, said system (<NUM>,<NUM>) comprising:
at least one receiver (<NUM>,<NUM>);
at least one transmitter (<NUM>,<NUM>); and
at least one processor (<NUM>,<NUM>) configured to:
- receive information from an audio streaming service (<NUM>) via said at least one receiver (<NUM>,<NUM>), said information being indicative of a median or an average of an audio characteristic for each section of a plurality of sections of said song,
- determine, from said information, a first median or average of said audio characteristic in a first section (<NUM>) of said plurality of sections and a second median or average of said audio characteristic in a second consecutive section (<NUM>) of said plurality of sections,
- determine whether a difference between said first and second medians or averages exceeds a threshold,
- determine said light effects based on said information,
- gradually reduce a light intensity (<NUM>) and/or color saturation of said light effects during a period (<NUM>) before the start (<NUM>) of said second section in dependence on said difference exceeding said threshold, and
- control, via said at least one transmitter (<NUM>,<NUM>), said lighting device (<NUM>,<NUM>,<NUM>) to render said light effects.