Patent Description:
The invention further relates to a method of controlling a light source based on an analysis of image content in an analysis region of said image content while a display displays said image content.

The Philips HueSync application for Window PCs and Apple computers brings the atmosphere of a game, music or a movie right into the room with the user. When the application is active, selected lights will play light effects to accompany the content. A new product called the HDMI Sync box, an HDMI module, has recently been added to the Hue entertainment portfolio. This new device does not require a computer and is intended for use with streaming and gaming devices connected to the TV.

A pixelated light strip may be mounted around a TV to create an immersive experience, such as the experience created by the TV described in <CIT>. One of the key issues with a pixelated strip around the TV is that TVs are typically placed at different distances before a surface by different consumers. The surface may be a wall or another surface on which the light is projected, e.g. the backside of a cabinet in which the TV has been placed.

For example, some consumers may mount the TV directly to the wall, while others may put it on a TV cabinet or table at <NUM>-<NUM> from the wall. Although <CIT> discloses changing the angle position of the pixelated light sources in the TV based on a distance to a wall, there is still a large difference between the quality of the entertainment light effects when the TV is near the wall and the quality of the entertainment light effects when the TV is farther away from the wall.

Document <CIT> discloses a system for determining one or more light effects based on an analysis of video content. Thereby, a selected color extraction method is applied to extract a color from one or more frames of said video content. Light effects based on said extracted color are subsequently rendered on a light source.

It is a first object of the invention to provide a system, which is able to control a light source to render entertainment light effects whose quality is less dependent on the distance between the light source and a surface.

It is a second object of the invention to provide a method, which can be used to control a light source to render entertainment light effects whose quality is less dependent on the distance between the light source and a surface.

The present invention is set out in the appended set of independent claims and dependent claims. Particularly, the invention is defined by a system as set out in claim <NUM> and by a method as set out in claim <NUM>. Further embodiments are set out in the dependent claims.

In a first aspect of the invention, a system for controlling a light source based on an analysis of image content in an analysis region of said image content while a display displays said image content, comprises at least one input interface, at least one output interface, and at least one processor configured to obtain said image content via said at least one input interface, obtain a distance between said light source and a surface, determine a size and/or location for said analysis region based on said distance, determine a characteristic of said image content by analyzing said image content in said analysis region, determine a a light effect based on said characteristic, and control, via said at least one output interface, said light source to render said light effect onto the surface. The light effect may comprise a color and/or intensity.

Said characteristic may comprise or be e.g. be a color, e.g. a pixel value, and/or e.g. an intensity. Said characteristic may e.g. be a color throughout the application, wherein said color may be determined by analyzing said image content in said analysis region, i.e. e.g. with a color extraction methodology.

When the light source is placed relatively far from the surface (e.g. a wall), the blending of light from different light sources of a pixelated lighting device, also referred to as pixels, will happen automatically by optical mixing. The larger the distance between the light source/pixel and the surface, the more the light emitted by the light sources/pixels will blend. At a short distance, there will be hardly any blending between the pixels. User perception tests have shown that users find the entertainment light effects to be of lesser quality when blending does not take place. For this reason, an algorithmic way of blending the colors from the pixels is used which depends on the distance between the light source and the surface. This distance may be obtained from a sensor or from a user device, for example. In the latter case, the distance is determined from user input. The user device may be a user input device, for example. The location of the analysis region may be the center of mass of the analysis region or the location of one or more of the corners, for example. The image content is typically video content. Hence, the system according to the invention may comprise the sensor or the user device. In aspects, the light source may comprise the sensor or the user device (such as a user input device or user interface device).

Said at least one processor may be configured to determine a first analysis region with a first size and a first location when said distance has a first value and a second analysis region with a second size and a second location when said distance has a second value, said second size being different than said first size and/or said second location being different than said first location, said first analysis region having a larger overlap with adjacent analysis regions than said second analysis region and said first value being smaller than said second value. Thus, as the distance between the light source/pixel and the surface gets smaller, the overlap between adjacent analysis regions increases to blend the light effects determined from the analysis regions, e.g. blend the colors extracted from the analysis regions, to a higher degree. Overlap between analysis regions may be increased by increasing the size of one or more of the analysis regions, for example.

In aspects, the size according to the invention may be predefined or constant, whereas the location may be determined based on said distance. In aspects, the location according to the invention may be predefined or constant, whereas the size may be determined based on said distance.

In aspects, throughout the application, but phrased alternatively, the processor according to the invention may be configured to determine an image analysis property for said analysis region based on said distance, wherein the image analysis property may comprise a size and/or a location for said analysis region. Hence, the size and/or location may be defined as an image analysis property.

Said light source may be comprised in a lighting device, said lighting device may comprise a further light source, said distance may also represent a distance between said further light source and said surface, and said at least one processor may be configured to determine a size and/or location for a further analysis region of said image content based on said distance, determine a further characteristic of said image content by analyzing said image content in said further analysis region, determine a further light effect based on said further characteristic, and control said further light source to render said further light effect. Hence, said distance may in examples be the distance between the lighting device and said surface; this may be considered a device distance in examples. The further light effect may comprise a color and/or intensity. Said further characteristic may e.g. comprise or be a color, e.g. a pixel value, and/or e.g. an intensity.

Often, the lighting device is placed such that the distance between each light source and the surface is the same for all light sources of the lighting device and making an assumption that the distance between multiple, e.g. all, light sources of the lighting device and the surface are the same therefore works in many situations. If the distances between the light sources and the surface are not the same but differ slightly, this assumption will usually still result in high quality entertainment light effects.

Said at least one processor may be configured to obtain a device distance between said lighting device and said surface, obtain a further device distance between said lighting device and said surface, and determine said distance by calculating an average of said device distance and said further device distance. This is beneficial if multiple device distances are obtained, e.g. the distances between the two edges of a lighting device and the surface.

Said light source may be comprised in a lighting device, said lighting device may comprise a further light source, and said at least one processor may be configured to obtain a further distance between said further light source and said surface, determine a size and/or location for a further analysis region of said image content based on said further distance, determine a further characteristic of said image content by analyzing said image content in said further analysis region, determine a further light effect based on said further characteristic, and control said further light source to render said further light effect. The further light effect may comprise a color and/or intensity. Said further characteristic may e.g. comprise or be a color, e.g. a pixel value, and/or e.g. an intensity. Determining different distances for different light sources of the same lighting device is beneficial if there is a substantial difference between the distances between the light sources and the surface. This may be the case when a lighting device that comprises vertically arranged light sources is placed leaning against a wall or when a light strip is attached horizontally to a curved display or display placed in a corner, for example,.

Said at least one processor may be configured to obtain a device distance between said lighting device and said surface, obtain a further device distance between said lighting device and said surface, determine said distance between said light source and said surface based on said device distance, said further device distance and a position of said light source on said lighting device, and determine said further distance between said further light source and said surface based on said device distance, said further device distance and a position of said further light source on said lighting device. If a distance between a light source and the surface is not obtained for each light source, it is still possible to determine a relatively accurate distance to the surface for each light source based on the (e.g. two) device distances that have been obtained.

Said at least one processor may be configured to estimate an amount of light overlap between light projected on said surface by said light source and light projected on said surface by said further light source based on said distance, determine an amount of desired region overlap between said analysis region and said further analysis region based on said estimated amount of light overlap, and determine said size and/or location for said analysis region and said size and/or location for said further analysis region based on said amount of desired region overlap. This may be beneficial, for example, if the user is able to change the angle of the light sources. If the user is not able to change the angle of the light sources, it is also possible to use predetermined mappings between distance and desired region overlap or between distance and analysis region size and/or location.

Said at least one processor may be configured to determine said size and/or location for said analysis region further based on a size of said light source, a size of said display, and/or a size of a lighting device comprising said light source. As a first example, the formula C=p+b/(<NUM>*d+<NUM>) may be used to calculate the size of the analysis region for a (e.g. LED) pixel in centimeters when d is smaller than a certain threshold T, C being the size of the analysis region in centimeters, b being a blending factor, d being an (estimated or measured) numeric distance value in centimeters and p being the size of the pixel in centimeters. In this example, a minimum analysis region size is used when d is greater than or equal to threshold T.

As a second example, a mapping table may be used, where the percentage of overlap is given for multiple ranges of distances to the surface. For example, a distance between <NUM> and <NUM> may be mapped to an overlap percentage of <NUM>%, a distance between <NUM> and <NUM> may be mapped to an overlap percentage of <NUM>%, etc. The size of the analysis region may then be determined based on the size of the pixel and the determined overlap percentage.

The ratio between pixel (light source) size and display size may further be used as parameter. For instance, the above-mentioned formula may be modified such that C is a function of p (size of a pixel), d (distance to the surface) and r (ratio between display size and pixel size). In a first implementation, the larger the size of the pixel relative to the size of the display, the smaller the overlap percentage. For example, with a distance of <NUM> to a wall, for a TV with a <NUM>-inch display (<NUM> inch being the diagonal dimension), the overlap between adjacent analysis regions might be <NUM>% for a <NUM> pixel and <NUM>% for a <NUM> pixel and for a TV with a <NUM>-inch display, the overlap between adjacent analysis regions might be <NUM>% for a <NUM> pixel and <NUM>% for a <NUM> pixel. Said at least one processor may be configured to determine said size and/or location for said analysis region further based on a distance between said light source and said display. For example, the entertainment light effects may look better if a larger analysis region is used when the distance between the light source and the display is larger.

Said light source may comprise a plurality of light elements which are not able to render different light effects. In other words, if these light elements render a light effect, they render the same light effect. A pixelated lighting device often comprises a plurality of such light sources. In a pixelated lighting device, the light source is also referred to as a pixel or a segment. The light element may be a LED, for example.

In a second aspect of the invention, a method of controlling a light source based on an analysis of image content in an analysis region of said image content while a display displays said image content comprises obtaining said image content, obtaining a distance between said light source and a surface, determining a size and/or location for said analysis region based on said distance, determining a characteristic of said image content by analyzing said image content in said analysis region, determining a light effect based on said characteristic, and controlling said light source to render said light effect onto the surface. Said method may be performed by software running on a programmable device. The light effect may comprise a color and/or intensity. This software may be provided as a computer program product. Said characteristic may e.g. comprise or be a color, e.g. a pixel value, and/or e.g. an intensity.

A non-transitory computer-readable storage medium stores at least one software code portion, the software code portion, when executed or processed by a computer, being configured to perform executable operations for controlling a light source based on an analysis of image content in an analysis region of said image content while a display displays said image content.

The executable operations comprise obtaining said image content, obtaining a distance between said light source and a surface, determining a size and/or location for said analysis region based on said distance, determining a characteristic of said image content by analyzing said image content in said analysis region, determining a light effect based on said characteristic, and controlling said light source to render said light effect onto the surface. The light effect may comprise a color and/or intensity. Said characteristic may e.g. comprise or be a color, e.g. a pixel value, and/or e.g. an intensity.

<FIG> shows a first embodiment of the system for controlling a light source based on an analysis of image content in an analysis region of the image content while a display displays the image content: an HDMI module <NUM>. The HDMI module <NUM> may be a Hue Play HDMI Sync Box, for example. In the example of <FIG>, the image content is rendered on a display <NUM>. Alternatively, the image content may be rendered on multiple displays, e.g. a video wall.

In the example of <FIG>, The HDMI module <NUM> can control the lighting devices <NUM>-<NUM> via a bridge <NUM>. The bridge <NUM> may be a Hue bridge, for example. The bridge <NUM> communicates with the lighting devices <NUM>-<NUM>, e.g. using Zigbee technology. The HDMI module <NUM> is connected to a wireless LAN access point <NUM>, e.g. via Wi-Fi. The bridge <NUM> is also connected to the wireless LAN access point <NUM>, e.g. via Wi-Fi or Ethernet.

Alternatively or additionally, the HDMI module <NUM> may be able to communicate directly with the bridge <NUM>, e.g. using Zigbee technology, and/or may be able to communicate with the bridge <NUM> via the Internet/cloud. Alternatively or additionally, the HDMI module <NUM> may be able to control the lighting devices <NUM>-<NUM> without a bridge, e.g. directly via Wi-Fi, Bluetooth or Zigbee or via the Internet/cloud.

The wireless LAN access point <NUM> is connected to the Internet <NUM>. A media server <NUM> is also connected to the Internet <NUM>. Media server <NUM> may be a server of a video-on-demand service such as Netflix, Amazon Prime Video, Hulu, Disney+ or Apple TV+, for example. The HDMI module <NUM> is connected to the display <NUM> and local media receivers <NUM> and <NUM> via HDMI. The local media receivers <NUM> and <NUM> may comprise one or more streaming or content generation devices, e.g. an Apple TV, Microsoft Xbox One and/or Sony PlayStation <NUM>, and/or one or more cable or satellite TV receivers.

In the example of <FIG>, the lighting devices <NUM> and <NUM> are vertically arranged arrays of light sources, like e.g. the Philips Hue Signe, and the lighting device <NUM> is a horizontally arranged array of light sources, e.g. a horizontally placed light strip. The lighting device <NUM> comprises four light sources (pixels) <NUM>-<NUM> and one distance sensor <NUM>, the lighting device <NUM> comprises four light sources (pixels) <NUM>-<NUM> and no distance sensor, and the lighting device <NUM> comprises five light sources (pixels) <NUM>-<NUM> and two distance sensors <NUM> and <NUM>. The distance sensors <NUM>-<NUM> may comprise one or more infrared distance sensors and/or one or more ultrasonic distance sensors, for example.

The lighting devices <NUM>-<NUM> are also referred to as pixelated lighting devices. In practice, a pixelated lighting device comprises more than four or five pixels. In the example of <FIG>, each light source (pixel) comprises two light elements, e.g. LEDs, which are not able to render light effects that are not the same. Light sources <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> are also referred to as individually addressable segments of light elements.

The HDMI module <NUM> comprises a receiver <NUM>, a transmitter <NUM>, a processor <NUM>, and memory <NUM>. The processor <NUM> is configured to obtain video content via the receiver <NUM>, e.g. from media receiver <NUM> or <NUM>, obtain a distance between the light sources of each lighting device and a surface, e.g. from one or more of sensors <NUM>-<NUM> or from a user device <NUM>, and determine, for each light source, a size and/or location for an analysis region associated with the light source based on the distance. User device <NUM> may be a mobile phone or a tablet, for example.

The processor <NUM> is further configured to determine a characteristic of the video content by analyzing the video content in the analysis region, determine a color and/or intensity for a light effect based on the characteristic, and control, via the transmitter <NUM>, the light sources <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> to render the light effects. Said characteristic may e.g. comprise or be a color, e.g. a pixel value, and/or e.g. an intensity.

In the example of <FIG>, a distance between the light sources and a surface is known for each for each of lighting devices <NUM>-<NUM> and the analysis regions can therefore be determined based on these distances. Analysis regions with distance-independent sizes and locations may be used for light sources of which no distance between the light sources and a surface is known.

In the example of <FIG>, two device distances are obtained from the lighting device <NUM> and one device distance is obtained from the lighting device <NUM>. A device distance is a distance between the lighting device and a surface on which the light is projected, e.g. a wall. A second device distance may be obtained for lighting device <NUM> from user device <NUM>. Alternatively, the single device distance obtained from the lighting device <NUM> may be considered to represent the distance between all light sources of the lighting device <NUM> and the surface, or, if the top of the lighting device <NUM> is intended to lean against the surface, a second device distance may be determined to be zero. One or more device distances for lighting device <NUM> are obtained from the user device <NUM>.

If only a single device distance is obtained for a lighting device, the distance between the surface and each light source of the lighting device is assumed to be this single device distance. If multiple device distances are obtained for a lighting device, the distance between the surface and each light source of the lighting device may be an average of the device distances or the distance between a light source and the surface may be determined based on the device distances and the position of the light source on the lighting device.

As an example of the former, if the distance sensor <NUM> measures a distance of <NUM> and the distance sensor <NUM> measures a distance of <NUM>, the distance between the surface and each light source of lighting device <NUM> may be considered to be <NUM>. As an example of the latter, if the distance sensor <NUM> measures a distance of <NUM> and the distance sensor <NUM> measures a distance of <NUM>, the distance between light sources <NUM>-<NUM> and the surface may be considered to be <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, respectively.

If the distance between each light source of a lighting device and the surface is assumed to be the same, then it is not necessary to determine a distance per light source after determining the (average) device distance. The processor <NUM> may then determine the sizes and/or locations for the analysis regions directly based on the (average) device distance.

In the embodiment of the HDMI module <NUM> shown in <FIG>, the HDMI module <NUM> comprises one processor <NUM>. In an alternative embodiment, the HDMI module <NUM> comprises multiple processors. The processor <NUM> of the HDMI module <NUM> may be a general-purpose processor, e.g. ARM-based, or an application-specific processor. The processor <NUM> of the HDMI module <NUM> may run a Unix-based operating system for example. The memory <NUM> may comprise one or more memory units. The memory <NUM> may comprise solid-state memory, for example.

The receiver <NUM> and the transmitter <NUM> may use one or more wired or wireless communication technologies such as Zigbee to communicate with the bridge <NUM> and HDMI to communicate with the display <NUM> and with local media receivers <NUM> and <NUM>, for example. In an alternative embodiment, multiple receivers and/or multiple transmitters are used instead of a single receiver and a single transmitter. In the embodiment shown in <FIG>, a separate receiver and a separate transmitter are used. In an alternative embodiment, the receiver <NUM> and the transmitter <NUM> are combined into a transceiver. The HDMI module <NUM> may comprise other components typical for a network device such as a power connector. The invention may be implemented using a computer program running on one or more processors.

In the embodiment of <FIG>, the system of the invention is an HDMI module. In an alternative embodiment, the system may be another device, e.g. a mobile device, laptop, personal computer, a bridge, a media rendering device, a streaming device, or an Internet server. In the embodiment of <FIG>, the system of the invention comprises a single device. In an alternative embodiment, the system comprises multiple devices.

The analysis of the video content may be performed in real-time, i.e. just before the light sources are controlled and the video content is displayed. Alternatively, the analysis of the video content may be performed earlier, e.g. by using automatic light scripting. Automatic light scripting may be performed by the above-mentioned Internet server, for example. In automatic light scripting, analysis of the video content is done prior to the user watching/streaming it (it could also be done near real-time with e.g. a buffer of <NUM> minutes), typically in the cloud. This may be used to ensure a perfect synchronization between content and light effects.

A processing system running in the cloud could use a user profile, indicating the distance to the surface amongst others, to generate a personalized script. Alternatively, the system might pre-generate a set of scripts for a few common distances (e.g. <NUM>-<NUM>, <NUM>-<NUM> and more than <NUM>), and when the user starts streaming a movie, the system could then choose the script that is closest to the user's setup. The latter will save resources in the cloud when popular movies are streamed.

<FIG> shows a second embodiment of the system for controlling a light source based on an analysis of image content in an analysis region of the image content while a display displays the image content: a mobile device <NUM>. The mobile device <NUM> may be a smart phone or a tablet, for example. The lighting devices <NUM>-<NUM> can be controlled by the mobile device <NUM> via the bridge <NUM>. The mobile device <NUM> is connected to the wireless LAN access point <NUM>, e.g. via Wi-Fi.

The mobile device <NUM> comprises a receiver <NUM> a transmitter <NUM>, a processor <NUM>, a memory <NUM>, and a display <NUM>. The image content is preferably displayed on the external display <NUM> but could also be displayed on display <NUM> of the mobile device <NUM>. The processor <NUM> is configured to obtain video content via the receiver <NUM>, e.g. from the media server <NUM>, and obtain a distance between the light sources of each lighting device and a surface, e.g. from one or more of sensors <NUM>-<NUM>, see <FIG>, or from an input interface (e.g. a touchscreen display or a microphone) of the mobile device <NUM> itself,.

The processor <NUM> is further configured to determine, for each light source, a size and/or location for an analysis region associated with the light source based on the distance, determine a characteristic of the video content by analyzing the video content in the analysis region, determine a color and/or intensity for a light effect based on the characteristic, and control, via the transmitter <NUM>, the light sources <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> to render the light effects. Alternatively, said "determine a color and/or intensity for a light effect" may be phrased as "determine a light effect", wherein said light effect may comprise a color and/or intensity.

In the embodiment of the mobile device <NUM> shown in <FIG>, the mobile device <NUM> comprises one processor <NUM>. In an alternative embodiment, the mobile device <NUM> comprises multiple processors. The processor <NUM> of the mobile device <NUM> may be a general-purpose processor, e.g. from ARM or Qualcomm or an application-specific processor. The processor <NUM> of the mobile device <NUM> may run an Android or iOS operating system for example. The display <NUM> may be a touchscreen display, for example. The display <NUM> may comprise an LCD or OLED display panel, for example. The memory <NUM> may comprise one or more memory units. The memory <NUM> may comprise solid state memory, for example.

The receiver <NUM> and the transmitter <NUM> may use one or more wireless communication technologies such as Wi-Fi (IEEE <NUM>) to communicate with the wireless LAN access point <NUM>, for example. In an alternative embodiment, multiple receivers and/or multiple transmitters are used instead of a single receiver and a single transmitter. In the embodiment shown in <FIG>, a separate receiver and a separate transmitter are used. In an alternative embodiment, the receiver <NUM> and the transmitter <NUM> are combined into a transceiver. The mobile device <NUM> may further comprise a camera (not shown). This camera may comprise a CMOS or CCD sensor, for example. The mobile device <NUM> may comprise other components typical for a mobile device such as a battery and a power connector. The invention may be implemented using a computer program running on one or more processors.

In the embodiment of <FIG>, the lighting devices <NUM>-<NUM> are controlled via the bridge <NUM>. In an alternative embodiment, one or more of the lighting devices <NUM>-<NUM> is controlled without a bridge, e.g. directly via Bluetooth. In the embodiment of <FIG>, the system of the invention comprises only local devices. In an alternative embodiment, the system of the invention comprises one or more Internet/cloud servers.

<FIG> shows an example of the lighting device <NUM> of <FIG> having a first distance <NUM> to a wall <NUM>. The lighting device <NUM> has been attached to the back of display <NUM> of <FIG>. <FIG> shows an example of the lighting device <NUM> having a second distance <NUM> to the wall <NUM>. <FIG> shows an example of the lighting device <NUM> having a third distance <NUM> to the wall <NUM>. The first distance <NUM> is shorter than the second distance <NUM>. The second distance <NUM> is shorter than the third distance <NUM>.

<FIG> depicts an example of a space in which the system of <FIG> is used. A floor <NUM> of a home comprises a hallway <NUM>, a kitchen <NUM> and a living room <NUM>. Lighting devices <NUM>-<NUM> have been installed in the living room <NUM>. Vertically arranged lighting devices <NUM> and <NUM> have been placed on respectively the left and right side of the display <NUM>, which may be a TV, for example. Horizontally arranged lighting device <NUM> has been attached to the back of display <NUM>.

The wireless LAN access point <NUM> has been installed in the hallway <NUM>. The HDMI module <NUM> has been installed next to the display <NUM> in the living room <NUM>. The bridge <NUM> has been installed in the living room <NUM> near the wireless LAN access point <NUM>. A person <NUM> is watching TV. The lighting device <NUM> has a second distance <NUM> to the wall, as shown in <FIG>. The lighting device <NUM> has distance <NUM> to the display <NUM>. The lighting device <NUM> has a distance <NUM> to the display <NUM>.

A first embodiment of the method of controlling a light source based on an analysis of image content in an analysis region of the image content while a display displays the image content is shown in <FIG>. A step <NUM> comprises obtaining video content.

A step <NUM> comprises obtaining lighting device information relating to the lighting devices. The lighting device information may comprise, per lighting device, the width of a horizontally arranged lighting device or the height of a vertically arranged lighting device and the size of the light sources/pixels, for example.

The lighting device information relating to a certain lighting device may be obtained, for example, when a user adds this lighting device to his lighting system. For example, when a new lighting device is added, an HDMI module, mobile device or bridge may download the lighting device information from the lighting device and store it. When this information is stored on a bridge, an HDMI module or mobile device may be able to later obtain this information from the bridge.

In a simpler embodiment, step <NUM> may be omitted. In a more advanced embodiment, the lighting device information may comprise further information about the lighting device, for example the pixel density (e.g. number of light pixels per meter), the pixel spacing (e.g. the distances between the centers of two individual pixels) and/or indicate optical properties of the pixels (e.g. the beam width and beam angle of the pixelated light sources).

A step <NUM> comprises obtaining a distance between each light source, i.e. each pixel, of the lighting device and a surface, e.g. a wall. Typically, one or more device distances are obtained from a sensor or from a user device. The distance between each light source of the lighting device and the surface may be the same. Alternatively, a different distance per light source may be determined based on the obtained one or more device distances.

The distance between the pixelated light sources and the to be illuminated area, i.e. the surface, can be determined in various ways. In a simple embodiment, the user provides input on the distance for example via a smart phone UI. This can be an input of a numeric value or an approximation e.g. by selecting an icon on how the TV is installed (wall-mount vs standing). In a more advanced embodiment, the distance between the light source and the to be illuminated area, i.e. the surface, is determined automatically via an external sensor, e.g. by using one or more sensors (e.g. time-of-flight sensors) embedded in or attached to the lighting device or by analyzing a (depth) image from the TV setup as captured by a smartphone camera.

A step <NUM> comprises determining, for each light source, a size and/or location for an analysis region associated with the light source based on the distance. A first analysis region with a first size and a first location is determined in step <NUM> for a first light source when the distance has a first value and a second analysis region with a second size and a second location is determined in step <NUM> for this first light source when the distance has a second value.

The first analysis region has a larger overlap with adjacent analysis regions than the second analysis region and the first value is smaller than the second value. The second size is different than the first size and/or the second location is different than the first location. Although it may be possible to create a larger overlap by only changing the location of at least one of the adjacent analysis regions, it may be easier to achieve this by (also) using a larger analysis region.

In the embodiment of <FIG>, the formula C=p+b/(<NUM>*d+<NUM>) is used to calculate the size of the analysis region for a pixel in centimeters when d is smaller than a certain threshold T, C being the size of the analysis region in centimeters, b being a blending factor, d being an (estimated or measured) numeric distance value in centimeters and p being the size of the pixel in centimeters. A minimum analysis region size is used when d is greater than or equal to threshold T.

A pixel may comprise multiple light elements, e.g. multiple LEDs. A pixel of <NUM> centimeters may comprise, for example, <NUM> LED packages, one placed each centimeter of the lighting device (e.g. light strip). All <NUM> LEDs are controlled in one color. The <NUM> LEDs may comprise three RGB LEDs and three white LEDs, for example.

For example, for horizontally arranged pixels with a blending factor of <NUM>, a pixel width of <NUM> and a distance of <NUM> to the wall, the width of a color analysis region could be calculated as follows: <NUM> + <NUM> / (<NUM>*<NUM>+<NUM>) = <NUM>. If the TV would be placed against the wall (<NUM>), the width of the color analysis region would be <NUM>. This would mean that it would take <NUM>% of both adjacent pixels. When put <NUM> from the wall, the color analysis region would be <NUM> wide (taking <NUM>% of both adjacent pixels). In this example, the threshold T is larger than <NUM>. If the threshold T is lower than <NUM> and the distance is <NUM> to the wall, the width of the color analysis region would be determined to be <NUM>.

For vertically arranged pixels, the height of the color analysis region may be determined in the manner described above in relation to the width of color analysis region for horizontally arranged pixels.

The color analysis region size in pixels may be determined by multiplying the width of the color analysis region size in centimeters with the amount of horizontal pixels of the video content divided by the width of the horizontally arranged lighting device in centimeters or multiplying the height of the color analysis region size in centimeters with the amount of vertical pixels of the video content divided by the height of the vertically arranged lighting device in centimeters.

For horizontally arranged pixels, the height of the color analysis region may be determined independent of the distance to the wall, e.g. may be a fixed value, or may be determined based on the width of the color analysis region, for example. For vertically arranged pixels, the width of the color analysis region may be determined independent of the distance to the wall, e.g. may be a fixed value, or may be determined based on the height of the color analysis region, for example.

The horizontally and vertically arranged pixels may be part of light strips attached the display device or may be part of a lighting device located a bit farther from the display device, e.g. a floor standing luminaire like Hue Signe that is put near a wall and joins the entertainment experience.

In a variant on this embodiment, the ratio between pixel (light source) size and display size is further used as parameter. For instance, the above-mentioned formula may be modified such that C is a function of p (size of a pixel), d (distance to the surface) and r (ratio between display size and pixel size). In a first implementation, the larger the size of the pixel relative to the size of the display, the smaller the overlap percentage. For example, with a distance of <NUM> to a wall, for a TV with a <NUM>-inch display (<NUM> inch being the diagonal dimension), the overlap between adjacent analysis regions might be <NUM>% for a <NUM> pixel and <NUM>% for a <NUM> pixel and for a TV with a <NUM>-inch display, the overlap between adjacent analysis regions might be <NUM>% for a <NUM> pixel and <NUM>% for a <NUM> pixel. Information indicating the display size may be obtained in step <NUM> or in a separate step performed before, after, or (partly) in parallel with step <NUM>, for example.

In a simpler embodiment, the determination of the color analysis region size may be a simple binary decision. If the TV (with attached or embedded light sources) is mounted on a stand, the color analysis region is one-to-one mapped to the pixel. So, if a pixel is <NUM> wide, the color analysis region has a corresponding width. If the TV is mounted against the wall, the color analysis region is enlarged and takes <NUM>% of the color analysis region of both adjacent pixels.

In a slightly less simple embodiment, a mapping table may be used, where the percentage of overlap is given for multiple ranges of distances to the surface. For example, a distance between <NUM> and <NUM> may be mapped to an overlap percentage of <NUM>%, a distance between <NUM> and <NUM> may be mapped to an overlap percentage of <NUM>%, etc. The size of the analysis region may then be determined based on the size of the pixel and the determined overlap percentage.

In a more advanced embodiment, a more complex function is used. Furthermore, the blend factor b may be variable and selected by a user or by the system, for example based on type of content.

A step <NUM> comprises checking whether there is a further lighting device to be controlled based on an analysis of video content for which the analysis region(s) have not been determined yet. If so, steps <NUM> and <NUM> are repeated for this further lighting device. If not, a step <NUM> is performed. Step <NUM> may be performed in parallel with at least part of one or more of steps <NUM>-<NUM>, before step <NUM> is performed or between steps <NUM> and <NUM>, for example.

Step <NUM> comprises determining a characteristic of the video content by analyzing the video content, typically a video frame of the video content, in an analysis region associated with one of the light sources. In the embodiment of <FIG>, a color is extracted from the video content in the analysis region. Various color extraction methods can be used, such as taking the average or trimean color in this region. The color extraction method could also vary depending on the amount of overall and absolute size of the analysis regions. For example, for smaller regions with no overlap, taking the average might work best (a stable but desaturating method), while for larger overlapping regions, a trimean method might work best (resulting in less stable but more saturated colors).

If desired, the size and/or locations of the color analysis region determined in step <NUM> may be adjusted in step <NUM>, e.g. based on color and/or brightness contrast and/or the number of edges in the content. For example, if a video frame contains high contrasting elements that are aligned with light pixels, the analysis regions could be reduced even in the case of a short distance to the wall. Similarly, if content is already very smooth, overlapping regions will not be beneficial and will only result in desaturation of the light effects. This analysis can be done per pixel and thereby allows an overlap to be reduced for some pixels and to be increased for other pixels. However, this will normally only be possible if the system can analyze the content fast enough, as it will need to be done every video frame.

A step <NUM> comprises determining a light effect to be rendered on this light source based on the characteristic. If a color is extracted in step <NUM>, this color may be used as color for the light effect to be rendered on the light source.

A step <NUM> comprises checking whether there is a further light source that has been associated with an analysis region and for which no light effect has been determined yet. If so, steps <NUM> and <NUM> are repeated for this further light source. If not, a step <NUM> is performed. Step <NUM> comprises controlling the light sources to render the light effects determined in step <NUM> by transmitting a light control command specifying one of the light effects to a corresponding light source, either directly or to the lighting device that comprises the light source.

A step <NUM> comprises checking whether the end of the video content has been reached. If not, steps <NUM>-<NUM> are repeated for the next part, e.g. next frame, of the video content.

<FIG> shows an example of a video frame <NUM> of the video content being rendered, e.g. on display <NUM> of <FIG>. <FIG> shows examples of analysis regions <NUM>-<NUM> that may be used to extract characteristics from video frame <NUM>. In this example, multiple regions of the screen are mapped to different lighting devices and each analysis region is analyzed separately, e.g. an average color is extracted from each analysis regions. For example, the analysis regions <NUM> to <NUM> may be mapped to pixels <NUM>-<NUM> of lighting device <NUM> of <FIG>, the analysis regions <NUM> to <NUM> may be mapped to pixels <NUM>-<NUM> of lighting device <NUM> of <FIG>, and the analysis regions <NUM> to <NUM> may be mapped to pixels <NUM>-<NUM> of lighting device <NUM> of <FIG>.

<FIG> shows further examples of analysis regions that may be used to extract characteristics from video frame <NUM>. In this example, the analysis regions <NUM> to <NUM> may be mapped to pixels <NUM>-<NUM> of lighting device <NUM> of <FIG>, the analysis regions <NUM> to <NUM> may be mapped to pixels <NUM>-<NUM> of lighting device <NUM> of <FIG>, and the analysis regions <NUM> to <NUM> may be mapped to pixels <NUM>-<NUM> of lighting device <NUM> of <FIG>.

The analysis regions <NUM>-<NUM> of <FIG> are larger than the analysis regions <NUM>-<NUM> of <FIG> and as a result, there is overlap between adjacent analysis regions in <FIG>, while there is no such overlap between adjacent analysis regions in <FIG>. Such an overlap is beneficial if the distance between the pixels of the pixelated lighting device and the surface, e.g. wall, is relatively small.

In the example of <FIG> and <FIG>, each analysis region has the same size. However, it is also possible to use differently sized analysis regions, as shown in <FIG>. Of the analysis regions <NUM>-<NUM> at the left side of the video frame, analysis region <NUM> is largest and analysis region <NUM> is smallest.

<FIG> also demonstrates that it is possible to increase the size of an analysis regions without increasing the overlap between adjacent analysis regions. For example, if a pixelated lighting device is far enough from the surface so that overlap between adjacent analysis regions is not needed, the size of an analysis region may be still increased to focus the light effect more towards ambiance rather than focusing solely on the colors that are close to the side of screen where the light source is located. So, without changing the overlap, the size of the analysis region is changed such that it takes up a bigger part of the video frame without impacting adjacent analysis regions.

In the example of <FIG>, the pixels/light sources of the lighting device have different distances to the surface. For example, a pixelated light device may lean against a wall, i.e. has an angle with respect to the wall, may be attached to the back of a curved display, or may be placed behind a display in a corner of two walls. In the example of <FIG>, the light source associated with analysis region <NUM> is farthest from the surface and the light source associated with analysis region <NUM> is closest to the surface. The farther away the light source is from the wall, the larger effect it creates (on the wall). So, it may sometimes be beneficial to analyze a larger part of the video frame even if the distance to the wall is large such that the large effect on the wall, instead of reflecting only small part of the video frame on the side, reflects a larger part of the video frame.

In the example of <FIG>, each pair of adjacent analysis regions has the same overlap. However, it is also possible to use different overlaps for different pairs of adjacent analysis regions, as is shown in <FIG>. Of the analysis regions <NUM>-<NUM> at the left side of the video frame, adjacent analysis regions <NUM> and <NUM> have the largest overlap and adjacent analysis regions <NUM> and <NUM> have the smallest overlap. This may be beneficial, for example, if the pixels/light sources of the lighting device have different distances to the surface.

By determining the distance to the surface for each light source individually, it becomes possible to use different overlaps for different pairs of adjacent analysis regions. In the example of <FIG>, the light source associated with analysis region <NUM> is farthest from the surface and the light source associated with analysis region <NUM> is closest to the surface. In the example of <FIG>, all analysis regions are of the same size but they stack on top of each other differently; the overlap area between the adjacent pixels that are farther from the wall is relatively small and the overlap area between the adjacent pixels that are closer to the wall is relatively large. Thus, only the position, and not the size, of the analysis region depends on the distance to the surface.

<FIG> also demonstrates that it is possible to change the overlap between two adjacent analysis areas without changing the total overlap between all analysis areas. When the angle of the pixelated lighting device with respect to the wall is increased and the distance between the light source associated with analysis region <NUM> is therefore increased, the overlap between adjacent analysis areas <NUM> and <NUM> may be decreased and the overlap between adjacent analysis areas <NUM> and <NUM> may be increased, thereby keeping the total overlap the same.

In the example of <FIG>, there is still a small overlap between adjacent analysis areas <NUM> and <NUM>. However, it is also possible for analysis area <NUM>, associated with the light source farthest from the surface, not to overlap with the adjacent analysis area <NUM> at all. Moreover, the overlap between adjacent areas <NUM> and <NUM> may even be larger than shown in <FIG>.

In the examples of <FIG>, all analysis regions have a rectangle shape. It is also possible to use a different shape or different shapes, as is shown in <FIG> with respect to analysis regions <NUM>-<NUM>.

A part of a second embodiment of the method of controlling a light source based on an analysis of image content in an analysis region of the image content while a display displays the image content is shown in <FIG>. In this second embodiment, step <NUM> of <FIG> is implemented by sub steps <NUM>-<NUM>. After step <NUM>, steps <NUM>-<NUM> are performed, as shown in <FIG>.

Step <NUM> comprises obtaining one or more device distances between the lighting device and the surface. Typically, these one or more device distances are obtained from a sensor or from a user device, e.g. a user input device. If one end of a vertically arranged lighting device needs to lean against the wall and a device distance is obtained for the other end, then a second device distance of zero may be obtained automatically.

Next, a step <NUM> comprises checking whether one device distance was obtained in step <NUM> or whether multiple device distances were obtained in step <NUM>. If is determined in step <NUM> that a single device distance was obtained in step <NUM>, a step <NUM> is performed next. Step <NUM> comprises determining the distances between the further light sources and the surface based on the single device distance determined in step <NUM>. Typically, the distances between the further light sources and the surface are equal to the single device distance. Step <NUM> is performed after step <NUM>.

If is determined in step <NUM> that multiple device distances were obtained in step <NUM>, a step <NUM> is performed next. Step <NUM> comprises calculating an average of the multiple device distances determined in step <NUM>. If two device distances, e.g. of the two ends of the lighting device, were determined in step <NUM>, then a single average is calculated in step <NUM>. If more than two device distances were determined in step <NUM>, then multiple averages may be calculated in step <NUM>. Next, a step <NUM> comprises determining the distances between the further light sources and the surface based on the average(s) calculated in step <NUM>. If a single average was calculated in step <NUM>, the distances between the further light sources and the surface are typically equal to this single average. Step <NUM> is performed after step <NUM>.

A part of a third embodiment of the method of controlling a light source based on an analysis of image content in an analysis region of the image content while a display displays the image content is shown in <FIG>. Compared to the second embodiment of <FIG>, steps <NUM> and <NUM> have been replaced with steps <NUM> and <NUM>, steps <NUM> and <NUM> have been added before step <NUM>, and step <NUM> is implemented by a step <NUM>.

Step <NUM> comprises determining the positions of the light sources on the lighting device. Step <NUM> comprises determining the distance between each light source and the surface based on at least two of the device distances determined in step <NUM> and the position of the light source determined in step <NUM>. The device distances are indicated with respect to a reference point on the lighting device, e.g. the edges of the lighting device. If the position of a light source is between two reference points, the distance between this light source and the surface is determined based on the two device distances corresponding to these reference points and the distances between the position of the light source and these reference points.

Step <NUM> comprises estimating an amount of light overlap between light projected on the surface by adjacent light sources based on the distances determined in step <NUM>. Obtained lighting device information, see step <NUM> of <FIG>, may be used to estimate the amount of overlap. For example, beam width, beam angle and distance to the wall may be used to more accurately calculate the amount of light overlap between the illumination regions of different pixels as projected on the wall. The beam width and beam angle normally have an effect on the optical mixing. For instance, more optical mixing occurs when using a wider beam and/or when the beam angle is sharper (due to the longer distance to the surface).

Step <NUM> comprises determining an amount of desired region overlap between the adjacent analysis regions of the adjacent light sources based on the estimated amount of light overlap. When more optical mixing occurs, the amount of desired region overlap may be lower. Step <NUM> comprises determining the sizes and/or locations for the analysis regions based on the amount of desired region overlap. Step <NUM> is performed after step <NUM>.

A fourth embodiment of the method of controlling a light source based on an analysis of image content in an analysis region of the image content while a display displays the image content is shown in <FIG>. In this fourth embodiment, additional steps <NUM>, <NUM> and/or <NUM> are optionally performed before step <NUM> and step <NUM> is implemented by a step <NUM>.

Step <NUM> comprises determining the size of the light source(s) of the lighting device. Step <NUM> comprises determining the size of the lighting device. Step <NUM> comprises determining a distance between the light source(s) and the display. Step <NUM> comprises determining the sizes and/or locations for the analysis regions based on the distances determined in step <NUM> and optionally based on the size of the light source(s) of the lighting device determined in step <NUM>, the size of the lighting device determined in step <NUM>, and/or the distance between the light source(s) and the display. Step <NUM> is performed after step <NUM>.

The embodiments of <FIG> and <FIG> differ from each other in multiple aspects, i.e. multiple steps have been added or replaced. In variations on these embodiments, only a subset of these steps is added or replaced and/or one or more steps is omitted. For example, one or more of steps <NUM>, <NUM> and <NUM> may be added to the embodiments of <FIG> and/or <FIG> and steps <NUM>, <NUM>, <NUM> of <FIG> may be added to the embodiment of <FIG> and/or omitted from the embodiment of <FIG>.

<FIG> depicts a block diagram illustrating an exemplary data processing system that may perform the method as described with reference to <FIG> and <FIG>.

Claim 1:
A system (<NUM>,<NUM>) for controlling a light source (<NUM>) based on an analysis of image content (<NUM>) in an analysis region (<NUM>,<NUM>) of said image content (<NUM>) while a display (<NUM>) displays said image content (<NUM>), said system (<NUM>,<NUM>) comprising:
the light source (<NUM>), wherein the light source (<NUM>) is configured to project light on a surface, wherein the surface is a wall;
at least one input interface (<NUM>,<NUM>);
at least one output interface (<NUM>,<NUM>); and
at least one processor (<NUM>,<NUM>) configured to:
- obtain said image content (<NUM>) via said at least one input interface (<NUM>,<NUM>), characterized in that the at least one processor is further configured to:
- obtain a distance (<NUM>-<NUM>) between said light source (<NUM>) and the surface (<NUM>) from a sensor (<NUM>-<NUM>) or from a user device (<NUM>),
- determine a size and/or location for said analysis region (<NUM>,<NUM>) based on said distance (<NUM>-<NUM>),
- determine a characteristic of said image content by analyzing said image content (<NUM>) in said analysis region (<NUM>,<NUM>),
- determine a light effect based on said characteristic of said image content, and
- control, via said at least one output interface (<NUM>,<NUM>), said light source (<NUM>) to render said light effect onto the surface.