Patent Description:
The present disclosure generally relates to video technology. More particularly, embodiments of the present disclosure relate to the display of video signals in a viewing environment with variable ambient light.

In a typical content creation pipeline, video is color graded in a low light ambient environment, typically with <NUM> nits ambient for color grading High Dynamic Range (HDR) video and <NUM> nits ambient for color grading Standard Dynamic Range (SDR) video. In practice, viewers may display content in a variety of ambient environments, say, at <NUM>-<NUM> nits (e.g., watching a movie in a relatively dark home theater), at <NUM>-<NUM> nits (e.g., watching a movie in a relatively bright living room), or higher (e.g., watching a movie on a tablet in a very bright room or outside, in daylight).

As appreciated by the inventors here, improved techniques are desired for displaying video to compensate for ambient light conditions in a viewing environment.

<CIT> Al relates to a method for adapting color appearance of a display for low luminance conditions includes operating a projection to display images on a display surface; detecting ambient light conditions and displayed image brightness; determining low luminance conditions based on the detected ambient light conditions and the detected display brightness; determining changes in color appearance to be applied to the displayed images based on the low luminance conditions, a model of photopic vision of the human eye, and a model of mesopic vision of the human eye; and applying the determined changes in the color appearance to image data using an image processor that alters the image data for the projected images.

In <CIT> Al, in one embodiment. an ambient light sensor with first and second photo-detectors is disclosed. The first mid second photo-detectors may have different depths and respond differently to light having a specific wavelength. The ambient lights sensor may further comprise a circuit configured to detect light of a specific wavelength. In another embodiment, the first and second photo-detectors may be configured such that each of the first and second photo-detectors detects coupling current from each other. In yet another embodiment, in addition to the first and second photo-detectors, a third photodetector may be formed proximate to at least one of the first and second photo-detectors such that coupling photo-current is detected therein by the at least one of the first and second photo-detectors.

The invention is set out by claim <NUM>. Preferable embodiments are set out by the dependent claims.

Embodiments of the present disclosure relate to a display management (DM) module and a corresponding method.

In an embodiment, a DM module is configured to receive an input video signal comprising a sequence of video frames, and a signal indicative of intensity of ambient light. The DM module is further configured to determine whether a current video frame of the sequence of video frames immediately follows a scene change. The DM module is further configured to transform the input video signal to an output video signal adapted for a target display apparatus. Transforming the input video signal to the output signal includes applying ambient light compensation to the input video signal in dependence on the signal indicative of the intensity of ambient light. The output video signal is output by the DM module for use by the target display apparatus. The DM module is configured to adjust said ambient light compensation applied to the input signal in dependence on the signal indicative of intensity of ambient light only in response to determining that the current video frame of the sequence of video frames immediately follows a scene change.

Further embodiments of the present disclosure relate to a display apparatus including said DM module, an ambient light sensor and a display, and to a computer program for executing said method.

Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the accompanying drawings, wherein like reference numerals refer to similar elements, and in which:.

Numerous specific details are described below to provide a thorough understanding of the present invention. However, the present invention may be practiced without these specific details. In addition, well-known parts may be described in less exhaustive detail. The figures are schematic and comprise parts relevant for understanding the present disclosure, whereas other parts may be omitted or merely suggested.

<FIG> show examples of display apparatuses <NUM>, <NUM>, <NUM> that apply ambient light compensation. In the context of this disclosure, a display apparatus may for example comprise a television, a laptop, a tablet, a smartphone, a projector or any other electronic device for displaying video.

As used herein, the term 'dynamic range' (DR) may relate to a capability of the human visual system (HVS) to perceive a range of intensity (e.g., luminance, luma) in an image, e.g., from darkest grays (darks or blacks) to brightest whites (highlights). In this sense, DR relates to a 'scene-referred' intensity. DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular breadth. In this sense, DR relates to a 'display-referred' intensity. Unless a particular sense is explicitly specified to have particular significance at any point in the description herein, it should be inferred that the term may be used in either sense, e.g. interchangeably.

As used herein, the term "display management" denotes the process of adapting images or pictures (e.g. frames) of an input video signal for a target display apparatus. Display management may comprise mapping images or pictures of an input video signal of a first dynamic range (e.g., <NUM> nits) to a display apparatus having a second dynamic range (e.g., <NUM> nits). Display management may comprise tone mapping and/or gamut mapping.

Display apparatus <NUM> (<FIG>) comprises a display <NUM>, an ambient light sensor <NUM> and a display management (DM) module <NUM>. In this example, the ambient light sensor <NUM> communicates with the DM module <NUM>, and the DM module <NUM> communicates with the display <NUM>. The DM module <NUM> receives an input video signal <NUM> comprising a sequence of video frames. Optionally, the DM module <NUM> further receives metadata <NUM> related to the input video signal <NUM>. For example, the DM module <NUM> may receive the input video signal <NUM> and metadata <NUM> from a decoder, internal or external to apparatus <NUM>.

The DM module <NUM> receives a signal <NUM> indicative of ambient light from sensor <NUM>. The signal is at least indicative of intensity of ambient light registered by the sensor <NUM>. Optionally, the signal may further be indicative of a color of ambient light registered by the sensor.

Typically, the sensor <NUM> is positioned on a front side of the apparatus <NUM>, that corresponds to the front side of the display <NUM>, to measure the illumination on the display screen, which is the ambient component that elevates the black level of the display as a function of reflectivity. In such a scenario, the sensor <NUM> measures the intensity, and optionally the color, of light present at the front side of the display <NUM>. However, the present disclosure is not limited to a particular position of the sensor <NUM>, and alternatively the sensor <NUM> may be positioned at a different location, e.g. along the edge of the display apparatus or on the back of the apparatus.

In the examples of the drawings, a single sensor <NUM> is shown. However, the present disclosure is not limited to the use of a single sensor, and multiple sensors <NUM> may be used to generate a signal <NUM> indicative of ambient light for use by the DM module <NUM>.

The DM module <NUM> is configured to transform the input video signal <NUM> to an output video signal <NUM> adapted for display <NUM>. The transformation may comprise tone mapping, e.g. for adjusting the dynamic range of the input signal <NUM> to the dynamic range supported by the display <NUM>, and/or gamut mapping e.g. for adjusting the colors of the input signal <NUM> to the color gamut supported by the display <NUM>.

The optional metadata <NUM> may include parameters for the tone mapping and/or the gamut mapping operations. The DM module <NUM> may thus adjust the tone mapping and/or the gamut mapping operations to be applied to the input video signal <NUM> based on metadata <NUM>. The tone mapping and/or the gamut mapping may further use parameters relating to the characteristics of the display <NUM>, which may be stored internally in apparatus <NUM> and are thus accessible by DM module <NUM>.

An example of display management including tone mapping is described in <CIT>, entitled "Display management for high dynamic range video". <CIT> describes a tone mapping curve that maps input luma values to output luma values according to <MAT> wherein C<NUM>, C<NUM> and C<NUM> are constants, Yin is the input luma value, Yout is the output luma value and n and m are parameters. C<NUM>, C<NUM> and C<NUM> are based on the brightness characteristics of the reference (or source) display and on the brightness characteristics of the target display. The brightness characteristics of the reference (or source) display are typically extracted from input metadata, e.g. a minimum brightness (Smin), an average or mid value (Smid) and a maximum brightness (Smax). The brightness characteristics of the target display, e.g. a minimum brightness (Tmin), an average or mid value (Tmid) and a maximum brightness (Tmax), are typically stored internally in the display apparatus and are therefore accessible by the DM module. The parameters Smin, Smid, Smax, Tmin, Tmid, Tmax define three anchor points of the tone mapping curve, namely (Smin, Tmin), (Smid, Tmid), (Smax, Tmax), the three anchor points in turn determining the value of the constants C<NUM>, C<NUM> and C<NUM>.

A further example of display management including tone mapping is described in <CIT>, entitled "Tone mapping for high dynamic range images". <CIT> describes a tone mapping curve that uses the same three anchor points, but has four segments in total: a first linear segment for values lower than (Smin, Tmin), a first spline from (Smin, Tmin) to (Smid, Tmid), a second spline from (Smid, Tmid) to (Smax, Tmax) and a second linear segment for values larger than (Smax, Tmax).

Other examples of display management including tone mapping are described in <CIT>, entitled "Display management methods and apparatus".

The transformation performed by DM module <NUM> may further comprise applying ambient light compensation to the input video signal <NUM>. The ambient light compensation is applied in response to the signal <NUM> indicative of at least the intensity of the ambient light, as received from sensor <NUM>. The DM module <NUM> applies the ambient light compensation to the input signal <NUM>, e.g. in addition to a tone mapping operation, to thereby generate the output video signal <NUM> for rendering by the display <NUM>. The ambient light compensation may be applied prior to or after applying a tone mapping and/or a gamut mapping. In another example, the ambient light compensation is combined with the tone mapping and/or gamut mapping into a single mapping operation. In such an example, the mapping applied by the DM module <NUM> to the input video signal <NUM> has parameters that depend on the signal <NUM>, and parameters related to the tone mapping and/or gamut mapping, that may e.g. be included or derived from metadata <NUM> or derived from the input video signal <NUM> itself.

The DM module <NUM> is further configured to determine whether a current video frame of the input video signal <NUM> immediately follows a scene change (also known as "scene cut").

In other words, DM module <NUM> may be configured to determine whether a current video frame starts a new scene, said new scene following a previous scene that includes the video frame immediately preceding the current video frame.

A scene comprises a set of consecutive video frames having similar brightness characteristics and/or similar color characteristics. Scenes may be defined during authoring of the video. For example, scene changes may be included in an Edit Decision List (EDL) which was used, e.g. by the director and/or editor, to create the movie from a number of different shots. An indication of these scene changes may be included in the metadata of the video signal.

Therefore, in an embodiment, metadata <NUM> comprises metadata indicating a scene change. DM module <NUM> is configured to determine whether a current video frame of the sequence of video frames included in video signal <NUM> follows a scene change using the metadata indicating the scene change. For example, the metadata may comprise frame-by-frame metadata including a flag that indicates whether the current frame immediately follows a scene change, i.e. whether the current frame is the first frame of a new scene. In another example, the metadata comprise frame-by-frame metadata including an identifier that identifies a scene, e.g. a scene number, with which the respective frame is associated. In such a scenario, the DM module <NUM> may determine whether a current video frame is in a different scene than a video frame immediately preceding the current video frame, by comparing the scene identifier of the current frame with the scene identifier of the previous frame. If the scene identifiers of the two consecutive video frames differ, then it is determined that the current video frame immediately follows a scene change.

In another embodiment, a scene change is determined by the DM module <NUM> by comparing characteristics of the current video frame to characteristics of one or more previous video frames. For example, the characteristics may comprise brightness characteristics and/or color characteristics.

In an example, the DM module <NUM> determines whether a variation of the characteristics between the current frame and the previous frame satisfies a predetermined threshold condition, e.g. exceeds a predetermined threshold. If the variation satisfies the threshold condition, then the DM module <NUM> determines that a scene change occurs between the current video frame and the previous video frame, i.e. the current scene immediately follows a scene change.

In another example, the DM module <NUM> determines for each frame j the average luminance value <Ij>, and compares the difference between the average luminance value of the current frame <Ij> and the average luminance value of the previous frame <If-<NUM>> with a threshold T. If the difference |<Ij> - <Ij-<NUM>>| exceeds the threshold T, i.e. |<If> - <If-<NUM>>| > T, then the DM module <NUM> determines that the current frame immediately follows a scene change. Instead of the average luminance value, a different function of the luma values of the frame may be used. For example, instead of the average luminance value, a mid luminance value may be used, that may e.g. be computed as the mean of the maximum luminance value (Imax) and the minimum value (Imin) of the video frame, i.e. (Imax + Imin)/<NUM>.

In a further example, the characteristics of the video frames are derived from metadata indicative of said characteristics. For example, referring to the examples of the previous paragraph, the mid luminance value or the average luminance value may be available as frame-by-frame metadata included in the metadata <NUM> received by the DM module <NUM>, e.g. in the form of the metadata value Smid described above.

In another embodiment, DM module <NUM> is capable of both determining a scene change based on said metadata indicating a scene change and determining a scene change by said comparison of characteristics of the current video frame with characteristics of one or more previous video frames. In a further example of this embodiment, the DM module <NUM> is configured to - in presence of metadata indicating a scene change - base the determination of the scene change on said metadata instead of determination by said comparison. An advantage of metadata indicating a scene change overruling automatic detection by the DM module <NUM> is that the display management process more closely follows the creator's intent.

In <FIG>, elements that are similar to elements of <FIG> have been given the same reference numeral, increased by <NUM> or <NUM>, respectively. Those elements have the same characteristics, configuration and functionality as described with respect to <FIG>, unless described otherwise in the description below.

Display apparatus <NUM> (<FIG>) differs from display apparatus <NUM> in that it comprises a display controller <NUM>, which controls the operation of the display <NUM>. The DM module <NUM> communicates with the display controller <NUM>, and the display controller communicates with the display <NUM>. The DM module <NUM> receives input video signal <NUM> that comprises a sequence of video frames. Further, the DM module <NUM> optionally receives metadata <NUM>, e.g. including metadata for dynamic range mapping and/or metadata indicating a scene change. The DM module <NUM> receives from display controller <NUM> information <NUM> on the characteristics of the display <NUM>.

In an example, information <NUM> comprises brightness characteristics of the target display, e.g. a minimum brightness (min), an average or mid value (mid) and a maximum brightness (max), for use by the DM module <NUM> in tone mapping. In this example, the DM module <NUM> applies a tone mapping to the input video signal <NUM> based on source brightness characteristics (e.g. Smin, Smid, Smax) included in metadata <NUM>, and target brightness characteristics (e.g. Tmin, Tmid, Tmax) included in or derivable from information <NUM> received from the display controller <NUM>. In addition, the DM module <NUM> receives from ambient light sensor <NUM> a signal <NUM> indicative of intensity of ambient light (and optionally indicative of color of ambient light), and applies ambient light compensation in dependence on signal <NUM>. The DM module <NUM> transmits the tone mapped (and optionally gamut mapped) and ambient light compensated output video signal <NUM> to the display controller <NUM>. The display controller <NUM> controls the display <NUM> to render the output signal <NUM>.

Instead of using metadata <NUM> to determine the source brightness characteristics, these characteristics may be determined by the DM module <NUM> by analyzing the video signal <NUM>, e.g. by determining the min, mid and max value of one or more images of the video signal <NUM>. Further, as described above in relation to display apparatus <NUM>, instead of using optional metadata <NUM> a scene change may be determined by the DM module <NUM> of display apparatus <NUM> by comparing characteristics of the current video frame to characteristics of one or more previous video frames.

In an example, display <NUM> comprises a backlight. For example, display <NUM> may be an LCD type display. In this scenario, the display controller <NUM> is configured to control the backlight of the LCD display based on the video signal <NUM> received from the DM module <NUM>. In other examples, display <NUM> may not include a backlight. For example, display <NUM> may be an OLED type display.

Display apparatus <NUM> (<FIG>) differs from display apparatus <NUM> in that the ambient light sensor <NUM> communicates with display controller <NUM>, instead of with DM module <NUM>. The display controller <NUM> receives signal <NUM> indicative of intensity of ambient light from sensor <NUM>. The display controller <NUM> transmits information <NUM> on the intensity of ambient light (and optionally information on the color of ambient light) to the DM module <NUM>. In the example shown, the signal <NUM> of sensor <NUM> is passed on to DM module <NUM>. Alternatively, the display controller <NUM> may process the signal <NUM> of sensor <NUM> to generate a parameter indicative of the intensity of ambient light, and communicate said parameter to DM module <NUM>.

Like in <FIG>, also in <FIG> the display controller <NUM> communicates information on the characteristics of display <NUM> to DM module <NUM>, e.g. brightness characteristics are communicated for a tone mapping operation of DM module <NUM>. DM module <NUM> applies a tone mapping operation and ambient light compensation to input video signal <NUM>, wherein the tone mapping operation is controlled via the parameters of source characteristics included in metadata <NUM> (or determined by the DM module <NUM> directly from the video signal <NUM>) and parameters of the characteristics of the display <NUM> included in signal <NUM>, and wherein the ambient light compensation is applied in dependence on the signal <NUM> indicative of intensity of ambient light received from display controller <NUM>.

Display apparatuses <NUM>, <NUM> and <NUM> comprise displays <NUM>, <NUM>, <NUM>. In some embodiments, a display apparatus <NUM>, <NUM>, <NUM> may comprise a projector instead of a display. For example, the projector comprises a laser projector.

In the above embodiments, ambient light sensors <NUM>, <NUM>, <NUM> are depicted as being part of the display apparatus <NUM>, <NUM>, <NUM>. Alternatively, the ambient light sensor may be external to the display apparatus <NUM>, <NUM>, <NUM>, in which case the DM module <NUM>, <NUM>, <NUM> is configured to receive a signal indicative of the intensity of ambient light from the external ambient light sensor, e.g. via a wired or wireless connection <NUM>, <NUM>, <NUM>.

As ambient light is typically not constant over time, also the signals <NUM>, <NUM>, <NUM> indicative of intensity of ambient light will typically vary over time. However, compensating for the instantaneous ambient light may result in "flicker": noticeable rapid fluctuations of the brightness of the displayed image. Measurement noise inherent to ambient light sensors may for example lead to adjusting the brightness in response to the measured ambient light intensity, even when the intensity of ambient light has not actually changed. In another example, if the ambient light sensor is momentarily blocked, e.g. by a person standing in front of the ambient light sensor, the brightness may go down significantly, only to be ramped up again immediately after the person steps away and no longer blocks the sensor.

In the embodiments of <FIG>, the DM modules <NUM>, <NUM>, <NUM> are configured to adjust the ambient light compensation only in response to determining that the current video frame of the sequence of video frames immediately follows a scene change. In other words, adjustment of ambient light compensation is synchronized with scene changes of the input video signal. The inventors have found that adjusting the ambient light compensation is not noticeable to viewers when restricting the adjustment to scene changes. In a typical video, scene changes occur every <NUM> to <NUM> seconds. The inventors have found that even when adjusting the ambient light compensation every <NUM> to <NUM> seconds, the changes in ambient light compensation are not noticeable if the adjustments are restricted to scene changes. Therefore, by synchronizing the ambient light adjustment to scene changes, flicker is avoided or at least reduced. Furthermore, a rapid response to changes in ambient light is ensured, as the next scene will have an ambient light adjustment adapted to the actual ambient light conditions.

The above description relates to sensor adaptive ambient light compensation, i.e. ambient light compensation that is automatically adjusted in response to a signal indicative of the intensity of ambient light. Further types of adjusting brightness and/or color of the displayed image may be available. For example, the DM module <NUM>, <NUM>, <NUM> and/or display controller <NUM>, <NUM> may allow a user to manually set a brightness level and/or color settings. In an embodiment, the DM module <NUM>, <NUM>, <NUM> does not restrict said further types of adjusting brightness and/or color to scene changes, e.g. a brightness setting may be adjusted by a user mid-scene.

Optionally, a form of temporal smoothing may be applied to the signal <NUM>, <NUM>, <NUM> of the sensor <NUM>, <NUM>, <NUM>, while at the same time restricting adjustment of the ambient light compensation to scene changes as described above. In an embodiment, a temporal smoothing filter is applied to the signal indicative of intensity of ambient light, wherein the filter is reset in response to determining that the current video frame immediately follows a scene change. For example, the smoothing filter comprises a moving average filter, a weighted moving average filter, an exponential smoothing filter. Resetting the filter may include clearing previously stored values of the filter, e.g. setting these values to zero or to the current measured value.

For example, a cumulative average filter is implemented as: <MAT> wherein I(n) denotes the ambient intensity measured by the sensor at discrete time instance n, and y(n) denotes the cumulative average, wherein y(<NUM>) may be initialized to I(<NUM>). In this example, in response to determining that the current video frame immediately follows a scene change, y(n) is reset to I(n).

<FIG> show exemplary display apparatus for ambient light compensation. However, the present disclosure is not limited to these examples. In particular, additional components of display apparatuses that are not relevant to ambient light compensation have not been described.

<FIG> illustrates a method for applying ambient light compensation. This method may be implemented by a display apparatus. In an embodiment, the method is executed by a display management module of a display apparatus. For example, the method may be executed by any of the DM modules <NUM>, <NUM>, <NUM> described above.

The method starts at step <NUM>. A current video frame of a sequence of video frames is received in step <NUM>. It is determined whether a scene change has occurred <NUM>. In particular, in step <NUM> it is determined whether the current frame immediately follows a scene change, which will be described in more detail with respect of <FIG>.

In response to determining that the current video frame immediately follows a scene change, the method moves to step <NUM>, wherein an ambient light signal is received. The ambient light signal comprises at least an indication of intensity of ambient light. Optionally, the ambient light signal further includes an indication of color of ambient light. In step <NUM>, the current settings of ambient light compensation are adjusted in dependence on the ambient light signal received in step <NUM>. The method then moves to step <NUM>, wherein the current settings of the ambient light compensation - updated in step <NUM> - are applied.

In response to determining that the current video frame does not immediately follow a scene change, the method moves from step <NUM> to step <NUM>. In such case, the settings for ambient light compensation are not adjusted, i.e. the previous settings for ambient light compensation are maintained. Therefore, in this case, at least the step <NUM> is omitted. At step <NUM>, the current settings are applied, which in this scenario are the same ambient light settings as for the previous frame.

After step <NUM>, it is determined whether the current frame is the final video frame in step <NUM>. If it is not, i.e. a next video frame is available, the next video frame is set as the current video frame and the method loops back to step <NUM>. If the current frame is the final video frame, the method ends at step <NUM>.

In the example of <FIG>, step <NUM> is performed immediately prior to step <NUM>. However, step <NUM> may alternatively be performed between steps <NUM> and <NUM>. In another example, step <NUM> is performed prior to step <NUM>, as a first step of the illustrated loop over the video frames, followed by step <NUM>.

<FIG> shows a first example of sub-steps of step <NUM>, i.e. the step of determining whether the current video frame immediately follows a scene change. In the example of <FIG>, metadata is received in sub-step <NUM>-<NUM>. The metadata includes metadata indicative of a scene change. In an example, the metadata indicative of a scene change comprises a frame-by-frame flag indicating whether the current frame starts a new scene. In sub-step <NUM>-<NUM>, the method determines whether the flag is set, i.e. whether the current frame starts a new scene. In response to determining that the flag is set, the method continues to step <NUM> as described above. In response to determining that the flag has not been set, the method continues to step <NUM> as described above. In another example, the metadata indicative of a scene change comprises a frame-by-frame identifier for a scene to which the current frame is associated. For example, the identifier may comprise a unique numeric, alphabetic or alphanumeric identifier to identify different scenes. In such a scenario, step <NUM>-<NUM> compares the identifier of the current video frame to the identifier of the previous video frame, i.e. the video frame immediately preceding the current video frame. If it is determined that the current video frame and the previous video frame have the same scene identifier, i.e. they are associated to the same scene, the metadata thus indicates that no scene change has occurred, and the method continues to step <NUM>. If it is determined that the current video frame and the previous video frame have a different scene identifier, i.e. they are associated with different scenes, the method continues to step <NUM>.

<FIG> shows a second example of sub-steps of step <NUM>. In sub-step <NUM>-A characteristics of the current video frame are determined. For example, brightness characteristics of the current video frame are determined. For example, a mid or average luminance value for the current video frame may be computed, or extracted from metadata, as described above.

In sub-step <NUM>-B the difference between one or more characteristics of the current video frame and the same one or more characteristics of at least one previous video frame is calculated. For example, the difference between a mid luminance of the current video frame and the mid luminance of the previous video frame is computed. In another example, the difference between a characteristic of the current video frame and an average (or weighted average) of the same characteristic of N previous video frames is computed, e.g. the difference between the mid luminance of the current video frame and an average (or weighted average) of the mid luminances of N previous frames.

In step <NUM>-C, it is determined whether the difference computed in step <NUM>-B satisfies a threshold condition. In the example shown, it is determined whether the difference exceeds a threshold. In response to determining that the difference exceeds the threshold, the method proceeds to step <NUM>. In response to determining that the difference does not exceed the threshold, the method proceeds to step <NUM>.

In another embodiment, the same functionality is implemented by determining whether the difference is smaller than a threshold. In response to determining that the difference is smaller than the threshold, the method proceeds to step <NUM>. In response to determining that the difference is not smaller than the threshold, the method proceeds to step <NUM>.

As described above, the DM module <NUM>, <NUM>, <NUM> applies ambient light compensation to the input video signal <NUM>, <NUM>, <NUM> in dependence on the ambient light intensity. For example, applying ambient light compensation comprises adjusting brightness as a function of ambient light intensity, wherein brightness is increased as ambient light intensity increases. Let vij denote a luma component of a video frame j having P pixels i = <NUM>. P - <NUM>, and let I denote the ambient light intensity. Then an output video frame may be computed as: v̂ij = vij + f(I), wherein f(I) denotes a function of ambient light intensity. Herein, the ambient light intensity I is computed on the basis of the signal <NUM>, <NUM>, <NUM> received from the ambient light sensor <NUM>, <NUM>, <NUM>. For example, the signal may be temporally smoothed as described above to obtain the ambient light intensity. The function f(I) can take on both negative and positive values. For example, when the input video signal is authored in a <NUM> nits reference environment, f(I) may take on negative values for values of I corresponding to an ambient light intensity of less than <NUM> nits, while f(I) = <NUM> for I values corresponding to an ambient light intensity of <NUM> nits and f(I) > <NUM> for I values corresponding to ambient light intensities exceeding <NUM> nits. The function f(I) may be a linear or a nonlinear function.

In an embodiment, applying ambient light compensation comprises adjusting brightness as a function of both the ambient light intensity and the intensity values of the input video frames, e.g. the pixel values of the luma component of the input video frames. Again, vij denotes the luma component of a video frame j having P pixels i = <NUM>. P - <NUM>, and I denotes the ambient light intensity. Then an output video frame may be computed as: v̂ij = g(vij, I), wherein g(vij, I) denotes a function of both ambient light intensity and luma value of pixel i of frame j.

In an embodiment, the function f(I) or g(vij, I) is implemented as a look up table (LUT).

<FIG> shows an example of a function g(vij, I) for four different levels of ambient light intensity I. In the graphs of <FIG>, the abscissa shows the input luminance expressed in units of candela per squared meter, which has a one to one correspondence to the input luma values vij. The ordinate shows the output luminance expressed in units of candela per squared meter, which has a one to one correspondence to output luma values v̂ij. The figure shows the relation between input luminance and output luminance at <NUM> nits (<NUM>), <NUM> nits (<NUM>), <NUM> nits (<NUM>), and zero nits (<NUM>).

As depicted in <FIG>, when the viewing environment matches the reference environment (e.g., <NUM> nits), function <NUM> represents a straight line with slope = <NUM>, i.e. no ambient compensation is applied. For darker (e.g. function <NUM>) or brighter (e.g. function <NUM> or function <NUM>) viewing environments, the ambient light compensation is either decreased or increased as needed. As can be seen from <FIG>, the ambient light compensation is dependent on both the ambient light intensity and the input luminance of the input video frame.

The DM module <NUM>, <NUM>, <NUM> may have access to a predetermined number of ambient-light compensation mappings, e.g. in the form of one or more LUTs or a 3D LUT, stored internally in display apparatus <NUM>, <NUM>, <NUM>, e.g. stored in a memory of DM module <NUM>, <NUM>, <NUM>. In this example, the DM module <NUM>, <NUM>, <NUM> may be configured to select one of a set of ambient light compensation mappings in dependence on the signal indicative of intensity of ambient light. For ambient light intensities not represented in the stored LUT(s), interpolation techniques may be applied to derive an appropriate LUT. For example, given pre-computed curves g(vij, I<NUM>) and g(vij, I<NUM>), for two ambient-light values, I<NUM> and I<NUM>, a new curve g(vij, I), for I<NUM> < I < I<NUM> may be generated by interpolating between the g(vij, I<NUM>) and g(vij, I<NUM>) values.

As described above, light sensors for ambient light compensation in a display apparatus are typically positioned on a front side of the display apparatus to obtain a measure of the light intensity that elevates the black level of the displayed image. However, the positioning of the sensor near the front of the display apparatus may cause the sensor to also register some of the light emitted by the display apparatus itself. Therefore, the signal generated by the sensor may not be an accurate measure of the ambient light. Instead, the signal of the sensor is dependent on both the ambient light and the light emitted by the display apparatus. As the light emitted by the display apparatus may change rapidly due to changes in the input video signal, this may also affect the output of the light sensor and - in conventional display apparatuses - may be another cause of the occurrence of flicker. The synchronization of the ambient adjustment to scene changes as described above is also effective in reducing or eliminating flicker caused by the sensor registering light emitted by the display apparatus itself. However, the signal generated by the sensor may be an inaccurate measure of the ambient light intensity. Therefore, the ambient light compensation applied based on the sensor output may increase or decrease the brightness of the displayed image more than necessary in view of the actual ambient light.

LCD displays may blank the screen at regular intervals. Therefore, the above problem of reduced accuracy may be addressed by only taking a measurement from the ambient light sensor during the time the screen is blanked. However, a disadvantage of this approach is that the amount of time the screen would be blanked is relatively short, otherwise average display brightness would suffer. If the screen is off for <NUM>% of the time, then the sensor would only be measuring the ambient light for <NUM>% of the time, and a 10x gain would need to be applied, resulting in more noise in the measurement. This is undesirable, as noise in the output signal of the sensor is a cause of flicker.

The problem of the display apparatus interfering with the measurement of the ambient light sensor increases for increasing brightness. Therefore, the interference is large for HDR display apparatuses that generally have a higher maximum brightness. Nevertheless, the problem is also present in SDR display apparatuses, and the present disclosure is therefore not limited to HDR, but covers both HDR and SDR implementations.

The inventors have realized that the accuracy of the ambient light compensation can be increased, without increasing the occurrence of flicker, by using an ambient light sensor that is less sensitive to light of wavelengths corresponding to primary colors of the display apparatus than to other wavelengths of the visible spectrum.

In order to display color images, display apparatuses comprise pixels in three or more different colors, referred to as primary colors of the display apparatus. For example, a display apparatus may comprise red (R), green (G) and blue (B) pixels. The ambient light sensor, e.g. sensors <NUM>, <NUM>, <NUM>, is of a type that is less sensitive to the primary colors of the display apparatus, e.g. to the wavelengths corresponding to the RGB colors of the pixels of a display apparatus.

The embodiment of <FIG> shows an example of such an ambient light sensor. The ambient light sensor <NUM> of <FIG> comprises a photodetector 603a and an optical filter 603b. The photodetector 603a may for example comprises a photodiode, a phototransistor, a photoresistor, one or more reverse-biased LEDs, any other suitable type of photodetector. The optical filter 603b reduces or blocks one or more of the primary colors of the display apparatus. For example, the optical filter 603b may be an optical band-stop filter (also known as optical notch filter) that attenuates light of wavelengths corresponding to the primary colors of the display apparatus. In other words, the stopbands of the optical band-stop filter correspond to the spectral power distribution of light emitted by the display apparatus. An example of such sensor characteristic is shown schematically in <FIG>. The abscissa corresponds to the wavelength, and the ordinate corresponds to sensor sensitivity. The sensor has generally a flat response across the visible spectrum, with the exception of three notches, each corresponding to one the primaries of the exemplary display apparatus.

The response of <FIG> may for example be realized using an optical band-stop filter as the optical filter 603b. In another example, instead of having a band-stop characteristic as shown in <FIG>, a sensor having a band-pass characteristic may be provided, wherein the pass bands of the sensor correspond to bands that do not include the primary colors of the display apparatus. For example, an optical band-pass filter may be provided as optical filter 603b. <FIG> shows schematically the sensitivity of a sensor having such band-pass characteristic. The solid line illustrates a first pass band, that corresponds to cyan light, e.g. light of a wavelength of <NUM> - <NUM>. The dashed line illustrates a second pass band, corresponding to yellow light, e.g. light of a wavelength of <NUM> - <NUM>. The response of the sensor may include only the first pass band, only the second pass band, or may include both the first pass band and the second pass band. For example, the band-pass characteristic of the sensor is realized by using a yellow filter as optical filter 603b, or by using a cyan filter as optical filter 603b, or by using an optical filter 603b that has a dual passband, or by using an optical filter 603b that comprises a mosaic of a yellow optical filter and a cyan optical filter.

In another example, the optical filter 603b may comprise a color filter mosaic comprising a red filter (R), a green filter (G), a blue filter (B), a yellow filter (Y) and a cyan filter (C). This can be labelled a RYGCB sensor. Optionally, a magenta filter (M) is further included. The R, G and B filter of the mosaic correspond to the R, G, B primaries of the display apparatus. For ambient light compensation, only the Y and C pixels of the RYGCB sensor are taken into account, to avoid adjusting the ambient light compensation to light emitted by the display apparatus itself. The R, G and B pixels can be used to determine a color of ambient light, for adjusting the color mapping of the video signal to be displayed. The present disclosure therefore further relates to such a sensor per se: a light sensor comprising an optical filter in the form of a RYGCB mosaic, optionally a RYGCBM mosaic.

Optical filter 603b may thus block or attenuate other wavelengths, e.g. red, green and blue light as used in RGB display apparatuses.

Use of an ambient light sensor that is less sensitive to the primary colors of the display apparatus is particularly effective for a display apparatus including one or more lasers, such as a laser projector, as the laser light sources have a very narrow bandwidth. The laser light impinging on the sensor can thus be filtered out very effectively.

An advantage of using a sensor that is sensitive only to yellow light is that in typical viewing environments the main contribution to the ambient light intensity is emitted by lamps that often emit predominantly yellowish light.

Some displays comprise a blue and yellow phosphor, in which case it is advantageous to use a sensor that is sensitive to cyan light only.

Instead of using an optical filter 603b, an ambient light sensor may be selected that is inherently less sensitive to the primary colors of the display apparatus. For example, the ambient light sensor comprises a light-sensitive semiconductor device (e.g. a photodiode or a reversed biased LED) that has a bandgap corresponding to wavelengths different from the wavelengths of the primary colors of the display apparatus. For example, a semiconductor device having a bandgap corresponding to a wavelength in the yellow part of the spectrum (e.g. <NUM> - <NUM>). In another example, two or more light-sensitive semiconductor devices are used, that have different bandgaps, each corresponding to a wavelength different from the wavelengths of the primary colors of the display apparatus. For example, the ambient light sensor comprises a photodiode having a bandgap corresponding to a wavelength in the yellow part of the spectrum (e.g. <NUM> - <NUM>) and another photodiode having a bandgap corresponding to a wavelength in the cyan part of the spectrum (e.g. <NUM> - <NUM>).

The measures of a) synchronizing the ambient light compensation with scene changes, and b) use of ambient light sensor having a reduced sensitivity to light of wavelengths corresponding to the display apparatus's primary colors have been described above in combination. However, the present disclosure is not limited to such a combination, and feature b) is implemented as a stand-alone measure. It is noted that in case an ambient light sensor having a reduced sensitivity to the primary colors of the display apparatus is used as a stand-alone measure, i.e. without synchronization of ambient light adjustment with scene change, also the effect of reducing flicker may be achieved, since flicker cause by detecting light emitted by the display apparatus is prevented.

In an embodiment, a display apparatus comprises an ambient light sensor configured to generate a signal indicative of intensity of ambient light, wherein the apparatus is configured to receive an input video signal and apply ambient light compensation to the input video signal in dependence on the signal indicative of the intensity of ambient light, wherein the ambient light sensor is less sensitive to light of wavelengths corresponding to primary colors of the display apparatus than to light of other wavelengths of the visible spectrum.

For example, the display apparatus comprises a projector, such as a laser projector, or a display, such as an LCD or OLED display.

For example, the display apparatus comprises a controller and/or a DM module configured to receive the input video signal and apply ambient light compensation to the input signal in dependence on the signal indicative of the intensity of ambient light.

As described above, by using an ambient light sensor with a reduced sensitivity to the display apparatus's primary colors, interference of light emitted by the display apparatus itself with the measurement performed by the ambient light sensor is avoided or at least reduced. Alternatively or additionally, the apparatus, e.g. the DM module of the apparatus, may be configured to determine an expected measuring bias based on the input video signal and take the expected measuring bias into account when applying ambient light compensation based on the sensors output.

<FIG> illustrates an example of a process for taking into account an expected measuring bias caused by the sensor registering light emitted by the display apparatus itself. The input video signal <NUM> comprises a sequence of video frames <NUM>-<NUM> to <NUM>-<NUM>. The current frame is depicted as "Frame n", and three preceding video frames are depicted as "Frame n-<NUM>", "Frame n-<NUM>" and "Frame n-<NUM>", respectively. The input video signal <NUM> may further comprise metadata (not shown), such as frame-based metadata and/or scene-based metadata, as described above in more detail.

An operation f is executed for each video frame <NUM> by function block <NUM>, to extract a measure of the overall intensity level of each video frame. The function block <NUM> is the same for each video frame <NUM>, so that the operation f is also the same for each video frame <NUM>. The operation f may be performed on the image data of each video frame or on the metadata of the video frame. For example, the operation f may analyze pixel intensity values of the image data to determine an overall intensity level of the video frame <NUM>. In another example, the operation f uses frame metadata, such as the Smin, Smid or Smax metadata described above, to determine an overall intensity level of the video frame <NUM>. The result of operation f is a scalar that is indicative of the overall intensity level of the corresponding video frame.

For example, overall intensity of a video frame <NUM> may be calculated as the average pixel intensity values of said video frame. Let P denote the number of pixels in each frame, i denote a pixel index with <NUM> ≤ i < P, and Li denote a pixel intensity value. Then in this example: <MAT>.

Alternatively, the average intensity of the intensity values of the video frame may be available in the frame metadata, and the operation f extracts said average intensity from the metadata of each frame.

In another example, the overall intensity of a video frame <NUM> is calculated as the mean or median of the pixel intensity values of said video frame. In yet another example, the overall intensity of a video frame <NUM> is calculated as the average of the maximum pixel intensity value and the minimum pixel intensity value of the video frame: <MAT>.

Alternatively, max(Li) or min(Li) may be available in the frame metadata, and the operation f extracts those values from the metadata, and their sum is computed and divided by two. In another example, instead of calculating the average of the max and min, the operation extracts an Smid value directly from the frame metadata.

The term pixel intensity value refers to a per pixel measure for the intensity of said pixel. For example, a pixel intensity value may correspond to a pixel value of a luma component of the image data. In another example, a pixel intensity value is calculated from the pixel values of the image data, e.g. in an RGB image the intensity value of the pixel may be calculated as a linear combination of the R, G and B pixel values.

In block <NUM>, an expected measuring bias <NUM> is computed, which includes the step of summing the intensity levels generated by blocks <NUM>. In an embodiment, the expected measuring bias <NUM> simply corresponds to the computed sum, as illustrated by the sigma sign in block <NUM>. In another embodiment, the expected measuring bias <NUM> is computed by dividing the sum by the number of video frames <NUM> used in the calculation, to obtain an average of the intensities of the video frames <NUM>. In another embodiment, the expected measuring bias <NUM> is computed by calculating a linear combination of the intensities of the video frames <NUM>, e.g. a weighted average. For example, a weighted average that has greater weighting factors for the intensities of more recent video frames (e.g. <NUM>-<NUM>) than for older video frames (e.g. <NUM>-<NUM>).

The measuring bias <NUM> calculated in block <NUM> is used in function block <NUM> to adjust an ambient light intensity value <NUM> that is based on the measurement performed by the ambient light sensor. The measured value <NUM> may be obtained directly from the ambient light sensor or preprocessing steps such as analog-to-digital conversion, scaling or offsetting may be performed on the sensor's signal to convert it into the measured value of ambient intensity <NUM>. In block <NUM> the measured value <NUM> is adjusted by compensating for the expected measuring bias <NUM> as determined based on the input video signal <NUM>. The output of block <NUM> is a corrected value for the ambient light intensity, on which subsequent ambient light compensation is to be based.

For example, the compensation performed in block <NUM> may comprise subtracting the expected measuring bias <NUM> from the measured value <NUM>: Icorrected = Imeasured - Ibias. In another example, the compensation performed in block <NUM> includes scaling the expected bias before subtracting: Icorrected = Imeasured - a * Ibias. For example, <NUM> < a ≤ <NUM>. In another example, the expected bias determined in block <NUM> is converted via a second order polynomial before subtracting: Icorrected = Imeasured - (αIbias<NUM> + βIbias + γ). In yet another example, the expected bias determined in block <NUM> is converted via a LUT before subtracting from Imeasured. In yet another example, the compensation performed in block <NUM> uses a 2D look-up table (LUT) to convert the data pair (Imeasured, Ibias) into a corrected intensity value Icorrected.

For example, the scaling factor a, the parameters α, β, γ, the LUT or the 2D LUT may be determined using a calibration procedure. In a first step of the calibration procedure, the display apparatus is configured to display a test image with minimal light emission. For example, the display apparatus is configured to display a black image. In an example of a display apparatus having a display with a backlight, the backlight may be switched off for said first step of the calibration procedure. The response from the ambient light sensor is measured. In a second step of the calibration procedure, that can be performed prior or after the first step, at least one test image is displayed, and the response from the ambient light sensor is measured. For example, a series of test images is displayed in sequence, each of the test images having a different overall intensity. The first step and second step are performed within a relatively small timeframe, not exceeding <NUM>-<NUM> minutes, but typically less than <NUM>. Therefore, the difference in the sensor output when displaying a black image and the sensor output when displaying a test image can be mainly attributed to the sensor registering the light emitted by the display apparatus itself. In an embodiment, said difference is calculated to obtain a calibrated bias level for each test image. Each test image has a known overall intensity. Therefore, a LUT can be constructed that relates the overall intensity of an image shown on the display apparatus to a calibrated sensor bias. In an example, interpolation is used to obtain further entries of the LUT. Additionally or alternatively, a function may be fitted to the measured pairs of overall image intensity and bias value, e.g. to obtain the parameters a, α, β or γ described above. Instead of measuring with a black screen once and subsequently measuring a series of test images, in an embodiment measuring with a black screen is alternated with measuring one or more test images.

In the example of <FIG>, the current video frame "Frame n" is also used for computing the sum in block <NUM>. However, as indicated by the dashed lines, including "Frame n" is optional, and in some embodiments the computation is based only on video frames that precede the current video frame. In the shown example, the summation is performed over four frames, but the present disclosure is not limited to summing over four frames. In an embodiment, the predetermined number of video frames to use for calculating a measuring bias may be in the range of <NUM>-<NUM> video frames, e.g. in the range of <NUM>-<NUM> video frames. Further, in the illustrated example the computation is performed on consecutive video frames. However, the present disclosure is not limited to using consecutive video frames for the computation of a measuring bias, e.g. every other video frame may be used.

The predetermined number of video frames used in the computation corresponds to a certain time duration. For example, at <NUM> frames per second, <NUM> video frames correspond to a time duration of <NUM> second. In an embodiment, the predetermined number of video frames to be used in the computation (and optionally also during calibration) corresponds to a time duration in the order of the sensor's integration constant. Typical values for the time duration are <NUM> - <NUM> seconds, e.g. <NUM> - <NUM> seconds.

Embodiments described herein may be implemented in hardware, software, firmware and combinations thereof. For example, embodiments may be implemented on a system comprising electronic circuitry and components, such a computer system. Examples of computer systems include desktop computer systems, portable computer systems (e.g. laptops), handheld devices (e.g. smartphones or tablets) and networking devices. Systems for implementing the embodiments may for example comprise at least one of an integrated circuit (IC), a programmable logic device (PLD) such as a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific IC (ASIC), a central processing unit (CPU), and a graphics processing unit (GPU).

Certain implementations of embodiments described herein may comprise a computer program product comprising instructions which, when executed by a data processing system, cause the data processing system to perform a method of any of the embodiments described herein. The computer program product may comprise a non-transitory medium storing said instructions, e.g. physical media such as magnetic data storage media including floppy diskettes and hard disk drives, optical data storage media including CD ROMs and DVDs, and electronic data storage media including ROMs, flash memory such as flash RAM or a USB flash drive. In another example, the computer program product comprises a data stream comprising said instructions, or a file comprising said instructions stored in a distributed computing system, e.g. in one or more data centers.

Claim 1:
A display apparatus, comprising:
- an ambient light sensor configured to generate a signal indicative of intensity of ambient light,
wherein the display apparatus is configured to:
- receive an input video signal; and
- applying ambient light compensation to the input video signal in dependence on the signal indicative of the intensity of ambient light,
characterised in that the ambient light sensor is less sensitive to light of wavelengths corresponding to primary colors of the display apparatus than to light of other wavelengths of the visible spectrum.