Multiple modulator displays and related methods

A dual modulator display has a first array of pixels that illuminates a second array of pixels with a pattern of light. The second array of pixels modulates the pattern of light to yield an image. A method for determining control values for pixels of the first array of pixels begins with an initial set of control values and refines the control values. The control values may be refined one at a time. Images may be displayed in real time.

TECHNICAL FIELD

The invention relates to electronic displays such as computer monitors, televisions, data projectors and the like. The invention relates more specifically to such displays in which light from a first array of pixels is modulated by a second array of pixels to yield an image.

BACKGROUND

Dual modulator displays have a first array of pixels that generates a controllable pattern of light at a second array of pixels. Examples of dual modulator displays are described in WO02/069030 (PCT/CA2002/000255) and WO03/077013 (PCT/CA2003/000350), both entitled HIGH DYNAMIC RANGE DISPLAY DEVICES. In some embodiments, image data specifying a desired image is supplied to a controller which operates the first array of pixels to yield a pattern of light at the second array of pixels. The pattern of light approximates the desired image. The controller operates the second array of pixels to modulate the pattern of light to yield an image that is closer to the desired image than the pattern of light. In some embodiments the first array of pixels has a lower resolution than the second array of pixels (i.e. there are more pixels in the second array of pixels than there are in the first array of pixels). The first array of pixels may comprise, for example, an array of individually-controllable light sources pixels of a spatial light modulator, or the like. The second array of pixels may comprise a reflective or transmissive spatial light modulator.

In some embodiments the first array of pixels comprises an array of light-emitting diodes (LEDs) and the second array of pixels comprises a liquid crystal display (LCD) panel.

A “dual modulator” display architecture can be suitable for use in high-end displays (some examples of high end displays are displays for viewing X-rays and other critical images; high-end cinema applications and the like). In such applications it is desirable to make the displayed image reproduce the desired image as closely as possible. In such applications, any perceptible deviation from the desired image is undesirable.

In a “perfect” dual modulator display the first set of pixels would have light outputs that are steplessly variable from zero to very bright and the second set of pixels would be steplessly controllable between transmitting zero light and passing all incident light. In the real world, the components practical for use as the first and second sets of pixels have limitations. For example, where the first or second set of pixels comprises pixels of an LCD (liquid crystal display) the pixels have a maximum transmission of less than 100%, a minimum transmission greater than 0%, and the transmission of each pixel is typically selectable from among a discrete set of values. Similarly, where the first set of pixels comprises an array of individually-controllable light sources (such as LEDs, for example) it is typical that the light sources have light outputs that can be adjusted in discrete steps up to some maximum value.

One problem is to determine how to control the first set of pixels so that the pattern of light on the second set of pixels will approximate the desired image in a way that can be corrected for by the second set of pixels to a high degree of accuracy.

SUMMARY OF THE INVENTION

One aspect of the invention provides a method for generating control values for pixels of a first array of pixels in a dual modulator display. The method comprises generating a desired light pattern and an initial set of control values from image data defining a desired image, and, refining the initial set of control values by determining for a pixel a change Δdjthat tends to reduce a difference between the desired light pattern and a light pattern estimated to be produced in response to the set of control values at a location corresponding to the pixel.

Further aspects of the invention and features of specific embodiments of the invention are described below.

DESCRIPTION

The inventors have determined that there is a need for:dual modulator displays;controllers for dual modulator displays;methods for operating dual modulator displays;methods for generating control values for controlling pixels of dual modulator displays; andprogram products containing software for operating dual modulator displays and generating control values for the pixels of dual modulator displays;
that display images while reducing loss of image information notwithstanding the constraints imposed by the capabilities of the first and second arrays of pixels.

For example, consider the case illustrated inFIG. 1. A display10comprises a first array12of pixels12A and a second array14of pixels14A. A controller16receives image data18and generates control signals19A and19B to control the pixels of the first and second arrays of pixels respectively. First array12may, for example, comprise an array of light-emitting diodes (LEDs) or other light sources. Second array14may, for example, comprise an LCD panel.

Controller16derives signals19A and19B from image data. Signals19A may comprise, for example, a set of control values d for pixels12A of first array12. Signals19B may comprise, for example, a set of control values p for pixels14A of second array14.

In response to control signals19A from controller16, first array12emits light. The light emitted by pixels12A of first array12yields a pattern20of light on second array14. Pattern20is an approximation of a desired image specified by image data18. Since first array12has significantly fewer pixels than second array14, aspects of the desired image that have higher spatial frequencies will be represented primarily by control signals19B. Pattern20will track aspects of the desired image that have relatively low spatial frequencies.

The characteristics of pattern20depend on the amounts of light emitted by pixels12A of first array12, as well as the point spread functions of pixels12A. Optical elements (not shown) such as diffusers, lenses, collimators, etc. may be provided between first and second arrays12and14. The effect of any such optical elements on pattern20may be taken into account by the point spread functions of pixels12A.

Pattern20can be characterized by an intensity value Biat each pixel14A of second array14(where i is an index that identifies a specific pixel14A). The way in which control signals19A are derived from the image data18can affect the quality of the image produced by display10. If control signals19A result in a “good” pattern20then second modulator14can modulate the light in pattern20to reproduce the desired image very accurately. On the other hand, if control signals19A produce a pattern20that is sub-optimal then it may not be possible to determine control signals19B that will cause second modulator14to modulate pattern20in a way that matches the desired image without perceptible artifacts.

Consider, for example, the case where, in an area of the desired image, the desired image depicts stripes having a spatial frequency too high to be reproduced in pattern20. The stripes are superposed on a background light intensity that varies at a lower spatial frequency. Curve30ofFIG. 2Ashows a variation in intensity of the desired image in a direction across the stripes. Curve32shows a variation in intensity of the light in an example pattern20. If the light in pattern20has an intensity that varies in a way which is a good match to the low spatial frequency component of curve30(as indicated by curve32) then second array14can be set to generate the high-spatial-frequency components of the desired image, as shown inFIG. 2B, so that the resulting image faithfully reproduces the desired image.

On the other hand, in some situations the light in pattern20has an intensity that varies in a way which is not such a good match to the low spatial frequency component of curve30. This may occur, for example, in a situation where the point spread function of pixels12A of first array12is such that the low spatial frequency of pattern20is lower than the low spatial frequency component of the desired image. Curve33inFIG. 2Cshows a variation in intensity of the light in another example pattern20, which illustrates such a mismatch between the low spatial frequency components of curve30and pattern20. In such situations, second array14is unable to accurately generate the high-spatial-frequency components of the desired image, as shown inFIG. 2D, so that the resulting image may fail to faithfully reproduce the desired image. Losses of high-spatial-frequency information can occur as a result of quantization of the signals controlling pixels14A. For example, in cases where Biis significantly greater than the desired light output of pixel14A, the value pifor the pixel14A may be rounded to zero. In cases where Biis equal to or less than the desired light output of pixel14A, the value pifor the pixel14A may be clamped at a maximum value. In either case, the resulting image does not contain all of the information in image data18.

It can be seen that obtaining the best possible image quality from a dual modulator display of the type described herein (in which the pattern of light from the first array of pixels does not contain the highest spatial frequency components of the image data) can require some optimization of the light pattern20.

FIG. 3illustrates steps in a method40for obtaining signals19A and19B from image data18according to an embodiment of the invention. Image data18comprises a set Ī of values Īi. In some embodiments, the values are color values. In such embodiments, the values are most conveniently expressed in a color space that has the same chromaticity, white point, and primaries as a display10on which the image will be displayed.

Block42derives an initial set of control signals19A for pixels12A of first array12. In an example embodiment, block42involves a process50as shown inFIG. 3A. Process50derives a target light patternBfrom image data18. The target light patternBmay comprise valuesBicorresponding to each pixel14A of second pixel array14. The values may be expressed in photometric units, for example. In some embodiments, the valuesBiare monochromatic values in photometric units, such as luminance. Block52may comprise clamping luminance values so that they do not fall outside of the range displayable by a display10on which the image will be displayed.

If image data18specifies a color image then block53may comprise extracting a monochrome (single channel luminance) representation of image data18. This may be achieved by taking the maximum of the three color channels (e.g. red, green and blue) for each pixel of the second array of pixels.

Block54determines values forBiby computing a function of the monochrome representation of image data18. In some embodiments, the function comprises determining a fractional power of the value of the corresponding pixel in the monochrome representation of image data18. The power is chosen so as to allocate the dynamic range between the first and second arrays. In some embodiments, block54comprises taking the square roots (the ½ power) of the values in the monochrome representation of image data18. The optimum division of the dynamic range between the first and second arrays of pixels will depend on the ratio of dynamic ranges between the first and second arrays. In some dual modulator displays such as the model DR 37 P display available from Brightside Technologies Inc. of Vancouver, Canada the ratio of dynamic ranges between the first and second arrays is about 1:1 and a power of approximately ½ is a good basis for obtaining signals19A.

A single light source that can be turned off entirely could be considered to have an infinite dynamic range. However, the dynamic range of a light source within any collection of such light sources that have overlapping point spread functions is determined by the point spread functions for the light sources and the light outputs of neighboring light sources.

Maintaining pixel values on the first and second pixel arrays to be of the same order of magnitude is preferable, other factors being equal, to a case in which one of the arrays is given large pixel values while the other array is given very small values because quantization artifacts are relatively large for small values. Also, if different combinations of values are used for adjacent pixels of the same intensity, the imperfect alignment present in real hardware systems could cause significant artifacts.

For purposes of calculation, it is convenient to express values forBand Ī in the range [0, 1]. Converting to this representation can be done by normalizing the image data. In block54the appropriate function of the normalized image data can be computed. The result can then be scaled to provide values forBin any desired units.

In the case where first array12has n pixels12A and second array14has m pixels14A with m>n then the problem of determining values for pixels12A of array12which will result in the desired pattern20can be expressed as an m×n system of equations. In cases where the first array12has a low spatial frequency (i.e., if the point spread function of first array12has no high spatial frequency components) it is expected that pattern20should also have a low spatial frequency. In such cases computation can be reduced without introducing significant artifacts by downsampling the values forBto a lower resolution (it is convenient to make the lower resolution the same as that of first array12). The resulting problem can be expressed as an n×n system of equations. Thus, the values forBmay comprise a set of n valuesBi, with a single valueBifor each group of pixels14A of second array14closest to a corresponding pixel12A of first array12.

In block56the resulting data is downsampled to a resolution which may be the same as that of first array12. Downsampling may be implemented in a range of ways. For example:downsampling may be implemented in software by any suitably filtered resize function.downsampling may be implemented in a logic circuit, such as a suitably-configured field progammable gate array (FPGA) as an average of pixel values in neighbourhoods around positions corresponding to positions of pixels in the first array of pixels.downsampling may be implemented in a graphics processor by recursively taking block averages of pixel values.

Block58returns the values ofBwhich specify the desired light pattern20(at the resolution of first array12).

Method40continues in block44to determine control values for the pixels of first array12that will result in a light pattern20having values close to or the same as the valuesBreturned by block58. This can be achieved by solving the minimization problem:

mind⁢Wd-B_(1)
where d is the set of control values for pixels12A, and W is the convolution of the point-spread functions for pixels12A with Dirac delta functions at the position of the pixels12A.

Computation can be saved by considering the pixels of first array12in neighborhoods that are smaller than array12instead of solving for all pixels of first array12at once. This can be done because, in typical dual modulator displays, a pixel.

12A of first array12does not contribute much light at pixels14A of second array14that are at coordinates distant from the pixel12A.

Also, the properties of the human visual system and the dynamic range of second pixel array14constrain which pixels of first array12can be adjusted to obtain a desired amount of light at a pixel14A of second pixel array14. The human visual system cannot detect changes in the luminance of a less-bright area that is very close to a brighter area. This effect is known as “veiling glare”. Veiling glare occurs in part because of light scattering in the eye.

FIG. 4illustrates veiling glare. A desired image is indicated by line75. Image75has a bright area75A adjacent to a dimmer area75B. Light emitted by three pixels12A is indicated by curves77A,77B and77C. Veiling glare caused by bright area75A affects an area76. Within area76the human eye cannot distinguish differences in luminance that fall below the level76A (which falls off with distance from bright area75A). Outside of area76, increasing the intensity of first array12beyond a locally-desired value would be detected. Pixels of first array12within area76can have increased intensities without perceptibly altering the image. For example, the light output77B could be boosted to the value indicated by77B′ without visibly affecting the resulting image (because the peak intensity of77B′ is still below curve76A.

While the weighting matrix W (see Equation (1)) is dense, the number of pixels12A that can be freely altered to control the amount of light at a given pixel14A of second pixel array14is quite limited. The resulting matrix of control values d is a relatively sparse, banded matrix which can be solved exactly or approximately in any suitable manner to obtain control signal values djfor pixels12A (j is an index that identifies specific pixels12A).

In one embodiment, djis given by:

dj=B_j-∑iN(δj)⁢wji⁢B_iwjj(2)
where:Bjis the value of the target light pattern at pixels14A of second array14in an area corresponding to the pixel12A identified by index j (the jth pixel);Biis the value of the target light pattern at pixels14A of second array14in an area corresponding to the pixel12A identified by index i (the ith pixel); N(δj) includes pixels12A in a neighborhood of the jth pixel; wjjis the value for the point spread function for the jth pixel at pixels14A in the area corresponding to the jth pixel; wjiis the value for the point spread function for the ith pixel at pixels14A in the area corresponding to the jth pixel.

The light pattern20that would result from control signals obtained as described above may not be optimal. The actual light pattern is characterized by values B corresponding to pixels14A. In general, B≠B. Typically, image quality can be improved by fine tuning the values for djto yield a light pattern that better approximates the optimum light pattern. Image quality can also be improved by fine tuning the values for djto permit more of the bit-depth of second array14to be applied to representing higher-spatial frequency details and color. Corrected control values are determined in block46of method40.

Assuming that the initial value forBobtained in block42is reasonably close to the optimum value, then only small changes to the values of d should be required for optimization. Such changes are referred to herein as Δdj. These changes can be determined by applying a “greedy” algorithm which treats pixels12A one at a time. The algorithm attempts to reduce the difference between B andBat locations corresponding to pixels12A.

In some embodiments, Δdjfor the jth pixel of first array12may be determined such that when the jth pixel is provided with a control value of dj+Δdj, pixels14A of second array14in an area corresponding to the jth pixel may be provided with control values p which are not subjected to rounding or clamping. This permits second array14to faithfully reproduce high frequency spatial components of the desired image without losses of information due to quantization of signals19B.

In one embodiment, the algorithm comprises determining a value for Δdjthat satisfies:

I_-Δ⁢⁢dj⁢WjWd=α(3)
where α is a constant (which may be identified with a desired average value for control signals of second pixel array14); Ī, W and d are as defined above; and Wjis the convolution of the point spread function for the pixel12A currently being processed with a Dirac delta function at the location of that pixel12A.

If B is substituted for Wd in Equation (3) then the problem can be expressed as finding a solution to:
∥Ī−ΔdjWj−αB(j)∥=0  (5)
where B(j)is the set of values that characterize light pattern20(including the effect of any changes Δdjfor previously-treated pixels12A).

Equation (4) can be represented in terms of the coordinates (x,y) of pixels12A to yield:

∑x,y⁢(I_⁡(x,y)-Δ⁢⁢dj⁢Sj⁡(x,y)-α⁢⁢B(x,y)(j))2⁢Mj,(x,y)=0(6)
where Sj(x,y) is a texture splat for the point spread image of the jthpixel12A which is at location (x,y); Mj(x,y)is a masking function which has a value 1 in a region surrounding the location of the pixel12A currently under consideration and 0 otherwise (the region may, for example, be a circular region with a defined radius). Equation (5) can be solved for Δdjto yield:

B(j)may be calculated from the values of d (which for convenient calculation are expressed in the range [0, 1]) including any Δdjthat have been already determined. Since the point spread functions for pixels12A will typically vary with low spatial frequencies, then calculation of B(j)may be implemented at a reduced resolution and subsequently upsampled. Where this is done it is desirable to ensure that pixels of the reduced resolution image are aligned with pixels12A to avoid rounding errors.

B(j)may be computed in various ways. For example, in some embodiments, B(j)is computed as a convolution. In some such embodiments, an all black image (pixel values are all zero) having individual pixels at the locations of pixels12A set to the respective control values (djor d+Δdj) as appropriate is convolved by the point spread function for pixels12A scaled in photometric units. Such embodiments can conveniently be implemented in software.

In other embodiments, the amount of light at each pixel14A is determined by computing the distances from that pixel14A to contributing pixels12A and for each such contributing pixel12A looking up the corresponding distance in a table to obtain a value of a point spread function for the contributing pixel12A at that pixel14A. The value of each point spread function is modulated (e.g. multiplied) by the current control value for the pixel12A (djor d+Δdj) as appropriate. Such embodiments may conveniently be implemented in logic circuits such as suitably-configured FPGAs.

Other embodiments apply a splatting approach, and draw screen-aligned quadrilaterals with textures of the applicable point spread functions into a frame buffer. Each texture is modulated by the corresponding control value (djor d+Δdj), as appropriate. Alpha blending may be applied to accumulate the results. Such embodiments may conveniently be implemented in graphic processor units (GPUs).

The tail of a point spread function applicable to pixels12A can be very long (that is, light from a particular pixel12A may reach pixels14A that are relatively far from the pixel12A (in terms of the coordinate spaces of the first and second pixel arrays). A compromise between accuracy and computational overhead may involve truncating the point spread functions at some reasonable distance. If this is done, it is desirable to blend the point spread function to zero in the area of the point of truncation as opposed to leaving a significant discontinuity in the point spread function.

Truncation of the point spread functions can result in the calculated intensities for pixels14A outside the truncation distance to be smaller than they ought to be. While insignificant when compared to the peak luminance of the display, this disparity can contribute to perceivable mismatches in dark regions. Because the spatial frequency of the truncated portions of the point spread function is very low, it is possible to compensate by adding a term u to each pixel of the backlight image to represent the light not taken into account as a result of truncation. The value of u may be chosen to be a fraction (or other suitable function) of the set of control values d. A suitable value for u may be based on the difference in energy between the actual point spread function and the truncated simulation of the point spread function.

A process for solving Equation (6) may comprise operating on pixels12A in scan-line order (i.e. starting at one corner of first pixel array12and working along one row of pixels at a time). For the point spread image Sjcorresponding to the current pixel12A, the corresponding areas of Ī and B are selected and their respective elements are multiplied and then summed together to yield Δdj. The corresponding control value d+Δdjis then written to the control values d and B is modified accordingly by accumulating the values ΔdjSjonto B.

The foregoing process may be iterated two or more times, if desired, to further refine B.

The calculations described above for refining the control values d for pixels12A assume that the correct set of values B of the actual light pattern is known (Equation (6) involves B for example). However, in almost all cases, at the time that the computation of Δdjis being performed for any particular pixel12A, Δdjwill not yet have been computed for other pixels12A. Even though Δdjis usually small (as long as a good approximation toBis obtained in block42) the accumulated error can be significant.

If pixels12A are processed in a known order such that an area for which Δdjhas already been processed can be distinguished from an area for which Δdjhas not yet been processed then it is possible to compensate for this error. For example, consider the case where pixels12A are processed in scan line order beginning at a top left corner of first array12. In this case, pixels12A above and to the left of the current pixel12A have already been updated, while pixels12A below and to the right of the current pixel12A have not yet been updated. Even if Δdkis not yet known for some k>j, the values for the desired image Ī and B(k)are known. One can assume that the control value for the kth pixel12A will change such that the light emitted by the kth pixel12A changes by an amount equal to the difference between Ī and B(k).

An image filter that performs the corrective measure can be added to Equation (5) to yield:

∑x,y⁢(I_(x,y)-Δ⁢⁢dj,(x,y)-α⁢⁢B(x,y)j)2⁢Mj,(x,y)⁢F(x,y)=0(7)
Where F is the image filter. Equation (7) can be solved for Δdjto yield:

Δ⁢⁢dj=∑x,y⁢Sj⁢I_⁢Mj⁢F-α⁢⁢∑x,y⁢Sj⁢B(j)⁢Mj⁢F∑x,y⁢Sj2⁢Mj⁢F(8)
Equation (8) may be solved as described above, for example. All that is required is a filter function F that has a positive value for pixels12A that have already been corrected (i.e. for which Δdjhas been computed and added to the corresponding control value) and a negative value for pixels12A that have not yet been corrected.

The value of a may be set to adjust the displayed image. A value of α=1 will cause the pixels12A to be the same intensity as Ī, causing the operation to match the targetBas best it can. Error diffusion can be performed as a second iteration to achieve the desired image. A value of α=0.5 causes the first array12to be twice as bright as Ī, resulting'in pavg=0.5. This typically maximizes the number of bits in the control values for pixels14A that are available for correction and minimizes artifacts arising from the quantization of control values p for second array14.

More complicated schemes, such as choosing the value of a depending on the local neighborhood, are also possible and can be employed to provide feature-specific tone scaling of light pattern20.

The control values d may be supplied in signals19A to drive the corresponding pixels12A of first array12of a display10. Control values p for pixels14A of second array14may be determined, for example, by:estimating the distribution B of light in pattern20that will result from applying control values d to the pixels of first array12; and,computing values p according to:

FIG. 5shows an example flow of data in an example controller16for controlling a display10. The functional blocks inFIG. 5may be implemented in software executed on general purpose data processors; logic circuits (for example, configured FPGAs), graphics processors or some combination thereof. Image data18is received at controller16. Desired image Ī is extracted from image data18. Desired light pattern generator60generates a desired light patternBto be produced by first array12. First array control values generator62generates control values d. First array control values generator62may generate control values d and adjustments Δd according to a method as described above.

Control signal generator64A generates control signals19A that are supplied to first array control circuit65A. First array control circuit65A operates the pixels of a first array12according to the first array control values d. Control signals19A may be provided directly to first array control circuit65A or may be delivered to first array control circuit65A after a delay. For example, signals19A may be recorded on a medium (not shown) and played back to first array control circuit65A at a later time.

First array control values d are also supplied to light pattern simulator66. Light pattern simulator66determines a simulated light pattern B. Second array control values generator68generates second array control values p based upon the simulated light pattern B and desired image Ī. Light pattern B determined by simulator66may also be used by first array control values generator62to generate B(j)for use in computing Δdjas described above, for example.

Control signal generator64B generates control signals19B that are supplied to second array control circuit65B. Second array control circuit65B operates the pixels of a second array14according to the second array control values p. Data19B may be provided to second array control circuit65B directly or after a delay as described above with reference to signal19A.

FIG. 6shows an example system70for generating control signals for a first array of pixels in a dual modulator display. The functional blocks inFIG. 6may be implemented in software executed on general purpose data processors; logic circuits (for example, configured FPGAs), graphics processors or some combination thereof. Desired image Ī is provided to an initial control values generator72. Initial control values generator72generates initial control values d. Initial control values generator72may generate initial control values d according to a method as described above. Initial control values generator72may generate a desired light patternBin the course of generating initial control values d.

Control values d are provided to a light pattern simulator74. Light pattern simulator74determines a simulated light pattern B. Simulated light pattern B is provided to a control value adjustment generator76, along with desired image Ī. Control value adjustment generator76generates control signal adjustments Δd. Control value adjustment generator76may generate a control signal adjustment Δdjfor each control value djaccording to a method as described above. Control signal adjustments Δd are combined with initial control values d to produce adjusted control values d+Δd.

Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in a display controller or one or more processors in a device that outputs a signal containing control signals19A and19B for use by a dual modulator display may implement the methods ofFIGS. 3 and 3Aby executing software instructions in a program memory accessible to the processors. The invention may also be provided in the form of a program product. The program product may comprise any medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of forms. The program product may comprise, for example, media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

As will be apparent to those skilled in the art in the light of the foregoing disclosure, many alterations and modifications are possible in the practice of this invention without departing from the spirit or scope thereof. For example:Control signals19A and19B may be generated in real time in response to image data18(which may comprise video data, for example) or may be generated in advance.
Accordingly, the scope of the invention is to be construed in accordance with the substance defined by the following claims.