Patent Description:
Current projectors are often based on so called "light valve technology". A light valve is an amplitude spatial modulator. The entire light valve is uniformly illuminated and the light valve (LCD, DLP or LCOS) blocks e.g. redirects light to a dump, i.e. light which is not needed. Two main disadvantages come with this:.

Currently there is a desire for displays, including projection displays to be capable of producing a Higher Dynamic Range (HDR). This means darker black levels and higher peak brightness levels. This will enable more details in black and a more adequate representation of the image highlights. It is however not the intention that the average picture brightness is much increased because this will mainly force eye adaptation to a different level, might even be painful, but does not benefit the perceived dynamic range.

Usually, when increasing peak brightness, the black level also is raised and since more information is encoded near black, this is highly undesirable. A cascade of <NUM> light valves has been proposed in <CIT>. While this approach is effective to lower the light leakage in black, it also significantly affects the light throughput efficiency as losses in the first light valve, the imaging optics, mirrors. easily reduce the peak brightness by <NUM>%.

Further, in a typical High Dynamic Range signal, the ratio between peak brightness and average brightness becomes even larger, so an even bigger amount of the light energy will be blocked.

A much more efficient approach towards an HDR projector is one where only the second modulator is of the light valve type and where the first modulator distributes the light where it is needed. The light being distributed, or redirected, by the first modulator is the steered light. This solution delivers both better blacks and higher peak whites for the same amount of illumination light input.

Such an approach where the first modulator is based on a phase modulating LCOS device has been proposed in <CIT>.

An approach where an analogue MEMS device is used as the first modulator is described in a paper from <NPL>.

<CIT> describes a projector design which combines one spatial light modulator that affects only the phase of the illumination, and one spatial light modulator that only affects its amplitude (intensity). The phase-only modulator curves the wavefront of light and acts as a pre-modulator for a conventional amplitude modulator.

An approach to predict via simulations the resulting brightness pattern on the second modulator starting from the phase pattern on the first modulator has been proposed by MTT Innovation in a paper from <NPL>. This model further introduces an amount of smoothness using a blur kernel to take into account the limited image sharpness caused by limitations in the beam quality of the laser sources and the additional blurring introduced by a diffuser at the intermediate image plane. As additional references will be made to this paper throughout the description, we will refer to it as <NPL>.

The geometric distortions introduced by the setup and the optics between the phase modulator and the amplitude modulator can be compensated by introducing a warping function to warp the image plane backwards onto the lens plane.

Both in the case of an LCOS phase modular and in the case of a MEMS light steering device there is an analogue first modulator used, where the response to a certain driving signal depends on many influencing factors, such as manufacturing tolerances in the device itself, its operating temperature, aging,. Deformable mirrors and MEMS additionally show some degree of hysteresis. The effect of a new driving signal applied depends to some extent on its previous conditions. For mirrors with a continuous membrane, there is also an influence of the position of adjacent elements.

The application of light steering requires very high accuracy of the deflection angle, because a small angular deviation can introduce a significant error in on screen brightness of a certain pixel. And if three separate channels are used per color, this will result in color errors that are even more visible.

Further, the final result from the light steering depends not only on the amount of steering by the first modulator but as well upon the spatial distribution as well as the angular distribution of the (laser) light that is incident on it. If there is any change in those illumination characteristics the steered light pattern will change. In the case where a phase modulator is used as the light steering device also a change in laser wavelength will affect the final result.

If multiple discrete laser sources are combined to illuminate the light steering, differential aging between those sources might also affect the light steered pattern.

Also, any drift in the optical path between the first and the second modulator, will create false assumptions on where exactly the light steered pattern is imaged on the second modulator.

As further described in the paper <NPL>, they make use of the forward image formation model from their simulations to predict the illumination profile present at the second, amplitude-only modulator. Given the phase function from the freeform lensing algorithm, the light distribution on the image plane is predicted using the simple model described in the paper. The amount of smoothness introduced at the diffuser at the intermediate image plane can be modelled using a blur kernel (e.g. system point spread function that can be either directly measured or computed via deconvolution for known targets) and the modulation pattern required for the amplitude modulator is then obtained to introduce any missing spatial information as well as additional contrast where needed.

They further describe the use of a one-time calibration and characterization of the entire optical system which is required to optimally control the spatial light modulator.

It is unlikely that the one-time calibration will be sufficient and adequate to compensate for all the effects described above. Frequent recalibrations are expected to be required for critical applications. Further calibration at different temperature points and different laser power levels might be needed.

The part of the light that still ends up in the active area, even if it is attempted to steer all the light outside the active area, is not fully static. It was found that some of this "unsteered light" depends upon the phase pattern used. A partial explanation for this behaviour can be found in the fringe field phenomenon known in LCOS devices. Even if the centre of each pixel is driven to deliver the desired phase retardation, at the transition between this pixel and the adjacent pixels, the resulting electrical field is influenced by both pixels. Light incident at these transition zones will not be steered correctly. Yet it is very difficult to predict where exactly it will end up.

A similar problem occurs when a pixel has to transition from a phase A in a first frame to a phase B in a second frame. A one-time calibration might yield the desired light steered pattern in a static situation at phase A and also in a static situation at phase B. In the transition period light could end up in undesired positions. And the precise transition time and what happens in between is almost impossible to predict. A solution could be to blank the laser source during the transition period, this however affects the light output available for light steering.

Adaptive optics has been used in high-end telescopes, where a deformable mirror or a MEMS device is dynamically driven to compensate for the aberrations imposed by the atmosphere. The mirror dynamically corrects the wavefront to deliver a better image. Such devices typically are also equipped with a wavefront sensor (sensing both amplitude and phase) and driven in closed loop to compensate for effects such as hysteresis and cross coupling of the MEMS pixels. The wavefront is adjusted in both amplitude and phase through an iterative process. However, such an iterative solution is very slow, and does not seem to be practical for video applications. Further again it would be required to blank the laser until the desired result is available.

The present invention provides a method for driving a projector system defined in independent claim <NUM>, a projector system defined in independent claim <NUM>, a computer program product defined in claim <NUM>, a projector system defined in claim <NUM> and comprising software which, when executed on one or more processing engines, performs any of the methods of claim <NUM> to <NUM>, and a controller, defined in claim <NUM>, for a projector system according to any of claims <NUM> to <NUM> and <NUM>, comprising a non-transitory signal storage medium storing the computer program product of claim <NUM>. Further details are defined by the dependent claims.

The present invention provides a method of driving a projector system that comprises a phase modulator and at least one amplitude modulator as defined in claim <NUM> and a corresponding projector system as defined in claim <NUM>.

These have the advantage of improving the image or output signal while avoiding exposing the audience to unsafe levels of laser light. The image sensor is used to provide feedback to the controller of the projection system and thereby improve the highlights projected by the projection system. This has the advantage that all the artefacts which are not taken into account in the target highlight image, such as a fixed texture, diffraction artefacts, and the DC component of the unsteered illumination, any drift in the opto-mechanics, the phase modulator and/or the laser source, are now incorporated into the feedback loop by means of the images acquired in real time or near real time.

Standard movies have an average brightness level of around <NUM>% (Information gathered by Real-D). Around <NUM>% of the light energy of the laser light source is thus blocked.

For HDR there is a desire to significantly increase the peak brightness level to offer a realistic impression of sunlight reflections and light sources, however without increasing the average brightness level. In this case the laser source has to become significantly more powerful while the light valve blocks even a higher percentage of the light. This is expensive and inefficient.

A light steering approach is promising to be much more effective, and the additional cost of the light steering modulator stage can easily be offset by the savings in laser cost and electricity consumption, especially for high brightness cinema projectors.

However, as the light steering stage is essentially an analogue stage, it is very prone to errors. A system with closed loop driving can compensate those errors and make for a reliable and accurate image reproduction.

A light steering projector has the potential to extend the dynamic range both in black and in white without requiring much additional laser power and therefore without a significant increase in cost. However, the light steering modulator is an analogue component (for MEMS as well as phase modulators) and stability of the highlight illumination pattern is a major concern. A distortion in the position or amplitude of the highlight pattern will create significant image artefacts. The invention enables to mitigate those artefacts.

Additionally or alternatively, at least one image sensor can run at a multiple of the frame frequency of the input image, in order to provide sub-frame resolution.

Advantages are that frames acquired by the image sensor during frame n of the input image are available to any of the block diagrams during the display of frame n, and thus the feedback calculated or measured from the sensor image for frame n can be applied during frame n, or before the frame finishes or for the next frame n+<NUM>.

Additionally, the projector comprises a second amplitude modulator and the step of generating a target highlight image can comprise a target image and power control signals from the input image which can generate a target base image.

Alternatively, and not part of the claimed invention, one of the at least one image sensor can comprise an addressable area and one of the at least one image sensor can comprise an active area, and the addressable area can be configured to provide real-time calibration patterns and the active area can be configured to provide periodic calibration patterns, and the step of generating the predicted backlight image from the target highlight image can further comprise as input, the real-time calibration patterns and the periodic calibration patterns.

This has the advantage of providing a more detailed image or output signal where the prediction is based on the actual implementation.

Using the background level has the advantage of providing information on the steered component of the light beam but also compensating any drift in the opto-mechanics, the phase modulator and/or the laser source.

Described herein, but not defined in the claims, one of the at least one image sensor can comprise an active area, and the image sensor can be configured to acquire a real-time backlight image within the active area, said real-time illumination profile image comprising slowly varying content between frame n and n+<NUM>, and the step of generating a target highlight image, a target image and power control signals from the input image for frame n+<NUM> further can use input from the slowly varying content from the real-time backlight image acquired during frame n of the input image.

This has the advantage of using the actual image of the previous frame.

Advantageously, motion detection means are configured to determine which part of the input image frame is static between two consecutive frames and which part is dynamic.

Advantageously, means to decide on a pixel or region basis which part of the image is static or slowly varying and which part of the image is dynamic are provided.

Advantageously, in regions or for the pixels of the input image where the content is static, the predicted illumination profile is compared to the actual illumination profile.

This allows to apply the correction only to regions of the images which are static or have slowly varying content.

Additionally or alternatively, in the step of generating a target highlight image, a target image and power control signals from the input image from the input image for frame n can use input from the predicted backlight image.

Additionally or alternatively, the step of generating an amplitude pattern for driving the amplitude modulator for frame n+<NUM> from the predicted backlight image and the target image can use as input the actual backlight image of the previous frame n or sub-frame when driven at a multiple frequency of the amplitude modulator. Additionally, when receiving the target base image, a base pattern can be generated.

This has the advantage of using the real signal to improve the final output.

Additionally or alternatively, a portion of the light can be processed by the phase modulator to generate a highlight image on the amplitude modulator and another portion of the light is uniformly distributed on the amplitude modulator to provide a base image.

This has the advantage of generating two types of illumination from light of the same light source.

Additionally or alternatively, the at least a portion of the illumination pattern substantially equivalent to the illumination pattern incident on the amplitude modulator is a scaled down version, optionally less than <NUM>:<NUM>.

Preferably the image sensor is positioned in the path of those optical elements that generate the highlight.

Any discrepancy between the amplitude modulator image and the camera image is preferably mitigated by image processing.

The image sensor can be arranged to receive light from reflected light of an inserted glass plate, for example, rather than the image sensor being placed behind a folding mirror.

The camera can operate at a higher frame rate and sampling is at lower resolution when the camera is used to monitor laser safety.

A switch can be arranged to provide switching to a higher resolution to perform calibration.

Described herein, but not defined in the claims, is a method for monitoring the light levels provided by at least one laser source in a projector system, said at least one laser source being driven by power control signals, the projector system can comprise a phase modulator and at least one amplitude modulator, the phase modulator can be configured to generate a highlight image incident on the amplitude modulator, the projector system can comprise at least one image sensor configured to receive at least a portion of an illumination pattern substantially equivalent to the illumination pattern incident on the amplitude modulator, the image sensor can comprise at least one of an active area and an addressable area, the method can comprise the steps of.

In an embodiment of the present invention there is provided a projector system as defined in claim <NUM>.

Additionally or alternatively, the phase modulator and the amplitude modulator can be arranged on an optical path; the phase modulator steering a light beam to an intermediate image, said image sensor and amplitude modulator can receive light from said intermediate image such that the optical path between said image sensor and intermediate image can be (e.g. substantially) optically equivalent to the optical path between the spatial light amplitude modulator and the intermediate image. Additionally, the intermediate image can be on a diffuser or on a moving diffuser.

Additionally or alternatively, the projector can comprise a second amplitude modulator configured to generate a base pattern for the second amplitude modulator.

Additionally or alternatively, said spatial light amplitude modulator can comprise at least one of a reflective spatial light amplitude modulator, and a transmissive spatial light amplitude modulator. Additionally, said spatial light amplitude modulator can comprise one of a liquid crystal device, a plurality of micro-mirrors.

Additionally or alternatively, the phase modulator can comprise one of a deformable mirror, MEMS, an LCoS device.

Advantageously, the image sensor is a CMOS sensor or a CCD sensor.

Such sensors can be driven at a sub-frame of the amplitude modulator or of the image sensor.

Additionally, the illumination brightness levels can be <NUM>-<NUM>% higher than the target image brightness level.

The at least a portion of the illumination pattern substantially equivalent to the illumination pattern incident on the amplitude modulator is a scaled down version, e.g. the scaled down version can be less than <NUM>:<NUM>. This has the advantage of using a smaller and cheaper sensor.

The image sensor can be positioned in the path of those optical elements that generate the highlight.

Any discrepancy between the amplitude modulator image and the camera image can be mitigated by means of image processing.

The image sensor can be arranged to receive light reflected from an inserted glass plate rather than the image sensor being placed behind a folding mirror.

The camera can be operated at a higher frame rate and sampling is at lower resolution when the camera is used to monitor laser safety.

A switch can be arranged to provide switching to a higher resolution at a lower framerate to perform calibration.

The technical effects and advantages of embodiments according to the present invention correspond mutatis mutandis to those of the corresponding embodiments of the method according to the present invention.

These and other technical aspects and advantages of embodiments of the present invention will now be described in more detail with reference to the accompanying drawings, in which:.

Where the term "comprising" is used in the present description and claims, it does not exclude other elements or steps.

The terms "about" or "approximate" and the like are synonymous and are used to indicate that the value modified by the term has an understood range associated with it, where the range can be +<NUM>%, +<NUM>%, +<NUM>%, +<NUM>%, or +<NUM>% or alternatively ±<NUM>%, ±<NUM>%, ±<NUM>%, ±<NUM>%, or ±<NUM>%The term "substantially" is used to indicate that a result (e.g., measurement value) is close to a targeted value, where close can mean, for example, the result is within <NUM>% of the value, within <NUM>% of the value, within <NUM>% of the value, or within <NUM>% of the value.

Target Base Image: In cases where the system uses a Dual Projector setup, the Target Base Image is the image to be reproduced by the Base projector. This signal is not used in a Highlighter projector only or Hybrid projector setup.

Target Image: This is the final image to be reproduced by the Highlighter projector or the Hybrid projector. In case of a dual projector setup, the Target Image is considered to be the part of the Image without the Target Base Image.

Highlight image: The highlight image is the image created by the phase modulator incident on the amplitude modulator. The highlight image can be seen as a detailed caustic image, which increases the brightness of the final image.

Illumination profile image: The illumination profile image is the equivalent of the highlight image with the optional addition of base illumination in the case of a Hybrid projector.

Predicted illumination profile or Predicted Highlight: The predicted illumination profile corresponds to the simulated illumination profile present at the second, amplitude modulator.

Target highlight or Target Illumination profile: The ideal illumination pattern to be created by the phase modulator incident on the amplitude modulator, assuming the phase modulator is illuminated by a perfectly collimated laser beam, there are no optical distortions and no parasitic reflections by the phase modulator and the diffuser is perfectly positioned in the intermediate image plane.

Phase modulator: A phase modulator is a device which introduces phase variations on an incident wavefront. In the context of the invention, it creates a smooth and low detail image on the amplitude modulator. Different technologies can be used to provide a phase modulator. These technologies include microelectromechanical system (MEMS) displays, which provide a very fast temporal response but a low spatial resolution. Deformable mirrors as used for example in the field of adaptive optics can also be used. LCD displays can also be used, which include liquid crystal on silicon (LCoS) devices, which have the advantage of offering a high spatial resolution, high speed and a high pixel fill factor. Liquid crystal displays used in transmission can also be used.

Base projector: A base projector is a traditional projector with uniform illumination which comprises an amplitude modulator per color. A single amplitude modulator may be used with a sequence of coloured parts of an image. Alternatively, three amplitude modulators are used in parallel, one for each colour.

Highlighter projector: A highlighter projector is a projector which comprises a dual modulation design which combines a phase modulator and an amplitude modulator per color.

Hybrid projector: A hybrid projector is a projector which combines the functionality of a Highlighter projector and a Base projector, while using only one amplitude modulator per color. A portion of the light is processed by the phase modulator to generate a highlight illumination onto the amplitude modulator, another portion of the light is uniformly distributed onto the amplitude modulator.

Dual Projector setup: A setup that combines a Base Projector and a Highlighter projector to create a combined image. The image can be combined onto the screen or the optical paths can be combined into a single projection lens. We will use the term Dual projector setup when the base image and the highlight image are each processed by a separate amplitude modulator.

Tone mapping: A method to compress or expand the dynamic range of the input signal such that it fits within the dynamic range of the projector.

Reference numbers and functional blocks: <FIG>, <FIG>, <FIG>, <FIG> and <FIG> show work flows disclosed as linked functional blocks. Reference numbers associated with these blocks are disclosed below in table <NUM> as well as text parts of the description that provide additional information with respect to each reference number:.

One aspect of the present invention is a dual modulation projection system providing a highlight image. This dual modulation projection system can further be combined with a base projector to form a dual projector. Alternatively, in a hybrid projector the highlight image and the base image are combined at the level of an amplitude modulator. Embodiments of the present invention are described first for a highlighter projector and the combination with a base projector, as a dual projector is illustrated in the flow diagrams with dotted lines. Collimated laser light is received by a phase modulator. Light leaving the phase modulator for a given channel can be combined with light from the remaining channels and relayed through a diffuser to a prism structure such as a Phillips prism. The Phillips prism splits the light into its component colours which are each modulated by a spatial amplitude modulator (such as a DMD or LCOS) mounted to the prism, recombined within the prism and directed to a projection lens. The phase modulator introduces phase variations to the incident wavefront. The aim of the phase modulator is to redistribute light, reallocate light, from the input illumination to a target illumination profile (or target highlight), so as to produce both a higher dynamic range as well as an improved local peak luminance, compared to traditional projectors. This redistributes light from dark areas to bright regions creating highlights. This results in using available light economically. The target highlight is chosen ideally to approximate an upper envelope of intensities from the target image.

Although the phase modulator steers the incoming wavefront to redistribute the light on the amplitude modulator, there is always a fraction of the light which remains unsteered and which ends up in the projected image, in the form of a background image, even if all the light is steered outside of the active area. This in a first approximation can be modelled as a DC component added to the steered fraction of the light. This DC component of unsteered light is a problem as it reduces the contrast ratio of the projector.

The phase modulator further introduces a number of artefacts such as a fixed texture, diffraction artefacts, and the DC component of the unsteered illumination.

It is advisable to incorporate a diffuser in the design of the highlighter projector that is positioned in an intermediate image. Preferably this diffuser is a moving diffuser.

The primary purpose of the diffuser is to introduce angular diversity for despeckling purposes and to widen the beam coming out of the projection lens for laser safety.

By positioning this (optionally moving) diffuser out of the intermediate image plane such that the relayed image from the diffuser onto the spatial amplitude modulator such as a DMD is not a sharp image, it can also be used to provide a smoothing effect by spatially averaging the image.

Aspects of the present invention can be used with any type of projector or projector system comprising light steering capabilities. Thus, aspects of the present invention include a two-projector system consisting of a base projector and a highlighter projector or a hybrid projector with a base projector and a highlighter projector integrated in one projector. The highlighter projector can be a dual phase/amplitude modulation projector.

<FIG> shows a flow diagram to illustrate the steps required to control a highlighter type projector (and a dual projector or hybrid projector system), and to generate, from an input image expressed in RGB laser primaries, a phase pattern, amplitude pattern(s) and power control signal(s) for the light source. Each rectangular block corresponds to a set of operations, parallelograms indicate input and output operations. The set of operations within each box can be executed by a computer program product. These set of operations or algorithms can be run on a projector controller, e.g. having a processing engine such as a microprocessor or an ASIC or an FPGA or similar. Solid arrows indicate required interactions between blocks while dashed arrows indicate optional interactions between blocks. Solid blocks are required for every configuration, dashed blocks are required only for the dual projector setup.

The input to the method is an input image expressed in linear RGB laser primaries <NUM>, which has been derived from an input image after color transformation and linearization. However, the invention is not limited thereto and other types of input images can be provided, for example if different types of light sources are being used.

This image <NUM> is an input to a content mapping block <NUM> responsible to determine the split between a Target Highlight <NUM>, a Target Image <NUM>, and optionally for a dual projector a Target Base Image <NUM> as well as Power Control Signals <NUM>. The Target Highlight <NUM> represents the illumination pattern that should be created by the phase modulator onto the surface of the amplitude modulator and the Target Image <NUM> represents the final image after modulation of the illumination pattern by the amplitude modulator. The Target Base Image <NUM> represents the image to be created by the base projector for a dual type projector setup.

The content mapping block <NUM> is also responsible for remapping content that is unfeasible given the system power budget and imaging parameters. The algorithm can for example first verify if the input image <NUM> is feasible given the system power budget. The content mapping block <NUM> thus generates the target Highlight <NUM>, the target image <NUM> and optionally the Target Base Image <NUM>.

Thus, in the content mapping block the input image <NUM> is processed to determine:.

To form the Target Image <NUM>, i.e. the image generated by the phase modulator at the level of the amplitude modulator, the target highlight <NUM> is used as an input to a phase pattern generation algorithm block. Algorithms such as the one described in <NPL> can be used for the calculation of the phase pattern. However, the invention is not limited thereto and the skilled person will appreciate that other methods are also suitable therefore. The output of the phase pattern generation algorithm block <NUM> is a phase pattern <NUM> which corresponds to the drive parameters needed to effect light-redistribution by the phase modulator. In an ideal world, when this phase pattern is applied to the phase modulator, it would yield an illumination profile illumination pattern that is exactly the same as the target highlight <NUM>. However, for all the reasons listed above, it will not be the case.

The target highlight <NUM> is also used as input to the forward model processing block <NUM>.

As described in <NPL>, the algorithm also makes use of the forward image formation model from simulations to predict the illumination profile present at the second, amplitude-only modulator. The forward image formation model corresponds to the Forward model block in the flow diagram of <FIG>.

Thus, the forward model processing block <NUM> generates a predicted illumination profile image which is further used to calculate the amplitude and base patterns.

The predicted illumination profile image can be generated with system calibration data to have a better prediction of the actual illumination profile illumination pattern. It can also be generated with simulations, although many effects would not be accounted for as they would have to be modelled.

The system calibration data can be captured by an external camera during a one-time calibration procedure, and characterizes how a small spot of light is blurred by the imperfect beam quality of the lasers and by the optical system parameters. The so-called Point Spread Function (PSF) can be characterized for the different colors (lasers) and at different positions. An example of PSF captured for different colors at different positions is shown in <FIG>. A second part of the calibration is capturing an image of the so-called unsteered light, which in a first approximation is a fixed pattern of illumination. The predicted illumination profile image (<NUM> in <FIG>) is based on a blurred version of the target highlight image (<NUM>) summed with the fixed pattern (<NUM>) of the unsteered light, as illustrated in the block diagram of <FIG>.

The forward model block also receives the power control signals. The predicted illumination profile image is multiplied with the power control signal for the highlight laser source. In case of a hybrid projector, also a prediction of the base illumination can be made, since this is a fixed, mostly uniform illumination, the pattern can be easily captured during calibration, and this pattern, multiplied with the power control signal for the base light source can be added to the predicted illumination profile image.

The predicted illumination profile image <NUM> (the predicted illumination profile is analogous to the brightness profile generated by an LCD backlight using an LED matrix with local dimming capability) in combination with the target image <NUM>, is used as input to the Amplitude Pattern Generation Block <NUM> to determine the necessary amplitude signal to drive the second modulator, the amplitude modulator of the highlighter projector or the hybrid projector i.e. the Amplitude Patterns <NUM>. In cases where there is also a base projector, the amplitude pattern generation also creates a Base Pattern. If the base illumination would be perfectly uniform, the Base Pattern could be identical to the Target Base Image. But when the base illumination uniformity is not perfect, color and brightness uniformity correction could be applied to derive the Base Pattern from the Target Base Image.

Optionally, the predicted illumination profile <NUM> from the forward model block <NUM> is used as a feedback signal to the content mapping block <NUM>, illustrated with arrow <NUM> in the flow diagram. This enables to verify if the predicted illumination profile brightness effectively exceeds the target image brightness, such that with the proper amplitude signal to the second modulator this target image brightness can effectively be achieved. Where this is not the case, the content mapping block could then increase the target highlight image and/or the laser power control signal.

One of the aspects of the present invention is a refinement of the one-time calibration with a feedback, e.g. intermittent or continuous feedback mechanism for a projector. To implement such an intermittent or continuous feedback mechanism, an image sensor is provided e.g. is integrated within the projector's optical path or can be brought into the optical path. The image sensor receives an image (or a fraction of the image) of the illumination pattern that is equivalent to the illumination pattern that is incident on the amplitude modulator. The illumination pattern can be related to or adapted to an image such as a video to be projected. Such an image or video is projected as frames.

The image sensor can be driven at the same frequency as the amplitude modulator or the input image or at a multiple of the frequency of the amplitude modulator or the input image. Driving the image sensor at a higher speed has the advantage increasing the speed of the feedback loop.

The image sensor, the phase modulator and the at least one amplitude modulator can be driven by the projector controller. The projector controller can further also drive the light source, i.e. laser sources.

The images of the image sensor are analyzed by means of image analysis software or digital image processing techniques known in the art to retrieve the required information from the images and provide said information as feedback to the controller configured to execute the feedback loop according to the present invention.

The intermittent or continuous feedback could be implemented at any level of the flow diagram of <FIG> wherein the image of the illumination pattern acquired by the image sensor can improve the driving scheme of the projector, thus at different levels, as further explained, e.g. any of:.

The image sensor could for example be integrated in the projector's optics behind a highly reflective dichroic folding mirror. This dichroic folding mirror could be specified to reflect <NUM>% till <NUM>% of the light. The <NUM>% respectively <NUM>% of the light leaking through the mirror would be sufficient to be forwarded to and received by the image sensor. The optical path of the image sensor, has characteristics which preferably resemble as closely as possible to the characteristics of the optical path towards the second spatial amplitude modulator, or in other words, the optical path of the image sensor is substantially optically equivalent (such that it has identical (or nearly identical) optical characteristics) to the optical path of the second spatial amplitude modulator. For example, if the second spatial amplitude modulator is a reflective spatial modulator such as a DMD device, the reflective spatial amplitude modulator such as a DMD is typically at an angle with respect to the optical axis of the incoming beam. In this case it is preferred that also the image sensor is at a similar angle with respect to the optical axis. A spatial amplitude modulator such as a DMD optical system typically uses a TIR prism, where the shape of the prism is optimized to minimize the path length differences incurred by putting the spatial amplitude modulator such as a DMD at an angle with respect to the optical axis. A dummy prism can be introduced in the light path towards the image sensor to create a similar situation. If the image sensor is of a different size than the reflective spatial modulator such as the DMD device the magnification of the imaging optics towards the sensor is preferably different than the magnification of the imaging optics towards the spatial amplitude modulator such as the DMD.

Alternatively, to optically try to replicate the characteristics of the optical path from the intermediate image to the second spatial amplitude modulator, such as a DMD, it would be possible to design the image sensing optics to only capture the intermediate image and apply an electronic correction to take into account the geometric distortion and blurring caused by the rotation of the second spatial amplitude modulator versus the optical axis.

Alternatively or additionally, the image of the spatial amplitude modulator such as a DMD can be a scaled down version. This can be less than <NUM>:<NUM> as this provides the advantage of a reduced cost of the image sensor.

Alternatively or additionally, the image sensor can be positioned in the path of those optical elements that generate the highlight. This may provide the advantage of a better utilization of available space. In case where there is a base illumination added to the highlight illumination, this base illumination is characterized in a calibration step and is considered to be constant over time.

Alternatively or additionally, any discrepancy between the amplitude modulator image, e.g. the DMD image and camera image can be mitigated by means of image processing such as for example image warping, flat field correction and/or location dependent blurring. In this case, such image processing may add a delay and true closed loop driving may no longer be possible. Even so the image sensor with image processing can still be used for calibration, static image feedback or for laser safety detection.

Alternatively or additionally, the image sensor can receive light from reflected light of an inserted glass plate rather than the image sensor being placed behind a folding mirror. This may provide the advantage of a better utilization of available space.

Alternatively or additionally, the camera can be at a high frame rate (e.g. <NUM>). Sampling can be at low resolution (e.g. 40x21) for example when the camera is used to monitor laser safety. A switch to a higher resolution at lower framerate can be provided to perform calibration, e.g. whenever time allows when the projector is not used and the amplitude modulator can be set to black, such as the time between the finish of one movie and the start of another.

It is important to note that the feature of providing substantially optically equivalent characteristics can be provided optically but also electronically. It is possible to have imperfect optical equivalence being corrected electronically. What is required is that the intermediate image is relayed onto both the second modulator and onto the sensor whereby most of the light ends up on the second modulator and only a very small part onto the sensor.

The resolution of the image in the intermediate image plane (in the plane of the phase modulator) is very limited. The point spread function will span hundreds to several hundreds of pixels on the second modulator (amplitude modulator). The image sensor can therefore capture the image with a resolution well below the native resolution of the second modulator, as the low-resolution image from the sensor, can be up-sampled with good accuracy.

<FIG> is a schematic representation of part of the optical path of a highlighter projector in accordance with an embodiment of the present invention. A light source such as a laser light source (not shown) provides a beam <NUM> which is incident on a phase modulator <NUM>. In the present embodiment, the phase modulator <NUM> is a reflective phase modulator but it could be a transmissive phase modulator. Illumination and sensing imaging optics <NUM> are provided along the optical path after the diffuser <NUM>. A mirror <NUM> reflects the intermediate image onto a prism structure <NUM>, after Illumination imaging optics means <NUM>. From the amplitude modulator <NUM> via prism structure <NUM> the final image is projected through a projection lens (not shown). A small amount of light also passes through the mirror <NUM> and falls onto sensing and imaging optics <NUM>, a dummy prism <NUM> and onto an image sensor <NUM>. The small amount of light for example can be provided by the mirror <NUM> being half-silvered.

For embodiments comprising a dual type projector, i.e. a highlighter and a base projector, the images generated by both projectors are superposed on the projection screen. For embodiments comprising a hybrid type projector, the beams of the highlighter and of the base are superposed upstream of the amplitude modulator <NUM>, for each color.

The maximum steer angle from the phase modulator <NUM> (or in fact from any such MEMS device) will typically be identical in horizontal and vertical directions. When the final image is wider than it is high (as is typically the case in modern <NUM>:<NUM> aspect ratio displays or cinema formats), and if the system is designed such that light can be steered by the phase modulator across the full width of the intermediate image <NUM> but beyond the active area representing that part of the intermediate image <NUM> that is for display, there is a possibility to steer the light beyond the top and bottom edges of the active area, e.g. beyond the edges of the active area on the intermediate image <NUM> which represents beyond the height of the image to be displayed.

This addressable area <NUM>, beyond the active area <NUM> can be used to generate calibration patterns (as shown in <FIG> as upper and/or lower calibration patterns, <NUM>, <NUM> respectively) in the factory or during normal operation of the projector. This would allow an intermittent or continuous refinement of the Forward Model <NUM>, compensating any drift in the opto-mechanics, the phase modulator and/or the laser source.

The level in the background of the calibration patterns can be used as an indicator for the amount of unsteered light. This information can then also be coupled back to the Forward Model <NUM>, and be used to calculate the next predicted illumination profile in cases where the content is predominantly static.

Additional laser energy will be required to create those calibration patterns outside the active area, but the calibration can be activated only if and when the image content does not require all the available laser energy. In this case, and if the laser source does not accommodate fast dimming, steering outside the active area might be the only solution to dump the excess light.

The optical path and mechanics have to be designed to image the parts of the phase modulator, which correspond to the entire addressable area on the image sensor, on the moving diffuser and from the moving diffuser onto the image sensor. In the imaging path towards the amplitude modulator however it would be preferred to block the light outside the active area (e.g. with a cooled aperture), before it reaches the position of amplitude modulator and could cause undesirable heating and straylight.

The image sensor can be a full color sensor (e.g. with Bayer filter) or a monochrome sensor. In the latter case the test patterns for calibration can be presented sequentially per primary color.

A calibration procedure using the active area <NUM> can be executed during startup or shutdown of the projector, or more generally when there is no need for image formation on the projection screen. This calibration is referred to as daily calibration (although it can also be performed only when required by the system or on a periodic basis).

A calibration data using the addressable areas <NUM> and <NUM> can be performed during projection, or in real time.

The flow diagram in <FIG> has functional blocks - see table <NUM>. As illustrated in the flow diagram of <FIG>, the Forward Model block <NUM>, can now consider multiple calibration inputs:.

The forward model thus now further receives (near) real-time calibration data from the internal image sensor <NUM> and can generate a more accurate predicted illumination profile <NUM> where slow and medium speed drift of the system parameter can be mitigated.

In the claimed invention, the sensor is also used to safeguard the maximum light level in the highlights in order to guarantee the light levels in front of the projection lens stay within the required limits to avoid exposing the audience to unsafe levels of laser light. What levels of highlights could be tolerated will be dependent upon the installation (such as the type of projection lens used, the position where the audience can interfere with the projected light beam. The maximum light level will therefore need to be calibrated upon installation of the projector. Thus, a threshold can be established during the calibration upon installation of the projector for example. The content mapping block will already tone map the content to stay below the set limit. However, the sensor could provide a second safeguard system in case something goes wrong in the algorithm such that too much light is concentrated in a certain location.

If the sensor detects brightness levels exceeding the set limit or the threshold, it will turn down or fully shutdown the laser source(s) via the power control signals.

It is described that the image (Sensor Image <NUM> Inside Active Area) captured by the image sensor <NUM> in the active area (the actual illumination profile image <NUM>) is taken as input to the Content Mapping block <NUM>.

For static and slowly varying content, the brightness levels of the images will vary slowly, and it can thus be assumed that the brightness levels in two successive frames will be substantially identical. It can thus be beneficial to correct the brightness error made in the target highlight for frame n (current frame for example) by adapting the brightness level of the target highlight used for frame n+<NUM> (following frame). A ratio between the predicted illumination profile brightness level and the actual measured illumination profile brightness level (from the actual illumination profile image, which can be processed to be linearized and normalized) is made and used as a multiplication factor for the static content in the next frame n+<NUM> of the illumination profile. A new drive signal for the first modulator will then be calculated to try to achieve the corrected target brightness level in the illumination profile. This new input to the Content Mapping block <NUM> can also be used to calculate the Target Image in the next frame and on the Power control signals for the next frame.

A motion detection mechanism can further be used to determine which part of the image frame is static and which part is dynamic and decide, on a per pixel basis or region basis, if the original image or the corrected image will be used. A weighted average between the two inputs can be used to obtain soft rather than hard transitions between adjacent pixels.

The image (Sensor Image <NUM> Inside Active Area) captured by the image sensor <NUM> in the active area (the actual illumination profile) and the predicted illumination profile from the previous image frame can also be taken as an input to the content mapping block, together with the input image of the current frame, as shown in the block diagram of <FIG>. The flow diagram in <FIG> has functional blocks - see table <NUM>.

In portions of the image where the content is static (input image for the current image frame is substantially identical or identical to the previous image frame), the predicted illumination profile is compared to the actual illumination profile.

Where the content is moving, motion vectors can be derived to select the corresponding image sections out of the actual illumination profile image and predicted illumination profile image from the previous frame, and apply the same correction mechanisms described above for static content.

Where content has no relation with the previous frame, the Content Mapping block <NUM> is executing the standard algorithm, ignoring both the predicted and actual illumination profile information from the previous frame.

The main problem with the sensor is that it supplies the information a frame too late. The illumination profile pattern needs to be there first before the image sensor can start sensing. Only at the end of the image sensor frame is the information available. Thus, the actual illumination profile information is only available <NUM> frame later. The image sensor can run faster than the projector (as explained later in the description, in reference to <FIG>) where the image sensor runs at sub-frame speed and a first feedback is available before the frame finishes. The information of the previous frame can also be used, but only on the condition that the content is static.

For this embodiment according to the present invention, the following approach is used:.

<FIG> shows the pixel brightness profiles across a row of pixels, where the x-axis is the horizontal position of the pixel. The solid line <NUM> indicates the target image brightness finally delivered after the second modulator. The dotted line <NUM> indicates the predicted illumination profile signal, while the dashed-dotted line <NUM> indicates the actual illumination profile signal as derived with the image sensor <NUM>.

During the first subframe(s) having a duration ti, the information from the image sensor <NUM> is not yet available. The drive signal for the second modulator (Amplitude Pattern) is now calculated as: <MAT>.

During the remaining time of the frame with a duration t frame - ti the actual illumination profile as measured by the image sensor can be used (after normalization to maximum white level) and a correction term to adjust for the error made during the initial sub-frame(s) is applied.

<FIG> illustrates an example, with a camera (i.e. the image sensor <NUM>) running at three times the frame period of the video signal. The camera output signal is delayed by <NUM>/<NUM> of the video frame period. Only when the output of the camera is available, the corrected PWM scheme can be calculated so it can be applied in the next sub-frame. The spatial amplitude modulator such as a DMD will be driven by an initial PWM scheme during the first <NUM> sub-frames. Only in the third sub-frame a corrected PWM scheme can be applied that takes into account the actual brightness level acquired by the camera in sub-frame <NUM>.

In practice, and especially when a slow modulator (like an LCOS phase modulator) is used to create the beam steering, the brightness might be changing over the duration of the frame. If the content is dynamic, then the information of the last sub-frame <NUM> is of no use. For static content however, it would therefore be beneficial to use the actual brightness level acquired by the camera in all sub-frames. Brightness deviations that might have occurred during the course of frame <NUM> can be compensated for during the course of frame <NUM>.

In the case of static content, the drive signal for the initial sub-frame(s) can be calculated as: <MAT>.

In this case the amplitude signal for the remaining time of the frame will be calculated as: <MAT> wherein the suffixes cf and pf stand for current frame and previous frame, respectively.

The generation of the drive signals towards the spatial amplitude modulator such as a DMD thus takes into account the illumination brightness level as predicted by the Forward Model <NUM> or by target brightness level (further called the predicted brightness level), the actual illumination brightness level as measured by the high-speed image sensor, the current target image and the target image from the previous frame. If the current target image is identical to the target image from the previous frame, it will start the frame by taking into account the ratio between the measured brightness level and the target image. If the target image is different from the previous frame, it will start the frame by taking into account the ratio between the predicted brightness level and the target image. Once the measured brightness level for the new frame is available the measured brightness level data will be used and a correction will be made for any error in the previous part of the frame because of a difference between the predicted brightness level and the measured brightness level.

The image sensor should measure the intensity in the three primary colors independently, and in this case, should be a full color sensor (e.g. with Bayer filter).

As illustrated in the flow diagram of <FIG>, in the algorithm the actual illumination profile image captured by the image sensor is now used as an additional input to the amplitude pattern generation block, next to the predicted illumination profile signal. The flow diagram in <FIG> has functional blocks - see table <NUM>.

For the calculation of the initial PWM frame, when the actual illumination profile signal is not yet available (the information available is still from the previous frame), the amplitude signal is determined from the predicted illumination profile signal and the target image.

For the calculation of the corrected PWM frame, the actual illumination profile signal is used as a basis, onto which a correction is applied to compensate for the difference between the actual illumination profile and the predicted illumination profile in the initial PWM frame. The latter compensation is weighted according to the relative time duration between the initial frame and the corrected frame(s).

As long as the actual measured illumination brightness is sufficiently high to deliver the target image brightness and compensate for any shortage in image brightness resulting from the wrong predictions used in the previous sub-frame(s), the end result should be a perfect reproduction of the target image. This is, however, on the condition that what is measured by the sensor is an accurate representation of the brightness level at the spatial amplitude modulator such as a DMD. Also, here a one-time calibration of uniformity, geometry and color sensitivity will be required. It is anticipated that such a calibration, however, would be much more stable than the highlight beam generation itself.

With this method limited drift over time and temperature of the opto-mechanical systems, the phase modulator or the laser source can be mitigated. In addition, a variation on a frame by frame basis of the unsteered component and temporal effects because of the response speed of the phase modulator can be taken into account.

In case of a hybrid projector, where both a highlight illumination and a base illumination are combined on the same amplitude modulator, the sensor <NUM> receives this same combination (as a fixed portion of the total illumination towards the amplitude modulator). In this case, we define the target image as the full image i.e. the combination of the target highlight and the target base. Then the method to derive the amplitude pattern generation is the same as for the highlighter projector.

While the previous method can address various types of drift, deal with a small variable of unsteered light component, and transition effects between frames, it cannot accommodate large drifts in the steering position or in the intensity.

A hybrid model can be used to combine the strengths of the previous methods, as illustrated in the flow diagram of <FIG>. The flow diagram in <FIG> has functional blocks - see table <NUM>.

Here slow but potentially large drifts can be addressed by a near real-time feedback on the parameters of the forward model. And those parameters of the forward model can further be updated (e.g. on a daily basis) using a set of calibration patterns in the active area during projector shutdown. For static content any residual errors can be compensated for in the next image frame. Further a final correction including a correction for dynamic effects can be implemented using a real-time feedback on the driving signal of the amplitude modulator.

A single image sensor can be used that captures both the active area and the test patterns generated outside the active area. In other embodiments, separate image sensors could acquire different portions of the light beam which impinges on the phase modulator, for example an image sensor which images the information in the active area (or the pupil of the projection lens) and an image sensor which images the information which is outside of the active area at the plane of the diffuser.

In other embodiments, an image sensor can also be implemented to image the light beam after being reflected or transmitted by the amplitude modulator.

In embodiments of the present invention, the projector or projection system can comprise one phase modulator and an amplitude modulator in every color channel. Embodiments of the present invention can also be implemented with a projector or projection system comprising color sequential operation with one phase modulator and one amplitude modulator however current LCOS phase modulators are not fast enough. In other embodiments according to the present invention, the projector or projection system can comprise a phase modulator per channel and a single spatial amplitude modulator such as a DMD amplitude modulator operating in color sequential mode. In both color sequential modes, one would then need to pulse the red, green and blue lasers sequentially, which can have drawbacks in terms of light efficiency.

Methods according to the present invention can be performed by a control unit such as a control unit or a processing device or any control unit for use with embodiments of the present invention including microcontrollers, either as a standalone device or embedded in a projector or as part of an optical subsystem for a projector. The present invention can use a processing engine being adapted to carry out functions. The processing engine preferably has processing capability such as provided by one or more microprocessors, FPGA's, or a central processing unit (CPU) and/or a Graphics Processing Unit (GPU), and which is adapted to carry out the respective functions by being programmed with software, i.e. one or more computer programs. References to software can encompass any type of programs in any language executable directly or indirectly by a processor, either via a compiled or interpretative language. The implementation of any of the methods of the present invention can be performed by logic circuits, electronic hardware, processors or circuitry which can encompass any kind of logic or analog circuitry, integrated to any degree, and not limited to general purpose processors, digital signal processors, ASICs, FPGAs, discrete components or transistor logic gates and similar.

Such a control unit or a processing device may have memory (such as non-transitory computer readable medium, RAM and/or ROM), an operating system, optionally a display such as a fixed format display, ports for data entry devices such as a keyboard, a pointer device such as a "mouse", serial or parallel ports to communicate other devices, network cards and connections to connect to any of the networks.

The software can be embodied in a computer program product adapted to carry out the functions of any of the methods of the present invention, e.g. as itemised below when the software is loaded onto the controller and executed on one or more processing engines such as microprocessors, ASICs, FPGAs etc. Hence a processing device control unit for use with any of the embodiments of the present invention can incorporate a computer system capable of running one or more computer applications in the form of computer software.

The methods described with respect to embodiments of the claimed invention can be performed by one or more computer application programs running on the computer system by being loaded into a memory and run on or in association with an operating system such as WindowsTM supplied by Microsoft Corp, USA, Linux, Android or similar. The computer system can include a main memory, preferably random-access memory (RAM), and may also include a non-transitory hard disk drive and/or a removable non-transitory memory, and/or a non-transitory solid state memory. Non-transitory removable memory can be an optical disk such as a compact disc (CD-ROM or DVD-ROM), a magnetic tape, which is read by and written to by a suitable reader. The removable non-transitory memory can be a computer readable medium having stored therein computer software and/or data. The non-volatile storage memory can be used to store persistent information that should not be lost if the computer system is powered down. The application programs may use and store information in the non-volatile memory.

The software embodied in the computer program product is adapted to carry out any of the steps of the methods according to any of the claims <NUM> to <NUM> when the software is loaded onto the respective device or devices and executed on one or more processing engines such as microprocessors, ASICs, FPGAs, etc..

Software embodied in the computer program product can be adapted to carry out the following functions when the software is loaded onto the respective device or devices and executed on one or more processing engines such as microprocessors, ASICs, FPGAs etc.:.

The software embodied in the computer program product is adapted to carry out the following functions when the software is loaded onto the respective device or devices and executed on one or more processing engines such as microprocessors, ASICs, FPGAs etc.:.

The software embodied in the computer program product is adapted to carry out the following functions when the software is loaded onto the respective device or devices and executed on one or more processing engines such as microprocessors, ASICs, FPGAs etc.:
the at least a portion of the illumination pattern substantially equivalent to the illumination pattern incident on the amplitude modulator can be a scaled down version, e.g. the scaled down version can be less than <NUM>:<NUM>.

Making use of the image sensor being positioned in the path of those optical elements that generate the highlight.

The software embodied in the computer program product is adapted to carry out the following functions when the software is loaded onto the respective device or devices and executed on one or more processing engines such as microprocessors, ASICs, FPGAs etc.:
Operating the camera at a higher frame rate and sampling is at lower resolution when the camera is used to monitor laser safety.

Arranging a switch to provide switching to a higher resolution to perform calibration.

Any of the above software may be implemented as a computer program product which has been compiled for a processing engine in any of the servers or nodes of the network. The computer program product may be stored on a non-transitory signal storage medium such as an optical disk (CD-ROM or DVD-ROM), a digital magnetic tape, a magnetic disk, a solid-state memory such as a USB flash memory, a ROM, etc..

Claim 1:
Method of driving a projector system comprising a laser light source, a phase modulator (<NUM>), and at least one amplitude modulator (<NUM>), the laser light source providing a beam incident on the phase modulator (<NUM>), and the phase modulator (<NUM>) being configured to generate a highlight image incident on the at least one amplitude modulator (<NUM>), the projector system further comprising at least one image sensor (<NUM>) configured to receive at least a portion of an illumination pattern optically equivalent to the illumination pattern generated by the phase modulator (<NUM>) and incident on the at least one amplitude modulator (<NUM>), the method comprising the steps of
<NUM>) receiving an input image (<NUM>),
<NUM>) generating a target highlight image (<NUM>), a target image (<NUM>) representing the final image to be reproduced after modulation by the at least one amplitude modulator (<NUM>) of the illumination pattern generated by the phase modulator (<NUM>), and power control signals (<NUM>) for the laser light source from the input image,
a) generating a phase pattern (<NUM>) for driving the phase modulator (<NUM>) from the target highlight image (<NUM>),
b) generating a predicted illumination profile image corresponding to a simulated illumination profile present at the at least one amplitude modulator (<NUM>),
c) generating an amplitude pattern (<NUM>) for driving the at least one amplitude modulator (<NUM>) from the predicted illumination profile image and the target image (<NUM>),
<NUM>) receiving an image from the at least one image sensor (<NUM>) to provide feedback to at least one of the method steps <NUM>), 2a), 2b), 2c) for driving the projector system; and wherein, if the at least one image sensor (<NUM>) detects brightness levels exceeding a threshold, turn down or fully shut down the laser light source via the power control signals (<NUM>).