Patent Description:
Multiple-modulation projection systems are known. Typical systems use two (or more) amplitude modulating spatial light modulators (SLMs) to produce high-dynamic range images. The first SLM, the pre-modulator, creates a low-resolution image on the second SLM, the primary modulator. The two images multiply optically to create very high contrast ratios. In dual modulation projection systems, dark areas of the image are realized by blocking light in both modulators. However, because some SLMs can only attenuate amplitude, the maximum brightness of a small image portion is the same as the maximum brightness of the full image (i.e. diffuse white).

Another method utilizes beam-steering rather than amplitude modulation to generate the low resolution image on the second SLM. Using a beam-steering SLM (e.g., a phase modulating SLM, tip-tilt mirror device, etc.) as the pre-modulator, light is steered to a location where greater intensity is needed, rather than being directed to a light dump. For example, if a small highlight area is required on a dark background, a beam-steering pre-modulator can steer the unused light from the dark background into the highlight area. Thus, highlighted areas may be displayed at a brighter intensity level than the diffuse white intensity level. Because most images only use high brightness in small areas, a beam-steering pre-modulator is useful for reducing cost (reducing the luminous flux requirement for the same effective peak brightness) and/or for improving the performance (e.g., dynamic range, color depth, etc.), thereby creating brighter and, hence, more compelling images.

However, beam-steering SLMs tend to switch at relatively slow frequencies as compared to most amplitude pre-modulators and primary modulators. They generally run at a lower frame rate than amplitude modulators, and they may change state relatively slowly between the images (i.e. spatial phase distributions) sent to them. Furthermore, because beam-steering SLMs use optical phase modulation and the resulting interference to steer light, the transition between two images might not represent a smooth crossfade between the two images arriving at the primary modulator.

Some dual-modulation systems use a light field simulation to model the image arriving on the primary modulator and determine how to drive the primary modulator to achieve a desired image. In a beam-steering system the light field simulation involves modeling the behavior of the light leaving the phase modulator. Because the physical image arriving at the primary modulator changes significantly between pre-modulator frames, and because the lightfield simulation is based on the phase image driving the pre-modulator (and the resulting complex interference patterns), the light field simulation during a transition is inaccurate and can cause visual artifacts.

Document <CIT> discloses a projection system combining one spatial light modulator that affects only the phase of the illumination, and one spatial light modulator that only affects its amplitude (intensity). The phase-only modulator curves the wavefront of light and acts as a pre-modulator for a conventional amplitude modulator.

Document <CIT> proposes a voltage overdrive of a phase-controlled siliconbased liquid crystal device to achieve a faster response speed.

The present invention overcomes the problems associated with the prior art by providing systems and methods for using temporal models of phasemodulation spatial light modulators (SLMs) to generate light field simulations. Aspects of the invention facilitate the generation of light field simulations at a higher frame rate than the rate at which phase modulation data is loaded into the phase-modulating SLM. As a result, a amplitude-modulating SLM can be driven at a much higher data frame rate than the maximum frame rate of the phase-modulating SLM.

According to a first aspect, a method for generating an image in a projector system includes receiving a first frame of image data and generating a first set of phase drive values for driving a phase-modulating spatial light modulator (SLM). The first set of phase drive values is based at least in part on the first frame of image data. The method additionally includes driving the phase-modulating SLM with the first set of phase drive values during a first time period, receiving a second frame of image data, and generating a second set of phase drive values for driving the phase-modulating SLM. The second set of phase drive values is based at least in part on the second frame of image data. The example method additionally includes driving the phase-modulating SLM with the second set of phase drive values during a second time period and modeling transitional states of the phase-modulating SLM during the second time period. The transitional states are based at least in part on the first set of phase drive values and the second set of phase drive values. The example method additionally includes generating a set of lightfield simulations of lightfields generated by the phase-modulating SLM and incident on an amplitude-modulating SLM. A first subset of the set of lightfield simulations corresponds to the first set of phase drive values, a second subset of the set of lightfield simulations corresponds to the second set of phase drive values. A third subset of the set of lightfield simulations corresponds to one or more of the transitional states of the phase-modulating SLM. The example method additionally includes generating sets of amplitude drive values for driving the amplitude-modulating SLM and driving the amplitude-modulating SLM with the sets of amplitude drive values. Each of the sets of amplitude drive values corresponds to an associated one of the lightfield simulations.

In a particular example method, the first frame and the second frame are temporally sequential, and the third subset of lightfield simulations includes exactly one lightfield simulation. Alternatively, the third subset of lightfield simulations includes more than one lightfield simulation. As another alternative, the first frame and the second frame are not temporally sequential, and the third subset of lightfield simulations corresponds to at least one intervening frame temporally between the first frame and the second frame.

In a particular example method, the step of generating a second set of phase drive values includes utilizing the first set of phase drive values as an initial approximation of the second set of phase drive values and altering the initial approximation based on the third frame of image data.

In another particular example method, the phase-modulating SLM comprises a plurality of pixels, and the step of modeling transitional states of the phase-modulating SLM during the second time period includes modeling individual transitional states of individual pixels of the plurality of pixels. In a more particular method, the step of driving the phase-modulating SLM with the first set of phase drive values includes asserting a first set of voltages across the plurality of pixels, each voltage of the first set of voltages indicated by an associated value of the first set of phase drive values. The step of driving the phase-modulating SLM with the second set of phase drive values includes asserting a second set of voltages across the plurality of pixels, each voltage of the second set of voltages indicated by an associated value of the second set of phase drive values. The step of modeling individual transitional states of the plurality of pixels includes determining an individual transition of each pixel of the plurality of pixels from a corresponding voltage of the first set of voltages to a corresponding voltage of the second set of voltages. In an even more particular example method, the step of modeling transitional states of the phase-modulating SLM includes determining a transitional state of each pixel of the plurality of pixels at a particular time during the individual transition of each pixel. Alternatively, the step of modeling transitional states of the phase-modulating SLM can include determining an average state of each of pixel of the plurality of pixels over a period of time during the individual transition of each the pixel.

The step of modeling transitional states of the phase-modulating SLM can include modeling transitional states of the phase-modulating SLM based at least in part on a physical characteristic of the phase-modulating SLM. The step of modeling transitional states of the phase-modulating SLM can also include modeling transitional states of the phase-modulating SLM based at least in part on a physical characteristic of a liquid crystal layer of the phase-modulating SLM.

The step of modeling transitional states of the phase-modulating SLM can also include modeling transitional states of the phase-modulating SLM based at least in part on a physical characteristic of a lightfield incident on the phase-modulating SLM. The step of modeling transitional states of the phase-modulating SLM based at least in part on a physical characteristic of a lightfield incident on the phase-modulating SLM can include modeling transitional states of the phase modulating SLM based at least in part on a history of the lightfield.

In a particular example method, the step of generating a set of lightfield simulations includes modeling each of a plurality of pixels of the phase-modulating SLM as a source of spherical waves having a phase delay determined at least in part based on a corresponding one of the transitional states. Alternatively, the step of generating a set of lightfield simulations can include modeling each of a plurality of pixels of the phase-modulating SLM as an origin point of a light ray having an angle with respect to a surface of the phase-modulating SLM. The angle is determined at least in part based on a corresponding one of the transitional states.

Example controllers for controlling a projection system are also disclosed. One example controller includes a processing unit configured to execute code, an interface coupled to receive a first frame of image data, a second frame of image data, and a third frame of image data, and memory electrically coupled to store data and the code. The data and the code include a phase drive module, a temporal lightfield simulation module, and an amplitude drive module.

The phase drive module is configured to generate a first set of phase drive values for driving a phase-modulating spatial light modulator (SLM). The first set of phase drive values is based at least in part on the first frame of image data. The phase drive module is also configured to generate a second set of phase drive values for driving the phase-modulating SLM. The second set of phase drive values is based at least in part on the second frame of image data. The phase drive module is also configured to drive the phase-modulating SLM with the first set of phase drive values during a first time period and drive the phase-modulating SLM with the second set of phase drive values during a second time period.

The temporal lightfield simulation module is configured to model transitional states of the phase-modulating SLM during the second time period. The transitional states are based at least in part on the first phase drive values and the second phase drive values. The temporal lightfield simulation module is also configured to generate a set of lightfield simulations of lightfields generated by the phase-modulating SLM and incident on an amplitude-modulating SLM. A first subset of the set of lightfield simulations corresponds to the first set of phase drive values, a second subset of the set of lightfield simulations corresponds to the second set of phase drive values, and a third subset of the set of lightfield simulations corresponds to one or more of the transitional states of the phase-modulating SLM.

The amplitude drive module is configured to generate sets of amplitude drive values for driving the amplitude-modulating SLM. Each of the sets of amplitude drive values corresponds to one lightfield simulation of the set of lightfield simulations. The amplitude drive module is additionally configured to drive the amplitude SLM with the sets of amplitude drive values.

In a particular example controller, the first frame and the second frame are temporally sequential, and the third subset of lightfield simulations includes exactly one lightfield simulation. Alternatively, the third subset of lightfield simulations includes more than one lightfield simulation. As another alternative, the first frame and the second frame are not temporally sequential, and the third subset of lightfield simulations corresponds to at least one intervening frame temporally between the first frame and the second frame.

In a particular example controller, the phase drive module is configured to utilize the first set of phase drive values as an initial approximation of the second set of phase drive values when generating the second set of phase drive values. The phase drive module is further configured to alter the initial approximation of the second set of phase drive values based on the third frame of image data.

In another particular example controller, the phase-modulating SLM comprises a plurality of pixels, and the temporal lightfield simulation module is configured to model individual transitional states of pixels of the plurality of pixels. The phase drive module is configured to assert a first set of voltages across the plurality of pixels. Each voltage of the first set of voltages is based on the first set of phase drive values. The phase drive module is also configured to assert a second set of voltages across the plurality of pixels. Each voltage of the second set of voltages is based on the second set of phase drive values. The temporal lightfield simulation module is configured to determine an individual transition of each pixel of the plurality of pixels from a corresponding voltage of the first set of voltages to a corresponding voltage of the second set of voltages. The temporal lightfield simulation module can be configured to determine a transitional state of each pixel of the plurality of pixels at a particular time during the individual transition. Alternatively, the temporal lightfield simulation module can configured to determine an average transitional state of each of the plurality of pixels over a period of time during the individual transition.

In an example controller, the temporal lightfield simulation module is configured to model transitional states of the phase-modulating SLM based at least in part on a physical characteristic of the phase-modulating SLM. For example, the temporal lightfield simulation module can be configured to model transitional states of the phase-modulating SLM based at least in part on a physical characteristic of a liquid crystal layer of the phase-modulating SLM. As another option, the temporal lightfield simulation module can be configured to model transitional states of the phase-modulating SLM based at least in part on a characteristic (e.g., intensity, wavelength, bandwidth, etc.) of a lightfield incident on the phase-modulating SLM. For example, the temporal lightfield simulation module is configured to model transitional states of the phase modulating SLM based at least in part on a history of the lightfield.

In a particular example controller, the temporal lightfield simulation module models each pixel of a plurality of pixels of the phase-modulating SLM as a source of spherical waves having a phase delay determined at least in part by a corresponding one of the transitional states. In another particular example controller, the temporal lightfield simulation module models each pixel of a plurality of pixels of the phase-modulating SLM as an origin point of a light ray having an angle with respect to a surface of the phase-modulating SLM. The angle is determined at least in part by a corresponding one of the transitional states.

Example non-transitory, computer-readable media are also disclosed. One example non-transitory, computer-readable medium has code embodied therein for causing a projection system to receive (n) frames of image data. The code additionally causes the projection system to generate (m) frames of phase drive values where m < n. Each frame of phase drive values is based at least in part on an associated frame of the image data and causes a phase-modulating spatial light modulator (SLM) to be in an associated phase state to generate a lightfield corresponding to one of the frames of image data. The code additionally causes the projection system to determine (p) transitional phase states of the phase-modulating SLM, where p > <NUM>. Each of the transitional phase states is indicative of a lightfield generated by the phase-modulating SLM during a transition between sequential ones of the phase states associated with the phase drive values. The code additionally causes a projection system to generate a set of lightfield simulations based on the phase drive values and the transitional phase states, each of the lightfield simulations is indicative of a lightfield generated by the phase-modulating SLM and incident on an amplitude-modulating SLM. The code additionally causes a projection system to generate a set of frames of amplitude drive values based on the set of lightfield simulations and the set of frames of image data.

Indeed, any of the methods disclosed herein may be implemented with a transitory or non-transitory, electronically-readable medium having code embodied therein that, when executed, will cause an electronic device to perform the disclosed method. Examples, of non-transitory electronically readable media include, but are not limited to, volatile memory, non-volatile memory, hardware, software, firmware, and/or any combination of the foregoing examples.

The present invention is described with reference to the following drawings, wherein like reference numbers denote substantially similar elements:.

The present invention overcomes problems associated with the prior art, by providing a system and method for generating lightfield simulations of lightfields produced by a phase-modulating SLM during transition periods. In the following description, numerous specific details are set forth (e.g., switching frequencies of modulators) in order to provide a thorough understanding of the invention. Those skilled in the art will recognize, however, that the invention may be practiced apart from these specific details. In other instances, details of well-known projection practices (e.g., data manipulation, routine optimization, etc.) and components have been omitted, so as not to unnecessarily obscure the present invention.

<FIG> is a block diagram of a dual-modulation projection system <NUM>. Projection system <NUM> generates high quality images from image data and includes a light source <NUM>, a phase modulator <NUM>, an amplitude modulator <NUM>, projection optics <NUM>, and a controller <NUM>. Light source <NUM> shines a flat lightfield onto phase modulator <NUM>. Phase modulator <NUM> is an analog liquid crystal phase modulator that selectively steers (i.e. spatially modulates the phase of) portions of the light comprising the flat lightfield through a set of intermediate optics <NUM> and onto amplitude modulator <NUM>, to form an intermediate image on the surface of amplitude modulator <NUM>. Phase modulator <NUM> could be a digital modulator, but analog modulators provide certain advantages, at least in this application, over digital modulators. Amplitude modulator <NUM> spatially modulates the intermediate image to form a final image, which is directed toward projection optics <NUM>. Projection optics <NUM> includes a set of lenses, prisms, and/or mirrors, which direct the final image toward a display screen or other surface (not shown) to be viewed by an audience.

Controller <NUM> controls and coordinates the other elements of projection system <NUM>, based on image data received from a data source (not shown). Controller <NUM> provides control instructions to light source <NUM>, phase modulator <NUM>, and amplitude modulator <NUM>, based at least in part on the received image data. The control instructions include, for example, phase drive values and amplitude drive values sent to phase modulator <NUM> and amplitude modulator <NUM>, respectively. Each phase drive value is a digital value corresponding to a voltage to be applied to a corresponding pixel of phase modulator <NUM>, in order to cause that pixel to impart a particular phase delay (e.g. π radians) on light that is incident on it. Each amplitude drive value is a multi-bit value corresponding to a time-averaged voltage (e.g., pulse width modulation)to be applied to a corresponding pixel of amplitude modulator <NUM>, in order to cause that pixel to impart a particular amplitude change (e.g., a grayscale level) to light incident on it. A set of phase drive values or amplitude drive values is a plurality of phase drive values or amplitude drive values, each corresponding to a pixel of phase modulator <NUM> or amplitude modulator <NUM>, respectively, and being represented by a matrix having the same resolution as phase modulator <NUM> or amplitude modulator <NUM>. One set of control instructions, having a drive value for each pixel of a modulator, is typically referred to as a frame of control/image data. These control instructions drive phase modulator <NUM> and amplitude modulator <NUM> in order to generate the intermediate and final images.

Controller <NUM> includes a temporal lightfield simulation module <NUM>, which is discussed in detail below, to model phase states of phase modulator <NUM> and to generate lightfield simulations of the intermediate image, based on the modelled phase states. The phase states of phase modulator <NUM> are collections of the physical phase delays imparted by each pixel of phase modulator <NUM> at a particular time, represented by a matrix having the same resolution as phase modulator <NUM>. Because phase modulator <NUM> is an analog device, each phase drive value corresponds to a particular phase delay, but each phase delay does not necessarily correspond to a particular phase drive value. Alternatively, phase modulator <NUM> can be a digital device.

In the example embodiment, light source <NUM> is a low-etendue light source including an array of tunable lasers. In alternate embodiments, light source <NUM> can be replaced by an array of light-emitting diodes (LEDs), a dimmable bulb, or any other suitable light source, including those now known or yet to be invented, in combination with suitable optics to provide a low-etendue lightsource. Additionally, phase modulator <NUM> and amplitude modulator <NUM> can be liquid crystal phase and amplitude spatial light modulators (SLMs), respectively. In alternate embodiments, amplitude modulator <NUM> can be a digital micromirror device (DMD), a reflective liquid-crystal-on-silicon (LCOS) device, or any other suitable amplitude modulating device, including those now known or yet to be invented.

In the description of example embodiments phase modulator <NUM> and amplitude modulator <NUM> are thus named to distinguish between an SLM that is used to steer light to create a lightfield on a primary modulator (phase modulator <NUM>) and an SLM that modulates selected portions of the lightfield to create an image for viewing (amplitude modulator <NUM>). However, these terms are not used in a limiting sense. For example, DMDs selectively steer light along or out of an optical path, but are used as amplitude modulators by time multiplexing the amount of light steered into or out of an image to create an intermediate gray level (perceived amplitude modulation). As another example, liquid crystal SLMs selectively alter the phase of light and can, therefore, be considered phase modulating or beam-steering devices. However, the birefringent property of liquid crystals also results in polarization rotation, and so liquid crystal SLMs can be used with internal or external polarizers to provide amplitude modulation. Therefore, devices referred to as "amplitude modulators", "phase modulators", or "beam-steering modulators" are understood to include any device capable of performing the titled function, either alone or in combination with other devices. Furthermore, although phase modulator <NUM> and amplitude modulator <NUM> appear as transmissive devices in <FIG>, phase modulator <NUM> and amplitude modulator <NUM> could also be, and more likely would be, reflective devices.

<FIG> is a timing diagram, illustrating an example method for generating images in projection system <NUM>. First, controller <NUM> receives image data at a rate of <NUM> frames per second (fps). Then controller <NUM> uses half of the frames of image data (e.g., every other frame) to generate sets of phase drive values at a decreased rate (<NUM> fps in the example embodiment), because phase modulator <NUM> is incapable of switching fast enough to receive sets of phase drive values at the same rate the image data is received. Controller <NUM> sends the sets of phase drive values to phase modulator <NUM> to cause phase modulator <NUM> to produce intermediate images on amplitude modulator <NUM> (<FIG>). Next, controller <NUM> models phase states of phase modulator <NUM> at a rate of <NUM> fps, based, at least in part, on the phase drive values. As used herein, the term "modeling" includes determining a particular phase state based on an existing model of a response of the pixels of phase modulator <NUM>. Half of the phase states, referred to as steady phase states, are indicative of the spatial phase delay distribution on phase modulator <NUM> while it is driven by the phase drive values at steady state (i.e. after all the pixels of phase modulator <NUM> have fully responded to the applied voltages indicated by the phase drive values). The other half of the phase states, referred to as transitional phase states, are indicative of the spatial phase delay distribution on phase modulator <NUM> at a time when it is transitioning between one set of phase drive values and the next (e.g., between consecutive steady states). Effectively, controller <NUM> converts the sets of phase drive values to steady phase states and temporally up-samples the phase states by generating additional transitional phase states at a rate of <NUM> fps, based on a transition of phase modulator <NUM> between consecutive steady phase states, in order to generate a phase state for each frame of image data (<NUM> fps). Next, controller <NUM> utilizes the phase states to generate lightfield simulations of the intermediate image generated by phase modulator <NUM> and incident on amplitude modulator <NUM> at a rate of <NUM> fps. Because the phase states indicate how phase modulator <NUM> will modulate an incident lightfield, they can be used to reliably estimate the resulting intermediate image (i.e., the lightfield incident on amplitude modulator <NUM>). Finally, controller <NUM> utilizes the lightfield simulations to generate sets of amplitude drive values for driving amplitude modulator <NUM>, at <NUM> fps, in order to generate a final image in accordance with the image data.

<FIG> is a timing diagram illustrating an alternative method for generating images in projection system <NUM>. Controller <NUM> receives image data at a rate of <NUM> fps (the framerate used for nearly all cinematography), uses all the frames of image data to generate phase drive values at an equal rate (<NUM> fps), and provides the phase drive values to phase modulator <NUM>. Controller <NUM> also generates phase states from the phase drive values, but at an increased rate (<NUM> fps in this example). Effectively, controller <NUM> converts the sets of phase drive values to steady phase states at a rate of <NUM> fps and temporally up-samples the phase states by generating additional transitional phase states at a rate of <NUM> fps, in order to generate two phase states for each frame of image data (<NUM> fps). Next, controller <NUM> utilizes the phase states to generate lightfield simulations of the intermediate image generated by phase modulator <NUM> and incident on amplitude modulator <NUM> at a rate of <NUM> fps. Finally, controller <NUM> utilizes the lightfield simulations to generate sets of amplitude drive values for driving amplitude modulator <NUM> at <NUM> fps, in order to generate final images free of artifacts that might otherwise be caused by the transitions of phase modulator <NUM>.

<FIG> is a timing diagram illustrating yet another alternative method for generating images in projection system <NUM>. <FIG> is substantially similar to <FIG>, except that controller <NUM> generates four phase states (at a rate of <NUM> fps) for each frame of image data received. Controller <NUM> also generates lightfield simulations and sets of amplitude drive values at a rate of <NUM> fps, thereby providing increased image quality. An arbitrary number of phase states can be generated for each frame of image data. As both the switching frequency of SLMs and current computing technologies improve, the number of phase states can be increased indefinitely. Additionally, it may be desirable to generate phase states at a higher frequency than amplitude modulator <NUM> is capable of switching at. For example, in a system with an amplitude modulator capable of switching twice as fast as the phase modulator, it might be desirable to generate sets of phase drive values and phase states at a <NUM>:<NUM> ratio and to calculate an average of the transitional phase states when generating the lightfield simulations.

<FIG> is a block diagram showing controller <NUM>, including a data transfer interface <NUM>, non-volatile data storage <NUM>, one or more processing unit(s) <NUM>, and working memory <NUM>. The components of controller <NUM> communicate with one another via a system bus <NUM>, which is interconnected between the components of controller <NUM>. Data transfer interface <NUM> controls the transfer of data, including image data and control instructions, to and from controller <NUM>. Non-volatile data storage <NUM> stores data and code and retains the data and code even when controller <NUM> is powered down. Processing unit(s) <NUM> impart(s) functionality to controller <NUM> by executing code stored in non-volatile data storage <NUM> and/or working memory <NUM>.

Working memory <NUM> provides temporary storage for data and code. Some functionality of controller <NUM> is represented by data and code modules shown within working memory <NUM>. The data and code modules are transferred (in whole or in part) into and out of working memory <NUM> from non-volatile data storage <NUM>, as determined by the execution of code by processing unit(s) <NUM>. The data and code modules can be implemented, for example, with any combination of hardware, software, and/or firmware.

Working memory <NUM> includes a control/coordination module <NUM>, a data buffer <NUM>, a communication module <NUM>, system configuration settings <NUM>, a phase drive module <NUM>, temporal lightfield simulation module <NUM>, and an amplitude drive module <NUM>. Control/coordination module <NUM> is a higher level program that provides overall coordination and control of the other functional aspects of controller <NUM>. Data buffer <NUM> temporarily stores data to be utilized by the other components of controller <NUM>. Communication module <NUM> facilitates communication with external devices in order to send/receive code and/or control instructions. System information module <NUM> includes information about projection system <NUM> (e.g. optical set-up, age of components, technical specifications of components, etc.), which is utilized by the other components of controller <NUM>. Phase drive module <NUM> includes data and algorithms for generating sets of phase drive values from image data. Temporal lightfield simulation module <NUM> includes data and algorithms for modeling the phase states of phase modulator <NUM> and generating lightfield simulations of the intermediate image produced by phase modulator <NUM>, based on the phase states. Amplitude drive module <NUM> includes data and algorithms for generating sets of amplitude drive values from the image data and the lightfield simulations.

<FIG> is a block diagram illustrating example data flow between some of the modules of the controller of <FIG>. In this example embodiment, the modules shown in <FIG> are stored and executed within working memory <NUM> (<FIG>) of controller <NUM>. First, phase drive module <NUM> receives image data from data transfer interface <NUM> (<FIG>). Using one (or more) of a variety of methods and/or algorithms, phase drive module <NUM> generates sets of phase drive values for driving phase modulator <NUM> (<FIG>) based on the image data. The sets of phase drive values are provided to drive phase modulator <NUM>, and also provided to temporal lightfield simulation module <NUM>, to be utilized for generating phase states and simulations of the corresponding lightfields on amplitude modulator <NUM>. Using one (or more) of a variety of methods and/or algorithms, lightfield simulation module <NUM> determines the phase states of phase drive module <NUM>, based on the sets of phase drive values, and generates the corresponding lightfield simulations, based on the phase states. The lightfield simulations are provided directly to amplitude drive module <NUM>. Amplitude drive module <NUM> uses one (or more) methods and/or algorithms to generate sets of amplitude drive values for driving amplitude modulator <NUM> based on the lightfield simulations and the image data. The sets of amplitude drive values are provided to drive amplitude modulator <NUM> via data transfer interface <NUM> (<FIG>).

In the example embodiment, phase drive module <NUM>, temporal lightfield simulation module <NUM>, and amplitude drive module <NUM> each utilize relevant data from system information module <NUM>. For example, temporal lightfield simulation module <NUM> can utilize information about phase modulator <NUM> from system information module <NUM> to model phase transitions of phase modulator <NUM> between consecutive steady phase states. Examples of relevant information about phase-modulator <NUM> include, but are not limited to, the age of phase-modulator <NUM>, physical characteristics of a liquid crystal layer of phase modulator <NUM>, the temperature of phase-modulator <NUM>, voltage / phase delay transition curves, and so on. Phase drive module <NUM> and temporal lightfield simulation module <NUM> can also utilize data from system information module <NUM> describing the characteristics of optics <NUM> and/or light source <NUM>, in order to simulate the lightfield generated by phase modulator <NUM>. Examples of relevant information about optics <NUM> and/or light source <NUM> include, but are not limited to, physical characteristics (e.g., intensity, wavelength(s), etc.) of a lightfield incident on phase modulator <NUM>, a history of the lightfield (e.g., total amount of incident light over lifetime of phase modulator <NUM>), and so on. Lightfield simulation module <NUM> can also use information about phase modulator <NUM> in combination with the phase drive values to generate improved light field simulations. For example, based on particular set of phase drive values, lightfield simulation module can estimate the effect of cross-talk between neighboring pixels of phase modulator <NUM>. Using any of the foregoing information, or combinations thereof, phase drive module <NUM>, temporal lightfield simulation module <NUM>, and amplitude drive module <NUM> are together able to generate beam-steering and amplitude drive values that produce higher quality images via projection system <NUM>.

<FIG> is a block diagram illustrating data flow between some of the modules of controller <NUM>, while operating according to the method described with reference to <FIG> above. Particularly, phase drive module <NUM> receives image data at a rate of <NUM> fps and outputs sets of phase drive values at a rate of <NUM> fps. The sets of phase drive values are received by temporal lightfield simulation module <NUM> and used to generate lightfield simulations at a rate of <NUM> fps (e.g. two lightfield simulations for each set of phase drive values). The lightfield simulations are received by amplitude drive module <NUM> along with the original frames of image data. Amplitude drive module <NUM> utilizes the lightfield simulations and the frames of image data to generate amplitude drive values at a rate of <NUM> fps.

<FIG> is a block diagram illustrating data flow between some of the modules of controller <NUM>, while operating according to the method described with reference to <FIG> above. Particularly, phase drive module <NUM> receives image data at a rate of <NUM> fps and outputs sets of phase drive values at a rate of <NUM> fps. The sets of phase drive values are received by temporal lightfield simulation module <NUM> and used to generate lightfield simulations at a rate of <NUM> fps (e.g. four lightfield simulations for each set of phase drive values). The lightfield simulations are received by amplitude drive module <NUM> along with the original frames of image data. Amplitude drive module <NUM> utilizes the lightfield simulations and the frames of image data to generate amplitude drive values at a rate of <NUM> fps.

<FIG> is a diagram illustrating the generation of drive values, phase states, and lightfield simulations corresponding to a plurality of frames of video data <NUM>(<NUM>-<NUM>). As will become clear, the various drive values, phase states, and lightfield simulations are arranged under associated frames of image data to which they correspond for display purposes. However, <FIG> is not a timing diagram. Therefore, the drive values, phase states, and lightfield simulations are not necessarily generated during the frame time in which they are displayed in <FIG>. Time is, however, a factor used to determine intermediate phase states and, therefore, affects the lightfield simulations and amplitude drive values based on those intermediate phase states. For example, some transitional phase states, which are used to calculate a lightfield simulation for a particular frame of image data, can be determined based in part on one or more subsequently received frames of image data. An example of relative timing for generating these values will be explained below, with reference to <FIG>.

In a first frame column <NUM>(<NUM>), a frame of image data <NUM>(<NUM>) is received and utilized to generate a set of phase drive values <NUM>(<NUM>). Phase drive values <NUM>(<NUM>) are utilized to determine a steady phase state <NUM>(<NUM>). Steady phase state <NUM>(<NUM>) is utilized to generate a lightfield simulation <NUM>(<NUM>), and lightfield simulation <NUM>(<NUM>) is utilized, along with image data <NUM>(<NUM>), to generate a set of amplitude drive values <NUM>(<NUM>).

In a second frame column <NUM>(<NUM>), a second frame of image data <NUM>(<NUM>) is received, but phase drive values are not generated based on second frame of image data <NUM>(<NUM>), because phase modulator <NUM> is incapable of switching at the relatively higher frequency of the image data receipt. Instead, a transitional phase state <NUM>(<NUM>) is determined based on steady phase state <NUM>(<NUM>) and a steady phase state <NUM>(<NUM>) of a third frame column <NUM>(<NUM>). Steady phase state <NUM>(<NUM>) is determined the same way as steady phase state <NUM>(<NUM>) (i.e. steady phase state <NUM>(<NUM>) is generated from image data <NUM>(<NUM>)). Transitional phase state <NUM>(<NUM>) is indicative of the phase state of phase modulator <NUM> during a transition between being driven by phase drive values <NUM>(<NUM>) and phase drive values <NUM>(<NUM>).

All odd frame columns (e.g. frames <NUM>(<NUM>), <NUM>(<NUM>), <NUM>(<NUM>), etc.) are similar, in that they include phase drive values <NUM>. All even frame columns (e.g. frames <NUM>(<NUM>), <NUM>(<NUM>), etc.) are similar, in that they do not include phase drive values <NUM>. Each of phase drive values <NUM> is utilized as an initial drive state (for the subsequent frame) and a final drive state (for the prior frame) for modeling the transition of phase modulator <NUM>. This pattern continues over all frames of image data <NUM>.

<FIG> is a timing diagram showing the relative timing of receiving image data and generating drive values, phase states, and lightfield simulations therefrom. For every other frame of image data, a set of phase drive values (labeled PDV) is generated. For every frame of image data, a phase state (labeled PS) of phase modulator <NUM> is calculated, a lightfield simulation (labeled LFS) is calculated, a set of amplitude drive values (labeled ADV) is calculated, the PDVs are asserted on phase modulator <NUM>, and the ADVs are asserted on amplitude modulator <NUM>. However, because the PDVs corresponding to a particular frame are required to calculate the PS of the prior frame, not all of these steps occur sequentially for a given frame.

<FIG> illustrates relative timing for performing each of the actions necessary for displaying images in projection system <NUM>. Each of a plurality of columns <NUM> (labeled "T1", "T2", etc.) refers to a time period (e.g., a frame time). Each of a plurality of rows <NUM> (labeled "Receive Data", "Generate PDV", etc.) refers to an action executed (by controller <NUM>, in the example embodiment) as part of displaying an image. During a first time period T1, a first frame of image data (Frame <NUM>) is received. Then, during a second time period T2, a second frame of image data is received and PDV1, PS1, LFS1, and ADV1, which all correspond to Frame <NUM>, are generated and/or calculated. During time period T2 (or at least by the beginning of T3), PDV1 is also asserted onto phase modulator <NUM>. Because there is a delay between asserting phase drive values on phase modulator <NUM> and phase modulator <NUM> reaching steady state, ADV1 is not asserted onto amplitude modulator <NUM> until a third time period T3. Instead amplitude modulator <NUM> is driven with a zero-state. Alternatively, a transitional phase state can be calculated, based on an initial state of phase modulator <NUM> and PDV1, with ADV1 being calculated based on the transitional phase state and asserted during second time period T2.

Next, during a fourth time period T4, a fourth frame of image data is received, and PDV3, which corresponds to the image data of Frame <NUM>, and PS2, LFS2, and ADV2, which all correspond to the image data of Frame <NUM>, are generated. Because PS2 is determined by both PDV1 and PDV3, it can be generated after PDV3 is determined. Additionally, PDV3 is asserted on phase modulator <NUM>, initiating a transition (modeled as part of determining PS2) between PS1 and PS3, and ADV2 (which corresponds to PS2 and LFS2) is asserted on amplitude modulator <NUM>.

Then, during a fifth time period T5, a fifth frame of image data is received, PS3, LFS3, and ADV3 are generated, and ADV3 is asserted on amplitude modulator <NUM>. The process of displaying images continues in this manner, with the generation of each phase state (and the corresponding lightfield simulation and set of amplitude drive values) lagging two time periods behind the receipt of the corresponding frame of image data. Additionally, the assertion of each set of amplitude drive values lags two time periods behind the receipt of the corresponding image data and one time period behind the assertion of the corresponding set of phase drive values.

<FIG> is a flow chart summarizing an example method <NUM> for generating drive values, phase states, and lightfield simulations from image data. In the following explanation, the current PDV corresponds to the most recent frame of image data received in step <NUM>, while the prior PS, prior LFS, and prior ADV correspond to a frame of image data received before the most recent frame of image data received in step <NUM>. In a first step <NUM>, a frame of image data is received. Then, in a second step <NUM>, it is determined whether a set of phase drive values was generated for the prior frame. If a set of phase drive values was generated for the prior frame, method <NUM> continues to a third step <NUM>, where a prior phase state, a prior lightfield simulation, and a prior set of amplitude drive values are calculated/determined based on the phase drive values of the prior frame. Then, in a fourth step <NUM>, the prior amplitude drive values are asserted on an amplitude modulator, and, in a fifth step <NUM>, it is determined whether there is any more incoming image data. If there is no more incoming image data, then method <NUM> ends. Otherwise, method <NUM> returns to first step <NUM>, where the next frame of image data is received.

If, in second step <NUM>, it is determined that a set of phase drive values was not generated for the prior frame, method <NUM> proceeds to a sixth step <NUM>, where a set of phase drive values is generated based on the current frame of image data (i.e. the frame received in the most recent iteration of step <NUM>). Next, in a seventh step <NUM>, a prior phase state, a prior lightfield simulation, and a prior set of amplitude drive values are calculated based, at least partially, on the current set of phase drive values. In seventh step <NUM> the calculated prior phase state is a transitional phase state of a phase modulator transitioning from a previous phase state to a phase state corresponding to the phase drive values generated in sixth step <NUM>. Then, in an eighth step <NUM>, the current set of phase drive values is asserted on a phase modulator, and method <NUM> proceeds to fourth step <NUM>.

<FIG> is a diagram illustrating the generation of drive values, phase states, and lightfield simulations corresponding to two frames of video data <NUM>(<NUM>-<NUM>), according to the methods described with reference to <FIG> and <FIG> above. Like <FIG>, <FIG> is not a timing diagram and simply illustrates how/which data is utilized to generate the drive values, phase states, and lightfield simulations utilized to display the images corresponding to the input image data. Time is, however, a factor used to determine the intermediate phase states and, therefore, affects the lightfield simulations and amplitude drive values based on those intermediate phase states.

A sixth frame of image data <NUM>(<NUM>) is utilized to generate a set of phase drive values <NUM>(<NUM>). Phase drive values <NUM>(<NUM>) are utilized to determine a steady phase state <NUM>(<NUM>). Steady phase state <NUM>(<NUM>) is utilized to generate a lightfield simulation <NUM>(<NUM>), and lightfield simulation <NUM>(<NUM>) is utilized, along with image data <NUM>(<NUM>), to generate a set of amplitude drive values <NUM>(<NUM>).

A seventh frame of image data <NUM>(<NUM>) is utilized to generate a set of phase drive values <NUM>(<NUM>). Phase drive values <NUM>(<NUM>) are utilized to determine a steady phase state <NUM>(<NUM>). Steady phase state <NUM>(<NUM>) is utilized to generate a lightfield simulation <NUM>(<NUM>), and lightfield simulation <NUM>(<NUM>) is utilized, along with image data <NUM>(<NUM>), to generate a set of amplitude drive values <NUM>(<NUM>).

Additionally, at least one transitional phase state <NUM>(<NUM>) is generated utilizing phase drive values <NUM>(<NUM>) and phase drive values <NUM>(<NUM>). Additional transitional phase states <NUM>(<NUM>. n) can also be generated. The transitional phase states <NUM>(<NUM>-<NUM>. n) describe the phase states of phase modulator <NUM> at various times between when phase modulator <NUM> is driven with steady phase state <NUM>(<NUM>) and when phase modulator <NUM> is driven with steady phase state <NUM>(<NUM>). Any number of transitional phase states <NUM> can be generated, based on the particular application of projection system <NUM>. Each of transitional phase states <NUM>(<NUM>-<NUM>. n) are utilized to generate corresponding lightfield simulations <NUM>(<NUM>-<NUM>. Finally, each of lightfield simulations <NUM>(<NUM>-<NUM>. n) are utilized, along with image data <NUM>(<NUM>) to generate corresponding amplitude drive values <NUM>(<NUM>-<NUM>.

<FIG> is a timing diagram showing the relative timing of receiving image data and generating drive values, phase states, and lightfield simulations in the case where a single transitional phase state is generated between sequential frames of image data. For every frame of image data, one set of phase drive values (PDVs), two phase states (PSs), two lightfield simulations (LFSs), and two sets of amplitude drive values (ADVs) are calculated. The PDVs are asserted on phase modulator <NUM>, and the ADVs are asserted on amplitude modulator <NUM>. However, because the PDVs corresponding to a particular frame are required to calculate the PSs of the prior frame, not all of these steps occur sequentially for a given frame.

<FIG> illustrates relative timing for performing each of the actions necessary for displaying images in projection system <NUM>. Each of columns <NUM> corresponds to a particular frame time of amplitude modulator <NUM> (<NUM> fps in this example). During a first time period T1, a first frame of image data (Frame <NUM>) is received. Then, during a second time period T2, PDV1 is generated and/or calculated. Optionally, a transitional phase state PS0. <NUM> can be calculated utilizing PDV1 and an initial (prior) state of phase modulator <NUM> (e.g. on, off, etc.). A corresponding LFS0. <NUM> and ADV0. <NUM> can be calculated also. During time period T2 (or at least by the beginning of T3), PDV1 and ADV <NUM> are also asserted onto phase modulator <NUM> and amplitude modulator <NUM>, respectively.

During a third time period T3, a second frame of image data is received. Additionally, PS1. <NUM> (corresponding to PDV1 at steady state), LFS <NUM>, and ADV <NUM> are generated, and ADV1. <NUM> is asserted on amplitude modulator <NUM> during time period T3. Next, during a fourth time period T4, PDV2, which corresponds to the image data of Frame <NUM>, and PS1. <NUM>, LFS1. <NUM>, and ADV1. <NUM>, which all correspond to the transitional state of phase modulator <NUM> between Frame <NUM> and Frame <NUM>, are generated. Because PS1. <NUM> is determined by both PDV1 and PDV2, it can be generated only after PDV2 is determined. Additionally, PDV2 is asserted on phase modulator <NUM>, initiating a transition (modeled as part of determining PS1. <NUM>) between PS1. <NUM> and PS2. <NUM>, and ADV1. <NUM> (which corresponds to PS <NUM> and LFS <NUM>) is asserted on amplitude modulator <NUM>.

Then, during a fifth time period T5, a third frame of image data is received, PS2. <NUM>, LFS2. <NUM>, and ADV2. <NUM> are generated, and ADV2. <NUM> is asserted on amplitude modulator <NUM>. The process of displaying images continues in this manner, with the generation of each steady phase state (and the corresponding lightfield simulation and set of amplitude drive values) lagging two time periods behind the receipt of the corresponding frame of image data. Additionally, the assertion of the amplitude drive values corresponding to a steady state lags one time period behind the assertion of the corresponding set of phase drive values.

<FIG> is a timing diagram showing the relative timing of receiving image data and generating drive values, phase states, and lightfield simulations in the case where three transitional phase states are generated between sequential frames of image data. For every frame of image data, one set of phase drive values, four phase states, four lightfield simulations, and four sets of amplitude drive values are calculated. The PDVs are asserted on phase modulator <NUM>, and the ADVs are asserted on amplitude modulator <NUM>. However, because the PDVs corresponding to a particular frame are required to calculate the PSs of the prior frame, not all of these steps occur sequentially for a given frame.

<FIG> illustrates relative timing for performing each of the actions necessary for displaying images in projection system <NUM>. Each of columns <NUM> corresponds to the framerate of amplitude modulator <NUM> (<NUM> fps in this example). During a first time period T1, a first frame of image data (Frame <NUM>) is received. Then, during a second time period T2, PDV1 is generated and/or calculated. Optionally, a transitional phase state PS0. <NUM> can be calculated utilizing PDV1 and an initial state of phase modulator <NUM> (e.g. on, off, etc.). A corresponding LFS0. <NUM> and ADV0. <NUM> are also calculated. During time period T2 (or at least by the beginning of T3), PDV1 and ADV <NUM> are asserted onto phase modulator <NUM> and amplitude modulator <NUM>, respectively. Next, during a third time period T3, another transitional phase state PS0. <NUM> and corresponding LFS <NUM> and ADV <NUM> are generated, and ADV <NUM> is asserted on amplitude modulator <NUM>. Then, during a fourth time period T4, yet another transitional phase state PS0. <NUM> and corresponding LFS <NUM> and ADV <NUM> are generated, and ADV <NUM> is asserted on amplitude modulator <NUM>.

During a fifth time period T5, a second frame of image data is received. Additionally, PS1. <NUM> (corresponding to PDV1 at steady state), LFS <NUM>, and ADV <NUM> are generated, and ADV1. <NUM> is asserted on amplitude modulator <NUM> during time period T5. Next, during a sixth time period T6, PDV2, which corresponds to the image data of Frame <NUM>, and PS1. <NUM>, LFS1. <NUM>, and ADV1. <NUM>, which all correspond to the transitional state of phase modulator <NUM> between Frame <NUM> and Frame <NUM>, are generated. Because PS1. <NUM> is determined by both PDV1 and PDV2, it can be generated only after PDV2 is determined. Additionally, PDV2 is asserted on phase modulator <NUM>, initiating a transition (modeled as part of determining PS <NUM>) between PS1. <NUM> and PS2. <NUM>, and ADV1. <NUM> (which corresponds to PS <NUM> and LFS <NUM>) is asserted on amplitude modulator <NUM>. Then, during a seventh time period T7, another transitional phase state PS1. <NUM> and corresponding LFS <NUM> and ADV <NUM> are generated, and ADV <NUM> is asserted on amplitude modulator <NUM>. Then, during an eighth time period T8, yet another transitional phase state PS1. <NUM> and corresponding LFS <NUM> and ADV <NUM> are generated, and ADV1. <NUM> is asserted on amplitude modulator <NUM>.

Then, during a ninth time period T9, a third frame of image data is received, PS2. <NUM>, LFS2. <NUM>, and ADV2. <NUM> are generated, and ADV2. <NUM> is asserted on amplitude modulator <NUM>. The process of displaying images continues in this manner, with the generation of each steady phase state (e.g. PS x. <NUM>) (and the corresponding lightfield simulation and set of amplitude drive values) lagging four time periods behind the receipt of the corresponding frame of image data. Additionally, the assertion of the amplitude drive values corresponding to a steady state lags three time periods behind the assertion of the corresponding set of phase drive values.

<FIG> is a flow chart summarizing an example method <NUM> for generating drive values, phase states, and lightfield simulations from image data. In a first step <NUM>, a frame of image data is received. Then, in a second step <NUM>, a current set of phase drive values (PDVs) is generated based on the current frame of image data. Next, in a third step <NUM>, the current PDVs are asserted on a phase modulator. Then, in a fourth step <NUM>, a phase state is generated based on at least one of the current set of PDVs and a prior set of PDVs. Next, in a fifth step <NUM>, a lightfield simulation (LFS) is generated based on the phase state. Then, in a sixth step <NUM>, a set of amplitude drive values (ADVs) is generated based on the LFS. Next, in a seventh step <NUM>, the ADVs are asserted on an amplitude modulator. Then, in an eighth step <NUM>, it is determined whether there are more intermediate phase states to be generated. If there are more intermediate phase states to be generated, method <NUM> returns to step <NUM>. If there are no more intermediate phase states to be generated, method <NUM> continues to a ninth step <NUM>, in which it is determined whether there are more frames of image data to be received. If there are more frames of image data to be received, method <NUM> returns to step <NUM>. Otherwise, method <NUM> terminates.

<FIG> is a block diagram illustrating a transitional phase state of phase modulator <NUM>. Phase modulator <NUM> includes a plurality of pixel groups <NUM> arranged in n rows <NUM> and m columns <NUM>. During a transition between steady phase states, each group of pixels transitions between an initial phase delay (corresponding to the prior set of phase drive values) and a final phase delay (corresponding to the next set of phase drive values) according to a phase equation, φn,m(t). The phase equations indicate how the phase delay of the pixels change over the time period of the transition. Each group of pixels (or each pixel, depending on the group size) has an individual phase equation, which is based on a number of factors, including, but not limited to, spatial cross-talk from neighboring pixels, the driving scheme used to apply the driving voltages (e.g. constant or varying voltage across the liquid crystal cell), the thickness of a liquid crystal layer of phase modulator <NUM>, the temperature of phase modulator <NUM>, properties of the liquid crystal material (e.g., chemical composition, viscosity, orientation, etc.), the age of phase modulator <NUM>, properties of the lightfield incident on phase modulator <NUM> (e.g., wavelengths, power, etc.), and the total amount of light incident on phase modulator <NUM> over its lifetime. Temporal lightfield simulation module <NUM> utilizes data, based on at least a subset of these factors, to use and/or generate the phase equations for each pixel. A user/manufacturer of projection system <NUM> can decide which factors to include in the phase equations to generate equations having a desired accuracy, based on tradeoffs between computational efficiency and image quality.

The phase state of phase modulator <NUM> at any time during a transition can be expressed mathematically by the following matrix: <MAT> Wherein φi,j(t) corresponds to the phase delay of a pixel in row i and column j on a phase modulator having a resolution of n x m pixels. Inputting a particular time to the matrix (i.e. evaluating each phase function at the particular time), provides the phase state of the phase modulator at that time. The phase state can then be used to determine the lightfield generated by the phase modulator at that time.

<FIG> is a graph <NUM> showing an example phase function <NUM>. Phase function <NUM> illustrates a phase delay response of a corresponding pixel to a new, higher driving voltage over time. Prior to an initial time, ti, a pixel corresponding to phase function <NUM> is driven with an initial voltage, corresponding to an initial (steady state) phase delay, φi. At time ti a new voltage is asserted onto the pixel, causing the liquid crystal layer of the pixel to be driven to a state corresponding to the new voltage. As a result, the phase delay imparted by the pixel also changes. At a final time, tf, the pixel has reached a new steady state, imparting a final phase delay, φf. In the example embodiment, phase function <NUM> has the form: <MAT> where φ(t) is the phase delay of the pixel at a time t, and A, B, and C are constants determined/used by temporal lightfield simulation module <NUM>, based on calculated factors that affect the transition. The constants A, B, and C can be determined, for example, empirically, by testing the modulator pixels under various conditions to determine the effects of each of the calculated factors on a transition. Lightfield simulation module <NUM> can then determine the constants based on the characteristics of the modulator and the effects on the transition resulting therefrom.

In the example embodiment, phase function <NUM> can be used to determine a transitional phase delay for generating a transitional phase state. An intermediate time (e.g. halfway between ti and tf) is selected for evaluating phase function <NUM>. The intermediate time is selected to coincide with the timing of amplitude modulator <NUM>. Then, the intermediate time is input to phase function <NUM>, which outputs a phase delay. The output phase delay is the phase delay of the pixel at the intermediate time. Indeed, phase function <NUM> can provide the phase delay of the pixel at any time between the initial and final times and can be used to determine an infinite number of phase delays at an infinite number of times. As an alternative, phase function <NUM> can be averaged over a time period by evaluating the following integral: <MAT> where t<NUM> and t<NUM> are the lower and upper bounds, respectively, of the time period.

<FIG> is a graph <NUM> showing another example phase function <NUM>. Phase function <NUM> illustrates a phase delay response of a corresponding pixel to a new, lower driving voltage over time. Prior to an initial time, ti, a pixel corresponding to phase function <NUM> is driven with an initial voltage, corresponding to an initial (steady state) phase delay, φi. At time ti a new voltage is asserted onto the pixel, causing the liquid crystal layer of the pixel to relax to a state corresponding to the new voltage. As a result, the phase delay imparted by the pixel also changes. At a final time, tf, the pixel has reached a new steady state, imparting a final phase delay, φf. In the example embodiment, phase function <NUM> has the form: <MAT> where φ(t) is the phase delay of the pixel at a time t, and A, B, and C are constants determined/used by temporal lightfield simulation module <NUM>, based on calculated factors that affect the transition.

During a transition between sets of phase drive values, the phase delay of any pixel of phase modulator <NUM> can be described by a phase function of the form of phase function <NUM> or phase function <NUM>, based on whether the pixel is being driven with a voltage that is higher or lower, respectively, than the prior driving voltage. Temporal lightfield simulation module <NUM> determines which form to utilize and the values of the required constants to determine the particular phase function for each pixel. Then, temporal lightfield simulation module <NUM> can calculate the phase delay for each pixel at any time (or the average phase delay for each pixel over any time period) during the transition to determine a phase state of phase modulator <NUM> at any time during the transition.

<FIG> is a flow chart summarizing an example method <NUM> (not being part of the present invention) for generating a lightfield simulation. In a first step <NUM>, a beam-steering SLM is driven with a first set of drive values to place the beam-steering SLM in a first state at a first time. Next, in a second step <NUM>, the beam-steering SLM is driven with a second set of drive values that cause the beam-steering SLM to transition from the first state to a second state of the beam-steering SLM at a second time. The second state of the beam-steering SLM is associated with the second set of drive values. Then, in a third step <NUM>, the transition of the beam-steering SLM from the first state to the second state is modeled. Next, in a fourth step <NUM>, a third state of the beam-steering SLM at a third time (occurring between the first time and the second time) is determined based at least in part on the model of the transition of the beam-steering SLM. Then, in a fifth step <NUM>, a lightfield simulation of a lightfield generated by the beam-steering SLM at the third time is generated based at least in part on the third state of the beam-steering SLM.

<FIG> is a flow chart summarizing a method <NUM> for generating images in a multi-modulation projection system. In a first step <NUM>, a first frame of image data is received. Then, in a second step <NUM>, a second frame of image data is received. Next, in a third step <NUM>, a first set of phase drive values for driving a phase-modulating spatial light modulator (SLM) are generated. The first set of phase drive values are based at least in part on the first frame of image data. Then, in a fourth step <NUM>, the phase-modulating SLM is driven with the first set of phase drive values during a first time period. Next, in a fifth step <NUM>, a third frame of image data is received. Then, in a sixth step <NUM>, a second set of phase drive values for driving the phase-modulating SLM are generated. The second set of phase drive values are based at least in part on the third frame of image data. Next, in a seventh step <NUM>, the phase-modulating SLM is driven with the second set of phase drive values during a second time period. Then, in an eighth step <NUM>, transitional states of the phase-modulating SLM are modelled during the second time period. The transitional states are based at least in part on the first set of phase drive values and the second set of phase drive values. Next, in a ninth step <NUM>, a set of lightfield simulations is generated. The set of lightfield simulations includes a first subset of the set of lightfield simulations corresponding to the first set of phase drive values, a second subset of the set of lightfield simulations corresponding to the second set of phase drive values, and a third subset of the set of lightfield simulations corresponding to one or more of the transitional states of the phase-modulating SLM. Then, in a tenth step <NUM>, sets of amplitude drive values are generated for driving the amplitude-modulating SLM. Each of the sets of amplitude drive values is based on and/or corresponds to one of the sets of lightfield simulations. Then, in an eleventh step <NUM>, the amplitude-modulating SLM is driven with the sets of amplitude drive values.

<FIG> is a flowchart summarizing an example method <NUM> (not being part of the present invention) of generating images in a multi-modulation projection system. In a first step <NUM>, (n) frames of image data are received. The frames of image data include a first group of frames interlaced with a second group of frames. Then, in a second step <NUM>, (m) frames of phase drive values are generated. Each frame of phase drive values is based, at least in part, on an associated frame of image data of the first group of frames and causes a phase-modulating spatial light modulator (SLM) to be in an associated phase state to generate a lightfield corresponding to the associated frame of image data. Next, in a third step <NUM>, (p) transitional phase states of the phase-modulating SLM are determined. Each of the transitional phase states are indicative of a lightfield generated by the phase-modulating SLM during a transition period and corresponds to one of the frames of image data of the second group of frames. Then, in a fourth step <NUM>, a set of lightfield simulations is generated based on the phase drive values and the transitional phase states. Each of the lightfield simulations is indicative of a lightfield generated by the phase-modulating SLM and incident on an amplitude-modulating SLM. Then, in a fifth step <NUM>, a set of amplitude drive values is generated based on the set of lightfield simulations and the frames of image data. Each of the sets of amplitude drive values causes the amplitude SLM to spatially modulate a corresponding one of the lightfields incident on the amplitude SLM to generate an image corresponding to the image data.

The description of particular embodiments of the present invention is now complete. Many of the described features may be substituted, altered or omitted without departing from the scope of the invention as defined in the appended claims.

Claim 1:
A method for generating images, said method implemented in a controller for controlling a dual-modulation projection system with a phase-modulating spatial light modulator (SLM) and an amplitude-modulating SLM, said method comprising:
receiving (<NUM>) a first frame of image data;
generating (<NUM>) a first set of phase drive values for driving the phase-modulating SLM, said first set of phase drive values being based at least in part on said first frame of image data;
driving (<NUM>) said phase-modulating SLM with said first set of phase drive values during a first time period;
receiving (<NUM>) a second frame of image data;
generating (<NUM>) a second set of phase drive values for driving said phase-modulating SLM, said second set of phase drive values being based at least in part on said second frame of image data;
driving (<NUM>) said phase-modulating SLM with said second set of phase drive values during a second time period;
modeling (<NUM>) transitional states of said phase-modulating SLM during said second time period, said transitional states being based at least in part on said first set of phase drive values and said second set of phase drive values;
generating (<NUM>) a set of lightfield simulations of lightfields generated by said phase-modulating SLM and incident on the amplitude-modulating SLM, a first subset of said set of lightfield simulations corresponding to said first set of phase drive values, a second subset of said set of lightfield simulations corresponding to said second set of phase drive values, and a third subset of said set of lightfield simulations corresponding to one or more of said transitional states of said phase-modulating SLM;
generating (<NUM>) sets of amplitude drive values for driving said amplitude-modulating SLM, each of said sets of amplitude drive values corresponding to one of the subsets of said set of lightfield simulations; and
driving (<NUM>) said amplitude-modulating SLM with said sets of amplitude drive values.