Abstract:
A multifocal display for rendering a 3D scene as a series of 2D images. In one aspect, the multifocal display includes a display, an optical imaging system, a refractive focus actuator and a controller. The display renders the 2D images. The optical imaging system is image-side telecentric and creates an image of the display. The refractive focus actuator is positioned at the pupil of the optical imaging system. Thus, adjusting the refractive focus actuator alters a location of the image of the display but does not significantly alter a size of the image. The controller coordinates adjustment of the refractive focus actuator with rendering of the 2D images on the display. The waveform driving the focus actuator is preferably designed to reduce ringing and jitter effects.

Description:
CROSS-REFERENCE TO RELATED APPLICATION(S) 
     This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application Ser. No. 62/084,315, “Design, Assembly and Operation of a Liquid Lens-Based Multifocal Display for Augmented Reality,” filed Nov. 25, 2014. The subject matter of all of the foregoing is incorporated herein by reference in its entirety. 
    
    
     BACKGROUND 
     1. Field of the Invention 
     This disclosure relates generally to multifocal displays. 
     2. Description of Related Art 
     Three-dimensional (3D) or stereoscopic displays are important for many applications, such as medical imaging, entertainment, virtual prototyping, and scientific visualization. Stereoscopic displays conventionally suffer from mismatched convergence and accommodation cues. This conflict hinders visual performance and causes visual fatigue. 
     Multifocal displays are one approach that has attempted to address the convergence-accommodation conflict. Multifocal displays (MFDs) typically use rapid temporal and focal modulation of a series of 2-dimensional (2D) images to render 3D scenes. This series of images is typically focused at parallel planes positioned at different, discrete distances from the viewer. The number of focal planes directly affects the viewers&#39; eye accommodation and 3D perception quality of a displayed scene. However, multifocal displays require a high-speed focus actuator. In addition, the shifting location of the focal plane can result in changes in magnification. Although this magnification dependence can be computationally corrected by adjusting the input images, this adds to the system complexity. 
     Therefore, there is a need for better multifocal displays. 
     SUMMARY 
     The present disclosure overcomes the limitations of the prior art by providing a multifocal display for rendering a 3D scene as a series of 2D images. In one aspect, the multifocal display includes a display, an optical imaging system, a refractive focus actuator and a controller. The display renders the 2D images. The optical imaging system is image-side telecentric and creates an image of the display. The focus actuator is positioned at the pupil of the optical imaging system. Thus, the focus actuator alters a location of the image of the display but does not significantly alter a size of the image. The controller coordinates adjustment of the focus actuator with rendering of the 2D images on the display. In some embodiments, the refractive focus actuator is a deformable liquid lens and the waveform driving the liquid lens is designed to reduce ringing in the liquid lens. 
     Other aspects include components, devices, systems, improvements, methods, processes, applications, computer readable mediums, and other technologies related to any of the above. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Embodiments of the disclosure have other advantages and features which will be more readily apparent from the following detailed description and the appended claims, when taken in conjunction with the accompanying drawings, in which: 
         FIG. 1 a    is a schematic of a multifocal display that implements augmented reality for a single eye. 
         FIG. 1 b    is the optical prescription for the multi-focal display of  FIG. 1   a.    
         FIG. 2 a    shows a Zemax model for a deformable liquid lens.  FIG. 2 b    is an MTF plot of the deformable liquid lens at one focal distance. 
         FIG. 3 a    is a schematic of the optical imaging system within the multi-focal display. 
         FIG. 3 b    illustrates change of image position when the liquid lens is deformed. 
         FIG. 4  illustrates increase of eye relief resulting from use of a negative field lens. 
         FIG. 5  is a timing diagram of a pyramid-shaped waveform used to drive the deformable liquid lens. 
         FIG. 6  shows a filtered version of the drive waveform of  FIG. 5 . 
     
    
    
     The figures depict various embodiments for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein. 
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     The figures and the following description relate to preferred embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed. 
       FIG. 1 a    is a schematic of a multifocal display  100  that implements augmented reality for a single eye. It uses a display  110 , an optical imaging system  120 , a refractive focus actuator  130  located in the pupil plane of the optical imaging system such that the display is doubly telecentric, a controller  140 , a negative field lens  160 , an eyepiece  150 , and a beamsplitter  170 . 
     In the example described below, the display  110  is a high-speed DLP (digital light processing) projector, although other displays such as OLEDs and LCDs could also be used. The optical imaging system  120 , which is the primary relay system of the multifocal display, is implemented by high-quality camera lenses in a 4-f configuration, and the refractive focus actuator is a deformable liquid lens. The display  110  in combination with the liquid lens  130  can project six focal planes at distances between 0 and 5 diopters, each at a speed of 60 frames per second (fps) or an aggregate speed of 360 fps for the display  110 . The eyepiece  150  and beamsplitter  170  allow the real world to be viewed alongside virtual content with a full field-of-view of 31 degrees in this example. The DLP  110  and liquid lens  130  are synchronized with signals output by a controller  140 , which in this case is a high-speed analog I/O device. The operating principles of this multifocal display are described in the following sections. 
     The following is a list of preferred performance for a multifocal display:
         No perceived flicker (2D image display speed preferably at least 60 fps for each focal plane)   Continuous perceived depth (number of focal planes N z  preferably at least 6)   No focal jittering (reduce settling time of the focus actuator)   Increase field-of-view for average pupil size of the eye   Maintain magnification with accommodation for both real and virtual content   Match spatial resolution of the eye (angular resolution with pixel spacing of &lt;1 arcmin)   Match virtual content f/# to the average eye f/#   Increase dynamic range and color accuracy   Allow for sufficient eye relief (&gt;=12 mm) and exit pupil diameter (&gt;=4 mm) for standard use       

     Optical Design 
       FIG. 1 b    is the optical prescription for the multi-focal display of  FIG. 1 a   . Consider each of the components in turn, beginning with a human eye. In this example, we use a model of the human eye, which accounts for curvature, thickness, diameter, and index of refraction for the cornea, aqueous humor, iris, lens, vitreous body, and retina. The model accounts for optical performance of the eye focusing anywhere between the near point to infinity. Intermediary accommodation models are desirable to determine the perceptual accuracy of the focus cues produced by the multifocal display. 
     The deformable liquid lens  130  is the Optotune EL-6-18.  FIG. 2 a    shows the optical layout of the liquid lens  130 , and  FIG. 2 b    shows a modulation transfer function (MTF) for this focus actuator. Placing the liquid lens in a pupil plane results in several advantages. First, this configuration results in the maximum focal sweep range. Second, the lens can be placed at the aperture stop of the system, resulting in image-side telecentricity for the optical imaging system  120 . The former is advantageous for modulation speed and settling time; the latter for constant magnification image display. 
       FIG. 3 a    is a schematic of the optical imaging system within the multi-focal display, and  FIG. 3 b    illustrates change of image position when the liquid lens is deformed. The optical imaging system  120  of the multifocal display uses two 35 mm camera lenses configured in a 4-f imaging system. This placement provides the double-telecentric design described in more detail below. The liquid lens is placed between the two camera lenses, such that focus modulation causes the position of the primary image to change, with a minimum change in magnification, as shown in  FIG. 3   b.    
     Returning to  FIG. 1 , the eyepiece  150  impacts the field of view and quality of the image on the retina, the size of the exit pupil, and the depth of field. Ideally, the exit pupil spans the eye&#39;s entire iris and remains fixed axially regardless of the additional power imparted by the liquid lens. In the case of an image-side telecentric optical relay with an eyepiece of focal length ƒ e , the size of the exit pupil is given by 
     
       
         
           
             
               
                 
                   
                     D 
                     XP 
                   
                   = 
                   
                     
                       
                         n 
                         e 
                       
                       ⁢ 
                       
                         f 
                         e 
                       
                       ⁢ 
                       
                         D 
                         stop 
                       
                     
                     
                       
                         
                           n 
                           o 
                         
                         ⁢ 
                         
                           f 
                           o 
                         
                       
                       ⁢ 
                       
                           
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     where n, ƒ, and D stop  respectively denote the index of refraction, focal length, and aperture stop diameter and the subscripts e and o respectively denote eyepiece and objective parameters. Since the exit pupil is located within the eye&#39;s anterior chamber, n e  is given by the refractive index of the aqueous fluid, which is about 1.34. 
     When using a single-lens eyepiece  150  with an image-side telecentric relay  120  from the DLP to the primary image plane, there exists a tradeoff between exit pupil size, focal modulation sensitivity, and image size. In the example multifocal display shown in  FIG. 1 , we use a 25 mm eyepiece  150 , 35 mm objective lenses for the optical imaging system  120 , and the liquid lens  130  has an aperture diameter of 6 mm. This yields an exit pupil of 4.3 mm, which matches the eye well for indoor lighting. 
     A negative field lens  160 , with f=−75 mm, is located between the tunable lens  130  and the eyepiece  150 . This field lens  160  is used to enlarge the eye relief, which is defined as the working distance between the display hardware and the eye.  FIG. 4  illustrates increase of eye relief resulting from use of a negative field lens. The top optical train in  FIG. 4  shows the design without the field lens  160 , and the bottom optical train in  FIG. 4  shows the design with the field lens  160 . An enlarged eye relief improves comfort and is beneficial for viewers wearing prescription glasses. Using a 12 mm eye relief, we obtain a 31-degree full field-of-view. 
     A non-polarizing plate beamsplitter  170  is used to combine real and virtual content. The beamsplitter  170  is 50% reflective and 50% transmissive. The proportions can be optimized based on the light throughput of the multifocal display. 
     A TI LightCrafter4500 DLP projector is used as the high-speed image display  110 . It can project 6-bit grayscale images at a frame rate of 400 fps, with a minimum exposure period of 2.5 ms. 
     In addition to the liquid lens  130 , the system  100  may include other active elements such as an adjustable aperture. An electronically adjustable aperture can change the f/# for a given focal plane, which affects the depth-of-field (focus cue strength) for the image content. Focus control and f/# control can both be used to optimize scene rendering with a given number of focal planes. Other adjustable elements could include fast modulated LEDs or LC shutters for illumination control, which can be used to adjust the brightness of each focal plane. 
     Controller Design 
     A frame rate of 360 fps is used to display six focal planes at 60 fps each, as described above. The frame rate applies to the projector display  110 , as well as the liquid lens  130 , which is modulated with an analog voltage signal. This waveform selection is influenced by the response time τ r  of the lens  130 , which in this example is on the order of 1-3 ms (depending on the dioptric input step size). Ideally, the lens should be stationary for the duration of the exposure period at every focal plane. In practice, the liquid lens settling time is on the order of 5 times the duration of the response time. Thus, the ideal waveform minimizes the “activation time” (sum of response time plus settling time), which is approximately 6τ r . The two design parameters responsible for minimizing the activation time are the waveform itself and any filtering applied to the waveform. 
     If the focal planes are numbered 1-6 in order of their location (either progressing from farthest away from the viewer to closest, or progressing from closest to farthest away), one possible drive signal realizes the focal planes in order from focal plane 1 to focal plane 6 and then resets to focal plane 1 (i.e., a staircase waveform). However, the large drop at the end of the staircase when returning from focal plane 6 to focal plane 1 can cause overshoot and increase the activation time. This can cause an incorrect focus and a blurred image for focal plane 1 (or focal plane 6 for a descending staircase). The artifacts are caused by the drive mechanism of the liquid lens and become more pronounced with increased dioptric spacing between focal planes. 
       FIG. 5  is a timing diagram of a pyramid-shaped waveform used to drive the deformable liquid lens. This waveform steps through the focal planes in the following order: 1, 3, 5, 6, 4, 2. In other words, the odd focal planes are realized in a temporal order that is monotonically ascending (i.e., focal planes 1, 3, 5) and then the even focal planes are realized in a temporal order that is monotonically descending (i.e., focal planes 6, 4, 2). Of course, the order can be reversed: descending for odd focal planes and ascending for even focal planes. By distributing the voltage change throughout the waveform, the effective focal range swept by the liquid lens can be increased, while maintaining an accurate depth position at each focal plane. Other orders of focal planes that reduce lens ringing or increase image quality can be used. 
     Even with a pyramid-shaped waveform, the settling time may still result in an unstable image or “focal jitter” for all planes. Jitter can be caused by underdamping of the liquid lens. A smoothed waveform can reduce the jitter. However, too much smoothing results in overdamping, which also increases the activation time of the liquid lens. Both of these cases result in perceptual artifacts of the virtual content. Ideally, the driving waveform critically damps the lens activation time. 
     In one approach, a method for artifact reduction in a liquid lens-based multifocal display is developed by deriving a model for image formation considering the step response of a liquid lens. Let the multifocal display project an image sequence ƒ(x,y,t). Let us also divide the time into subintervals T k : (t k , t k +1), where k=1, . . . , N z  and t 1 =0. Within each interval T k , we assume the target image data to be constant in time. That is,
 
ƒ( x,y,t )≡ƒ k ( x,y ),
 
 t   k   ≦t&lt;t   k+1 ,
 
 k= 1, . . . , N   z ,  (2)
 
We also assume that N z  is equal to the number of focal planes available. Namely the 3D scene image data is divided into N z  planar images and displayed on N z  depth planes. In this case, to get projection on N z  depth planes, we drive the liquid lens by a voltage waveform that is a piecewise constant function:
 
                       v   ⁡     (   t   )       =       ∑     k   =   1       N   2       ⁢       Δ   k     ⁢     u   ⁡     (     t   -     t   k       )             ,           (   3   )               
where u(t) is a unit step function and Δ k =v(t k )−v(t k −1) is a voltage change step at time t k  (either a positive or negative voltage change). Given the impulse response of the liquid lens, h(t), and the driving voltage waveform v(t), the depth z at which the liquid lens forms an image can be modeled as:
 
 z ( t )=α( v*h )( t )=α∫ −∞   ∞   v ( t−t′ ) h ( t ′) dt′=αW ( t ),  (4)
 
where * denotes the convolution and α is a constant that translates the voltage values to physical depths. To simplify notation, we have introduced W(t)=(v*h)(t). The lens response h(t) might exhibit ringing due to the settling time of the lens.
 
     Given the waveform W(t), and hence the depth z(t)=αW(t) at which the stimulus ƒ is displayed, the image perceived by the eye accommodating to a distance z e  of a virtual multifocal display volume is given by
 
 g ( x′,y′,z   e )=∫ x ∫ y ∫ t ƒ( x,y,t ) p ( x−x′,y−y′,z   e   −αW ( t ) dxdydt,   (5)
 
where p represents the system&#39;s depth-dependent point spread function (PSF). This PSF includes the characteristics of the optical display system and the response of the human eye. We assume an aberration-free PSF p. That means that for each accommodation distance z e , i.e. the eye focusing at distance z e , the PSF is the same. However, we still consider the PSF of the system at a distance z(t) that is different from the accommodation of the eye z e , i.e., the defocused PSF p(x, y, z e −z(t))=p(x, y, z e −αW(t)).
 
     If we now split the integration over time into N z  integrations over time interval T k , we obtain: 
                           g   ⁡     (       x   ′     ,     y   ′     ,     z   e       )       =       ⁢       ∫   x     ⁢       ∫   y     ⁢       ∑     k   =   1       N   2       ⁢       ∫     t   k       t     k   +   1         ⁢       f   ⁡     (     x   ,   y   ,   t     )       ⁢     p   (       x   -     x   ′       ,     y   -     y   ′       ,       z   e     -                                       ⁢     α   ⁢           ⁢   W   ⁢     (   t   )       )     ⁢   d   ⁢           ⁢   x   ⁢           ⁢   d   ⁢           ⁢   y   ⁢           ⁢   d   ⁢           ⁢   t     ,               =       ⁢       ∑     k   =   1       N   z       ⁢       ∫   x     ⁢       ∫   y     ⁢       ∫     t   k       t     k   +   1         ⁢         f   k     ⁡     (     x   ,   y     )       ⁢     p   (       x   -     x   ′       ,     y   -     y   ′       ,       z   e     -                                       ⁢     α   ⁢           ⁢     W   ⁡     (   t   )         )     ⁢   d   ⁢           ⁢   x   ⁢           ⁢   d   ⁢           ⁢   y   ⁢           ⁢   d   ⁢           ⁢   t     ,                 (     6   ⁢   a     )                 where   ⁢           ⁢   we   ⁢           ⁢   have   ⁢           ⁢   used   ⁢           ⁢     Eq   .           ⁢   2.     ⁢           ⁢   By   ⁢           ⁢   simple   ⁢           ⁢   regrouping     ,     we   ⁢           ⁢   obtain   ⁢     :                                       g   ⁡     (       x   ′     ,     y   ′     ,     z   e       )       =       ⁢       ∑     k   =   1       N   z       ⁢       ∫   x     ⁢       ∫   y     ⁢         f   k     ⁡     (     x   ,   y     )       [       ∫     t   k       t     k   +   1         ⁢     p   (       x   -     x   ′       ,     y   -     y   ′       ,       z   e     -                                           ⁢     α   ⁢           ⁢     W   ⁡     (   t   )         )     ⁢   d   ⁢           ⁢   t     ]     ⁢   d   ⁢           ⁢   x   ⁢           ⁢   d   ⁢           ⁢   y     ,                 =       ⁢       ∑     k   =   1       N   z       ⁢       ∫   x     ⁢       ∫   y     ⁢         f   k     ⁡     (     x   ,   y     )       ⁢       p   k     ⁡     (       x   -     x   ′       ,     y   -     y   ′       ,     z   e       )       ⁢   d   ⁢           ⁢   x   ⁢           ⁢   d   ⁢           ⁢   y             ,                 (     6   ⁢   b     )               
where p k  is the defocused PSF integrated over the interval T k , which also accounts for the ringing of the liquid lens. We can see that this integrated PSF depends on the waveform driving the lens, i.e., depends on W(t). When the voltage waveform v(t) is simply piecewise continuous and contains sharp discontinuities, as in  FIG. 5 , the lens response will contain ringing due to the Gibbs effect. In order to reduce this effect and obtain focused images g(x′,y′,z e ) on the retina, we modify the driving voltage waveform v(t) by filtering it with another waveform γ(t). By convolving v(t) with γ(t), we can smoothen v(t) to avoid Gibbs effect.  FIG. 6  shows a filtered version of the drive waveform of  FIG. 5 .
 
     Mathematically, the modified lens response becomes: 
                           z   ⁡     (   t   )       =       ⁢       α   ⁡     (       (     v   *   γ     )     *   h     )       ⁢     (   t   )                   =       ⁢       α   ⁡     (     v   *     (     γ   *   h     )       )       ⁢     (   t   )                   =       ⁢       α   ⁡     (     v   *     (     h   *   γ     )       )       ⁢     (   t   )                   =       ⁢     α   ⁡     (       (     v   *   h     )     *   γ     )                   =       ⁢       α   ⁡     (     W   *   γ     )       ⁢       (   t   )     .                     (   7   )               
The equalities above are due to commutativity and associativity of convolution. With this new filtering, the image captured on the retina of an accommodated eye becomes:
 
     
       
         
           
             
               
                 
                   
                     
                       g 
                       γ 
                     
                     ⁡ 
                     
                       ( 
                       
                         
                           x 
                           ′ 
                         
                         , 
                         
                           y 
                           ′ 
                         
                         , 
                         
                           z 
                           e 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       ∑ 
                       
                         k 
                         - 
                         1 
                       
                       
                         N 
                         z 
                       
                     
                     ⁢ 
                     
                       
                         ∫ 
                         x 
                       
                       ⁢ 
                       
                         
                           ∫ 
                           y 
                         
                         ⁢ 
                         
                           
                             
                               
                                 f 
                                 k 
                               
                               ⁡ 
                               
                                 ( 
                                 
                                   x 
                                   , 
                                   y 
                                 
                                 ) 
                               
                             
                             ⁡ 
                             
                               [ 
                               
                                 
                                   ∫ 
                                   
                                     t 
                                     k 
                                   
                                   
                                     t 
                                     
                                       k 
                                       + 
                                       1 
                                     
                                   
                                 
                                 ⁢ 
                                 
                                   
                                     p 
                                     ⁡ 
                                     
                                       ( 
                                       
                                         
                                           x 
                                           - 
                                           
                                             x 
                                             ′ 
                                           
                                         
                                         , 
                                         
                                           y 
                                           - 
                                           
                                             y 
                                             ′ 
                                           
                                         
                                         , 
                                         
                                           
                                             z 
                                             e 
                                           
                                           - 
                                           
                                             
                                               α 
                                               ⁡ 
                                               
                                                 ( 
                                                 
                                                   W 
                                                   * 
                                                   γ 
                                                 
                                                 ) 
                                               
                                             
                                             ⁢ 
                                             
                                               ( 
                                               t 
                                               ) 
                                             
                                           
                                         
                                       
                                       ) 
                                     
                                   
                                   ⁢ 
                                   d 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   t 
                                 
                               
                               ] 
                             
                           
                           ⁢ 
                           d 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           x 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           d 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             y 
                             . 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
           
         
       
     
     Finally, we would like to maximize the sharpness of the retinal images when the eye accommodates to each designed depth plane of the multifocal display. If we denote the distances of these depth planes as z i , i=1, . . . N z , a quality metric Q can be defined as: 
                     Q   ⁡     (   γ   )       =       ∑     i   =   1       N   z       ⁢       Ω   ⁡     [       g   γ     ⁡     (       x   ′     ,     y   ′     ,     z   i       )       ]       .               (   9   )               
where Ω can be any image quality metric, such as sharpness, for example MTF20, MTF30, or some variation of a contrast, spatial frequency content, or entropy-based image sharpness metric. Therefore, finding the optimal filter γ can be posed as the following optimization problem:
 
     
       
         
           
             
               
                 
                   
                     OPT 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     1 
                     ⁢ 
                     
                       : 
                     
                     ⁢ 
                     
                       γ 
                       ^ 
                     
                   
                   = 
                   
                     
                       arg 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           max 
                           γ 
                         
                         ⁢ 
                         
                           Q 
                           ⁡ 
                           
                             ( 
                             γ 
                             ) 
                           
                         
                       
                     
                     = 
                     
                       arg 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           max 
                           γ 
                         
                         ⁢ 
                         
                           
                             ∑ 
                             
                               i 
                               = 
                               1 
                             
                             
                               N 
                               z 
                             
                           
                           ⁢ 
                           
                             
                               Ω 
                               ⁡ 
                               
                                 [ 
                                 
                                   
                                     g 
                                     γ 
                                   
                                   ⁡ 
                                   
                                     ( 
                                     
                                       
                                         x 
                                         ′ 
                                       
                                       , 
                                       
                                         y 
                                         ′ 
                                       
                                       , 
                                       
                                         z 
                                         i 
                                       
                                     
                                     ) 
                                   
                                 
                                 ] 
                               
                             
                             . 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   10 
                   ) 
                 
               
             
           
         
       
     
     Alternatively, one can use a different metric for optimizing γ, such as projected depth plane error E(γ)=∥z−{tilde over (z)}∥ 2   2  where z={z k } k=1   N     z    is a vector containing the design focal plane distances and {tilde over (z)}={z k } k=1   N     z    is a vector containing the focal plane distances achieved by the system. In this case, the optimization problem becomes: 
     
       
         
           
             
               
                 
                   
                     OPT 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     2 
                     ⁢ 
                     
                       : 
                     
                     ⁢ 
                     
                       γ 
                       ^ 
                     
                   
                   = 
                   
                     
                       arg 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           min 
                           γ 
                         
                         ⁢ 
                         
                           E 
                           ⁡ 
                           
                             ( 
                             γ 
                             ) 
                           
                         
                       
                     
                     = 
                     
                       arg 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           min 
                           γ 
                         
                         ⁢ 
                         
                           
                             
                                
                               
                                 z 
                                 - 
                                 
                                   z 
                                   ~ 
                                 
                               
                                
                             
                             2 
                             2 
                           
                           . 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   11 
                   ) 
                 
               
             
           
         
       
     
     Finally, one can use a combination of both metrics: 
                     OPT   ⁢           ⁢   3   ⁢     :     ⁢     γ   ^       =         arg   ⁢           ⁢       min   γ     ⁢     E   ⁡     (   γ   )           -     λ   ⁢           ⁢     Q   ⁡     (   γ   )           =       arg   ⁢           ⁢       min   γ     ⁢            z   -     z   ~            2   2         -     λ   ⁢       ∑     i   =   1       N   z       ⁢       Ω   ⁡     [       g   γ     ⁡     (       x   ′     ,     y   ′     ,     z   i       )       ]       .                     (   12   )               
where λ is a regularization parameter that weighs the importance of maximizing image sharpness (the second term) versus projecting at the correct depths (the first term). Note that sharpness may be reduced for two reasons. First, the filter attenuates high frequencies too much, resulting in over-damping, and the liquid lens sweeps through the focal plane, causing a hazy appearance of focal sweep data. Second, the filter does not attenuate high frequencies enough, resulting in underdamping and strong ringing and an integration of the projected depth slices in- and out of focus.
 
     Since Eq. 12 has a large solution space, we narrow it by considering only Gaussian filters of the form 
                         γ   σ     ⁡     (   t   )       =     e     -       π   ⁢           ⁢     ⅈ   2         σ     2   ⁢                       ,           (   13   )               
where σ is the filter&#39;s standard deviation. The optimization problem from Eq. 12 reduces to
 
                       σ   ^     =       arg   ⁢           ⁢       min   γ     ⁢            z   -     z   ~            2   2         -     λ   ⁢       ∑     i   =   1       N   s       ⁢     Ω   ⁡     [       g     γ   ⁢           ⁢   σ       ⁡     (       x   ′     ,     y   ′     ,     z   i       )       ]               ,           (   14   )               
Possible fluctuations due to the projection of individual bit planes were ignored. Projection at the granularity of bit planes can be included in the model by adding bit plane dependency into the function ƒ and the optical response function p.
 
     In one scenario, the exact waveform W(t) may not be known, because the temporal impulse response of the lens h(t) may be unknown. Estimating the true temporal impulse response would involve synchronization with a fast photodiode or high-speed camera and measuring the response time directly for all voltage changes. Given such estimation or otherwise knowledge of W, the solution space of potential waveform filters can be enlarged by including other filters in addition to Gaussian filters, e.g. FIR filters by optimizing over their filter coefficients and their filter taps. 
     In the case that W is not known, g(x′,y′,z i ) can still be measured by simulating the observation of a human eye with a camera. By searching through the solution space of σ, the {circumflex over (σ)} that provides the best sharpness measured by the sharpness metric Ω can be determined. The evaluation metric Ω could also measure temporal behavior of the displayed image. 
     Although the detailed description contains many specifics, these should not be construed as limiting the scope of the invention but merely as illustrating different examples and aspects of the invention. It should be appreciated that the scope of the invention includes other embodiments not discussed in detail above. For example, the sharpness metric could be implemented as a form of contrast, spatial frequency content, or entropy; the waveform step size can be optimized to fit nonuniform focal plane locations and spacing, and the adjustable aperture&#39;s size could be incorporated into these metrics. Various other modifications, changes and variations which will be apparent to those skilled in the art may be made in the arrangement, operation and details of the method and apparatus of the present invention disclosed herein without departing from the spirit and scope of the invention as defined in the appended claims. Therefore, the scope of the invention should be determined by the appended claims and their legal equivalents. 
     In alternate embodiments, aspects of the invention (e.g., the controller) may be implemented in computer hardware, firmware, software, and/or combinations thereof. Apparatus of the invention can be implemented in a computer program product tangibly embodied in a machine-readable storage device for execution by a programmable processor; and method steps of the invention can be performed by a programmable processor executing a program of instructions to perform functions of the invention by operating on input data and generating output. The invention can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. Each computer program can be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language if desired; and in any case, the language can be a compiled or interpreted language. Suitable processors include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory and/or a random access memory. Generally, a computer will include one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks. Any of the foregoing can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits) and other forms of hardware. 
     The term “module” is not meant to be limited to a specific physical form. Depending on the specific application, modules can be implemented as hardware, firmware, software, and/or combinations of these. Furthermore, different modules can share common components or even be implemented by the same components. There may or may not be a clear boundary between different modules.