Patent Description:
Unfortunately, the modulation efficiency with which the individual pixels are able to generate these photovoltaic signals typically decreases with increasing image height on the array of pixels. This is partly due to inefficiencies with which conventional designs for the arrays of micro lenses focus the backscattered light into a region of interest of a corresponding pixel.

It is with respect to these considerations and others that the disclosure made herein is presented. <CIT> relates to a time-of-flight (ToF) camera that is configured to operate in a manner that reduces power consumption of the ToF camera. For a key frame, a key-frame depth image is generated based on a plurality of sets of key-frame IR images. Each set of key-frame IR images is acquired using a different modulation frequency of active IR light. For a P-frame after the key frame, a P-frame depth image is generated based on a set of P-frame IR images acquired using a single modulation frequency of active IR light. <CIT> relates to an imaging device that includes: a microlens that focuses subject light; a light receiving element that receives the subject light focused by the microlens to thereby generate a signal for making focus determination through phase-difference detection; and a light blocking portion that is disposed between the microlens and the light receiving element so as to block part of the subject light focused by the microlens, wherein the distance between the light blocking portion and the microlens in an optical axis direction of the microlens is set so as to decrease as an image height increases.

This object is solved by the subject matter of the independent claim.

Technologies described herein provide for a three-dimensional time-of-flight (3D TOF) camera having a micro lens (ML) array configured with variable ML height and variable ML shift. In particular, the ML array includes a plurality of micro lens that are configured to direct backscattered light that is transmitted through an image lens into corresponding pixels. An image height for various individual pixels and/or individual micro lenses may be determined based on an axis of the image lens. In an exemplary embodiment, the height of individual micro lenses within the ML array vary according to the image height. For example, the height of micro lenses at the center of the ML array, near the axis of the image lens, may be relatively larger than the height of other micro lenses toward the perimeter of the ML array. Furthermore, the shift of individual micro lenses with respect to corresponding pixels may also vary according to the image height. For example, the shift of micro lenses at the center region of the ML array may be relatively smaller than the shift of the other micro lenses toward the perimeter of the ML array. As described in detail herein, the variable ML height and variable ML shift may be selected to maximize a volumetric optic power Hit Rate Rhit experienced at individual pixels across various image heights. The result of maximizing this volumetric optic power Hit Rate Rhit is increased modulation efficiency as compared to existing TOF pixel-based 3D cameras.

In an exemplary embodiment, a 3D TOF image camera includes a signal generator to generate a modulated electrical signal. The 3D TOF camera further includes a light emitter that is configured to emit modulated light based on the modulated electrical signals. The signal generator may simultaneously dispatch the modulated electric signals to a phase shifter. The 3D TOF camera further includes an image lens that receives backscattered light that is reflected by a physical object onto which the modulated light was emitted. Thus, the backscattered light that is received at the image lens includes at least a portion of the modulated light in addition to ambient light. The portion of the modulated light that is received at the image lens experiences a time delay td from the time t<NUM> at which the modulated light is emitted by the light emitter. Specifically, the time delay td corresponds to an amount of time that it takes the modulated light to travel from the light emitter to the physical object and, upon being reflected, from the physical object to the image lens.

The 3D TOF camera further includes a micro lens time-of-flight (ML-TOF) sensor having a ML array and a pixel array. Specifically, the ML array is disposed between the image lens and onto the pixel array. The ML array includes a plurality of micro lenses (MLs) and the pixel array includes a plurality of pixels. Individual ones of the MLs may be positioned and/or sized to optimally focus rays of the reflected light with respect to individual ones of the pixels. For example, a first micro lens may be sized and positioned to focus rays of the reflected light with respect to a first pixel whereas an Nth micro lens may be sized and positioned to focus rays of the reflected light with respect to an Nth pixel. More specifically, the individual MLs may be sized and positioned with respect to the corresponding pixels to maximize a volumetric optic power Hit Rate Rhit as defined herein.

In some embodiments, heights of the individual MLs within the ML array decrease with increasing radial distance from the axis. Stated alternatively, the heights of the individual MLs within the ML array are inversely related to the radial distance of those individual MLs from the axis. Additionally, or alternatively, shifts of the individual MLs with respect to a center of a corresponding pixel may increase with increasing radial distance from the axis. Stated alternatively, the positional shifts of the individual MLs with respect to a corresponding pixel are directly (i.e., as opposed to inversely) related to the radial distance of those individual MLs from the axis.

Individual ones of the pixels may include one or more photodetector cells that generate photoelectric signals when stricken with incident light. These photoelectric signals may be provided directly from the ML-TOF sensor to a controller which may analyze the photoelectric signal to determine per-pixel depth information associated with the physical object and any other objects in a Field-of-View of the 3D TOF camera.

This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended that this Summary be used to limit the scope of the claimed subject matter.

The same reference numbers in different figures indicates similar or identical items.

References made to individual items of a plurality of items can use a reference number followed by a parenthetical containing a number of a sequence of numbers to refer to each individual item. Generic references to the items may use the specific reference number without the sequence of numbers. For example, the items may be collectively referred to with the specific reference number preceding a corresponding parenthetical containing a sequence number.

<FIG> illustrates a schematic diagram of a three-dimensional (3D) time-of-flight (TOF) camera <NUM> having a micro lens (ML) array <NUM> configured with variable ML height and shift, according to an exemplary embodiment of the present disclosure. In some embodiments, the 3D TOF camera <NUM> may include a controller <NUM> that is configured to provide output signals to and receive input signals from various components of the 3D TOF camera <NUM>. As illustrated, the controller <NUM> is shown to be receiving one or more inputs signals <NUM> from an application <NUM>. For example, the input signals <NUM> may include a request from the application <NUM> for depth data that may be generated by the 3D TOF camera <NUM>. In plain terms, depth data may refer to one or more maps of per-pixel data containing depth-related information.

In response to the input signals <NUM> being received from the application <NUM>, the controller <NUM> may output various control signals to cause operations to be performed by one or more other components of the 3D TOF camera <NUM>. For example, as illustrated, the controller <NUM> is shown to be outputting control signals <NUM> to a signal generator <NUM> that is configured to generate modulated electrical signals <NUM>. The signal generator <NUM> may generate the modulated electrical signals <NUM> in response to the control signals <NUM> being received from the controller <NUM>. In some embodiments, the modulated electrical signals <NUM> may be any periodic modulated electrical signals. Additionally, or alternatively, the modulated electrical signals <NUM> may be frequency-modulated electrical signals. In embodiments in which the modulated electrical signals <NUM> are frequency modulated, the modulated electrical signals <NUM> may modulate in frequency. In embodiments in which the modulated electrical signals <NUM> are amplitude modulated, amplitude modulation may occur with a predetermined modulation frequency.

The 3D TOF camera <NUM> further includes a light emitter <NUM> that is configured to emit modulated light <NUM> based on the modulated electrical signals <NUM>. For example, as illustrated, the modulated electrical signals <NUM> are output from the signal generator <NUM> to the light emitter <NUM> which in turn emits modulated light <NUM> in response to the modulated electrical signals <NUM>. As further illustrated and describe in detail below, the signal generator <NUM> may simultaneously dispatch the modulated electric signals <NUM> to a phase shifter <NUM>. An exemplary light emitter <NUM> may include, for example, a light-emitting diode (LED), a laser diode, or any other light source suitable for emitting modulated light <NUM> based on the modulated electrical signals <NUM>. The modulated light <NUM> may be periodic-modulated in embodiments in which the modulated electrical signals <NUM> are periodic-modulated and frequency-modulated in embodiments in which the modulated electrical signals <NUM> are frequency-modulated. As illustrated, the modulated light <NUM> is emitted from the 3D TOF camera <NUM> toward a physical object <NUM> for which the requested depth data is to be generated. As further illustrated, ambient light <NUM> may also be emitted toward the physical object <NUM> from one or more ambient light sources <NUM>. For example, the ambient light <NUM> may be emitted toward the physical object <NUM> from a light source (e.g., a light bulb, the sun, etc.) that generates light. Additionally, or alternatively, the ambient light <NUM> may be reflected toward the physical object <NUM> from an indirect passive light source (e.g., a wall, a mirror, etc.) that reflects at least a portion of the electromagnetic (EM) spectrum.

The 3D TOF camera <NUM> further includes an image lens <NUM> that receives backscattered light <NUM> that is reflected by the physical object <NUM>. The backscattered light <NUM> that is received at the image lens <NUM> includes both a portion of the modulated light <NUM> and a portion of the ambient light <NUM>. The portion of the modulated light <NUM> that is received at the image lens <NUM> experiences a time delay td from the time t<NUM> at which the modulated light <NUM> is emitted by the light emitter <NUM>. Specifically, the time delay td corresponds to an amount of time that it takes the modulated light <NUM> to travel from the light emitter <NUM> to the physical object <NUM> and, upon being reflected, from the physical object <NUM> to the image lens <NUM>.

The 3D TOF camera <NUM> further includes a micro lens time-of-flight (ML-TOF) sensor <NUM> having a ML array <NUM> and a pixel array <NUM>. As illustrated, the ML array <NUM> is disposed between the image lens <NUM> and onto the pixel array <NUM>. The ML array <NUM> includes a plurality of micro lenses (MLs) <NUM> and the pixel array <NUM> includes a plurality of pixels <NUM>. As described in more detail below, individual ones of the MLs <NUM> may be positioned and/or sized to optimally focus rays <NUM> of the reflected light <NUM> (which have passed through the image lens <NUM>) with respect to individual ones of the pixels <NUM>. For example, as illustrated, a first micro lens <NUM>(<NUM>) is sized and positioned to focus rays <NUM> of the reflected light <NUM> with respect to a first pixel <NUM>(<NUM>) whereas an Nth micro lens <NUM>(N) is sized and positioned to focus rays <NUM> of the reflected light <NUM> with respect to an Nth pixel <NUM>(N). As described in detail below, the individual MLs <NUM> may be sized and positioned with respect to the corresponding pixels <NUM> to maximize a volumetric optic power Hit Rate Rhit as defined by Equations <NUM> and <NUM> below.

With respect to the individual MLs <NUM> within the ML array <NUM> varying in size, the first ML <NUM>(<NUM>) may be manufactured to a first height whereas the Nth ML <NUM>(N) may be manufactured to an Nth height that is different than the first height. In some embodiments, heights of the individual MLs <NUM> within the ML array <NUM> may be varied according to a distance from an optical axis <NUM> of the image lens <NUM> (or any other suitable reference datum of the 3D TOF camera <NUM>). As illustrated, for example, the first ML <NUM>(<NUM>) is shown to be substantially centered on the axis <NUM> whereas the Nth ML <NUM>(N) is shown to be offset from the axis <NUM> by some distance. Thus, the radial distance of the first ML <NUM>(<NUM>) from the axis <NUM> is relatively less than the radial distance of the Nth ML <NUM>(N) from the axis <NUM>. In some embodiments, heights of the individual MLs <NUM> within the ML array <NUM> decrease with increasing radial distance from the axis <NUM>. Stated alternatively, the heights (and/or width and/or curvature) of the individual MLs <NUM> within the ML array are inversely related to the radial distance of those individual MLs <NUM> from the axis <NUM>. To illustratively convey this point, in <FIG> the first ML <NUM>(<NUM>) is drawn at a relatively larger size as compared to the Nth ML <NUM>(N). It is worth noting that although the MLs <NUM> are depicted as circular, this is a simplistic representation that intended to emphasis the variations between MLs. It will be appreciated by one skilled in the art that the actual shape of the MLs may be square with a spherical lens top. It is also worth noting that although the MLs <NUM> are depicted in <FIG> and <FIG> with gaps therebetween, in some embodiments the ML array <NUM> is a gapless-type ML array in which individual ones of the MLs <NUM> abut other ones of the MLs. Furthermore, it is worth noting that in a gapless-type ML array that is configured with a linear ML shift, the square pitch size of the individual MLs may be constant or, alternatively, may vary. For example, in a gapless-type ML array with linear shift (e.g., the same amount of shift for each ML with respect to a corresponding pixel), the square pitch size of the individual MLs may be constant across the entire ML array. As another example, in a gapless-type ML array with non-linear shift (e.g., where the amount of shift for the individual MLs varies based on image height), the square pitch size of the individual MLs may vary across the ML array.

With respect to the individual MLs <NUM> within the ML array <NUM> varying in position (i.e., shift in relation to a center of a corresponding pixel <NUM>), the first ML <NUM>(<NUM>) may be radially shifted a first amount with respect to a center of the first pixel <NUM>(<NUM>) whereas the Nth ML <NUM>(N) may be radially shifted an Nth amount with respect to a center of the Nth pixel <NUM>(N) - the first amount being different than the Nth amount. In some embodiments, positional shifts of the individual MLs <NUM> with respect to corresponding pixel <NUM> may be varied according to a distance from the axis <NUM> of the image lens <NUM> (or any other suitable reference datum of the 3D TOF camera <NUM>). To illustrate this point, recall that the radial distance of the first ML <NUM>(<NUM>) from the axis <NUM> is relatively less than the radial distance of the Nth ML <NUM>(N) from the axis <NUM>. Thus, due to these different radial distances from the axis, the position of the first ML <NUM>(<NUM>) and the Nth ML <NUM>(N) may be shifted different amounts with respect to centers of the first pixel <NUM>(<NUM>) and the Nth pixel <NUM>(N), respectively. In an exemplary embodiment, shifts of the individual MLs <NUM> with respect to a center of a corresponding pixel <NUM> may increase with increasing radial distance from the axis <NUM>. Stated alternatively, the positional shifts of the individual MLs <NUM> with respect to a corresponding pixel <NUM> are directly (i.e., as opposed to inversely) related to the radial distance of those individual MLs <NUM> from the axis <NUM>. To illustratively convey this point, the first ML <NUM>(<NUM>) is shown in <FIG> to be substantially centered over the first pixel <NUM>(<NUM>) due to being substantially centered on the axis <NUM>. In contrast, however, the Nth ML <NUM>(N) is shown in <FIG> to be shifted away from a center of the Nth pixel <NUM>(N) and toward the optical axis <NUM> due to the Nth pixel <NUM>(N) and, therefore, the Nth ML <NUM>(N) being along the outer perimeter of the ML array <NUM> (e.g., distant from or not centered over the axis <NUM>).

As described in more detail below, individual ones of the pixels <NUM> may include one or more photodetector cells that emit electric current (e.g., via the photoelectric effect) when stricken with incident light (e.g., rays <NUM>). Within the present disclosure, electric current (or electric voltage for that matter) that is emitted by the one or more photodetector cells in response to incident light may be referred to as a photoelectric signal. In some embodiments, this photoelectric signal may be provided directly from the ML-TOF sensor <NUM> to the controller <NUM> which may analyze the photoelectric signal to determine per-pixel depth information associated with the physical object <NUM> and any other objects in a Field-of-View of the 3D TOF camera <NUM>.

In some embodiments, the 3D TOF camera <NUM> may further include a phase shifter <NUM>. The phase shifter <NUM> may also receive the modulated electrical signal <NUM> from the signal generator <NUM> and apply a phase shift including one or more phase shift steps to the modulated electrical signal <NUM>. Then, the phase shifter <NUM> may transmit the modulated electrical signal <NUM> (or some variant thereof) that is received from the signal generator <NUM> to the ML-TOF sensor <NUM>. The modulated electrical signal <NUM> that is provided to the ML-TOF sensor <NUM> may correlate with the photoelectric signal generated in response to the incident light striking the one or more photodetector cells (e.g., due to the backscattered light <NUM> including a portion of the modulated light <NUM>). Thus, the modulation signal after the phase shifter <NUM> may demodulate the modulated electric signal <NUM> (which is provided from the signal generator <NUM> and in accordance with which the modulated light <NUM> is emitted) from the photoelectric signal to extract one or more components of the photoelectric signal that are specifically associated with modulated light <NUM> that is reflected off the physical object <NUM> toward the image lens <NUM>. It will be appreciated that the signal resulting from the demodulation of the modulated electric signal <NUM> from the photoelectric signal may be a correlation electrical signal <NUM>.

As illustrated, after the modulated electrical signal <NUM> has been demodulated from the photoelectric signal, the controller <NUM> may receive the correlation electrical signal <NUM> from the ML-TOF sensor <NUM>. In some embodiments, the photoelectric signal and/or the correlation electrical signal <NUM> may be amplified prior to transmission to the controller <NUM>. The controller <NUM> may be further configured to determine, based on a phase difference between the correlation electrical signal <NUM> and the modulated electrical signal <NUM>, a time-of-flight of the reflected modulated light. It should be appreciated that the phase difference and time-of-flight may be determined on a per-pixel basis to generate the depth data requested by the application <NUM>. The depth data may then be provided to the application <NUM> with an output signal <NUM>.

An example algorithm by which the phase difference may be determined in some embodiments is provided below. The example algorithm is an algorithm for determining the phase difference between the correlation electrical signal <NUM> and a periodic-modulated electrical signal, e.g., a sinusoidal signal. In this example, the phase difference is determined for a simplified correlation between one frequency component of the photoelectric signal associated with modulated light <NUM> that is reflected off the physical object <NUM> and one frequency component associated with the modulated electrical signal <NUM>. The correlation of the frequency components for one frame captured by a pixel <NUM> is given by Equation <NUM> as follows: <MAT> In Equation <NUM>, CM<NUM> is a common mode voltage signal corresponding to a direct current (DC) signal received from the pixel <NUM>. CM<NUM> includes signals associated with both the modulated light <NUM> emitted by the light emitter <NUM> and ambient light <NUM> emitted by the ambient light source <NUM>. A formula for CM<NUM> is given below in Equation <NUM>: <MAT> In Equation <NUM>, N is the total number of phase shifting steps.

Returning to Equation <NUM>, AB<NUM> is the amplitude of an alternating voltage of the modulated light <NUM>, ϕd<NUM> = <NUM>πftd<NUM> is the phase of the time of flight td<NUM>, and ψk is the kth phase shift, being equal distributed within <NUM>π. A formula for AB<NUM> is given below in Equation <NUM>: <MAT> In addition, a formula for ϕd<NUM> is given below in Equation <NUM>: <MAT> In Equation <NUM>, Ik is the correlation result of voltage output contributed by the photoelectric signal at the kth phase shifting step from each pixel <NUM>.

Although the above example is provided for a single frequency, the above example may be extended to signals including multiple components with different frequencies by summing over the components. Thus, a single pixel <NUM> may concurrently provide time-of-flight data for a plurality of different wavelengths of light.

Turning now to <FIG>, illustrated is a top view and a cross-sectional side view, respectively, of an exemplary micro lens time-of-flight (ML-TOF) sensor <NUM> having a ML array <NUM> configured with a plurality of individual MLs <NUM> that are manufactured with variable ML height and shift with respect to corresponding pixels <NUM>. The cross-sectional side view shown in <FIG> is taken along the line A-A shown in <FIG>. In the illustrated example, the ML array <NUM> includes thirty-six individual MLs <NUM> that are each positioned with respect to an individual region-of-interest (ROI) <NUM> of a corresponding pixel <NUM> within a pixel array <NUM>. It is worth noting that in each of <FIG>, the MLs <NUM> are illustrated in dotted circle to represent its location and spherical or aspherical feature (rather than solid) lines. It is worth noting that the pitch of ML in general are square formed to have maximum fill factor assisted with gapless ML process technology. Individual ones of the MLs <NUM> uniquely correspond to and are specifically positioned with respect to individual ones of the pixels <NUM>. For example, as illustrated in <FIG>, a first ML <NUM>(<NUM>) uniquely corresponds to and is specifically positioned with respect to a first ROI <NUM>(<NUM>) of a first pixel <NUM>(<NUM>), a second ML <NUM>(<NUM>) uniquely corresponds to and is specifically positioned with respect to a second ROI <NUM>(<NUM>) of a second pixel <NUM>(<NUM>), and so on.

In some embodiments, the ROI <NUM> of an individual pixel <NUM> is a particular volume of photodetector material that is configured to emit electric signals in response to being stricken by incident backscattered modulated light. In this respect, a typical TOF pixel has a limited electrical field modulating area that projects to a particular depth (thereby forming a volumetric region-of-interest). Furthermore, in a typical TOF pixel this "volumetric" ROI represents a portion of the photodetector material of the TOF pixel that most effectively transfers electrons in response to photon stimulation. Thus, the ROI <NUM> of a pixel <NUM> is the region of that pixel <NUM> which is best suited for producing photoelectric signals with a high modulation frequency (e.g., <NUM> or higher) such as that used in typical TOF sensors. Exemplary such ROIs <NUM> may be formed of an Epitaxy layer of Silicon.

Referring specifically to <FIG>, optical axes <NUM> for individual ones of the MLs <NUM> are graphically represented as a cross-mark that is bound by a square (<IMG>). In the illustrated embodiment, the optical axes <NUM> for each individual one of the MLs <NUM> are positioned with respect to an individual ROI <NUM> of a corresponding pixel <NUM> by an amount of shift that varies according to a distance of the corresponding pixel <NUM> from a reference datum <NUM> ( graphically represented as <IMG>). An exemplary such reference datum <NUM> may be, for example, the optical axis <NUM> of the image lens <NUM> described in <FIG>. To illustrate this variable shift that varies according to distance from the reference datum <NUM>, within <FIG> individual ones of the optical axes <NUM> for the four MLs <NUM> at the four corners of the ML array <NUM> are shifted away from the center of the corresponding ROI <NUM> and toward the reference datum <NUM> by a greater distance than are the optical axes <NUM> for the four MLs <NUM> that touch the reference datum <NUM>. Thus, in the illustrated embodiment, the amount of shift (e.g., measured in micrometers "µm") of each ML <NUM> with respect to a center of a corresponding pixel <NUM> increases with increasing distance from the reference datum <NUM>. That is, the amount of shift of each ML <NUM> with respect to a corresponding pixel <NUM> is positively related to the distance of that ML <NUM> from the reference datum <NUM>.

In some embodiments, the individual MLs <NUM> have geometric parameters (e.g., size, height, width, lens curvature, etc.) that vary according to distance from the reference datum <NUM>. For example, in the embodiment illustrated in <FIG>, a first ML <NUM>(<NUM>) through a sixth ML <NUM>(<NUM>) are each shown to have geometric parameters of height and width that vary according to a distance from the reference datum <NUM>. Here, a first height H<NUM> is shown for the first ML <NUM>(<NUM>) and a third height H<NUM> is shown for the third ML <NUM>(<NUM>), where the first height H<NUM> is less than the third height H<NUM>. In the illustrated embodiment, the heights of the individual MLs <NUM> decrease with increasing distance from a reference datum <NUM>. Specifically, the third height H<NUM> (which corresponds to MLs <NUM>(<NUM>) and <NUM>(<NUM>) - since these MLs are the same distance from the reference datum <NUM>) is greater than a second height which corresponds to MLs <NUM>(<NUM>) and <NUM>(<NUM>) (note the second height is not labeled). Furthermore, the first height H<NUM> is less than the third height H<NUM> and is also less than the second height H<NUM>. Thus, in the illustrated embodiment, geometric size parameters (e.g., height and/or width) of the individual MLs <NUM> is inversely related to the distance of that ML <NUM> from the reference datum <NUM>. As described in more detail below, the varying positional shifts and/or geometric parameters for the individual MLs may be optimized to maximize a volumetric optic power Hit Rate Rhit as defined by Equations <NUM> and <NUM> below.

Turning now to <FIG>, illustrated is an exemplary optical framework <NUM> between an image lens <NUM> and a micro lens time-of-flight (ML-TOF) sensor <NUM>. The optical framework <NUM> is used to build a ML-TOF optimization model as described below. More specifically, by jointly modeling characteristics of the image lens <NUM> and the ML-TOF sensor <NUM> using the geometrical framework <NUM> in conjunction with the following mathematical framework, the geometric parameters (e.g., heights, shifts, etc.) of the individual MLs within an ML array <NUM> can be optimized to maximize the volumetric optic power Hit Rate Rhit of the backscattered light into the silicon in the pixel modulation regions (i.e., the volumetric deep depletion region).

The present framework accounts for a variety of design differences between the characteristics of TOF pixels and normal image pixels. One such design difference corresponds to typical TOF sensors using active infrared (IR) light to stimulate photoelectric signals from which depth information may be correlated. For example, a TOF sensor may actively emit IR light at a wavelength range at or near <NUM> nanometers which has a penetration depth into silicon of roughly <NUM>. This penetration depth is more than six times that of visible light (e.g., roughly <NUM>) that normal image pixels are designed to detect. Furthermore, rather than using the chief ray angle <NUM> as typically used in conventional optical models, the present framework uses projections of rim rays <NUM> (i.e., rays projected from the marginal portion of the image lens <NUM>) which have relatively larger incident angles as compared to the chief ray angle <NUM> as outer boundaries for cones of randomly generated input rays. In particular, input rays on a multitude of points that span the entire surface area of the ML array <NUM> are generated and constrained within an optical ray cone that is bound by the upper rim ray <NUM>(U) and lower rim ray <NUM>(L) around the chief ray angle <NUM> (as described in relation to <FIG>). Using this framework, photon volume density metrics may be calculated based on a Fresnel power hitting rate to optimize the design of the ML-TOF sensor <NUM>. For example, and as described in more detail below in relation to <FIG>, multiple different ROI projection slices may be defined and then weighted by their corresponding absorption behaviors.

Turning now to <FIG>, illustrated is an exemplary optical framework <NUM> for a single pairing of a pixel and corresponding micro lens within the exemplary micro lens time-of-flight (ML-TOF) sensor <NUM>. In the following discussion, the exemplary optical framework <NUM> is discussed in terms of a sequence of four regions that are identified with solid black arrows that are individually numbered one through four.

Referring specifically to the first region, a plurality of rays <NUM> are modeled at a multitude of points on the surface of a micro lens. In particular, at individual ones of these multitude of points, a cone-shaped bundle of input rays <NUM> is formed that is bound by the upper and lower rim rays (described in relation to <FIG> and <FIG>). These cone-shaped bundles of input rays <NUM> may be generated by Monte Carlo methods or various other repeated random sampling techniques. That is, the specific parameters for each individual cone-shaped bundle of input rays <NUM> is determined by the upper rim ray <NUM>(U) and the lower rim ray <NUM>(L) of the image lens <NUM> as varied by different image heights as shown in <FIG> (e.g., radial distances from the axis <NUM> of the image lens <NUM>). It is worth noting that although merely <NUM> input rays <NUM> are shown in <FIG>, practical implementations of the techniques described herein may include generating hundreds or even thousands of input rays <NUM> in association with any individual micro lens.

Referring specifically now to the second region, the curvature (e.g., height) and offset (e.g., shift) of the micro lens is optimized to maximize the volumetric optic power Hit Rate Rhit as described below. Referring specifically now to the third region, the rays <NUM> are modeled as propagating through a back side of the optical framework <NUM> such as, for example, a Backside Illuminated (BSI). In various implementations, the optimization techniques described herein account for various refractive indexes such as a first refractive index n<NUM> of the micro lens, a second refractive index n<NUM> of an oxidation layer that forms a ML Pedestal Height within the micro lens layer, and an mth refractive index nm within back side of the optical framework <NUM>. Referring specifically now to the fourth region, the volumetric region of interest (ROI) of an exemplary pixel is represented with seven ROI projection slices <NUM> shown. It should be appreciated that these several slices represent the deep depletion region of the illustrated TOF pixel. As described in detail below, these so-called ROI projection slices <NUM> are used for volumetric optimization.

Next, an advanced formulation for a volumetric optic power Hit Rate Rhit is defined based on the foregoing optical frameworks <NUM> and <NUM> of <FIG>. The following volumetric optic power Hit Rate Rhit is usable to optimize the photon volumetric density within the regions of interest of the individual pixels within a ML-TOF sensor <NUM> based on the following parameters. Within the advanced formulation, the power transmittance <MAT> of individual rays <NUM> within the various coned bundles (e.g., that are bound by the upper rim ray <NUM>(U) and the lower rim ray <NUM>(L)) through all optical mediums is defined by Equation <NUM> as follows: <MAT> where <MAT>, which is denoted by ray vector <MAT>, is related to the Fresnel's transmittance with S and P polarized waves carried by the kth ray vector at mth medium boundary of the optical framework <NUM> (e.g., a Backside Illuminated "BSI" process stack). The boundary number m is counted from the outer surface of the micro lens at which m = <NUM> to the volumetric region of interest (ROI) (e.g., silicon) at which m = M. In some implementations in which the various mediums, e.g., through which the rays <NUM> propagate, have absorption, optical complex indexes may be applied.

Next, the transmittance summation TAs (j), which is the summation of the transmittance for all of the rays <NUM> in the volumetric "pipe" of the photodetector material (e.g., silicon) bounded by the projection of the optically active area As (i.e., the ROI at a given depth j inside of the photodetector material), is defined by Equation <NUM> as follows: <MAT> Then, an average of the transmittance T with all ROI projection slices <NUM> of varying depth number Nj is defined by Equation <NUM> as follows: <MAT> In implementations in which absorption of the photodetector material is taken into account, a depth weighted version of Equation <NUM> can be defined by Equation <NUM> as follows: <MAT> where α the silicon absorption coefficient at given wavelength, if the depth j in µm, then <MAT> at a wavelength of <NUM>.

Based on the foregoing, the volumetric optic power Hit Rate Rhit is defined by Equation <NUM> as follows: <MAT> where N is the total number of rays generated for the optimization. Furthermore, for an absorption weighted case, the volumetric optic power Hit Rate Rhit is defined by Equation <NUM> as follows: <MAT> where again N is the total number of rays generated for the optimization.

Deploying the foregoing advanced formulation for maximizing volumetric optic power Hit Rate Rhit provides several benefits. One such benefit is maximizing the number of rays <NUM> that become directed through or constrained within the ROI of each pixel. Due to the longer wavelength and, therefore deeper penetration into the photodetector material, of the IR modulated light, maximizing the number of rays <NUM> constrained within the ROI of each pixel is especially important for TOF sensors as compared to normal image sensors. Another such benefit is providing for minimization of the Fresnel reflection by optimizing curvature of each micro lens to best respond to all of the modeled rays within the cone bundle bound by the larger angular upper rim ray <NUM>(U) and lower rim ray <NUM>(L) as opposed to the chief ray angle <NUM>. The ultimate result of deploying the foregoing formulation to optimize the ML array <NUM> is a maximized photon power within the relevant volume of the ROI - thereby maximizing the modulation efficiency while minimizing crosstalk.

Turning now to <FIG>, illustrated are a plurality of ray tracing responses <NUM> that are modeled based on various ML lens parameters and with incoming light modeled as columnated light that is parallel to the axis of the image lens. Stated alternatively, the modeled incoming rays of light are all parallel to the axis of the image lens prior to striking the illustrated micro lens within each scenario. Each of these four different scenarios are optimized in accordance with various design criteria. More specifically, illustrated in <FIG> are four different scenarios (scenarios 5A through 5D), each of which have a corresponding ray tracing response (labeled <NUM>(A) through <NUM>(D)) which is modeled based on corresponding ML lens parameters as defined in Table <NUM> below. In each scenario, the incoming light is modeled as columnated light being received at a zero ("<NUM>") incident angle condition (i.e., propagating straight down as shown in <FIG>). Furthermore, the ML lenses are modeled as being spherical lenses in each of scenarios 5A through 5D. With respect to the optimization criteria used in each of these scenarios: scenario 5Ais optimized to have a focal point <NUM>(A) designed at the top of the ROI <NUM>(A), scenario 5B is optimized to have a focal point <NUM>(B) at the mid-point of the ROI <NUM>(B), scenario 5C is optimized to have a focal point <NUM>(C) at the bottom of the ROI <NUM>(C), and scenario 5D is optimized to have a volumetric criterion in which the focal point <NUM>(D) is somewhere below the ROI <NUM>(D) - as indicted by the arrow directed downward. Table <NUM> which tabulates the ML lens parameters is shown below as follows:.

As can be appreciated from <FIG>, designing to a single focal point is non-optimal for a TOF pixel due to the deep penetration of the long wavelength of IR light into the photodetector material.

Turning now to <FIG>, illustrated are a plurality of ray tracing responses <NUM> that are modeled based on the same ML lens parameters of <FIG> and with incoming light modeled as non-columnated light that is bound by the upper rim ray <NUM>(U) and lower rim ray <NUM>(L) described in relation to <FIG>. It can be appreciated that modeling the incoming rays as bound between the upper rim ray <NUM>(U) and lower rim ray <NUM>(L) (e.g., + <NUM> degrees to - <NUM> degrees, or some other degree range), yields significantly different ray tracing responses than those shown in <FIG>. Moreover, since in real-life TOF implementations incoming light is received at the image lens and micro lens from many different angles (not simply the CRA <NUM>), the ray tracing responses <NUM> of <FIG> represent significantly more accurate predictions of real-life optical behavior than the ray tracing responses <NUM> of <FIG>.

As illustrated, each of the four scenarios modeled in <FIG> effectively direct some portion of the modeled incoming light into the region of interest <NUM> (i.e., the photoactive region of the pixel). This portion of the modeled incoming light that is directed into the corresponding region of interest specifically corresponds to the volumetric optic power Hit Rate Rhit as defined by Equations <NUM> and <NUM> above. As illustrated in <FIG>, each of ML Type A and ML Type B as shown in Scenario 6A and Scenario 6B direct a substantial amount of incoming light outside of the corresponding region of interest <NUM>. This could potentially result in undesirable crosstalk between neighboring pixels. In contrast, each of ML Type C and ML Type D as shown in Scenario 6C and Scenario 6D direct a substantial amount of incoming light into the corresponding region of interest <NUM> - although as shown in <FIG> not all incoming rays are perfectly contained within the ROI <NUM>(D). Furthermore, with specific reference to Scenario D, as shown in <FIG>, the ML Type D effectively re-directs and constrains nearly all incoming light into the region of interest <NUM>(D). Thus, Scenario D represents the highest volumetric optic power Hit Rate Rhit shown in <FIG>.

Turning now to <FIG>, illustrated is a graph that shows the volumetric optic power Hit Rates for various optical scenarios - including those that are modeled and described in relation to <FIG>. More specifically, each of the lines that are graphed in <FIG> correspond to an optical model in which incoming light rays are modeled as striking each point of a micro lens from a multitude of different angles (e.g., however many are randomly modeled using Monte Carlo methods) that are bound by an upper rim ray and lower rim ray of +<NUM> degrees and -<NUM> degrees, respectively. Furthermore, each individually graphed line corresponds to a model of a micro lens have a particular ML size, ML radius curvature, and ML pedestal height (e.g., referred to as FOC in <FIG>) as described in Table <NUM> above. Then, for each optical scenario modeled, the volumetric optic power Hit Rates are determined across a range of ML Heights as indicated by the lower axis of the graph in <FIG>. In this way, optical parameters for each individual micro lens within a micro lens array of a TOF sensor can be identified so as to achieve the highest volumetric optic power Hit Rate.

On each one of the lines graphed in <FIG> is a specifically labeled point that corresponds to one of the scenarios described in relation to <FIG>. In particular, the point labelled A within <FIG> corresponds to Scenario 6A which is described in relation to <FIG>, the point labelled B within <FIG> corresponds to Scenario 6B which is described in relation to <FIG>, and so on. Furthermore, it can be appreciated from <FIG> that the highest volumetric photon hit rate that was modeled during the generation of the graph of <FIG> corresponds to Scenario 6D of <FIG>. Thus, an examination of a graph similar to that shown in <FIG> reveals the optical parameters (e.g., the ML Height, ML Pedestal Height, ML Radius Curvature) which are usable to achieve an optimal (e.g., maximized) volumetric photon hit rate for each pixel within a pixel array based on the image height of that pixel within the pixel array as described in relation to <FIG>.

In an exemplary implementation, an optimization procedure for identifying the optimal TOF-ML parameters (e.g., ML Curvature, Pedestal Height, etc.) at each different image array location (e.g., each pixel) includes deploying Monte Carlo methods to generate a predetermined number of input rays (e.g., <NUM>,<NUM> input rays) with random incident angles that are constrained by the lens output rim ray cone (e.g., rather than the chief ray angle) as described in relation to <FIG>. For example, for a centrally located pixel within a pixel array, the upper rim ray <NUM>(U) and lower rim ray <NUM>(L) may form a lens output rim ray cone that extends from +<NUM> degrees to -<NUM> degrees. Thus, for this centrally located pixel, the optimization procedure may include generating <NUM>,<NUM> input rays that strike this pixel at a randomly determined angle that is somewhere between +<NUM> degrees to -<NUM> degrees. It should be appreciated that the angular constraints of these randomly generated input rays will vary at different sensor locations (e.g., image heights). For example, generally speaking the lens output rim ray cone will have different tiling angles related to the chief ray angle at different sensor locations. With respect to this point, <FIG> graphically illustrates the angles for each of the lower rim ray <NUM>(L), the upper rim ray <NUM>(U), and the chief ray angle (CRA) <NUM> vs image height for the exemplary optical framework <NUM> of <FIG>.

Turning now to <FIG>, illustrated are a plurality of ray tracing responses <NUM> that demonstrate an effect of shifting micro lenses with respect to a region of interest <NUM> of a pixel on the volumetric optic power Hit Rate. More specifically, <FIG> shows three ray tracing responses (labeled <NUM>(A) through <NUM>(C)) that each correspond to different optical circumstances. For purposes of the present discussion, each of Scenarios 9A through 9C have common optical circumstances of being modeled with a ML Height of <NUM>, a ML Radius Curvature of <NUM>, and a ML Oxidation Pedestal of <NUM>.

Referring now specifically to Scenario 9A, illustrated is a ray tracing response <NUM>(A) that corresponds to a pixel that is centrally located on a pixel array <NUM> of a ML-TOF sensor <NUM> (e.g., a pixel at the Image Height = <NUM>). Under such circumstances, it can be appreciated the chief ray angle for this centrally located pixel is zero. Therefore, referring to <FIG>, the optical ray cone that is generated (e.g., using Monte Carlo methods to generate <NUM>,<NUM> input rays of varying incoming angles and that strike varying points on the micro lens) for this centrally located pixel has been constrained within a rim-ray range of +<NUM> degrees to -<NUM> degrees (as defined in <FIG>). Furthermore, in Scenario 9A which has been modeled to generate the ray tracing response <NUM>(A), the micro lens is modeled as having zero shift in relation to the region of interest <NUM>(A). That is, the micro lens in Scenario 9A is modeled as being centered over the region of interest <NUM>(A) of the corresponding pixel. As illustrated, in the ray tracing response <NUM>(A) modeled in association with Scenario 9A, the bulk of the rays are well contained within the region of interest <NUM>(A).

Referring now specifically to Scenarios 9B and 9C, illustrated are ray tracing responses that corresponds to a pixel that is located around the peripheral region (outer region) of the pixel array <NUM> of the ML-TOF sensor <NUM> (e.g., a pixel at an Image Height greater than zero). More specifically, in Scenarios 9B and 9C the pixel is offset from the axis <NUM> of the image lens <NUM> such that the CRA is <NUM> degrees. It should be appreciated that this specific CRA is arbitrarily chosen for illustrative purposes only and that many of CRAs could also be used to convey the concepts described in relation to <FIG>. Based on this specific CRA of <NUM> degrees, <FIG> reveals that the optical ray cone that is to be generated for a pixel with this specific image height is constrained within a rim-ray range of +<NUM> degrees to +<NUM> degrees.

Referring specifically now to Scenario 9B of <FIG>, the illustrated ray tracing response <NUM>(B) is generated based on the micro lens being modeled as having zero shift in relation to the region of interest <NUM>(B). That is, the micro lens in Scenario 9B is modeled as being centered over the region of interest <NUM>(B) of the corresponding pixel - just as it was in Scenario 9A. As illustrated, in the ray tracing response <NUM>(B) that results from the optical model of Scenario 9B, the bulk of the rays are not directed into the region of interest <NUM>(B) - the result being a much lower volumetric optic power Hit Rate Rhit as compared to Scenario 9A. In particular, based on the specific optic parameters described above and used to generate the ray tracing responses <NUM>(A) and <NUM>(B), the volumetric optic power Hit Rate Rhit of Scenario A is <NUM> whereas the volumetric optic power Hit Rate Rhit of Scenario B is a substantially lower <NUM>. Thus, Scenario 9B would yield worse modulation efficiency as compared to Scenario 9A.

Scenario 9C is similar to Scenario 9B with the exception that the micro lens is not modeled as being modeled as being shifted <NUM> toward the axis <NUM> of the image lens <NUM>. As illustrated, in the ray tracing response <NUM>(C) modeled in association with Scenario 9C, the bulk of the rays are directed into the region of interest <NUM>(B) (Note: the region of interest in both Scenario 9B and 9C is labeled <NUM>(B) because they are each modeled in association with the same pixel). In plain terms, the shifting of the micro lens from Scenario 9B to 9C results in a corresponding shift of the resulting ray tracing response <NUM>(C) into the region of interest <NUM>(B) - the result being a much higher volumetric optic power Hit Rate Rhit in Scenario 9C as compared to Scenario 9B. In particular, the addition of the <NUM> shift to the micro lens placement results in the volumetric optic power Hit Rate Rhit of Scenario C increasing to <NUM>.

Turning now to <FIG>, illustrated is a graphical representation of the effect of shifting the micro lens within an ML-TOF Sensor for a pixel having a predetermined image height (e.g., as indicated by the chief ray angle) on the volumetric photon hit rate experienced at that pixel. Here, the optical parameters used to generate the graphical representation shown in <FIG> are the same as those used to generate the ray tracing responses <NUM>(B) and <NUM>(C) shown in <FIG>. As illustrated, for a pixel having an image height of greater than zero (and therefore having a CRA of greater than zero), the maximum volumetric photon hit rate does not result from the micro lens being perfectly centered over the region of interest of a corresponding pixel (i.e., having a shift of zero). Rather, for TOF pixels having such an offset from the axis of an image lens, the volumetric photon hit rate increases with increasing ML shift until a maximum is reached - after which additional shift results in the volumetric photon hit rate decreasing. As illustrated in <FIG>, under the specific optical parameters used to model the scenario (e.g., CRA = <NUM> degrees, ML Pedestal = <NUM>, ML Height = <NUM>, etc.), this maximum volumetric photon hit rate is achieved at an ML Shift of <NUM> toward the axis of the image lens.

Turning now to <FIG>, illustrated is a graphical representation of the effect of shifting two different micro lenses, each having a different height, within an ML-TOF Sensor for a pixel having a predetermined image height on the volumetric photon hit rate experienced at that pixel. In particular, the graphical representation of <FIG> is similar to that of <FIG> with the exception that ML shift optimization results are graphed for a pixel having a CRA of <NUM> degrees under circumstances where a corresponding micro lens has a height of <NUM> and also under alternate circumstances where a corresponding micro lens has a height of <NUM>. It is worth noting that while only two lines are graphed - each corresponding to a particular ML Height - many more lines could potentially be included and examined to identity optimal optical parameters. As a specific but nonlimiting example, the graph of <FIG> could potentially include a line corresponding to a ML Height of <NUM>, another line corresponding to a ML Height of <NUM>, yet another line corresponding to a ML Height of <NUM>, and so on. As can be appreciated from <FIG>, under the modeled circumstances for a pixel residing on a pixel array at the particular image height, utilizing a micro lens having an ML height of <NUM> at an ML shift of <NUM> yields roughly the same volumetric photon hit rate as does utilizing a micro lens having a relatively larger ML height of <NUM> at a relatively larger ML shift of <NUM>.

Turning now to <FIG>, illustrated is a graph that represents a relationship between the micro lens shift and image height for two different optical models that result in similar volumetric optic power Hit Rate Rhit. In particular, the graph of <FIG> includes a line with points marked as black circles and that corresponds to an optical model in which the height of micro lenses within a micro lens array are varied in accordance with image height as described above. Adj acent to each black circle on this line is a numerical indication of the ML Height selected at a corresponding image height (e.g., using the techniques above). The graph of <FIG> further includes a line with points marked as black triangles and that corresponds to an optical model in which the height of micro lenses within a micro lens array remain constant (e.g., at <NUM>) regardless of image height. As can be seen from <FIG>, the shift needed to achieve a similar volumetric optic power Hit Rate Rhit at each image height is relatively less for the variable ML Height optical model as compared to the constant ML Height optical model. <FIG> is a graph that shows the similarity in volumetric optic power Hit Rate Rhit for the two optical models described and shown in <FIG>. It can be appreciated from <FIG> that by decreasing the ML Heights of the micro lenses within an ML array with increasing image height, a similar volumetric optic power Hit Rate Rhit can be achieved as if the ML Heights remained constant but with less ML Shift. Thus, it can be appreciated that there is no performance lost by decreasing both the ML Height and ML Shift as compared to an implementation in which ML Lenses of constant Height are shifted varying amount based on increasing Image Height. In this way, each ML may have different curvature and shift to achieve best modulation efficiency and QE as well to reduce ML shift amount. It will be appreciated by one skilled in the art that an ability to reduce ML shift amount is highly beneficial in terms of manufacturability, optical lens-ML-TOF array axis alignment robustness, as well the ML design flexibility because in many conventional manufacturing processes the ability to shift these parameters may be limited by the process quanta/resolution. The foregoing techniques, therefore, allow ML shift amount to be adjustable to fit the manufacturing and/or design parameters that are possible with the designed parameters closing to the optimized results.

In some embodiments, the heights of the individual MLs <NUM> within the ML array are linearly proportional to the radial distance of the individual MLs <NUM> from the axis <NUM>. As a specific but nonlimiting example, suppose that at a center point of the ML array that intersects with the axis a ML Height is set to <NUM> and that at an edge of the ML array an ML height is set to <NUM>. Under these circumstances, the ML Height for the individual MLs would linearly vary from a maximum ML height of <NUM> at the center of the pixel array to a minimum ML height of <NUM> at the outermost edge of the ML array.

In some embodiments, the heights of the individual MLs <NUM> within the ML array are non-linearly proportional to the radial distance of the individual MLs <NUM> from the optical axis <NUM>. To illustrate this point, presume that the variable ML heights shown in <FIG> along the "ML Height Variable" line are selected for a particular ML-TOF Sensor design. Under these circumstances, a polynomial-based best fit line may be generate based on an ML Height of <NUM> at an Image Height of <NUM>, an ML Height of <NUM> at an Image Height of <NUM>, and so on. It is worth noting that this Optimized Numeric Data shown in <FIG> is also represented in Table <NUM> herein. Then, this newly determined polynomial-based best fit line may be used to prescribe specific ML Heights at the various Image Heights.

In some embodiments, the shifts of the individual MLs <NUM> within the ML array are linearly related to the radial distance of the individual MLs <NUM> from the axis <NUM>. As a specific but nonlimiting example, suppose that at a center point of the ML array that intersects with the axis a ML shift is set to <NUM> (e.g., the ML, is perfectly centered over a corresponding pixel) and that at an edge of the ML array an ML shift is set to <NUM>. Under these circumstances, the ML shift for the individual MLs would linearly vary from a minimum ML shift of <NUM> at the center of the pixel array to a maximum ML shift of <NUM> at the outermost edge of the ML array.

In some embodiments, the shifts of the individual MLs <NUM> within the ML array are non-linearly proportional to the radial distance of the individual MLs <NUM> from the optical axis <NUM>. To illustrate this point, presume that the variable ML shifts shown in <FIG> along the "ML Height Variable" line are selected for a particular ML-TOF Sensor design. Under these circumstances, a polynomial-based best fit line may be generate based on an ML shift of <NUM> at an Image Height of <NUM>, an ML shift of <NUM> at an Image Height of <NUM>, and so on. That is, a best fit line may be generated based on the <NUM> unique ML Shift vs. Image Hight Data points shown in Table <NUM>. Then, this newly determined polynomial-based best fit line may be used to prescribe specific ML shifts at the various Image Heights.

Turning now to <FIG>, illustrated is an exemplary geometrical framework for defining an interpolation function for determining optimal micro lens shifts based on a relationship between image height and ML Height. Furthermore, <FIG> illustrates an exemplary geometrical framework that is usable for determining a new ML height and pitch size based on the new central location (i', j') for the ML that is determined based on the shift illustrated in <FIG>. With reference to <FIG>, an interpolation function for Variable ML shift according to simulation data presented in <FIG> (also represented in Table <NUM> below) is given by Equation <NUM> as follows: <MAT> where (i, j) is the location of the center of a pixel in relation to a reference datum <NUM>. In Equation <NUM>, ΔS(i,j) represents the ML shift distance away from the center of the pixel along the radius image height lr(i, j) (i.e., toward the center of the sensor array with regards to the position of ML without shift). In various implementations, the interpolation function Func1(Ir(i, j)) is a polynomial-type best fit function. Other types of functions suitable for determining a best fit are also suitable and contemplated for use in accordance with the disclosed techniques.

Based on the Optimized Numerical Data in Table <NUM>, one of the approximated ML shift functions can be formed as follows in Equation <NUM>: <MAT> Then, after the approximated ML shift above, the Optimized Numerical Data in Table <NUM> is used to make an interpolation of ML height in related to the simulation data (labeled in <FIG> or in Table <NUM>) based on Equation <NUM> as follows: <MAT> where (i',j') is the pixel location index representing the pixel central point after shifting from location (i, j), and where H(i', j') is the shifted variable ML height at the radius image height Ir(i', j') to the center of the sensor (i.e., the datum <NUM>). In various implementations, the interpolation function Func2 (Ir(i', j')) is a polynomial-type best fit function or any other type of function suitable for best fitting the ML Height Data Points of <FIG> (also in Table <NUM>). In this way, one of the approximated variable ML height functions can be generated from Table <NUM> as follows in Equation <NUM>: <MAT> Here, it should be appreciated that the variable ML height related to the ML shift at location (i, j), as well the shift direction and quantity is illustrated in <FIG>. After the ML shift as shown in <FIG> is determined, the ML pitch size centered at (i',j') is confined by the adjacent central points of the shifted ML: A rectangular surrounded by four lines; each line is passing through the middle of the adjacent points. In general, each shifted ML is with different pitch size, as well different height.

Turning now to <FIG>, illustrated is an exemplary pictorial flow diagram that graphically illustrates an exemplary process <NUM> for manufacturing a ML array in accordance with the characteristics determined based on the frameworks described above in relation to <FIG> and <FIG>.

At block <NUM>, a substrate <NUM> is provided that has a photoresist layer <NUM> deposited thereon. The photoresist layer <NUM> may have a uniform thickness and still be used to form a ML array having MLs with variable height and/or curvatures as described below.

At block <NUM>, a mask <NUM> is disposed adjacent to the photoresist layer <NUM> to mask a select portion of the photoresist layer. The mask may be a suitable opaque material to prevent the select portion from being exposed to some form of light such as, for example, Ultra-Violet light.

At block <NUM>, an unmasked portion of the photoresist layer is exposed to light that is suitable for shaping the photoresist layer based on the mask <NUM>. For example, UV light may be emitted toward the substrate and photoresist layer with the mask blocking some portion of this UV light from reaching the select portion of the photoresist layer.

At block <NUM>, a thermal reflow process is initiated on a remaining portion <NUM> of the photoresist layer. The result of the thermal reflow process is formation of a ML array <NUM> having individual MLs <NUM> which vary in size and height.

The following detailed methodology is provided to facilitate using the thermal reflow process to achieve the ML array <NUM> with variable ML height and size from a photoresist layer <NUM> having a constant thickness. More specifically, based on the simulation results that are achieved for each individual pixel as described in relation to <FIG> and <FIG>, a ML volume can be calculated and converted into a carriable photoresist pitch size at a given photoresist layer thickness condition.

The following methodology is described with respect to a single representative pixel at original location (i, j), while the ML shifted center is located at (i', j') with the variable gapless ML size being, while the ML shifted center is located at (i', j') with the variable gapless ML size A'(i',j') and its photo resist size a(i', j'), a(i', j')<A'(i', j'). Based on these defined parameters, the ML processed by thermal reflow can be approached as square shaped ML with spherical surface, thus the ML volume V(i', j') at the said location (i', j') can be expressed as equation <NUM> below: <MAT> where R(i', j') is the ML curvature at location (i', j'), the variable (x, y) represent ML area integrating variables over the ML pitch size area A'(i', j'), and the terms dx and dy represent the corresponding integration infinitesimals. It will be appreciated that the gapless ML pitch size is generally smaller than the pixel size due to the ML shift, typically the ML pitch size reduction with regards to pixel size is round <NUM>-<NUM>. Thus, it may be acceptable to use the original pixel size (e.g., <NUM>) as a good approximation for the integral.

Based on the foregoing, the radius of curvature to variable ML high H(i', j') at the same location as associate with Eq.<NUM> can be expressed by Equation <NUM> as follows: <MAT> where d05(i', j') is a half of the diagonal dimension of the ML pitch at location (i', j'). Therefore, the photo resist pitch size can determine by the volume balance as defined in Equation <NUM> below: <MAT> Thus, the variable photo resist pitch area size can be expressed as <MAT>.

In embodiments in which the photoresist layer is square shaped, the pitch side dimension is then <MAT>. It should be appreciated that the integrals in Eq.<NUM> - <NUM> can be resolved by numeric method or any other available methods. Furthermore, the ML photoresist pitch size is also radial symmetrical. Based on several characterized values of a(i', j') along radius Image height Ir(i', j'), an interpolation function for pitch size can be defined by Equation <NUM> as follows: <MAT> The interpolated function Func3(H(i, j)) can be a polynomial or any form of the best fitting function representing the variable ML photoresist pitch size as shown in <FIG>. For convenience of pattern design algorithm Eq. <NUM> can be also expressed as <MAT>.

Based on the foregoing equations, the variable ML photoresist size has been defined as function of Image height Ir(i', j'). Assuming the photoresist thickness T=<NUM> for <NUM> pixel process, we have the characterized data given in Table.

An example of characterized data of variable ML height and photoresist pitch size vs. image height is provided in Table <NUM> as follows;.

By using the data in Table <NUM>, an exemplary approximated ML photoresist pitch sizing functions can be defined as follows in Equation <NUM>: <MAT>.

Turning now to <FIG>, illustrated is an exemplary portion of a gapless-type square pitched ML array that can formed with variable height and shift in accordance with the techniques described herein. In particular, the portion of the ML array illustrated in <FIG> includes variable heights and variable shifts that are determined as described above in relation to <FIG> and <FIG>. It should be appreciated that the illustrated surfaces shown in <FIG> represent a spherical lens geometry for individual MLs within the ML array.

Turning now to <FIG>, illustrated is an exemplary process <NUM> for designing a micro lens time-of-flight (ML-TOF) sensor in accordance with the techniques described herein.

At block <NUM>, an optical framework between an image lens and the ML-TOF sensor being designed is determined. An exemplary such optical framework is shown in <FIG> which graphically illustrates a relationship between a ML-TOF sensor <NUM> and an image lens <NUM>. As illustrated, the optical framework <NUM> is usable to determine one or more of a chief ray angle between the image lens and a particular image height, a lower rim ray for the particular image height, and an upper rim ray for the particular image height.

At block <NUM>, the optical framework determined at block <NUM> is used to generate a plurality of bundles of input rays in association with a plurality of image heights on the ML-TOF sensor. Exemplary such bundles of input rays are illustrated in <FIG> and <FIG>. Furthermore, as described above, various optical parameters such as, for example, ML Height, ML shift, ML Pedestal Height, and/or ML curvature can be strategically varied in association with individual pixels to calculate a plurality of different potential volumetric optic power hit rates that could be achieved at the individual pixels.

At block <NUM>, micro lens heights and micro lens shifts are selected for individual image heights of the ML-TOF sensor under design. More specifically, the micro lens heights and micro lens shifts are selected based on the different calculated potential volumetric optic power hit rates that could be achieved at the individual pixels.

In some embodiments, the process <NUM> includes determining other micro lens heights and/or other micro lens shifts based on discrete numbers of micro lens heights and micro lens shifts selected at block <NUM> for a discrete number of image heights. For example, with particular reference to <FIG> and Table <NUM>, the <NUM> micro lens heights and <NUM> micro lens shifts determined for the <NUM> image heights (e.g., tabulated in Table <NUM>) can be used to generate best fit lines.

At block <NUM>, a best fit line that is generated based on a discrete number of micro lens heights is used to determine a plurality of other micro lens heights across the ML-TOF sensor under design. The best fit line may be linear or non-linear.

At block <NUM>, a best fit line that is generated based on a discrete number of micro lens shifts is used to determine a plurality of other micro lens shifts across the ML-TOF sensor under design. The best fit line may be linear or non-linear.

Claim 1:
A three-dimensional time-of-flight image camera (<NUM>), comprising:
a signal generator (<NUM>) configured to generate a modulated electrical signal (<NUM>);
a light emitter (<NUM>) configured to emit modulated light (<NUM>) in accordance with the modulated electrical signal;
an image lens (<NUM>) configured to receive backscattered light (<NUM>) that includes at least a portion of the modulated light;
a pixel array (<NUM>) that includes a plurality of pixels (<NUM>) that are configured to generate photoelectric signals in response to the portion of the modulated light; and
a micro lens array (<NUM>) that includes a plurality of micro lenses (<NUM>) having a plurality of different micro lens heights and widths, wherein individual micro lenses are configured with individual micro lens heights and widths that are inversely related to image heights of the individual micro lenses with respect to an axis (<NUM>) of the image lens