Patent Description:
Traditional imaging hardware involves the projection of complex three-dimensional (3D) scenes onto simplified two-dimensional (2D) planes, forgoing dimensionality inherent in the incident light. This loss of information is a direct result of the nature of square-law detectors, such as charge-coupled devices (CCD) or complementary metal-oxide-semiconductor (CMOS) sensor arrays, which can only directly measure the time-averaged intensity I of the incident light, not its phase, ϕ, or wave vector, k, or angular frequency, ω: <MAT>.

Working within this constraint, plenoptic cameras are forced to recover depth information through either the comparative analysis of multiple simultaneously acquired images, complicated machine learning and/or reconstruction techniques, or the use of active illuminators and sensors. Plenoptic cameras generally describe a scene through the "plenoptic function" which parameterizes a light field impingent on an observer or point by: <MAT> where the x and y coordinates define a certain image plane at time t, for wavelength λ, and polarization angle p, as witnessed by an observer at location (Vx, Vy, Vz). While they may be single- or multi-sensor based systems, current plenoptic cameras can rely, at minimum, solely on the intensity of light detected by any given pixel of a sensor array. More practically, existing solutions, such as stereovision or microlensing, sacrifice overall image quality and sensor footprint by employing multiple sensors or sensor segmentation to accommodate the various fields of view required to discern depth.

Random binary occlusion masks and coded apertures are other existing approaches that provide single-sensor solutions with minimal impact on packaging or overall footprint. However, despite advances in compressed sensing and non-linear reconstruction techniques, these solutions remain hindered by the massive image dictionaries and computational expense involved.

Time-of-flight and structured-light based techniques actively illuminate a scene with pulsed, patterned, or modulated continuous-wave infrared light, and determine depth via the full return-trip travel time or subtle changes in the illuminated light pattern. While these techniques do not suffer from image segmentation, they generally require additional active infrared emitters and detectors which both increase power consumption as well as overall device footprint. Similarly, these techniques tend to be sensitive to interfering signals, specular reflections, and ambient infrared light, thus limiting their viability outdoors.

<CIT>discloses an angle-sensitive pixel (ASP) device that uses the Talbot effect to detect the local intensity and incident angle of light. The ASP device includes a phase grating disposed above a photodiode assembly or a phase grating disposed above an analyzer grating that is disposed above a photodiode assembly. When illuminated by a plane wave, the upper grating generates a self-image at a selected Talbot depth.

<CIT> relates to an array of diffraction-pattern generators employ phase anti-symmetric gratings to projects near-field spatial modulations onto a closely spaced array of photoelements. Each generator in the array of generators produces point-spread functions with spatial frequencies and orientations of interest. The generators are arranged in an irregular mosaic with little or no short-range repetition. Diverse generators are shaped and placed with some irregularity to reduce or eliminate spatially periodic replication of ambiguities to facilitate imaging of nearby scenes.

<CIT> relates to a light-receiving device including a light-trapping sheet, and a photoelectric conversion section optically coupled thereto. The light-trapping sheet includes: a light-transmitting sheet and a plurality of light-coupling structures arranged in an inner portion of the light-transmitting sheet. At least a part of the photoelectric conversion section is located along an outer edge of a surfaces of the light-transmitting sheet.

Challenges therefore remain in the field of light field imaging.

The present description generally relates to light field imaging techniques for depth mapping and other 3D imaging applications.

In accordance with an aspect, there is provided a light field imaging device for capturing light field image data about a scene, the light field imaging device including:.

In some implementations, the diffracted wavefront has an intensity profile along the grating axis, and the pixel array is separated from the diffraction grating by a separation distance at which the intensity profile of the diffracted wavefront has a spatial period that substantially matches the grating period.

In accordance with another aspect, there is provided a backside-illuminated light field imaging device for capturing light field image data about a scene, the backside-illuminated light field imaging device including:.

In accordance with another aspect, there is provided a light field imaging device including:.

In accordance with another aspect that is not claimed individually, there is provided a diffraction grating assembly for use with an image sensor including a pixel array having a plurality of light-sensitive pixels to capture light field image data about a scene, the diffraction grating assembly including a diffraction grating having a grating axis and a refractive index modulation pattern having a grating period along the grating axis, wherein the diffraction grating is a binary phase grating and the refractive index modulation pattern comprises a series of ridges periodically spaced-apart at the grating period, interleaved with a series of grooves periodically spaced-apart at the grating period, the grating period being larger than a pixel pitch of the pixel array along the grating axis, wherein a ratio of the grating period to the pixel pitch along the grating axis is substantially equal to two, the diffraction grating being configured to receive and diffract an optical wavefront originating from the scene to generate a diffracted wavefront for detection by the light-sensitive pixels as the light field image data, the diffraction grating assembly being configured to be disposed over the pixel array. In some implementations, the diffraction grating assembly is configured to be separated from the pixel array by a separation distance at which the diffracted wavefront has an intensity profile along the grating axis with a spatial period that substantially matches the grating period.

In accordance with another aspect, there is provided a method of capturing light field image data about a scene, the method including:
diffracting an optical wavefront originating from the scene with a diffraction grating having a grating period along a grating axis to generate a diffracted wavefront; and
detecting the diffracted wavefront as the light field image data with a pixel array including a plurality of light-sensitive pixels disposed under the diffraction grating, the pixel array having a pixel pitch along the grating axis that is smaller than the grating period.

In accordance with another aspect, there is provided a method of providing three-dimensional imaging capabilities to an image sensor viewing a scene and including a pixel array having a plurality of light-sensitive pixels, the method including:.

In some implementations, disposing the diffraction grating assembly in front of the image sensor includes positioning the diffraction grating assembly at a separation distance from the pixel array at which the diffracted wavefront has an intensity profile along the grating axis with a spatial period that substantially matches the grating period.

In some implementations, the light field imaging device can include an array of light-sensitive elements; an array of color filters overlying and aligned with the array of photosensitive elements such that each color filter covers at least one of the light-sensitive elements, the color filters being spatially arranged according to a mosaic color pattern; and a diffraction grating structure extending over the array of color filters.

In some implementations, the light field imaging device can include a diffraction grating structure exposed to an optical wavefront incident from a scene, the diffraction grating structure diffracting the optical wavefront to produce a diffracted wavefront; an array of color filters spatially arranged according to a mosaic color pattern, the array of color filters extending under the diffraction grating structure and spatio-chromatically filtering the diffracted wavefront according to the mosaic color pattern to produce a filtered wavefront including a plurality of spatially distributed wavefront components; and an array of light-sensitive elements detecting the filtered wavefront as light field image data, the array of light-sensitive elements underlying and being aligned with the array of color filters such that each light-sensitive element detects at least a corresponding one of the spatially distributed wavefront components.

In some implementations, the method can include diffracting an optical wavefront incident from a scene to produce a diffracted wavefront; filtering the diffracted wavefront through an array of color filters spatially arranged according to a mosaic color pattern, thereby obtaining a filtered wavefront including a plurality of spatially distributed wavefront components; and detecting the filtered wavefront as light field image data with an array of light-sensitive elements underlying and aligned with the array of color filters such that each light-sensitive element detects at least part of a corresponding one of the spatially distributed wavefront components.

In some implementations, the method can include diffracting an optical wavefront incident from a scene to produce a diffracted wavefront; spectrally and spatially filtering the diffracted wavefront to produce a filtered wavefront including a plurality of spatially distributed and spectrally filtered wavefront components; and detecting as light field image data the plurality of spatially distributed and spectrally filtered wavefront components at a plurality of arrayed light-sensitive elements.

Other features and advantages of the present description will become more apparent upon reading of the following non-restrictive description of specific embodiments thereof, given by way of example only with reference to the appended drawings.

In the present description, similar features in the drawings have been given similar reference numerals, and, to not unduly encumber the figures, some elements may not be indicated on some figures if they were already identified in a preceding figure. It should also be understood that the elements of the drawings are not necessarily depicted to scale, since emphasis is placed upon clearly illustrating the elements and structures of the present embodiments.

In the present description, and unless stated otherwise, the terms "connected", "coupled" and variants and derivatives thereof refer to any connection or coupling, either direct or indirect, between two or more elements. The connection or coupling between the elements may be mechanical, optical, electrical, operational or a combination thereof. It will also be appreciated that positional descriptors and other like terms indicating the position or orientation of one element with respect to another element are used herein for ease and clarity of description and should, unless otherwise indicated, be taken in the context of the figures and should not be considered limiting. It will be understood that such spatially relative terms are intended to encompass different orientations in use or operation of the present embodiments, in addition to the orientations exemplified in the figures. More particularly, it is to be noted that in the present description, the terms "over" and "under" in specifying the relative spatial relationship of two elements denote that the two elements can be either in direct contact with each other or separated from each other by one or more intervening elements.

In the present description, the terms "a", "an" and "one" are defined to mean "at least one", that is, these terms do not exclude a plural number of items, unless specifically stated otherwise.

The present description generally relates to light field imaging techniques for acquiring light field information or image data about an optical wavefront emanating from a scene. In accordance with various aspects, the present description relates to a light field imaging device for capturing light field image data about a scene, for example a backside-illuminated light field imaging device; a diffraction grating assembly for use with an image sensor to obtain light field image data about a scene; a method of capturing light field image data about a scene; and a method of providing three-dimensional (3D) imaging capabilities to an image sensor array viewing a scene.

In some implementations, the present techniques enable the specific manipulation and comparison of the chromatic dependence of diffraction by means of one or many diffractive optical elements paired with an appropriate chromatic encoding mechanism, as well as its use in 3D imaging. In some implementations, the light field imaging devices and methods disclosed herein are sensitive to not only the intensity and angle of incidence of an optical wavefront originating from an observable scene, but also the wavelength, through a specific spatio-spectral subsampling of a generated interference pattern allowing for direct measurement of the chromatic dependence of diffraction. Light field information or image data, can include information about not only the intensity of the optical wavefront emanating from an observable scene, but also other light field parameters including, without limitation, the angle of incidence, the phase, the wavelength and the polarization of the optical wavefront. Therefore, light field imaging devices, for example depth cameras, can acquire more information than traditional cameras, which typically record only light intensity. The image data captured by light field imaging devices can be used or processed in a variety of ways to provide multiple functions including, but not limited to, 3D depth map extraction, 3D surface reconstruction, image refocusing, and the like. Depending on the application, the light field image data of an observable scene can be acquired as one or more still images or as a video stream.

The present techniques can be used in imaging applications that require or can benefit from enhanced depth sensing and other 3D imaging capabilities, for example to allow a user to change the focus, the point of view and/or the depth of field of a captured image of a scene. The present techniques can be applied to or implemented in various types of 3D imaging systems and methods including, without limitation, light field imaging applications using plenoptic descriptions, ranging applications through the comparative analysis of the chromatic dependence of diffraction, and single-sensor single-image depth acquisition applications. Non-exhaustive advantages and benefits of certain implementations of the present techniques can include: compatibility with passive sensing modalities that employ less power to perform their functions; compatibility with single-sensor architectures having reduced footprint; enablement of depth mapping functions while preserving 2D performance; simple and low-cost integration into existing image sensor hardware and manufacturing processes; compatibility with conventional CMOS and CCD image sensors; and elimination of the need for multiple components, such as dual cameras or cameras equipped with active lighting systems for depth detection.

In the present description, the terms "light" and "optical" are used to refer to radiation in any appropriate region of the electromagnetic spectrum. More particularly, the terms "light" and "optical" are not limited to visible light, but can also include invisible regions of the electromagnetic spectrum including, without limitation, the terahertz (THz), infrared (IR) and ultraviolet (UV) spectral bands. In some implementations, the terms "light" and "optical" can encompass electromagnetic radiation having a wavelength ranging from about <NUM> nanometers (nm) in the deep ultraviolet to about <NUM> micrometers (µm) in the terahertz range, for example from about <NUM> at the blue end of the visible spectrum to about <NUM> at telecommunication wavelengths, or between about <NUM> and about <NUM> to match the spectral range of typical red-green-blue (RGB) color filters. Those skilled in the art will understand, however, that these wavelength ranges are provided for illustrative purposes only and that the present techniques may operate beyond this range.

In the present description, the terms "color" and "chromatic", and variants and derivatives thereof, are used not only in their usual context of human perception of visible electromagnetic radiation (e.g., red, green and blue), but also, and more broadly, to describe spectral characteristics (e.g., diffraction, transmission, reflection, dispersion, absorption) over any appropriate region of the electromagnetic spectrum. In this context, and unless otherwise specified, the terms "color" and "chromatic" and their derivatives can be used interchangeably with the term "spectral" and its derivatives.

Referring to <FIG> and <FIG>, there is provided a schematic representation of an exemplary embodiment of a light field imaging device <NUM> for capturing light field or depth image data about an observable scene <NUM>. In the present description, the term "light field imaging device" broadly refers to any image capture device capable of acquiring an image representing a light field or wavefront emanating from a scene and containing information about not only light intensity at the image plane, but also other light field parameters such as, for example, the direction from which light rays enter the device and the spectrum of the light field. It is to be noted that in the present description, the term "light field imaging device" can be used interchangeably with terms such as "light field camera", "light field imager", "light field image capture device", "depth image capture device", "3D image capture device", and the like.

In the illustrated embodiment, the light field imaging device <NUM> includes a diffraction grating assembly or structure <NUM> configured to receive an optical wavefront <NUM> originating from the scene <NUM>. The diffraction grating assembly <NUM> can include at least one diffraction grating <NUM>, each of which having a grating axis <NUM> and a refractive index modulation pattern <NUM> having a grating period <NUM> along the grating axis <NUM>. In <FIG> and <FIG>, the diffraction grating assembly <NUM> includes a single diffraction grating <NUM>, but as described below, in other embodiments the diffraction grating assembly can include more than one diffraction grating. The diffraction grating <NUM> is configured to diffract the incoming optical wavefront <NUM>, thereby generating a diffracted wavefront <NUM>. The diffraction grating <NUM> in <FIG> and <FIG> is used in transmission since the incident optical wavefront <NUM> and the diffracted wavefront <NUM> lie on opposite sides of the diffraction grating <NUM>.

Referring still to <FIG> and <FIG>, the light field imaging device <NUM> also includes a pixel array <NUM> comprising a plurality of light-sensitive pixels <NUM> disposed under the diffraction grating assembly <NUM> and configured to detect the diffracted wavefront <NUM> as the light field image data about the scene <NUM>. In color implementations, the light field imaging device <NUM> can also include a color filter array <NUM> disposed over the pixel array <NUM>. The color filter array <NUM> includes a plurality of color filters <NUM> arranged in a mosaic color pattern, each of which filters incident light by wavelength to capture color information at a respective location in the color filter array <NUM>. The color filter array <NUM> is configured to spatially and spectrally filter the diffracted wavefront <NUM> according to the mosaic color pattern prior to detection of the diffracted wavefront <NUM> by the plurality of light-sensitive pixels <NUM>. Therefore, as mentioned above, by providing a color filter array to perform a direct spatio-chromatic subsampling of the diffracted wavefront generated by the diffraction grating assembly prior to its detection by the pixel array, the light field imaging device can be sensitive to not only the angle and intensity of an incident wavefront of light, but also its spectral content.

It is to be noted that a color filter array need not be provided in some applications, for example for monochrome imaging. It is also to be noted that the wavefront detected by the light-sensitive pixels will be generally referred to as a "diffracted wavefront" in both monochrome and color implementations, although in the latter case, the terms "filtered wavefront" or "filtered diffracted wavefront" may, in some instances, be used to denote the fact that the diffracted wavefront generated by the diffraction grating assembly is both spatially and spectrally filtered by the color filter array prior to detection by the underlying pixel array. It is also to be noted that in some implementations where a color filter array is not provided, it may be envisioned that the diffraction grating could act as a color filter. For example, the diffraction grating could include a grating substrate with a top surface having the refractive index modulation pattern formed thereon, the grating substrate including a spectral filter material or region configured to spectrally filter the diffracted wavefront according to wavelength prior to detection of the diffracted wavefront by the plurality of light-sensitive pixels. For example, the spectral filter material or region could act as one of a red pass filter, a green pass filter and a blue pass filter.

Depending on the application or use, embodiments of the light field imaging device can be implemented using various image sensor architectures and pixel array configurations. For example, in some implementations, the light field imaging device can be implemented simply by adding or coupling a diffraction grating assembly on top of an already existing image sensor including a pixel array and, in color-based applications, a color filter array. For example, the existing image sensor can be a conventional 2D CMOS or CCD imager. However, in other implementations, the light field imaging device can be implemented and integrally packaged as a separate, dedicated and/or custom-designed device incorporating therein all or most of its components (e.g., diffraction grating assembly, pixel array, color filter array).

More detail regarding the structure, configuration and operation of the components introduced in the preceding paragraphs as well as other possible components of the light field imaging device will be described below.

In the embodiment illustrated in <FIG> and <FIG>, the diffraction grating <NUM> includes a grating substrate <NUM> extending over the color filter array <NUM>. The grating substrate <NUM> has a top surface <NUM>, on which is formed the periodic refractive index modulation pattern <NUM>, and a bottom surface <NUM>. The grating substrate <NUM> is made of a material that is transparent, or sufficiently transparent, in the spectral operating range to permit the diffracted wavefront <NUM> to be transmitted therethrough. Non-limiting examples of such material include silicon oxide (SiOx), polymers, colloidal particles, SU-<NUM> photoresist, glasses. For example, in some implementations the diffraction grating <NUM> can be configured to diffract the optical wavefront <NUM> in a waveband ranging from about <NUM> to about <NUM>.

As known in the art, diffraction occurs when a wavefront, whether electromagnetic or otherwise, encounters a physical object or a refractive-index perturbation. The wavefront tends to bend around the edges of the object. Should a wavefront encounter multiple objects, whether periodic or otherwise, the corresponding wavelets may interfere some distance away from the initial encounter as demonstrated by Young's double slit experiment. This interference creates a distinct pattern, referred to as a "diffraction pattern" or "interference pattern", as a function of distance from the original encounter, which is sensitive to the incidence angle and the spectral content of the wavefront, and the general size, shape, and relative spatial relationships of the encountered objects. This interference can be described through the evolving relative front of each corresponding wavelet, as described by the Huygens-Fresnel principle.

In the present description, the term "diffraction grating", or simply "grating", generally refers to a periodic structure having periodically modulated optical properties (e.g., a refractive index modulation pattern) that spatially modulates the amplitude and/or the phase of an optical wavefront incident upon it. A diffraction grating can include a periodic arrangement of diffracting elements (e.g., alternating ridges and grooves) whose spatial period - the grating period - is nearly equal to or slightly longer than the wavelength of light. An optical wavefront containing a range of wavelengths incident on a diffraction grating will, upon diffraction, have its amplitude and/or phase modified, and, as a result, a space- and time-dependent diffracted wavefront is produced. In general, a diffracting grating is spectrally dispersive so that each wavelength of an input optical wavefront will be outputted along a different direction. However, diffraction gratings exhibiting a substantially achromatic response over an operating spectral range exist and can be used in some implementations. For example, in some implementations, the diffraction grating can be achromatic in the spectral range of interest and be designed for the center wavelength of the spectral range of interest. More particularly, in the case of a Bayer patterned color filter array, the diffraction grating can be optimized for the green channel, that is, around a center wavelength of about <NUM>. It is to be noted that when the diffraction grating is achromatic, it is the mosaic color patter of the color filter array that provides the chromatic sub-sampling of the diffraction pattern of the diffracted wavefront.

Depending on whether the diffracting elements forming the diffraction grating are transmitting or reflective, the diffraction grating will be referred to as a "transmission grating" or a "reflection grating". In the embodiments disclosed in the present description, the diffracting gratings are transmission gratings, although the use of reflection gratings is not excluded a priori. Diffraction gratings can also be classified as "amplitude gratings" or "phase gratings", depending on the nature of diffracting elements. In amplitude gratings, the perturbations to the initial wavefront caused by the grating are the result of a direct amplitude modulation, while in phase gratings, these perturbations are the result of a specific modulation of the relative group-velocity of light caused by a periodic variation of the refractive index of the grating material. In the embodiments disclosed in the present description, the diffracting gratings are phase gratings, although amplitude gratings can be used in other embodiments.

In the illustrated embodiment of <FIG> and <FIG>, the diffraction grating <NUM> is a phase grating, more specifically a binary phase grating for which the refractive index modulation pattern <NUM> includes a series of ridges <NUM> periodically spaced-apart at the grating period <NUM>, interleaved with a series of grooves <NUM> also periodically spaced-apart at the grating period <NUM>. The spatial profile of the refractive index modulation pattern <NUM> thus exhibits a two-level step function, or square-wave function, for which the grating period <NUM> corresponds to the sum of the width, along the grating axis <NUM>, of one ridge <NUM> and one adjacent groove <NUM>. In some implementations, the grating period <NUM> can range from about <NUM> to about <NUM>, although other values are possible in other implementations. In the illustrated embodiment of <FIG> and <FIG>, the grooves <NUM> are empty (i.e., they are filled with air), but they could alternatively be filled with a material having a refractive index different from that of the ridge material. Also, depending on the application, the diffraction grating <NUM> can have a duty cycle substantially equal to or different from <NUM>%, the duty cycle being defined as the ratio of the ridge width to the grating period <NUM>. Another parameter of the diffraction grating <NUM> is the step height <NUM>, that is, the difference in level between the ridges <NUM> and the grooves <NUM>. For example, in some implementations the step height <NUM> can range from about <NUM> to about <NUM>. It is to be noted that in some implementations, the step height <NUM> can be selected so that the diffraction grating <NUM> causes a predetermined optical path difference between adjacent ridges <NUM> and grooves <NUM>. For example, the step height <NUM> can be controlled to provide, at a given wavelength and angle of incidence of the optical wavefront (e.g. its center wavelength), a half-wave optical path difference between the ridges and the grooves. Of course, other optical path difference values can be used in other implementations.

It is to be noted that while the diffraction grating <NUM> in the embodiment of <FIG> and <FIG> is a linear, or one-dimensional, binary phase grating consisting of alternating sets of parallel ridges <NUM> and grooves <NUM> forming a square-wave refractive index modulation pattern <NUM>, other embodiments can employ different types of diffraction gratings. For example, other implementations can use diffraction gratings where at least one among the grating period, the duty cycle and the step height is variable; diffraction gratings with non-straight features perpendicular to the grating axis; diffraction gratings having more elaborate refractive index profiles; 2D diffraction gratings; and the like. It will be understood that the properties of the diffracted wavefront can be tailored by proper selection of the grating parameters. More detail regarding the operation of the diffraction grating and its positioning relative and optical coupling to the other components of the light field imaging device will be described further below.

Referring still to <FIG> and <FIG>, as mentioned above, the pixel array <NUM> includes a plurality of light-sensitive pixels <NUM> disposed under the color filter array <NUM>, which is itself disposed under the diffraction grating assembly <NUM>. The term "pixel array" refers generally to a sensor array made up of a plurality of photosensors, referred to herein as "light-sensitive pixels" or simply "pixels", which are configured to detect electromagnetic radiation incident thereonto from an observable scene and to generate an image of the scene, typically by converting the detected radiation into electrical data. In the present techniques, the electromagnetic radiation that is detected by the light-sensitive pixels <NUM> as light field image data corresponds to an optical wavefront <NUM> incident from the scene <NUM>, which has been diffracted and, optionally, spatio-chromatically filtered, prior to reaching the pixel array <NUM>. The pixel array <NUM> can be embodied by a CMOS or a CCD image sensor, but other types of photodetector arrays (e.g., charge injection devices or photodiode arrays) could alternatively be used. As mentioned above, the pixel array <NUM> can be configured to detect electromagnetic radiation in any appropriate region of the spectrum. Depending on the application, the pixel array <NUM> may be configured according to either a rolling or global shutter readout design. The pixel array <NUM> may further be part of a stacked, backside, or frontside illumination sensor architecture, as described in greater detail below. The pixel array <NUM> may be of any standard or non-standard optical format, for example, but not limited to, <NUM>/<NUM>", <NUM>", <NUM>/<NUM>", <NUM>/<NUM>", <NUM>/<NUM>", <NUM>", <NUM>/<NUM>", <NUM>/<NUM>", <NUM>/<NUM>", <NUM>, and the like. The pixel array <NUM> may also include either a contrast or a phase-detection autofocus mechanism and their respective pixel architectures. It is to be noted that in the present description, the term "pixel array" can be used interchangeably with terms such as "photodetector array", "photosensor array", "imager array", and the like.

Each light-sensitive pixel <NUM> of the pixel array <NUM> can convert the spatial part of the diffracted wavefront <NUM> incident upon it into accumulated charge, the amount of which is proportional to the amount of light collected and registered by the pixel <NUM>. Each light-sensitive pixel <NUM> can include a light-sensitive surface and associated pixel circuitry for processing signals at the pixel level and communicating with other electronics, such as a readout unit. Those skilled in the art will understand that various other components can be integrated into the pixel circuitry of each pixel. In general, the light-sensitive pixels <NUM> can be individually addressed and read out.

Referring still to <FIG> and <FIG>, the light-sensitive pixels <NUM> can be arranged into a rectangular grid of rows and columns defined by two orthogonal pixel axes <NUM>, <NUM>, the number of rows and columns defining the resolution of the pixel array <NUM>. For example, in some implementations, the pixel array <NUM> can have a resolution of at least <NUM> pixels, although a wide range of other resolution values, including up to <NUM> megapixels or more, can be used in other embodiments. It is to be noted that while the light-sensitive pixels <NUM> are organized into a 2D array in the embodiment of <FIG> and <FIG>, they may alternatively be configured as a linear array in other embodiments. It is also to be noted that while the light-sensitive pixels <NUM> are square in the embodiment of <FIG> and <FIG>, corresponding to a pixel aspect ratio of <NUM>:<NUM>, other pixel aspect ratio values can be used in other embodiments.

The pixel array <NUM> can also be characterized by a pixel pitch <NUM>. In the present description, the term "pixel pitch" generally refers to the spacing between the individual pixels <NUM> and is typically defined as the center-to-center distance between adjacent pixels <NUM>. Depending on the physical arrangement of the pixel array <NUM>, the pixel pitch <NUM> along the two orthogonal pixel axes <NUM>, <NUM> may or may not be the same. It is to be noted that a pixel pitch can also be defined along an arbitrary axis, for example along a diagonal axis oriented at <NUM>° with respect to the two orthogonal pixel axes <NUM>, <NUM>. It is also to be noted that, in the present techniques, a relevant pixel pitch <NUM> is the one along the grating axis <NUM> of the overlying diffraction grating <NUM>, as depicted in <FIG> and <FIG>. As described in greater detail below, the grating period <NUM> of the diffraction grating <NUM> is selected to be larger than the pixel pitch <NUM> of the pixel array <NUM> along the grating axis <NUM>. For example, in some implementations the pixel pitch <NUM> along the grating axis <NUM> can range from <NUM> or less to <NUM>, although different pixel pitch values can be used in other implementations.

In the present description, the term "pixel data" refers to the image information captured by each individual pixel and can include intensity data indicative of the total amount of optical energy absorbed by each individual pixel over an integration period. Combining the pixel data from all the pixels <NUM> yields light field image data about the scene <NUM>. In the present techniques, because the optical wavefront <NUM> incident from the scene <NUM> is diffracted and, possibly, spatially and spectrally filtered prior to detection, the light field image data can provide information about not only the intensity of the incident wavefront <NUM>, but also other light field parameters such as its angle of incidence, phase and spectral content. More particularly, it will be understood that the present techniques can allow recovery or extraction of depth or other light field information from the intensity-based diffraction pattern captured by the pixel array <NUM>, as described further below.

Referring still to <FIG> and <FIG>, the color filter array <NUM> is spatially registered with the pixel array <NUM>, such that each color filter <NUM> is optically coupled to a corresponding one of the light-sensitive pixels <NUM>. That is, each color filter <NUM> covers a single light-sensitive pixel <NUM>, such that there is a one-to-one relationship, or mapping, between the color filters <NUM> and the light-sensitive pixels <NUM>. However, in other implementations, each color filter can be optically coupled to at least two corresponding ones of the plurality of light-sensitive pixels. For example, turning briefly to <FIG>, there is shown another embodiment of a light field imaging device <NUM> in which each color filter <NUM> of the color filter array <NUM> overlies a group or subset of light-sensitive pixels <NUM>, namely a <NUM>×<NUM> block of light-sensitive pixels <NUM>. In both the embodiment of <FIG> and <FIG> and the embodiment of <FIG>, the color filter array <NUM> and the pixel array <NUM> together enable the direct spatio-chromatic sampling of the diffracted wavefront produced by the overlying diffraction grating assembly <NUM>, as detailed and explained below.

As mentioned above regarding the terms "color" and "chromatic", terms such as "color filter" and "color filtering" are to be understood as being equivalent to "spectral filter" and "spectral filtering" in any appropriate spectral range of the electromagnetic spectrum, and not only within the visible range. Depending on the application, the color filters can achieve spectral filtering through absorption of unwanted spectral components, for example using dye-based color filters, although other filtering principles may be used without departing from the scope of the present techniques.

Returning to <FIG> and <FIG>, the color filters <NUM> are physically organized according to a mosaic color pattern or configuration. In some implementations, each color filter <NUM> is one of a red pass filter, a green pass filter and a blue pass filter. For example, in the illustrated embodiment, the mosaic color pattern of the color filter array <NUM> is a Bayer pattern, in which the color filters arranged in a checkerboard pattern with rows of alternating red (R) and green (G) filters are interleaved with rows of alternating green (G) and blue (B) filters. As known in the art, a Bayer pattern contains twice as many green filters as red or blue filters, such that the green component of the mosaic color pattern is more densely sampled than red and blue components. In alternative implementations, the mosaic color pattern can be embodied by more elaborate Bayer-type patterns, for example Bayer-type patterns with an n-pixel unit cell, where n is an integer greater than <NUM>. Of course, the present techniques are not limited to Bayer-type patterns, but can be applied to any appropriate mosaic color pattern including, but not limited to, RGB, RGB-IR, RGB-W, CYGM, CYYM, RGBE, RGBW #<NUM>, RGBW #<NUM>, RGBW #<NUM>, and monochrome. It is to be noted that in some implementations, the color filter array <NUM> may be extended beyond the standard visible Bayer pattern to include hyperspectral imaging and filtering techniques or interferometric filtering techniques. In such embodiments, the design of the diffraction grating <NUM> (e.g., the grating period <NUM>) can be adjusted to accommodate the increased spectral sampling range.

Referring now to <FIG> and <FIG>, there is shown another embodiment of a light field imaging device <NUM>, which is suitable for monochrome imaging applications. This embodiment shares many features with the embodiment described above and illustrated in <FIG> and <FIG>, insofar as it generally includes a diffraction grating assembly <NUM> including at least one diffraction grating <NUM> and disposed over a pixel array <NUM> including a plurality of light-sensitive pixels <NUM>. These components can generally be similar in terms of structure and operation to like components of the embodiment of <FIG> and <FIG>. The light field imaging device <NUM> of <FIG> and <FIG> differs from that of <FIG> and <FIG> mainly in that it does not include a color filter array disposed between the diffraction grating assembly <NUM> and the pixel array <NUM>. As a result, the light-sensitive pixels <NUM> directly detect the diffracted wavefront <NUM> transmitted by the diffraction grating <NUM>.

Referring to <FIG>, there is shown another embodiment of a light field imaging device <NUM>, which shares similar features with the embodiment of <FIG> and <FIG>, but differs in that it further includes a microlens array <NUM> disposed over the pixel array <NUM> and including a plurality of microlenses <NUM>. Each microlens <NUM> is optically coupled to a corresponding one of the light-sensitive pixels <NUM> and is configured to focus the spatial part of the diffracted wavefront <NUM> incident upon it onto its corresponding light-sensitive pixel <NUM>. It is to be noted that in embodiments where an array of color filters is provided, such as in <FIG> and <FIG>, the microlens array would be disposed over the color filter array such that each microlens would be optically coupled to a corresponding one of the color filters. In some variants, the light imaging device may also include an anti-reflection coating (not shown) provided over the pixel array <NUM>.

Referring now to <FIG>, there is shown a schematic partially exploded side view of an embodiment of a light field imaging device <NUM> suitable for monochrome imaging applications. The light field imaging device <NUM> shares similarities with the one shown in <FIG> and <FIG>, in that it includes a diffraction grating <NUM> disposed on top of a pixel array <NUM> of light-sensitive pixels <NUM>. The diffraction grating <NUM> is a binary phase transmission grating having a duty cycle of <NUM>% and a periodic refractive index modulation pattern <NUM> consisting of alternating sets of ridges <NUM> and grooves <NUM>. <FIG> also depicts schematically the propagation of light through the device <NUM>. In operation, the light field imaging device <NUM> has a field of view encompassing an observable scene <NUM>. The diffraction grating <NUM> receives an optical wavefront <NUM> (solid line) incident from the scene <NUM> on its input side, and diffracts the optical wavefront <NUM> to generate, on its output side, a diffracted wavefront <NUM> (solid line) that propagates toward the pixel array <NUM> for detection thereby. For simplicity, the incoming optical wavefront <NUM> in <FIG> corresponds to the wavefront of a plane wave impinging on the diffraction grating <NUM> at normal incidence. However, the present techniques can be implemented for an optical wavefront of arbitrary shape incident on the diffraction grating <NUM> at an arbitrary angle within the field of view of the light field imaging device.

Referring still to <FIG>, the diffracted wavefront <NUM> can be characterized by a diffraction pattern whose form is a function of the geometry of the diffraction grating <NUM>, the wavelength and angle of incidence of the optical wavefront <NUM>, and the position of the observation plane, which corresponds to the light-receiving surface <NUM> of the pixel array <NUM>. In the observation plane, the diffraction pattern of the diffracted wavefront <NUM> can be characterized by a spatially varying intensity profile <NUM> along the grating axis <NUM> in the light-receiving surface <NUM> of the pixel array <NUM>. It is to be noted that in <FIG>, the grating axis <NUM> is parallel to the pixel axis <NUM>.

In the present techniques, the diffraction grating <NUM> and the pixel array <NUM> are disposed relative to each other such that the light-receiving surface <NUM> of the pixel array <NUM> is positioned in the near-field diffraction region, or simply the near field, of the diffraction grating <NUM>. In the near-field diffraction regime, the Fresnel diffraction theory can be used to calculate the diffraction pattern of waves passing through a diffraction grating. Unlike the far-field Fraunhofer diffraction theory, Fresnel diffraction accounts for the wavefront curvature, which allows calculation of the relative phase of interfering waves. Similarly, when detecting the diffracted irradiance pattern within a few integer multiples of the wavelength with a photosensor or another imaging device of the same dimensional order as the grating, higher order-diffractive effects tend to be limited simply by spatial sampling. To detect the diffracted wavefront <NUM> in the near field, the present techniques can involve maintaining a sufficiently small separation distance <NUM> between the top surface <NUM> of the diffraction grating <NUM>, where refractive index modulation pattern <NUM> is formed and diffraction occurs, and the light-receiving surface <NUM> of the underlying pixel array <NUM>, where the diffracted wavefront <NUM> is detected. In some implementations, this can involve selecting the separation distance <NUM> to be less than about ten times a center wavelength of the optical wavefront <NUM>. In some implementations, the separation distance <NUM> can range between about <NUM> and about <NUM>, for example between <NUM> and about <NUM> if the center wavelength of the optical wavefront lies in the visible range.

In the near-field diffraction regime, the intensity profile <NUM> of the diffracted wavefront <NUM> produced by a periodic diffraction grating <NUM> generally has a spatial period <NUM> that substantially matches the grating period <NUM> of the diffraction grating <NUM> as well as a shape that substantially matches the refractive index modulation pattern <NUM> of the diffraction grating <NUM>. For example, in the illustrated embodiment, the diffraction pattern of the diffracted wavefront <NUM> detected by the light-sensitive pixels <NUM> of the pixel array <NUM> has a square-wave, or two-step, intensity profile <NUM> that substantially matches that of the refractive index modulation pattern <NUM> of the binary phase diffraction grating <NUM>. In the present description, the term "match" and derivatives thereof should be understood to encompass not only an "exact" or "perfect" match between the intensity profile <NUM> of the detected diffracted wavefront <NUM> and the periodic refractive index modulation pattern <NUM> of the diffraction grating <NUM>, but also a "substantial", "approximate" or "subjective" match. The term "match" is therefore intended to refer herein to a condition in which two features are either the same or within some predetermined tolerance of each other.

Another feature of near-field diffraction by a periodic diffraction grating is that upon varying the angle of incidence <NUM> of the incoming optical wavefront <NUM> on the diffraction grating <NUM>, the intensity profile <NUM> of the diffracted wavefront <NUM> is laterally shifted along the grating axis <NUM>, but substantially retains its period <NUM> and shape, as can be seen from the comparison between solid and dashed wavefront lines in <FIG>. It will be understood that in some implementations, the separation distance between the diffraction grating <NUM> and the pixel array <NUM> can be selected to ensure the spatial shift experienced by the intensity profile <NUM> of the diffracted wavefront <NUM> remains less than the grating period <NUM> as the angle of incidence <NUM> of the optical wavefront <NUM> is varied across the angular span of the field of view of the light field imaging device <NUM>. Otherwise, ambiguity in the angle of incidence <NUM> of the optical wavefront <NUM> can become an issue. For example, consider for illustrative purposes, a light field imaging device <NUM> whose field of view has an angular span of ±<NUM>° and in which varying the angle of incidence <NUM> of the incoming optical wavefront <NUM> by <NUM>° produces a spatial shift of the intensity profile <NUM> of the diffracted wavefront <NUM> equal to the grating period <NUM>. In such a case, light incident on the diffraction grating <NUM> with an incidence angle of, for example, +<NUM>° would be undistinguishable, from phase information alone, from light incidence on the diffraction grating <NUM> with an incidence angle of +<NUM>°.

It is also to be noted that upon being optically coupled to an underlying pixel array <NUM>, the diffraction grating <NUM> convolves lights phase information with a standard 2D image, so that the intensity profile <NUM> of the diffraction pattern of the detected diffracted wavefront <NUM> can generally be written as a modulated function I ~ Imod(depth info)×Ibase(2D image) including a modulating component Imod and a base component Ibase. The base component Ibase represents the non-phase-dependent optical wavefront that would be detected by the pixel array <NUM> if there were no diffraction grating <NUM> in front of it. In other words, detecting the base component Ibase alone would allow a conventional 2D image of the scene <NUM> to be obtained. Meanwhile, the modulating component Imod, which is generally small compared to the base component Ibase (e.g., ratio of Imod to Ibase ranging from about <NUM> to about <NUM>), is a direct result of the phase of the incident optical wavefront <NUM>, so that any edge or slight difference in incidence angle will manifest itself as a periodic electrical response spatially sampled across the pixel array <NUM>. It will be understood that the sensitivity to the angle of incidence <NUM> of the optical wavefront <NUM>, and therefore the angular resolution of the light field imaging device <NUM>, will generally depend on the specific design of the diffraction grating <NUM>.

Referring still to <FIG>, as mentioned above, in the present techniques, the pixel array <NUM> has a pixel pitch <NUM> along the grating axis <NUM> that is smaller than the grating period <NUM> of the diffraction grating <NUM>. This means that when the light-receiving surface <NUM> of the pixel array <NUM> is in the near field of the diffracting grating <NUM>, the pixel pitch <NUM> of the pixel array <NUM> along the grating axis <NUM> is also smaller than the spatial period <NUM> of the intensity profile <NUM> along the grating axis <NUM> of the detected diffracted wavefront <NUM>. It will be understood that when this condition is fulfilled, a complete period of the intensity profile <NUM> of the detected diffracted wavefront <NUM> will be sampled by at least two adjacent pixel banks of the pixel array <NUM>, each of these pixel banks sampling a different spatial part of the intensity profile <NUM> over a full cycle. In the present description, the term "pixel bank" refers to a group of light-sensitive pixels of the pixel array that are arranged along a line which is perpendicular to the grating axis of the overlying diffraction grating. That is, two adjacent pixel banks are separated from each other by a distance corresponding to the pixel pitch along the grating axis. For example, in <FIG>, each pixel bank of the pixel array <NUM> extends parallel to the pixel axis <NUM> oriented perpendicular to the plane of the page.

It will be understood that depending on the application, the ratio R of the grating period <NUM> of the diffraction grating <NUM> to the pixel pitch <NUM> of the pixel array <NUM> along the grating axis <NUM> can take several values. In some implementations, the ratio R can be equal to or greater than two (i.e., R ≥ <NUM>); or equal to a positive integer greater than one (i.e., R = (n + <NUM>), where n = {<NUM>, <NUM>,. }); or equal to an integer power of two (i.e., R = 2n, where n = {<NUM>, <NUM>,. }); or the like. In some implementations, it may be beneficial or required that the grating period <NUM> be not only larger than, but also not too close to the pixel pitch <NUM> along the grating axis <NUM>. For example, in some implementations, it may be advantageous that the grating period <NUM> be at least about twice the underlying pixel bank pitch <NUM> to allow for each pair of adjacent pixel banks to sufficiently subsample the resultant modulated diffracted wavefront <NUM>, whose spatial modulation rate is dictated by the properties of the diffraction grating <NUM>, near or at Nyquist rate. This Nyquist, or nearly Nyquist, subsampling can allow for the direct removal of the modulating component Imod from the measured signal I by standard signal processing techniques. Once removed, the modulating signal Imod may be manipulated independently of the base component Ibase. In some implementations, undersampling effects can arise if the pixel pitch <NUM> along the grating axis <NUM> is not sufficiently smaller than the grating period <NUM>. In such scenarios, it may become useful or even necessary to alter the grating design to provide two different sub-gratings with a sufficient relative phase offset between them to allow for signal subtraction.

For example, in the illustrated embodiment according to the invention of <FIG>, the ratio R of the grating period <NUM> to the pixel pitch <NUM> along the grating axis <NUM> is substantially equal to two. It will be understood that in such a case, adjacent pixel banks will sample complimentary spatial phases of the intensity profile <NUM> of the detected diffracted wavefront <NUM>, that is, spatial parts of the intensity profile <NUM> that are phase-shifted by <NUM>° relative to each other. This can be expressed mathematically as follows: |φbank,n+<NUM> - φbank,n| = π, where ϕbank,n+<NUM> and φbank,n are the spatial phases of the intensity profile <NUM> measured by the (n + <NUM>)th and the nth pixel banks of the pixel array <NUM>, respectively. Such a configuration can allow for a direct deconvolution of the modulating component Imod and the base component Ibase through the subsampling of the interference pattern resulting from the incident wave fronts interaction: <MAT> <MAT>.

Referring still to <FIG>, in the illustrated embodiment, the diffraction grating <NUM> has a duty cycle of <NUM>% (i.e., ridges <NUM> and grooves <NUM> of equal width), and each light-sensitive pixel <NUM> is positioned under and in vertical alignment with either a corresponding one of the ridges <NUM> or a corresponding one of the grooves <NUM>. However, other arrangements can be used in other embodiments, non-limiting examples of which are shown in <FIG>. First, in <FIG>, according to the invention the diffraction grating <NUM> has a duty cycle of <NUM>%, but is laterally shifted by a quarter of the grating period <NUM> compared to the embodiment <FIG>. As a result, each light-sensitive pixel <NUM> is positioned under and in vertical alignment with a transition <NUM> between a corresponding one of the ridges <NUM> and a corresponding adjacent one of the grooves <NUM>. Second, in <FIG>, not falling under the scope of the invention the diffraction grating <NUM> has a duty cycle of <NUM>%, but compared to the embodiment of <FIG>, the ratio R of the grating period <NUM> to the pixel pitch <NUM> along the grating axis <NUM> is equal to four rather than two. There are therefore two light-sensitive pixels <NUM> under each of the ridges <NUM> and each of the grooves <NUM>. Finally, in <FIG>, according to the invention the ratio R of the grating period <NUM> to the pixel pitch <NUM> along the grating axis <NUM> is equal to two, as in <FIG>, but the duty cycle of the diffracting grating is different from <NUM>%.

In some implementations, for example in backside-illuminated architectures with high chief-ray angle optical systems, the diffraction grating may be designed to follow the designed chief-ray-angle offset of the microlens array relative to their light-sensitive pixel so that each corresponding chief ray will pass through the center of the intended grating feature and its subsequent microlens. Such a configuration can ensure appropriate phase offsets for highly constrained optical systems. This means that, in some embodiments, the degree of vertical alignment between the features of the diffraction grating (e.g., ridges and grooves) and the underlying light-sensitive pixels can change as a function of position within the pixel array, for example as one goes from the center to the edge of the pixel array, to accommodate a predetermined chief-ray-angle offset. For example, in some regions of the pixel array, each light-sensitive pixel may be positioned directly under a groove or a ridge of the diffraction grating, while in other regions of the pixel array, each light-sensitive pixel may extend under both a portion of a ridge and a portion of a groove. In the implementations of <FIG> and <FIG>, the diffraction grating <NUM> is oriented with respect to the underlying pixel array <NUM> so that the grating axis <NUM> is parallel to one of the two orthogonal pixel axes <NUM>, <NUM> (and thus perpendicular to each other). However, referring to <FIG> and <FIG>, not falling under the scope of the invention there are illustrated two other possible embodiments in which the grating axis <NUM> is oblique to both the two orthogonal pixel axes <NUM>, <NUM>. This is, in <FIG>, the grating axis <NUM> is oriented at an angle θ = <NUM>° with respect to each one of the pixel axes <NUM>, <NUM>, while in <FIG>, the grating axis is oriented at angle θ ≈ <NUM>° with respect to the pixel axis <NUM>. It is to be noted that in the oblique configurations illustrated in <FIG> and <FIG>, the pixel pitch <NUM> along the grating axis <NUM> remains smaller than the grating period. It is also to be noted that pixel banks such as defined above, that is, groups of pixels arranged along a line transverse to the grating axis <NUM> of the overlying diffraction grating <NUM> can also be defined in oblique configurations. For example, <FIG> includes a first group of pixels <NUM><NUM> that belong to a first pixel bank located under ridge <NUM>, and a second group of pixels <NUM><NUM> that belongs to a second pixel bank located an adjacent groove <NUM>.

Referring now to <FIG>, there is shown a schematic partially exploded side view of an embodiment of a light field imaging device <NUM> suitable for color imaging applications. The light field imaging device <NUM> shares similarities with the one shown in <FIG> and <FIG>, in that it includes a diffraction grating <NUM> disposed on top of a color filter array <NUM>, which is itself disposed on top of a pixel array <NUM> of light-sensitive pixels <NUM>. The diffraction grating <NUM> is a binary phase transmission grating having a duty cycle of <NUM>% and a periodic refractive index modulation pattern <NUM> consisting of alternating sets of ridges <NUM> and grooves <NUM>. The color filter array <NUM> has a Bayer pattern, of which <FIG> depicts a row of alternating green (G) and blue (B) filters. <FIG> also depicts schematically the propagation of light through the device <NUM>. In operation, the diffraction grating <NUM> receives and diffracts an optical wavefront <NUM> originating from the scene <NUM> to generate a diffracted wavefront <NUM>. For simplicity, it is assumed that the diffraction grating <NUM> of <FIG> is achromatic in the spectral range encompassing green and blue light. The color filter array <NUM> receives and spatio-spectrally filters the diffracted wavefront <NUM> prior to its detection by the underlying pixel array <NUM>. The operation of the light field imaging device <NUM> is therefore based on a directly spatio-and-chromatically sampled diffracted wavefront <NUM> enabled by the provision of a periodic diffraction grating <NUM> deposed on top of a sensor structure including a color filter array <NUM> and an underlying pixel array <NUM>.

As in <FIG>, the diffracted wavefront <NUM> produced by the diffraction grating <NUM> in <FIG> defines a diffraction pattern characterized by a spatially varying intensity profile <NUM> along the grating axis <NUM>. Also, the diffraction grating <NUM> and the pixel array <NUM> are disposed relative to each other such that the light-receiving surface <NUM> of the pixel array <NUM> is positioned in the near field of the diffraction grating <NUM>, where the spatial period <NUM> of the intensity profile <NUM> of the detected diffracted wavefront <NUM> substantially matches the grating period <NUM> of the diffraction grating <NUM>.

It will be understood that the intensity profile <NUM> of the diffracted wavefront <NUM> that is detected by the pixel array <NUM> after spatio-spectral filtering by the color filter array <NUM> is a combination or superposition of the portions of the diffracted wavefront <NUM> filtered by the red filters, the portions of the diffracted wavefront <NUM> filtered by the green filters, and the portions of the diffracted wavefront <NUM> filtered by the blue filters. As such, using a standard RGB Bayer pattern as an example, the modulating component Imod and the base component Ibase of the intensity profile I can be split into their respective color components as follows: <MAT> <MAT> <MAT>.

In <FIG>, the intensity profiles IG and IB are depicted in dashed and dotted lines, respectively.

As in <FIG>, the ratio R of the grating period <NUM> of the diffraction grating <NUM> to the pixel pitch <NUM> of the pixel array <NUM> along the grating axis <NUM> is equal to two in the embodiment of <FIG>, and the relationship |φbank,n+<NUM> - φbank,n| = π introduced above applies. In a standard RGB Bayer pattern, the red and blue filters are always located in adjacent pixel banks in a Bayer pattern, the signals IR and IB, which are associated with the sparsely sampled red and blue components, will be in antiphase relative to each other. Meanwhile, because green filters are present in all pixel banks, the signal IG, which is associated with the densely sampled green components, will contain both in-phase and out-of-phase contributions.

In the implementations described so far, the diffraction grating assembly was depicted as including only one diffracting grating. However, referring to <FIG>, in other implementations, the diffraction grating assembly <NUM> includes a plurality of diffracting gratings 28a, 28b, where the diffracting gratings 28a, 28b are arranged in a two-dimensional grating array disposed over the color filter array <NUM>. In <FIG>, the diffracting grating assembly <NUM> includes sixteen diffraction gratings, but this number is provided for illustrative purposes and could be varied in other embodiments. For example, depending on the application, the number of diffraction gratings 28a, 28b in the diffraction grating assembly <NUM> can range from one to up to millions (e.g., a <NUM>-megapixel pixel array <NUM> could have up to <NUM> million diffraction gratings on top of it). It is to be noted that other than their grating axis orientation, every diffraction grating <NUM> of the diffraction grating assembly <NUM> depicted in <FIG> is a binary phase grating including alternating sets of parallel ridges <NUM> and grooves <NUM> having the same duty cycle of <NUM>%, the same grating period <NUM>, and the same number of repetitions of the grating period <NUM>, although in other embodiments each of these parameters can be varied from one diffraction grating <NUM> to the another. More particularly, each one of the diffraction gratings <NUM> in <FIG> includes two repetitions of the grating period <NUM>. However, it will be understood that this number can be varied depending on the application, for example between two and ten repetitions in some embodiments.

In some implementations, the plurality of diffraction gratings <NUM> includes multiple sets 80a, 80b of diffraction gratings <NUM>, where the grating axes 30a, 30b of the diffraction gratings <NUM> of different ones of the sets 80a, 80b have different orientations. For example, in <FIG>, the multiple sets 80a, 80b consist of a first set 80a of diffraction gratings <NUM> and a second set 80b of diffraction gratings <NUM>, the grating axes 30a of the diffraction gratings <NUM> of the first set 80a extending substantially perpendicularly to the grating axes 30b of the diffraction gratings <NUM> of the second set 80b. The first grating axes 30a are parallel to the first pixel axis <NUM>, while the second grating axes 30b are parallel to the second pixel axis <NUM>. In the illustrated embodiment, the diffraction gratings <NUM> of the first set 80a and second set 80b are arranged to alternate in both rows and columns, resulting in a checkerboard pattern. Of course, any other suitable regular or irregular arrangement, pattern or mosaic of orthogonally oriented gratings can be envisioned in other embodiments. For example, the orthogonally oriented gratings could be arranged to alternate only in rows or only in columns or arranged randomly. Furthermore, other embodiments can include more than two sets of diffraction gratings, which may or may not be orthogonal with respect to one another. For example, in some implementations, the diffraction grating assembly can include up to <NUM> different sets of diffraction gratings.

It will be understood that providing a diffraction grating assembly with diffracting gratings having different grating axis orientations can be advantageous or required in some implementations since diffraction occurs along the grating axis of an individual diffraction grating. This means that when only a single grating orientation is present in the diffraction grating assembly, light coming from objects of the scene that extend perpendicularly to this single grating orientation will not be diffracted. In some implementations, providing two sets of orthogonally oriented gratings (e.g., horizontally and vertically oriented gratings) can be sufficient to capture sufficient light field image data about the scene. The concept of using diffraction grating assemblies with two or more grating orientations can be taken to the limit of completely circular diffraction gratings having increasing periodicity radially form the center, which would provide a near perfect Fourier plane imager.

Referring to <FIG>, there are illustrated other examples of grating arrangements in diffraction grating assemblies including a plurality of diffraction gratings. In <FIG>, the diffraction grating assembly <NUM> includes two sets 80a, 80b of orthogonally oriented diffraction gratings <NUM> that alternate only in columns. The grating axis orientation of one set 80a is along one pixel axis <NUM>, and the grating axis orientation of the other set 80b is along the other pixel axis <NUM>. In <FIG>, the diffraction grating assembly <NUM> includes four sets 80a to 80d of diffraction gratings <NUM> whose grating axes 34a to 34d are oriented at <NUM>°, <NUM>°, <NUM>° and <NUM>° with respect to the horizontal pixel axis <NUM>. In <FIG>, the diffraction grating assembly <NUM> includes four sets 80a to 80d of diffraction gratings <NUM> whose grating axes 34a to 34d are oriented at <NUM>°, <NUM>°, <NUM>° and -<NUM>° with respect to the horizontal pixel axis <NUM>. It will be understood that in each of <FIG>, the depicted diffraction gratings <NUM> can represent a unit cell of the diffraction grating assembly <NUM>, which is repeated a plurality of times.

Referring now to <FIG>, there is shown an embodiment of a light field imaging device <NUM> that is suitable for color-based applications, but does not include a color filter array disposed between the diffraction grating assembly <NUM> and underlying pixel array <NUM>. Rather, in the illustrated embodiment, the diffraction grating assembly <NUM> includes an array of diffraction gratings <NUM>, each of which includes a grating substrate <NUM> having a refractive index modulation pattern <NUM> formed thereon (e.g., made of alternating series of ridges <NUM> and grooves <NUM>). The grating substrate <NUM> of each diffraction grating <NUM> also includes a spectral filter material or region <NUM> configured to spectrally filter the diffracted wavefront <NUM> prior to its detection by the plurality of light-sensitive pixels <NUM>. In some implementations, each one of the diffraction grating <NUM> can be made of a material tailored to filter a desired spectral component, for example by incorporating a suitable dye dopant in the grating substrate <NUM>.

Referring still to <FIG>, the plurality of diffraction gratings <NUM> of the diffraction grating assembly <NUM> thus forms a color filter array in which each color filter is embodied by a corresponding one of the diffraction gratings <NUM>. In other words, each one of the diffraction gratings <NUM> can be individually designed and tailored so that it forms to its own respective color filter in the color filter array. In <FIG>, the color filter array formed by the plurality of diffraction gratings <NUM> is arranged in a Bayer pattern, so that the grating substrate <NUM> of each diffraction grating <NUM> acts as a red pass filter, a green pass filter or a blue pass filter. Of course, the color filter array defined by the plurality of diffraction gratings <NUM> can be operated outside the visible region of the electromagnetic spectrum and its mosaic color pattern is not limited to Bayer-type patterns, but can be applied to any appropriate mosaic color pattern, including those listed above.

In some implementations, the light field imaging device can include wavefront conditioning optics in front of the diffraction grating. The wavefront conditioning optics can be configured to collect, direct, transmit, reflect, refract, disperse, diffract, collimate, focus or otherwise act on the optical wavefront incident from the scene prior to it reaching the diffraction grating assembly. The wavefront conditioning optics can include lenses, mirrors, filters, optical fibers, and any other suitable reflective, refractive and/or diffractive optical components, and the like. In some implementations, the wavefront conditioning optics can include focusing optics positioned and configured to modify the incident wavefront in such a manner that it may be sampled by the light field imaging device.

Referring now to <FIG>, another possible embodiment of a light field imaging device <NUM> is illustrated and includes dispersive optics <NUM> disposed in a light path of the optical wavefront <NUM> between the scene and the diffraction grating assembly. The dispersive optics <NUM> is configured to receive and disperse the incoming optical wavefront <NUM>. The dispersive optics <NUM> can be embodied by any optical component or combination of optical components in which electromagnetic beams are subject to spatial spreading as a function of wavelength as they pass therethrough (e.g., by chromatic aberration). In the embodiment of <FIG>, the dispersive optics <NUM> is a focusing lens, for simplicity. However, it will be understood that, in other embodiments, the dispersive optics <NUM> can be provided as an optical stack including a larger number of optical components (e.g., focusing and defocusing optics) that together act to disperse the optical wavefront <NUM> before it impinges on the diffraction grating assembly <NUM> (e.g., due to their intrinsic chromatic aberration).

For exemplary purposes, it is assumed in <FIG> that the optical wavefront <NUM> originating from the scene <NUM> is a superposition of waves containing multiple wavelengths of light, for example a green component (dashed line) and a blue component (dotted line). Each color components of the optical wavefront <NUM>, by the nature of its energy-dependent interaction with the dispersive optics <NUM>, will follow a slightly different optical path, leading to a chromatic dependence in the phase-shift introduced by the diffraction grating <NUM>. In other words, the chromatic spread of the optical wavefront <NUM>, as sampled through the angle-dependent diffraction produced by the diffractive grating <NUM>, can provide coarse depth information about the optical wavefront <NUM>. In such scenarios, the finer details of the depth information can be obtained from a comparative analysis of the modulating components Imod,R and Imod,B, which are phase-shifted relative to each other due to their optical path differences, as sampled by the color filter array <NUM>.

It is to be noted that in the case of monochromatic plane optical wavefront impinging on a focusing lens such as shown in <FIG>, the focusing lens gradually refracts and focuses the wavefront as it traverses the lens. It will be understood that the cross-sectional area of the wavefront reaching the diffraction grating assembly will be larger if the diffraction grating assembly is located out (either before or after) of the focal plane of the focusing lens that if it is located in the focal plane. Accordingly, the diffracted wavefront will be sampled by a greater number of light-sensitive pixels in the out-of-focus than in the in-focus configuration.

Referring to <FIG> and <FIG>, in some implementations, the light field imaging device <NUM> can include pixel array circuitry <NUM> disposed either between the diffraction grating assembly and the pixel array, in a frontside illumination configuration (<FIG>), or under the pixel array <NUM>, in a backside illumination configuration (<FIG>). More particularly, the diffraction grating assembly <NUM> can be directly etched into overlying silicon layers in the case of a frontside illumination architecture (<FIG>), or placed directly atop the microlens array <NUM> and the color filter array <NUM> in the case of a backside illumination architecture (<FIG>). In frontside illumination technology, the pixel array circuitry <NUM> includes an array of metal wiring (e.g., a silicon layer hosting a plurality of metal interconnect layers) connecting the color filters <NUM> to their corresponding light-sensitive pixels <NUM>. Meanwhile, backside illumination technology provides opportunities for directly sampling the diffracted wavefront <NUM> produced by diffraction of an optical waveform <NUM> by the diffraction grating assembly <NUM>. As light does not have to pass through the array of metal wiring of the pixel array circuitry <NUM> before reaching the pixel array <NUM>, which otherwise would result in a loss of light, more aggressive diffraction grating designs with increased periodicity can be implemented. Also, the shorter optical stack configuration, as shown in <FIG>, can allow for the diffraction grating assembly <NUM> to be positioned in much closer proximity to the light-receiving surface <NUM> of the pixel array <NUM>, thereby decreasing the risk of higher-order diffractive effects which could cause undesirable cross-talk between pixel banks. Similarly, the decreased pixel size can allow for direct subsampling of the diffraction grating by the existing imaging wells.

Referring now more specifically to <FIG>, there is shown a backside-illuminated light field imaging device <NUM> for capturing light field image data about a scene <NUM>. The device <NUM> includes a substrate <NUM> having a front surface <NUM> and a back surface <NUM>; a diffraction grating assembly <NUM> disposed over the back surface <NUM> of the substrate <NUM> and configured to receive an optical wavefront <NUM> originating from the scene <NUM>; a pixel array <NUM> formed in the substrate <NUM>; and pixel array circuitry <NUM> disposed under the front surface <NUM> and coupled to the pixel array <NUM>. The diffraction grating assembly <NUM> includes at least one diffraction grating <NUM> having a grating axis <NUM> and a refractive index modulation pattern <NUM> having a grating period <NUM> along the grating axis <NUM>. The diffraction grating <NUM> diffracts the optical wavefront <NUM> to generate a diffracted wavefront <NUM>. The pixel array <NUM> includes a plurality of light-sensitive pixels <NUM> configured to receive, through the back surface <NUM>, and detect, as the light field image data, the diffracted wavefront <NUM>. As mentioned above, the pixel array <NUM> has a pixel pitch <NUM> along the grating axis <NUM> that is smaller than the grating period <NUM>. As mentioned above, an advantage of backside illumination sensor technology in the context of the present techniques is that the diffraction grating assembly <NUM> can be positioned closer to the light-receiving surface <NUM> of the pixel array <NUM> than in frontside illumination applications. For example, in some backside illumination implementations, a separation distance <NUM> between the refractive index modulation pattern <NUM> of the diffraction grating <NUM> and the light-receiving surface <NUM> of the pixel array <NUM> can range from about <NUM> to about <NUM>, for example between <NUM> and <NUM>.

In color imaging applications, the backside-illuminated light field imaging device <NUM> can include a color filter array <NUM> disposed over the back surface <NUM> and including a plurality of color filters <NUM> arranged in a mosaic color pattern, for example a Bayer pattern. The color filter array <NUM> spatially and spectrally filters the diffracted wavefront <NUM> according to the mosaic color pattern prior to its detection by the plurality of light-sensitive pixels <NUM>. The device <NUM> also includes a microlens array <NUM> disposed over the color filter array <NUM> and including a plurality of microlenses <NUM>, each of which is optically coupled to a corresponding one of the plurality of the color filters <NUM>. In <FIG>, the diffraction grating <NUM> also includes a grating substrate <NUM> including a top surface <NUM> having the refractive index modulation pattern <NUM> formed thereon and a bottom surface <NUM> disposed over the microlens array <NUM>. It is to be noted that the diffraction grating assembly <NUM>, the pixel array <NUM>, the color filter array <NUM> and the microlens array <NUM> of the backside-illuminated light field imaging device <NUM> can share similar features to those described above.

It is to be noted that backside illuminated and stacked-architecture devices are often employed in situations where sensor footprint is an issue (e.g., smartphone modules, tablets, webcams) and are becoming increasingly complex in design. In some implementations, the present techniques involve positioning a diffraction grating assembly directly on top of an existing sensor architecture as an independent process. Therefore, using the present techniques with backside illumination sensor technology can represent a flexible opportunity for sensor-level depth sensing optics, as it does not require a complete sensor or packaging redesign as is the case for microlens or coded aperture approaches. Furthermore, the modest z-stack increase of the order of micrometers resulting from the integration of the diffraction grating assembly on top of the sensor can similarly simplify packaging requirements and implementation in the overall optical stack of the sensor module. Additionally, the backside illumination manufacturing process itself does not require a direct etch into existing silicon layers as would be the case in frontside illumination technology. It is to be noted that for backside-illuminated devices with larger pixel pitch values and certain frontside illuminated devices, the diffraction grating assembly itself can act as a color filter array (see, e.g., <FIG>), which can reduce the manufacturing complexity and/or the overall height of the optical stack. It is also to be noted that the different layers of the light field imaging device may be stacked and spaced-apart according to geometrical parameters supporting the desired optical functionalities.

Referring to <FIG>, in accordance with another aspect, the present description also relates to a diffraction grating assembly <NUM> for use with an image sensor <NUM> including a pixel array <NUM> having a plurality of light-sensitive pixels <NUM> to capture light field image data about a scene <NUM>. The diffraction grating assembly <NUM>, which is configured to be disposed over the pixel array <NUM>, can share many similarities with those described above in the context of light field imaging device implementations, insofar as it includes a diffraction grating <NUM> having a grating axis <NUM> and a refractive index modulation pattern <NUM> having a grating period <NUM> along the grating axis <NUM>, the grating period <NUM> being larger than a pixel pitch <NUM> of the pixel array <NUM> along the grating axis <NUM>. For example, a ratio of the grating period <NUM> to the pixel pitch <NUM> along the grating axis <NUM> can be equal to two or an integer multiple of two. In some implementations, the diffraction grating <NUM> can be a binary phase grating and the refractive index modulation pattern <NUM> can include alternating ridges <NUM> and grooves <NUM>. The diffraction grating <NUM> is configured to receive and diffract an optical wavefront <NUM> originating from the scene <NUM> to generate a diffracted wavefront <NUM> for detection by the light-sensitive pixels <NUM> as the light field image data. In some implementations intended for color imaging applications, the diffraction grating assembly <NUM> is configured to be disposed over a color filter array <NUM> of the image sensor <NUM>. The color filter array <NUM> is disposed over pixel array <NUM> and configured to spatially and spectrally filter the diffracted wavefront <NUM> prior to its detection by the plurality of light-sensitive pixels <NUM>.

Depending on the application, the diffraction grating assembly <NUM> can include a single diffraction grating <NUM> or a plurality of diffraction gratings <NUM> arranged in a two-dimensional grating array disposed over the pixel array <NUM>.

In accordance with another aspect, the present description also relates to various light field imaging methods, including a method of capturing light field image data about a scene as well as a method of providing 3D imaging capabilities to a conventional 2D image sensor. These methods can be performed with light field imaging devices and diffraction grating assemblies such as those described above, or with other similar devices and assemblies.

Referring to <FIG>, there is provided a flow diagram of an embodiment of a method <NUM> of capturing light field image data about a scene. The method includes a step <NUM> of diffracting an optical wavefront originating from the scene with a diffraction grating. The diffraction grating has a grating axis and a grating period along the grating axis. The diffraction grating is configured to diffract the incident optical wavefront to generate a diffracted wavefront. The diffracted wavefront can be characterized by an intensity profile along the grating axis. In some implementations, the diffracting step <NUM> can include diffracting the optical wavefront in a waveband ranging from <NUM> (blue end of visible spectrum) to <NUM> (telecommunication wavelengths), for example from <NUM> to <NUM>. In some implementations, the diffraction grating is one of a plurality of diffraction gratings that together form a diffraction grating assembly. In such implementations, the method <NUM> of <FIG> can be performed simultaneously for each diffraction grating of the diffraction grating assembly.

In some implementations, the method <NUM> can include a step of providing the diffraction grating as a phase grating, for example a binary phase grating. The binary phase grating can include alternating ridges and grooves periodically spaced-apart at the grating period. The method <NUM> can include a step of selecting the grating period in a range between <NUM> to <NUM>. The method <NUM> can also include a step of setting a step height of the ridges relative to the grooves to control an optical path difference between adjacent ridges and grooves. For example, in some implementations, the step height can be set to provide, at a given wavelength of the optical wavefront, a half-wave optical path difference between the ridges and the grooves. Of course, other values of optical path difference can be used in other implementations.

Referring still to <FIG>, the method <NUM> also includes a step <NUM> of spatio-spectrally filtering the diffracted wavefront with a color filter array to produce a filtered wavefront. It is to be noted that this step <NUM> is optional and can be omitted in some implementations, for example in monochrome imaging applications.

The method <NUM> can further include a step <NUM> of detecting the spatio-spectrally filtered wavefront as the light field image data. The detecting step <NUM> can be performed with a pixel array comprising a plurality of light-sensitive pixels disposed under the color filter array. However, when the spatio-spectral filtering step <NUM> is omitted, there is no color filter array disposed between the diffraction grating assembly and the pixel array, and the detecting step <NUM> involves the direct detection of the diffracted wavefront with the plurality of light-sensitive pixels. As mentioned above with respect to device implementations, the grating period of the diffraction grating is selected to be larger than the pixel pitch of the pixel array along the grating axis. As also mentioned above, the separation distance between the top surface of the diffraction grating (i.e., the refractive index modulation pattern) and the light-receiving surface of the underlying pixel array is selected so that the filtered or diffracted wavefront is detected in a near-field diffraction regime, where the intensity profile of the diffracted wavefront along the grating axis has a spatial period that substantially matches the grating period. For example, in some implementations, the method can include a step of setting the separation distance to a value that is less than about ten times a center wavelength of the optical wavefront to detect the filtered or diffracted wavefront in the near field.

In some implementations, the diffraction grating can be provided with a duty cycle of about <NUM>%, and the method <NUM> can include a step of positioning each light-sensitive pixel under and in alignment with either a ridge or a groove of the diffraction grating, or under and in alignment with a transition or boundary between a ridge and an adjacent groove. In some implementations, the method <NUM> can include a step of setting a ratio of the grating period to the pixel pitch along the grating axis to be substantially equal to two or an integer multiple of two.

Referring still to <FIG>, in some implementations, the plurality of light-sensitive pixels can be arranged in a rectangular pixel grid defined by two orthogonal pixel axes, and the method <NUM> can include a step of orienting the grating axis either parallel to one of the two orthogonal pixel axes or oblique to both the two orthogonal pixel axes. For example, in some orthogonal implementations, one half of the diffraction gratings can be oriented along one pixel axis, and the other half can be oriented along the other pixel axis. One possible oblique configuration can include orienting the diffraction gratings at an angle of <NUM>° with respect to each pixel axis.

In some implementations, the method <NUM> can further include an optional step of spectrally dispersing the optical wavefront prior to diffracting the optical wavefront.

Referring now to <FIG>, there is provided a flow diagram of a method <NUM> of providing 3D imaging capabilities, for example depth mapping capabilities, to an image sensor viewing a scene and including a pixel array having a plurality of light-sensitive pixels. For example, the image sensor can be a conventional or custom-designed frontside- or backside illuminated CMOS or CCD sensor.

The method <NUM> includes a step <NUM> disposing a diffraction grating assembly in front of the image sensor. The diffraction grating assembly includes at least one diffraction grating, each of which having a grating axis and a grating period along the grating axis. The grating period is selected to be larger than a pixel pitch of the pixel array along the grating axis. For example, in some implementations, the grating period can be larger than the pixel pitch along the grating axis by a factor of two or more. In some implementations, the disposing step <NUM> can include positioning the diffraction grating assembly at a separation distance from the pixel array which is selected such that an optical path length of the diffracted wavefront prior to detection by the light-sensitive pixels is less than about ten times a center wavelength of the optical wavefront. Such a configuration allows detection of the diffracted wavefront in a near-field diffraction regime. In some implementations, the disposing step <NUM> can include orienting the grating axis either parallel to one of two orthogonal pixel axes of the pixel array or oblique (e.g., at <NUM>°) to the pixel axes.

In some implementations, the method <NUM> can include a step of providing the diffraction grating as a phase grating, for example a binary phase grating. The binary phase grating can include a series of ridges periodically spaced-apart at the grating period, interleaved with a series of grooves also periodically spaced-apart at the grating period. The method <NUM> can include a step of selecting the grating period between <NUM> to <NUM>. The method <NUM> can also include a step of setting a step height of the ridges relative to the grooves to control an optical path difference between adjacent ridges and grooves. As mentioned above, the step height can be selected to provide a predetermined optical path difference between the ridges and the grooves. In some implementations, the diffraction grating can be provided with a duty cycle of about <NUM>% and the diffraction grating assembly can be positioned over the pixel array such that each ridge and each groove extends over and in alignment with a corresponding one of the light-sensitive pixels, or alternatively such that each transition or junction between adjacent ridges and grooves extends over and in alignment with a corresponding one of the light-sensitive pixels.

Referring still to <FIG>, the method <NUM> also includes a step <NUM> of receiving and diffracting an optical wavefront originating from the scene with the diffraction grating to generate a diffracted wavefront, and a step <NUM> of detecting the diffracted wavefront with the light-sensitive pixels. In color imaging applications, the method <NUM> can include an optional step <NUM> of spatio-spectrally filtering the diffracted wavefront with a color filter array prior to the detecting step <NUM>. In some implementations, the method <NUM> can further include an optional step of spectrally dispersing the optical wavefront prior to diffracting the optical wavefront.

Claim 1:
A light field imaging device (<NUM>) for capturing light field image data about a scene (<NUM>), the light field imaging device (<NUM>) comprising:
a diffraction grating assembly (<NUM>) configured to receive an optical wavefront (<NUM>) originating from the scene (<NUM>), the diffraction grating assembly (<NUM>) comprising a diffraction grating (<NUM>) having a grating axis (<NUM>) and a refractive index modulation pattern (<NUM>) having a grating period (<NUM>) along the grating axis (<NUM>), the diffraction grating (<NUM>) being configured to diffract the optical wavefront (<NUM>) to generate a diffracted wavefront (<NUM>), wherein the diffraction grating (<NUM>) is a binary phase grating and the refractive index modulation pattern (<NUM>) comprises a series of ridges (<NUM>) periodically spaced-apart at the grating period (<NUM>), interleaved with a series of grooves (<NUM>) periodically spaced-apart at the grating period (<NUM>); and
a pixel array (<NUM>) comprising a plurality of light-sensitive pixels (<NUM>) disposed under the diffraction grating (<NUM>) assembly (<NUM>) and configured to detect the diffracted wavefront (<NUM>) as the light field image data, the pixel array (<NUM>) having a pixel pitch (<NUM>) along the grating axis (<NUM>) that is smaller than the grating period (<NUM>), wherein a ratio of the grating period (<NUM>) to the pixel pitch (<NUM>) along the grating axis (<NUM>) is substantially equal to two.