Imaging device and method

The imaging device includes: a modulator configured to modulate a light intensity in accordance with a real pattern; an image sensor configured to create a sensor image in accordance with the modulated light; and a micro lens array including a plurality of micro lenses arranged to correspond to a plurality of pixels of the image sensor. The imaging device has a distribution property of a relative positional difference amount between a center position of a light receiver of each pixel of the plurality of pixels and a center position of each micro lens of the plurality of micro lenses of the micro lens array in a plane of the image sensor. This property has at least one point or more with a changing difference value of the difference amount between the adjacent pixels from a positive value to a negative value or from a negative value to a positive value.

TECHNICAL FIELD

The present invention relates to a technique of an imaging device.

BACKGROUND ART

As a related art example of the imaging device, International Patent Publication No. WO/2017/149687 (Patent Document 1) is exemplified. The Patent Document 1 (Abstract) describes that “an imaging device achieving high functionality is provided by easiness in detection of an incident angle of light ray penetrating a grid substrate. This is achieved by an imaging device including: an image sensor configured to convert and output an optical image taken in a plurality of array-formed pixels on an imaging plane into an image signal; a modulator arranged in a light receiving plane of the image sensor and configured to modulate an optical intensity; an image storage configured to temporarily store the image signal output of the image sensor; and a signal processer configured to perform an image process to the image signal output of the image storage, the modulator has a first grid pattern made of a plurality of concentric circles, and the signal processer creates a moire stripe image by modulating the image signal out of the image storage to be a virtual second grid pattern made of a plurality of concentric circles, and changes a size of the concentric circles of the second grid pattern in accordance with a focus position.”

RELATED ART DOCUMENT

Patent Document

Patent Document 1: International Patent Publication No. WO/2017/149687

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

As described in the related art example, a lensless imaging device including a lensless camera not using a lens has been proposed as an imaging device including a camera, and has been expected as an imaging device achieving downsizing and low cost. And, network communication of the imaging device has been essential for widening an application range of the imaging device such as image analysis.

However, the lensless imaging device of the related art example has a room for studies on suitable properties of the image sensor for the imaging. The Patent Document 1 does not describe the suitable properties of the image sensor for the imaging using the lensless imaging device.

A purpose of the present invention relates to a technique of a lensless imaging device, and is to provide a technique for the suitable properties, structures, process methods and others of the image sensor for the imaging.

Means for Solving the Problems

A typical embodiment of the present invention provides the following configuration. An imaging device of one embodiment includes: a modulator configured to modulate an intensity of incident light in accordance with a real pattern; an image sensor configured to convert the modulated light into an electric signal and create a sensor image; and a micro lens array including a plurality of micro lenses arranged to correspond to a plurality of pixels of the image sensor, and has a distribution property of a relative positional difference amount between a center position of a light receiver of each pixel of the plurality of pixels and a center position of each micro lens of the plurality of micro lenses of the micro lens array in a plane of the image sensor, the property having at least one point or more with a changing difference value of the difference amount between the adjacent pixels from a positive value to a negative value or from a negative value to a positive value.

Effects of the Invention

The typical embodiment of the present invention can provide the suitable properties, structures, processing methods and others of the image sensor for the imaging in the lensless imaging device.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the same components are denoted by the same reference symbols in principle throughout all the drawings, and the repetitive description thereof will be omitted. In the embodiments described below, the invention will be described in a plurality of sections or embodiments when required. These sections or embodiments are relevant to each other as a modification example, details, or a supplementary explanation thereof. The number, the amount, the range, the size, the shape of the components, the positional relation thereof, and the like that are mentioned are not limited to the specific number or others. The shape of the component or others includes the approximate or similar shape or others.

First Embodiment

With reference toFIGS.1to33and others, an imaging device and an imaging method of the first embodiment of the present invention will be explained. The imaging method of the first embodiment is a method including steps executed by the imaging device of the first embodiment. The imaging device of the first embodiment is the imaging device to which the imaging method of the first embodiment is installed.

<Principle of Imaging of Object at Infinite Distance>

A principle of imaging of an object at infinite distance and others will be mainly explained below.

FIG.1shows a configurational example of an imaging device of the first embodiment. The imaging device inFIG.1is a lensless camera capturing an image of an object in outside while not using an imaging lens. The imaging device1includes an imaging portion102, a fringe scan processer106, an image processer107and a controller108. The imaging portion102includes an image sensor103, a micro lens array140and a modulator110. The modulator110includes a pattern substrate104and a real pattern105. The image sensor103is a device that converts light into an electric signal and creates a sensor image. The modulator110is a combined portion of the pattern substrate104and the real pattern105, and is a device that modulates an intensity of light entering the image sensor103and being detected, in accordance with the real pattern105.

FIG.2shows an exploded view of components in a configurational example of the imaging portion102. Note that an x-axis direction, a y-axis direction and a z-axis direction are used as explanatory directions. The x-axis direction and the y-axis direction are two orthogonal directions making a principal plane of the image sensor103or others, and the z-axis direction is a direction being vertical to the principal plane, corresponding to an optical axis, a thickness or others and being up and down directions in the drawing. On a light receiving plane of the image sensor103(on an upper plane of the drawing), a plurality of pixels30are arranged as an array. Above the light receiving plane of the image sensor103, a micro lens array140described later is arranged. Above the light receiving plane of the image sensor103, the pattern substrate104is arranged through the micro lens array140. The pattern substrate104is tightly fixed to, for example, the micro lens array140. However, the present invention is not limited to this arrangement, and these components may be arranged to separate.

The real pattern105is formed on an upper surface of the pattern substrate104in the drawing. The pattern substrate104is made of a material such as a glass or a plastic that is transmittable for visible light. The real pattern105may be formed by, for example, a sputtering method or others used in a semiconductor process to deposit a metal such as aluminium or chromium. Contrast of the real pattern105for modulating a transmissivity is made of, for example, a pattern with the deposited aluminium and a pattern without the deposited aluminium. Regarding the method and the means for the formation of the real pattern105, the present invention is not limited to this, and the method or the means may be a means of achieving the modulation of the transmissivity such as a method of making the contrast by, for example, printing using an inkjet printer or others. And, in the present specification, the visible light has been explained for the explanation. However, the present invention is not limited to this, and a material having a transmissivity to a target wavelength for the imaging may be used for the pattern substrate104while a material that can transmit or shield the light may be used for the real pattern105. For example, in a case of an imaging device1performing far infrared-ray imaging, a material such as germanium, silicon, chalcogenide or others having a transmissivity to the far infrared ray may be used for the pattern substrate104while a material that can transmit or shield the far infrared ray may be used for the real pattern105. Note that the present specification has described an aspect of the formation of the real pattern105on the pattern substrate104. However, the present invention is not limited to this, and a different aspect such as usage of a display element may be used.

As a modification example,FIG.3shows a configuration of the imaging portion102in a different aspect. The real pattern105is formed as a thin film in the imaging portion102inFIG.3, and this real pattern105is held to the image sensor103and others by a support member301.

An angle of view of the imaging of the imaging device1in the aspect ofFIGS.1and2varies depending on a thickness of the pattern substrate104. Therefore, for example, when the pattern substrate104has the configuration ofFIG.3to have a function capable of changing a length of the support member301in the z-axis direction in the drawing, a function capable of changing the angle of view in the imaging is achieved.

On an upper surface of the image sensor103inFIG.2, pixels30that are light receiving elements are orderly arranged in, for example, a grid formation. This image sensor103converts light images received by the pixels30into image signals that are electric signals. As shown inFIG.1, the image signals out of the image sensor103are subjected to the image processing through the fringe scan processer106by the image processer107, and are output through the controller108.

In the configuration of the imaging device1, at the time of the imaging, a light intensity of the light penetrating the real pattern105is modulated by the real pattern105. The modulated light having penetrated the modulator110penetrates the micro lens array140, and is received by the image sensor103. Noises of the image signals out of the image sensor103are cancelled by the fringe scan processer106. The noise-cancelled signals are subjected to the image processing by the image processer107to restore the image. The image-processed image data is output to the controller108. When the image data out of the image processer107is output to a host computer, an external storage medium or others, the controller108converts a data format of the image data to be fixed to an interface such as a USB or others and outputs it. Note that the image signals, the image data or others may be stored in a memory inside the imaging device1.

Next, a principle of the imaging in the imaging device1will be explained. First, the real pattern105is made of a concentric circles pattern having a pitch reducing in inverse proportion to a radius positioned from its center. The real pattern105is defined by the following equation 1 using a radius “r” positioned from reference coordinates at the center of the concentric circles and a coefficient “β”. The real pattern105is designed to have the transmissivity that is modulated in proportion to a term “I(r)” of the equation 1.
[Equation 1]
I(r)=1+cos βr2Equation 1:

A plate having such a stripe is called a Gabor zone plate or a Fresnel zone plate.FIG.4shows an example of a Gabor zone plate based on the equation 1 as a configurational example of the real pattern105. This Gabor zone plate is a transmissivity-modulating pattern having multiple-valued contrasts. As another configurational example of the real pattern105,FIG.5shows an example of a Fresnel zone plate expressed by binarization of the equation 1 taking 1 as a threshold. This Fresnel zone plate is a transmissivity-modulating pattern having binarized contrasts. White parts correspond to transparent parts while black parts correspond to non-transparent parts. A virtual pattern corresponding to the real pattern105also has the same pattern. Note that only the x-axis direction ofFIG.2will be mainly expressed by expressions below for simplification. However, two-dimensional expansion can be considered in the similar consideration about the y-axis direction.

FIG.6is an explanatory diagram about the pattern substrate104. With reference toFIG.6, occurrence of in-plane shift of a shadow from the real pattern105on the surface of the pattern substrate104to the image sensor103due to oblique incident collimated light will be explained. As shown inFIG.6, it is assumed that, at an angle “θ0” in the x-axis direction, the collimated light enters the pattern substrate104having a thickness “d” on which the real pattern105is formed. When a refraction angle in the pattern substrate104is “θ”, light having a transmissivity of the real pattern105on the surface of the pattern substrate104geometrically refracts by an amount of “k=d·tan θ” in the drawing, and enters the image sensor103. In this case, a shadow having an intensity distribution as expressed by the following equation 2 is detected on the image sensor103. Note that a term “φ” of the equation 2 represents an initial phase of a transmissivity distribution of the equation 1.
[Equation 2]
IF(x)=1+cos[β(x+k)2+Φ]  Equation 2:

FIG.7shows an example of a shadow701of this real pattern105. This shadow701shifts by “k” (shifts by a shift amount “k”), and is projected as expressed by the equation 2. This shadow701becomes an output of the imaging portion102.

FIG.8shows a virtual pattern801. The virtual pattern801inFIG.8is an example corresponding to the real pattern150inFIG.5. The imaging processor107performs reconstruction while using such a virtual pattern801.

Next, regarding a process in the imaging processor107, reconstruction processes based on a correlation reconstruction method and a moire reconstruction method will be explained. Either the reconstruction process based on the correlation reconstruction method or the reconstruction process based on the moire reconstruction method may be used. Alternatively, both methods may be installed, and the methods may be switched if needed.

In the correlation reconstruction method, the imaging processor107calculates a cross-correlation function between the shadow701of the real pattern105inFIG.7and the virtual pattern801inFIG.8. This manner provides a bright spot902having the shift amount “k” as shown inFIG.9.FIG.9shows a configurational example of a reconstructed image901based on the correlation reconstruction method.

When the cross-correlation calculation is performed as a two-dimensional convolution calculation, a calculation amount is generally large, and therefore, a principle of a calculation example using Fourier transform will be explained with reference to equations. First, for the virtual pattern801inFIG.8, the Gabor zone plate or the Fresnel zone plate is used as similar to the real pattern105. Therefore, the virtual pattern801can be expressed by the following equation 3 using the initial phase “φ”.
[Equation 3]
IB(x)=cos(βx2+Φ)  Equation 3:

The virtual pattern801is used in an imaging process, and therefore, does not need to be offset at 1 as different from the equation 1, and has no problem when having a negative value. The Fourier transform in the equations 1 and 3 are as shown in the following equations 4 and 5.

In the equations, a term “F” represents the calculation of the Fourier transform, a term “u” represents frequency coordinates in the x-axis direction, and a term “δ” with parentheses represents a delta function. What is important in these equations is that the Fourier-transformed equations also show the Gabor zone plate or the Fresnel zone plate. Therefore, the imaging device1may directly create and store the Fourier-transformed virtual pattern to be used. In this manner, the calculation amount can be reduced. Next, the following equation 6 is provided by multiplication of the equations 4 and 5.

A term “e−iku” expressed by an exponential function of the equation 6 represents a signal component. By the Fourier transform of this term “e−iku”, this term is transformed as expressed by an equation 7, and the bright spot can be provided at a position shifting by the shift amount “k” in the initial x axis.
[Equation 7]
−1[e−iku]=2πδ(x+k)  Equation 7:

This bright spot represents light flux at infinite distance, and corresponds to the captured image created by the imaging device1inFIG.1. Note that the correlation reconstruction method may be achieved by a pattern such as a random pattern not limited to the Gabor zone plate or the Fresnel zone plate if a self correlation function of this pattern has a single peak.

Next, in the moire reconstruction method, the imaging processor107creates a moire stripe1000as shown inFIG.10by multiplying the shadow701of the real pattern105inFIG.7and the virtual pattern801inFIG.8. Then, the imaging processor107provides bright spots1102and1103shifting by a shift amount “kβ/π” in a reconstructed image1101inFIG.11by performing the Fourier transform of this moire stripe1000. The bright spot1102is a bright spot with “−kβ/π”, and the bright spot1103is a bright spot with “+kβ/π”. This moire stripe is expressed by the following equation 8.

The third term in the expanded equation in the equation 8 represents a signal component. It is understood that this signal component creates a stripe pattern such as the moire stripe1000inFIG.10over a two-pattern overlapping region. In the moire stripe1000inFIG.10, stripe patterns that are linearly extending in a direction (y-axis direction) orthogonal to the two-pattern shift direction (x-axis direction) are formed at an equal interval (interval1001) in the x-axis direction. A stripe that is created at a relatively low spatial frequency by such overlap between the stripes is called moire stripe. In the example of the first embodiment, the moire stripe is created by the overlap between the real pattern501and the virtual pattern801. The two-dimensional Fourier transform of the third term in the equation 8 is as expressed by the following equation 9.

In this equation, a term “F” represents the calculation of the Fourier transform, a term “u” represents the frequency coordinates in the x-axis direction, and a term “δ” with parentheses represents the delta function. This result shows that a peak of the spatial frequency in a spatial frequency spectrum of the moire stripe is at a position of “u=±kβ/π”. A bright spot corresponding to this peak represents the light flux at infinite distance, and corresponds to the image captured by the imaging device1inFIG.1.

Note that the moire reconstruction method may be achieved by not only the Gabor zone plate or the Fresnel zone plate but a pattern such as an ellipsoidal pattern if the moire stripe resulted from shift of this pattern has a single frequency.

Noize cancel based on fringe scan or others will be explained below.

The transformations from the equation 6 to the equation 7 and from the equation 8 to the equation 9 have been explained along with the focus on the signal component. However, other terms from the signal component term practically inhibit the reconstruction. Accordingly, a fringe scan processor106of the imaging device1performs noise cancelling based on fringe scan. For the fringe scan, it is necessary to use a plurality of patterns having different initial phase φ as the real pattern105.

FIG.12shows a configurational example of combination of the plurality of patterns having different initial phase φ as the real pattern105in the fringe scan.FIG.12(a) to (d)show four phase patterns1201to1204having “φ=0, π/2, π, 3π/2”. In this case, in accordance with the following equation 10, the fringe scan processor106calculates the sensor image captured along with the usage of these four phases. As a result, a sensor image having a complex number (referred to as complex sensor image in some cases) is provided.

The virtual pattern801having a complex number is expressed by the following equation 11. The virtual pattern801has no problem even when having the complex number because of being used in the fringe scan process performed by the fringe scan processor106.
[Equation 11]
ICB(x)=exp(−iβx2)  Equation 11:

In the moire reconstruction method, the following equation 12 is provided by multiplication of the equations 10 and 11. A term “exp(2iβkx)” expressed by an exponential function of the equation 12 represents the signal component. From this signal component, it is understood that the noise canceling process without an unnecessary term as shown in the equation 8 is performed.
[Equation 12]
ICF(x)·ICB(x)=exp[iβ(x+k)2]·exp(−iβx2)=exp[2iβkx+iβk2]  Equation 12:

The correlation reconstruction method is similarly reviewed. The Fourier transforms of the equations 10 and 11 are expressed by the following equations 13 and 14.

Next, the following equation 15 is provided by multiplication of the equations 13 and 14. A term “exp(−iku)” expressed by an exponential function of the equation 15 represents the signal component. From this signal component, it is understood that the noise canceling process without an unnecessary term as shown in the equation 8 is performed.

The above-described examples have been explained with reference to the four-phase plural patterns. However, the initial phase φ only needs to be designed to equally divide an angle ranging from 0 to 2π, and is not limited to the four phases.

The method for achieving the imaging based on the plural patterns is roughly classified into a method of switching the patterns in time division and a method of switching the patterns in spatial division. Either the method of switching the patterns in time division or the method of switching the patterns in spatial division may be applicable, or a method of installing both methods and using a selected method may be applicable.

In order to achieve the time-division fringe scan for handling the method of switching the patterns in the time division, for example, the plurality of initial phases ofFIG.12may be electrically switched, and displayable display elements such as liquid crystal display elements may be used as the real pattern105inFIG.1. For example, the fringe scan processor106inFIG.1controls the switching timing of the liquid crystal display elements showing the real pattern105and a shutter timing of the image sensor103to be synchronized. In this manner, after acquisition of four images corresponding to the image sensor103, the imaging device1executes the fringe scan calculation in the fringe scan processor106.

On the other hand, in order to achieve the spatial-division fringe scan for handling the method of switching the patterns in the spatial division, for example, a real pattern1300having the plurality of initial phases in the spatial division is used as the real pattern105as shown inFIG.13. The real pattern1300inFIG.13includes the four-phase patterns in regions1301to1304resulted from division of the entire quadrangular region into four pieces in the x-axis direction and the y-axis direction.

After the imaging device1acquires one image from the image sensor103, the imaging device1divides this image in the fringe scan processor106into four pieces corresponding to the respective initial-phase patterns, and executes the fringe scan calculation.

Next, the fringe scan calculation in the fringe scan processor160will be explained.FIG.14is a flowchart showing a process outline including the fringe scan calculation in the fringe scan processor160.FIG.14shows steps S1to S7that will be explained below in an order of the steps. First, in the step S1, the fringe scan processor106acquires the sensor image based on the plurality of patterns out of the image sensor103. In this case, the fringe scan process106divides this sensor image if the spatial-division fringe scan is used, or does not divide it in this step S1if the time-division fringe scan is used.

Next, in the step S2, the fringe scan processor106initializes the complex sensor image to be output. The fringe scan processor106acquires the sensor image having the first initial phase φ in the loop in the step S3, multiplies this sensor image with “exp(iφ)” that is the exponential function corresponding to this initial phase φ in the step S4, and adds the multiplication result to the complex sensor image to be output in the step S5.

The fringe scan processor106confirms whether the process for all initial phases cp to be used ends or not in the step S6, and similarly repeats the processes of the steps S3to S5the number of times corresponding to the number of the initial phases φ to be used. For example, in the fringe scan using the four phases inFIGS.12and13, the processes are repeated four times corresponding to the number of the initial phases “φ=0, π/2, π, and 3π/2”.

Lastly, in the step S7, the fringe scan processor106outputs the processed complex sensor image. The above-described main processes in the fringe scan processor106are equivalent to the above-described equation 10.

Next, an imaging process in the imaging processor107will be explained.FIG.15is a flowchart showing a process outline in the case of usage of the correlation reconstruction method in the imaging processor107.FIG.15shows steps S21to S26. First, in the step S21, the imaging processor107acquires the complex sensor image out of the fringe scan processor106, and executes the two-dimensional fast Fourier transform (FFT: Fast Fourier Transform) calculation to this complex sensor image.

Next, in the step S22, the imaging processor107creates the virtual pattern801used for the reconstruction. Alternatively, the imaging processor107may refer to the virtual pattern801that is previously set and stored in a memory or others. Then, the imaging processor107multiplies the two-dimensional FFT calculated complex sensor image with this virtual pattern801. In the step S24, the imaging processor107performs an inverse two-dimensional FFT calculation to the multiplied image.

This calculation result shows a complex number. Therefore, in the step S24, the imaging processor107performs a real number conversion process to the complex number that is the calculation result. The real number conversion process is a process for conversion to an absolute value or a process for extraction of a real part. When the complex number is expressed as “C=A+Bi”, the process for conversion to an absolute value is expressed as “|A+Bi|=√(A2+B2)”. The process for extraction of a real part is expressed as “Re[A+Bi]=A”. In this manner, the imaging processor107rationalizes the image, and reconstructs it.

Then, the imaging processor107performs a contrast enhancement process to the resultant image in the step S25, and adjusts color balance or others in the step S26. The imaging processor107outputs data of the process result as the captured image. The imaging process in the imaging processor107ends here, and the captured image that is visually recognizable for the user can be provided.

In comparison to the above description,FIG.16shows a process outline in the case of the usage of the moire reconstruction method.FIG.16shows steps S31to S36. First, in the step S31, the imaging processor107acquires the complex sensor image out of the fringe scan processor106, creates the virtual pattern801used for the reconstruction, and multiplies the complex sensor image with this virtual pattern801. The imaging processor107performs a two-dimensional FFT calculation to the multiplied image to provide a frequency spectrum in the step S32, and extracts data of a necessary frequency region from this frequency spectrum in the step S33. The processes in the subsequent steps S34to S36are the same as the processes in the steps S24to S26inFIG.15.

Note that the imaging device1is not limited to a camera capturing the image based on the visible light, and can be an imaging device targeting a frequency band of infrared ray or others or an imaging device (referred to as distance image sensor or others in some cases) imaging a distance image to an object (that is an image having three-dimensional distance information in a depth direction).

<Principle of Imaging of Object at Finite Distance>

A principle of imaging of an object at finite distance and others will be explained below.

[Imaging of Object at Infinite Distance and Imaging of Object at Finite Distance]

Next,FIG.17shows a state of projection of the real pattern105to the image sensor103in the above-described case of the far object (in other words, the object at infinite distance). A spherical wave from a point1701configuring the far object becomes a planar wave while travelling in a sufficiently long distance, and is emitted to the real pattern105. When a shadow1702created by this emission is projected to the image sensor103, the shadow has almost the same shape as that of the real pattern105. As a result, a single bright spot can be provided by the reconstruction to the shadow1702along with the usage of the virtual pattern.

On the other hand, the imaging of the object at finite distance for handling a case of a near object will be explained.FIG.18is an explanatory diagram in the case of the object to be imaged at finite distance, showing that the shadow of the real pattern105on the image sensor103is wider than the real pattern105. A spherical wave from a point1801configuring the object is emitted to the real pattern105, and a shadow1802created by this emission is projected to the image sensor103. In this case, the shadow is almost uniformly expanded. Note that an expansion rate “α” in this case can be calculated by the following equation 16 along with usage of a distance “f” from the real pattern105to the point1801in the z-axis direction.

Therefore, the reconstruction along with the usage of the virtual pattern that is designed for the collimated light as it is cannot provide the single bright spot. Accordingly, expansion of the virtual pattern801to match the uniformly-expanded shadow of the real pattern105as shown inFIG.18can provide the single bright spot again in the expanded shadow1802. For this, correction is achieved by changing the coefficient “β” of the virtual pattern801into “β/α2”. In this manner, the imaging device1can selectively reproduce the light on the point1801at a distance that is not always infinite. Therefore, the imaging device1can perform the imaging to focus on any position.

<Configuration of Image Sensor>

Next, with reference toFIG.19and others, a configuration of the image sensor103will be explained.

FIG.19shows cross-sectional structures of a center pixel1901and an end pixel1902of the image sensor103a(103) in the x-axis direction as a comparison example of the first embodiment. In the explanation, an image sensor made of a general imaging element such as CMOS or CCD is considered. A pixel30of this image sensor103acannot receive the light on the whole surface but can receive the light only on a light receiver1903that is a part of the surface based on a wiring or a mask structure. A width1912of the light receiver1903is smaller than a width1911of the pixel30.

Therefore, in the general image sensor, micro lenses1904are arranged as the micro lens array in front of the light receiver1903in the z-axis direction as shown in this image sensor103ain order to improve the light use efficiency. In this manner, the incident light1905illustrated with a dashed dotted line is gathered to be light1906illustrated with a dotted line by the micro lenses1904, and is efficiently taken into the light receiver1903. Each of the micro lenses1904is a lens that is convexly curved on the incident side.

FIG.20shows a state of the light incidence on one pixel30inFIG.19. Since the light receiver1903of the image sensor103ainFIG.19is smaller than the pixel30as described above, the imaging is possible only in an angle range in which the light is gathered in the light receiver1903as shown with a light ray2001inFIG.20. Sensitivity for light emitted from other angles than this angle range is rapidly reduced.

FIG.21shows an example of dependency of the sensitivity to the angle, andFIG.22shows an example of practical measurement. Such sensitivity is called Chief Ray Angle (CRA) property. A horizontal axis of a graph inFIG.21represents the angle θ, and a vertical axis of the same represents the sensitivity. The sensitivity corresponds to a light receiving amount and a light intensity. A horizontal axis inFIG.22represents an incident angle corresponding to the angle θ, and a vertical axis of the same represents a normalized light intensity. If the structure of the image sensor is no problem, the CRA property generally reduces by the fourth power of cos θ. However, if the structure of the image sensor becomes disincentive as described above, the light amount rapidly reduces by an amount that is more than the fourth power of cos θ as shown inFIG.22.

Therefore, in a comparison example, an image-side telecentric optical system is used in a general camera not being the lensless camera as shown inFIG.23. In this manner, the light ray is designed to always orthogonally enter the image sensor and to avoid obliquely entering the image sensor (the corresponding pixel).FIG.23shows a configurational example of the image-side telecentric optical system in the general camera in the comparison example. InFIG.23, lenses2301and2302are arranged in front of the image sensor103ain the z-axis direction. The lens2302is a field lens that is a lens for changing a travelling direction of the light near the object. The incident light ray2303that is oblique to the optical axis in the z-axis direction is converted to the light ray travelling in the z-axis direction, that is the collimated light, by the lenses2301and2302.

However, as a result of advancement of downsizing and high density observed in apparatuses such as smartphones in recent years, non-telecentric optical systems as shown inFIG.24have been used often.FIG.24shows a configurational example of the non-telecentric optical system in another general camera as a comparison example. InFIG.24, a lens2401is arranged in front of the image sensor103ain the z-axis direction. The incident light ray2403that is oblique by an angle θ1to the optical axis in the z-axis direction is received as the oblique light ray by the light receiver even after traveling in the lens2401. In a case of a camera with a lens as shown inFIG.24, a pixel property is different depending on the incident angle of the light ray among a plurality of pixels in the plane of the image sensor103a. The light obliquely enters a pixel at an end, and therefore, the light amount is less, and a dark pixel value is observed.

In recent cameras, a configuration as shown inFIG.25fitting with such a property of the non-telecentric optical system is a mainstream. As a comparison example of the first embodiment,FIG.25shows a configurational example of arrangement of the pixel30of an image sensor103b(103) and a micro lens1904. In the configuration ofFIG.25, an optical axis of the micro lens1904of the image sensor103and an optical axis of the light receiver1903purposely shift to be different from each other. If it is desirable to equalize the light amount on each pixel30in the plane of the image sensor103in this configuration, the light ray is corrected by a relative positional relation with the micro lens1904to be set to the end pixel1902. In this configuration, the relation between the micro lens1904and the light receiver1903is different between the center pixel1901and the end pixel1902of the image sensor103in the x-axis direction. This image sensor103bis configured to change the relative position between the micro lens1904and the light receiver1903, and to monotonously increase or decrease a relative-position difference amount. At the center pixel1901, the optical axis of the micro lens1904and the optical axis of the light receiver1903match each other. At the end pixel1902, there is a distance2502in the difference between the optical axis of the micro lens1904and the optical axis of the light receiver1903. Also at an opposite end pixel, there is a distance in difference in an opposite direction although not illustrated. Therefore, this image sensor103bhas an imaging system in which attenuation of the light amount reduction near the image is small. Cameras having the configuration as shown inFIG.25have been already inexpensively achieved.

However, the lensless optical system does not have the lens as shown inFIG.24, and the light from the point1701creates the shadow1702that is almost the collimated light at an outgoing position from the pattern substrate104, and is uniformly emitted to the image sensor103. Basically, the light ray orthogonally enters each pixel in the plane of the image sensor103, and the light amounts of the respective pixels are almost the same. In this case, when the image sensor103bthat is the type as shown inFIG.25is used, shading is strongly observed in a sensor image corresponding to the captured image signal as shown inFIG.26. In other words, a peripheral part of the sensor image is darkened.FIG.26shows an example of a sensor image2601with the shading in the case of the usage of the image sensor103b. Under the occurrence of such shading, noises having a low frequency are strongly observed in the course of the later-stage reconstruction, and therefore, the reconstructed image is deteriorated.

Therefore, for the lensless optical system, not the configuration of the image sensor103bas shown inFIG.25but the configuration with the uniform relation between the light receiver1903and the micro lens1904as shown in the image sensor103aof the type inFIG.19is desirable in order to avoid the deterioration of the image. However, under the application of this configuration, the CRA property depending on the pixel structure is a problem as shown inFIGS.21and22, and a field of view (a corresponding angle range of view) of the camera is narrowed.

Accordingly, a system for solving the problems of the shading and the CRA property in the imaging device1of the first embodiment will be explained. The purpose of the first embodiment is to achieve both of the suppression of the image deterioration and the wide angle range of view.FIG.27(A)shows top views of one pixel30and the micro lens1904inFIG.20. In the case ofFIG.27(A), there is no relative positional difference between the micro lens1904and the light receiver1903. On the other hand,FIG.27(B)shows a case with the relative positional difference between the micro lens1904and the light receiver1903. The relative positional difference is expressed as a difference (Δ)2701using vectors. When there is the relative positional difference2701as shown inFIG.27(B), the difference is regarded to be divided into a difference “Δx” in the x-axis direction and a difference “Δy” in the y-axis direction.

In this case,FIG.28shows a relation between a position of the pixel30and the difference “Δx” in the x-axis direction in the configuration of the image sensor103binFIG.25.FIG.28(A)shows a distribution property of the difference Δx, andFIG.28(B)shows a graph resulted from differentiation of the property ofFIG.28(A). InFIG.28(A), regarding the pixel30at each position in the x-axis direction, a center position is represented by a position “L1”, a right end position that is in a positive direction is represented by a position “L2”, and a left end position that is in a negative direction is represented by a position “L3”. As seen in the drawing, at the center position L1of the image sensor103b, there is no difference Δx, that is, 0. At the right end position L2, there is the difference Δx in the positive direction (+Δ×1). At the left end position L3, there is the difference Δx in the negative direction (−Δ×1). InFIG.28(A), the change of the relative positional difference is monotonous increase change. InFIG.28(B), a value of the difference between the adjacent pixels that is the change amount of the difference Δx is constant. Note thatFIG.28(A)shows the linearly changing difference Δx. However, the present invention is not limited to this configuration, and a curved change expressed by a quadratic function or others is also applicable.

This configuration of the image sensor103bis achieved by designing a pitch of the micro lens1904and a pitch of the pixel30to be almost equal to each other and relatively shifting, that is, changing the pitches to have a difference of one pixel or smaller between the center and the end of the image sensor130b.

FIG.29shows a pitch “p1” between the pixels30and between the light receivers1903, a pitch “p2” between the micro lenses1904, a relative positional difference2502and others for explanation about the configuration of the image sensor103b.FIG.29shows an example of the left end pixel1902inFIG.25. The pitch p1and the pitch p2are almost equal to each other. Regarding the relative positional difference2502, a center position of the micro lens1904with respect to a center position of the pixel30and the positive direction in the x-axis direction (that is a direction from left to right in the drawing) are illustrated as reference. For example, the pitch p2between the micro lenses1904at the center pixel1901inFIG.25is represented by a pitch “p21”, and the pitch p2between the micro lenses1904at the left end pixel1902is represented by a pitch “p22”. The above-described configuration is designed to have, for example, a relation of “p21>p22”, and the pitches p2between the micro lenses1904from the center to the end shift to be reduced from the pitch p21to the pitch p22. Besides, in this configuration, the change of the relative positional difference between the pixel30and the micro lens1904is designed to be one pixel or smaller.

On the other hand,FIG.30shows a configurational example of an image sensor103c(103) in the imaging device1of the first embodiment. InFIG.30, a configurational example of a pixel30and a micro lens140at the center position L1in the plane of the image sensor103is illustrated to be enlarged. In this configuration, the pitch p2between the micro lenses1904is larger than the pitch p1between the pixels30(p1<p2). The center pixel30is represented by a pixel “PX1”, and pixels on both sides of this pixel are represented by “PX2” and “PX3”. The micro lenses140corresponding to these pixels are represented by micro lenses “ML1”, “ML2” and “ML3”, respectively. At the center pixel PX1, the relative positional difference Δx from the micro lens ML1is 0. At the right pixel PX2, the relative positional difference Δx (3001) from the micro lens ML2is a value that is larger than 0 in the positive direction. At the left pixel PX3, the relative positional difference Δx (3002) from the micro lens ML3is a value that is larger than 0 in the negative direction. The change in the relative positional difference becomes larger when the position is closer from the center to the outer end in the plane.

In the case of the large difference between the pitch p2of the micro lens140and the pitch p1of the pixel30as shown inFIG.30, in other words, in the case of the certain difference (p2−p1) or larger between the pitch p1and the pitch p2, the relative positional difference (Δx, Δy) in the plane rapidly changes. When the relative positional difference between the adjacent pixels30is larger than one pixel in accordance with the difference design (p2−p1), a point changing from positive to negative or from negative to positive occurs. The larger the difference (p2−p1) is, the larger the number of such points is. As described above, in the first embodiment, for example, since the pitch p1and the pitch p2are largely made different from each other, the relative positional difference changes in the plane of the image sensor103to provide various values. This manner provides an effect of the improvement of the camera properties as described later.

To put it in extreme words, the image sensor103in the first embodiment may be configured so that the relative position between the pixel30and the micro lens140shifts/changes at random in the plane as shown inFIG.31. A schematic graph inFIG.31(A)shows the distribution property of the relative positional difference amount of the image sensor103in the first embodiment. A horizontal axis of the graph represents, for example, the pixel30at each position in the x-axis direction, and a vertical axis of the same represents the relative positional difference Δx between a center position of the pixel30and a center position of the micro lens140. InFIG.31(A), the difference Δx has random values ranging from −Δ×1 to +Δ×1.FIG.31(B)shows a schematic graph resulted from differentiation of the property ofFIG.31(A), the graph taking the difference value between the adjacent pixels as the change amount of the relative positional difference. As shown inFIG.31(B), in the first embodiment, the difference value of the relative positional difference amount also has the random distribution. This property shows a point at which the difference value that is the change of the difference amount between the adjacent pixels changes from a positive value to a negative value or a point at which the difference value changes from a negative value to a positive value. Such a random property is achieved by a configuration or others designed to, for example, make the random of at least either one of the pitch p1of the pixels30and the pitch p2of the micro lens140in accordance with a position in the plane.

The configuration as shown inFIGS.30and31is a configuration including various relative positional differences in the plane of the image sensor103, that is a configuration including the pixels30corresponding to the receptions of the light ray at various incident angles. By the configuration as shown inFIGS.30and31, the CRA property as shown inFIG.32is achieved.FIG.32shows an example of the entire CRA property of the image sensor103in the first embodiment. A CRA property3200inFIG.32is a variously-combined property including a CRA property3201, a peak of which is on the incident light ray in an orthogonal direction (z-axis direction) to the image sensor103, and CRA properties3202and3203, each peak of which is on the oblique incident light ray and others. The entire image sensor103is configured to have sensitivity to light with a wide incident angle as shown in the CRA property3200.

However, for example, pixels having a sensitivity to the incident light ray in the orthogonal direction to the image sensor103and pixels not having or having a low sensitivity thereto exist, and therefore, an image with missing some pieces is adversely created as the sensor image. In a case of a system of forming the image on the image sensor such as a general camera, this is a problem because of pixel defect. On the other hand, in the case of the lensless imaging device including the lensless optical system as described in the first embodiment, the information of the incident light from certain points entirely diffuses in the plurality of pixels30of the image sensor103(the corresponding sensor image). Therefore, even if the defect exists in the sensor image, the information (the corresponding image) can be restored by the process. In accordance with such properties and such configurations of the random relative positional difference of the lensless imaging device, the imaging device1of the first embodiment can perform the reconstruction while suppressing the image quality deterioration.

In order to achieve the effect as described above, it is only necessary to randomly change, for example, the relative position between the light receiver1903and the micro lens140. Therefore, the same effect can be provided by not only the configuration as shown inFIG.31but also the configurational example as shown inFIG.33. The configuration inFIG.33is a configuration as a modification example of the first embodiment in which an optical element such as a scattering member3301randomly scattering the light is inserted. In the configuration inFIG.33, the scattering member3301is arranged on the incident side of the micro lens140in the z-axis direction. The configuration inFIG.33does not have the distribution of the relative positional difference between the pixel30and the micro lens140in the image sensor103. As another configuration, the scattering member3301may be arranged in addition to the configuration having the distribution of the relative positional difference in the image sensor103.

As described above, the imaging device1and the imaging method of the first embodiment provide the configuration having the random change of the relative positional difference between the light receiver1903and the micro lens140in the plane of the image sensor103, in other words, the configuration having the distribution of various relative positional differences. This manner can provide the property, the structure, the processing method and others of the image sensor103that are suitable for the imaging, and can achieve the lensless imaging device having the wide angle of view of the imaging, not generating the noises due to the shading but generating the high image quality and improving the camera property. Further, according to the configuration of the first embodiment, the existing inexpensive image sensor can be used as image sensor103, and therefore, the manufacturing cost can be also suppressed.

Second Embodiment

With reference toFIG.34and subsequent drawings, an imaging device and an imaging method of the second embodiment will be explained. Configurational parts of the second embodiment that are different from those of the first embodiment will be explained below.

[Missing Pieces of Sensor Image]

In the arrangement of the configuration of the imaging device1of the first embodiment, for example, the relative position between the light receiver1903and the micro lens140in the image sensor103randomly changes. This configuration has a possibility of the random occurrence of the missing pieces of the sensor image when the object is imaged at a certain angle, which results in the influence of the random missing pieces on the image quality of the reconstructed image. Accordingly, in the second embodiment, devised configuration, processing method and others will be explained in consideration of this possibility.

Even in the imaging device1of the second embodiment, the CRA property of the entire lensless optical system is improved by the configuration having the change of the relative positional difference between the light receiver and the micro lens in the plane of the image sensor103. However, the second embodiment does not provide the random arrangement as described in the first embodiment, but provides arrangement with the previously-known relative positional difference amount and distribution property such as periodic arrangement. In this manner, as the process performed in the second embodiment, the sensor image captured by the image sensor103is divided for each angle corresponding to the incident angle of the light ray.

FIG.34shows a configurational example of the periodic arrangement of the relative positional difference in the image sensor103of the imaging device1of the second embodiment.FIG.34(A)shows the distribution property of the relative positional difference amount of the image sensor103of the second embodiment, the distribution property being periodic as shown in the drawing. A horizontal axis ofFIG.34(A)represents, for example, the pixel30at each position in the x-axis direction, and a vertical axis of the same represents the relative positional difference Δx between the light receiver1903and the micro lens140. The difference Δx periodically varies depending on the pixel position in the x-axis direction in a range from −Δ×1 to +Δ×1. For example, the difference Δx at a center position L1is “−Δ×1”, and the difference Δx at a position3402that shifts in a right direction from the position L1by a predetermined pixel distance is “+Δ×1”. In a region3401from the position L1to the position3402, the difference Δx linearly increases from “−Δ×1” to “+Δ×1”. In a right region3403that shifts in the right direction from the position3402by the same pixel distance, the difference Δx linearly increases as similar to the region3401.FIG.34(B)shows a graph resulted from differentiation of the property ofFIG.34(A), the graph taking the difference value between the adjacent pixels30as the change amount of the relative positional difference amount. As shown in theFIG.34(B), in the second embodiment, the difference values of the relative positional difference amount includes a portion having a certain positive value and a plurality of portions each having largely changing from a positive value to a negative value or from a negative value to a positive value. The property of the first embodiment inFIG.31and the property of the second embodiment inFIG.34are different from the monotonous increase or monotonous decrease property of the comparison example inFIG.28, and include a portion having the difference value of the relative positional difference amount changing from a positive value to a negative value or from a negative value to a positive value. The property of the imaging device1as shown inFIG.34is already known, and is used for the process in the imaging processor107or others.

On the image sensor103having the arrangement of the periodic change of the relative position between the pixel30and the micro lens1904as described above, the incident light ray in the orthogonal direction (z-axis direction) to the image sensor103creates an image as shown inFIG.35.FIG.35shows an example of a sensor image3500captured by the image sensor103and having the shading. The sensor image3500(the corresponding image sensor103) includes a plurality of regions3501on an x-axis/y-axis plane. One of the regions3501is a region corresponding to one period (that is a portion such as the region3401inFIG.34) of the change of the relative positional difference, and this one-period region is defined as one unit. Therefore, the relative positional difference Δx of this unit is as shown inFIG.36.

FIG.36shows an example of a process dividing method for the relative positional difference of the unit. A horizontal axis of a graph ofFIG.36(A)represents a position of the pixel30of one unit in the x-axis direction, and a vertical axis of the same represents the relative position difference Δx. A center of one unit is represented by “M1”, a right end is represented by “M2”, and a left end is represented by “M3”. In the imaging device1of the second embodiment, a plurality of units of all regions of the sensor image (the corresponding image sensor103) are divided into a plurality of regions such as regions A1to A5for each interval3601of a certain relative positional difference amount. In this example, the units are divided into five regions (A1to A5) within a range from the minimum value (−Δ×1) of the difference amount to the maximum value (+Δ×1) of the same. The larger the number of divisions is, the more the image quality is improved. The interval3601and the number of divisions are designed values, and are changeable. Similarly, as shown inFIG.36(B), also for the relative positional difference amount Δy of the unit in the y-axis direction, the units are divided into a plurality of regions such as regions B1to B5for each interval3602. The image processor107performs such a division process to the sensor image.

Regarding the division ofFIG.36, mapping of one unit (such as a unit “U1”) as shown in the region3501is as shown inFIG.37.FIG.37(A) shows a configurational example of the division in one unit. A smaller region than the unit is made by crossing of each region in the x-axis direction and each region in the y-axis direction. For example, a region3701is a region of “A1-B1”, and a region3702is a region of “A5-B5”. As the divided sensor image, the image processor107creates images that are classified and gathered for each same-type region in the plurality of units of the sensor image3500ofFIG.35. For example, divided sensor images are created to have “5×5=25” types that are a divided sensor image (referred to as “G11”) created by gathering of only the “A1-B1” regions3701to a divided sensor image (referred to as “G55”) created by gathering of only the “A5-B5” regions3702.

The image processor107executes the reconstruction shown inFIGS.15and16as described above for each created divided sensor image of the plurality of divided sensor images by using the corresponding reconstruction pattern. Then, the image processor107synthesizes the reconstruction results of the respective divided sensor images into one image. By such a process dividing method, each divided sensor image can be processed without the pixel missing. Therefore, the image process is enabled without the increase in the noises, the information can be restored in the synthesized image, and the deterioration of the image quality can be suppressed.

In the example, note that the process method of dividing each unit into 5×5 regions has been described. However, the present invention is not limited to this process method.FIG.38(B)shows a process example of dividing the unit only in the x-axis direction, andFIG.38(C)shows a process example of dividing the unit only in the y-axis direction. In these manners, an optimal design such as the process with the change in the direction is possible in accordance with the property of the image sensor.

In the above description, the configurational example having the periodic relative positional difference Δx as shown inFIG.34has been explained. In such a periodic design, the PSF (Point Spread Function) of the reconstructed image has a plurality of peaks as shown inFIG.38(A), and ghost phenomenon is possibly observed in the reconstructed image.FIG.38(A)shows an example of the PSF of the reconstructed image. For example, the maximum peak3800represents an original point, but a peak3801represents the ghost point.

Accordingly, as a modification example of the second embodiment, the following configuration may be applicable. In the modification example, while the relative positional difference of the image sensor103is randomly arranged as similar to the first embodiment, the random arrangement is a previously-known design. In other words, in the imaging device1, the relative positional difference included in each pixel30is previously known. In the modification example, as shown inFIG.39, the divided region is assigned for each unit in accordance with the relative positional difference amount.FIG.39shows a process dividing method for the relative positional difference in the modification example. For example, regarding the relative positional difference Δx in the x-axis direction, the regions A1to A5corresponding to the difference amount are assigned. The image processor107creates the divided sensor image corresponding to the region as similar to the second embodiment, and performs the process to each divided sensor image.

FIG.38(B)shows an example of the PSF of the reconstructed image resulted from the image process of the modification example. It is found that a peak ofFIG.38(A)based on periodicity such as a peak3801is suppressed as shown in a peak3802ofFIG.38(B)by the configuration of the modification example.

As described above, the second embodiment provides the configuration having the periodic change of the relative positional difference in the plane of the image sensor103, and correspondingly provides the configuration in which the sensor image is divided and processed into the plurality of regions. This manner can achieve the lensless imaging device having the wide angle of view of the imaging, and not generating the noises based on the shading but generating the high image quality. Note that the periodic distribution property of the relative positional difference in the image sensor103is not limited to the above-described examples, and may be defined by a different periodic function such as a sine function.

In the foregoing, the present invention has been concretely described on the basis of the embodiments. However, the present invention is not limited to the foregoing embodiments, and various modifications can be made within the scope of the present invention. In the present invention, for example, elements can be added to, eliminated from, replaced with, or combined with all elements of the embodiment. In the present invention, combination of one example and another example may be also applicable. Each element (such as the controller108, the image processor107the fringe scan processor106and others inFIG.1) of the present invention may be made of a hardware such as an integrated circuit or FPGA, or a software program process or others. In the case of the program process, the functionality is achieved when, for example, a processor using a CPU, a ROM, a RAM or others executes a process in accordance with a program while using a memory or others. The programs and related data and information may be stored in a storage device or a record medium such as an HDD, a DVD, an SSD, an SD card or others, or may be stored on an external communication network.

FIG.40shows a configurational example of cross-correlation calculation in the above-described correlation reconstruction method for supporting each embodiment. The image processor107performs FFT402to the sensor image401that is input from the image sensor103through the fringe scan processor106to provide the FFT-calculated image403. Meanwhile, the image processor107performs FFT412to a previously-created virtual pattern411(such as the virtual pattern801) to provide the FFT virtual pattern413. The image processor107performs convolution404to the FFT-subjected image403with the FFT virtual pattern413to provide a convolution-calculated image405. The image processor107performs inverse FFT406to the convolution-calculated image405to provide a restored image407. Note that the image processor107may execute the process such as the FFT for each imaging in real time. However, the data (such as the FFT virtual pattern413) that has been previously subjected to the FFT or others may be created and stored in the memory, and the data may be read in the imaging, and then, the calculation may be performed. This manner can achieve the increase in the speed.

EXPLANATION OF REFERENCE CHARACTERS