Super resolution imaging of convolved light

A system and method for imaging is disclosed wherein light that is convolved by a blade is received by an imaging sensor. The received light may be convolved by a blade moving laterally across the image plane. The received light may be recorded as light data. The light data may be processed by rotations, collapses, normalizations, and applying one or more derivative filters to generate enhanced result images.

BACKGROUND

As light radiates through space it spreads due to diffraction, a wave effect. This wave effect limits the resolution of the human eye and other electromagnetic imaging devices, such as microscopes and telescopes. In 1873, physicist Ernest Abbe expressed this limit as: d=λ/2(n sin θ), where “d” is the diameter of a resolvable spot, “λ” is the wavelength of light, “n” is the index of refraction of the transmitting medium, and “θ” is the spot angle. This limit was later expressed by astronomer W. R. Dawes in terms of aperture: R=11.6/D, where “R” is the angular separation in arc seconds between two resolvable points, and “D” is the aperture in centimeters of the viewing device.

Since then, astronomers have addressed the diffraction limit by building telescopes with increasingly larger apertures, primary mirrors, and objective lenses. However, as the size of the telescope increases, the costs associated with building and maintaining the telescope skyrocket to prohibitive levels. Using these large powerful telescopes is far out of the reach of amateur astronomers, and even governments struggle with the costs of running large telescopes. Thus, there is a demand in astronomy for a new practical low-cost method for increasing telescope resolving power.

In microscopy, information on relevant attempts to increase resolution can be found in U.S. Pat. No. 5,043,570 and U.S. Pat. No. 5,731,588. However, each one of these references disclose systems that require highly controlled scanning environments, ill-suited for applications where the imaging object is not easily manipulated, such as star systems hundreds of parsecs from Earth. These systems are further limited by their complex controls, unwieldy size, extreme delicacy, slow imaging speed, and the high costs associated with building and running the systems. Thus, there is a demand in microscopy for a new practical low-cost method for increasing microscope resolving power.

DETAILED DESCRIPTION

Here below, modes for caring out mechanical convolution imaging shall be explained with reference to drawings.

Summary of the Present Embodiment

In the present embodiment, a convolving blade is used to distort light waves coming from imaging objects to reveal characteristics about the waves to enable higher resolution imaging of the objects.FIG. 1Ais a diagram showing one possible variation of the imaging system. In this variation, imaging objects135are electromagnetic sources, such as stars, that create an electromagnetic or light path125, that may be convolved by a convolution blade system145comprising one or more blades140,150and an optional labeling filter130. As the imaging objects135are imaged through a lensing system120by a sensor110, such as a CCD/CMOS image sensor, lux meter, or phototransistor, one of the blades in the system, the first blade140for example, is inserted into the light path125to create recorded wave data105which is a recording of the light path as it is changed by the blade insertion. The recorded wave data105is then sent to a computer system100for processing to generate useful output data103about the imaging objects135.

InFIG. 1Aimaging objects135are denoted by point maximas e and f, which are at a distance AD from the sensor110, where two points side-by-side (“AD”) represent the distance between them, as shown on reference line160. Point maximas can be thought of as point spread function local maximas of radiating objects. Point maximas e and f spread out on the path from A to D. At point C on the reference line160, e and f point maximas are represented by points c and d, respectively. At point B on reference line160, the same point maximas are represented by a and b. Thus, the distance between the point maximas increases as the light travels from imaging objects135to the sensor110, such that: ef<cd<ab.

The light path125defines the boundary of light or electromagnetic waves radiating from imaging objects135. The light path125may be labeled using an optional labeling filter130such as a polarizer or color filter, as explained below. The convolving blade system145in this variation may consist of materials that have characteristics opposite that of the labeling filter130which may also be considered part of the convolving blade system145. If the labeling filter130consists of a vertically aligned polarizer, then the blades140,150may consist of horizontally aligned polarizers. Similarly, where the labeling filter130may be a green light frequency pass filter, the blades140,145may consist of filters that are notched to filter out green light. If a labeling filter130is omitted from the convolving blade system145, the blades140,150may consist of opaque or translucent materials, such as steel razor blades or liquid crystals. Alternatively, a convolving blade system145may consist of a single first blade140with or without the labeling filter130, inserted once to create one result, inserted at an angle to create a flux gradient result, or inserted at different places in the light path125. The blades140,150in this variation are designed to move on an axis155that is perpendicular to the light path axis165in discrete or continuous movements. The blades140,150may be actuated using linear actuators or equivalent means of movement known in the art.

A simplified example method using this system is shown inFIG. 1B. At the first step170, the first blade140is inserted into the light path125in the direction of the reference arrow axis155at a first position B. As the blade travels farther into the light path, it cuts off more light in the path thus changing the signal received by the sensor110. At the second step175, the sensor110records this movement of the blade through the light path and sends it as recorded wave data105to the computer system100, after which the first blade is removed. At the third step180, the second convolving blade150is inserted into the light path125in the direction of the reference arrow axis155at a second position C. As the second blade150travels farther into the light path, it cuts off more light in the path thus changing the signal received by the sensor110. At the fourth step185, the sensor110records the second blade's movement through the light path and sends it as a second set of recorded wave data105to the computer system100, after which the second blade is removed. Next at the fifth step190, the computer system100uses the first and second set of recorded wave data to generate output data103regarding the imaging objects, such as their range, shape, size, plots, surfaces, or images. Another equal variation includes starting the convolving steps170,180with the blades fully inserted and measuring the light path125with the sensor110as the blades are removed. Another variation includes having only one convolving step170where the blade is inserted perpendicular to the light path125or at an angle, and one recording step180then immediately starting the fifth step190.

The following is another example of how this variation works referring toFIG. 1A, though one skilled in the art will appreciate that the ordering can again be changed and elements modified. The sensor110is directed towards the imaging objects135to record the incoming light path125. The light path125from the imaging objects135is convolved by filtering through the labeling filter130, which here is a horizontally aligned polarizer. Then, the first blade140at point B, here comprised of a vertical polarizer, is moved along an axis155so it blocks the now horizontally polarized light incident on the blade. The first blade140continues its movement along the axis155until it is fully blocking the light path125. The first blade's movement through the light path125creates a signal representing a convolution of the light path as convolved by the first blade140. Sensor110records the incoming light path as it is convolved and sends this as recorded wave data105, which may be a plurality images or a video stream, to computer system100for processing. Next, the process may be repeated with the second blade150at point C. The computer system100may then use the recorded wave data105to output object dimension data such as size, shape, and range to the imaging objects135as output data103. Alternatively the computer system100may use the recorded wave data105to enhance pre-recorded images (images of the light path un-convolved by the blades) of the imaging objects135through image processing and output that as output data103. Alternatively the computer system100may use the recorded wave data105to generate entirely new computer graphic representations of the imaging objects135as the output data103.

FIG. 2A-Dshow possible sensor outputs of convolved light path data.FIG. 2Ashows a graph plot200where the distance AD between the sensor110and the imaging objects135is greater than the distance set by the diffraction limit. In this situation, point maxima's e and f are essentially indistinguishable from one another and lie at the center point210of the graph plot200. Note, however that this graph would be virtually identical to a graph plot of a single point maxima system of just e for example, where f is non-existent. Thus an observer (not depicted) using imaging sensor110at this great a distance would have a difficult time determining whether there are two point maximas, e and f, or just one, e.FIG. 2Bshows a graph plot220, where the light path125has been convolved by the first convolution blade140at a distance of AB from the sensor110. Due to diffractive spreading, where ab>ef, point maximas a and b are further spread out on the graph plot220, and lie at an approximate first point225and second point226, respectively.FIG. 2Cshows a graph plot230, where the light path125is convolved by a second convolution blade150at a distance AC from the sensor. Due to diffractive spreading, where ab>cd, point maximas c and d are more spread out on the result slope plot230than their a and b counterparts, and lie at an approximate first point225and second point226as shown on graph plot230, respectively. Further, as the distance of AC approaches that of AD (i.e., AC→AD), point maximas c and d grow further apart if magnified properly, thereby allowing more accurate measurements of imaging objects135to be made. For example, if the imaging objects135consist of a far away binary star system where AD is many parsecs long, the second blade150at AC should appropriately be as far from the sensor110as possible to capture as much diffractive spreading of the light path125as possible. Other types of far-away blades are disclosed inFIG. 11BandFIG. 15and are discussed below.

One advantage of the imaging system disclosed is that using a blade to convolve a light path is a highly versatile way of increasing resolution. Whereas telescopes may increase resolution by increasing apertures “horizontally”, the system disclosed here increases the resolution “vertically”, positioning the blade nearer and nearer to the imaging objects, convolving them one or more times to yield useful data. Another advantage of the blade system disclosed here is that it is far easier to use than a scanning system because with the system here, the imaging objects are imaged all at once, instead of scanning them little by little which is not possible in many imaging applications. Yet another advantage of the imaging system disclosed is it may be built and customized using off the shelf parts and computer software which can keep costs very low compared to the prior art.

In equation 1-1, the function ƒ may represent the convolution blade system145. One possible function that may mathematically represent ƒ is a negative variation of the Heaviside step function wherein the Dirac delta function, δ(s), is integrated over ds, but where the limits, x and ∞, are reversed to produce a negative step-function. See equation 1-2 for ƒ in general form.
ƒ=∫x∞δ(s)ds1-2

In equation 1-1, the function g may represent the imaging objects135. One possible function that may represent g is a superposition of two Gaussian point spread functions as shown in equation 1-3, and approximated inFIG. 2D. In this equation, A and B are amplitude normalization variables; α and β, are diffraction spread normalization variables; and η is the shift variable that shifts the second point spread function laterally. Further, Ae−αx2can be thought to correspond to a point spread function of one of the imaging objects135with a point maxima e at its peak, and Be−β(x±η)2can be through to correspond to point spread function of the other of the imaging objects135with a point maxima fat its peak, but shifted laterally by η. In some applications, where there is a lack of information about the point spread functions, simple discrete functions such as unit triangles may be used to represent the point spread functions. Computer system100can model the graph plots200,220,230of the imaging objects135by applying curve fitting techniques as are known in the art, wherein A, B, α, β, and η are adjusted to allow the convolution integral, ΔB, to best fit recorded wave data105. In particular, a person of ordinary skill in the art can readily apply these techniques using software with imaging modules, such as MATLAB and Mathematica, using linear regression or similar techniques, but may also apply the above methods using custom coded imaging software as well.
g(x)=Ae−αx2±Be−β(x±N)21-3

FIG. 2Dshows a deconvolution graph plot240produced from result slope230through methods described here. On the deconvolution graph plot240, point maximas a and b now lie at the first point225and the second point226which coincide with the local maximas of plot240. The deconvolution plot may be generated by taking the derivative of ΔB with respect to the lateral distance x. The deconvolution plot is not limited to representation by derivatives of ΔB. Other methods known in the art may be applied such as dividing the Laplacian Transform of ΔB by the Laplacian Transform of ƒ or g and taking the Inverse Laplacian Transform of the quotient, as shown in equation 1-4, whereis an intermediary dimension.

FIG. 3shows a three dimensional surface300, two of space (“Range” and “Lateral Distance”) and one of energy (“Energy”), of the convolution produced by a blade, such as the first blade140, and light path (125) at a plurality of ranges denoted by points A, B, C, D. Light point maximas e and f, c and d, and a and b are also labeled on surface300along point maxima axes310. The convolution surface300may be used in correlation with the techniques described above (as shown inFIG. 2A-D) to give the complete energy-space dimensions of electromagnetic radiation from the imaging objects135. In particular, the actual distance between points e and f may be found using the point maxima proportion equation 1-5, and the range from the sensor110to the imaging objects135, AD, may be found using the point maxima equation 1-6.
ef=cd2/ab1-5
AD=AB+BC+BC(cd−ef/ab−cd)  1-6

The principles that create the two-dimensional graph plots ofFIG. 2A-D, which use one dimension of energy (“Energy”) and one of space (“Lateral Distance”) may similarly be extended to two-dimensional plots using two dimensions of space, also known as images, as shown inFIG. 4A-G. The image shownFIG. 4Ais an example image or video stream capture400from a CCD camera imaging a two-object system405.FIG. 4Bshows an image or video stream capture410of the two-object system405as it is convolved from right to left with a blade415, which is denoted by black pixels of value 0. Using the computing system100, the recorded wave data105of images or a video stream may undergo image processing, such as image stacking, or matrix operations as are known in the art to create the convolved image output420with the two-object system405now convolved with a blade140, as shown inFIG. 4C. The process may be repeated with a second blade150at different range, CD, to the two body system405to yield a second convolved image output (not shown), thereby allowing a determination of the two-object system's405dimensions as using the above disclosed techniques, which in image and video processing are analogous to motion estimation vectors, to yield a higher resolution image435as shown inFIG. 4D.

While the images inFIG. 4A-D, are grayscale with pixel values between 1 (white) and 0 (black),FIG. 4E-Gshow how the same principles may be applied for color images with channel RGB data. InFIG. 4Ea green light spot430and red light spot435have point light maximas (here spot centers) that are 28 microns apart as indicated by a reference line440. At great distances, such as AD, the two spots430,435mix together due to wave diffraction and interference, as shown inFIG. 4F. Using the convolution blade processes described above, an image sensor will receive the output shown inFIG. 4G, where color the channel data green is highly prevalent on the left side the convolved spot431in a bowed arc which follows the perimeter of the diffracted green spot432and color channel data red is highly prevalent on the right side of the convolved spot436in a bowed arc, which again follows the perimeter of the diffracted red spot437. A person of ordinary skill in the art may then use trigonometry and geometry methods to find the center points of the red and green spots using the bowed arcs.

FIG. 5A-Bshow possible result graphs where one point maxima, for example e, is of a smaller energy magnitude than point maxima f.FIG. 5Ashows a result of the convolution process in a graph plot500with a reduced plot slope around e at first point225, whereas there is a larger slope around f at second point226. A deconvolution result slope510inFIG. 5Bmore readily shows the nature of the system. Point maxima e, at the first point225, lies at a smaller local maxima than point maxima f, which lies at the second point226at a larger local maxima. This indicates that point maxima f comes from a higher or closer energy source, such as a brighter or nearer star.

FIG. 6shows three-dimensional surface300ofFIG. 3, but from a different perspective, that allows the computer system100to avoid perspective errors. From this perspective, perspective plot600corresponds to a convolution blade movement (not shown) through the surface300. The plot600appears to have a smaller magnitude point maxima605near a larger magnitude point maxima610, which may cause the computer system100to conclude that the apparent smaller point maxima605was generated from a dimmer or farther energy source and the larger point maxima610was generated by a brighter or closer energy source, but this it is not the case. Rather, the smaller point maxima605is due to a perspective skew. In reality, the graph surface300was generated by the two point maximas of the same energy magnitude and range.

FIG. 7A-Bshow a surface700where perspective may be used by the computer system100to determine ranges.FIG. 7Ashows a first point spread function710at a range of 2, a second point spread function720at a range of 6, and a third point spread function730at a range of 8, all viewed from a first perspective740where it is not readily apparent which of the point spread functions is closer or farther in the absence of the shading and numbered graph axes. A reference distance750between the peak maximas of the first and second point spread functions710,720provides a measureable data value which can be found using the above described convolution techniques. Viewed from a different perspective760inFIG. 7B, where a shift has occurred, the reference distance750between the two point maximas is now demonstrably larger, thus allowing ranges to the point spread functions710,720to be more readily determined. A method exemplifying this processes would be to first convolve spots710and720from the first perspective740to record the convolution curves shown inFIG. 2A-D, for example, and measure the distances between the maximas750, then move the imaging system laterally to the second perspective760and repeat the process of recording the distance between point maximas750, thereby revealing that the first point maxima710is at a closer range of 2, than the second point maxima720with range 6.

FIG. 8A-Cshow a similar technique using perspective to find object shapes. InFIG. 8A, a surface800represents a point spread function of an object805with an elongated form. Convolving the object using the techniques disclosed from its narrow face as indicated by a reference arrow810, will yield a convolution graph plot830shown inFIG. 8B. While convolving from its wide face as indicated by reference arrow820will yield a convolution graph plot840shown inFIG. 8C, where length face point maxima840is a plateau which indicates that the object805is longer than it is wider. The computer system100can then use the plots830,840to recreate a more accurate image of the object805using the techniques disclosed above.

FIG. 9A-Cshow another possible variation of the imaging system and possible convolution surface outputs. InFIG. 9A, imaging objects900, represented by e and f, create an electromagnetic or light path920that has a flux gradient930that decreases in intensity as D→A, which are points on reference line940. In this variation, the convolution blade943is inserted along an axis945at an insertion angle θ950to the light path axis960.FIG. 9Bshows an example surface output965where the blade935(here represented by a vertical plane) is at insertion angle of 0 degrees (degrees not diagrammed).FIG. 9Cshows an example surface output965where the convolution blade935is at an insertion angle950of 45 degrees.FIG. 9Cshows the effect that changing the convolution blade angle θ950has on the data recorded by the sensor975. Particularly, the cross-section ofFIG. 9Cfeatures an elongated lower cross-section slope971that will have a different rate of change (i.e. different first and second derivatives) than the cross section slope972inFIG. 9B. By using a single blade at an angle, the imaging system is able to forgo taking multiple measurements at different positions along light path, and instead take one measurement at an angle where the convolution of the flux gradient discloses how light path125evolves as D→A. However, even at an angle of 0 degrees (perpendicular) convolving the light path data may still yield more information than imaging the imaging objects without any blade processes.

FIG. 10A-Cshow different configurations and types of convolution blades that may be used with the imaging system. InFIG. 10A, a steel razor blade with a straight edge1000is inserted into light path1003in the direction of a reference arrow1004. Due to steel's opaque nature, no labeling filter130(as shown inFIG. 1) is required. Alternatively, a steel razor blade with partially jagged edge1005may be used to customize the convolution output, such that the jagged edge convolves the light path before the straight edge does, acting as a specialized spatial filter. Another variation utilizes polarizers as shown inFIG. 10B. There, a convolution blade1010is a vertically aligned polarizer and a labeling filter1020is a horizontally aligned polarizer. An advantage of using polarizer blades is that they cause less “smear” of the light as it is convolved.

FIG. 10Cshows another variation wherein the convolution blade consists of two orthogonal polarizers1030joined along an adjacent edge1035, where the labeling filter1040consists of a polarizer parallel to one of the polarizers used in orthogonal blade1030.FIG. 10Dshows a variation of the imaging system wherein the blade1045is held tangent to the light path axis1055. Using this configuration, single images or multiple images combined using image stacking will yield better resolution than had no blade1045been inserted. Alternatively, the blade1045held at the tangent of1055may undergo a range convolution by allowing sensor1060to record the light path data as the blade1045is moved along the direction indicated by the reference arrow1050.

FIG. 11Ashows an ocular view of another convolution blade variation. The blade1100is mounted on an aperture1110of a lensing system (ocular view) such as a telescope, and remains stationary with respect to the aperture1110. A plurality of stars1120, visible through the un-obstructed portion of the aperture1105, is then allowed to move the towards the blade1100in an apparent motion indicated by the reference arrow1130due to Earth's rotation around its axis. Alternatively, if a telescope is utilized, the apparent motion of the stars may be caused by movement of the telescope's tripod motors moving the aperture1110into the plurality of stars1120. Similarly, large objects such a cliffs, asteroids, or a school tower1140may be used to convolve the plurality of stars1120, as shown inFIG. 11B. Using the school tower1140, or other far object, as a convolution blade140(inFIG. 1) allows an observer to create a larger distance between the sensor110and the blade150thereby allowing more accurate measurements to be recorded.

FIG. 12A-Bshow a variety of optional filtering techniques. To improve the signal to noise ratio (SNR) of the imaging target1215, such as the stars in Orion's belt, a variety of spatial filters1250may be applied to block out signal from noisy nearby objects, such as the moon1230, as demonstrated inFIG. 12A. Similarly, the light coming from imaging objects135(inFIG. 1) may consist of particular frequencies1240, as shown inFIG. 12B, which is a frequency/amplitude graph of a notch filter. In this case, a notch filter (not pictured) is devised to block out unwanted noise1250. A person of ordinary skill in the art can similarly apply other filters, such as light color filters, polarizers high-frequency pass, and low-frequency pass filters generally.

FIG. 13shows a method for increasing signal energy received by a sensor, such as an image sensor1330. Often, the light path or signal energy1300coming from the imaging objects1310is highly attenuated by the time it reaches sensor1330. To improve the signal amplitude and definition, a lensing system1340may direct an out-of-focus image plane1360on the image sensor, instead of using an in-focus image plane1350, because the out of focus image plane1360causes more sensor elements1370to receive signal energy. Similarly, the image sensor could be replaced by a lux meter with a single recording element, and lensing system1340may focus all of the light path1300on the signal lux meter element.

FIG. 14shows a variation of the imaging system wherein a plurality of light sources1400are used to illuminate a target object1410. In this variation, the light sources1400may be lasers, arc lamps, or other sources of illumination. The advantage of this variation is the ability to determine dimensions, such as range, of an object with no defined features or very low SNR.

FIG. 15shows a variation of the imaging system using Earth's orbit1500. The imaging objects1510are on an imaging axis1530that intersects Earth's solstice position on June 21 at a solstice position point1540and Earth's equinox position on March 21 at an equinox position point1550. At the equinox position1550, a sensor110, lensing system120and a convolving blade system145, as shown inFIG. 1, take a measurement of a blade140as it moves through the light path1520at an angle perpendicular to the imaging axis1530, or at an angle θ therefrom. Later, at the solstice position1540, the process is repeated. It is noted that the same blade140may be used in both processes, first at the equinox position1550and second at the solstice position1540whereby the second blade150is simulated by the first blade at a different position.

After the equinox and solstice processes have been recorded producing two sets of recorded wave data105, a computer system100may generate the plots and images shown inFIG. 2A-D,FIG. 3, andFIG. 4A-Gusing the techniques disclosed above. A person of ordinary skill in the art that will appreciate that possible variations of this system include using imaging objects that are not aligned with the equinox and solstice positions1550,1540or are out of the plane of Earth's orbit1500and using trigonometry to calculate proper distances; or another variation comprising generating plots and images immediately after the equinox position measurement and foregoing the solstice position measurement. The latter method, whereby a second blade measurement is not applied, is of interest because one of ordinary skill in the art can appreciate that even a single convolution blade140measurement will reveal higher resolution than if no blade has been applied.

FIG. 16shows a flowchart1600for an approach for manipulating and restructuring image data to generate enhanced images, as according to some embodiments. At1602, the image sequence is received at the computing device100for processing. In some embodiments, the image sequences corresponds to light that has been convolved laterally. The convolved light may be recorded using an image sensor (e.g.110,FIG. 1) to generate convolved light data (e.g.105,FIG. 1). In some embodiments, the convolved light data corresponds to one or more frames (e.g. two-dimensional image data) captured over time intervals. In this way, the convolved light data may be aligned in a temporal axis (e.g. time arrow1716,FIG. 17A).

At1604, the image sequence may be treated as 3-D image block (e.g. matrix, tensor) that may be rotated by an amount of degrees, such as 90 degrees along one or more axes. In some embodiments, no rotation is implemented. In some embodiments, the image block may be rotated by slicing the block into frames of dimensions different than their original dimensions (e.g. as captured dimensions) and then recomposing the image block from the new sliced frames to rotate the image block. As an example of the frame slice rotation operation, if a two-dimensional image (e.g. frame) corresponds to the X axis (e.g. horizontal axis) and Y axis (e.g. vertical axis); multiple frames may be captured to generate an image block of X, Y, and T axes (where T corresponds to time). To rotate the image block by slicing data, the X, Y, T, axis may be sliced into individual frames of Y and T dimensions. Next, the new rotate image block may be generated (e.g. recomposed) by using the Y and T frames using a image compose operation (e.g. Image3D( ) in Mathematica).

At1606, the image block may be collapsed to generate a 2-D image matrix. In some embodiments, the image block may be collapsed by totaling the rows of the matrix along an axis to reduce dimensions. In some embodiments, the convolved light data is collapsed to generate collapse light data. Generally, a collapsing operation adds elements of the image block together to reduce the object's (e.g. image block's) dimensionality. For example, if multiple image frames are captured in the X and Y dimensions, the 3D block may be generated to correspond to the X, Y, and T dimensions. The XYT block may then be collapsed or totaled in the XY direction (parallel to T), or YT direction (parallel to X) to generate collapsed light data (e.g.1742,FIG. 17D) of one or more dimensions smaller than the image block.

At1608, the images may be pre-filtered to remove unwanted artifacts, such as noise and oversaturation. At1610, one or more enhanced or more resolved images may be generated by applying derivatives to generate enhanced images with greater resolution. In some embodiments, the derivative filters may be applied in a direction parallel to the temporal dimension (e.g. applied in a direction parallel to the T dimension). In some embodiments, derivative filters may be applied in a direction orthogonal to the temporal dimension (e.g. applied in a direction orthogonal to the T dimension). At1612, the enhanced images may be further filtered, modified, or adjusted (e.g. cropping, denoising, binarization) to generate a final output. Though the approach above recites the process in a specific order, one of ordinary skill in the art appreciates that the steps can be modified, reordered, added or removed as appropriate for a given implementation. For example, in some implementations rotating a large image block (e.g.1604), which may have dimensions such as 980×640×100000 (where 980×640 is the image resolution and “100000” denotes 100000 frames or seconds), may place too great a strain on the computing system. In some embodiments, one solution is to first collapse the image block (e.g.1606) into a smaller data set, then apply the rotation. Further combinations are modifications are explained with in conjunction withFIGS. 17A-F,19, and20.

FIG. 17A-Fillustrate example data manipulations and restructuring for processing image data to generate enhanced images, as according to some embodiments.FIG. 17Aillustrates an image block1700that has been convolved by a blade (not depicted). The illustrated image block is spread out by frames (e.g. image slices, images)1702-1712for greater clarity. Image frame1702, for example, illustrates a 5×4 pixel image where “1” may represent a white pixel and “0” may represent a black pixel. Thus for example, image frame1702may represent an image captured of a far away star where the 1's represent pixels with captured photon data and zero's indicated the absence of light.

In this example, time flows in the direction of the time arrow1716. That is, image frame1702may be a picture taken at time, t=0; image frame1704may be a picture taken at time, t=1 (e.g. 1 second), and so on, until the last example frame is captured, frame1712which is taken at t=6 (e.g. 6 seconds). Thus, the image block1700may correspond to a video sequence or a plurality of frames aligned along a temporal axis (e.g. time arrow1716).

As mentioned, the image block1700corresponds to light coming from objects convolved by a blade; in particular, in this example, the light has been convolved along a direction indicated by the blade movement arrow1714. For example, in frame1702, none of the pixels have been blocked by the blade, as indicated by the right-most column, which still has image information (e.g. the 1's in the right-most column) from the hypothetical star(s) being imaged. However, at frame1704the blade has been incremented to block the right-most column, thus the right-most column in1704is all 0's and appears black. At1706, two right-most columns have been blocked by the blade, as indicated by 0's in those columns. The blade movement continues until frame1712(the last in this example sequence), where the blade has completely occluded the star being imaged, as indicated by the 0's in image block1712. (Another illustrative example of the blade incrementing is shown inFIG. 23A-K, discussed in further detail below.) The process described here is merely an example of some embodiments for producing enhanced images. One of ordinary skill in the art appreciates that while frames, images slices or samples may be used here, continuous data (such as the data illustrated inFIG. 2A-D) may likewise be implemented. That is, instead of the incremental movements illustrated here, the blade may move forward continuously (e.g. infinitesimally, effectively infinitesimally) and the frames may be captured according to a frames per second (FPS) capture rate. Similarly, while 5×4 pixel images are sampled one per second, for a duration of 6 second total, one of ordinary skill in the art appreciates that the number of samples taken, the sampling rate, and the duration of the imaging run (e.g. total elapsed capture time) may be modified or adjusted to optimize the approach per implementation. Finally, while 0's and 1's are illustrated here as an example for black and white pixels, one of ordinary skill in the art appreciates that other data-values and/or color-spaces, such as RGB or CMYK, can likewise be implemented.

FIG. 17Billustrates an example image block rotation sequence, as according to some embodiments. There, the image block1700is rotated 90 degrees counter clockwise with respect to the Y-axis (where Y is up/down with respect to font orientation inFIG. 17B). The output of this example rotation sequence is illustrated as rotated image block1732inFIG. 17C. There, frame1728comprises each left-most column from each frame1702to1712. In other words, the left-most column of frame1728corresponds to the left most column in frame1702. Likewise, the second left-most column in frame1728corresponds to the left-most column in frame1704; the third left-most column in frame1728corresponds to the left-most frame in frame1706, etc. Similarly, frame1730comprises the right-most column of each frames1702-1712. In some embodiments, the image block1718may be rotated one or more degrees along the Y axis as illustrated in arrow1724. In some embodiments, the rotation may rotate the image block one or more degrees along the T axis (e.g. the axis along time arrow1716inFIG. 17A).

In some embodiments, the rotation may be performed as part of a image slice rotation sequence wherein the image block is partitioned into frames of different dimensions than as the frames were captured, and then recombining the frames to compose a rotated image block. For example, referring toFIG. 17A, the frames may have been captured along the X axis (horizontally) and Y axis (vertically), and over time-intervals along the temporal dimension T (e.g. the dimensions parallel to1716). Thus, in this example, image block1718may comprise XYT dimensions. The image slice rotation sequence may first partition image block1718into YT slices. For example the first YT frame (e.g. YT1) comprises the the left most columns of each frames1702-1712(e.g. YT1={{0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}} and the last YT frame (YT5) comprises the right most columns of frames1702-1712(e.g. YT5={{0, 1, 1, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}}). Next, a rotated image block (e.g. image block1726,FIG. 17C) may be generated by combining the YT frames (e.g., in this example: YT1-YT5).

FIG. 17Cillustrates the collapse operation (e.g. totaling the matrix) as indicated by the collapse arrow1732, as according to some embodiments. In some embodiments, the collapse operation collapses an n-dimensional image data object by at least n−1. In some embodiments, the collapse operation collapses an n-dimensional image data object by more than n−1 (e.g. n−2). In some embodiments, the convolved data (e.g. image block1726,FIG. 17C) is collapsed to generate collapsed light data (e.g. frame1742,FIG. 17D).

For example, the operation may reduce a 3-D image sequence into a 2-D image by summing all the elements in the desired direction. As an example, if matrix_1 is a 2×2 array: {{c1, c2,}, {c3, c4}}; then a row collapse/totaling operation on matrix_1 may sum elements together row-wise to yield: {c1+c2, c3+c4}; similarly, a column collapse/totaling operation on matrix_1 would sum elements together column-wise to yield: {c1+c3, c2+c4}. Thus the collapse/totaling operation, at least in this example, reduces the dimensions by one, from 2-D to 1-D. In this way, for example, the number of dimensions of the image block may be reduced by collapsing.

Applied here to the image block1726, image frame1728, image frame1730, and the image frames between the two (denoted by the dotted lines) are summed in the direction indicated by1732. (Note: the actual output of the collapse operation for the dataset depictedFIG. 17Cwould be the array: {{0, 0, 3, 2, 0},{0, 4, 3, 2, 1},{0, 0, 3, 2, 1},{0, 0, 0, 0, 0}}. However, inFIG. 17D-Fa different dataset, which more clearly illustrates the filtering processes, is explained/used instead.)

FIG. 17Dillustrates a filtering process that may be applied to the image block or collapse image data to enhance the result output by removing artifacts (e.g. unwanted features). In particular, the filtering operation depicted is normalization, or normalizing over a range. After the collapse operation, the resultant image may be oversaturated. In a gray-scale image, where the pixel values range from 0 (black) to 1 (white) (a gray pixel, for example, may be represented as “0.5”; or a light gray pixel may be represented as a number closer to white, such as “0.9”), pixel summation results over “1” are oversaturated. One operation to remove oversaturation is normalizing over the range, wherein all the pixel elements are divided by the largest element in the array. The filtering process1740inFIG. 4Dillustrates an example normalization process. There, after the collapse, the resultant 2-D frame1742has values over “1” which will appear oversaturated and overly “white”. To normalize the image1744and increase detail, each element is divided by the highest/maximum element value in the range. Applied here, the element with “9” is the highest value. Accordingly, each element may be divided by 9 to remove oversaturation. Thus, the pixel element with 9 becomes 1 (i.e. 9/9=1), the element with 6 becomes “0.7” (6/9=˜0.7), and so on, through-out the array, until the normalized image frame1746is generated. In some embodiments, after normalization a result image is directly output without applying derivative filters. In some embodiments, normalizing the data is skipped and the derivative filters are used to reduce artifacts since in some cases derivative filters produce the same change-based output without normalization of data.

In some embodiments, derivative filters may be applied to the image data (e.g.1748, or image block1718) to deconvolve the image, as according to some embodiments. For example, inFIG. 17Ethe normalized image frame1746may have two derivatives applied. One possible image output is generated by applying the horizontal derivative filter1752(e.g. ∂x) to the normalized image frame to deconvolve the image with respect to x. In some embodiments, an output image is generated by applying the vertical derivative filter1750(e.g. ∂y) to deconvolve the image with respect to y. In some embodiments, an image output is generated by applying both horizontal and vertical derivatives to generate a deconvolved image with respect to both dimensions. Further additional image processing (e.g. filtering, lighting, darkening, removing noise, applying additional derivatives) may be applied to modify or optimize the image, as required per implementation.

FIG. 17Fillustrates an example image frame after derivatives filters are been applied. Notably, the image may appear enhanced or in higher resolution. For example, if the imaging targets are diffraction limited binary stars, the resultant image may feature an enhanced gap1758between the two stars. Though only two points (e.g. dots, stars) are manipulated here, one of ordinary skill in the art appreciates that the same techniques (e.g. convolving with a blade, and filtering using the above approaches) yields increased resolution when the objects being imaged are not two dots. For instance, more complex objects such as microorganisms may be imaged using the same techniques and approaches.

FIG. 18A-Billustrates two binary stars (e.g. two spots, dots, points) before and after the processing using approaches, such as those described above. InFIG. 18A, the combined image of the binary stars appears as one unresolvable spot1800. As explained, due to the diffractive spreading, light from the two stars coalesces and mixes with one another. At some distance, the two stars cease appearing as two points and instead appear as one combined spot1800. Conventional approaches, such as increasing the aperture of the telescope, may be used to “split” (e.g. resolve) the two spots. However, as explained these approaches can be very expensive and/or impractical: as the distance between the observer and the binary system increases (or, as in some cases in microscopy, as the distance between the two spots decreases), the sophistication, size, cost, and complexity of the lensing systems can quickly become impractical.

FIG. 18Bshows the binary star system resolved using the above-disclosed approaches. There, a blade has been inserted into the light path from the binary star system (not depicted), and the resulting information was processed as described. The result are two distinct spots,1802aand1802b, which correspond to the physical binary star system. The example image inFIG. 18Bhas been further post processed by clipping (e.g. trimming) some of the resulting smear that the convolving processed produced. In some embodiments, the smearing may be reduced by changing the location of the blade to the imaging targets (e.g. putting the blade closer to the objects being imaged), increasing the number of increments the blade makes to pass the object (or inserting the blade in one continuous movement), increasing the number of captures per time interval (e.g. FPS), and/or reducing the amount of light radiating from the targeting objects (to increase the definition of the imaging objects, e.g. removing the glowing/saturation effects).

FIG. 19illustrates an image block1900that has under gone image processing in a different order than as illustrated inFIG. 16. In particular, the flowFIG. 16discloses first collapsing the imaging block1900, then performing derivative filtering. In contrast,FIG. 19illustrates performing derivative filtering, such as applying the vertical derivative filter1902and/or the horizontal derivative filter1904directly to the image block1900, before a collapse. Depending on the parameters of the implementation (e.g. whether computational environment can support the heavier load of performing filtering on a image block instead of a 2-D image, or whether the implementation requires dynamic image manipulation of a dynamic video feed), it may be more appropriate to perform derivative filtering directly on the image block1900, instead of first collapsing the image block.

FIG. 20illustrates performing the collapse on the image block in different directions, than as illustrated inFIG. 17C. As is known in the art, matrices can be rotated and collapsed in different directions. WhileFIG. 17Cillustrates collapsing an image block out of the page (towards the viewer), one of ordinary skill in the art appreciates that collapses may occur in different directions. For instance, inFIG. 20, depending on the imaging objects and and implementation goals, it may be optimal to collapse in the vertical direction2002or from left-to-right2004(as opposed to from right-to-left, as illustrated inFIG. 17C). Rotations, as illustrated inFIG. 17B, may be used in concert with collapses in various directions to optimize the approach describedFIG. 16, per implementation or environment.

FIG. 21Aillustrates imaging objects2100that may be imaged enhanced using the approaches disclosed herein. There, a small imaging object2104(e.g. a planet) orbits a larger imaging object2102(e.g. a host star). Generally, host stars are far more luminous than their orbiting planets. In such situations, it is typically extremely difficult to image the darker planets as they orbit around their host stars. One technique to detect such planets, the transit method, works by plotting the total amount of light emanating from the host star on a light curve (e.g. a 2-D plot with light intensity as a function of time) and then looking for dips in the light curve that occur when the planet transits (passes in front of) the host star. While this method can be used to detect far-away exo-planets, it produces only light curve plots, not richer image outputs.

The approaches discussed above can be used to generate a richer image output of transiting targeting objects, such as those shown inFIG. 21A. In particular, instead of using a blade (e.g.140,150), the transiting planet2104and/or the host star2102can be used as a blade. For instance, if the transiting planet2104is used as a blade, multiple images may first be captured as the transiting planet moves from darkness (outside the disk of the host star2102), moves across the face of the host star2102, and then moves back into darkness. Because, in this example, the blade (e.g. the transiting planet2104) is much smaller than the object it is obscuring or convolving (e.g. the host star2102), the resultant image output may not be as detailed as an image output when the blade completely blots out the imaging targets (See imaging outputFIG. 18B); however, useful information and image details can still be generated applying the approaches discussed above (e.g.FIG. 16).

FIG. 21Billustrates an image output for the transiting objects2100, as according to some embodiments. There, the image output2106is result of a rotation1604(with respect to T), collapse (seeFIG. 17C), and normalization (seeFIG. 17D), however no derivative filtering has been applied (seeFIG. 17E). The vertical height (relative to the text orientation inFIG. 21B) of the image corresponds to how many frames (e.g.1702-1712) were imaged as the planet transited the host star. From the resultant image2106information can be determined and/or output. In particular, for example, in resultant image2106“Dp” corresponds to the diameter of the planet2104and Ds corresponds to the diameter of the host star2102.

FIG. 21Cillustrates an image output for the transiting objects2100, wherein a horizontal filter has been applied. There, the resultant image output2108is a result of a rotation1604(seeFIG. 17B), collapse (seeFIG. 17C), normalization (seeFIG. 17D), and application of a horizontal derivative filter1752(seeFIG. 17E). From the resultant image2108information can be determined and/or output. In particular, for example, in resultant image2108“Rp” corresponds to the radius of the planet2104and Ds corresponds to the radius of the host star2102. Although the information such as the radius, or diameter are observable inFIG. 21B, as illustrated inFIG. 21Cthe information is more apparent due to the changes emphasized by the derivative filtering. For instance, the border of the host star is made more prominent by a light band2140(created by applying the derivative to the transition of the black background area to the light star area). Similarly, the border of the planet is made more prominent by light/dark bands2142. The color/grayscale value of the bands (e.g. features) inFIG. 21Cis based at least in part on the direction of derivative filtering applied, as discussed in more detail with reference toFIG. 21D.

FIG. 21Dillustrates an image output for the transiting objects2100, wherein a vertical filter has been applied. There, the resultant image output2110is a result of a rotation1604, collapse (seeFIG. 17C), normalization (seeFIG. 17D), and application of a vertical derivative filter1750(seeFIG. 17E). From the resultant image2110information can be determined and/or output. In particular, for example, in resultant image2108“Dp” corresponds to the diameter of the planet2104(the diameter of the host star, Ds, has been removed by the vertical derivative filter because, as shown inFIG. 21B, there is little change in the image vertically with respect to the host star).

As mentioned the pixel values (e.g. color values, grayscale values) can yield useful information about the objects being imaged and their conditions. For instance, dark disk2114corresponds to the planet2104moving from complete darkness into “light” (obscuring part of the host star2102). Similarly, light disk2112corresponds to the planet2104moving from the light-face of the star into darkness.

FIG. 21Eillustrates an image output for the transiting objects2100, wherein a vertical and horizontal filter has been applied. There, the resultant image output2120is a result of a rotation1604(seeFIG. 17B), collapse (seeFIG. 17C), normalization (seeFIG. 17D), and application of a vertical derivative filter1750and horizontal derivative filter1752(seeFIG. 17E). From the resultant image2120information can be determined and/or output. In particular, for example, in resultant image2120the diameter of the planet can be ascertained as explained above (e.g. the diameter corresponds to the width of the disks). Additionally, because both the horizontal and vertical derivatives are applied additional information can be extracted observing the pixel values of the disks2122,2124; such as, for example the characteristics of the planet (e.g. width as correlated to disk2122dimensions), and characteristics of the transit itself (e.g. time duration as correlated to the rate of change from dark to light pixels).

FIG. 22illustrates an approach for using a blade moving relative to the imaging target frame of reference in order to enhance imaging results, as according to some embodiments. There, an ocular view2200is illustrated comprising two imaging objects2202and a blade2204. In this example, for clarification purposes, it can be assumed that the blade2204is mounted directly on telescope as a stationary blade (such that, in order to convolve imaging objects the telescope is pivoted/rotated, thereby moving the imaging objects into the blade, or the stars may be allowed to move into the blade due to Earth's rotation). As illustrated, the imaging objects2202can be two binary stars that move from left to right at the apparent speed of 465 m/s. As the Earth rotates, an observer on a night-side will observe the imaging objects move from the left portion of the side, through a celestial equatorial arc.

However, though the imaging objects2202appear to move slowly, under magnification in the ocular view2200they can move quite fast, staying in view for only a matter of seconds. As such, the imaging objects will convolve—going from visible to behind the blade—quickly. In this situation a camera imaging at 30 frames per second (FPS) may not capture enough frames or information of the convolution process and a “thin” image with little may result. One approach for enhancing the image is to increase the amount of frames per second (e.g. 210 FPS) as the imaging objects go behind the blade; thereby increasing the amount of information, detail, and/or “thickness” of the resultant image output.

In some embodiments, moving the blade system in relation to the imaging objects may be used increase the frames and/or information captured to generate enhanced result images. For instance, as illustrated inFIG. 22, the blade2204is moved along the same celestial East-to-West path as the imaging objects2202but at a reduced speed. In particular, while imaging targets2202move through the sky at an apparent speed of 465 m/s, the telescope system with the blade2204is tracking it at a speed that corresponds to 465 m/s, as relative to the imaging objects. As such, the imaging objects are tracked in a lagging manner (e.g. the imaging objects innate speed is used in concert with the blades speed to produce a combined convolved rate/speed). The objects will stay in the field-of-view of the oculus (e.g.2200) longer, and will be observed as moving slowly toward the blade2204. In this way, the speed at which the imaging objects2202can be slowed down and/or controlled. Thus, an imaging camera can yield more frames per convolution process and a enhanced resultant image can be generated.

FIG. 23A-Killustrate a blade convolving imaging objects in a grid of rows and columns, as according to some embodiments. There, two imaging objects (e.g. a binary star system) are illustrated in a field of view of an imaging device (not depicted). A blade2304convolves the imaging objects, in this example, from right to left in increments denoted by the column widths (e.g. C1, C2, . . . C10). In some embodiments, the resolution of the output (e.g.FIG. 18B) is a function of column width. That is, for example, the smaller the column widths or increments that the blades move forward may yield higher resolution outputs (assuming that for every increment forward an image frame is captured).

In some embodiments a frame capture balance exists per imaging sensor, if the cells (e.g. R1, C0correspond to pixels. For instance, if a CCD sensor is 480 pixels wide (480 1-pixel wide columns), a blade may move in increments of 1-pixel (omitting dot-pitch distances). In such cases, multiple frames per increment may be captured, the resolution of the sensor may be increased, or the frames may be captured using analogue imaging methods and later converted into digital image data. In some embodiments, one column per increment is occluded using a blade consisting of liquid crystals or other mediums/mechanisms to occlude one column at a time. In some embodiments, in cases where the blade moves at increments smaller than a pixel width, an increase in resolution may still occur due to the wavelengths of light (which are far smaller than a pixel width) still diffracting around the blade and producing useful convolution data per pixel.

In some embodiments, for every increment/column the blade moves forward, the number of frames captured varies proportionally. For example, the blade can “move” faster than an imaging device captures: at C1, capture frame; at C2, no capture; at C3, capture frame, etc. This process, where the blade moves faster than the device may yield an image result output (e.g.FIG. 18B) of lower resolution (e.g. more errors in the image).

In some embodiments, the blade may move “slower” than the imaging device captures: at C1, capture two frames; at C2, capture two frames; at C3, capture two frames. This process, where the blade moves slower than the imaging device may yield image result outputs (e.g.FIG. 18B) that contain potentially more details (e.g. increased visibility of two distance objects split up), however the image may be distorted or smeared (e.g convolved) in the direction of the blade movement. For instance, the apparent lateral distance between the two imaging objects may be exaggerated.

In some embodiments, the blade may move in step with the imaging devices: at C1, capture one frame; at C2, capture one frame; at C3, capture one frame. This process, where the blade moves in step (e.g. unity) with the imaging device captures, may produce an accurate image result: the distance between two imaging objects corresponds to actual lateral distance between one another. In some embodiments, when the blade moves faster or slower than the capturing image device, post processing operations, such as compressing/expanding the image or using interpolation techniques, may be implemented to correct the image to proper dimensions and/or details.

Referring toFIG. 23F, in some embodiments, one or more frames are captured when the blade2304is located at or near the center point between imaging objects (e.g. blade at C5, inFIG. 23F). In some embodiments, two or more frames are captured when the blade2304is located at or near the center point between imaging objects (e.g. blade at C5, inFIG. 23F) so as to increase the resolution (e.g. the objects split, seeFIG. 18B).

In some embodiments, the angle at which the blade approaches (e.g. the blade's angle of attack) the imaging objects is orthogonal, as illustrated inFIG. 23A-K. In some embodiments, the imaging objects form an axis that is not orthogonal to the blade. For instance, if two binary stars have a 0 degree position angle and a stationary mounted blade is used, the blade's angle of attack will be parallel to the binary stars. In cases such as this, the blade may be adjusted and/or rotated so that the angle of attack is orthogonal or closer to orthogonal to the binary stars. In some embodiments, the blade's angle of attack will be at non-orthogonal angles (e.g. 45 degrees) to the imaging objects as the imaging is performed. In cases such as this, such as compressing/expanding the image or using interpolation, may be implemented to correct the image to proper dimensions and/or details, such as those that would result if the blade's angle of attack is orthogonal.

Generally, the more frames that are collected as a blade convolved the targeting objects, the less “thin” the resultant image output may be. In some embodiments, to increase the “thickness” (e.g. image height, inFIG. 21B) of the resultant image multiple frames per blade position may be captured. For instance, assumeFIG. 23A-Kcorresponds to a sample sequence wherein one frame is captured per column increment moved forward. In this case the final result image block (e.g.1718) will comprise ten frames: a first frame capture at C1, a second frame captured at C2, a third frame captured at C3, etc., to C10. After applying the approaches disclosed above (e.g.FIG. 16), the final result image will be 10 pixels wide; that is, the image result will be very thin. In cases such as this, more frames may be captured (e.g. higher FPS used in the imaging device) to increase the thickness, details, and/or resolution of the result image. For instance, if ten frames were taken per column increment (e.g. 10 frames captured at C1, 10 frames captured at C2) the resultant image will be 100 pixels wide (10 frames captured*10 columns).

In some embodiments a continuous smooth moving blade is implemented that does not move forward with discernable increments/movements. In cases such as this, the thickness, details, and/or resolution of the result image may be increased by increasing the FPS of the imaging device. For instance, if the imaging objects are two stars and the blade is a building (seeFIG. 11B), increasing the FPS to210from 30 will result in an increase in resolution (after the processes such as those shown inFIG. 16are applied).

FIG. 24shows a flowchart2400for an approach for producing enhanced imaging, such as those inFIG. 18B, without operations such as matrix rotations and collapses, as according to some embodiments.FIG. 24is a high-level view of multiple approaches that may use flat-based schemes (e.g. algorithms). For instance, instead of collecting one or more images into an image block, collapsing, rotating and filtering (e.g. as illustrated inFIG. 16), the approaches below enable direct data manipulations of the frames, as according to some embodiments.

Generally, at2402the blade is incremented forward. For instance, inFIG. 23Cthe blade2304is incremented forward into the C3column. The result isFIG. 23Dwhere the blade2304obfuscates the C3column. Though the blade is illustrated moving in discrete increments, one of ordinary skill in the art appreciates that the blade may move in continuous movements as well. In that case the increment captures may correspond to the frame capture rate of the imaging device. That is, instead of: increment blade, capture frame, increment blade, capture frame; the process may comprise: move blade forward continuously, capture multiple instances of frames per unit time (e.g. FPS). However for the sake of simplicity in this example, inFIG. 24every increment forward by the blade is followed by one frame capture by the imaging device. Thus, at2402the blade moves forward one increment (e.g. over the C3column) and one frame of the imaging objects is captured.

At2404an approach based at least in part on a flat-based scheme is applied. In some embodiments, the term flat or flatter connotes that data manipulations are performed directly on the images as they are captured (or in post-processing) without waiting to accumulate an image block (e.g.1718), as according to some embodiments. In some embodiments, the term flat may be treated dimensionally. For instance, if the the result image is in n dimensions, the flat process may result in an image output in n dimensions without accessing or manipulating operations in n+1 dimensions. For example, an image (e.g.FIG. 18A) is in two dimensions (e.g. X and Y), thus n=2. Using a flat process (e.g. flat-ased scheme) a result image can be generated (e.g.FIG. 18B) without using an image block, which is in three dimensions (e.g. n=3; X, Y, and T (time)). The result of the flat-based scheme generates information about what was in the column just incremented over. That is, for example, as the C3column was incremented over, the flat-based scheme at this point generates information and image detail for what will be in the C3column of the final result image. These intermediate results may be stored in a database as the process continues, as according to some embodiments.

At2406, the imaging system determines whether there are more increments to move through. Applied to the series depicted inFIG. 23A-K, since the C3column was just obfuscated and C4-C10remain, the determination at2406is that there are move increments to move through. Thus the process may loop to2402where the blade may be incremented, imaged, processed using a flat-based scheme, and looped again accordingly. If there are no more increments to move through, then the process may continue to2408. For example, once the blade increments to C10, an image is captured, and/or a flat-based scheme is applied, a final image may be generated using the information generated for each increment/column (that was stored at each step).

FIG. 25shows a flowchart for an approach using a baseline comparison (e.g. baseline comparator) scheme for generating enhanced imaging, as according to some embodiments. At2502a baseline data result is generated. In some embodiments a baseline result can be generated using the approach shown inFIG. 16but where the imaging target is a single star (e.g. point, dot). For example, inFIG. 23Bonly one star2560is visible. Applying the process shown inFIG. 16to the imaging object2560inFIG. 23Bwill still result in a single point image output, such as those illustrated inFIG. 2B, orFIG. 18A. In some embodiments, the baseline result is generated and stored as a data structure for later use/analysis.

At2504, the blade is incremented. For example, inFIG. 23Cthe blade2304can be incremented over the C3column; the result is shown inFIG. 24Dwhere the C4column is obfuscated by the blade. Once the blade has incremented a frame may be captured (as explained above, though one frame capture per increment is used here for simplicity, multiple captures and/or continuous blades can be implemented).

At2506the difference between the C3column in the baseline (single-star) result is compared to the C3column of the capture currently underway (e.g. the imaging process of double-stars2302aand2302b). At2308if the pixel values (e.g. a pixel located at Row1, Column3) of the current capture are the same as the pixel values in the baseline image, then no change is made to the pixel values of the current capture at2512. However, if the pixel values of the current capture are different (as evidenced by such processes as subtraction, or comparing the pixel values for changes/difference) than the pixel values of the current capture, then deductions may be made to the C3column of the current capture. After the deductions are made, the column's values may be stored as a result column data structure.

At2514a determination is made whether there are more columns to increment through. If there are, then the process loops to2504where the blade is incremented and the process repeats. Once there are no more increments to be made, then at2516a result is generated. In some embodiments, a result may be generated by retrieving the result columns and using them to create a composite image.

As an example of this process, assume, as above, the C3column is incremented over. The determination/adjustment process may proceed as follows (where Rows and Columns are referred in short: “R1, C1”=row1, column1). Is R1, C3in the current capture (seeFIG. 23C) different from R1, C3of the baseline (seeFIG. 25B). Analyzing R1, C3ofFIG. 23Cand R1, C3ofFIG. 25Bit is determined they are the same. Thus, at2512no change occurs to that pixel value. The same process is repeated for other pixels in the column: is R2, C3ofFIG. 23Cthe same as R2, C3ofFIG. 25B, etc. The result of the example analysis at2508for the C3column is that the entire C3column ofFIG. 23Cis the same asFIG. 25B; accordingly, no changes are made (e.g.2512).

However, if there are changes between the baseline data and the current capture data deductions may be made. For example, if the blade is incremented over the C5column there is a difference between the C5areas (e.g. pixels) in the baseline image data (e.g.FIG. 25B) and current capture (e.g.FIG. 23EtoFIG. 23F). In particular, for example, the R3, C5pixel inFIG. 23Eis different than the R3, C5pixel inFIG. 25Bbecause some light from the second object2302ais visible to R3, C5inFIG. 23E, but missing from R3,C5inFIG. 25B(asFIG. 25Bdoes not contain a second imaging object). In some embodiments the two pixel values (imaging areas) can be subtracted to determine the difference. The difference, or a value proportional to the difference may then be deducted from the R3, C5pixel value inFIG. 23Eand the result stored as a new result column C5′ with modified values. Once the process is complete (there are no more iterations and all result column data has been generated), the result column data items can be combined (e.g. added, placed side-by-side) to form a new composite image (e.g.FIG. 18B).

As one of ordinary skill in the art appreciates, of the approach shown inFIG. 25Acan be varied or customized. For instance the baseline image inFIG. 25Bassumes that the objects in the baseline image and current capture are aligned (e.g.2560inFIG. 25Bis aligned with2302binFIG. 23E). Additionally, object2560andFIG. 2302bare of the same size. Additionally, object2560and object2302bradiate light the same amount (e.g. denoted in the figures by the sharp transition from black to white at the stars border, in other words, they shine the same amount). Accordingly, per implementation and/or objects being imaged in the current capture process, the baseline data may be adjusted to yield a more enhanced image. For example, the baseline object's size may be increased, its radiation flux/shine values may be adjusted to be more or less luminous, and/or its alignment with respect to the frame being captured (e.g.2300) can adjusted. Additionally, the amount of deduction per change detected can be modified per implementation. In some embodiments the difference between R3/C5inFIGS. 23E and 25Bmay be determined and used directly to modify the values of the current capture R3/C5. In some embodiments the number detected may be a function (e.g. proportional) of the difference detected.

In some embodiments, no baseline image may be implemented; instead the flat-based scheme may comprise examining row data (e.g. the pixel values of row3, inFIG. 23A) after the blade as incremented through the frame (e.g. from C1to C10). From the analysis, it can be determined whether the row data corresponds to single star 1-dimensional plot (e.g. which would appear similar toFIG. 2A) or a double star plot (e.g. which may appear similar toFIG. 2C). In other words, the row data is examined to determine if the row data before the capture is different than the row data after the capture. For example, before the capture process R3, C5inFIG. 23Amay appear as completely filled by the combined output/shine from2302aand2302b. However, because the blade2304physically occludes part of2302bas shown inFIG. 23E, the value of R3/C5will be different than that of the R3/C5value inFIG. 23A. As such, the difference can be analyzed and an appropriate modification (e.g. deduction) can be made to the R3/C5pixel value. The modified pixel values may then be store as column result data and combined to generate a composite image, as explained above.

FIG. 26shows an flowchart for an approach for determining the position, orientation, and/or alignment of objects being imaged, as according to some embodiments. At2602the imaging system is rotated by θ degrees. For example, referring toFIG. 23A, assume that the blade2304and imaging objects2302aand2302bare orthogonal to one another (e.g. an axis through the centers of2302aand2302bforms a right angle with the edge of the blade2304). The orthogonal orientation can be assigned as 0 degrees. Then at2602the blade is rotated by some amount, such as one degree, so that there is a one-degree angle between an axis through the objects (2302aand2302b) and blade2304.

At2604an approach such as the sequence illustrated inFIG. 16may be applied. That is, the blade is moved through the line of sight to the imaging objects as the imaging device captures one or more image frames; the resulting image block is rotated, collapsed, and filtered to output a result image, such as the one illustrated inFIG. 18B. At2606, the result image may be stored as data for later analysis. At2608a determination is made whether there are more rotations to be made. In some embodiments, there are more rotations if the blade system has not rotated 360 degrees. In some embodiments, there are more rotations to be made if the blade system has not rotated to 90 degrees, 180 degrees, or other amounts. Assuming in this example, there are more rotations if the blade has not rotated to 90 degrees (relative to the axis through2302aand2302b), then the flow loops to2602where the next rotation is made. For example, in the first iteration the blade orthogonal to the imaging objects, the imaging process (e.g.FIG. 16) is performed on the orthogonal blade and the image result is stored; in the second iteration the blade is rotated by one degree, the imaging process is performed, and the image result is stored; in the third iteration the blade is rotated by another degree so that it forms a two-degree angle with the axis through the objects, the imaging process is performed, and the image result is stored; and the process continues to the desired or selected rotation limit: here 90 degrees.

At2610the stored images are analyzed to determine imaging object orientation, as according to some embodiments. The analysis may include determining which angle or alignment generates a result image with the greatest amounts of change. At2616data may be output as a result. In some embodiments the data output may include an image of the angle that produced the greatest change. In some embodiments, the data output may include descriptive data that describes the angles, orientation, and/or sizes of the objects being imaged.

FIGS. 27A-Dshow example outputs of the process shown inFIG. 26, as according to some embodiments. The imaging objects imaged inFIGS. 27A-Dare an orbiting exoplanet system, similar to the objects illustrated inFIG. 21A. In particular,FIG. 27Ais an result (e.g.2604,FIG. 26) of the planet2104transiting the host star from left to right (with respect to the font orientation inFIG. 26). The left to right action corresponds to an orthogonal blade2304approaching objects2302aand2302bat 90 degrees. The result of the result/collapse/filter process (e.g.2604) generates the result image inFIG. 27Awith two prominate light and dark bands2701and2700, that correspond to the planet2104entering the disk of the host star and leaving the disk of the host star, respectively.

Similarly,FIG. 27Bis an example result with the planet2104transiting the host star2102at 30 degrees, with respect to the font orientation inFIG. 21A;FIG. 27Cis an example result with the planet2104transiting the host star2102at 60 degrees, with respect to the font orientation inFIG. 21A; andFIG. 27Dis an example result with the planet2104transiting the host star2102at 30 degrees, with respect to the font orientation inFIG. 21A.

As is illustrated, the dark bands (e.g.2701,2703,2705,2707) that correspond to the planet2104entering the disk of the host star2102and the light bands (e.g.2700,2702,2704,2706) that correspond to the planet2104leaving the disk of the host star2102become thinner, and the position may change as the orientation of the blade (e.g. planet2104) changes in relation to the targeting objects (e.g. host star2102). In this way, the analysis at2610(FIG. 26) can analyze how imaging object orientation with respect to the blade changes. For example, comparingFIG. 27A(e.g. 0 degree, orthogonal blade) withFIG. 27Cthe dark band2701is broader than the dark band2705. Since the planet transited at the same speed inFIGS. 27A and 27C, the broadness of the dark band is then not a result of planet more slowly crossing the cusp (e.g. edge) of the host star. Rather, the broadness of the dark band is due to the angle in which the matrix collapse process occurred in relation to transiting angle of the planet across the star. In other words, at the orthogonal angle the change caused by the transit is the greatest; however as the transit angle goes from 0 to 90 degrees, the output caused by the matrix collapse process lessens trigonometrically, the components of which may be calculated using cosines and/or sines. In this way, object alignment may be determined.

FIG. 28illustrates an example blade with varying characteristics, as according to some embodiments. There, the blade2800is illustrated as a diffraction grating with varying grating widths. In particular, the blade2800comprises gratings (e.g. grooves, slits)2802that become closer and closer to one another from left-to-right (with respect to the fonts inFIG. 28). That is, the width between the gratings (e.g. X1) becomes smaller as X1goes to Xn.

As is known in the art, diffraction gratings cause incident light to diffract through gratings at angles proportional to the wavelength of light. Implementing the blade2800by itself or using labeling filters (seeFIG. 1) results in image output effects that increase as the objects are increasingly obscured by the blade2800. For instance, as imaging objects2804move behind the blade, the way light coming from the objects diffracts through the blade changes in proportion to the decreasing width between the grating. In some embodiments, the distance between the gratings is varied, as illustrated inFIG. 28. In some embodiments, the width of the grating slits is varied and the distance between the gratings is not varied. In some embodiments, other characteristics such as the amount by which light is filtered out through color filters or polarization increases or decreases laterally across the blade (e.g. in the same way the grating distances vary inFIG. 28). In this way, the blade may remain stationary and the objects may be allowed to move into the blade. This action will still result in changing image output that contain image information about the imaging. The results of the imaging sequence is then filtered and processed using the above methods (e.g.FIG. 16).

In some embodiments, the blade corresponds to light illumination that sweeps across the imaging objects. That is, referring toFIG. 23A, the imaging objects can be illuminated with a light (e.g. laser light) such that the entire grid2300is illuminated. Next the light may removed from the grid column by column, in a sweeping manner. That is, C10is de-illuminated, then C9is de-illuminated, etc, until the entire image is dark. The resulting images can be processed using the above approaches.

In some embodiments, the blade corresponds to an opaque or filter-able flowing object/blade (e.g. liquid) that flows across the surface of the imaging objects. As flowing blade flows across the imaging objects light from the imaging objects is blocked or filtered in proportion to the movement of the flowing blade across the imaging objects. One or more imaging frames may be captured and filtered to generate enhanced image results, using the approaches above (e.g.FIG. 16).

In some embodiments, the blade corresponds to a linear blade of light that sweeps the imaging objects. That is, for instance, the blade may correspond to a single column, e.g. “C10” inFIG. 23A, the illuminates the objects in a sweeping movement, e.g. from C10to C1, illuminating column by column.

In some embodiments, the blade is placed at different points in the light path, such as between an objective lens (e.g.120,FIG. 1) and the imaging device (e.g.110,FIG. 1). In some embodiments, the aperture of the imaging device is increased to compensate for the size of the increments the blade can move forward. For example, if the blade is mounted on a liner actuator that moves forward in increments of 0.03 mm, the rate at which the imaging objects can be sampled (e.g. one frame per 0.03 mm movement of the blade) may be limited. As described above, generally, finer movements of the blade (or the faster frame rates using a continuous blade) result in higher resolutions in imaging outputs (e.g.FIG. 18B).

Thus, what is disclosed is an improved approach for generating higher resolution images using blades and light path convolutions.

FIG. 29is a block diagram of an illustrative computing system2900suitable for implementing an embodiment of the present invention for performing intrusion detection. Computer system2900includes a bus2906or other communication mechanism for communicating information, which interconnects subsystems and devices, such as processor2907, system memory2908(e.g., RAM), static storage device2909(e.g., ROM), disk drive2910(e.g., magnetic or optical), communication interface2914(e.g., modem or Ethernet card), display2911(e.g., CRT or LCD), input device2912(e.g., keyboard), and cursor control.

According to one embodiment of the invention, computer system2900performs specific operations by processor2907executing one or more sequences of one or more instructions contained in system memory2908. Such instructions may be read into system memory2908from another computer readable/usable medium, such as static storage device2909or disk drive2910. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and/or software. In one embodiment, the term “logic” shall mean any combination of software or hardware that is used to implement all or part of the invention.

The term “computer readable medium” or “computer usable medium” as used herein refers to any medium that participates in providing instructions to processor2907for execution. Such a medium may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as disk drive2910. Volatile media includes dynamic memory, such as system memory2908.

In an embodiment of the invention, execution of the sequences of instructions to practice the invention is performed by a single computer system2900. According to other embodiments of the invention, two or more computer systems2900coupled by communication link2915(e.g., LAN, PTSN, or wireless network) may perform the sequence of instructions required to practice the invention in coordination with one another.

Computer system2900may transmit and receive messages, data, and instructions, including program, i.e., application code, through communication link2915and communication interface2914. Received program code may be executed by processor2907as it is received, and/or stored in disk drive2910, or other non-volatile storage for later execution.