Abstract:
An image processing method. At least one light ray presented by a vector in a first affine coordinate system with a tip of the probe as an origin is projected from the tip of the probe. The light ray is intercepted from a projection surface satisfying a function in the first affine coordinate. A distance between the tip of the probe and an interception point of the light ray on the projection surface is obtained based on a rotation angle of the probe, a wavelength of the light ray, and a deflection angle of the light ray from the probe. A relationship between the first coordinate and a second affine coordinate system defined with the projection surface as a reference is obtained. Image data are acquired from the light ray reflected from the target surface presented in the first affine coordinate. The image data presented in the first affine coordinate are converted into image data presented in the second affine coordinate, and the image data in the second coordinate system are resampled by interpolating or extrapolating a gray scale.

Description:
BACKGROUND 
     Field of the Disclosure 
     The present disclosure relates in general to a spectrally encoded endoscope (SEE), and more particularly, to image rectification and image registration of image data acquired by the spectrally encoded endoscope. 
     Description of the Related Art 
     The first endoscope was invented more than 50 years ago and consisted of a bundle of optical fibers. Since then, significant progress in minimally invasive surgeries and thereby reducing the risk of complications, costs, and recovery times has been made. 
     With the advent of inexpensive and miniature CMOS sensors which are mainly used in smart phones, endoscopes are shifting from fiber bundles into designs with imaging sensors at the distal tip of a probe. One significant drawback on these CMOS sensor based endoscopes is the tradeoff between the scope diameter and the resolution. The spectrally encoded endoscope (SEE) is one of the smallest endoscope that has shown a great potential for the use in minimally invasive surgeries; however, the images of target obtained by SEE probes suffer from distortion in a two-dimensional view. Therefore, to obtain an accurate investigation of a target while maintaining the minimal invasive characteristics provided by the SEE probes, the distortion issue has to be resolved. 
     SUMMARY 
     According to aspect of the invention, an image processing method for a spectrally encoded endoscope is provided. At least one light ray is projected from a probe. The light ray is presented by a vector in a first affine coordinate system with a tip of the probe, more specifically, the geometric center of the light covered area on the grating surface at the tip, as an origin. The light ray is intercepted with a projection surface which satisfies a function in the first affine coordinate system. A distance between the tip of the probe and an interception point of the light ray on the projection surface is obtained based on a rotation angle of the probe, a wavelength of the light ray, and a deflection angle of the light ray from the probe. A relationship between the first coordinate and a second affine coordinate system is obtained, while the second coordinate system is defined with the projection surface as a reference. Image data is acquired from the light ray reflected from the target surface presented in the first affine coordinate. The image data presented in the first affine coordinate is then converted into image data presented in the second affine coordinate. The image data is then resampled in the second coordinate system by interpolating or extrapolating a gray scale. 
     In one embodiment, the first coordinate may include a Cartesian coordinate including an x-axis, a y-axis, and a z-axis extending along the probe, such that the vector of the light ray may be presented as: 
             {             x   =     r   ⁢           ⁢   sin   ⁢           ⁢   θ   ⁢           ⁢   cos   ⁢           ⁢   φ                 y   =     r   ⁢           ⁢   sin   ⁢           ⁢   θ   ⁢           ⁢   sin   ⁢           ⁢   φ                 z   =     r   ⁢           ⁢   cos   ⁢           ⁢   θ             ,           
where r is the length of the light ray, θ is the deflection angle of the light ray, and φ is an azimuth angle of the light ray. The function of the target plane satisfies ƒ(x, y, z)=0 at the interception point between the light ray and the projection plane. The azimuth angle φ may be determined based on a series of rotation angles of the probe with respect to scanning time between the rotation angles in a calibration or mapping process; the wavelength of the light ray λ may be determined based on interpolation or extrapolation from at least two distinct wavelengths and their corresponding pixel indices of pixels configured to receive the reflected light ray; and the deflection angle θ may be determined based on the wavelength of the light ray λ, the light incident angle on the grating, the grating constant, the diffraction order, and refractive indices at two opposing sides of the grating.
 
     In one embodiment, the second affine coordinate system may also include a Cartesian coordinate system. The projection plane may be a target surface to be imaged and analyzed. Alternatively, the projection plane is separate from a target surface to be imaged and analyzed. When being a separate plane from the target surface, the projection plane may include a conjugate plane of a hypothetical image plane of the target surface to be imaged and analyzed. 
     In one embodiment, multiple light rays may be projected from the tip of the probe onto the projection plane. A grayscale value may be interpolated to resample the image data presented in the second affine coordinate. 
     According to another aspect of the invention, an image processing method comprising the following steps is provided: a) projecting at least one light ray of a first color from a probe, wherein the light ray is presented by a vector in a first affine coordinate system with a tip of the probe as an origin; b) intercepting the light ray with a projection plane, the projection plane satisfying a function in the first affine coordinate; c) obtaining a distance between the tip of the probe and an interception point of the light ray and the projection plane based on a rotation angle of the probe, a wavelength of the light ray, and a deflection angle of the light ray from the probe; d) obtaining a relationship between the first coordinate and a second affine coordinate system, the second coordinate system being defined with the projection plane as a reference; e) acquiring image data from the light ray reflected from the target plane presented in the first affine coordinate; f) converting the image data presented in the first affine coordinate into image data presented in the second affine coordinate; g) repeating steps a) to f) for a light ray of second and third colors; h) overlaying the image data acquired from the light rays of the first, second, and third colors; and i) resampling the overlaid image by interpolating or extrapolating the image data presented in the second affine coordinate. 
     According to another aspect of the invention, a spectrally encoded endoscopic apparatus is provided. The apparatus comprises a light source; a probe having a proximal end optically coupled to the light source and a distal end; a grating attached to the distal end; and a spectrometer. The light source is configured to generate a light propagating through the grating, intercepting with an object surface in a three dimension, and then project onto a projection plane. The spectrometer is configured to receive the light reflected from the 3-dimensional object surface and hence the projection plane; obtain location information of image data carried by the reflected light presented in a first coordinate system with reference to the probe based on a rotation angle of the probe, a wavelength of the light, and a deflection angle of the light deflecting from the probe; transform the location information of the image data from the first coordinate system into a second coordinate system with reference to the projection plane; and resample the image data by interpolating or extrapolating a grayscale value. 
     The apparatus may further comprises a galvo motor or a motor configured to rotate the probe; a processor, and an encoder configured to control and calibrate the galvo motor or the motor, so as to determine the rotation angle of the probe. The projection plane may include a target surface to be imaged and analyzed by the spectrometer. Alternatively, the projection plane may be separate from a target surface to be imaged and analyzed by the spectrometer, and the spectrometer is further configured to generate a perspective view or a fisheye view of the image data in accordance with an angle and a distance of the projection plane with respect to the probe. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a schematic drawing of a spectrally encoded endoscopic (SEE) system and a probe according to one embodiment of this invention; 
         FIG. 2  shows the propagating path of the light projected from the probe of the SEE system as illustrated in  FIG. 1 ; 
         FIG. 3  is a graph showing the displacement curve of an oscillation motor driving the probe; 
         FIG. 4  shows the relationship between the incident angle, the diffractive angle, and the deflection angle of a light ray propagating through the grating as shown in  FIG. 1 ; 
         FIG. 5  shows the spectral distribution of the light at the spectrometer; 
         FIG. 6  shows the propagation path of the light in a three-dimensional SEE system when the target plane is a plane normal of (0, b, c); 
         FIG. 7  shows the process flow of an image rectification method according to one embodiment of this invention; 
         FIG. 8  shows the propagation path of light in a perspective view application of the SEE system; 
         FIG. 9  shows a fisheye view; 
         FIG. 10  is a flow chart of the perspective view application of the SEE system; and 
         FIG. 11  is a flow chart of the color registration process. 
     
    
    
     DETAILED DESCRIPTION 
     The following description is of certain illustrative embodiments, although other embodiments may include alternatives, equivalents, and modifications. Additionally, the illustrative embodiments may include several novel features, and a particular feature may not be essential to practice the devices, systems, and methods described herein. 
       FIG. 1  shows a schematic drawing of a spectrally encoded endoscopic (SEE) apparatus. The SEE apparatus includes a light source  10 , preferably a broadband light source such as a supercontinuum laser coupled with a proximal end of a fiber  20   a , preferably single mode fiber (SMF). The distal end of the fiber  20   a  is attached with an SEE probe  30 . The SEE probe  30  is preferably formed by attaching the fiber  20   a  to a focusing element (not shown) to form a spot several millimeters away from a tip of the probe  30 . The distal end of the SEE probe  30  includes a grating  31  to disperse the broadband light spot into a rainbow line. The sample is scanned orthogonal to the wavelength axis by rotating the SEE probe  30 . As shown in  FIG. 1 , the rotation of the SEE probe  30  is controlled by a galvo motor  40 . The diffracted light from the grating  31  is then incident on a target plane  50 . The light reflected from the target plane P may be collected by another fiber  20   b , for example, a multimode fiber. Three-dimensional information of image data of the target plane  50  carried by the light reflected from the target plane  50  is then analyzed by the spectrometer  70 . A computer  60  may control and acquire data from the spectrometer and control the motor  40 . The three-dimensional information includes the location, angle, and shape of a target to be imaged. In one embodiment, the fiber  20   b  to collect the light reflected from the target plane  50  may be coupled with the fiber  20   a  for transmitting light from the light source  10  into the fiber bundle  20  by a coupler  15  as shown in  FIG. 1 . If a non-oscillating motor is used instead, a rotary junction (not shown) is needed to connect the light source  10  or spectrometer  70  or both to the probe  30 . 
     Due to its smaller diameter of about hundred microns, the probe  30  is flexible and can be maneuvered to inspect hard-to-reach areas with a minimum bending radius of several millimeters. Other color lights and fluorescent light can also be used as the light source for the SEE apparatus. The SEE apparatus as shown in  FIG. 1  provides image rectification as a transformation process used to project light onto a designated plane. Once the projection is done, it is possible to display the image data of the reflected light of the projection in different views. The image registration is defined as the process of transforming different sets of image data into one coordinate system. The processes of image rectification and registration will be described in details as follows. 
     Image rectification is an image transformation process used to project one or more images onto a designated image plane. As shown in  FIG. 2 , the light source  10  generates a light traveling through the fiber  20   a  and the SEE probe  30 . The light is then diffracted by the grating  31  and incident on a target plane, for example, the target plane P as shown in  FIG. 2 . The propagation path of the light may be presented by a vector in a three-dimensional affine coordinate system. In the embodiment as shown in  FIG. 2 , a Cartesian coordinate system is selected with the tip of the probe  30  as the origin and the rotation axis of the probe  30  defined as the z-axis. The light projecting from the tip of the probe  30  may be presented by a vector in a three-dimensional coordinate system as Equation (1): 
                   {           x   =     r   ⁢           ⁢   sin   ⁢           ⁢   θ   ⁢           ⁢   cos   ⁢           ⁢   φ                 y   =     r   ⁢           ⁢   sin   ⁢           ⁢   θ   ⁢           ⁢   sin   ⁢           ⁢   φ                 z   =     r   ⁢           ⁢   cos   ⁢           ⁢   θ                     (   1   )               
where r is the length of the light ray, θ is the deflection angle of the light ray with respect to the z-axis, and φ is the azimuth angle, which is measured from the angle between x-axis and the projection of the light ray on the XY plane. In general, the target plane P can be presented by a function of x, y, and z, that is,
 
ƒ( x,y,z )=0  (2).
 
     From Equations (1) and (2), the length of the light ray, that is, the distance between the tip and the interception point of the light ray and the target plane r can be solved and thus the interception point of each light ray in the three-dimension is determined. This is evident if one considers a plane in the three-dimension of the interception point (x,y,z) satisfies: 
                       (     a   ,   b   ,   c     )     ⁢     (           x   -   a               y   -   b               z   -   c           )       =   0           (   3   )               
where (a, b, c) is the surface normal of the plane as shown in  FIG. 2 . It is not unlikely that the light ray may intercept with a three-dimensional target plane multiple times. In such situation, the smallest positive r is preferably selected as the interception point of the light ray and the target plane P.
 
     As discussed above, the probe  30  rotates about the z-axis. The azimuth angle φ can be determined by an encoder of a motor driving the probe, for example, the galvo motor  40  as shown in  FIG. 1 . The motor may be controlled by a processor such as a personal computer  60  as shown in  FIG. 1 .  FIG. 3  shows a displacement curve of an oscillator motor that satisfies dφ(t)/dt≈0 at time t 1  (A) and time t 2  (B). The slope of the curve between A and B is close to a constant v. The real steps of the motor are presented by the solid line, while the ideal steps of the motor are presented by the dashed line in  FIG. 3 . Calibration may be needed to determine the relationship between the real scanning steps and the ideal scanning steps of the rotation of the probe. When the curve is linear, the scanning steps φ m  satisfies: 
                     φ   m     =       Γ   N     ⁢     (     m   -     N   2       )               (   4   )               
where Γ is the total scanning angle, for example, 70° in one embodiment of the current invention, N is the number of linear portion of pixels, for example, 800 and m is the step index between 1 and N.
 
     Alternatively, it might be preferable to include an encoder to motor to record the curve as shown in  FIG. 3 , that is, what the rotation angle of the motor will be at a specific time. It also may be preferred to have a continuous rotation rather than oscillation for many applications. 
     Each wavelength of the light propagating through the grating  31  is diffracted to a distinct angle towards the target plane. Equation (5) shows the relationship between the spectral distribution of the light ray projected from the probe  30  and the incident angle and the diffractive angle of the light propagating through grating  31 :
 
 n   i  sin θ i   +n   d  sin θ d   =lGλ   (5),
 
where n i  and n d  are the refractive indices of the media through which the light propagates, including the incident side and the diffractive side of the grating  31 , respectively; θ i  is the incident angle of the light onto the grating  31 ; θ d  is the diffractive angle of the light projecting from the grating  31 ; l is the diffraction order, G is the grating constant of the grating  31 , and λ is the wavelength of the light. Further, as shown in  FIG. 4 , the deflection angle θ as a function of the wavelength of the light satisfies the Equation (6) as:
 
θ(λ)=θ i −θ d (λ)  (6)
 
     The wavelength λ of the light at the spectrometer may be calibrated based on interpolation or extrapolations from two or more wavelengths, that is, two or more color lights, and the pixel index P(λ) of each pixel by Equation (7): 
                   λ   =             λ   2     -     λ   1           P   ⁡     (     λ   2     )       -     P   ⁡     (     λ   1     )           ⁡     [       P   ⁡     (   λ   )       -     P   ⁡     (     λ   1     )         ]       +     λ   1               (   7   )               
where λ 1  and λ 2  are the wavelengths of known spectra, for example, blue and red lasers. The linearity of the spectral distribution at the spectrometer is shown in  FIG. 5 . In another embodiment, three reference wavelengths, for example, blue, red, and green lasers, or even more reference wavelengths, can be applied to a polynomial for correcting the wavelength of the collected light.
 
     By applying the deflection angle θ and the azimuth angle φ obtained from Equations (4) and (6), the distance between the tip of the probe  30  and the coordinate of the interception point (x, y, z) can be obtained. With the information derived from the light reflected from the target plane P, the image data of the target plane P can be analyzed at the spectrometer  70  and the image of the target plane P can be presented in the coordinate with the tip of the probe  30 . However, as the image (image data) is presented in the coordinate with the tip of the probe as the origin, distortion and deviation can be expected from the actual image presented in the coordinate of the target plane P itself. Therefore, the image data presented in the coordinate of (x, y, z) are resampled into a coordinate system of (α, β, γ) of the target plane P as follows. The coordinate system having its origin on the target plane P can be any Affine coordinate, Cartesian or non-Cartesian. When an affine coordinate system is selected, a transformation between the coordinate system with reference to the probe and the coordinate system of or with reference to the target plane P can be presented as: 
                     (         x           y           z         )     =       C   ⁡     (         α           β           γ         )       +     (           x   0               y   0               z   0           )               (   8   )               
where C is the transition matrix between the coordinate system of (x, y, z) and the new coordinate system of (α, β, γ) as:
 
             C   =     [           C   11           C   12           C   13               C   21           C   22           C   23               C   31           C   32           C   33           ]           
and (x 0 , y 0 , z 0 ) is the original of the new coordinate system. The transition matrix C can be derived from Equation (9):
 
     
       
         
           
             
               
                 
                   
                     ( 
                     
                       
                         α 
                         → 
                       
                       , 
                       
                         β 
                         → 
                       
                       , 
                       
                         γ 
                         → 
                       
                     
                     ) 
                   
                   = 
                   
                     
                       ( 
                       
                         
                           x 
                           → 
                         
                         , 
                         
                           y 
                           → 
                         
                         , 
                         
                           z 
                           → 
                         
                       
                       ) 
                     
                     ⁡ 
                     
                       [ 
                       
                         
                           
                             
                               C 
                               11 
                             
                           
                           
                             
                               C 
                               12 
                             
                           
                           
                             
                               C 
                               13 
                             
                           
                         
                         
                           
                             
                               C 
                               21 
                             
                           
                           
                             
                               C 
                               22 
                             
                           
                           
                             
                               C 
                               23 
                             
                           
                         
                         
                           
                             
                               C 
                               31 
                             
                           
                           
                             
                               C 
                               32 
                             
                           
                           
                             
                               C 
                               33 
                             
                           
                         
                       
                       ] 
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
     
     In the situation where the target plane  50  is a plane normal of (0, b, c), the surface of the target plane  50  can be presented as: 
                       (     0   ,   b   ,   c     )     ⁢     (         x             y   -   b               z   -   c           )       =   0           (   10   )               
From Equations (1) and (10), the length of the light ray r can be solved as:
 
                   r   =       d   2         b   ⁢           ⁢   sin   ⁢           ⁢   θ   ⁢           ⁢   sin   ⁢           ⁢   φ     +     c   ⁢           ⁢   cos   ⁢           ⁢   θ                 (   11   )               
Equation (8) can be modified as:
 
                     (         x           y           z         )     =       C   ⁡     (         α           β           γ         )       +     (         0           b           c         )               (   12   )               
The relationship between these two coordinate systems can be presented by Equation (13):
 
                     (       α   →     ,     β   →     ,     γ   →       )     =         (       x   →     ,     y   →     ,     z   →       )     ⁡     [         1       0       0           0         cos   ⁢           ⁢     θ   p             sin   ⁢           ⁢     θ   p               0           -   sin     ⁢           ⁢     θ   p             cos   ⁢           ⁢     θ   p             ]       =       (       x   →     ,     y   →     ,     z   →       )     ⁢   C               (   13   )               
where
 
               sin   ⁢           ⁢     θ   p       =     b   d           
and
 
               cos   ⁢           ⁢     θ   p       =       c   d     .           
According to Equation (11), the coordinate transformation between the coordinate systems of (α, β, γ) and (x, y, z) satisfies:
 
                     (         α           β           γ         )     =         [         1       0       0           0         cos   ⁢           ⁢     θ   p             sin   ⁢           ⁢     θ   p               0           -   sin     ⁢           ⁢     θ   p             cos   ⁢           ⁢     θ   p             ]     ⁢     (         x             y   -   b               z   -   c           )       =     (         x                 c   ⁡     (     y   -   b     )       -     b   ⁡     (     z   -   c     )         d             0         )               (   14   )               
As shown in  FIG. 6 , the deflection angle θ can be solved by Equation (6), where the diffractive angle θ d  is a function of the wavelength λ of the diffractive light ray. For a forward-view application, θ p =0 and thus b=0, c=d, the solution is simplified as:
 
                     (         α           β           γ         )     =     (         x           y           0         )             (   15   )               
In this situation, no coordinate transformation is needed from (x, y, z) to (α, β, γ) is needed. It is possible to derive from the spherical coordinate [R, θ, φ] to Cartesian (x, y, z) and resample the plane (x,y).
 
       FIG. 7  shows the process flow of the image rectification as discussed above. Assuming the SEE apparatus includes a number of pixels, for example, 1024 pixels, for receiving the image of the target plane carried by the light reflected from the target plane. The pixel index of each of the 1024 pixels is denoted as P n (λ), where n=1 to 1024 in step S 701 . In step S 702 , the wavelength λ of each light ray incident on each pixel is calibrated based on the pixel index P n (λ) of the specific pixel n as Equation (7). The deflection angle θ is derived as a function of the calibrated wavelength λ from Equations (5) and (6) in step S 703 . The azimuth angle φ can be derived based on the rotation angle of the probe presented in Equation (4) in step S 704 . In step S 704 , the deflection angle θ and the azimuth angle φ obtained in steps S 703  and S 704  is applied to Equations (1) and (2) to derive the distance R between the tip of the probe and the target plane. Steps S 701  to S 704  provide information of image data of the target plane in the coordinate system with reference to the probe. That is, the image of the target plane is presented in the coordinate system with reference to the probe. In step S 705 , information of image data presented in the coordinate system with reference to the probe is obtained. A mapping is thus established as g(P n ,m)→g(x,y,z), where g is the grayscale value. In step  706 , a transformation matrix C between the coordinate system with reference to the probe and a coordinate system with reference to the target plane is obtained. The image data is then converted based on the transformation and presented by the coordinate system with reference to the target plane in this step afterwards. The image data presented in the coordinate system with reference to the target plane are then resampled by interpolation or extrapolation for proper display in step S 707 . 
     The above computation and visualization of the image of the target plane can be done by parallel computing or GPU computing for faster processing and display. 
     In additional to the planar surface, the target may have a surface with more complex shapes such as a quadratic surface as:
 
 ax   2   +by   2   +cz   2 +2 fyz+ 2 gzx+ 2 hxy+ 2 px+ 2 qy+ 2 rz+d= 0  (16)
 
where a, b, c, f, g, h, p, q, r, d are known parameters related to the quadratic surface. In general, the surface can be presented as:
 
Σ i     x     =0   N     x   Σ i     y     =0   N     y   Σ i     z     =0   N     z     a   i     x     a   i     y     a   i     z     x   i     x     y   i     y     z   i     z   =0  (17)
 
where a i     x     i     y     i     z    is the known coefficient related to the higher order surface of the target. Depending on the target to be imaged, the three-dimensional surface can take any form of interest.
 
     In the embodiment discussed above, it is assumed that the target plane overlaps with the surface of a target object to be imaged and analyzed. In the situation that the surface of the target object is curved or even irregularly shaped, the target plane may be separate from the surface of the target object as shown in  FIG. 8 . In  FIG. 8 , considering a human eye ball is placed at the tip of the probe  30  or the camera sensor if a CMOS camera is placed, the solution of the interception point P(x,y,z) on the surface of the target object can be projected to a projection plane P′ to obtain perspective view of the target surface P. The perspective view provides the depth perception to the observer, i.e., the closer object will appear bigger. The projection plane is a plane conjugate to the retina of the human eyeball or the camera sensor. It is even possible to construct other types of views if the projection plane is a curved surface as discussed in Equations (16) and (17). One specific such application includes a wide angle camera lens, for example, a fisheye lens as shown in  FIG. 9 . 
     The image rectification process for the perspective view application is similar to those as described in  FIG. 7 . However, different from the process in  FIG. 7 , the function of the projection plane P′ is predetermined, and the interception point P′(x,y,z) is to be derived. As shown in  FIG. 10 , in step  1001 , a plurality of pixels, for example, 1024 pixels, for receiving the image data of the projection plane P′ is provided. The pixel index of each of the pixels is also provided in step  1001 . The pixel index of each of the 1024 pixels is denoted as P n (λ), where n=1 to 1024 in this embodiment. In step S 1002 , the wavelength λ of each light ray incident on each pixel is calibrated based on the pixel index P n (λ) of the specific pixel n as Equation (7). The deflection angle θ is derived as a function of the calibrated wavelength λ from Equations (5) and (6) in step S 1003 . The azimuth angle φ can be derived based on the rotation angle of the probe presented in Equation (4) in step S 1004 . In step S 1005 , the deflection angle θ and the azimuth angle φ obtained in steps S 1003  and S 1004  is applied to Equations (1) and (2) to derive the interception point P′(x,y,z). The image of the target object presented in the coordinate with reference to the probe is thus obtained. For the perspective view application, the function ƒ(x,y,z) of the projection plane P′ is predetermined and is not related to the target object to be image and inspected. If the projection surface P′ is a planar surface, a perspective view can be generated. If the projection surface is a general surface, other views including the fisheye view can be simulated. The different types of views can be generated by adjusting the distance and angle with respect to the probe  30  as shown in  FIG. 1 . For a general surface, more steps to project the three-dimensional surface to a two-dimensional plane are required as shown in  FIG. 10 . The image data is then converted and presented in the coordinate system with reference to the projection plane P′ as Equation (7) in step S 1006 . The image data presented in the coordinate system with reference to the projection plane are then resampled by interpolation or extrapolation for proper display in step S 1007 . 
     Image registration is a process of transforming different sets of data into one coordinate system. In one embodiment, the SEE image rectification discussed above is the first step for the color image registration. Once the image for an individual color, that is, an individual channel image, is properly rectified, three or more of the channel images on the same plane are mapped and overlay with each other. Steps S 1101  to S 1106  are the same as steps S 1001  to S 1004 . However, as shown  FIG. 11 , the steps S 1101  to S 1106  are repeated for three different channels, for example, for λ R , λ G , λ B  for red, green, and blue colors. Each of these three channel images are then converted and presented in the new coordinate with reference to the target or projection planes. The image data for these three channels are then overlaid in step S 1107  and the overlay result is then resampled by interpolation or extrapolation with three grayscale values for red (R), green (G), and blue (B) channels for each pixel in step S 1108 . Similarly, perspective view and fisheye view application can be generated, respectively. 
     The above embodiments describe the transformation of two Cartesian coordinate systems ({right arrow over (x)}, {right arrow over (y)}, {right arrow over (z)}) and ({right arrow over (α)}, {right arrow over (β)}, {right arrow over (γ)}). These coordinate system in general can be Affine coordinate systems. In geometry, an Affine transformation is a function between Affine spaces which preserves points, straight lines, and planes. Sets of parallel lines remain parallel after an Affine transformation. An Affine transformation does not necessarily preserve angles between lines or distances between points, though it does preserve ratios of distances between points lying on a straight line. For many purposes an Affine space can be thought of as Euclidean space, though the concept of Affine space is far more general. That is, all Euclidean spaces are Affine, but there are Affine spaces that are non-Euclidean. In an Affine coordinate system, each output coordinate of an Affine map is a linear function of all inputs coordinates. Another way to deal with Affine transformations systematically is to select a point as the origin, and then any Affine transformation is equivalent to a linear transformation (of position vectors) followed by a translation. The linear transformations discussed above can be naturally extended to Affine coordinate systems and displayed in Affine coordinate systems. However, the transformation matrix C in Equation (9) will no longer be an orthogonal matrix if the coordinate systems are not Cartesian. 
     While the above disclosure describes certain illustrative embodiments, the invention is not limited to the above-described embodiments, and the following claims include various modifications and equivalent arrangements within their scope.