METHOD, DEVICE, AND SYSTEM FOR COMPUTING A SPHERICAL PROJECTION IMAGE BASED ON TWO-DIMENSIONAL IMAGES

An image projection method for generating a panoramic image, the method including the steps of accessing images that were captured by a camera located at a source location, and each of the images being captured from a different angle of view, the source location being variable as a function of time, calibrating the images collectively to create a camera model that encodes orientation, optical distortion, and variable defects of the camera; matching overlapping areas of the images to generate calibrated image data, accessing a three-dimensional map, first projecting pixel coordinates of the calibrated image data into a three-dimensional space using the three-dimensional map to generate three-dimensional pixel data, and second projecting the three-dimensional pixel data to an azimuth-elevation coordinate system that is referenced from a fixed virtual to generate the panoramic image.

Herein, identical reference numerals are used, where possible, to designate identical elements that are common to the figures. Also, the images in the drawings are simplified or illustration purposes and may not be depicted to scale.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1depicts diagrammatically a method of generating panoramic images according to a first embodiment of the present invention, withFIG. 2depicting a scenery300that is viewed by a camera107of camera unit100of an imaging system1000(FIG.4) for performing the method ofFIG. 1. As shown schematically inFIG. 2, two-dimensional (2D) images411-413,421-423, and431-433of a scene200are captured and stored by imaging system1000by using camera unit100that captures images411-413,421-423, and431-433from camera location T. Camera unit100may be composed of a plurality of cameras107that may rotate or may be stationary, for example as shown inFIG. 2a camera107that is rotating with a rotational velocity Ω by use of rotational platform105installed as a payload on an aerostat such as but not limited to a blimp, a balloon, or aerodynes such as but not limited to flight drones, helicopters, or other manned or unmanned aerial vehicles (not shown). In addition to rotational velocity Ω being the azimuthal rotation, there is another rotation R of the inertial navigation system (INS) that parameterizes the overall orientation of imaging system1000, including the parameters roll r, pitch p, and heading or yaw h (not shown). R(t)=[r, p, h] can specify the current orientation of imaging system1000in the same way as location T(t)=[x, y , z] specifies the position. It is also possible that multiple images are captured simultaneously from multiple cameras107circularly arranged around location T with different angles of view all pointing away from location T. The actual geographic position of camera unit100will usually not be stationary, but will follow a trajectory [x,y,z]=T(t) that varies over time. This is due to the fact that the aerial vehicle carries camera unit100cannot be perfectly geostationary and will move due to wind gusts, thermal winds, or by the own transversal movement of the aerial vehicle. Also, rotational velocity Ω of camera unit100may be influenced by INS rotation R of the imaging system1000.

Typically, 2D images411-413,421-423, and431-433that compose a scene200are captured during one scanning rotation by camera unit100. For example, if camera unit100rotates at Ω=1 Hz, one camera107is used, and the image capturing frequency is f=100 Hz, then 100 images411-413will be captured for a scene210. In case multiple parallel operated cameras107are viewing the entire scene200, 2D images411-413,421-423, and431-433are captured at one capturing event at the same time. Also, it is not necessary that scene200covers a full rotation of 360°, and it is also possible that scene200is only composed of one or more sectors that are defined by azimuthal angles φ.

Captured 2D images411-413,421-423, and431-433picture portions of a panoramic scene200, the size of scene200being defined by the elevation view angles of the camera unit100. Scene200may be defined by upper and lower elevation angles θupper, θlowerof the scene200itself, and these angles will depend on the elevation angles of cameras107and the field of view of the associated optics. Preferably, upper elevation angle θupperis in a range between −10° and 30°, the negative angle indicating an angle that is above the horizon that is assumed at 0°, and lower elevation angle θloweris in a range between 40° and 75°. In the variant shown inFIG. 1, the angle range is approximately from 20° to 60°.

Also, adjacent images, for example411and412, are preferably overlapping. By using additional cameras in camera unit100having a different elevation angle θcas compared to the first camera, or by changing an elevation angle θcof a sole camera that is rotating with rotational velocity Ω, it is possible to capture one or more additional panoramic scenes210,220,230that will compose the viewed panoramic scene200with images411-413(upper panoramic scene210), images421-423(middle panoramic scene220), and images431-433(lower panoramic scene230) associated thereto. While images411-413,421-423, and431-433are represented inFIG. 2in an imaginary sphere represented as scene200with partial panoramic scenes210,220,230so as to show them references to an azimuth-elevation spherical coordinate system, they are actually viewing a corresponding surface of scenery300. For example, image411is representing a viewed surface511of scenery300, while image423is representing a viewed surface523.

Preferably, the sequentially captured images of a respective panoramic scene210,220,230are overlapping, for so that a part of image411will overlap the next captured image412, and image412overlaps partially with next captured image413, etc. However, this is not necessary, it is also possible that images411-413are taken with rotational velocity Ω, image capturing frequency f, and camera viewing angles that do not produce overlapping images, but that images captured from a first rotation of camera unit100to capture images from panoramic scene210overlap with images captured from a subsequent rotation of camera unit100in imaging system1000of the panoramic scene210.

Moreover, preferably camera unit100is arranged such that the adjacent panoramic scenes210,220,230overlap with an upper or lower neighboring panoramic scene in a vertical direction, for example, upper panoramic scene210overlaps with middle panoramic scene220, and lower panoramic scene230overlaps with middle panoramic scene220. Thereby, in a subsequent process, it is possible to stitch images411-413,421-423, and431-433together to form a segmented panoramic image having a higher resolution of the viewed scenery. Preferably, images are captured along a full rotation of 360° by rotation of camera system100with rotational speed Ω or from a plurality of cameras with different viewing angles, but it is also possible that images are merely captured from a sector or a plurality of sectors without capturing image along a portion of the 360° view of the panoramic scene.

Moreover, camera unit100also captures images from scenery300that may have objects or world points310,320,330, that are geostationary and are located on scenery so as to be viewable by camera unit100, such as buildings310, antennas320, roads330, etc., and these world points310,320,330can be either recognized by feature detection and extraction algorithms from captured images411-413,421-423, and431-433, or can simply be manually located within the images by having access to coordinate information of these world points310,320,330. Also, imaging system1000can dispose of a topographical map of the part of scenery300that is within viewable reach of camera unit100, for example a three-dimensional (3D) map that is preferably based on a orthographic coordinate system.

Generally, while images411-413,421-423, and431-433in an azimuth/elevation (Az-El) coordinate system represent a natural view of the viewed surfaces of scenery300by camera unit100having pixels representing a substantial similar view angle, if the same images would be viewed in the orthographic coordinate system to represent surfaces511and523of scenery300, these images would represent surfaces511and523in a very distorted way, with an decreased resolution with an increasing radial distance R from a rotational axis RA of camera unit100. For oblique angles, it is often more appropriate to view the image data from the perspective of the image capturing camera107, or in close proximity thereof, in particular when data from other cameras107will be used to compare the image data. In such a case, the Az-El coordinate system for projection presents a more natural solution to view the image data. In addition, the use of an Az-El coordinate system will also make the images appear more natural and is more efficient for image processing. The projection to a fixed azimuth/elevation camera location is an important aspect of the present invention which allows to generate stable imagery and to make subsequent processing easier.

For example, assuming the altitude A of camera unit100is 2 km, and a radial distance R of viewed surface523is 15 km, every pixel of image411will represent a narrow strip550of surface523that is extended in radial direction away from rotational axis RA. This distorted projection is the result of an affine transformation of the pixel response function. Generally, projection result being a narrow strip550has a trapezoidal shape, but the angles are in the order of 10−4rad. This distortion can be neglected especially in contrast to the distortion from the foreshortening that can be a factor of 100, tan−15°. Therefore, when viewed in the orthographic coordinate system, images411-413,421-423, and431-433of the scenery300would appear very distorted at distances that are far from a center projection of camera unit100compared to its altitude. In addition, because the location of camera unit100is not constant, images that appear directly under camera unit100in direction of the rotational axis RA and a certain angular range will have a parallax errors, and artifacts may occur as a result of differences in time of capture between images411-413,421-423, and431-433.

Therefore, the present invention aims to represent images411-413,421-423, and431-433in a spherical Az-El coordinate system, to provide a more natural viewing projection for the user, and to avoid the generation of strongly distorted images for scenery portions that are located for from camera unit100that would be of little use for a human user or image processing software for object recognition and tracking. In addition, another goal of the present invention is to project the captured image data from a fixed virtual camera source location V=[x_v, y_v, z_v] that is geostationary, despite the movements of camera unit100by trajectory [x_t, y_t, z_t]=T(t). This way, issues of parallax and other image distortions can be at least partially eliminated. This projection from the virtual camera source also substantially eliminates the effects of motion of camera unit100for an image sequence, so that the compressibility of the image sequence is improved, and also the performance of tracking and change detection algorithms are improved.

Next, data from the step S100of capturing images with camera unit100, images411-413,421-423, and431-433are associated to metadata with information on time of capture, location of capture, and geometrical arrangement of camera at time of capture, in a step S300. For example by associating the trajectory position T, elevation angle θc, and azimuth angle φc, and rotational speed Ω, INS rotation R, camera lens information, at time of the image capture, to the respective image.FIG. 3depicts the geometry of an Az-El coordinate system depicting azimuth angle φcand elevation angle θcof a spherical coordinate system that characterizes the viewing angle of camera system100at time of image capture. In this step S300, the image data is stored together with an association to the relevant metadata. This step can be performed with a processing unit that is located at the camera unit100. Every captured 2D image411-413,421-423, and431-433is also associated with the location T where in space the images were taken, and a series of such locations can be expressed as a trajectory [x_t, y_t, z_t]=T(t) that is variable in time.

In an additional step S200, the virtual camera source location V=[x_v, y_v, z_v] can be determined by an algorithm, for example by determining a location V that is in close proximity of a real camera location T, for example by using estimation techniques to predict a location V that will be close to present location T based on data of the past trajectory [x_t, y_t, z_t]=T(t). In addition, it is also possible to use a location V that is somewhat different than the location T, based on the user's viewing preference, for example by using a virtual camera source location V that is independent of the actual trajectory. The virtual camera source location V need not be a permanently fixed location, but can be refreshed at regular intervals, or for example when location T is outside a certain geographic range, preferably once camera unit100moves more than 10% of the distance to the ground. This allows to take global movements of camera unit100into account, for example if there is a dominant transversal movement when camera unit100is carried by a flying drone, or winds are pushing an aerostat in a certain direction.

As an example, virtual camera source location V=[x_v, y_v, z_v] can be determined by using the immediately past trajectory [x_t, y_t, z_t]=T(t) during a certain time period, for example a period of the past 10 seconds, and then generate a median or mean value of all the samples of trajectory T that will serve as location V. Data for trajectory T can be generated by using a satellite receiver115of the Global Positioning System (GPS) that is located at the same place as the camera unit100. This calculated location V can be refreshed at periodic interval that is different from the time period that is used for gathering passed data on trajectory T. Such way of calculating the virtual camera source location V=[x_v, y_v, z_v] is especially useful is a location of imaging system1000is substantially stationary and is not subject to any predictable transversal movement, such as it would be the case if a balloon or a blimp is used to carry imaging system1000.

In case camera unit100is performing a substantially transversal movement, for example when camera unit100is part of a payload that is installed on an aircraft moving at a certain speed over viewed scenery, the virtual camera source location V=[x_v, y_v, z_v] can be predicted for periods of time, for example by calculating an average motion vector of trajectory T for past periods, to gather period information on how much the camera unit100will move during a certain time period. This information can be further completed by having access to the speed of the aircraft, and speeds and directions of winds. Next based on this information, a virtual camera source location V for a next time period can be predicted that would correspond to a mean or median location if camera unit100would continue to move at the same average motion. It is also possible to estimate a virtual camera source location V=[x_v, y_v, z_v] by using maximum-likelihood estimation techniques, based on data on past camera source location T, present and past wind data, and flight speed of aircraft carrying camera unit100.

Next, based on data of images411-413,421-423, and431-433, a first bundle adjustment is performed in step S400that results in a camera model152for calibrating image data for all cameras107of the camera unit100. This is a calibration step that calibrates all the cameras together to form a unified camera model152that can take into account all internal camera parameters such as pixel response curves, fixed pattern noise, pixel integration time, focal length, aspect ratio, radial distortion, optic axis direction, and other image distortion. For this purpose, a processor performing step S400also disposes of a generic camera models of the camera107that was used from camera unit100to capture the respective image. Preferably, the generic camera models have a basic calibration capacity that is specific to the camera107and lens used, but has parameters that can be adjusted depending on variances of camera107, image sensor, lenses, mirrors, etc.

Preferably, the first bundle adjustment is done only once before operating the imaging system1000, but can also be repeated to update camera model152after a predetermined period of time, or after a certain trigger event, for example after camera unit100was subject to a mechanical shock that exceeded a certain threshold value. Therefore, the adaptation of the existing camera model152by a step S400allows to take variable defects into account, for example certain optical aberrations that are due to special temperature, mechanical deformation effects of scanning mirrors and lenses used, and other operational conditions of camera unit100. The camera model152generated by step S400are represented as a list of parameters which parameterize the nonlinear mapping from three-dimensional points in the scenery to two-dimensional points in an image.

Based on camera model152, every image that is later captured by camera unit100will be calibrated by a step S500to generate calibrated image data based on the camera model152for camera107that captured the image. The camera model calibration step S500takes into account optical distortions of the lenses of the cameras, image sensor distortions, so that for every pixel of each image411-413,421-423, and431-433a camera-centered azimuth and elevation angle can be established. This also allows to establish the viewing angles between the pixels of images411-413,421-423, and431-433, for each pixel. Therefore, the first bundle adjustment generates a data set of directional information for each pixel on real elevation angle θc, azimuth angle φc, and the angular difference between neighbouring pixels. This camera model calibration step S500does not take into account any dynamic effects of imaging system due to rotation Ω, INS rotation R, movement of location by trajectory T, and other distortions that are not internal to the capturing camera.

Next, the images that were processed by camera model calibration step S500are subject to a processing with a second bundle adjustment step S600, that includes an interframe comparison step S610that attempt to match overlapping parts of adjacent images, and a world point matching step S620where overlapping parts of adjacent images are matched to each other or to features or world points310,320,330of scenery300. The second bundle adjustment step S600allows to estimate with higher precision where the individual pixels of cameras107of camera unit100are directed to. Due to the motion of trajectory T of camera unit100, consecutively captured images are rarely captured from exactly the same location, and therefore the second bundle adjustment step S600can gather more information of the displacement and orientation of the imaging system1000. Thereby, it is possible to refine the directional information of each pixel, including relative elevation angle θc, azimuth angle φc, and the angular difference between neighbouring pixels, based on image information from two overlapping images.

In the interframe processing step S610on the overlapping parts of adjacent images411and412, image registration is performed where matching features in the overlapping part between two images411and412are searched for, for example by searching for image alignments that minimize the sum of absolute differences between the overlapping pixels or calculate these offsets using phase correlation. This processing allows to create data on corresponding image information of two different images that overlap, to further refine the pixel information and the viewing angle of the particular pixels. Also, interframe processing step S610can apply corrections to colors and intensity of the pixels to improve visual appearance of the images, for example by adjusting colors of mapping pixels and changing pixel intensity of exposure differences. Interframe processing step S610can prepare the images for later projection processing to make the final projected images more appealing to a human user.

Moreover, in the world points matching step S620, based directional information in which direction the camera of camera unit100that captured respective image is pointing, pre-stored world points510,520,530can be located in overlapping part of images411,412, so that a matching feature can be matched in order to improve the knowledge of orientation and position. This is particularly useful if it is desired to maintain geoaccuracy by matching to imagery with known geolocation. In this processing step, it is also possible to further match the non-overlapping part of images with certain world points510,520,530, to further refine the directional information. This step can access geographic location data and three-dimensioning modeling data of world points, so that an idealized view of the world points510,520,530can be generated from a virtual view point. Because the location of camera unit100at time of image capture and the location of world points510,520,530is precisely known, a projected view onto world points510,520,530can be compared with captured image data from a location T, so that additional data is available to refine the directional information that is associated with pixel data of images411-413,421-423, and431-433.

As explained above, the geographic location of the world points510,520,530is usually stored in a database in the orthographic coordinate system references to a 3D map, but a coordinate transformation can be performed on data of world points in step S620to generate Az-El coordinates that match the elevation angle θc, and azimuth angle φc, of the captured image, so that the world points510,520,530can be located on overlapping or non-overlapping parts of images411-413,421-423, and431-433. However, it is also possible that world points are newly generated without receiving such data from an external mapping database, for example by performing a feature or object detection algorithm on overlapping parts of adjacent images411and412, so that overlapping parts of an image can be better matched. Such object detection algorithm can thereby generate new world points that appear conspicuously on the images411,412for matching. Accordingly, the results of both the interframe processing step S610and the world points matching step S620will further calibrate the images to an Az-El coordinate system.

Next, the image data that was subject to the second bundle adjustment in step S600is then projected to an existing 3D map in step S700. Preferably, this step requires that coordinate data of the scenery300is available as 3D coordinate mapping data, for example in the orthographic or Cartesian coordinate system that is accessed from a database. In a variant, if the landscape of scenery300is very flat, for example a flat desert or in maritime applications, it may be sufficient to project the image data to a flat surface for which the elevation is known, or a curved surface that corresponds to the Earth's curvature, without the use of a topographical 3D map. With this projection in step S700, the pixel data is projected by using associated coordinates on elevation angle θc, and azimuth angle φc, and camera source capture location T for each pixel towards but a 3D topographical map or a plane in the orthographic coordinate system, so that each pixel is associated with an existing geographic position in x, y, and z coordinate system on the map. Based on this projection, ground coordinates for the image data referenced to the orthographic coordinate system is generated. Step S700is optional, and in variant it is possible to pass directly from the second bundle adjustment step S600to a projection step S800that generates a panoramic image based on a spherical coordinate system, as further described below.

The thus generated image data and is associated ground coordinates can be further processed based on stored data of the topographical map, so as to adjust certain pixel information and objects that are located in the ground image. For example, the image data can be processed to complement the image data content with data that is available from the 3D maps, for example color patterns and textures of the natural environment and buildings such as roads, houses, as well as shadings, etc. can be added. In addition, if three dimensional weather data is available, for example 3D data on clouds that intercept a viewing angle of camera unit100, this information could be used to mark corresponding pixels as not being projectable to the 3D topographical map.

In addition, in a variant, it is also possible that 3D on weather patterns are available from a data link or database for projection step S700, for example geographic information on location of clouds or fog. The projection step S700would thereby be able to determine whether a particular view direction from location [x_t, y_t, z_t]=T(t) is obstructed by clouds and fog. If the processing step confirms that this is the case, it would be possible to either replace or complement pixel data that are located in those obstructed view directions with corresponding data that is available from the topographical 3D map to complete the real view with artificial image data, or to mark the obstructed pixels of the image with a special pattern, color, or label, so that a viewer is readily aware that these parts of the images are obstructed by clouds or fog. This is advantageous if the image quality is low, for example in low lighting conditions, or homogenous scenes in a desert, ocean, etc.

Because the ground coordinates of the image data associates pixel data to an orthographic coordinate systems, this data could theoretically be displayed as a map on a screen and viewed by a user. But as explained above, pixel information on map portions that are located far away from the camera location will appear as a narrow strip550to the viewer. In addition, the orthographic coordinate system does not take into account movements of camera source location T, and many artifacts would be present due to parallax for image that point downwards along the rotational axis RA. Such orthographic ground image would therefore be of poor quality for a human user for viewing scenery300. In addition, depending on the lower elevation angle θlowerof scene200, there may be no image data available for parts of the scenery300that are located under the camera unit300around the rotational axis RA.

Accordingly, the thus generated ground image that is based on ground coordinates and image data is subject to a reprojection step S800that generates a panoramic image based on a spherical coordinate system with coordinates having elevation angle θpand azimuth angle φpthat are again associated to each pixel as shown inFIG. 3, but as seen from a virtual camera source location V=[x_v, y_v, z_v]. As explained above, the virtual camera source location V can be fixed, estimated, calculated, and can be periodically updated, but will have at least for a certain period a fixed geographic position, as discussed with respect to step S200. The pixels of the reprojected image that will be composed from many 2D images will therefore be references in the Az-El coordinate system, as an Az-El panoramic image, from a fixed virtual viewpoint.

Because imaging system1000is configured to view a segment or a full circle of a panoramic scene200that is define by an upper and a lower elevation angle θupper, θlower, this form of projection of the data corresponds more naturally to the originally captured data, but the initially captured 2D image data from images411-413,421-423,431-433has been enhanced by data and information from the pre-existing 3D map, world points510,520,530, geometric calibration, and have been corrected to appear as if the images were taken from a fixed location V. Such Az-El panoramic image is also more suitable for persistent surveillance operations, where a human operator has to use the projected image to detect events, track cars that are driving on roads, etc. This coordinate transformation that was performed in step S800is used to warp the image data for projection and display purposes to from the image to the Az-El coordinate system.

As described above with reference toFIG. 1, the steps of the image projection method appear in a certain order. However, it is not necessary that all the processing steps are performed in the above described order. For example, the intra-frame world point matching step S610need to be a sub-step of the second bundle adjustment step S600, but may be performed as a separate step before the matching of the world points5620.

FIG. 4shows an exemplary imaging system1000to perform the method described above with respect toFIG. 1. Imaging system1000includes a camera unit100with one or more cameras107that may either rotate at a rotational speed Ω to continuously capture images, or be composed of cameras that are circularly arranged around position T to capture image from different view angles simultaneously to capture overlapping images of panoramic scene200. In a variant, camera unit100or individual cameras107of the camera unit100are not rotated, by a rotating scanning mirror (not shown) is used for the rotation, or a plurality of cameras107are used that are circularly arranged around location T and optically configured to substantially cover either panoramic scene200, or a sector thereof. In a variant, three pairs of cameras107are rotating, each pair of cameras being composed of a 1024 to 1024 pixel visible light charge-coupled device (CCD) image sensor camera, and a focal plane array (FPA) thermal image camera, and each pair having a different elevation angles (β1, β2and β3so that visible light images and thermal images are captured simultaneously captured from the same partial panoramic scene210,220, and230.

A controller110controls the capturing of the 2D images, but also captures simultaneously data that is associated to conditions of each captured image, for example a precise time of capture, GPS coordinates of the location of camera unit100at time of capture, elevation angle θcand azimuth angle φcof the camera at time of capture, weather data including temperature, humidity and visibility. Elevation angle θc and azimuth angle cpscan be determined from positional encoders from motors rotating camera unit or scanning mirrors that is accessible by controller110, and based on GPS coordinates and orientation of an aircraft carrying the camera unit100. Moreover, controller110is configured to associate these image capturing conditions as metadata to the captured 2D image data. For this purpose, the controller110has access to a GPS antenna and receiver115. 2D image data and the associated metadata can be sent via a data link120to a memory or image database130for storage and further processing with central processing system150. Data link120may be a high-speed wireless data communication link via a satellite or a terrestrial data networking system, but it is also possible that memory130is part of the imaging system1000and is located at the payload of the aircraft for later processing. However, it is also possible that the entire imaging system1000is arranged in the aircraft itself, and therefore data link120may only be a local data connection between controller110and locally arranged central processing system150.

Moreover, in a variant, cameras107of camera unit100are each equipped with a image processing hardware, so called smart or intelligent cameras, so that certain processing steps can be performed camera-internally before sending data to central processing system150. For example, certain fixed pattern noise calibration, the first bundle adjustment of step S400, the association of image data with certain data related to image capture of step S300can all be performed within each camera107, so that less processing is required in central processing system150. For this purpose, each camera107would have a camera calibration model stored in its internal memory. The camera model152could also be updated, based on results of the second bundle adjustment step S500that can be performed on central processing system150. In a variant, the world point matching step S520that matches world points to non-overlapping parts of a captured image could also be performed locally inside camera107.

Central processing system150is usually located at a remote location from camera unit100at a mobile or stationary ground center and is equipped with image processing hardware and software, so that it is possible to process the images in real-time. For example, processing steps S500, S600and S700can be performed by the central processing system150with a parallel hardware processing architecture. Moreover, the imaging system1000also includes a memory or map database140that can pre-stores 3D topographical maps, and pixel and coordinate information of world points510,520,530. Both map database140with map information and image database130with the captured images are accessible by the image processing system150that may include one or more hardware processors. It is also possible that parts of the map database be uploaded to individual cameras107, if some local intra-image processing of cameras107requires such information.

Moreover, central processing system150may also have access to memory that stores camera model152for respective cameras107that are used for camera unit100. Satellite or other type of weather data156may also be accessible by central processing system so that weather data can be taken into consideration for example in the projection steps S700and S800. Central image processing system150can provide the Az-El panoramic image data projection that results from step S800to an optimizing and filtering processor160, that can apply certain color and noise filters to prepare the Az-El panoramic image data for viewing by a user. The data that results from the rendering and filtering processor160can then be subjected to a graphics display processor170to generate images that are viewable by a user on a display180. Graphics display processor170can process the data of the pixels and the associated coordinate data that is based on the Az-El coordinate system to generate regular image data by warping, for regular display screen. Also, graphics display processor170can render the Az-El panoramic image data for display on a regular display monitor, a 3D display monitor, or a spherical or partially curved monitor for user viewing.

Moreover, the present invention also encompasses a non-transitory computer readable medium that has computer instructions recorded thereon, the non-transitory computer readable medium being at least one of CD-ROM, CD-RAM a memory card, a hard drive, FLASH memory drives, Blue Ray™ disks or any other type of portable data storage mediums. The computer instructions configured to perform an image processing method as described with reference toFIG. 1when executed on a central processing system150or other suitable image processing platform. Portions or entire parts of the image processing algorithms and projection methods described herein can also be encoded in hardware on field-programmable gate arrays (FPGA), complex programmable logic devices (CPLD), dedicated digital signal processors (DSP) or other configurable hardware processors.