Patent Publication Number: US-8125406-B1

Title: Custom, efficient optical distortion reduction system and method

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application relates to the following applications filed on even date herewith and each incorporated herein by these references in their entirety: 
     Multiprocessor Discrete Wavelet Transform by John K. Gee, Jennifer A. Davis, David W. Jensen and James Potts, Ser. No. 12/572,600; Optical Helmet Tracking System by Jaclyn A. Hoke and David W. Jensen, Ser. No. 12/572,542; and Multiple Aperture Video Image Enhancement System by David W. Jensen and Steven E. Koenck, Ser. No. 12/572,492. 
     FIELD OF THE INVENTION 
     The present invention relates to optical distortion reduction and more specifically to custom systems and methods for reduction of optical distortion in individual helmet-mounted displays in an efficient manner. 
     BACKGROUND OF THE INVENTION 
     When an image is projected onto a curved optical surface, such as a windshield or a helmet visor or is captured through a curved optical surface, such as a camera lens, it is common for the image to become optically distorted. In the past, mathematical algorithms containing sets of polynomials have been used to pre-correct an image before it is projected onto a curved surface and/or through the optical system of a helmet-mounted display (HMD) with predetermined optical distortion producing characteristics. This is done to reduce the observable distortion in the finally viewed image. Traditionally, the HMD optics and the inside surface of a helmet visor, which is used to project images to a pilot, are characterized with a sophisticated test rig, and high-order polynomials (e.g. 10 th  order) have been used to characterize the shape and, therefore, the correction needed to achieve the desired image. However, these high-order polynomial algorithms are computationally intensive and are difficult to implement in a real time system. 
     The problems associated with high-order polynomials are indeed not new. Before the advent of computer aided design (CAD) systems, draftsmen were known to attempt to plot polynomials by breaking them up into numerous segments, each segment being approximated by simpler polynomials, and a set of French curves was an essential tool in this endeavor. 
     But with the advent of modern computing, higher order polynomials could be handled, but they often were too slow and required too much computing resources to be used with military video applications. 
     Today various tricks are known in the art as ways to overcome the problems associated with dealing with such high-order polynomials. One approach is to precompute the pixel distortion and store the amount in a large table. Memory access speeds and memory cost make this approach prohibitive for some applications. 
     Also, it has been known that other optical systems, such as camera lenses, can have their images improved by digitally correcting problems, such as a drop in brightness as one moves further from the center of a lens. 
     The present invention overcomes long-standing problems associated with distortion of optical images, especially when images are being processed at video rates. 
     SUMMARY OF THE INVENTION 
     Computer images are mathematically distorted before being projected onto the visor and/or through the HMD optics. The distorted image appears rectangular to the person viewing the display. To accurately distort the image, the computer algorithm uses a high-order (e.g., 10th order) set of polynomials. These polynomials are computationally intensive and difficult to implement in a real time system. This invention provides an efficient alternative implementation to those high-order polynomials. 
     Accordingly, the present invention can be summarized as follows: a custom system for efficiently reducing optical distortion in a particular helmet-mounted display comprising: 
     an optical element comprising a first curved optical surface, which surface is associated with optical element distortion; 
     said first curved optical surface being logically subdivided into a plurality of smaller curved optical surface segments, each segment associated with a segment distortion; 
     wherein said optical element distortion can be characterized across the optical element with an optical element polynomial; 
     wherein at least one of said segment distortions of said plurality of smaller curved optical surface segments, can be characterized with a segment polynomial which is a lower order polynomial than said optical element polynomial; 
     a computer processor configured to apply distortion correction to each of said smaller curved optical segments based upon the segment polynomial associated with said smaller curved optical segments; 
     said computer processor further configured to combine data associated with each smaller curved optical segments to collectively improve image quality of images associated with said first curved optical surface. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  shows a system of the prior art used to implement a pre-distortion error correcting image projected onto a curved optical surface with a predetermined distortion causing characteristic, where the dotted lines show a distorted image or distorted image data. 
         FIG. 2  shows a system of the prior art where a camera has post-processing of an image which has been optically altered by passing through a lens, where the dotted lines refer to altered images or altered image data. 
         FIG. 3  shows a system of the present invention where the dotted lines show a distorted image or distorted image data. 
         FIG. 4  is a system of the present invention where the dotted lines refer to altered images or altered image data. 
     
    
    
     DETAILED DESCRIPTION 
     Now referring to the drawings, wherein like numerals refer to like matter throughout and more particularly referring to  FIG. 1 , there is shown a Helmet-Mounted Display (HMD) system of the prior art, generally designated  100 , including a human viewer  1  which may be a helicopter pilot or any person looking through an optical element or a more complex optical system of elements  10  which may include a curved surface of a shield or visor mounted on a helmet. A projector  20  is shown projecting a pre-distorted image toward the optical element or system  10  where it is reflected back to the viewer  1 , in the predetermined way which will cancel out the intentionally pre-distorted image output by the projector  20 , so that the viewer sees a non-distorted image. The projector  20  receives its signals from an image source  30 , which could be a camera or other sensor, but the projector does not project the camera output directly; instead, the processors  40 , which could be a single processor or multiple processors pre-distort or apply a forward error correcting distortion to the signal before it is sent to the projector  20 . 
     This problem of the prior art is exacerbated by the fact that the image seen of the outside scene by viewer  1  looking through the optical element is also distorted. The forward error correcting pre-distortion of the signal to the projector must also attempt to align the image up with the distorted image that the viewer  1  is receiving through the optical element  10 . The prior art system of  FIG. 1  shows nothing disposed between the projector and the optical element  10 . It should be understood by those skilled in the art that relay optics often are used there, and these relay optics can pre-distort the projected image in a known manner. 
     Now referring to  FIG. 2 , there is shown a digital camera system of the prior art, generally designated  200 , including a camera lens  230  receiving light from a real world scene  220 . The light is affected by the camera lens and strikes a sensor  240  where electronic data is generated of the optically altered image; a computing system  250  corrects the altered image data to achieve an improved image at output  260 . It has been known to correct for the decrease in amplitude in the light as a function of distance from the center of the lens. 
     Now referring to  FIG. 3 , the present invention is shown to include a system quite similar to the prior art  FIG. 1  except for the differences noted below. The optical system  10 , the projector  20  and the camera  30  all may be the same as those well known in the prior art. The computer processor  340  may be quite different, depending upon system requirements and design limitations. 
     The preferred embodiment relates to the correction or reduction of optical distortion that occurs when an image is projected onto a curved optical surface or through an HMD optical system. 
     A curved optical surface is part of the optical system and refers to any non-planar surface. The curved optical surface may cause significant optical element distortion. 
     Optical distortion refers to the distortion of an image when projected onto or captured through a curved optical surface or an HMD optical system. There are various kinds of optical element distortion which can be addressed by the present invention. 
     Each curved optical element  10  can be logically viewed as a surface which includes a plurality of smaller surface segments. Each smaller surface segment is then analyzed and associated with a segment distortion that is further related to a specific region of an image. Distortion data for each of the smaller segments is determined by the test rig developed data associated with that portion of the display field of view. 
     The optical distortion of the HMD can be represented with an optical element polynomial or also referred to as an optical distortion polynomial. An optical distortion polynomial consists of coefficients and variables representing the angles viewed at the pilot&#39;s eye. The more complex the distortion, the higher the order of polynomial that is necessary to characterize it. For example, an optical element that is a portion of a perfect sphere would be much easier to compensate for than an optical element that has a much more complex radius of curvature changing shape which would require more than a 3 rd  order polynomial. When there are even very small manufacturing tolerances or anomalies in the optical elements  10 , the necessary order of the polynomial can be much higher. 
     However, segment distortion can be characterized by a segment polynomial. A segment polynomial is a polynomial associated with a smaller segment and is often a lower order than the order of the entire optical distortion polynomial. For example, if the distortion polynomial associated with a curved optical system is of 10 th  order, a 4 th  order polynomial associated with a specific smaller segment is a segment polynomial. The lower the order of the polynomial, the easier the computation. 
     In order to meet HMD accuracy requirements, 10 th  order polynomials might be used to represent the HMD optical distortion. 
     In one exemplary embodiment of the present invention, four sets of 10 th  order polynomial are stored on the HMD EEPROM—two for the left eye and two for the right eye. Each pair of polynomials represents the following transfer functions:
 
Az out   =f ( H,V )Azimuth angle(Az out )as a function of LCD horizontal and vertical pixel position( H,V )
 
EL out   =f ( H,V )Elevation angle(El out )as a function of LCD horizontal and vertical pixel position( H,V ).
 
     The two polynomials for each eye are used to accurately predict the outside world angles (Az out , El out ) for any LCD pixel position that falls within the monocular field of view. The polynomial is optimized over an area that extends to half a degree beyond the edges of the 40° by 30° monocular field of view of each display. Typically, the performance of the polynomial is not controlled outside of the defined field of view, and it is likely that the behavior outside of his region will be erratic, as is typical of high-order polynomials. 
     Because high-order polynomial algorithms are mathematically intense, they can take considerable time and power to perform. This makes it difficult to implement optical distortion reduction in a real time military video system. The following equations illustrate the complexity of the 10 th  order polynomials.
 
Az out   ′=C   1   C   2   V+C   3   V   2   +C   4   V   5   +C   5   V   4+C   6   V   5   +C   7   V   6   +C   8   V   7   +C   9   V   8   +C   10   V   9   +C   11   V   IU   +C   12   H+C   13   H V+C   14   H V   2   +C   15   H V   3   +C   16   H V   4   +C   17   H V   5   +C   18   H V   6   +C   19   H V   7   +C   20   H V   8   +C   21   H V   9   +C   22   H   2   +C   23   H   2   V+C   24   H   2   V   2   +C   25   H   2   V   3+C   26   H   2   V   4   +C   27   H   2   V   5   +C   28   H   2   V   6   +C   29   H   2   V   7   +C   30   H   2   V   8   +C   31   H   3   +C   32   H   3   V+C   33   H   3   V   2   +C   34   H   3   V   3   +C   35   H   3   V   4   +C   36   H   3   V   5   +C   37   H   3   V   6   +C   38   H   3   V   7   +C   39   H   4   +C   40   H   4   V+C   41   H   4   V   2   +C   42   H   4   V   3   +C   43   H   4   V   4   +C   44   H   4   V   5   +C   45   H   4   V   6   +C   46   H   5   +C   47H   5   V+C   48   H   5   V   2   +C   49   H   5   V   3   +C   50   H   5   V   4   +C   51   H   5   V   5   +C   52   H   6   +C   53   H   6   V+C   54   H   6   V   2   +C   55   H   6   V   3   +C   56   H   6   V   4   +C   57   H   7   +C   58   H   7   V+C   59   H   7   V   2   +C   60   H   7   V   3   +C   61   H   8   +C   62   H   8   V+C   63   H   8   V   2   +C   64   H   9   +C   65   H   9   V+C   66   H   10 
 
 E 1 out   ′=C   67   +C   68   V+C   69   V   2   +C   70   V   3   +V   3   +C   71   V   4   +C   72   V   5   +C   73   V   6   +C   74   V   7   +C   75   V   8   +C   76   V   9   +C   77   V   10   C   78   H+C   79   H V+C   80   H V   2   +C   81   H V   3   +C   82   H V   4   +C   83   H V   5   +C   84   H V   6   +C   85   H V   7   +C   86   H V   8   +C   87   H V   9   +C   88   H   2   +C   89   H   2   V+C   90   H   2   V   2   +C   91   H   2   V   3   +C   92   H   2   V   4   +C   93   H   2   V   5   +C   94   H   2   V   6   +C   95   H   2   V   7   +C   96   H   2   V   8   +C   97   H   3   +C   98   H   3   V+C   99   H   3   V   2   +C   100   H   3   V   3   +C   101   H   3   V   4   +C   102   H   3   V   5   +C   103   H   3   V   6   +C   104   H   3   V   7+C   105   H   4   +C   106   H   4   V+C   107   H   4   V   2   +C   108   H   4   V   3+C   109   H   4   V   4   +C   110   H   4   V   5   +C   111   H   4   V   6   +C   112   H   5   +C   113   H   5   V+C   114   H   5   V   2   +C   115   H   5   V   3   +C   116   H   5   V   4   +C   117   H   5   V   5   +C   118   H   6   +C   119   H   6   V +C   120   H   6   V   2   +C   122   H   6   V   3   +C   122   H   6   V   4   +C   123   H   7   +C   124   H   7   V+C   125   H   7   V   2   +C   126   H   7   V   3   +C   127   H   8 +C   128   H   8   V+C   129   H   8   V   2   +C   130   H   9   +C   131   H   9   V+C   132   H   10 
 
     Each 10th order polynomial contains 66 coefficients where Azout and Elout are the outside world angles, H and V are the horizontal and vertical LCD pixel positions, and Cn represents the coefficient number. In comparison, a 5 th  order polynomial only has 21 coefficients, a 4 th  order polynomial only has 15 coefficients, and a 2 nd  order polynomial only has 6 coefficients. The number of mathematical operations is proportional to the number of coefficients. A 2 nd  order polynomial requires 1/10 the number of mathematical operations compared to a 10 th  order polynomial. 
     Efficient optical distortion reduction solves this computation problem by splitting the image into multiple smaller regions and using separate and different lower-order (e.g. 3 rd  order) polynomials for each region or segment. The test rig is used to determine the 10 th  order polynomials to represent the optical distortion for the helmet. Lower-order polynomials can be used to correct distortion in each smaller segment and its associated image region. The coefficients for the lower-order polynomial equations are computed with linear algebra to solve simultaneous equations. The solution to those equations provides the best coefficients to closely match the higher polynomial for each of the segment regions. Coefficients for multiple lower-order equations (2 nd , 3 rd , 4 th , etc.) can be produced. A comparison is made between each of the lower-order polynomials and the original optical element polynomial using a root mean square (RMS) error. The lowest-order segment polynomial providing an acceptable RMS error compared with the higher-order polynomial is the best order for that region. Normally, the RMS error is less with higher-order polynomials; however, subdividing the data over even more partitions allows for a reduced order polynomial to be used while maintaining an equivalent or reduced RMS error. In other words, the more and the smaller the image regions or segments, the lower the polynomial order required to achieve the desired distortion correction. Design trade-offs will need to be made in each application. It may be that the distortion polynomial has an order of 10 and the segment polynomial has an order of 4 or 3. In such cases, the decrease in complexity in computing resources needed for each segment can be substantial. As noted above, a 2 nd  order polynomial requires 1/10 the number of mathematical operations compared to a 10 th  order polynomial. 
     Once the best order polynomial is determined for each image region, a computer processor is used to apply the lower order segment polynomials to the associated image region. Because these are simpler operations involved with the segment polynomials, the computer processor is able to run the algorithms more rapidly. The processing of these algorithms on independent image regions also executes well in distributed or parallel processing and in a multicore environment. Using multiple processors reduces the amount of computing that each processor has to perform; for example, with sixteen processors each rendering its own polynomial function. Each processor is responsible for computing a subset of the entire data. Even if there were no reduction in the order of the polynomials needed to characterize a segment, with sixteen processors, each processor would be responsible for one sixteenth of the computing. Each would process one-fourth of the region in the x direction and one-fourth in the y direction. This would equal 6.25% of the total computing per processor, which implies a speed-up factor of 16. Combined with the reduction in operations with the lower order segment polynomials, the speed factor can be 100 to 200. This approach enables real time processing of the distortion algorithm on lower power, lower performance, and lower cost processors. It also makes it possible to implement optical distortion reduction in a real time system. The system latency is greatly reduced when the benefits of lower order polynomials are simultaneously achieved. The flow of this process is described below. 
     Finally, after each segment polynomial is applied to its respective image region, the computer processor combines the image regions to recreate a corrected version of the original image by stitching the image regions together. The removal of distortion is in itself helpful in the stitching together process. 
     The present invention is described for use with a helmet-mounted display, but it can be used with any optical system that induces distortion. Now referring to  FIG. 4 , there is shown a simple example of the present invention used in a camera  400  to process the image detected by a digital sensor  240  after the image has experienced some distortion by the camera lens  230 . The same process that was described above to pre-distort the image before it interacts with the optical element  10  ( FIGS. 1 and 3 ) can be used to remove distortion after it is caused by an optical element and prepare a distortion corrected output  460 . Of course, a simple single lens  230  is shown for simplicity, but it should be understood that the term “optical element” is intended herein to be any lens, mirror, reflecting surface, combination of the same or any structure which is capable of imparting upon light incident thereon with a known and predictable alteration or distortion. 
     Throughout this discussion, it has been assumed that the curved optical surface would be logically subdivided, a preferred embodiment might be addressed similarly but slightly differently, For example, in one embodiment, it could be the displayed (pre-distorted) image that is broken into rectilinear segments at the image source (AMLCD), which means that each of these segments is an irregular shape by the time it hits the visor. I.e. it may actually be the source image that has to be broken into segments. The basic concept of the present invention remains, that the higher order polynomials can be avoided by segmenting the source image and characterizing those segments with lower order polynomials and then parallel processing the segments. It will be further appreciated that functions or structures of a plurality of components or steps may be combined into a single component or step, or the functions or structures of one-step or component may be split among plural steps or components. The present invention contemplates all of these combinations. Unless stated otherwise, dimensions and geometries of the various structures depicted herein are not intended to be restrictive of the invention, and other dimensions or geometries are possible. Plural structural components or steps can be provided by a single integrated structure or step. Alternatively, a single integrated structure or step might be divided into separate plural components or steps. In addition, while a feature of the present invention may have been described in the context of only one of the illustrated embodiments, such a feature may be combined with one or more other features of other embodiments, for any given application. It will also be appreciated from the above that the fabrication of the unique structures herein and the operation thereof also constitute methods in accordance with the present invention. The present invention also encompasses intermediate and end products resulting from the practice of the methods herein. The use of “comprising” or “including” also contemplates embodiments that “consist essentially of” or “consist of” the recited feature. 
     The explanations and illustrations presented herein are intended to acquaint others skilled in the art with the invention, its principles, and its practical application. Those skilled in the art may adapt and apply the invention in its numerous forms, as may be best suited to the requirements of a particular use. Accordingly, the specific embodiments of the present invention as set forth are not intended as being exhaustive or limiting of the invention. The scope of the invention should, therefore, be determined not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. The disclosures of all articles and references, including patent applications and publications, are incorporated by reference for all purposes.