Abstract:
The present invention relates generally to various arrangements of optical and electronic components to form a high-resolution helmet mounted display (HMD) or other compact display device. In particular, the current invention is designed in such a way as to allow it to operate utilizing several different kinds of image generation device and to incorporate many different features including the option of “eye tracking”.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application claims priority of U.S. provisional patent No. 60/213,888 titled “Multi-Mode Display Device” filed Jun. 26, 2000 by Angus Duncan Richards. 
    
    
     BACKGROUND OF THE INVENTION 
     The present invention relates generally to various arrangements of optical and electronic components to form a high-resolution helmet mounted display (HMD) or other compact display device. In particular, the current invention is designed in such a way as to allow it to operate utilizing several different kinds of image generating devices and to incorporate many different features including the option of “eye tracking”. 
     Recently there have been several major breakthroughs in display generation technology. These breakthroughs have occurred in the fields of active matrix liquid crystal display (AMLCD), ferroelectric display (FED) technology and a new technology known as liquid crystal on silicon (LCOS). Display devices will soon be emerging with extended graphics array (XGA) and super extended graphics array (SXGA) performance. At this time, it is uncertain which technology will prevail in the HMD market. 
     SUMMARY OF THE INVENTION 
     The HMD design of this invention works equally well with both transmissive and reflective technologies. When used with a transmissive AMLCD type display device, the HMD design allows the generation of a full stereoscopic (separate image to each eye) image from a single display device. This is achieved by “time-multiplexing” the use of the display device between the two eyes. When used with a reflective display technology such as FED or LCD on silicon device, a separate display chip is required for each eye. In addition to the versatility of this display configuration to utilize either of these display technologies, it is also possible to utilize both of these display technologies simultaneously. In such a situation, the images produced would consist of an “optical overlay” of the images generated by the FED and AMLCD devices. Under certain circumstances this configuration can yield great benefits. For example, it would be possible to provide a large panoramic field of view in relatively low resolution (perhaps in monochrome) and use the reflective displays to provide a smaller high-quality field of view as a subset of this larger field image. It is also possible utilizing this unique HMD design to combine multiple fields of view utilizing pairs of similar display technology. For example, one can utilize a pair of transmissive AMLCD displays (2 displays for each eye) or a pair of FED displays for each eye. 
     With the growing interest in “-augmented reality-” and virtual reality based CAD there is a growing demand for new and more efficient man-machine interface tools. One of the most sophisticated user interface tools is that of the “eye cursor” or “eye tracking device.” Such “eye cursor” technology calculates the precise direction that each eye is pointing and mathematicallly calculates the “target” that is being viewed. This technology is inherently three-dimensional and represents an ideal interface tool for 3-D virtual worlds and augumented reality. The unique design of this HMD allows the incorporation of eye tracking in addition to the display tasks normally associated with an HMD. In addition, the imaging technology that comprises the eye tracking system can be used to overcome one of the major problems with lens-based HMD designs, which is the ability to automatically accommodate viewers with different inter-ocular spacing (spacing between the eyes). Most off-the-shelf HMD&#39;s which are utilizing a lens-based design require manual inter-ocular adjustment for each viewer. This adjustment is often difficult and time-consuming, both of which are factors that make such HMD&#39;s inappropriate for public use. By utilizing the imaging hardware of the eye tracking system as feedback, an automatic or servo-controlled inter-ocular adjustment can be readily achieved. Interestingly, such precise inter-ocular adjustment are an essential requirement for the correct operation of most eye tracking hardware. In this way, both hardware requirements are met and are mutually symbiotic. 
     In certain situations, a “see-through” augmented reality HMD design is preferable to a fully enclosed virtual reality configuration. In such a situation it is possible to configure the HMD design described in this specification to “see-through-mode” by replacing the eye tracking video cameras with corrective optics. This will allow the user to see virtual objects superimposed upon the real world images at the cost of losing the eye tracking and automatic inter-ocular adjustment features of the design. An alternative configuration which achieves many of the same objectives is simply to include two miniature video cameras as part of the HMD design. These cameras would provide “real-time” video feeds which can be digitally combined with the “virtual objects” displayed in the HMD. It would of course be possible to provide the same monoscopic video signal from a single miniature video camera to both eyes of the HMD. However, this would preclude the use of eye tracking for all but the virtual objects displayed in the system because eye-tracking inherently requires the stereoscopic images to provide the required three-dimensional information. 
     In many ways, this electronic equivalent of “see-through mode” is actually superior to the simplistic optical approach as it inherently avoids the problems of “transport lag” and “-positional inaccuracies-” associated with optical superimposing of virtual objects. This is possible because the same “live video feeds” are being used to determine the spatial points of reference for the projection of the virtual objects. Therefore, any “transport lag” and/or “positional inaccuracies between the “real world” and the “virtual world” are not perceived by the user because they are common to both the background and the virtual objects projected upon it. In effect, the user is working entirely within a “virtual environment”. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIG. 1 is a plan view of a certain presently preferred embodiment of a multi-mode display device according to this invention and illustrates both the reflective mode (Mode 1) and the transmissive mode (Mode 2) of the invention. 
     FIG. 2 is an elevation view of the multi-mode display device shown in FIG.  1 . 
     FIG. 3 is a plan view of an alternative certain presently preferred embodiment of a multi-mode display device according to this invention and illustrates the “see through” mode (Mode 3) of the invention. 
     FIGS. 4 and 5 are elevation views of display devices similar to the display devices illustrated by FIGS. 1 and 2, but show polarizing options. 
     FIG. 6 and 7 are elevation views of display devices similar to the display devices illustrated by FIGS. 1 and 2, but show focusing options. 
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     The preferred embodiment of this design is shown in FIGS. 1 and 2. The HMD consists of two separate optical assemblies and one optional common optical device. Optical assembly  1  consists of mirror M 1 , beamsplitter BS 1 , beamsplitter BS 3 , eyepiece E 1 , light source LS 3  and optional components consisting of reflective display device RD 1 , collimating lens CL 1 , infrared filter F 1 , camera ETC 1 , and infrared light sources LS 1 . Optical assemably  2  is basically identical and consists of mirror M 2 , beamsplitter BS 2 , beamsplitter BS 4 , eyepiece E 2 , light source LS 4  and optional components consisting of reflective display device RD 2 , collimating lens CL 2 , infrared filter F 2 , camera ETC 2 , and infrared light sources LS 2 . The common optical component is transmissive display device TD. 
     Mode 1 
     (Reflective Mode—Virtual Reality) 
     In this mode, two reflective display devices (such as FED or reflective LCOS) are used to display a separate image to each eye. The video information delivered to each display device may be purely synthetic or it may be a combination of virtual imagery and real-time video feeds. Although different configurations are possible, beamsplitters BS 1  and BS 2  would generally be of non-polarized type and beamsplitters BS 3  and BS 4  would be of broadband polarizing design. Light sources LS 3  and LS 4  can be of any appropriate type but would most probably be of an LED design. In this configuration, light from light source LS 3  would pass through beamsplitter BS 3 . Only light with one plane of polarization will be allowed through beamsplitter BS 3 , the other plane of polarization being reflected form the hypotenuse of the beamsplitter. In this particular configuration, the optional transmissive display element TD would simply be replaced by a non-reflective baffle. Light being reflected towards this baffle would simply be absorbed. The remainder of the plane polarized light incident upon reflective display device RD 1  will then selectively undergo rotation of the plane of polarization (phase retardation) depending upon the state of each individual pixel. The light that has undergone this rotation (pixels which are on) will them be reflected from the hypotenuse of beamsplitter BS 3  towards beamsplitter BS 1 . As beamsplitter BS 1  is non-polarizing, half of the light incident upon it will pass through the beamsplitter towards mirror M 1  the other half of the light will reflect off the hypotenuse of beamsplitter BS 1  towards infrared filter F 1 . As light source L 3  contains no infrared component, this light will be absorbed by F 1 . The light that strides concave mirror M 1  will form a real image at some distance from M 1  such that the light from this image will once again be partially reflected from the hypotenuse of beamspliter BS 1 . This reflected light will enter eyepiece E 1  and form an image which is viewable by the user of the HMD. 
     Ideally, beamsplitters BS 3  and BS 1  should be optically bonded together to reduce reflections. BS 1  and BS 2  could be of polarizing design if an additional phase retarder is placed between beamsplitter BS 1  and mirror M 1  (and between beamsplitter BS 2  and mirror M 2 ) as shown in FIG.  7 . In fact, such a configuration will yield significantly higher optical performance. The reason why this is not considered the preferred embodiment is that polarizing beamsplitters are relatively expensive devices. It is also possible (at the cost of further reducing sytem optical efficiency) to replace beamsplitters BS 3 , BS 4  with simple non-polarizing beamsplitters and by adding additional polarizers P 1 -P 4  as shown in FIG. 4, which will operate in an optically similar manner. Equally valid is a somewhat simpler configuration utilizing single polarizing elements P 1 , P 2  as shown in FIG.  5 . These configurations may be preferable in a production model given that in a further modification the cube beamsplitters BS 1 -BS 4  could potentially be replaced by simple partially silvered mirrors which are inexpensive. Reducing the overall optical efficiency may not pose a significant problem if the light sources LS 3  and LS 4  are sufficiently powerful. The replacement of cube beamsplitters with partially silvered mirror style beamsplitters does, however, have the disadvantage of double reflections. 
     The miniature video cameras ETC 1 , ETC 2  work in conjunction with eyepiece lenses E 1 , E 2  to form images of the user&#39;s eyes. Infrared filters F 1 , F 2  are used to block stray light from the display chips from entering the cameras directly. Light sources LS 1 , LS 2  can be of a variety of different sources and/or a different number of lights. However, a configuration of four infrared LED&#39;s is probably preferred. A simple configuration of 4 LED&#39;s in a square can be used to achieve both eye tracking and eye positioning (for inter-ocular adjustment). In such a configuration, the LED&#39;s will produce four distinct reflections from the cornea of the viewer&#39;s eyes. This information, in addition to the relative position of the pupil of the eye, can be used to determine the direction that the eye is pointing in three dimensions (i.e. a line of sight can be determined). The two separate lines of sight (one for each eye) can be used to locate a specific point in 3-D space. Infrared light (or near infrared) is used in preference to visible light, firstly because it allows the stray from the image projection to be easily eliminated and secondly because the use of visible light would be a constant source of irritation to the user. 
     In this embodiment of the HMD, eyepiece E 1 , beamsplitters BS 1 , BS 3 , light source LS 3 , and reflective display device RD 1 , collimating lens CL 1  and camera assembly consisting of infrared filter F 1 , light source or sources LS 1 , and camera ETC 1  would be physically joined together and would be free to move laterally as shown in FIGS. 1 and 2. Mirror M 1  is free to move laterally on the same axes as this assembly and may or may not be physically connected to the assembly. The lateral adjustment of the aforementioned optical assembly (and its counterpart) allows for automatic adjustment of different eye spacing (inter-ocular adjustment). This adjustment would ideally be performed by some form of an electro-mechanical means such as a servo system. The independent adjustment of the position of mirror M 1  will provide focus or diopter adjustment. To reduce costs, the adjustment of mirrors M 1 , M 2  may be done manually, or alternatively the position of M 1 ,M 2  may be fixed with respect to the separate optical assemblies and focus/diopter adjustment may be made directly at the eyepieces E 1 , E 2 . 
     Mode 2 
     (Transmissive Mode—Virtual Reality) 
     In an alternative configuration, the reflective display devices RD 1 ,RD 2  can be replaced by a single transmissive display device (such as an AMLCD). In this configuration, stereoscopic images (a separate image to each eye) can be achieved with a single display element by “time multiplexing” the two images (i.e. by alternating quickly between each light source). This “time multiplexing” is achieved by utilizing two separate light sources LS 4 , LS 3  for the illumination of the left and right eye images respectively. It is of course also possible to provide the same image to the left and right eyes simultaneously, simply by switching light sources LS 3 , LS 4  on at the same time. In many ways the transmissive configuration is more efficient because only a single display element is required for the generation of two separate images. However, at the present time, reflective display technology is significantly more advanced than similar transmissive display technology. One of the major differences in the hardware configuration when a transmissive element is utilized is that there can no longer be a physical connection between beamsplitters BS 1 , BS 3  and BS 2 , BS 4 . In this configuration beamsplitters BS 3 , BS 4 , light sources LS 3 , LS 4  and transmissive display device TD form a single optical assembly. This leaves optical assembly-1 consisting of beamsplitter BS 1 , eyepiece E 1  and optional components infrared filter F 1 , light sources LS 1 , and miniature camera ETC 1 . Optical assembly-2 consists of beamsplitter BS 2 , eyepiece E 2 , and optional components infrared filter F 2 , light sources LS 2 , and miniature camera ETC 2 . The operation of this modified optical assembly will be similar to that previously described, with the exception that it is now an absolute requirement that mirrors M 1 ,M 2  are adjustable independently from optical assemblies 1,2 in order to maintain the correct focal position. 
     Additional Enhancements 
     With the reflective design, the inclusion of collimating optics CL 1 , CL 2  (as shown in FIGS. 1 to  7 ) can greatly improve the overall optical efficiency of the system without seriously degrading the image quality. 
     Mode 3 
     Reflective focusing optics M 1 , M 2  can be replaced by refractive focusing optics consisting of lenses FL 1 , FL 2  as shown in FIG.  6 . Although theoretically the mirror elements could be now be eliminated from the design by simply orienting beamsplitters BS 1 , BS 2  to reflect the light directly into eyepieces E 1 , E 2 , respectively, in practice, this would require the focusing optics FL 1 , FL 2  to have an excessively short focal length. A more practical solution to this configuration is to replace concave mirrors M 1 , M 2  with plain mirrors, leaving the beamsplitters BS 1 , BS 2  as in the previous configuration. This variation in the design greatly increases the required focal length of optics FL 1 , FL 2  which eases the design requirements. 
     Color Generation 
     The generation of color in this HMD design can be achieved with either spatial color (red/green/blue picture elements) or by temporal color (display of rapid succession of red, green, and blue image fields, usually by changing the color of the light incident on the display device), and is dependent upon the display devices utilized. 
     Alternative Configurations 
     The alternative configurations of this HMD as shown in FIGS. 1 to  7  simply highlight alternative configurations for a specific component of the overall system. It is of course possible to combine any of the configurations shown in FIGS. 1 to  7  to form an overall system with the desired characteristics. 
     The rotational orientation of the sub-assembly consisting of light source LS 3 , beamsplitter BS 3 , and optional components RD 1 , CL 1  and its counterpart consisting of light source LS 4 , beamsplitter BS 4 , and optional components RD 2 , CL 2  is not a factor defining the intellectual property, and the illustrations showing a particular configuration should not be considered a reduction in the generality of this specification.