Abstract:
A graphical user interface is implemented as a human interface device which can directly sense where on a domed screen a user is pointing. The human interface device can be an optical pointing device which emits a beam of light, preferably produced using a laser, which can be tracked by a camera appropriately located inside the dome. As an alternative, the human interface device can be a hand held device capable of tracking position and orientation. Data produced by the human interface device is then detected and mapped to the location for a corresponding cursor.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    The present invention generally relates to systems for displaying images on dome-shaped screens, and in particular, to human interface devices for use with such systems. 
         [0002]    Numerous systems have been developed for displaying images on any of a variety of dome-shaped screens. For example, hemispherical screen display systems have commonly been used in planetariums and other immersive environments. Such displays are typically 180 degrees from their front, through their topmost or zenith point to their back, although many are built with screens that are less than or greater than 180 degrees. 
         [0003]    Such display systems have recently migrated to digital technology, leading to the need for increased interactivity with the display. Traditional human interface devices (HID&#39;s) used to operate such digital systems have, in practice, been found to be very problematic in this environment. The problems associated with such devices, for example, a computer mouse, a graphics tablet or a trackball, can be separated into two categories. 
         [0004]    First, traditional HID&#39;s typically operate in a planar coordinate system, whereas the dome display is better suited to a spherical coordinate system. In particular, it has been found that in practice, linear movements of a mouse, for example, will not naturally produce a corresponding movement of the cursor on a domed screen. As an example, the vertical movement of a mouse will naturally move the cursor on a flat display directly from the bottom to the top of the screen. On a hemispherical screen, however, similar vertical movement of a mouse will not typically translate to a straight line path of the cursor on the screen. As a result of this, an operator will quickly find themselves very disoriented in attempting to predict how inputs from a planar HID will translate into cursor movements on the domed screen. 
         [0005]    Second, the curved surface of a dome display makes the incremental nature of most traditional HID&#39;s problematic. On a flat screen, cursor movements shown when a user moves an HID device correspond naturally to the user&#39;s frame of reference. When a normally flat image is curved onto a dome, however, this correspondence is lost so that, for example, a movement of the HID from left to right might become a movement on the display from right to left, depending on the cursor&#39;s location on the screen. 
         [0006]    In order for the user to place the cursor in a desired location, the user must first know the cursor&#39;s current location to determine which way to move it. This is easy to determine with a traditional computer monitor, having a substantially flat display, but much more difficult with a dome display system that is radically different from the user&#39;s frame of reference. Put simply, it is very easy on a dome display to lose track of the cursor&#39;s location. Digitizer tablets do provide a rigid mapping between the pen position and the cursor position on the display, which is helpful. However, the size and shape of the digitizer tablet still makes it very difficult for a user to translate a position on the tablet to a position on the domed screen. 
       SUMMARY OF THE INVENTION 
       [0007]    In accordance with the present invention, these and other problems associated with existing HID&#39;s for dome display systems are resolved by providing a graphical user interface implemented as a human interface device which can directly sense where on a domed screen a user is pointing. 
         [0008]    A human interface device in accordance with the present invention can be implemented as an optical pointing device which emits a beam of light, preferably produced using a laser, which can be tracked by a camera appropriately located inside the dome. With appropriate optics and software, the spot of the beam on the domed screen can be detected and mapped to the corresponding location of a rectangular computer display. 
         [0009]    As an alternative, a hand held device which includes position and orientation tracking can be used. Such a device would be continually aware of the location, pointing direction and velocity of movement of the hand held device, which can similarly be detected and mapped to the corresponding cursor location. Examples of such devices can include Wii® controllers, SmartPhones and Air Mice. Given the state of today&#39;s technology, using such devices is currently considered to be less desirable because such use would then require regular calibration and high precision for the detection of location and acceleration in order to determine, with sufficient accuracy, the user&#39;s position and pointing direction within the domed screen area. Consequently, given the current state of the technology, the use of an optical pointing device is currently considered to be more practical. As the technology develops, however, the potential for using hand held devices is expected to increase. 
         [0010]    For a further description of the human interface device of the present invention, reference is made to the description of preferred embodiments which is provided below, together with the following drawings. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0011]      FIG. 1  is a schematic view showing a human interface device produced in accordance with the present invention. 
           [0012]      FIG. 2  is a flow chart showing image processing for implementing a human interface device produced in accordance with the present invention. 
       
    
    
     DESCRIPTION OF PREFERRED EMBODIMENTS 
       [0013]      FIG. 1  schematically illustrates an example of a system for projecting images onto a domed screen, and a human interface device in accordance with the present invention for controlling functions associated with the images projected onto the domed screen. 
         [0014]    In the illustrated embodiment, the images are projected onto a hemispherical screen  1  using a centrally located projector  2 . It is to be understood, however, that the human interface device of the present invention can also be used with other types of domed screens, and with one or more projectors which can be located at any of a variety of different locations with respect to the domed screen which is used. 
         [0015]    For use of the display system in a digital environment, a computer  3  is coupled with the projector  2  for appropriately delivering digitally stored images to the projector  2 , via the connection  4 . A fisheye lens  5  is conventionally coupled with the projector  2  for the appropriate display of projected images onto the screen  1 . 
         [0016]    Interactive operation of the display system, which can include cursor movements, operational instructions, for example, to “click” on a displayed operating parameter, or other desired operations, is achieved in accordance with the present invention using a portable human interface device  10  which can be carried by a user  11 , and directed at the screen  1  irrespective of the user&#39;s location within the domed space defined by the screen  1 . 
         [0017]    In the illustrated embodiment, the human interface device  10  is an optical pointing device which transmits a beam of light  12  onto the screen  1 , producing a spot at  13 . A camera  14  is provided for receiving signals reflected from the surface of the screen  1 , for detecting the spot  13 , which is schematically represented at  15 , and is operationally coupled with the computer  3  via the connection  16 . The connection  16  can be implemented using any of a variety of wired connections, such as a GigE, FireWire® or USB3 connection, or using wireless communications, if preferred. 
         [0018]    As will be more fully described below, the computer  3  is a general purpose computer provided with appropriate software for detecting and mapping the spot  13  on the domed screen  1 . A desired pointer, such as a cursor, can then be projected onto the screen  1 , at  17 , responsive to mapping of the spot  13 , using the projector  2  and the fisheye lens  5 . 
         [0019]    As an alternative, the human interface device  10  can also be a hand held device incorporating a position locating circuit for transmitting position and orientation, such as a Wii® controller, SmartPhone or Air Mouse. In such case, a suitable receiver would be provided for communicating with the hand held device selected for use, which could either replace the camera  14  and communicate with the computer via the connection  16 , or which could be incorporated in the computer  3 , as desired. 
         [0020]    Implementing the human interface device  10  using an optical pointing device requires various special accommodations. 
         [0021]    For example, the extreme field of view of the display system, particularly for hemispherical screens, requires that the camera  14  have an equally wide field of view to effectively track the pointer. This makes it necessary to fit the camera  14  with a fisheye lens  18  capable of imaging the entire screen  1 . This is preferably implemented with a Fujinon FE185C046HA-1 lens. 
         [0022]    To provide acceptable resolution over a large screen, such as the hemispherical screen  1 , the display system will typically employ extremely high resolution projectors. Often, more than one projector is employed to cover the entire screen surface. The cursor needs to be accurately positioned within that very high resolution image. Employing an equally high resolution camera  14  to achieve this is, however, typically impractical. The image processing computer  3  needs to process at least 20 frames per second in order to maintain smooth and responsive cursor motion, which is more difficult to achieve with extremely high resolution imagery. 
         [0023]    For this reason, and in order to achieve an appropriate level of performance, the use of a camera such as a JAI BM-141GE C-Mount camera is presently considered to be preferred, coupled with image processing for determining the beam location with sub-pixel accuracy. This allows the cursor to be positioned with a much higher degree of accuracy than the camera would natively provide. 
         [0024]    Such image processing is preferably implemented in software, employing the available computer  3 . Sub-pixel accuracy is derived through calculation of the image centroid across multiple pixels, where: 
         [0000]    
       
         
           
             
               Centroid 
                
               
                   
               
                
               
                 ( 
                 
                   x 
                   , 
                   y 
                 
                 ) 
               
             
             = 
             
               
                 
                   ∑ 
                   
                     ( 
                     
                       pixel 
                        
                       
                           
                       
                        
                       coordinate 
                        
                       
                           
                       
                        
                       
                         ( 
                         
                           x 
                           , 
                           y 
                         
                         ) 
                       
                     
                     ) 
                   
                 
                 
                   number 
                    
                   
                       
                   
                    
                   of 
                    
                   
                       
                   
                    
                   pixels 
                 
               
               . 
             
           
         
       
     
         [0000]    The values for the input pixels x and y will always be integers defining a specific x and y coordinate pair on the camera&#39;s grid. The output centroid value will in most instances be a fractional number, thereby giving the desired sub-pixel accuracy. 
         [0025]    Referring to  FIG. 2 , prior to calculation of the image centroid, the image received by the camera  14  is first scanned, at  20 , to find all pixels having a value greater than or equal to a selected threshold. If the detected number of pixels having a value greater than or equal to the selected threshold exceeds a selected number, which is determined by the test  21 , the assumption is made that the room lights are on and the event is ignored, at  22 . If the detected number of pixels having a value greater than or equal to the selected threshold is less than the selected number, as determined by the test  21 , the image centroid is then calculated, at  23 , in accordance with the above described calculation. The calculated image centroid is then suitably mapped to coordinates and a mouse event is generated, at  24 , using techniques which are otherwise known. 
         [0026]    Even with the high processing frame rate and sub-pixel accuracy which is achieved, it is anticipated that with such wide angles there will be some visible discrepancy between the spot  13  on the screen  1  and the true position for the cursor. In practice, this will only be recognizable to the user if they are able to see both the spot  13  and the displayed cursor. For this reason, it is preferable to use a source which produces light outside the visible spectrum and a camera  14  which is capable of sensing such light. The user is then unable to see the spot  13  and does not detect any slight positional deviations from the displayed cursor position. This can be accomplished using the previously specified JAI BM-141GE C-Mount camera  14 , coupled with a Semrock FF01-780-12-20 filter, in combination with an infrared (“IR”) laser light source  12 . 
         [0027]    In order to maximize image contrast and to minimize cross-reflectance, it is typical for spherical projection screens to be finished with a low gain surface. This is a finish which reflects only a limited amount of the incident light. Flat screens, on the other hand, are typically finished with a unity gain surface which reflects virtually all of the incident light. The lower gain finishes found on spherical projection screens (typically ranging from 38% to 80% reflective) make identifying the spot  13  considerably more difficult. To improve the ability of the camera  14  and the image processing software to discriminate the spot  13  from the background image, the camera  14  is preferably fitted with an optical filter which restricts the light reaching the camera&#39;s imaging chip to a narrow bandwidth. This allows the camera  14  to see the spot  13  without seeing the visible light associated with the projected image, or other background noise sources. This also allows the system to be operated with a Class I laser device, eliminating safety concerns for the audience and for the operator. 
         [0028]    Because of the extremely wide field of view involved in operating a domed display, particularly for hemispherical displays, it is difficult to ensure that the only light the camera  14  will detect is light from the beam  12 , such as the spot  13 , as is intended when the operator is pointing at the screen  1 . It is very possible, and indeed likely, that in some instances the camera  14  will see light from another source, such as a stray light source that happens to include energy inside the bandwidth of the light source used to display the spot  13  (for example, an incandescent exit light) or a reflection of the light source off of some surface other than the projection screen  1 . This can lead to errors in determining the desired cursor position. To filter these undesired signals, the image processing software preferably includes a software mask which can be configured by the user to determine what portion of the camera&#39;s field of view corresponds to the surface of the screen  1 . Any light detected outside of this area is ignored. 
         [0029]    To enhance the flexibility of the foregoing system, it is preferable for the camera  14  to be capable of being positioned at any desired location within the region defined by the domed screen  1 . For this reason, and to avoid the need to place the camera  14  in a single, critical position within the dome, it is preferable to provide for an adjustment of the pointer mapping based on the location selected for the camera  14 . To that end, an initial calibration is performed, in software, by displaying several reference points on the screen  1  and having the user aim the beam  12  at each of the reference points, in sequence. The spot detected by the camera  14  can then be correlated with the reference points. The number of calibration points is variable, with a minimum of two. Four calibration points have been found to work well in practice. Such calibration does not need to be repeated unless the detector camera  14  is repositioned. 
         [0030]    It will be understood that while the present invention has been described based on specific embodiments incorporating specified parts, the present invention further encompasses all enabled equivalents of the parts described, and that various changes in the details, materials and arrangement of parts which have been herein described and illustrated in order to explain the nature of this invention may be made by those skilled in the art within the principle and scope of the invention as expressed in the following claims.