Abstract:
A physical element may be caused to appear to interact with an image displayed on a computer display screen. The position of the element with respect to the display screen may be determined automatically. The user can then manipulate the element to cause an image, which may appear to be connected to the element, to be altered. Therefore, the user gets the impression that the element is capable of interacting and altering an image displayed on the display screen.

Description:
BACKGROUND  
       [0001]     This invention relates generally to processor-based systems.  
         [0002]     A processor-based system may include a display having a display screen. The display screen may display images generated by a processor-based system. Generally, there is no way to interact with the images generated on that system in any extensive fashion.  
         [0003]     For example touch screens are available which enable the user to touch the display screen and thereby to select an icon displayed on the screen. However, this operation requires a specialized display screen. The display screen must include a sensor which detects the presence of the user&#39;s finger and thereby correlates that presence to the selection of an appropriate icon. Thus, the interaction that is possible is a direct result of the special configuration and construction of the display screen itself. Such interaction is not possible with any display screen. Moreover, additional expense may be incurred in providing a display screen which is also sensitive to touch.  
         [0004]     With the advent of three dimensional graphics, relatively life-like images may be produced in computer displays. Ideally, the user would like to interact with those graphics. Currently, electronic interaction is possible through keyboards and other input devices.  
         [0005]     Thus, there is a need for a way to physically interact with the images displayed on a computer display screen. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0006]      FIG. 1  is a perspective view of one embodiment of the present invention;  
         [0007]      FIG. 2  is a perspective view of the embodiment shown in  FIG. 1 , with the user interacting with an image displayed on a display screen;  
         [0008]      FIG. 3  is a block diagram in accordance with one embodiment of the present invention; and  
         [0009]      FIG. 4  is a flow chart for software in accordance with one embodiment of the present invention. 
     
    
     DETAILED DESCRIPTION  
       [0010]     Referring to  FIG. 1 , an element  10  enables a user whose hand is indicated at A to interact with images being displayed on a computer display screen. In one embodiment, the element  10  includes a handle  12  that fits in the palm of the user&#39;s hand and presents a trigger  24  for operation by the user&#39;s index finger. Transversely connected to the handle  12 , a telescoping shaft may include a proximal portion  14  and a distal portion  16 .  
         [0011]     The shaft portions  14  and  16  may be splined to prevent relative rotation. In one embodiment, the portions  14  and  16  are spring biased to extend apart unless constrained.  
         [0012]     A sensor housing  18  may be coupled to the distal portion  16 . The sensor housing  18  may include detectors  22  that may detect the positions of spring biased light pens  20 . The spring biased light pens  20  telescope in and out of the housing  18  in the direction of the arrows. As the pens  20  extend in and out of the housing  18 , the detectors  22  detect the position of each pen  20  relative to the housing  18 .  
         [0013]     Thus, turning to  FIG. 2 , the element  10  is shown in position pressed against the display screen  28  of a computer display  26 . The display  26  may be any type of computer display including a computer monitor.  
         [0014]     In this case, the pens  20  are pressed against the screen  28 . Because three equally spaced pens  20  are utilized, the angular orientation of the element  10  with respect to the display screen  28  may be determined based on the extension of each of the pens  20  with respect to the housing  18 . However in the alternative embodiments, the element orientation detectors and the light pens may be separate devices.  
         [0015]     In some embodiments, the display screen  28 , which may be glass, may be covered by another surface which may be flat and transparent. However, other shapes and transparencies may be used.  
         [0016]     In the illustrated embodiment, the element  10  interacts with an image  30  displayed on the display screen  28 . In one embodiment, the image  30  may be a scissors-type gripper. The gripper image  30  may grip a second image  32  such as a test tube. The user may press against the screen  28  to cause the images  30  and  32  to appear to extend “deeper” into the display screen  28 . In actuality, the images  30  and  32  are altered to create this effect under computer control.  
         [0017]     Thus, the exact position of the pens  20  on the display screen  28  may be determined in a fashion described hereinafter. The angular orientation and extension of the portions  14  and  16  may also be determined. As a result, the orientation of the element  10  with respect to the image  30  may be determined. This information may be utilized to allow the element  10  to seemingly interact with and actually alter the image  30 . For example, as the user presses the element against the screen  28  against the spring bias between the portions  14  and  16 , the image  30  may appear to extend further into the screen  28  as if the image  30 , were actually part of the physical element  10 . This allows the user, whose hand is indicated at A, to apparently physically interact with images  30 ,  32  displayed on the display screen  28 .  
         [0018]     Referring to  FIG. 3 , the element  10  may be coupled to a processor-based system  39  through a video pass through box  38  in one embodiment. The video pass through box  38  may receive video control signals from the processor-based system  39  headed for the display  26 . Thus, the pass through box  38  may receive the vertical and horizontal sync signals that the system  39  may generate to control the display  26  in one embodiment of the present invention. In addition, the pass through box  38  receives the detector  22  signals from the element  10 .  
         [0019]     The pass through box  38  may be of the type conventionally utilized with light pens to determine a location on a display screen selected by a light pen. An example of such a box is the PXL-2000 USB External Interface available from FastPoint Technologies, Inc., Stanton, Calif. However, other techniques for identifying the location of the light pens  20  on the display screen  28  may also be used.  
         [0020]     The pass through box  38  may receive signals from the light pens  20   a ,  20   b  and  20   c.  Each light pens  20  may detect light signals generated by one or more pixels making-up the display screen  28 . The optical transducers  34  convert those light signals into electrical signals and provide them to the video pass through box  38 . In one embodiment, the signals pass through the video pass through box  38  into a serial port such as a Universal Serial Bus hub  50  coupled to the processor-based system  39 .  
         [0021]     The detectors  22   a - c  that detect the extension of the pens  20  with respect to the housing  18  may also generate signals. In one embodiment, the detectors  22  may be rheostats that generate a signal indicative of the extent of spring biased extension of the pens  20  from the housing  18 . Those analog signals may be converted to digital signals by the analog-to-digital converters  36 . The digital signals may also pass through the pass through box  38  and the hub  50  to the processor-based system  39 .  
         [0022]     A detector  22   d  may be associated with the portions  14  and  16  to determine their relative extension. Thus, the detector  22   d  determines the relative positions of the portions  14  and  16  which may be the result of the user pressing the element  10  into the screen  28  or releasing that force. In one embodiment, all of the detectors  22  may be rheostats.  
         [0023]     Finally, user operation of the trigger  24  may generate signals. Each time the user operates the trigger  24 , the extent of trigger deflection and its duration may be encoded into an analog signal. That analog signal may be converted into a digital signal by the analog-to-digital converter  36   e.  This digital signal, like the other signals discussed above, is passed through the video pass through box  38  and the hub  30  to the processor-based system  39 .  
         [0024]     The processor-based system  39  may include a processor  44  coupled to system memory  46 . The processor  44  may be coupled to a bus  42  which in turn is coupled to a graphics interface  40  in one embodiment. Signals generated by the processor  40  may be passed through the graphics interface  40  and the video pass through box to the video display  26 .  
         [0025]     A bridge  48  may also be coupled to the bus  42 . In one embodiment, the bridge  48  may be coupled to the hub  50  as well as a storage device  52  which may be a hard disk drive. The storage device  52  may store software  54  for controlling the interaction between the element  10  and the display screen  28 .  
         [0026]     The video pass through box  38  may receive the graphics signals intended for display on the video display  26 . Thus, the box  38  may receive the vertical and horizontal sync signals as well. By extracting those vertical and horizontal sync signals, and comparing their timing to the timing of signals received from the light pens  20 , the system  39  can determine the location of the light pens  20  on the display screen  28 . In particular, the light pens  20  receive a flash of light when a particular underlying display screen  28  pixel is activated. The pixels may be sequentially activated in response to vertical and horizontal sync signals generated by the system  39 . Thus, the time from vertical and horizontal sync signal to light flash is indicative of screen  28  position of the pens  20 .  
         [0027]     In one embodiment, a vertical sync signal is generated to start each new frame. A horizontal sync signal is generated with the beginning of each line. Thus, by knowing when a light signal is received by a light pen  20  relative to when a corresponding vertical sync signal and horizontal sync signal was detected, the system  39  may determine the vertical and horizontal coordinates of each light pen  20 . The pass through box  38  may do the initial analysis to determine the pen  20  position or may simply forward the raw information onto the system  39  for analysis.  
         [0028]     The software  54 , shown in  FIG. 4 , may begin in one embodiment, by determining whether the light pen data has been received as indicated in diamond  56 . If so, that data may be correlated to the vertical and horizontal sync signals as indicated in block  58 . The frame and screen coordinates for each particular received light pen signal may then be determined as indicated in block  60 .  
         [0029]     Next, a check at diamond  62  indicates whether the detector  22  data was received, as indicated in diamond  62 . If so, the angle of the element  10  with respect to the display screen  26  may be calculated. In addition, the distance of a handle  12  from the display screen is also calculated as indicated in block  64 , using the shaft data from the portions  14  and  16 .  
         [0030]     A check a diamond  66  determines whether trigger activation has occurred. If so, an image such as the gripper image  30  may be altered. For example, the gripper image  30  may appear to “open”. For each unit of trigger activation in terms of time, a corresponding operation may be created virtually on the display screen  28  in one embodiment.  
         [0031]     Based on the change in relative position between the portions  14  and  16 , relative motion of a handle  12  with respect to the display screen, or rotation of the handle  12  relative to the display screen, the orientations or images  30  and  32  may be recalculated. The signals that generate the images  30  and  32  may be received and the revised signals may be transmitted to the display screen  26  for a display as indicated in block  70 .  
         [0032]     The images  30  and  32  may be caused to move inwardly, as if they were coupled to the element  10 , by pressing the element  10  harder against the screen  28 . This action is detected by the detector  22   d . Similarly, the element  10  may be rotated or angularly adjusted with respect to the screen causing a corresponding change in position of the images  30  and  32 . This action is detected by the detectors  22   a - c.  Similarly, operation of the trigger  24  may cause the preprogrammed change in one or both images  30  and  32 .  
         [0033]     In each case, three dimensional manipulation of the element  10  may result in a corresponding three dimensional alteration of an image  30  or  32 . As a result, it may seem that the element  10  is physically linked to an image  30  or  32 .  
         [0034]     While the present invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of this present invention.