Abstract:
The invention is related to a system for interacting with a display through a display window, comprising a recording unit configured to be placed behind the display window for recording a pointer object in front of said display window, at least one deflection unit configured to direct a beam path from a space in front of the display window to said recording unit, a computer unit configured to be connected to said recording unit for determining a position of the pointer object with respect to the display window, and shield means by means of which beam paths that are not deflected by the deflection unit are suppressed, and to a corresponding method. The invention achieves a good contrast between the pointer object and the background and therefore an improved recognition of the pointer object.

Description:
CROSS REFERENCE TO RELATED APPLICATION 
       [0001]    This application claims the benefit under 35 USC 119 of German Patent Application No. 10 2006 006 343.0 filed Feb. 8, 2006, the entire disclosure of which is incorporated herein by reference. 
       FIELD OF THE INVENTION 
       [0002]    The invention concerns a system and method for interacting with a display through a display window. 
       BACKGROUND OF THE INVENTION 
       [0003]    In conventional touchscreens, a user moves a pointer object, e.g. his fingers, on a display that is identical to a touch surface. The position on the display and/or the movement of the pointer object with respect to the display is detected by mechanical sensors. A predetermined event is triggered depending on the position and/or on other actions of the pointer like pushing onto a certain location on the display. Such systems are susceptible to mechanical wear and thus because of increased maintenance requirements in many cases not suited for continuous public use. 
         [0004]    Systems that simulate such a mechanical touchscreen by means of optical detection of the pointer object with respect to a passive touch surface are known. For example, U.S. Pat. No. 7,034,807 discloses a configuration for user interaction with a display visible through a display window (shop window). The area of the display as visible through the window acts as passive touch surface. The system comprises a recording unit for recording a pointer object in the vicinity of the touch surface by optical means. A computer unit connected to the recording unit is capable of determining a position of the pointer object in the vicinity of the touch surface from the signals recorded by the recording unit. The display is located behind the display window, i.e. separated from the user by the display window. The recording unit comprises two cameras, for example. It is also located behind the display window. By means of a deflection unit, e.g. mirrors or prisms, located above or beneath the touch surface, a beam path is directed from the recording unit to a space in front of the touch surface. In order to enhance contrast, the space in front of the display/touch surface is illuminated with infrared light having the same beam path as the recording unit. Reflective surfaces are arranged opposite the deflection unit with respect to the touch surface. They reflect the infrared light and act as background in front of which the pointer object can be detected with increased contrast. The position of the pointer is then determined, e.g. by means of triangulation. 
         [0005]    Similar camera-based touch systems are for example disclosed by WO-A 02/03316, EP-A 1420335, and DE-A 10163648. These documents teach to use a recording unit with at least two cameras having overlapping fields of view that encompass a touch surface bordered by a frame. The presence of a pointer on or in the vicinity of the touch surface is determined by analyzing the difference between two consecutive images of the touch surface. If a pointer is present, the acquired image data is processed such that only selected subregions of the image are evaluated. 
         [0006]    Known camera-based touch systems have the problem that the images acquired by the recording unit are often distorted by light from other sources than the pointer, e.g. directly incident, scattered or reflected light from the environment. Light from such “external” sources often has a much higher intensity than light emerging from the pointer object. As the camera control adapts to the maximum intensity, the ability to detect the pointer is often greatly reduced. The image recorded by the recording unit is also very sensitive to change in external illumination, e.g. day and night. These circumstances lead to errors in the determination of the position of the pointer object. 
         [0007]    To reduces these errors, it is common to use an additional IR-illumination of the space in front of the touch surface, and to use IR sensitive cameras to acquire the image. Contrast is further increased by the above mentioned reflective surfaces. The illuminated pointer object can then be detected with higher contrast. This, however, makes the system very complex and costly. 
         [0008]    A further problem arising with known systems, e.g. as shown in U.S. Pat. No. 7,034,807, is that the field of view of the recording unit is such that not only the region around the touch surface but even parts of the surrounding space are recorded. Therefore, an image recorded by the recording unit comprises a lot of side information that complicates extraction of the correct position of pointer object. It is difficult to distinguish which of the objects within the field of view are supposed to act as pointer object, whether a pointer object indeed touches the window or whether it is at a distance therefrom and no interaction is intended. 
         [0009]    A further problem arises with double glazed display windows. Light can be reflected between the panes of the window, reach the recording unit, distort the image acquired and thus falsify the determination of the position of the pointer object. 
       SUMMARY OF THE INVENTION 
       [0010]    It is therefore an object of the present invention to provide a system for interacting with a display behind a display window with improved capacity of detecting the position of a pointer object with respect to a touch surface of the display window. In particular, the setup of the system shall be simple so that it can be installed at any window without much effort and costs. Furthermore, the image generation and evaluation shall be simple and have an increased reliability and stability with respect to a change in external lightning conditions. 
         [0011]    It is a further object of the invention to provide a system for interacting with a display behind a display window that can be implemented with standard components, in particular with a standard camera. 
         [0012]    It is a further object of the invention to provide a method for interacting with a display behind a display window with improved capacity of detecting the position of a pointer object with respect to a touch surface of the display window. 
         [0013]    These and further objects are achieved by a system for interacting with a display through a display window comprising: a recording unit configured to be placed behind the display window for recording a pointer object in front of said display window; at least one deflection unit configured to direct a beam path from a space in front of the display window to said recording unit; a computer unit configured to be connected to said recording unit for determining a position of the pointer object with respect to the display window; and shield means by means of which beam paths that are not deflected by the deflection unit are suppressed. 
         [0014]    The objects are also achieved by a method for interacting with a display through a display window comprising the steps of recording a pointer object in front of said display window by means of a recording unit placed behind the display window; directing a beam path from a space in front of the display window to said recording unit by means of at least one deflection unit; determining a position of the pointer object with respect to the display window by means of a computer unit connected to said recording unit; suppressing beam paths that are not deflected by the deflection unit by means of shield means. 
         [0015]    The system for interacting with a display through a display window comprises a recording unit configured to be placed behind the display window for recording a pointer object in front of said display window, in particular on or in the immediate vicinity of a touch surface on the display window. The touch surface normally corresponds to the surface area of the display window through which the display located behind the display window is visible, or to a part of said surface area. The recording unit preferably comprises one or more cameras or other optical sensors that are capable to produce an image or spatial information in other ways, e.g. a CCD array. 
         [0016]    The system further comprises at least one deflection unit configured to direct light emerging from a space in front of the display window, in particular from the immediate vicinity of the touch surface, to said recording unit. In other words, the deflection unit deflects light in such a way that the field of view of the recording unit located behind the display window encompasses said space in front of the display window, and preferably encompasses the touch surface. The deflection unit may comprise one or more mirrors or prisms. 
         [0017]    The system further comprises a computer unit configured to be connected to said recording unit for determining a position of the pointer object with respect to the display window, in particular with respect to the touch surface. Preferably, at least two images or parts thereof taken by the recording unit from different perspectives are evaluated. From the location of the (image of the) pointer object within the images taken by the recording unit, the location of the pointer object on or with respect to the touch surface is calculated. 
         [0018]    The system further comprises shield means by means of which beam paths that are not deflected by the deflection unit are suppressed. Light traveling along these “suppressed” beam paths thus does not reach the recording means at all or contributes to the image generated by the recording means with reduced intensity. These shield means ensure that there is a good contrast between the pointer object and its background, while reducing the intensity of light from other sources than the pointer object. Preferably they ensure that direct incidence of light onto the recording means is not possible, to prevent overdriving the recording means. Also, reflections at a surface of the display window contribute to the image at most with reduced intensity. The shield means according to the invention make it possible to do without external illumination of the pointer object and without reflective surfaces. It is sufficient to use the display itself as illumination of the pointer object and to detect light scattered by the pointer object. 
         [0019]    In a preferred embodiment of the invention, the shield means are configured such that beam paths corresponding to a direct incidence of light on the recording unit are suppressed, and preferably do not reach the recording means at all, preventing overdrive of the control of the recording means and thus ensuring a good contrast for detection of the pointer object. For example, this shield means may comprise at least one shading element that is arranged at a distance from a front surface of the display window with a deflection surface of the deflection means located between the front surface and the shading element. The shape of the shading element, its distance to the display window and its location with respect to the recording means are chosen such that no direct incidence of light onto the recording means, e.g. a lens of a camera, is possible. Further protective screens arranged at a front and/or a rear surface of the display window in the vicinity of the recording means are possible. 
         [0020]    In a further preferred embodiment of the invention, the shield means are configured such that beam paths corresponding to reflections at a surface of the display window are suppressed. This is especially beneficial in double glazed windows, where reflections at one of the surfaces of the two window panes can otherwise reach the recording means with full intensity. Preferably, the shield means in this embodiment comprise one or more protective screens, e.g. semi-transparent foils, arranged on the front and/or rear surface of the display window in the vicinity of the recording means. The protective screen has preferably only the function to attenuate the light reflected at one of the surfaces, e.g. by scattering. Preferably, transparency ranges from 40 to 90%. Though complete blocking of the reflected light is possible, this could lead to an overdrive of the camera control which possibly reduces the contrast of the pointer object in front of the background. 
         [0021]    The protective screen is preferably shaped in such a way that undistorted transmission of light deflected by the deflection means onto the recording means is possible. For example, it comprises an opening or aperture. 
         [0022]    Preferably, the recording means acquire images that encompass the touch surface. Pixels that correspond to other regions than the touch surface are preferably disregarded. To prevent that reflections at the touch surface under a small angle of incidence with respect thereto are erroneously identified as a pointer object, the touch surface is preferably encircled by a frame. This is not necessary if the display window itself already comprises a frame. 
         [0023]    In a further preferred embodiment of the invention, the computer unit generates an average background by averaging a plurality of images acquired by the recording unit. Differences with respect to this background on a small time scale, e.g. movement of the pointer on the touch surface, are detected in real time, thus fullfilling the requirements of reliable interaction with the display. They do not significantly contribute to the average background. Slow variations of the image, e.g. caused by change of night and day, or permanent alteration thereof, e.g. by dirt on the touch surface or the deflection unit, on the other hand, become part of the average background and do not affect detection of the “true” pointer object. 
         [0024]    The recording unit, e.g. the camera(s), can be attached directly at the rear surface of the display window. This has the advantage that oscillations of the display window do not affect the analysis of the acquired data. The display itself can be exchanged without having to calibrate the system. However, the system has to be calibrated in the place of use. 
         [0025]    A display having cameras integrated into its frame has the advantage that the position and orientation of the cameras with respect to one another is known. The calibration of the system can be done in the factory. Only a mount for the display, and not a separate one for the camera, is necessary. 
         [0026]    To reduce damage caused by vandalism, the cameras are arranged behind the display window. Furthermore, the deflection unit and any other elements arranged on the user-side of the window are preferably shaped with rounded edges in order to reduce a contact area, e.g. for climbing. 
         [0027]    The inventive technology can be implemented at virtually any kind of window glass in an extremely cost efficient manner. Double glazed or even mirrored windows can be utilized. The invention does not require any special components like specific cameras, additional light sources or reflectors. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0028]    Embodiments of the invention are shown in the drawings 
           [0029]      FIG. 1  shows schematically a setup of the inventive system in a view on a display window; 
           [0030]      FIG. 2  shows the setup as shown in  FIG. 1  in a view parallel to the display window; 
           [0031]    FIG.  3 A+ 3 B show a deflection unit and shield means in one embodiment of the invention; 
           [0032]      FIG. 4  shows a flow chart of different steps of a method of processing acquired image data; 
           [0033]      FIG. 5A-C  show how spatial information is extracted from an image; 
           [0034]      FIG. 6  shows a routine for detecting a pointer object in front of a background. 
       
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0035]      FIG. 1 and 2  show a setup of the inventive system in a view on a display window  10  and parallel to the display window  10 , respectively. The system comprises a recording unit  18  with here two cameras  20 ,  20 ′. The cameras are located behind the display window  10 . “Behind the display window” denotes the space that is separated from a user  60  by the display window. “In front of the display window” is the space that is accessible by the user  60 . The cameras  20 ,  20 ′ are arranged at a distance from one another and have overlapping fields of view  24 ,  24 ′. By means of a deflection unit  28 ,  28 ′ arranged in front of the display window  10 , the beam paths are shaped such that the fields of view  24 ,  24 ′ encompass a part of a front surface  12  of the display window  10 . Near the cameras  20 ,  20 ′, their optical axes  25 ,  25 ′ are generally perpendicular to the display window  10 . They are then deflected such that they run generally parallel to the display window  10 . The optical axes  25 ,  25 ′ are arranged at an angle α of about 90° with respect to one another. Other angles are possible, as long as the fields of view  224 ,  24 ′ do not coincide. 
         [0036]    A display  16  is located behind the display window  10  in the immediate vicinity of a rear surface  14  thereof, preferably mounted directly to the display window  10  (mount not shown). It, is located such that it lies within the field of view  24 ,  24 ′ of both cameras  20 ,  20 ′, as seen through the display window  10 . The area of the front surface  12  where the display  16  is visible and that is contained in the fields of view  24 ,  24 ′ of both cameras  20 ,  20 ′ can act as touch surface  17 . This means that the location of a pointer object  30 , here a user&#39;s finger, on or in the immediate vicinity of the touch surface  17  can be detected and transformed into a movement of a pointer object, e.g. a mouse pointer, on the display  16 . It is also possible to use only a part of the display  16  as visible through the display window  10  as touch surface  17 . As the cameras  20 ,  20 ′ view generally parallel to the display window, the touch surface  17  is represented as a narrow strip in the images taken by the cameras  20 ,  20 ′. The touch surface  17  may be encircled by a frame  26 . This prevents distortion of the images taken by objects passing by the display window  10 . 
         [0037]    A computer unit  40  is arranged behind the display window  10 . It receives data from the recording unit  18  and is able to extract therefrom an information on whether a pointer is present, where it is located and/or how it is moved. This information is used to move a pointer object on the display  16  and/or to trigger a predetermined action, e.g. after detecting a movement corresponding to a “click” or “double click”. 
         [0038]    The computer unit  40  may be part of a standard computer used in connection with the display  16 , e.g. by running a certain program thereon. The inventive system preferably does not occupy more than 15% of the overall computing capacity of the computer. It is discussed below how the performance is optimized. 
         [0039]    The computer unit  40  is able to detect the position x′/y′ of the pointer object  30  on the touch surface  17  and to transform these world coordinates x′/y′ into screen coordinates x/y. In this example, transformation into screen coordinates x/y is particularly easy, because analysis of images taken by the left-hand side camera  20  yields coordinate y′ measured with respect to the optical axis  25 , and analysis of images taken by the right-hand side camera  20 ′ yields coordinate x′ measured with respect to the optical axis  25 ′. This will be described in more detail in connection with  FIGS. 5A-C . 
         [0040]    According to the invention, there are shield means that serve to reduce the contribution of beam paths not deflected by the deflection unit  28 ,  28 ′ to the image acquired by the recording unit  18 . They comprise a shading element  32 ,  32 ′ arranged at or in the vicinity of the deflection unit  28 ,  28 ′ and shown in more detail in  FIG. 3A . 
         [0041]    They further comprise protective screens  34 ,  34 ′,  35 ,  35 ′ arranged at the front and/or rear surface  12 ,  14  of the display window  10  in the vicinity of the cameras  20 ,  20 ′ and having an opening  36 ,  37  in the immediate vicinity of an objective  22  of the cameras  20 ,  20 ′. The function of these shield means will be described in connection with FIG.  3 A+B. 
         [0042]    FIG.  3 A+B show one example of a deflection unit  28  comprising a deflection surface  29 , e.g. a mirror, arranged at an angle of about 45° with respect to the optical axis of the camera  20  and to the front/rear surface  12 ,  14 . Beam paths  50  running in a plane parallel to the front/rear surface  12 ,  14  comprising the optical axis  25  are thus deflected onto the objective  22  of the camera. After deflection at the deflection surface  29 , the angle of incidence β with respect to a normal to the front/rear surface  12 ,  14  is small, e.g. smaller than 30°. 
         [0043]    A further beam path  53  corresponds to direct incidence of light without deflection at the deflection surface  29 , e.g. incidence of sun light or automobile headlights with a generally quite high intensity. Without shield means beam path  53  would reach the objective  22  and cause the camera control to adjust to the higher intensity, thus darkening the overall image corresponding to a smaller aperture. Beam path  53  is blocked by shading element  32  which is here formed in one piece with the deflection means  28 . Shading element  32  comprises a planar face  32 ′ running parallel to the front/rear surface  12 ,  14 . The deflection surface  29  is arranged between the face  32 ′ and the front/rear surface  12 ,  14 . The planar face  32 ′ is preferably shaped and arranged such that directly incident beam paths  53  having an angle of incidence β within a predetermined range, e.g. of 0 to 60° or 0 to 78-80°, are blocked. The face  32 ′ may be rectangular and run parallel to an edge of the deflection surface  29 . Generally, the complete body  38  of the deflection unit  28  may serve as shading element  32  blocking all beam paths having an angle of incidence β in a predetermined range. Preferably this angular range is chosen such that the complete field of view of the camera—if deflections at the deflection surface  29  are not taken into account—is covered. This is indicated, by further beam path  53 ′. Generally, all beam paths corresponding to direct incidence of light having an angle of incidence β in a predetermined range are blocked by a shading element arranged at a distance from the front surface, potentially in combination with a protective screen on one of the surfaces  12 ,  14 . 
         [0044]    Another beam path  52  having an angle of incidence β outside the range blocked by the shading element  32  is attenuated by protective screen  34  arranged on the front surface  14 . The protective screen  34  comprises a semi-transparent foil that does not completely block but attenuate the light. Directly incident beam paths  52  with angles of incidence β within a further predetermined range, e.g. 45 to 80°, are affected by the protective screen  34 . Furthermore, protective screen  34  attenuates beam paths  54  traveling inside the display window  10 , e.g. by reflections between two panes  10 ′,  10 ″ of a double glazed window. Beam path  54  as shown here does thus reach the deflection surface  29  and the objective  22  with reduced intensity and has thus less potential of distorting the image. 
         [0045]    A further beam path  55  that is reflected at the inside of outer pane  10 ′ is attenuated by a further protective screen  35  arranged at the rear surface  14 . 
         [0046]    Generally, all beam paths corresponding to light reflected at one of the faces of the display window is attenuated by a protective screen arranged at the front and/or rear surface or at a distance therefrom. 
         [0047]    The size of the protective screens  34 ,  35 ,  34 ′,  35 ′ is determined from the aperture of the cameras and the thickness of the display window or the distance between two window panes  10 ′,  10 ″, such that no light reflected within the display window itself can reach the cameras  20 ,  20 ′. Furthermore, it is avoided that directly incident light beams reach the cameras. 
         [0048]    By the combination of the protective screens  34 ,  35  with the shading element  32  it is achieved that the image acquired by the camera  20  is less distorted by reflections within the display window  10  and direct incidence of light. The contrast within the actual region of interest, i.e. the space in front of the touch surface  17 , is thus enhanced. 
         [0049]    As shown in  FIG. 1 , the protective screens  34 ,  35  may as well be arranged projecting laterally beyond the body of the deflection unit  28  at all sides. 
         [0050]    The body  38  of the deflection unite  28  is rounded in order to give as little area to serve as a grip or to step on as possible. Furthermore, the shading element  32  is tapered such that it cannot be easily gripped. 
         [0051]      FIG. 4  shows a flow chart of different steps of a method of processing acquired image data.  FIG. 5A-C  illustrate this method in more detail. 
         [0052]    In a first step  102 , image or frame  42  is taken by the recording unit  18 , here a digital camera, and transmitted to the computer unit  40 . Each camera  20 ,  20  generates an image  42  in a format 320×240, or more. Only a subset of the pixels constituting the image  42  are used for further evaluation. Two parameters z 1 , z 2  define this subset ( FIG. 5B ), e.g. z 1 =118 and z 2 =124, that is extracted in step  104 . All other pixels are discarded, thereby achieving a better performance of the computer unit  40 . In this example, only (z 2 −z 1 )×320=6×320=1020 as compared to 240×320=76800 pixels have to be analyzed. The subset corresponds to a region of interest  44  that contains the touch surface  17  in a projection from a small angle of view. The discarded pixels correspond to an image of the face  32 ′, the screens  34 ,  35  and/or parts of the environment. The touch surface  17  basically appears as a narrow strip  17 ′ in the image  42  (see  FIG. 5B ). The z 1 -, z 2 -values can be determined during the installation of the system. In step  104 , the image  42  usually taken in color is converted into gray scale (R+G+B/3) in order to enhance contrast, especially in twilight or at night. 
         [0053]    Step  106  comprises a realtime motion detection and is further illustrated in  FIG. 6 . A motion detection algorithm detects where motions are present and calculates their position x′, y′ with respect to the optical axes  25 ,  25 ′. From these world coordinates x′, y′ a transformation into screen coordinates x, y is made. 
         [0054]    The software analyses 14 images per second, for example. An object is identified as pointer object  30  if the value of a pixel changes by a predetermined amount within a predetermined time interval. To avoid that a change in external lightning conditions falsifies this recognition, an average value for each pixel is determined, e.g. by taking the average of the past 10-20 images, and the difference with respect to this average is determined (step  108 ). This average background is constantly updated. Consequently, small but continuous changes in external lightning conditions enter into the average background and will not be recognized as pointer object. Furthermore, sudden but then constant alteration of the image acquired, e.g. by a chewing gum onto the touch surface, will only temporarily be identified as pointer object, but not affect the recognition of the true pointer object after a certain time. 
         [0055]    As illustrated in  FIG. 5C , step  106  comprises summing up the intensity differences to the average background for all pixels within the region of interest  44  having the same x′. The maximum of this curve Δ(x′) yields the coordinate x′. 
         [0056]    Step  110  comprises a transformation of the world coordinates x′/y′ into screen coordinates x/y. The input parameters are x′, y′ (y′ is derived in the same way as x′) and several constant parameters gained during calibration of the system, like the position and orientation of the two cameras with respect to one another. Preferably, aberrations within the optical system are corrected by the computer unit within step  110 . 
         [0057]    In step  112 , the screen coordinates x/y then enter into the control of the display, here by a mousepointer simulator. The mousepointer simulator calculates off-screen coordinates and causes a mouse event if the coordinates are within the touch surface or within the screen. Additionally, mouse-up, mouse-down and mouse-click events are simulated. Any standard software can be handled with the inventive system without needing to adapt it. 
         [0058]    During installation of the system, the computer unit—a specific driver of the system—has to receive information on the size of the display and on the aperture of the cameras. In a calibration step, a set of spaced apart points appears on the display, e.g. in the right and left upper corner and in the middle of the lower part of the display. The user has to touch each of this points as visible on the touch surface in a predetermined order. From the world coordinates retrieved in this step and the known screen coordinates, the computer unit calculates and saves the position and the distance of the cameras. These parameters are used to convert all further pointer positions x′/y′ into screen coordinates. 
         [0059]      FIG. 6  shows the evaluation of the individual pixels of the images generated in further detail. First, it is checked whether a pixel lies within subset  44 , i.e. has a z-coordinate between z 1  and z 2 . If yes, the value of the pixel is converted in gray scale (It(x)). Then the median Mt(x)−1 for the pixel is calculated. This corresponds to the averaged value of the pixel in a predetermined number of past images, e.g. 14 images. The difference T(x) between the median and the actual value is evaluated. If it lies within a predetermined range (within the color scale), the variance is calculated. Otherwise, this step is skipped and the variance not updated. Then in a further step, the difference of T(x) to a predetermined threshold T (horizontal dashed line in  FIG. 5C ) is calculated. This difference is compared to the variance. This yields the information whether the pixel has been altered (1 if T(x)−T&gt;variance) or not (0 if T(x)−T&lt;=variance). Hereby, statistical variations in the measured pixel intensity are taken into account.