Abstract:
The present invention provides a multi-touch position tracking technique and an interactive system and a multi-touch interactive image processing method using the same. In the present invention, a light guide element is designed to comprise frustrating structures to frustrate total internal reflection (TIR) so that the light beam therein can be dispersed to form a dispersed optical field distribution. The dispersed optical field is used to respond a physical relation between an object and the light guide element.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    1. Field of the Invention 
         [0002]    The present invention generally relates to a multi-touch position tracking technique and, more particularly, to a multi-touch position tracking apparatus, an interactive system and an image processing method. 
         [0003]    2. Description of the Prior Art 
         [0004]    In a multi-touch system, the user is able to interact with the multi-media interactive system using multiple objects (such as fingers) to touch the interface. Conventionally, the single-touch scheme is used in the touch system so that the touch system is restrictedly used. However, since the consumer digital products have been developed towards compactness and the interactions between the user and the products have changed, the multi-touch approach has attracted tremendous attention to replace conventional single-touch technique. 
         [0005]    In  FIG. 1 , which is a cross-sectional view of a conventional multi-touch display device disclosed in U.S. Pat. Appl. No. 20080029691, the multi-touch display device comprises a light guide plate  10  with a light source  11  on one side to receive an incoming light beam from the light source  11  into the light guide plate  10 . Since the refractive index of the air outside the light guide plate  10  is smaller than that of the light guide plate  10 . With a pre-designed incoming angle, the light beam entering the light guide plate  10  is confined inside the light guide plate  10  due to total internal reflection (TIR). However, as the user uses an object (such as the skin on a finger) with a higher refractive index to touch the surface of the light guide plate  10 , total internal reflection is frustrated at the point where the object touches the light guide plate  10  so that a dispersed optical field  13  is formed due to light leaking into the air. The dispersed optical field  13  is then received by a sensor module  14  to be further processed. 
         [0006]    Moreover, U.S. Pat. No. 3,200,701 discloses a technique to image fingerprint ridges by frustrated total internal reflection. In U.S. Pat. No. 3,200,701, the light from a light source is introduced into a light guide element (such as glass) with a refractive index higher than the air so that total internal reflection takes place in the light guide element. When the skin of a finger touches the light guide element, total internal reflection will be frustrated because the refractive index of the skin is higher than that of the light guide element. A sensed image with patterns formed by the dispersed light beam dispersed by the skin is then sensed by the sensor module to identify the fingerprint on the skin of the finger. 
         [0007]    Furthermore, U.S. Pat. No. 6,061,177 discloses a touch-sensing apparatus incorporating a touch screen panel adapted for use with a rear-projected computer display using total internal reflection. In U.S. Pat. No. 6,061,177, a sensor module is disposed on one side of the touch screen panel. A polarizer is disposed between the sensor module and the touch screen panel to filter out the non-TIR light so that the sensor module will not receive dispersed light due to frustrated total internal reflection by the skin of the finger (or other material with a higher refractive index than the touch screen panel). Accordingly, a dark zone is formed on a position where the skin of the finger the touch screen panel to be used as a basis for interactive touch-sensing. 
       SUMMARY OF THE INVENTION 
       [0008]    The invention provides a multi-touch position tracking apparatus, using a light guide element designed to comprise frustrating structures to frustrate total internal reflection (TIR) so that the light beam therein can be dispersed to form a dispersed optical field distribution over the light guide element. The dispersed optical field is used to respond a physical relation between a contact/non-contact object and the light guide element. 
         [0009]    The invention provides a multi-touch interactive system, using a light guide element designed to comprise frustrating structures to frustrate total internal reflection (TIR) so that the light beam therein can be dispersed to form a dispersed optical field distribution over the light guide element. The dispersed optical field is used to respond a physical relation between a contact/non-contact object and the light guide element. An interactive program is controlled according to the physical relation to interact with the user. 
         [0010]    The invention provides a multi-touch interactive image processing method for processing a sensed image detected from the dispersed optical field, and determining the physical relation between the object and the light guide element. 
         [0011]    The present invention provides a multi-touch position tracking apparatus, comprising: a light source; a light guide element capable of receiving an incoming optical field from the light source and enabling the incoming optical field to go out from a side surface of the light guide element to form a dispersed optical field; a sensor module capable of sensing the light from the dispersed optical field being dispersed or reflected to acquire a sensed image; and a processing unit capable of determining a physical relation between at least an object and the light guide element corresponding to the sensed image. 
         [0012]    The present invention provides a multi-touch interactive system, comprising: a light source; a light guide element capable of receiving an incoming optical field from the light source and enabling the incoming optical field to go out from a side surface of the light guide element to form a dispersed optical field; a sensor module capable of sensing the light from the dispersed optical field being dispersed or reflected to acquire a sensed image; a processing unit capable of determining a physical relation between at least an object and the light guide element corresponding to the sensed image and generating a control signal corresponding to the physical relation or the variation of the physical relation; and a display device capable of generating an interactive image according to the control signal. 
         [0013]    The present invention provides a multi-touch interactive image processing method, comprising steps of: (a) providing a light guide element and a sensor module, the light guide element capable of receiving an incoming optical field and enabling the incoming optical field to go out therefrom to form a dispersed optical field being incident on at least an object so that a dispersed/reflected light beam from the object is received by the sensor module to form a sensed image; (b) filtering the sensed image according to at least a threshold value to form at least a filtered image; (c) analyzing the filtered image to acquire at least a group of characteristic values corresponding to the filtered image and the object; (d) determining a physical relation between the object and the light guide element according to the characteristic values; and (e) tracking the variation of the physical relation. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0014]    The objects, spirits and advantages of the preferred embodiments of the present invention will be readily understood by the accompanying drawings and detailed descriptions, wherein: 
           [0015]      FIG. 1  is a cross-sectional view of a conventional multi-touch display device; 
           [0016]      FIG. 2A  is a schematic diagram of a multi-touch position tracking apparatus according to a first embodiment of the present invention; 
           [0017]      FIG. 2B  is a schematic diagram of a multi-touch position tracking apparatus according to another embodiment of the present invention; 
           [0018]      FIG. 3A  and  FIG. 3B  are cross-sectional views showing the operation of a multi-touch position tracking apparatus according to a first embodiment of the present invention; 
           [0019]      FIG. 4  is a flowchart of a multi-touch interactive image processing method according to a first embodiment of the present invention; 
           [0020]      FIG. 5  is a flowchart of a multi-touch interactive image processing method according to a second embodiment of the present invention; 
           [0021]      FIG. 6  is a schematic diagram of a multi-touch position tracking apparatus according to a second embodiment of the present invention; 
           [0022]      FIG. 7A  and  FIG. 7B  are cross-sectional views showing the operation of a multi-touch position tracking apparatus according to a second embodiment of the present invention; 
           [0023]      FIG. 8A  is a schematic diagram of a multi-touch interactive system according to a first embodiment of the present invention; 
           [0024]      FIG. 8B  is a schematic diagram of a multi-touch interactive system according to a second embodiment of the present invention; 
           [0025]      FIG. 9  is a flowchart of a multi-touch interactive image processing method according to a third embodiment of the present invention; and 
           [0026]      FIG. 10  is a schematic diagram of a multi-touch interactive system according to a third embodiment of the present invention. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
       [0027]    The present invention can be exemplified but not limited by the embodiment as described hereinafter. 
         [0028]    Please refer to  FIG. 2A , which is a schematic diagram of a multi-touch position tracking apparatus according to a first embodiment of the present invention. The multi-touch position tracking apparatus  2  comprises at least a light source  20 , a light guide element  21 , a sensor module  22  and a processing unit  23 . The light source  20  can be an infrared light source, but is not restricted thereto. For example, the light source  20  can also be an ultra-violet light source. Generally, the light source  20  is implemented using a light emitting diode (LED), a laser or other non-visible light source. In the present embodiment, the light source  20  is an infrared light emitting diode (LED). The light guide element  21  is capable of receiving an incoming optical field from the light source  20 . The light guide element  21  comprises a dispersing structure  210  on a surface to frustrate total internal reflection (TIR) so that the incoming optical field is dispersed to form a dispersed optical field with a distribution having a specific height. The specific height is not restricted and is dependent on the intensity of the light source  20 . 
         [0029]    The sensor module  22  is capable of sensing the light from the dispersed optical field being dispersed or reflected to acquire a sensed image. The sensor module  22  further comprises an image sensor  220  and a lens set  221 . In the present embodiment, the image sensor  220  is an infrared CCD image sensor. 
         [0030]    The lens set  221  is disposed between the image sensor  220  and the light guide element  21  to form the sensed image on the image sensor. In order to prevent the image acquired by the image sensor  220  from being interfered by other light source, an optical filter  222  is further disposed between the lens set  221  and the image sensor  220 . In the present embodiment, the optical filter  222  is an infrared band-pass optical filter to filter out non-infrared light (such as background visible light) to improve sensing efficiency of the image sensor  220 . The number of image sensors  220  is determined according to practical use and is thus not restricted as shown in  FIG. 2A . 
         [0031]    As shown in  FIG. 2B , which is a schematic diagram of a multi-touch position tracking apparatus according to another embodiment of the present invention. In the present embodiment, the optical filter  222  is disposed between the light guide element  21  and the lens set  221 . 
         [0032]    Please refer to  FIG. 3A  and  FIG. 3B  for are cross-sectional views showing the operation of a multi-touch position tracking apparatus according to a first embodiment of the present invention. In  FIG. 3A , since the dispersed optical field  90  is formed with a specific height from the surface of the light guide element  21 , the light from the dispersed optical field  90  is dispersed or reflected by the surfaces of the objects  80  and  81  to issue a sensing optical field  91  when the objects  80  and  81  (such as fingers or other pointing devices) are coming closer. The sensing optical field  91  passes through the light guide element  21  and is received by the sensor module  22  so as to be processed to form a sensed image. Moreover, as shown in  FIG. 3B , the objects  82  and  83  contact the surface of the light guide element  21 . Similarly, the light from the dispersed optical field is dispersed by the objects  82  and  83  contacting the surface of the light guide element  21  to form a sensing optical field  92 . The sensing optical field  92  is received by the sensor module  22  to be processed to form a sensed image. 
         [0033]    Returning to  FIG. 2A , the processing unit  23  is coupled to the sensor module  22  to receive the sensed image, determines a physical relation between at least an object and the light guide element  21  corresponding to the sensed image according to the sensed image and tracks the variation of the physical relation. The physical relation represents the three-dimensional position of the non-contact objects  80  and  81  as shown in  FIG. 3A  or the two-dimensional position of the objects  82  and  83  contacting the light guide element  21  as well as the pressure applied to the light guide element  21  as shown in  FIG. 3B . 
         [0034]    The process for the processing unit  23  to process the sensed image to analyze the physical relation between the object and the light guide element is described hereinafter. Please refer to  FIG. 2A  and  FIG. 4 , wherein  FIG. 4  is a flowchart of a multi-touch interactive image processing method according to a first embodiment of the present invention. In the present embodiment, the method  3  comprises steps as follows. First in Step  30 , the processing unit  23  receives a sensed image transmitted from the image sensor  20 . Then, Step  31  is performed to filter the sensed image according to a threshold value to form at least a filtered image. The threshold value is a luminance threshold value. The object of the present step is to determine at least a luminance threshold value and to compare the luminance value of each pixel in the sensed image to the threshold value. The luminance value is kept if it is larger than the threshold value. Therefore, the filtered image with a luminance value larger than or equal to the threshold value is acquired after the luminance value is compared to the threshold value. 
         [0035]    Step  32  is then performed to analyze the filtered image to acquire at least a group of characteristic values corresponding to each filtered image. The characteristic values represent the luminance in an image pixel. The undesired noise has been filtered out in Step  31 . However, it is likely that a plurality of objects (such as a plurality of fingers of a hand or two hands) touch the light guide element  21  at the same time in a contact/non-contact fashion to determine the position or the pressure. Different objects result in different luminance values. Therefore, the luminance values larger than the threshold value have to be classified to identify the positions of the objects or the contact pressure applied to the light guide element. According to Step  32 , the number of classified group of characteristic values is capable of determining the number of objects touching the light guide element  21 . 
         [0036]    Then, Step  33  is performed to determine a physical relation between each object and the light guide element  21  according to the group of characteristic values. Since the luminance ranges corresponding to each group of characteristic values and the positions sensed by the image sensor  220  are not the same, therefore the object of the present step is to obtain the physical relation between the object corresponding to the group of characteristic values and the light guide element  21  according to the luminance range and the position information sensed by the image sensor  220 . The physical relation comprises the position between the object and the light guide element and the contact pressure applied to the light guide element  21 . 
         [0037]    After Step  33 , Step  34  is performed to determine if there is any signal missing from the group of characteristic values. The object of the present step is to determine whether there is any signal missing from the group of characteristic values due to the variation of the pressure on the light guide element  21  resulting from the sliding of the object contacting the light guide element  21 . Therefore, Step  35  is performed if there is any signal missing to update the threshold value. Step  31  is re-performed to form an updated filtered image according to the updated threshold value. After returning to Step  34 , Step  36  is performed to determine the variation between the present physical relation and the previous physical relation if there is no signal missing. By repeating from Step  30  to Step  36 , it is possible to keep tracking the position of each (contact/non-contact) object on the light guide element  21  or the pressure and variation thereof. 
         [0038]    Please refer to  FIG. 5 , which is a flowchart of a multi-touch interactive image processing method according to a second embodiment of the present invention. In the present embodiment, the operation of the processing unit  23  when there are both contact and non-contact objects is described hereinafter. In the present embodiment, the method  4  comprises steps as follows. First in Step  40 , the processing unit  23  receives a sensed image transmitted from the image sensor  20 . Then, Step  41  is performed to filter the sensed image according to a first threshold value to form at least a first filtered image. Step  42  is then performed to filter the first filtered image according to a second threshold value to form at least a second filtered image. In Step  41  and Step  42 , the first threshold value and the second threshold value represent luminance threshold values and the first threshold value is smaller than the second threshold value. The first and the second threshold values are different to distinguish the images formed due to the contact object and the non-contact object, respectively. Since the image due to the contact object is formed directly on the light guide element, the luminance of the light dispersed by the contact object is higher than that by the non-contact object. In Step  41  and Step  42 , the first filtered image corresponding to the non-contact object and the second filtered image corresponding to the contact object can be acquired according to the difference between the first threshold value and the second threshold value. 
         [0039]    Step  43  is then performed to analyze the first filtered image and the second filtered image. In Step  43 , the first filtered image corresponding to the non-contact object and the second filtered image corresponding to the contact object are distinguished so that the first filtered image and the second filtered image are then analyzed in Step  44  and Step  45 , respectively. 
         [0040]    In Step  44 , Step  440  is first performed to analyze the first filtered image to acquire at least a group of first characteristic values corresponding to the first filtered image and the geometric position of the group of first characteristic values. Each group of first characteristic values corresponds to an object, while the characteristic values correspond to the luminance of an image pixel. For example, in  FIG. 3A , two groups of characteristic values correspond to two non-contact objects  80  and  81 . In Step  440 , since the heights between different objects and the light guide element are different, the luminance values of the corresponding dispersed optical fields are different. Therefore, in order to distinguish three-dimensional positions of the plurality of non-contact objects, the luminance values larger than the threshold value have to be classified. Then, Step  441  is performed to determine a 3-D position between a non-contact object and a light guide element according to the group of first characteristic values. Since the distances between different objects and the light guide element are different, the luminance values corresponding to the groups of characteristic values are not the same. Therefore, in the present step, the positions between the light guide element and the objects corresponding to each group of characteristic values can be determined according to the luminance information. On the other hand, the positions of the group of characteristic values corresponding to the first filtered image represent the positions sensed by the image sensor, which can be interpreted as positions corresponding to the light guide element. Therefore, two-dimensional positions of the objects on the light guide element can be acquired according to geometric positions corresponding to each group of characteristic values. Furthermore, three-dimensional positions of the objects relative to the light guide element can be determined according to the two-dimensional positions and the heights. Then, Step  442  is performed to analyze the variation of the 3-D positions corresponding to each group of first characteristic values according to the next detection and analysis. 
         [0041]    Moreover, the second filtered image is analyzed in Step  45 . In Step  45 , Step  450  is first performed to analyze the second filtered image to acquire at least a group of second characteristic values corresponding to the second filtered image. Each group of second characteristic values corresponds to an object, while the characteristic values correspond to the luminance of an image pixel. In order to distinguish the two-dimensional positions and contact pressures of the plurality of objects, the luminance values larger than the threshold value have to be classified. In Step  450 , even though all the objects contact the light guide element, the contact pressures for each object on the light guide element are not necessarily identical. For example, in  FIG. 3B , two contact objects  82  and  83  contact the light guide element and the contact pressure of the object  83  on the light guide element is larger than the contact pressure of the object  82  on the light guide element. Therefore, the luminance values of the dispersed optical fields corresponding to the objects  82  and  83  are different. Accordingly, the groups of characteristic values corresponding to the contact objects  82  and  83  can be respectively acquired. 
         [0042]    Returning to  FIG. 5 , after Step  450 , Step  451  is performed to determine a 2-D position and a contact pressure between a contact object and a light guide element according to the group of second characteristic values. Since the contact pressures of different objects on the light guide element are different, the luminance values corresponding to the groups of characteristic values are not the same. Therefore, in the present step, the contact pressures of different objects on the light guide element corresponding to each group of characteristic values can be determined according to the luminance information. On the other hand, the positions of the group of characteristic values corresponding to the second filtered image represent the positions sensed by the image sensor, which can be interpreted as positions corresponding to the light guide element. Therefore, two-dimensional positions of the objects contacting the light guide element can be acquired according to geometric positions corresponding to each group of characteristic values. 
         [0043]    After Step  451 , Step  452  is performed to determine if there is any signal missing from the group of characteristic values. The object of the present step is to determine whether there is any signal missing from the group of characteristic values due to the variation of the pressure on the light guide element  21  resulting from the sliding of the object contacting the light guide element  21 . Therefore, Step  453  is performed if there is any signal missing to update the second threshold value. Step  42  is re-performed to form an updated second filtered image according to the updated second threshold value. After returning to Step  452 , Step  454  is performed to determine the variations between the present two-dimensional position and pressure and the previous two-dimensional position and pressure to acquire the variations of the 2-D positions and pressures of the objects on the light guide element if there is no signal missing. By repeating from Step  40  to Step  45 , it is possible to keep tracking each (contact/non-contact) object on the light guide element  21 . 
         [0044]    Please refer to  FIG. 6 , which is a schematic diagram of a multi-touch position tracking apparatus according to a second embodiment of the present invention. In the present embodiment, the light guide element  21  comprises a light guide plate  211  and a light guide sheet  212 . The light guide plate  211  is capable of receiving an incoming optical field. The light guide sheet  212  is connected one side surface of the light guide plate  211 . The refractive index of the light guide sheet  212  is larger than that of the light guide plate  211 . The light guide sheet  212  comprises a dispersing structure  213  on the surface to enable the incoming optical field to go out to form a dispersed optical field. 
         [0045]    Please refer to  FIG. 7A  and  FIG. 7B  for are cross-sectional views showing the operation of a multi-touch position tracking apparatus according to a second embodiment of the present invention. In  FIG. 7A , since the dispersed optical field  93  is formed with a specific height from the surface of the light guide sheet  212 , the light from the dispersed optical field  93  is dispersed or reflected by the surfaces of the objects  80  and  81  to issue a sensing optical field  94  when the objects  80  and  81  (such as fingers or other pointing devices) are coming closer. The sensing optical field  94  passes through the light guide sheet  212  and the light guide plate  211  and is received by the sensor module  22  so as to be processed to form a sensed image. Moreover, as shown in  FIG. 7B , the objects  82  and  83  contact the surface of the light guide element  21 . Similarly, the light from the dispersed optical field is dispersed by the objects  82  and  83  contacting the surface of the light guide sheet  212  to form a sensing optical field  94 . The sensing optical field  94  is received by the sensor module  22  to be processed to form a sensed image. 
         [0046]    Please refer to  FIG. 8A , which is a schematic diagram of a multi-touch interactive system according to a first embodiment of the present invention. In the present embodiment, the multi-touch interactive system  5  uses the multi-touch position tracking apparatus  2  in  FIG. 2A  and a display device  6 . The light source  20 , the light guide element  21  and the sensor module  22  are similar to those as described and thus descriptions thereof are not repeated. The processing unit  23  is capable of determining a physical relation between at least an object corresponding to the sensed image and the light guide element  21  according to the sensed image and is capable of tracking the variation of the physical relation to issue a control signal corresponding to the physical relation or the variation of the physical. The display device  6  is disposed between the sensor module  22  and the light guide element  21 . The display device  6  is capable of generating an interactive image according to the control signal. In the present embodiment, the display device  6  is coupled to the light guide element  21  so that the user is able to watch and interact with the image displayed on the display device  6  through the light guide element  21 . Moreover, the display device  6  is a distance away from the light guide element  21 . The distance is not restricted as long as the user is able to watch the image displayed on the display device  6 . Generally, the display device  6  can be a rear-projection display device or a liquid-crystal display device. 
         [0047]    Please refer to  FIG. 8B , which is a schematic diagram of a multi-touch interactive system according to a second embodiment of the present invention. In the present embodiment, the multi-touch position tracking apparatus  2  in  FIG. 6  is combined with the display device  6 . In other words, the light guide element  21  comprises a light guide plate  211  and a light guide sheet  212 . The other elements in  FIG. 8B  are similar to those as described in  FIG. 8A , and thus descriptions thereof are not repeated. 
         [0048]    Please refer to  FIG. 9 , which is a flowchart of a multi-touch interactive image processing method according to a third embodiment of the present invention. In the present invention, the image processing method is similar to the method in  FIG. 5  to identify the contact/non-contact objects except that the method in  FIG. 9  further comprises Step  46  to issue a control signal to an application program according to the variations of the physical relations. The application program can be a game or application software in the display device. Alternatively, as shown in  FIG. 10 , the application program can also be executed in a game device  7  coupled to the display device  6 . Returning to  FIG. 9 , Step  47  is performed, in which the application program is capable of interacting with the object according to the control signal. 
         [0049]    According to the above discussion, it is apparent that the present invention discloses a multi-touch position tracking apparatus, an interactive system and an image processing method using frustrated total internal reflection (FTIR) to detect information of an object. Therefore, the present invention is novel, useful and non-obvious. 
         [0050]    Although this invention has been disclosed and illustrated with reference to particular embodiments, the principles involved are susceptible for use in numerous other embodiments that will be apparent to persons skilled in the art. This invention is, therefore, to be limited only as indicated by the scope of the appended claims.