Abstract:
The present invention relates to haptic augmented reality, and more specifically, to a method for providing haptic augmented reality that provides mixed results of a real environment and a virtual environment to a user. 
     The present invention provides an apparatus for providing haptic augmented reality comprising a controller that calculates a replusive force from an object and determinces a driving force in consideration of the calculated repulsive force and preset stiffness; and a driver that drives a haptic device with the determined driving force.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    1. Technical Field 
         [0002]    The present invention relates to haptic augmented reality, and more particularly, to a method for providing haptic augmented reality that provides mixed results of a real environment and a virtual environment to a user. 
         [0003]    2. Related Art 
         [0004]    Imagine that you are holding a pen-shaped tool in your hand and writing something on a table. Would it be possible that you may feel as though you were writing on a smooth piece of paper with a ball-point pen, or on a soft rubber pad with a marker? The tool may also guide your hand to teach the art of East Asian calligraphy, preferably with the feel of using a brush on a piece of traditional East Asian paper. Creating such haptic illusions belongs to the realm of haptic augmented reality. 
         [0005]    Augmented Reality (AR), or more generally, Mixed Reality (MR) environment provides the user with sensations resulted from a mix of real and synthetic stimuli that are generated by computer. As conceptualized by Milgram et al., a mixed environment can be located at a reality-virtuality continuum (Milgram, P., Colquhoun Jr., H.: A Taxonomy of Real and Virtual World Display Integration. In: Tamura, Y. (ed.) Mixed Reality. Merging Real and VirtualWorlds, pp. 1.16. Springer, Berlin (1999)) Whether an environment in this continuum is closer to reality or virtuality is determined by the amount of knowledge that the computer is required to manage for the environment. For example, the display with head of a human in an aircraft/automobile cockpit is an application of augmented reality by coating a real photograph of a face image of a celebrity on a virtual body employed in a computer game is classified to be an example of augmented virtuality. Although the continuum was primarily defined for vision, the same continuum can be applied to touch. Given a mixed reality application including both vision and touch, the degrees of reality (or virtuality) for the two sensory modalities can be identified and mapped to the composite visuo-haptic continuum of reality and virtuality. 
         [0006]    Earlier research on haptic mixed reality can be categorized using the continuum proposed by Milgram, et al. Although the taxonomy in the continuum is not strictly adhered, even in the literature of visual mixed reality, applying it to haptic mixed reality can be instrumental for elucidating associated concepts. Firstly, “haptic reality” corresponds to applications wherein a user directly touches real objects and feels only a touch from the real object. A typical example is the so-called tangible AR where a real prop held in the hand is used as a tangible interface for visually mixed environments. Secondly, “haptic virtuality” is located at the other end of the continuum, and has received the most attention from the research community. In this case, purely virtual haptic objects are added in a vision mixed reality environment, and the touch of the virtual object is rendered based on the conventional haptic rendering methods under the virtual environment. Thus, accurate positional registration between the haptic and visual coordinate frames is a key issue. The third category is “haptic augmented virtuality”. In this environment, a real prop attached to the haptic device to provide the sensation of surface real material and the haptic device performs only a role of a force-feedback device of the virtual object. 
       SUMMARY OF THE INVENTION 
       [0007]    The present invention proposes haptic augmented reality. It is an object of the present invention to provide augmented haptic information by coating virtual tactile information on tactile information generated when a haptic device collides with a real object. 
         [0008]    Therefore, a technical problem to be achieved by at least one embodiment of the present invention is to provide an apparatus for providing haptic augmented reality enabling a user, which is integrated with a haptic device colliding with an object, to bodily sense real stiffness and other stiffness of the object. 
         [0009]    Another technical problem to be achieved by at least one embodiment of the present invention is to provide a method for providing haptic augmented reality enabling a user, which is integrated with a haptic device colliding with an object, to bodily sense real stiffness and other stiffness of the object. 
         [0010]    Another technical problem to be achieved by at least one embodiment of the present invention is to provide a recording medium readable with a computer stored with a computer program enabling a user, which is integrated with a haptic device colliding with an object, to bodily sense real stiffness and other stiffness of the object. 
         [0011]    In order to achieve the above technical problems, the present invention provides an apparatus for providing haptic augmented reality comprising a controller that calculates a replusive force from an object and determines a driving force in consideration of the calculated repulsive force and preset stiffness; and a driver that drives a haptic device with the determined driving force. 
         [0012]    Preferably, the controller measures displacement of the object and determines the driving force in consideration of the measured displacement, the calculated replusive force, and the preset stiffness. 
         [0013]    Preferably, the controller senses the collision between the object and the haptic device and generates control signals while the object is deformed due to the collison with the haptic device and the driver is operated in response to the control signals. 
         [0014]    Further, the controller checks whether the replusive force from the object exceeds a threshold to sense the collision between the haptic device and the object. 
         [0015]    Meanwhile, according to another embodiment of the present invention, there is provided a method for providing haptic augmented reality comprising: calculating a repulsive force from an object; determining a driving force in consideration of the calculated repulsive force and preset stiffness; and driving a haptic device with the determined driving force. 
         [0016]    Preferably, the method for providing haptic augmented reality further comprises measuring displacement of the object and the determining determines the driving force in consideration of the calculated repulsive force, the measured displacement, and the preset stiffness. 
         [0017]    Preferably, the method for providing haptic augmented reality further comprises sensing the collision between the object and the haptic device and the calculating, the determining, and the driving are performed while the collision between the object and the haptic device is sensed. 
         [0018]    Moreover, the sensing checks whether the replusive force from the object exceeds a threshold to sense the collision between the haptic device and the object. 
         [0019]    On the other hand, according to still another embodiment of the present invention, there is provided a recording medium recordable with a computer stored with computer program that enables the computer to run steps of comprising: calculating a repulsive force from an object; determining a driving force in consideration of the calculated repulsive force and preset stiffness; and driving a haptic device with the determined driving force. 
         [0020]    According to the present invention, the user can feel the augmented haptic information by increasing and decreasing the repulsive force to and from the real repulsive force using the haptic device rather than directly feeling the repulsive force generated due to the collision with the real object, when the user feels the repulsive force generated by the interaction with the real object. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0021]      FIG. 1  is a block diagram of an apparatus for providing haptic augmented reality according to at least one embodiment of the present invention; 
           [0022]      FIG. 2  is a diagram showing one example of a haptic device and explaining a method for calculating a driving force according to one embodiment of the present invention; 
           [0023]      FIG. 3  is a flow chart of the method for providing haptic augmented reality according to at least one embodiment of the present invention; 
           [0024]      FIG. 4  is a diagram showing a real implementation example for applying the apparatus and method for providing haptic augmented reality according to the present invention; 
           [0025]      FIG. 5  is a diagram showing experimental results obtained by performing the collision check using the apparatus for providing haptic augmented reality according to  FIG. 4 ; 
           [0026]      FIG. 6  is a diagram showing a relationship between forces of four objects used-in the experiment and their positional displacement; and 
           [0027]      FIG. 7  is a diagram showing augmented stiffness from the experimental results. 
       
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0028]    Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. First of all, it should be noted that in giving reference numerals to elements of each drawing, like reference numerals refer to like elements even though like elements are shown in different drawings. Further, in describing the present invention, well-known functions or constructions will not be described in detail since they may unnecessarily obscure the understanding of the present invention. Also, although the exemplary embodiments of the present invention are described below, the technical aspects of the present invention are not limited thereto and can be changed and variously practiced by those skilled in the art. 
         [0029]    Hereinafter, an apparatus and method for providing haptic augmented reality according to at least one embodiment of the present invention will be described with reference to the accompanying drawings. 
         [0030]      FIG. 1  is a block diagram of an apparatus  100  for providing haptic augmented reality according to at least one embodiment of the present invention. The apparatus for providing haptic augmented reality includes a controller  110  and a driver  120 . At least one of the controller  110  and the driver  120  can be integrated with a haptic device. 
         [0031]    In the specification, the haptic device is a device that includes a plurality of articulations connected to each other in a link structure and if one articulation moves, then other articulations connected to the one articulation can move. The haptic device means a device when at least a part of the plurality of articulations (for example, an articulation positioned at a tip among the plurality of articulations) is integrated with at least a part (for example, fingers) of a user&#39;s body so that it can be operated in the user&#39;s intention. For example, the user holds the articulation positioned at the tip thereof, making it possible to move the articulation positioned at the tip thereof in the user&#39;s intention. 
         [0032]    The user moves the haptic device such that the haptic device can collide with the object. At this time, a part colliding with an object among the parts of the haptic device is a tip end of any one articulation of the plurality of articulations that configures the haptic device. In the specification, the colliding end means the tip end. Meanwhile, the “object” may be a virtual object in the specification. However, for convenience of explanation, it is assumed that the object is a real object. 
         [0033]    The controller  110  senses whether the ‘colliding end of the haptic device’ collides with the ‘object’. To this end, the controller  110  checks whether the repulsive force from the object exceeds a ‘preset threshold’ to sense the collision between the colliding end of the haptic device and the object. In other words, if the repulsive force from the object exceeds the threshold, the controller  110  recognizes that the collision between the colliding end and the object is generated. 
         [0034]    The controller  110  senses the collision between the object and the colliding end of the haptic device, generates the control signals while the object is deformed due to the collision, and calculates the repulsive force from the object while the collision between the object and the colliding end is sensed. 
         [0035]    When the controller  110  calculates the repulsive force, it calculates the repulsive force from the object by using the inertia force of the colliding end and force sensed by the sensor provided between the colliding end of the haptic device and a last link. 
         [0036]    If the repulsive force from the object generated by the collision between the colliding end of the haptic device and the object is calculated, the controller  110  controls the driver  120  by determining the driving force that drives the driver  120 . Thereby, the driver  120  can transfer the repulsive force augmented by the controller  110 , not the repulsive force due to the collision with the real object, to the user. 
         [0037]      FIG. 2  is a diagram showing one example of the haptic device and explaining the method for calculating a driving force according to one embodiment of the present invention. 
         [0038]    The apparatus  100  for providing haptic augmented reality according to  FIG. 1  may be integrated with a haptic device  200  shown in  FIG. 2  as described above. 
         [0039]    As shown in  FIG. 2 , the haptic device  200  includes a plurality of links  212 ,  214 , and  216 . The first link  212  and the second link  214  are connected with a first articulation  213  and the second link  214  and the third link  216  are connected with a second articulation  215 , in the state where the first link  212  is rotatably connected with a base part  210 . As shown in  FIG. 1 , the driver  120  may be included in a base part of the first link  212  connected with the base part  210  and the first articulation  213  and the second articulation  215 . The end of the third link  216  is provided with a force sensor  217  and a colliding end  218  is provided at a lower part of the force sensor  217 . 
         [0040]    A user  220  holds the third link  216  and moves it so that the colliding end  218  of the haptic device  210  collides with an object  230 . This situation is represented by the following equations 1 and 2. 
         [0000]        f   r ( t )= f   s ( t )− f   i ( t )   [Equation 1] 
         [0000]        f   i ( t )= m   t    a ( t )   [Equation 2] 
         [0041]    In the above Equations 1 and 2, t means time, f r (t) means the repulsive force due to the collision with the object  230 , f s (t) means the force sensed by the force sensor  217 , m t  means a mass of the colliding end  218 , a(t) means acceleration in a direction that presses the (object) of the colliding end  218 , and f i (t) means the inertia force of the colliding end. 
         [0042]    The controller  110  can determine that the collision is generated if f r (t) is larger than a decision threshold for determining the collision. 
         [0043]    However, in real implementation, it may be difficult to obtain acceleration a(t) of the colliding end  218 . In this case, in order to obtain a(t), it is possible to use a specialized acceleration sensor. In the case of using the acceleration sensor, there is a problem in that it causes the system to be complicated and increases cost. As an easy method, there may be a method to obtain the acceleration from positional information of the colliding end  218  that is provided by the haptic device. However, differential for a discontinuous positional value provides results including noise, such that it is preferable to use a low frequency passing filter for each differential to obtain the acceleration. 
         [0044]    The controller  110  controls the driving force of the haptic device  200  with respect to the repulsive force f r (t) obtained by Equation 1, thereby changing the force that is realistically felt by the user. In other words, the user can feel force augmented by the apparatus  100  for providing haptic augmented reality rather than the real force caused by repulsion. 
         [0045]    The relationship between the force and positional displacement is generally called stiffness. In a particular time t, assume that the stiffness of the real object is k(t). If the apparatus  100  for providing haptic augmented reality does not perform the control, this value may be the stiffness felt by the user. Our goal is to change k(t) into our own desired value k*(t) by appropriately using the haptic device  200 . Assume that the force generated from the haptic device  200  is f d (t) and the force generated from the user&#39;s hand is f h (t). The surface of the real object is deformed due to the above two forces, thereby causing the positional displacement by X r (t). At this time, the force received from the real object is a sum of the two forces. This is represented by the following Equation 3. 
         [0000]        f   r ( t )= k ( t ) x   r ( t )= f   h ( t )+ f   d ( t )   [Equation 3] 
         [0046]    In Equation 3, the force f h (t) felt by the user may be represented by the following Equation 4 and the force generated from the haptic device  200  may be represented by the following Equation 5. 
         [0000]        f   h ( t )= k *( t ) x   r ( t )   [Equation 4] 
         [0000]        f   d *( t )= f   r ( t )− k *( t ) x   r ( t )   [Equation 5] 
         [0047]    The controller  110  performs the control so that f d *(t) is generated by the driver  120  according to Equation 5. Thereby, the user feels the force that subtracts the driving force f d *(t) generated from the haptic device  200  from the repulsive force f r *(t) generated by the collision between the colliding end  218  and the object, such that the user can feel the force augmented by the haptic device  200 . 
         [0048]    When the driver  120  drives the haptic device  200  with the driving force f d *(t), it generates a force command f c *(t) for driving the haptic device  200  and the haptic device  200  generates the driving force f d *(t) according to the generated force command. 
         [0049]    Herein, f c (t) may be identical with f d (t). In this case, the noise generated while the force sensor  217  provided in the colliding end  218  senses force directly influences the force command, such that the driver  120  cannot stably provide the driving force. 
         [0050]    In order to solve this risk, a typical closed-loop stiffness method using proportional and differential control can be applied to f c (t). The closed-loop stiffness control may be represented by the following Equation 6. 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       f 
                       c 
                     
                      
                     
                       ( 
                       t 
                       ) 
                     
                   
                   = 
                   
                     
                       
                         f 
                         c 
                       
                        
                       
                         ( 
                         
                           t 
                           - 
                           1 
                         
                         ) 
                       
                     
                     + 
                     
                       
                         K 
                         p 
                       
                        
                       
                         
                           f 
                           e 
                         
                          
                         
                           ( 
                           t 
                           ) 
                         
                       
                     
                     + 
                     
                       
                         K 
                         d 
                       
                        
                       
                         
                            
                           
                             
                               f 
                               e 
                             
                              
                             
                               ( 
                               t 
                               ) 
                             
                           
                         
                         
                            
                           t 
                         
                       
                     
                   
                 
               
               
                 
                   [ 
                   
                     Equation 
                      
                     
                         
                     
                      
                     6 
                   
                   ] 
                 
               
             
           
         
       
     
         [0051]    where f c (t)=f d *(t)−f d (t). This means a force error and K p  and K d  each mean a gain of proportional derivative. However, the problem of Equation 6 is that it is difficult to accurately measure f d (t). Of course, f d (t) can be obtained if the force sensor is further attached between the third link  216  and the second link  214  with reference to  FIG. 2 . However, this increases the complication of the system. 
         [0052]    Therefore, the present invention proposes f d (t)=f c (t−1). This is based on the assumption that the force command before one sampling time is equal to the force generated from the current haptic device  200  through a heuristic method. In other words, a driving force determined at a current point in time by the driver  120  may be a driving force according to the force command generated at a previous point in time by the driver  120 . When the user stops the motion of his/her own hand in order to feel stiffness, this effect is assumed but these two forces are generally not consistently the same. If the user applies force to an object to move a tool in the object in order to feel stiffness or if the user feels stiffness and then removes force to move the tool out of the object, these two forces are slightly different from each other. However, since a person is limited in a speed range when moving objects, only a phase delay for the order of several milliseconds occurs. For example, when the user provides a sine wave having a period of 2.5 Hz and a stiffness of 2N as the force command, a value of f d (t) and f c (t−1) generates only a delay of 4 ms in the case of a phantom haptic device. The force error generated due to this delay is much smaller than force recognition just noticeable difference (JND) of a person. 
         [0053]    As a result, the apparatus for providing haptic augmented reality according to at least one embodiment of the present invention allows the user integrated with the haptic device colliding with the object to bodily sense the stiffness different from the real stiffness of the object. For example, according to at least one embodiment of the present invention, even if the colliding end of the haptic device collides with a sponge, the user (moving one articulation of the haptic device to collide with a sponge) may have a feeling that the colliding end of the haptic device collides with a hard object like concrete, not a feeling that it collides with sponge. 
         [0054]      FIG. 3  is a flow chart showing the method for providing haptic augmented reality according to at least one embodiment of the present invention. This will be described below with reference to  FIG. 1 . 
         [0055]    First, the apparatus  100  for providing haptic augmented reality checks whether the object and the colliding end collide with each other (step  310 ). 
         [0056]    At step  310 , if it is checked that they collide with each other, the apparatus  100  for providing haptic augmented reality calculates the repulsive fore from the object (step  320 ). 
         [0057]    After step  320 , the apparatus  100  for providing haptic augmented reality determines the driving force in consideration of the repulsive force calculated in step  320 , the displacement of the object due to collision, and the preset stiffness (step  330 ). 
         [0058]    After step  330 , the apparatus  100  for providing haptic augmented reality drives the haptic device by using the driving force determined at step  330  (step  340 ). 
         [0059]    As described above, a program that enables a computer to run the method for providing haptic augmented reality according to at least one embodiment of the present invention can be stored in a recording medium readable with the computer. Herein, the recording medium readable with the computer includes a storage medium, such as a magnetic storage medium (for example, ROM, floppy disc, hard disc, etc.) and an optical reading medium (for example, CD-ROM, digital versatile disc (DVD)). 
         [0060]    Meanwhile, the inventors configured a system shown in  FIG. 4  and performed an experiment by directly applying the present invention to four real objects, in order to verify the performance according to the present invention. 
         [0061]      FIG. 4  is a diagram showing a real implementation example for applying the apparatus and method for providing haptic augmented reality according to the present invention. 
         [0062]    In obtaining f r (t) according to the above Equations 1 and 2 using the apparatus for providing haptic augmented reality, the positional information provided by the haptic device is differentiated twice to obtain acceleration a(t).  FIG. 5  shows the results of calculating the force based on the above obtained acceleration. In order to measure the delay degree generated by using the low frequency pass filter, the difference does not exceed 0.015N when f r (t) is compared with f i (t) in the situation where there is no collision. This value does not exceed the noise level of the force sensor. Further, it can be confirmed from this value that the error degree due to the delay according to the use of the low frequency passing filter is not higher than the noise level due to the force sensor. 
         [0063]    The four real objects tested by using the apparatus for providing haptic augmented reality according to  FIG. 4  are a sponge block, a foam ball, a rubber ball, and a rubber eraser. The relationship between the force and the positional displacement of each real object is shown in  FIG. 6 .  FIG. 6  shows the results of a well controlled tapping experiment by using the phantom haptic device and the force sensor. The force generated from the haptic device is changed from 0 N to 4 N at a speed of 0.5 N/s at the time of performing the tapping. As a result, it is shown that the rubber eraser has the most linear result and the sponge block has the most non-linear result and a large hysteresis. 
         [0064]    Thereafter, the tapping is then progressed and the positional displacement of the tool and the repulsive force received from the real object are measured. As described above, the force generated from the haptic device is replaced with the force command value transferred to the haptic device. The really operating stiffness is confirmed by calculating the force applied to the real user using these values and is compared with our desired stiffness. As shown in  FIG. 7 , the stiffness of the real objects is effectively deformed together. In  FIG. 7 , the sponge block is shown at the upper left, the foam ball is shown at the upper right, the rubber ball is shown at the lower left, and the rubber eraser is shown at the lower right. 
         [0065]    Although the preferred embodiment of the present invention is described, it will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the inventions. Thus, it is intended that the present invention covers the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.