Patent Publication Number: US-2016239087-A1

Title: Content-aware haptic system and associated control method

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit of U.S. Provisional Application No. 62/116,710 filed on Feb. 16, 2015, the entirety of which is incorporated by reference herein. 
    
    
     BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The invention relates to haptic systems, and, in particular, to a content-aware haptic system and an associated control method. 
     2. Description of the Related Art 
     In recent years, haptic technologies have emerged in academic reports and industrial products. However, current haptic technologies mostly consider how to address the haptic experience of the end user, but haptic experiences are not easy to record or to duplicate in different haptic systems. In addition, the patterns of haptic feedback are usually edited or predetermined in a specific manner, such as using a predefined mapping table to map specific symbols into corresponding haptic patterns. Conventional haptic technologies do not control the haptic feedback according to the content of an input signal such as a video signal, image signal, audio signal, etc. 
     Accordingly, there is demand for a haptic system and an associated control method to solve the aforementioned problems. 
     BRIEF SUMMARY OF THE INVENTION 
     A detailed description is given in the following embodiments with reference to the accompanying drawings. 
     In an exemplary embodiment, a haptic system is provided. The haptic system includes: a processor, and control circuitry. The processor is configured to receive an input signal and perform haptic conversion on the input signal according to content of the input signal to generate a first control signal. The control circuitry is configured to generate a driving signal according to the first control signal so as to provide haptic feedback through a haptic device. 
     In another exemplary embodiment, a control method for a haptic system is provided. The method comprises the steps of: receiving an input signal; performing haptic conversion on the input signal according to the content of the input signal to generate a first control signal; and generating a driving signal according to the first control signal so as to provide haptic feedback through a haptic device. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention can be more fully understood by reading the subsequent detailed description and examples with references made to the accompanying drawings, wherein: 
         FIG. 1  is a block diagram of a haptic system in accordance with an embodiment of the invention; 
         FIG. 2  is a flow chart of a control method for a haptic system in accordance with an embodiment of the invention; 
         FIGS. 3A and 3B  are diagrams illustrating enhancing haptic feedback on the touch position in accordance with an embodiment of the invention; 
         FIG. 4  is a flow chart of a control method for a haptic system in accordance with another embodiment of the invention; and 
         FIG. 5  is a flow chart of a control method for a haptic system in accordance with yet another embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     The following description is of the best-contemplated mode of carrying out the invention. This description is made for the purpose of illustrating the general principles of the invention and should not be taken in a limiting sense. The scope of the invention is best determined by reference to the appended claims. 
     The present disclosure is directed to systems and methods for providing content-aware haptic controls. Haptic systems may be used for actuation such as vibration, shape deformation (e.g., contouring a flat surface, or deformation), friction change, volumetric haptic shape, transcutaneous nerve stimulation, temperature stimulation, or other suitable actuations or combination of actuations which could provide tactile feedback to a user. Haptic systems may also be used for sensing stimuli such as, for example, contact on a display screen, patterns of contact on a screen, shape changes, physical changes of a system or component, or other suitable stimuli or combinations of stimuli which may be received. Haptic systems may sense particular stimuli, change one or more characteristics of a shape change element, or both. Haptic systems may perform sensing functions and actuating functions by a integrated haptic device. In some embodiments, haptic systems may be coupled to a display screen, audio processing circuitry, image processing circuitry, device hardware, or other system to provide for any combination of tactile, visual, and audio interactions. Actuation may occur, in some embodiments, substantially normal to a substantially planar surface, which may allow for three dimensional contouring of the planar surface. 
       FIG. 1  is a block diagram of a haptic system in accordance with an embodiment of the invention. As shown in  FIG. 1 , the haptic system  100  includes a processing unit  110 , control circuitry  120 , and a haptic device  130 . The processing unit  110  is configured to receive an input signal  10 , and performs a haptic conversion to convert the content of the input signal  10  into a control signal  20  that is transmitted to the control circuitry  120 . For example, the processing unit  110  may comprises one or more processors, or digital signal processors (DSP), but the invention is not limited thereto. The haptic conversion may include operations selected from at least one of decoding, processing, analyzing, converting the content of the input signal, retrieving, transforming, calculating relevant information of the input signal, and any combination operations mentioned in above, so as to generate the control signal  20  accordingly. The control signal  20  may be one single signal or a set of signals, and may include information for indicating the strength of the haptic feedback to be provided as well as the location information of rendering the haptic feedback on the haptic device  130 . The input signal  10  may be an image, video, acoustic signal, message, or sign, but the invention is not limited thereto. The details for converting different types of input signals into an associated control signal will be described later. 
     The control circuitry  120  is configured to receive the control signal  20  and generate a driving signal  30  so as to control the haptic device  130  to provide haptic feedback. For example, the control circuitry  120  may comprise a digital-to-analog converter (DAC)  121  that is configured to convert a digital control signal  20  received from the processing unit  110  into the driving signal  30 , e.g. an audio waveforms. The haptic feedback provided by the haptic device  130  may include vibration, shape deformation, friction change, volumetric haptic shape, transcutaneous nerve stimulation, temperature stimulation etc., but the invention is not limited thereto. 
     In an embodiment, given that the input signal  10  is a video signal and the haptic device  130  including vibrator to provide vibration haptic feedback, the processing unit  110  may analyze the video signal to obtain the motion vectors of macroblocks of an image frame in the video signal. The magnitude and/or frequency of the driving signal  30  depend on the motion vectors of the macroblocks of the image frame. For example, when the scene of the video signal moves fast, it indicates that the magnitude of the motion vectors of the image frame in the video signal would be larger, and thus the processing unit  110  may provide a control signal  20  indicating a higher amplitude and/or frequency of vibration to the control circuitry  120 , so that the control circuitry  120  may provide a driving signal  30  to control the haptic device  130  to provide vibration haptic feedback having a corresponding amplitude and/or frequency. It should be noted that the control circuitry  120  may provide a driving signal  30  to control the haptic device  130  so as to provide a global vibration haptic feedback for the whole haptic device  130 . In yet another embodiment, the control circuitry  120  may provide a plurality of driving signals  30  to locally control a plurality of vibrators located in specific regions of the haptic device  130  so as to provide regional vibration haptic feedbacks respectively. 
     In addition, the processing unit  110  may analyze the video signal to obtain the foreground object, its position, and motion vectors of the foreground object. The processing unit  110  may generate the control signal  20  to control at least a local vibrator of the haptic device  130  on the position at which the foreground object is presented. In some embodiments, the processing unit  110  may also detect the content change of the video signal, and generate the control signal  20  corresponding to the content change. 
     Moreover, the processing unit  110  may also analyze the acoustic content of the video signal, and may generate the control signal  20  based on the acoustic content: For example, when there is a huge explosion scene in the video signal, and the associated acoustic content may have a sudden explosion sound with a high magnitude. Accordingly, the control signal  20  generated by the processing unit  110  may indicate a sudden strong vibration on the haptic device  130  while the explosion occurs. 
     In another embodiment, the haptic device  130  includes electrostatic generator which is capable of providing electrostatic haptic feedback. For example, the haptic device  130  may be an electrostatic passive haptic display or an electrostatic passive haptic touchpad and the input signal  10  is a video signal, but the invention is not limited thereto. Given that the haptic device  130  is an electrostatic passive haptic display, the user may sense friction change while sliding his/her finger or fingers on the surface of the haptic device  130 , and the friction force sensed by the user corresponds to the content of the image. Additionally, the friction force (i.e. sensing slippery or rough) sensed by the user can be controlled by the driving signal  30  applied to the haptic device  130 . In some embodiments, given that the haptic device  130  is an electrostatic passive haptic display, the user may place his/her finger or fingers directly on the surface of the haptic device  130 , and the voltage difference occurs between the finger (as an electrode) and the haptic device  130 . This allows a direct touch without the use of any additional components. In addition, the surface of the haptic device  130  may include several segmented electrodes placed in a plurality of different regions. In some other embodiments, given that the haptic device  130  is an electrostatic passive haptic touchpad, the user may experience tactile feedback by putting his/her finger or fingers on the touchpad and explores the surface of the haptic device  130 . For example, there may be a cursor or pointer on a display of the haptic system  100 , and the user may sense the friction haptic feedback where the cursor or pointer is located when putting his finger on the surface of the haptic device  130 . 
     In the aforementioned embodiments, the processing unit  110  may perform a two-dimensional frequency-domain transformation (e.g. digital wavelet transform (DWT) or digital cosine transform (DCT)) on the image frame of the video signal, and thus a map in the frequency domain can be obtained. When pixel values in a specific image block vary a lot, it indicates that the frequency in the specific image block is higher, and thus the control signal  20  generated by the processing unit  110  may control the control circuitry  120  to provide the driving signal  30  with higher voltage or frequency, so that the user may sense that the friction force on the specific image block is larger (i.e. sensing a rough surface) while sliding on the corresponding specific image block of the haptic device  130  via his finger. 
     In some embodiments, the haptic device  130  (e.g. an electrostatic passive haptic display) could be able to provide friction haptic feedback. The applied driving signal  30  to drive the haptic device  130  for generating a friction haptic feedback depends on the magnitude of motion vectors of the image frame. Additionally, the control circuitry  120  may provide a plurality of driving signals  30  to control different segmented electrodes in the haptic device  130  to provide a friction haptic feedback at which the foreground object or an obvious image feature (e.g. a human face) is appeared in the image frame. Alternatively, the friction force provided by the haptic device  130  may also depend on the magnitude or frequency of the acoustic content of the incoming video signal, and the details can be found in the aforementioned embodiment. 
     Similarly, given that the haptic device  130  is further capable of providing deformation haptic feedback (e.g. through a display screen having a deformation function on its surface), the processing unit  110  may also generate the control signal  20  associated with the content of the video signal. For example, as described above, the degree of deformation haptic feedback also corresponds to the magnitude of the motion vectors and/or the content change in the video signal. 
     In an embodiment, the haptic device  130  is a haptic deformation display having a plurality of deformation components, e.g. micro-fluidics display panel, and the processing unit  110  is capable of converting an incoming stereoscopic video signal or 3D image into a control signal  20  indicating deformation magnitude and shape of the deformation components of different regions on the haptic device  130 . It should be noted that the surface of the haptic device  130  can be divided into a plurality of regions, and the control circuitry  120  may generate a plurality of driving signals  30  to control the deformation component of respective region according to the control signal  20  from the processing unit  110 . For example, the haptic deformation display is originally in a first-shape configuration, and the deformation component may at least partially define the shape of the haptic deformation display, thereby causing the shape of the haptic deformation display to deform into a second-shape configuration that is different from the first-shape configuration. The second-shape configuration may substantially be maintained. 
     The processing unit  110  may calculate depth information (e.g. a depth map) associated with image frame in the stereoscopic video signal or 3D image, and the deformation height of the deformation component of a specific region corresponds to the associated depth information. In addition, the processing unit  110  may perform foreground segmentation on the incoming video signal or image to obtain a foreground object. The control signal  20  controls the individual deformation component of each region of the haptic device  130  to generate the deformation shape corresponding to the segmentation result of the foreground object. 
     In some embodiments, control signal  20  is generated according to the temporal motion prediction results of the incoming stereoscopic video signal or 3D image when the haptic system  100  is used in a video codec system. Similarly, the haptic feedback (e.g. slippery or rough) may also correspond to the magnitude, or correspond to motion vectors in the stereoscopic video signal or 3D image. In addition, the processing unit  110  may perform foreground segmentation on the stereoscopic video signal or 3D image to obtain a foreground object so as to generate the control signal  20  accordingly by the processing unit  110 . The control circuitry  120  may generate the driving signals  30  to control the corresponding deformation components where the foreground object is presented on the haptic deformation display. Furthermore, the deformation behavior may also correspond to the acoustic content in the stereoscopic video signal, and the aforementioned embodiments can be referred to for the details. 
     In an embodiment, the haptic device  130  equipped with transcutaneous electrical nerve stimulation device having a plurality of electrodes. The processing unit  110  is capable of converting an incoming video signal or audio signal into a control signal  20 . The control circuitry  120  may provide a plurality of driving signals  30  to control the plurality of electrodes in the haptic device  130  to provide a transcutaneous nerve stimulation haptic feedback. The driving signals  30  indicate amplitude, frequency and/or duration of electrical impulses which would be applied by the plurality of electrodes on the haptic device  130 . 
     In yet an embodiment, the haptic device  130  equipped with heaters. The processing unit  110  is capable of converting an incoming video signal or audio signal into a control signal  20 . The control circuitry  120  may provide a plurality of driving signals  30  to control the heaters in the haptic device  130  to provide thermal stimulation haptic feedback. The driving signals  30  control thermal energy, and/or heating duration of the heaters of the haptic device  130 . 
     In view of the above, the haptic system  100  is able to convert the input content into various types of haptic feedback, e.g. vibration, shape deformation, friction change, volumetric haptic shape, transcutaneous nerve stimulation, and/or temperature stimulation mentioned in the embodiments in above.  FIG. 2  is a flow chart of a control method for a haptic system in accordance with an embodiment of the invention. In  FIG. 2 , in step S 210 , the input signal  10  is received. In step S 220 , the haptic conversion is performed based on the content of the input signal  10  to generate a control signal  20  indicating associated haptic feedback. In step S 230 , a haptic feedback driving signal  30  is provided to the haptic device  130  according to the control signal  20 . 
     In yet another embodiment, the input content can be processed so as to generate driving signal to provide various haptic feedback with the help of sensor data. For example, the haptic system  100  may further includes at least a sensor  140  such as GPS sensor, ambient light sensor, proximity sensor, thermal sensor, accelerometer, pressure sensor, heart-rate sensor, or UV sensor, etc. The sensor  140  provides sensor data  40  to the processing unit  110 . In an embodiment, the processing unit  110  receives the input video signal  10  and GPS data from the GPS sensor and generates a control signal  20  according to the input video signal  10  and GPS data, where the control signal  20  may be converted to the driving signal  30  to control the haptic device  130  to provide the haptic feedback, e.g. vibration haptic feedback. For example, the vibration magnitude may depend on the motion vectors of the image frame of the video signal, and also depend on the GPS data e.g. geographical location of the user such as at home or office. The vibration haptic feedback may be weaker when the user is at home, and the vibration haptic feedback may be stronger when the user is in the office. Additionally, the processing unit  110  may also generate the control signal  20  according to the video signal and a plurality of sensor data  40  so as to control the haptic feedback of the haptic device  130 . 
       FIGS. 3A and 3B  are diagrams illustrating enhancing haptic feedback on the touch position in accordance with an embodiment of the invention. In an embodiment, the haptic device  130  includes a touch module  131  is configured to detect the touch position  310  where the user&#39;s finger is touching on the haptic device  130  or is touching the mid-air space above the haptic device  130 , as shown in  FIG. 3A . And the touch module  131  is further configured to send the touch position information  50  to the processing unit  110 , and thus the processing unit  110  may adjust the control signal  20  to enhance the haptic feedback corresponding to the touch position  310 . For example, when the user&#39;s finger touches on the haptic device  130 , the user may sense that there is a first haptic feedback on the touch position  310 . After receiving the touch position information  50 , the processing unit  110  may increase the strength (e.g. magnitude or frequency) of the haptic feedback on or near the touch position  310 , and thus the user may sense a stronger haptic feedback on the touch position  310 , as shown in  FIG. 3B . 
     In an embodiment, the haptic device  130  further comprises an ultrasound haptic component. The ultrasound haptic component is capable of producing a volumetric haptic shape to the user by vibrating the air particles near the ultrasound haptic component, so that the user may sense the friction haptic feedback by “touching” the mid-air where the ultrasound haptic component renders the volumetric haptic shape. In addition, the ultrasound haptic component is capable of detecting the touch position where the user is touching in mid-air, and sends the touch position information  50  to the processing unit  110 . Thus, the processing unit  110  may enhance the ultrasound haptic feedback on the touch position. 
     In view of the above, the haptic system  100  is able to convert the input content and the sensor data  40  into various types of vibration, shape deformation, friction change, and/or volumetric haptic shape mentioned in the embodiments in above.  FIG. 4  is a flow chart of a control method for a haptic system in accordance with another embodiment of the invention. Referring to  FIG. 4 , in step S 410 , the input signal  10  and sensor data  40  are received. In step S 420 , the haptic conversion is performed based on the content of the input signal  10  and the sensor data  40  from the sensors  140  of the haptic system  100  to generate a control signal  20  indicating associated haptic feedback. In step S 430 , the haptic feedback driving signal  30  is provided to drive the haptic device  130  according to the control signal  20 . 
     In another embodiment, the sensors  140  of the haptic system  100  may further comprise a microphone for detecting the acoustic signal performed by the user. And the sensors  140  of the haptic system  100  may further comprise a camera for detecting the motions performed by the user. For example, the processing unit  110  can be further controlled by the user through a gesture, a voice command, an ultrasound signal, or a pose captured by the microphone or the camera. Specifically, when the processing unit  110  detects the gesture, voice command, ultrasound signal, or pose performed by the user, the processing unit  110  may apply a specific haptic signal instead of the original control signal, so that the control circuitry  120  would generate the driving signal  30  according to the specific haptic signal. It should be noted that the haptic signal includes vibration, friction, or deformation, but the invention is not limited thereto. 
     In an embodiment, the haptic system  100  may further comprise a network interface unit  150  configured to communicate with other devices using wired or wireless protocols such as LAN, Bluetooth, Wifi, LTE etc. For example, the processing unit  110  of the haptic system  100  may receive a status signal  60  indicating a remote user&#39;s action or a remote object&#39;s status from a remote electronic device (not shown in  FIG. 1 ), and the processing unit  110  may generate the control signal  20  corresponding to the status signal  60 , and convert the control signal  20  to a driving signal  30  through control circuitry  120 . It should be noted that the remote electronic device may also comprise sensors such as accelerometer and gyroscope (not limited), that detect the remote user&#39;s action or a remote object&#39;s status, and the status signal  60  is generated based on the sensor data  40  from the sensors of the remote electronic device. 
     In yet another embodiment, the processing unit  110  may analyze a input image signal and identify objects having different materials from the image signal by using object recognition techniques or material data base from an external source. And the processing unit  110  is configured to be capable of generating haptic signals corresponding to different materials such as cloth, glass, paper, metal, air, but the invention is not limited thereto. For example, given that the haptic device  130  provides friction haptic feedback, the user may sense that the cloth may be rough and the metal may be slippery when touching on the associated portion on the haptic device  130  e.g. a electrostatic passive haptic display. That is, different material has a corresponding haptic signal that causes an individual haptic feedback, e.g. different friction haptic feedback driven by driving signals  30  having different magnitude and/or frequency. 
     Furthermore, the haptic signals of different materials can be recorded using analog signals such as audio signals. Specifically, each material has its own acoustic material haptic file. When the processing unit  110  recognizes the materials in the video signal, the processing unit  110  may retrieve the acoustic material haptic files of the recognized materials from a database (not shown in  FIG. 1 ), and generate the control signals  20  associated with the recognized materials according to the retrieved acoustic material haptic files. The database could be stored in the haptic system  100 , or the data base could be received from outside of the haptic system  100  through network interface unit  150 . With the help of acoustic material haptic files, the haptic properties of different materials can be easily duplicated and distributed, and different haptic systems may share the same acoustic material haptic files, so that the user may sense the same haptic feedback on different haptic systems. 
     In an embodiment, the input signal  10  may include an analog content, e.g. an analog audio signal, which can be regarded as the control signal  20 . Thus, the analog content could be treated as the driving signal  30  and be sent to the haptic device  130  for proving haptic feedback directly. For example, the processing unit  110  may determine whether the input signal  10  is an analog signal. If the input signal  10  is an analog signal, the processing unit  110  may directly send the analog signal to the haptic device  130 . For example, the direct output haptic feedback from the input audio signal can also reflect the vibration from the air wave such as collision between materials, or the explosion of fireworks. 
     If the input signal  10  is a digital signal, the processing unit  110  may perform haptic conversion to convert the digital signal into an analog control signal that is transmitted to the control circuitry  120 . However, both the digital content and the analog content may exist in a multimedia file. For example, when the input signal is a video signal including video images and an analog acoustic signal, the processing unit  110  may perform haptic conversion on the video signal to generate a first control signal, and directly send the analog acoustic signal to the control circuitry  120  as the second control signal. Accordingly, the haptic device  130  performs haptic feedback according to the first control signal and the second control signal. 
       FIG. 5  is a flow chart of a control method for a haptic system in accordance with yet another embodiment of the invention. In step S 510 , an input signal comprising a digital signal and an analog signal is received. In step S 520 , the haptic conversion is performed on the digital signal to generate a first control signal. In step S 530 , analog signal is directly determined as a second control signal. In step S 540 , haptic feedback driving signal  30  is provided according to the first control signal and the second control signal. In an embodiment, the first control signal is converted by a DAC  121  then to be mixed with the second control signal to generate the driving signal  30 . 
     While the invention has been described by way of example and in terms of the preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements (as would be apparent to those skilled in the art). Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.