Patent Description:
A haptic device is a human machine interface which enables a way of interaction other than visual and sound, creating a sense of touch through different technologies. A sense of touch could for instance be created through vibrations or force feedback.

Traditionally, haptic devices have been utilized in tactile displays, gaming and simulators. However as virtual reality (VR) and mixed reality (MR) (encompassing augmented reality, AR, and augmented virtuality, AV) devices are rapidly growing, haptic devices are becoming an integral part of such systems to close the interaction loop. In VR and MR systems, haptic devices take the role of providing touch feedback for virtual content. An immersive VR or MR system could not be realized without effective and precise haptic feedbacks.

Haptic perception consists of kinaesthetic and tactile sense and relates to the sensation of the size, shape, mass, texture and stiffness of physical objects, surfaces, etc. Kinaesthetic information refers to the information perceived when moving joints, muscles and tendons, while tactile information refers to information retrieved via the skin. For example, haptic devices can create vibrations through direct contact to the user's skin (e.g. Avatar VR); they could use motors to steer the movement of a tool which is held by the user (e.g. Geomagic Touch) or they could create mid-air haptic sensations (e.g. Ultrahaptics, MyVox) which is part of the environment in which embodiments presented herein can be applied.

The mid-air haptic technologies provide users with the freedom of not having to wear or hold anything, therefore being easier to use and enhancing user experience.

Mid-air haptic technologies could be applied to a vast spectrum of industries including automotive, healthcare, marketing and advertisement, gaming and more. They can be used as feedback reinforcement of graphical user interfaces such as buttons, slide bars and pull-down menus. For instance, by embedding a mid-air haptic device into the infotainment system of a car, it is possible to interact with the system in mid-air, without the driver needing to take eyes from the road.

Another example would be publicly available machines such as cash machines, elevators vending machines. Mid-air feedback would provide a cleaner and more hygienic interaction while reducing the needed maintenance of the machine.

Mid-air haptic devices mainly use ultrasound to create tactile sensations. These ultrasonic haptic devices utilize an array of ultrasonic transducers to create ultrasonic waves which coincide at a point in the space, an ultrasonic focal point. An ultrasonic focal point that is moved in space or changed in intensity could create a tactile sensation on the hand of the user. In order to track and determine the position of the hand in space to be able to project a focal point on it, hand tracking systems are used, such as those provided by the company Leap Motion.

Precise hand tracking is used to ensure that the user feels the tactile sensation in the right location. The hand tracking device derives the position of the user hand and its elements and this information is then used to define where the actuation should take place (the ultrasonic focal point used for the haptic actuation). The hand tracking also needs to be accurate in order to provide a higher resolution tactile actuation in specific point(s) of the user hand that are in "contact" with the (virtual) object to be rendered.

<CIT> refers to a method and apparatus for Modulating Haptic Feedback, using ultrasound, where an acoustic field is modulated for providing tactile sensations.

<NPL> disclose a system for multi-point haptic feedback above an interactive surface, which employs focused ultrasound to project discrete points of haptic feedback through the display and directly on to a users unadorned hands.

<CIT> disclose a three dimensional user interface cursor control which is activated by gestures performed by a user positioned within a field of view of a sensing device coupled to a computer, where 3D coordinates of a 3D coordinate system of the sensing device are transformed into 3D coordinates of a 3D coordinate system of the user.

<CIT> refers to a system for producing an acoustic field from a plurality of ultrasonic transducer arrays, each of which has known relative positions and orientations and for calibrating the ultrasonic transducer arrays by using the relative position of each of the plurality of ultrasonic transducer arrays.

In experiments, the inventors have discovered that the tactile feedback is not sufficiently precise. The deviation between ultrasonic focal point and the point tracked by the hand tracking sensor could be significant, e.g. one centimetre or more, which drastically affects the user experience.

One objective is to improve calibration between a first coordinate system of an ultrasonic haptic device and a second coordinate system of a visual sensor device.

According to a first aspect, it is provided a method for determining a transformation between a first coordinate system of an ultrasonic haptic device and a second coordinate system of a visual sensor device as defined in claim <NUM>.

The step of determining a position of the ultrasonic focal point may be based on a body part tracker.

The body part tracker may form part of the visual sensor device.

The step of determining a position of the ultrasonic focal point may be based on depth information forming part of the images.

The method may further comprise the step of: instructing the user to find the ultrasonic focal point with the body part.

According to a second aspect, it is provided a transformation determiner for determining a transformation between a first coordinate system of an ultrasonic haptic device and a second coordinate system of a visual sensor device as defined in claim <NUM>.

The instructions to determine a position of the ultrasonic focal point may be based on data from a body part tracker.

The instructions to determine a position of the ultrasonic focal point may be based on depth information forming part of the images.

The transformation determiner may further comprise instructions that, when executed by the processor, cause the transformation determiner to instruct the user to find the ultrasonic focal point with the body part.

The instructions to instruct the user may comprise instructions that, when executed by the processor, cause the transformation determiner to trigger three-dimensional rendering of a region encompassing the ultrasonic focal point.

The instructions to determine the position of the ultrasonic focal point may comprise instructions that, when executed by the processor, cause the transformation determiner to detect a skin vibration on the body part.

The first transformation may be in the form of a transformation matrix.

The transformation determiner may further comprise instructions that, when executed by the processor, cause the transformation determiner to apply calibration, wherein the calibration is based on the transformation calculated in the instructions to calculate a first transformation.

According to a third aspect, it is provided a computer program for determining a transformation between a first coordinate system of an ultrasonic haptic device and a second coordinate system of a visual sensor device as defined in claim <NUM>.

According to a fourth aspect, it is provided a computer program product, as defined in claim <NUM>, comprising a computer program according to the third aspect and a computer readable means on which the computer program is stored.

In embodiments presented herein, a solution is provided which allows calibration of an ultrasonic haptic device and a visual sensor device. The solution allows determination of the pose of the hand of the user in the coordinate system of the ultrasonic haptic device. In this way, the actuation by the ultrasonic haptic device can be provided in the correct location of the hand of the user. As ultrasonic waves hit the skin of a hand, they create vibrations on the skin. Images of the hand are used to locate the vibrations of the ultrasonic haptic feedback on the hand of the user and an estimation of the position of the ultrasonic focal point in the coordinate system of the visual sensor device is performed. Then, the position of the ultrasonic focal point in the ultrasonic haptic device coordinate system is set as a known position. After repeating this a number of times, the positions of the ultrasonic focal point in both coordinate systems are used to calculate the transformation between the two coordinate systems, as explained in more detail below.

In the prior art, there is no way to automatically find the transformation between the coordinate systems of the ultrasonic haptic device and the body part tracking sensor (e.g. hand tracking sensor). In the case of Ultrahaptics device, the transformation between the transducers board and the body part tracking sensor was computed once when the device was manufactured and is hardcoded in the device SDK (Software Development Kit), which can be applied since the body part tracking sensor is fixed and cannot be moved. However, this prevents other body part tracking sensors to be used with the Ultrahaptics device, and the provided tracking sensor cannot be repositioned for improved tracking.

<FIG> is a schematic diagram illustrating an environment in which embodiments presented herein can be applied. An ultrasonic haptic device <NUM> is used to provide haptic feedback via an array of ultrasonic transducers to the user (e.g. Ultrahaptics). The ultrasonic waves created by the transducers are controlled to coincide at a point in the space, at an ultrasonic focal point <NUM>. The ultrasonic focal point <NUM> can be moved in space or changed in intensity to control a tactile sensation on a body part <NUM> of the user. In the examples presented herein, the body part <NUM> exemplified as a hand. However, it is to be noted that the embodiments presented herein are not restricted to the body part being a hand; any suitable body part of the user can be used. In order to track and determine the position of the hand <NUM> in space, to be able to project an ultrasonic focal point on it, a body part (in this example hand) tracking system <NUM> is provided, e.g. from Leap Motion. The body part tracker <NUM> can be a sensor which tracks a body part (e.g. the hand) of the user, e.g. using an RGB (Red Green Blue) visual sensor, a depth camera, etc..

A visual sensor device <NUM> comprises a visual sensor which is able to capture images. Hence, the visual sensor device <NUM> comprises a camera. Optionally, the visual sensor device <NUM> can be combined with the body part tracker <NUM> in a single physical device. The visual sensor device <NUM> can be placed next to the ultrasonic haptic device <NUM>, facing the active region of the ultrasonic haptic device <NUM>. The visual sensor device <NUM>, either alone or in combination with the body part tracker <NUM>, can be used to determine depth, e.g. distance, to the skin of a tracked hand.

A transformation determiner <NUM> is provided to calibrate the conversion between the coordinate systems of the visual sensor device <NUM> and the ultrasonic haptic device <NUM>, as explained in more detail below.

<FIG> are schematic diagrams illustrating the effect of an ultrasonic focal point on a body part <NUM>, in this case a hand. Each marking on the hand, in the form of a small line, indicates a small vibration on the skin. In <FIG>, vibrations of the skin of the hand are shown when the hand approaches, but is not yet at, the ultrasonic focal point. In <FIG>, the hand is closer to the ultrasonic focal point, increasing the intensity of the vibrations, but not yet to a maximum. In <FIG>, the skin of the hand is in the ultrasonic focal point <NUM> and vibrations are at its maximum.

A sequence of images (e.g. as a video stream) is captured of the hand <NUM> as it moves through an ultrasonic focal point. When the vibrations are at a maximum, the position of the ultrasonic focal point <NUM> of that frame can reliably be determined. In this frame, the amplitude of the vibrations is at its maximum and the vibrations are most focused, i.e. in the smallest region. As long as the user moves the hand through the ultrasonic focal point at reasonable speed, this will result in capturing the frame containing the ultrasonic focal point.

The margin of error of this determination depends on the frame rate of the video capture and how focused the ultrasonic focal point <NUM> is.

<FIG> are flow charts illustrating embodiments of methods for determining a transformation between a first coordinate system of an ultrasonic haptic device and a second coordinate system of a visual sensor device. The method is performed in the transformation determiner.

In a trigger focal point generation step <NUM>, the transformation determiner triggers generation of an ultrasonic focal point by the ultrasonic haptic device. The position of the ultrasonic focal point is defined in the first coordinate system, i.e. the coordinate system of the ultrasonic haptic device.

This step can make use of an API (Application Programming Interface) provided by the manufacturer of the ultrasonic haptic device. The ultrasonic haptic device then renders an ultrasonic focal point P_uhd in the first coordinate system where P_uhd is a point in three dimensions X, Y, Z. See <FIG>, where X and Y axes are extended along the edges of the ultrasonic haptic device and the Z axis is perpendicular to the XY plane.

In one embodiment, points are selected to be on top of the centre and corners of the device and at different heights.

In an obtain images step <NUM>, the transformation determiner obtains images, captured by the visual sensor device. The images depict a body part of the user while the ultrasonic focal point is active. The images can be in the form of a set of sequential images, e.g. a video stream at a specific frame rate. The body part can be a hand, which is convenient for the user to move as requested. However, there is nothing preventing the body part to be any other suitable body part, such as arm, elbow, foot, leg, etc..

In a determine position of focal point step <NUM>, the transformation determiner determines, based on the images, a position of the ultrasonic focal point in the second coordinate system, i.e. the coordinate system of the visual sensor device. This is performed when the body part of the user is, within a margin of error, in the ultrasonic focal point.

Ultrasonic focal points create air pressure, hence vibrations on the skin, changing the composition of the skin. The changes in the skin surface are thus captured in the images taken by the visual sensor device. The images are used to detect the vibrations, and optionally their amplitude and frequency. Optionally, interpolation between images is used to improve the resolution of this determination.

In one embodiment, the vibration frequency response of different parts of the skin of the hand is estimated. Projection of the ultrasonic focal point on the skin of the hand creates a certain range of vibration frequency response. For instance, the vibration frequency response of the centre of the palm is different from the vibration frequency response of a fingertip when an ultrasonic focal point is projected on these different parts of the hand. This range of frequency responses could be measured in advance and a model corresponding to the correct frequency response given an ultrasonic focal point in a hand of the user can be built.

Furthermore, the frequency response on an ultrasonic focal point is not the same as the frequency response when you have multiple dispersed points of impact (i.e. when the body part is not in the ultrasonic focal point). The reason for this is that multiple actuators (of the ultrasonic haptic device <NUM>) all impact the ultrasonic focal point in a very small region and in an overlapping manner. Similarly, since a lot of actuations are performed in the same area, the amplitude resulting from these actuations is larger. This is also something that can be measured in the sequence of images.

For every captured frame, an estimated vibration frequency response of different parts of the hand is compared to the vibration frequency response range of the ultrasonic focal point. For instance, if the centre of the palm is exposed to the ultrasonic waves, as the hand gets closer to the ultrasonic focal point (see <FIG>), the frequency response of the centre of the palm gets closer to the frequency response range of centre of the palm to an ultrasonic focal point. According to this embodiment, the frame in which an estimated vibration frequency response falls into ultrasonic focal point frequency response range is the frame that accommodates the ultrasonic focal point.

It is to be noted that the detected ultrasonic focal point on the skin is not an exact point in space, but a region which depends on the resolution of the ultrasonic haptic device. At present, this region may be an area of a few square millimetres to a few square centimetres. For example, for the Ultrahaptics device, this region is of about one square centimetre. The centroid of this shape is considered to be the position of the ultrasonic focal point.

After detecting the frame which contains the highest intensity of the ultrasonic waves, the position of the ultrasonic focal point is obtained in the second coordinate system. To obtain this position, P_vs, depth information is used.

The depth information can be provided by a depth sensor (e.g. Intel Realsense D435) which contains a depth camera and a single RGB camera. The RGB image information can be used to detect the vibration point in the palm (2D point), and then the depth camera is used to track the palm and the corresponding 2D point in the 3D coordinate system of the camera (P_vs).

The position of the ultrasonic focal point can be determined based on a body part tracker. The body part tracker can form part of the visual sensor device or be separate devices. In any case, in this embodiment, the geometric relationship between the visual sensor device and the body part tracker is known, either by fixed colocation or fixed geometric relationship, or through a deduced transformation, e.g. as explained in more detail below.

The position of the ultrasonic focal point can be determined based on depth information forming part of the images.

The position of the ultrasonic focal point can be determined based on detecting a skin vibration on the body part, as illustrated in <FIG> and described above.

In one embodiment, the body part tracking information could be computed from images using deep neural networks as known in the art per se, see e.g. <NPL> at the time filing this patent application. The position of the ultrasonic focal point projected on the hand of the user is calculated for the frame (i.e. image) mentioned above, in which the ultrasonic focal point is identified. This position, P_vs, is the position of the ultrasonic focal point in the second coordinate system.

In one embodiment, the visual sensor device forms part of the same device as the body part tracking sensor. In this case, the transformation between the visual sensor device and the body part tracking sensor is known (e.g. provided by the manufacturer). The position of the ultrasonic focal point in the second coordinate system could be retrieved from the body part tracking sensor.

In one embodiment, the body part tracking sensor is not integrated in the visual sensor device. The body part tracking sensor and the visual sensor device could then be calibrated separately, as described below.

In a conditional done step <NUM>, the transformation determiner determines when the ultrasonic focal point positioning is done, i.e. whether a sufficient number of ultrasonic focal points have been positioned. At least four points are needed in total for a three-dimensional coordinate system transformation. However, the method can use more points to improve accuracy and reduce effect of noisy measurements and/or outliers. When the ultrasonic focal point positioning is done, the method proceeds to a calculate transformation step <NUM>. Otherwise, the method returns to the trigger focal point generation step <NUM>. For each iteration of step <NUM>, each ultrasonic focal point is in a different location compared to the other (previously generated) ultrasonic focal points.

In the calculate transformation step <NUM>, the transformation determiner calculates a first transformation between the first coordinate system and the second coordinate system. This calculation is based on the positions of the ultrasonic focal points in the first coordinate system and the second coordinate system. In this way, a relationship between the ultrasonic focal point and the visual sensor device is calibrated. The first transformation can e.g. be in the form of a transformation matrix. Alternatively, the first transformation can be in the form of a set of equations or any other suitable form.

In order to be able to calculate the transformation between the ultrasonic haptic device and the visual sensor device, as explained above, at least four linearly independent points in the first coordinate system (P_uhd) and their corresponding points in the second coordinate system (P_vs) are used. When more than four sets of points are available, outlier rejection methods such as RANSAC can be applied and/or a least squares methods can be applied for computing the best transform (in the linear square sense), given the set of measurements.

For instance, the transformation matrix T_uhd_vs applied in the following equation (<NUM>) can be calculated:
<MAT>.

Looking now to <FIG>, only new or modified steps compared to the steps of <FIG> will be described.

In an optional instruct user step <NUM>, the transformation determiner instructs the user to find the ultrasonic focal point with the body part. In one embodiment, this comprises triggering three-dimensional rendering of a region encompassing the ultrasonic focal point. The three-dimensional rendering can be achieved e.g. using mixed reality or VR (Virtual Reality) hardware. The guidance can also be performed using auditory and/or two-dimensional graphical user interface, e.g. on a computer and/or smartphone.

In other words, the user is guided to move the hand (or other body part) in the area above the ultrasonic haptic device to thereby locate the ultrasonic focal point.

It is indicated to the user to place an open hand above the ultrasonic haptic device in parallel to the transducers array in the whereabouts of the point P_uhd in the first coordinate system.

The user can choose which part of the hand is going to be exposed to the ultrasound. It could be the distal phalange (the last joint of a finger) of the index or long finger, palm or back of the hand.

In one embodiment, it is indicated to the user to move the hand up along the Z axis where the ultrasonic focal point is projected, such that the user feels the ultrasonic pressure on the hand strengthen when approaching the ultrasonic focal point, after which it weakens. Then the user lowers the hand along the Z axis. For instance, if the ultrasonic focal point is created <NUM> centimetres above the top left corner of the device, the user begins by placing the hand on the top left corner. The user gradually raises the hand to around <NUM> centimetres above the starting point and then starts to lower it on the same path until the hand reaches the device. At this point ultrasonic haptic device stops projecting and visual sensor device stops capturing images.

In one embodiment, it is indicated to the user to move the hand until the user finds the area in which the ultrasonic focal point feels more intense. It is indicated to the user to keep moving the hand within this area, in a virtual sphere of a few cubical centimetres. After a few seconds of the hand hovering in the sphere around the ultrasonic focal point, the ultrasonic haptic device stops projecting and visual sensor device stops capturing images and the position of ultrasonic focal point can be determined.

In one embodiment the user has access to an AR device, which visualizes the position of the point in the ultrasonic haptic device workspace. The AR device also visualizes the virtual region around the ultrasonic focal point in which the hand should be moved. After a few seconds of the hand hovering around the ultrasonic focal point, the ultrasonic haptic device stops projecting and the visual sensor device stops capturing images.

In an optional apply calibration step <NUM>, the calibration based on the transformation calculated in the calculate transformation step <NUM> is applied. In this way, the calibration is applied for whenever the first coordinate system of the ultrasonic haptic device and the second coordinate system of the visual sensor device need are used together, e.g. for tracking the body part to ensure that the user feels the tactile sensation in the right location.

When the ultrasonic haptic device is equipped with a body part tracking sensor, the body part tracking sensor for the ultrasonic haptic device can be calibrated according to the following.

The body part tracking sensor and the visual sensor device are configured to have similar optical characteristics such as sampling frequency, scale and focus. Steps <NUM> and <NUM> from above are performed. Also, step <NUM> is performed, but here the body part tracking sensor also captures images of the hand together with the visual sensor device. At this point, the frame containing most intense ultrasonic focal point has been detected and the position of the ultrasonic focal point in the second coordinate system (P_vs) is acquired.

The body part tracking sensor information is obtained that corresponds to the detected frame of the visual sensor device, thus acquiring the position of the ultrasonic focal point in the first coordinate system, P_hts.

The mentioned steps are then repeated for at least four points.

In a way similar to step <NUM> mentioned above, a transformation matrix T_hts_vs is then calculated between the coordinate systems of the body part tracking sensor and the second coordinate system. This computation is based on both the body part sensor and the visual sensor device capturing the same corresponding points in a synchronized manner, after which a transformation can be computed from these corresponding points in both coordinate systems according to the following.

The calculated transformation T_hts_vs could be used to directly map a point in the first coordinate system to the body part tracking sensor coordinate system when a user is using the calibrated ultrasonic haptic device, according to the following:
<MAT>
<MAT>.

From (<NUM>) and (<NUM>), equation (<NUM>) is derived:
<MAT>.

According to embodiments presented herein, an automatic calibration between the coordinate systems of the visual sensor device and the ultrasonic haptic device is achieved. The presented solution does not rely on hardcoding of such a transformation. Hence, freedom of placing the visual sensor device at any desirable location around the ultrasonic haptic device is provided, after which a new transformation and calibration, applicable for the present location of the visual sensor device, can be executed.

No additional hardware is needed than what is used in a regular setup. All that is needed is the integral component of the system (ultrasonic haptic device and visual sensor device), the hand (or other body part of the user) of the user and the transformation determiner.

<FIG> is a schematic diagram illustrating components of the transformation determiner <NUM> of <FIG>. A processor <NUM> is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions <NUM> stored in a memory <NUM>, which can thus be a computer program product. The processor <NUM> could alternatively be implemented using an application specific integrated circuit (ASIC), field programmable gate array (FPGA), etc. The processor <NUM> can be configured to execute the method described with reference to <FIG> above.

The memory <NUM> can be any combination of random-access memory (RAM) and/or read only memory (ROM).

The transformation determiner <NUM> further comprises an I/O interface <NUM> for communicating with external and/or internal entities. Optionally, the I/O interface <NUM> also includes a user interface.

Other components of the transformation determiner <NUM> are omitted in order not to obscure the concepts presented herein.

<FIG> is a schematic diagram showing functional modules of the transformation determiner <NUM> of <FIG> according to one embodiment. The modules are implemented using software instructions such as a computer program executing in the transformation determiner <NUM>. Alternatively or additionally, the modules are implemented using hardware, such as any one or more of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or discrete logical circuits. The modules correspond to the steps in the methods illustrated in <FIG>.

A focal point trigger <NUM> corresponds to step <NUM>. A user instructor <NUM> corresponds to step <NUM>. An image obtainer <NUM> corresponds to step <NUM>. A position determiner <NUM> corresponds to step <NUM>. A completeness evaluator <NUM> corresponds to step <NUM>. A transformation calculator <NUM> corresponds to step <NUM>.

<FIG> shows one example of a computer program product <NUM> comprising computer readable means. On this computer readable means, a computer program <NUM> can be stored, which computer program can cause a processor to execute a method according to embodiments described herein. In this example, the computer program product is an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. As explained above, the computer program product could also be embodied in a memory of a device, such as the computer program product <NUM> of <FIG>. While the computer program <NUM> is here schematically shown as a track on the depicted optical disk, the computer program can be stored in any way which is suitable for the computer program product, such as a removable solid-state memory, e.g. a Universal Serial Bus (USB) drive.

Claim 1:
A method for determining a transformation between a first coordinate system of an ultrasonic haptic device (<NUM>) and a second coordinate system of a visual sensor device (<NUM>), the method being performed in a transformation determiner (<NUM>) and being characterized by:
triggering (<NUM>) generation of an ultrasonic focal point (<NUM>) by the ultrasonic haptic device (<NUM>), the position of the ultrasonic focal point (<NUM>) being defined in the first coordinate system;
obtaining (<NUM>) images captured by the visual sensor device (<NUM>), the images depicting a body part (<NUM>) of a user (<NUM>), where the body part (<NUM>) is being moved through the ultrasonic focal point while the ultrasonic focal point is active;
determining (<NUM>), based on the content of the images, a position of the ultrasonic focal point in the second coordinate system when the body part (<NUM>) of the user is, within a margin of error, considered to be in the ultrasonic focal point (<NUM>), by determining when the ultrasonic focal point creates a maximum of vibrations on the body part (<NUM>);
repeating (<NUM>) the steps of triggering (<NUM>), obtaining (<NUM>) and determining (<NUM>) for at least three additional ultrasonic focal points, wherein each ultrasonic focal point is in a different location compared to the other ultrasonic focal points; and
calculating (<NUM>) a first three -dimensional coordinate system transformation between the first coordinate system and the second coordinate system based on the determined (<NUM>) positions of the ultrasonic focal points in the first coordinate system and the second coordinate system to thereby calibrate a relationship between the ultrasonic haptic device (<NUM>) and the visual sensor device (<NUM>).