Patent Description:
Haptic devices and Mixed or Virtual Reality (MR or VR) devices will be an increasing part of the future device ecosystem used by society for interpersonal communication, gaming and other applications. Mixed reality computer systems augment what a user sees in the real world with computer generated virtual objects having a pose that makes the virtual objects appear to the user as if they exist in the real world. In contrast, a virtual reality computer system generates entirely virtual world of objects, instead of overlaying the objects on the real world. Both types of systems can enable a user to interact in a seemingly real or physical way using special electronic equipment, such as a head mounted display that displays computer generated objects to a user and a haptic device that tracks movement of the user and which may provide haptic feedback to the user. The term computer generated reality (CGR) is used herein to collectively refer to MR type systems and VR type systems. Accordingly, a CGR system described herein may therefore be a MR system or a VR system unless if defined otherwise in a particular example below.

In a CGR system, a haptic device can provide haptic feedback to a user that enables the user to feel virtual objects while viewing the virtual objects through a display of a CGR device. A challenging problem with CGR systems is how to provide alignment between the coordinate system of the haptic device, such as the workspace of the device, and the coordinate system of the CGR device which can be used to display 3D computer generated objects that are generated based on camera images.

Some known mixed reality systems that integrate haptic devices in a CGR world use an external tracking system. For example, the GO TOUCH VR brand system and the Ultrahaptics brand system use the Leap Motion brand hand tracking device. The HTC Vive brand product tracks its peripheral devices and headset device using an external tracking system, which removes the need for a coordinate system alignment since the same tracking system is employed for tracking all devices. A disadvantage of the HTC Vive is that it must be used within a single room and, therefore, is not mobile beyond those confines.

A need therefore exists for providing a CGR system that provides improved automated alignment of coordinate systems of haptic and CGR devices.

<CIT> discloses mixed reality computing devices that utilize remote sensors and local sensors as input devices. In one example, a mixed reality computing device comprises an image sensor, a remote input device, a processor, and storage comprising stored instructions. The stored instructions are executable by the processor to perform object motion tracking and environmental tracking based on output from the image sensor, and in response to detecting that the remote input device is in use, adjust a parameter of the motion tracking while maintaining the environmental tracking. US patent application <CIT> is further prior art.

Some embodiments disclosed herein are directed to a second electronic device for measuring a position of a first electronic device relative to the second electronic device. The second electronic device comprises a display device, a user-facing camera, a processor, and a memory. The display device allows a user to see real-world objects through the display device. The memory stores program code that is executed by the processor to perform operations comprising capturing a digital picture of a reflection from a user's eye using the user-facing camera. The reflection from the user's eye includes a first component that is a reflection of a virtual computer-generated object displayed on the display device and a second component that is a reflection of light from a real-world object. The operations further comprise processing the digital picture to extract a measure of misalignment between the virtual computer-generated object and the real-world object. The operations further comprise, responsive to the measure of misalignment not satisfying a defined alignment rule, initiating operations to generate an updated transformation matrix.

In some further embodiments, the first electronic device includes a haptic device that is configured to perform at least one of measuring movement of the haptic device by a user and providing haptic feedback to a user. The request is transmitted to the second electronic device that includes a CGR device which is configured to display graphics as an overlay on real world objects. A measurement of the position of the haptic device is received from the CGR device. A transformation matrix is then determined for transforming a pose referenced in a first coordinate system to a pose referenced in a second coordinate system based on the position of the haptic device retrieved from the memory and based on the measurement of the position of the haptic device received from the CGR device. One of the first and second coordinate systems is used to reference a pose of one of the haptic and CGR devices and the other one of the first and second coordinate systems is used to reference a pose of the other one of the haptic and CGR devices.

Some other related embodiments are directed to a method by a second electronic device for measuring a position of a first electronic device relative to the second electronic device. The method includes receiving a request to send a measurement by the second electronic device of a position of the first electronic device. Responsive to the request, the method initiates operations for generating a measurement of the position of the first electronic device, and storing in a memory the sensor data output by a sensor that can indicate the position of the first electronic device. An acknowledgement is transmitted that indicates the sensor data has been stored. A measurement of the position of the first electronic device is generated based on the sensor data stored in the memory. The measurement of the position of the first electronic device is transmitted from the second electronic device.

Other electronic devices, servers, and corresponding methods according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such additional electronic devices, servers, and corresponding methods be included within this description and protected by the accompanying claims.

Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying drawings. In the drawings:.

Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of various present inventive concepts to those skilled in the art.

Coordinate systems of haptic and CGR devices can be aligned through use of transformation matrices, which enable transformation of the location and/or rotational angles (e.g., location and rotation of a <NUM> degree of freedom (DOF) coordinate system) of an object or device referenced in one ordinate system to another coordinate system. A transformation matrix may be generated using a sensor of a CGR device, such as a front facing camera of a MR device, to detect the position (location) of the haptic device in the coordinate system of the MR device, and the onboard sensors of the haptic device which are used to track its motion (e.g., based on the Geomagic Touch or GO TOUCH VR devices) and detect its position in the coordinate system of the haptic device. Based on the position of the haptic device in the haptic device coordinate system and the position of the haptic device in the CGR device coordinate system a transformation matrix can be generated which relates the two coordinate systems. Accuracy of the transformation matrix may be increased by taking several position measurements by the sensor in the haptic device and/or several position measurements by the sensor in the CGR device. A transformation matrix that better approximates the relation between the two coordinate systems is computed, using e.g. least squares or other regression methods.

As used herein, the term "pose" refers to the position and/or the rotational angle of a device relative to a defined coordinate system. A pose may therefore be defined based on only the multidimensional position of the device, the multidimensional rotational angles of the device, or a combination thereof. The term "position or pose" refers to position, rotational angle, or combination thereof.

As explained above, aligning the coordinate systems of two electronic devices, such as a haptic device and a CGR device, can be challenging. The challenge increases when the two devices are not synchronized in their sensor measurements, such as when the haptic device measure its position at <NUM> (e.g. as done by Geomagic Touch) and when the CGR device takes images at <NUM> (e.g., as done by Microsoft Hololens). For example, the Hololens device can take between <NUM> to <NUM> to process the image it takes of a haptic device to measure the position of the haptic device, depending on the other tasks running in the Hololens device. In order to accurately calculate transformations between the coordinate systems, a system should ensure that the timing of the measurements from both devices which are used to compute the transformation are the same, i.e. the position pairs correspond to measurements taken at the same time, or be aware of the time offset between the measurement timings.

One approach to ensuring near-simultaneous sensing of the devices' positions is to configure the CGR system to operationally synchronize the sensing operations by the devices and calculate a measurement latency at the Hololens and then find the position measurement from the haptic device that corresponds to the same time as the position measurement captured by the Hololens. This synchronized sensing approach requiring a complex synchronization operation and algorithm, is prone to synchronization errors (e.g., drift in the synchronization clocks of the devices), is subject to the deleterious effects on synchronization accuracy due to latency variability in the communication network between the devices, and requires frequent message passing between the devices related to maintaining and measuring synchronization.

Embodiments of the present disclosure are directed to providing improved alignment of coordinate systems between two electronic devices, such as a haptic device and a MR device or other CGR device. The haptic device can sense its position in its own coordinate system, and the CGR device is able to sense the position of the haptic device in the CGR device's coordinate system which may be the same as that used for a camera of the CGR device.

Some embodiments of the present disclosure can perform alignment of coordinate systems of two devices using a reduced complexity operational algorithm and reducing the need for communications between the devices. In some embodiments, a haptic device uses its relatively fast position measurement to identify when the haptic device is static, i.e. the speed of the device is below a noise level threshold, and responsive thereto sends a request to the CGR device to sense the position of the haptic device (in the CGR device coordinate system) and the haptic device itself also captures its own position (in the haptic device coordinate system). Since the haptic device is static, the position sensed by both the CGR device and haptic device is the same and, thereby, eliminates the need for the alignment operations to use any timing information of when each device completed its respective position sensing. Several sensed measurements of different static positions can be taken at each device, and used to compute the transformation between the two coordinate systems to improve alignment of the coordinate systems. As will be explained in further detail below, the haptic device can also trigger sensing of its position by itself and sensing of its position by the CGR device although the haptic device is moving, such as when haptic device determines that it has a substantially constant velocity (i.e., when it's translational and/or rotational velocity does not change more than a defined threshold amount during a time interval).

Although various embodiments are disclosed herein in the context of MR devices and other CGR devices used in combination with haptic devices, these embodiments are not limited thereto. Embodiments of the present disclosure can operate to provide alignment of coordinate systems used as references for any two types of electronic devices such as between any display device (e.g. MR device, VR device, smartphone screen) which is configured to sense the position of another device that is configured to sense its own position (e.g. a haptic device or a gaming object that may move by user, such as a game controller or a gaming sword, gun steering wheel, etc.).

<FIG> illustrates a CGR system that includes a haptic device <NUM> and a CGR device <NUM> which operate in accordance with some embodiments of the present disclosure.

Referring to <FIG>, the example CGR device <NUM> can be a MR device having a forward-facing camera <NUM> which is configured to optically sense the position of the haptic device <NUM>. The CGR device <NUM> can be configured to display graphical objects as an overlay on real-world objects that are viewable through the camera <NUM> and/or that are viewable through a see-through display (e.g., Google Glass). Graphical objects may be generated by the MR device and/or by the server <NUM>. The CGR device <NUM> may include one or more speakers <NUM>.

The haptic device <NUM> includes onboard sensors that sense the present position of the haptic device <NUM>. The haptic device <NUM> may include a haptic feedback generation device <NUM> (<FIG>) that is configured to provide haptic feedback to a user, such as force feedback and/or vibration feedback. The onboard sensors may include a motion sensor <NUM> (<FIG>), rotational and/or translational position encoders, an Infra-red (IR) positioning system, and/or other sensor configured to sense the present position or pose of the haptic device <NUM>. The haptic device <NUM> may include components of Geomagic Touch product. In one embodiment, the position or pose is measured at <NUM> using encoders in the motors of the haptic device <NUM>. The haptic device <NUM> and the CGR device <NUM> include network interface circuits, which may be configured to communicate through wired and/or wireless communication links directly with each other and/or via a server <NUM> or another device (e.g., a network router or relay).

The CGR device <NUM> may sense its present position or pose using the camera <NUM> and/or using a motion sensor, such as an Inertial Measurement Unit, configured to sense the present position or pose of the haptic device <NUM>. The CGR device <NUM> can alternatively sense the present position or pose of the haptic device <NUM> using the camera <NUM>, spaced apart RF transmitters and receivers (e.g., Ultra-Wide Band or Wi-Fi radios) configured to perform triangulation of RF signals transmitted to and/or received from the haptic device <NUM>, and/or using another sensor configured to sense the present position or pose of the haptic device <NUM>. The CGR device <NUM> may include components of the Microsoft Hololens product, such as its camera and communication circuitry.

In the description of example embodiments below, the position of the haptic device <NUM> in homogeneous coordinates is referred to as P, where P = [p <NUM>]T ([. ]T represents the transpose of a matrix [. The term p is a vector and p = [X,Y,Z] denotes the three-dimensional (3D) position of the haptic device <NUM> in cartesian coordinates. Additionally, the term S(P) is defined as a set of all recorded haptic device <NUM> positions in the coordinate system of the haptic device <NUM>. The position of the haptic device <NUM> relative to a coordinate system of the CGR device <NUM> is referred to as PCGR, in homogenous coordinates, where PCGR = [pCGR <NUM>]T and pCGR = [XCGR,YCGR,ZCGR] is in cartesian coordinates. The set of positions of the CGR device <NUM> is SCGR(PCGR). The set of each pair of haptic device positions sensed by both the haptic device <NUM> and the CGR device <NUM> are stored in a set Q(P,PCGR).

Furthermore, the term VP refers to the translational velocity and the term Valpha refers to the rotational velocity. The term V̂(t) refers to an estimate of the velocity V, t seconds in the future, which may be estimated based on the analysis of the motion acceleration and/or previous motion patterns which have been performed by the user.

The term epsilon refers to a constant value that may be defined by the user and/or by an executable application.

In the example of <FIG>, the haptic device <NUM> is configured to sense its pose in a first <NUM> DOF coordinate system, illustrated as translational directions X', Y', Z' and rotational directions Θx', Θy', Θz', which can correspond to roll, pitch, and yaw. Similarly, the CGR device <NUM> is configured to sense the position of the haptic device <NUM> in a second <NUM> DOF coordinate system, illustrated as translational directions X, Y, Z and rotational directions Θx, Θy, Θz, which can correspond to roll, pitch, and yaw. Although, embodiments may be used with any multi-dimensional coordinate system.

In accordance with various embodiments herein, the haptic device <NUM>, the CGR device <NUM>, and/or the server <NUM> is configured to generate a transformation matrix for transforming a pose (e.g., position, rotational angle, and/or a combination thereof) referenced in one of the first and second coordinate systems to the other one of the first and second coordinate systems, and may furthermore generate another transformation matrix for transforming a pose in the other direction from the other one of the first and second coordinate systems to the one of the first and second coordinate systems. The haptic device <NUM> may generate the transformation matrix and communicate it to the CGR device <NUM>, so that the CGR device <NUM> can adjust the pose of objects it displays on a display device <NUM> (<FIG>) for a user.

For example, a virtual object can have haptic properties which are defined in the first coordinate system referenced by the haptic device <NUM>. The CGR device <NUM> may receive metadata about the virtual object, such as from the haptic device <NUM> itself and/or the server <NUM>, where the metadata may include the shape, color shading, and pose of the virtual object relative to the first coordinate system. The haptic device <NUM> can transmit the transformation matrix to the CGR device <NUM> to cause the CGR device <NUM> to transform the metadata (e.g., pose) of the virtual object from the first coordinate system of the haptic device <NUM> to the second coordinate system of the CGR device <NUM>. The CGR device <NUM> can then display the virtual object using the transformed metadata on a display device for viewing by a user. These operations enable the virtual object to be more accurately illustrated and have its haptic properties operationally tracked relative to real objects, such as to the user's fingers, hand, arm or other physical object.

The haptic properties of the virtual object may cause the CGR device <NUM> to control the haptic device <NUM> to provide haptic feedback to a user when haptic properties are satisfied, such as when a user's finger is determined to have positionally touched a surface of the virtual object. Alternatively or additionally, the haptic properties of the virtual object may cause the CGR device <NUM> to move and/or rotate the displayed virtual object or otherwise change its displayed appearance responsive to determining that the haptic properties are satisfied, such as when user's finger is determined to have positionally touched the surface of the virtual object.

Conversely, a new virtual object that is created in a second coordinate system of the CGR device <NUM> can have its metadata sent to the haptic device <NUM> to, for example, control feedback provided by the haptic device <NUM> to a user and/or to control position determination of the haptic device <NUM> relative to the new virtual object. The haptic device <NUM> uses the metadata to compute a transformation matrix that is used to transform the pose of the virtual object from the second coordinate system of the CGR device <NUM> to the first coordinate system of the haptic device <NUM>.

<FIG> illustrate a data flow diagram and flowchart of further example operations that can be performed by a haptic device <NUM> and a CGR device <NUM> which operate in accordance with some embodiments of the present disclosure. <FIG> is a block diagram of haptic device components that are configured in accordance with some other embodiments of the present disclosure, and <FIG> is a block diagram of CGR device components that may be used in the CGR device <NUM> and/or CGR device <NUM> described above in accordance with some other embodiments of the present disclosure.

Referring to <FIG> with further reference to <FIG>, which will be described in further detail below, the haptic device <NUM> controls the CGR device <NUM> to measure a position of the haptic device <NUM>. The haptic device <NUM> of <FIG> is configured to determine a transformation matrix between the first and second coordinate systems of <FIG>.

The haptic device includes a motion sensor <NUM>, network interface <NUM>, processor <NUM>, and a memory <NUM>. The motion sensor <NUM> is configured to sense motion of the haptic device <NUM>. The network interface circuit <NUM> is configured to communicate with the CGR device <NUM>, such as through direct wireless and/or wired communications, and/or with the server <NUM>. The processor <NUM> is connected to the motion sensor <NUM> and the network interface circuit <NUM>. The memory <NUM> stores program code that is executed by the processor <NUM> to perform operations that are explained below with regard to <FIG>.

The CGR device <NUM> measures a position of a haptic device <NUM> relative to the CGR device <NUM>, includes a sensor <NUM>, a network interface circuit <NUM>, processor <NUM>, and a memory <NUM>. The sensor <NUM> is configured to output sensor data that can indicate a position of the haptic device <NUM>. The network interface circuit <NUM> is configured to communicate with the haptic device <NUM>. The processor <NUM> is connected to the sensor <NUM> and the network interface circuit <NUM>. The memory <NUM> stores program code is executed by the processor <NUM> to perform operations that are also explained below with regard to <FIG>.

The haptic device may communicate <NUM> a request to pair to the CGR device <NUM>, which can responsively receive <NUM> the request. Operation <NUM> may be initiated by the user by pressing a button or by start-up of an application executed by the haptic device <NUM>. Operation <NUM> may additionally or alternatively be triggered when coordinate system alignment or updated alignment is determined to be needed, such as when triggered by a user and/or triggered automatically by the CGR device <NUM> as will be described below with regard to <FIG>. The CGR device <NUM> may use an eye facing camera to capture corneal images from the user which it analyzes to determine when there is excessive misalignment between graphics for virtual computer-generated object displayed on a display device <NUM> of the CGR device <NUM> and a real world object viewed by the user. The CGR device <NUM> can then trigger updating of the transformation matrix when there is excessive misalignment.

As will be explained in further detail below with regard to <FIG>, the corneal image can be analyzed to identify occurrence of excessive misalignment between the first and second coordinate systems, such as when a virtual object that should be aligned with a specific real object has excessive misalignment therebetween.

The sensor, e.g. the front facing camera <NUM>, may be turned off when not used to conserve power. The CGR device <NUM> may respond to the request by turning-on <NUM> the sensor that it will use to measure the position of the haptic device <NUM>. The haptic device <NUM> operates the motion sensor <NUM> to sense <NUM> its motion. Responsive to determining <NUM> that the haptic device <NUM> has a level of motion as sensed by the motion sensor <NUM> that satisfies a defined rule, the haptic device <NUM> transmits <NUM> a request for the CGR device <NUM> to measure a position of the haptic device <NUM>.

The haptic device <NUM> may determine that it's level of motion satisfies the defined rule when, for example, one or more the following conditions exist:.

In one embodiment, the epsilon value is about <NUM> or another defined threshold value. An epsilon value of <NUM> can be hard to observe since the haptic device <NUM> may experience very small motions and/or the output of the motion sensor <NUM> may be affected by noise and/or drift. Accordingly, the value of epsilon can be chosen based on a desired level of accuracy for the resulting transformation matrix used for coordinate system alignment. The haptic device position P is then stored in S and Q. The defined rule may further include determining whether the current measured position P is already in a set of recorded positions S and/or whether the current measured position has expired because of an elapsed threshold time since its determination.

Condition (b) above may be more generally advantageous since one can perform a request for a measurement in the future which reduces the latency to receive a position measurement from the CGR device <NUM>. A request for a measurement from the haptic device <NUM> to the CGR device <NUM> and in operation for capturing an image incurs a non-zero latency. The latency may be enough for a movement of the haptic device <NUM> to occur. However, using a predictive request enables computational compensation for such latency and more consistent alignment of the coordinate systems despite motion of the haptic device <NUM>.

It is positional measurement of the haptic device <NUM> by the CGR device <NUM> may be improved by limiting the measurement to occurring when haptic device <NUM> has a rotational speed that is below a defined threshold.

In an additional embodiment, the user may be requested, e.g., via the display of the CGR device <NUM>, to stop moving the haptic device <NUM> when it is recognized that a position measurement is required. Similarly, the user may be requested to move the haptic device <NUM> into a position or to a pair of different positions, which is not yet part of the set S and Q, respectively. Such guidance may be provided to the user via haptic feedback provided through the haptic device <NUM>, by audio feedback via the speaker <NUM> of the CGR device <NUM>, and/or by information displayed on the display device <NUM> of the CGR device <NUM>.

With further reference to <FIG>, when the defined rule is satisfied the haptic device <NUM> senses and stores <NUM> the position of the haptic device <NUM> in the memory <NUM>. The CGR device <NUM> receives <NUM> the request, and responsively initiate <NUM> operations for generating a measurement of the position of the haptic device <NUM>. The CGR device <NUM> stores <NUM> in the memory <NUM> sensor data that is output by the sensor, e.g., digital picture from the camera <NUM>, that can indicate the position of the haptic device <NUM>. The CGR device <NUM> then transmits <NUM> an acknowledgement indicating that the sensor data has been stored.

The haptic device <NUM> receives <NUM> the acknowledgement from the CGR device <NUM> indicating that it has stored the sensor data that can be used to measure the position of the haptic device <NUM>. The haptic device <NUM> may determine <NUM> whether an abort condition has occurred in which a level of motion of the haptic device <NUM>, which is sensed during a time interval between when the request was transmitted <NUM> to the CGR device <NUM> and when the acknowledgement was received <NUM> from the CGR device <NUM>, has ceased to satisfy the defined rule. Responsive to the abort condition occurring, the haptic device <NUM> transmits <NUM> an abort message to the CGR device <NUM> to abort operations for measuring the position of the haptic device <NUM>.

In one illustrative embodiment, when the haptic device <NUM> has not moved or otherwise has motion that satisfies the defined rule, further operations are performed to determine a transformation matrix between the first and second coordinate systems. In contrast, when the haptic device <NUM> is determined to have moved or to otherwise have had motion that did not satisfy a defined rule, a message is transmitted to the CGR device <NUM> to cancel its current measurement of the position of the haptic device <NUM>.

Let T1 be the time at which the request for a measurement was transmitted <NUM> to the CGR device <NUM> and let T2 be the time at which the ACK was received <NUM>, when the haptic device <NUM> is determined to have had motion that violated the defined rule during the time interval between T1 and T2, a message is transmitted <NUM> to the CGR device <NUM> to abort the measurement since the stored sensor data indication of the haptic device's position is no longer valid. In contrast, when the haptic device <NUM> is determined to not have had motion that violated the defined role during the time interval between T1 and T2, the haptic device <NUM> proceeds with operations to determine the transformation matrix.

The time T2-T1 may be based on or equal to Delta_N*<NUM> + Delta_C, where Delta_N is the network latency, and Delta_C is the time between when the request is transmitted <NUM> for a new image and when the image is acquired and stored in memory of the CGR device <NUM> for subsequent processing to determine the position of the haptic device <NUM>. The Delta_C may be defined based on the camera frame rate (e.g. <NUM>), and the network latency may be about <NUM> to about <NUM> depending on the network communication protocol and pathway used for communications whether communications are delayed due to sharing of communication resources with other devices and/or other applications on the same devices. Hence T2-T1 may be about <NUM>, which is then the acceptable duration for which the haptic device should not move or should not otherwise have motion that exceeds the defined rule.

For the case where a server <NUM> is used to collect the measured position from both the haptic device <NUM> in the CGR device <NUM> and to perform operations to determine the transformation matrix used for coordinate system alignment transformations, if a message from the haptic device <NUM> is received with information that the haptic device <NUM> has moved, Delta_B seconds after receiving the ACK from the CGR device <NUM>, where Delta_B is higher than the maximum admissible Delta_N, then the server <NUM> transmits a message to the CGR device <NUM> to abort its collecting the measured positions and performing operations to determine the transformation matrix.

With further reference to <FIG>, the CGR device <NUM> determines <NUM> whether an abort message has been received that indicates that a level of motion of the haptic device <NUM>, which is sensed during a time interval between when the request was received <NUM> and when the acknowledgement was transmitted <NUM>, has ceased to satisfy the defined rule. Responsive to no abort message having been received, the CGR device <NUM> completes generation <NUM> of a measurement of the position of the haptic device <NUM> based on the sensor data stored in the memory, and transmits <NUM> the measurement of the position of the haptic device <NUM>. Accordingly, transmission of the measurement of the position of the haptic device <NUM> is performed responsive to the determination <NUM> being that no abort message has been received.

In one embodiment, the CGR device <NUM> uses a position estimation algorithm, such as a vision-based position identification operation the processes a digital photo of the haptic device <NUM> by the camera <NUM> stored <NUM> in the memory, to estimate the position of the haptic device <NUM> in the second coordinate system of the CGR device <NUM>. The acknowledgement (ACK), which indicates that the digital photo has been captured, is sent <NUM> to the haptic device <NUM>. The position estimation algorithm continues to be performed to estimate the position of the haptic device <NUM> using the stored digital photo, although the acknowledgment message is transmitted <NUM> as soon as it is determined that the digital photo is stored in memory.

The vision-based position estimation algorithm may determine the position based on identifying one or more markers connected to or visible on the haptic device <NUM>, a portion of the housing of the haptic device <NUM>, and/or another visually identifiable feature of the haptic device <NUM>. The algorithm may be based on an algorithm used by the Hololens product, which uses the HololensARToolKit that takes between <NUM> and <NUM> from capturing an digital picture to outputting the position estimate of a marker attached to the haptic device <NUM>. The vision-based position estimation algorithms may alternatively or additionally be based on one or more algorithms provided through the Open Source Computer Vision Library (OpenCV).

In some other embodiments, the position of the haptic device is estimated using spaced apart RF transmitters and receivers (e.g., Ultra-Wide Band or Wi-Fi radios) configured to perform triangulation of RF signals transmitted to and/or received from the haptic device <NUM>.

The haptic device <NUM> receives <NUM> the measurement of the position of the haptic device <NUM> which is referenced in the second coordinate system PAR of the CGR device <NUM>. The measured haptic device position in the second coordinate system PCGR can be stored in SCGR and Q.

The haptic device <NUM> responsively determines <NUM> a transformation matrix for transforming a pose (i.e., position, rotational angle, and/or combination thereof) referenced in one of the first and second coordinate systems to a pose that is referenced in the other one of the first and second coordinate systems, based on the position of the haptic device <NUM> retrieved from the memory of haptic device <NUM> and based on the position of the haptic device <NUM> received from the CGR device <NUM>.

In one embodiment, when the number of the linearly independent position vectors in set Q is larger than N, i.e. rank(Q)>=N, the calculation of the transformation matrix between the first coordinate system of the haptic device <NUM> and the second coordinate system of the CGR device <NUM> can be performed. For the calculation of the transformation matrix to be performed for both position and orientation, the number of linearly independent vectors in the set Q is larger than N >= <NUM> (which is the minimum number of parameters to be identified in the transformation matrix). The calculation of the transformation matrix may be performed via least-squares operations where the operations find the transform T, which relates the two coordinate systems as PCGR = T*P, where PCGR = [pCGR <NUM>]T, P = [p <NUM>]T, and T = <MAT>, where <MAT> with <MAT>, and t = [t<NUM>, t<NUM>, t<NUM>] with <MAT>. Let M = [P(<NUM>)T; P(<NUM>)T;. ; P(n)T] and MCGR = [PCGR(<NUM>)T; PCGR(<NUM>)T;. ; PCGR(n)T] be a column matrix composed the pose measurements P and PCGR for all measurements i in the set Q, respectively. The transform T is obtained as the transform which minimizes ∥M*TT - MCGR∥. Other methods such as Direct Linear Transformation (DLT) may also be used to compute the transformation matrix.

The transformation matrix may be used <NUM> to control the haptic device <NUM> and/or can be sent to the CGR device <NUM> and/or to the server <NUM>. For example, the transformation matrix may be used to transform a pose of the haptic device <NUM> from one of the first and second coordinate systems to the other one of the first and second coordinate systems, to generate a transformed pose. Generation of haptic feedback by the haptic device <NUM> may be controlled based on the transformed pose. Alternatively or additionally, the CGR device <NUM> may be controlled <NUM> based on the transformation matrix, such as by displaying on a display device a graphical object with a pose that is determined based on the transformation matrix.

In some embodiments, the operations for determining <NUM> that the level of motion sensed by the motion sensor <NUM> satisfies the defined rule, such as when the haptic device <NUM> remains stationary during a time interval between when the request was transmitted <NUM> (time T1) to the CGR device <NUM> and when the acknowledgement was received <NUM> (time T2) from the CGR device <NUM>.

In contrast, in some other embodiments, the defined rule is satisfied when the haptic device <NUM> remains stationary or has a velocity that does not change more than a defined threshold amount during the time interval between T1 and T2. The operation for estimating an updated position of the haptic device <NUM> is then performed based on an amount that the position of the haptic device <NUM> retrieved from the memory <NUM> is determined to have changed due to the velocity of the haptic device <NUM> over at least a portion of the time interval between T1 and T2. The haptic device <NUM> receives <NUM> a measurement of the position of the haptic device <NUM> from the CGR device <NUM>. The haptic device <NUM> can then determine <NUM> a transformation matrix for transforming a pose referenced in one of the first and second coordinate systems to a pose referenced in the other one of the first and second coordinate systems based on the updated position of the haptic device <NUM> and based on the measurement of the position of the haptic device <NUM> received from the CGR device <NUM>.

<FIG> and <FIG> illustrate a data flow diagram and flowchart of operations by a haptic device <NUM>, a server <NUM>, and a CGR device <NUM> which operate in accordance with some embodiments of the present disclosure. In the illustrated operations, the server <NUM> generates the transformation matrix between the first and second coordinate systems of the haptic device <NUM> and the CGR device <NUM>.

Referring to <FIG> and <FIG>, the haptic device <NUM> generates <NUM> a pair request that is communicated to the CGR device <NUM> for receipt <NUM>, and which may be relayed <NUM> through the server <NUM>. The haptic device <NUM> senses <NUM> its motion and waits for a determination <NUM> that the level of motion satisfies a defined rule, such as by being substantially motionless or having an acceptable constant velocity. Responsive to that determination, the haptic device <NUM> transmits <NUM> a request to the CGR device <NUM> to send a measurement of the haptic device position to the server <NUM>. The CGR device <NUM> receives <NUM> the request for measurement, which may be forwarded or otherwise observed <NUM> by the server <NUM>.

The haptic device <NUM> senses <NUM> its current position, e.g., via motion sensor <NUM>, and transmits <NUM> the sensed position to the server <NUM>. The server initiates <NUM> operations for generation of a transformation metrics responsive to observing <NUM> the request or responsive to receiving <NUM> the measurement from the haptic device <NUM>. The CGR device <NUM> responds to the received <NUM> request by initiating <NUM> operations to generate a measurement of the haptic device position, and stores <NUM> sensor data that is output by the sensor, e.g., a digital picture from the camera <NUM>, that can be used by the CGR device <NUM> or the server <NUM> to determine the haptic device position. Responsive to storing the sensor data, the CGR device <NUM> transmits <NUM> an acknowledgment for receipt <NUM> by the haptic device <NUM>. The server <NUM> may forward or otherwise observe <NUM> the acknowledgment.

The CGR device <NUM> transmits <NUM> a measurement of the haptic device position to the server <NUM>, which is received <NUM> by the server <NUM>. The haptic device <NUM> determines <NUM>, either before or after receiving <NUM> the haptic device position measurement, whether it moved during the time interval between when the request was transmitted <NUM> (time T1) and when the acknowledgment was received <NUM> (time T2), or otherwise had motion during the time interval that did not satisfy a defined rule (e.g., having substantially constant translational and/or rotational velocity that satisfied the defined rule).

When the haptic device <NUM> had motion that did not satisfy the defined rule, an abort message may be transmitted <NUM> to the server <NUM> and, when received <NUM> by the server <NUM>, triggers the server <NUM> to abort generation of the transformation matrix based on the position measurement by the haptic device <NUM> and the position measurement by the CGR device <NUM>. In contrast, when the haptic device <NUM> remained static or otherwise had motion that satisfied the defined rule, the server <NUM> completes generation <NUM> of the transformation matrix between the first and second coordinate systems.

The transformation matrix can then be used to control <NUM> generation of haptic feedback by the haptic device. Alternatively or additionally, the transformation matrix can be used to control <NUM> display on a display device of the CGR device <NUM> of a graphical object with a pose is determined based on the transformation matrix. For example, the CGR device <NUM> may use the transformation matrix to manipulate a graphical object that is displayed, such as by positioning, rotating, adjusting color shading of the object, and/or adjusting a shape of the object based on processing metrics of the object through the transformation matrix.

<FIG> and <FIG> illustrate another data flow diagram and flowchart of operations by a haptic device <NUM>, a server <NUM>, and a CGR device <NUM> which operate in accordance with some other embodiments of the present disclosure. In the illustrated operations, the CGR device <NUM> and/or the server <NUM> generates the transformation matrix between the first and second coordinate systems of the haptic device <NUM> and the CGR device <NUM>.

Referring to <FIG> and <FIG>, the haptic device <NUM> generates <NUM> a pair request that is communicated to the CGR device <NUM> for receipt <NUM>, and which may be relayed <NUM> through the server <NUM>. The haptic device <NUM> senses <NUM> its motion, and waits for a determination <NUM> that the level of motion satisfies a defined rule, such as by being substantially motionless (static) or having an acceptable constant velocity. Responsive to that determination, the haptic device <NUM> transmits <NUM> a request to the CGR device <NUM> to generate a measurement of the haptic device position. The CGR device <NUM> receives <NUM> the request for measurement, which may be forwarded or otherwise observed <NUM> by the server <NUM>.

The haptic device <NUM> senses <NUM> its current position, e.g., via the motion sensor <NUM>, and transmits <NUM> the sensed position to the server <NUM> and/or to the CGR device <NUM>. The CGR device <NUM> responds to the received <NUM> request by initiating <NUM> operations to generate a measurement of the haptic device position, and stores <NUM> sensor data that is output by the sensor, e.g., a digital picture from the camera <NUM>, that can be used by the CGR device <NUM> or the server <NUM> to determine the haptic device position. Responsive to storing the sensor data, the CGR device <NUM> transmits <NUM> an acknowledgment for receipt <NUM> by the haptic device <NUM>. The server <NUM> may forward or otherwise observe <NUM> the acknowledgment.

The haptic device <NUM> determines <NUM>, whether it moved during the time interval between when the request was transmitted <NUM> (time T1) and when the acknowledgment was received <NUM> (time T2), or otherwise had motion during the time interval that did not satisfy a defined rule (e.g., having substantially constant translational and/or rotational velocity that satisfied the defined rule).

When the haptic device <NUM> had motion that did not satisfy the defined rule, an abort message may be transmitted <NUM> to the CGR device <NUM> and/or the server <NUM>, which may forward <NUM> the report message. If the abort message is received, it triggers the CGR device <NUM> to abort <NUM> generation of the measurement of the haptic device position. In contrast, when the haptic device <NUM> remained static or otherwise had motion that satisfied the defined rule, the CGR device <NUM> does not receive the abort message and therefore completes generation <NUM> of the measurement of the haptic device position, and may operate to transmit <NUM> the measurement to the server for receipt <NUM>.

The CGR device <NUM> may receive <NUM> the position of the haptic device <NUM> that is measured by the haptic device <NUM>, and determine the transformation matrix between the first and second coordinate systems. The CGR device <NUM> may use the transformation matrix to control display <NUM> of a graphical object with a pose that is determined based on the transformation matrix. For example, the CGR device <NUM> may manipulate a graphical object that is displayed, such as by positioning, rotating, adjusting color shading of the object, and/or adjusting a shape of the object based on processing metrics of the object through the transformation matrix.

Alternatively or additionally, the server <NUM> may receive <NUM> the measurement of the haptic device position, and determine <NUM> the transformation matrix between the first and second coordinate systems. The transformation matrix can then be used to control <NUM> generation of haptic feedback by the haptic device. Alternatively or additionally, the transformation matrix generated by the server <NUM> can be used to control <NUM> display on a display device of the CGR device <NUM> of a graphical object with a pose is determined based on that transformation matrix.

<FIG> illustrates another type of CGR device <NUM> (e.g., Google Glass) having a user facing camera <NUM> which displays computer-generated objects on a display device <NUM> and which allows a user to see the real-world objects through the display device <NUM> and/or to see the real-world objects outside an area of the display device <NUM>, in accordance with some embodiments of the present disclosure. A digital image from the user facing camera <NUM> can be processed to automatically determine when an updated transformation matrix needs to be generated, and responsively initiate generation thereof. <FIG> illustrates a flowchart of related operations by the CGR device <NUM> to initiate generation of an updated transformation matrix in accordance with some embodiments of the present disclosure.

Referring to <FIG>, the CGR device <NUM> operates the user facing camera <NUM> to capture <NUM> a digital picture of a reflection from the user's eye <NUM>. The reflection from the user's eye <NUM> includes a first component that is a reflection of a virtual computer-generated object displayed on the display device <NUM> and a second component that is a reflection of light from a real world object. The CGR device <NUM> processes <NUM> the image to extract a measure of misalignment between the virtual computer-generated object and the real world object. The CGR device <NUM> responds to the measure of misalignment not satisfying a defined alignment rule, by initiating <NUM> operations to generate an updated transformation matrix, such as the operations shown in <FIG>, <FIG>, and/or <FIG>.

Some or all operations described above as being performed by the haptic device <NUM>, the server <NUM>, and/or the CGR device <NUM> may alternatively be performed by another node that is part of a network operator cloud computing resource. For example, those operations can be performed as a network function that is close to the edge, such as in a cloud server or a cloud resource of a telecommunications network operator, e.g., in a CloudRAN or a core network.

<FIG> is a block diagram of components of a haptic device <NUM> that are configured in accordance with some other embodiments of the present disclosure. The haptic device <NUM> can include a motion sensor <NUM>, a network interface circuit <NUM>, at least one processor circuit <NUM> (processor), and at least one memory <NUM> (memory). The motion sensor <NUM> may include an inertial measurement unit. The network interface circuit <NUM> is configured to communicate with another electronic device through a wired (e.g., ethernet, USB, etc.) and/or wireless (e.g., Wi-Fi, Bluetooth, cellular, etc.) network interface. The haptic device <NUM> may further include a haptic feedback generation device <NUM> that is configured to provide haptic feedback to a user, such as force feedback and/or vibration. The processor <NUM> is connected to the motion sensor <NUM>, the network interface circuit <NUM>, the haptic feedback generation device <NUM>, and the memory <NUM>. The memory <NUM> stores program code that is executed by the processor <NUM> to perform operations. The processor <NUM> may include one or more data processing circuits, such as a general purpose and/or special purpose processor (e.g., microprocessor and/or digital signal processor), which may be collocated or distributed across one or more data networks. The processor <NUM> is configured to execute computer program instructions among program code <NUM> in the memory <NUM>, described below as a computer readable medium, to perform some or all of the operations and methods for one or more of the embodiments disclosed herein for a haptic device <NUM>.

<FIG> is a block diagram of CGR device components <NUM> that may be used in the CGR devices <NUM> and/or <NUM> described above, and which operate according to at least some embodiments of the present disclosure. The CGR device components <NUM> can include a motion sensor <NUM>, a network interface circuit <NUM>, a speaker <NUM>, at least one processor circuit <NUM> (processor), a display device <NUM>, a front facing camera <NUM>, the user facing camera <NUM>, and at least one memory <NUM> (memory). The motion sensor <NUM> may include an inertial measurement unit. The network interface circuit <NUM> is configured to communicate with another electronic device through a wired (e.g., ethernet, USB, etc.) and/or wireless (e.g., Wi-Fi, Bluetooth, cellular, etc.) network interface. The processor <NUM> is connected to the motion sensor <NUM>, the display device <NUM>, the front facing camera <NUM>, the user facing camera <NUM>, the speaker <NUM>, the network interface <NUM>, and the memory <NUM>. The memory <NUM> stores program code that is executed by the processor <NUM> to perform operations. The processor <NUM> may include one or more data processing circuits, such as a general purpose and/or special purpose processor (e.g., microprocessor and/or digital signal processor), which may be collocated or distributed across one or more data networks. The processor <NUM> is configured to execute computer program instructions among program code <NUM> in the memory <NUM>, described below as a computer readable medium, to perform some or all of the operations and methods for one or more of the embodiments disclosed herein for a CGR device <NUM>.

<FIG> is a block diagram of components of a server <NUM> that are configured in accordance with some other embodiments of the present disclosure. The server <NUM> can include a network interface circuit <NUM>, at least one processor circuit <NUM> (processor), and at least one memory <NUM> (memory). The network interface circuit and <NUM> is configured to communicate with another electronic device through a wired (e.g., ethernet, USB, etc.) and/or wireless (e.g., Wi-Fi, Bluetooth, cellular, etc.) network interface. The processor <NUM> is connected to network interface <NUM> in the memory <NUM>. The memory <NUM> stores program code <NUM> that is executed by the processor <NUM> to perform operations. The processor <NUM> may include one or more data processing circuits, such as a general purpose and/or special purpose processor (e.g., microprocessor and/or digital signal processor), which may be collocated or distributed across one or more data networks. The processor <NUM> is configured to execute computer program instructions among program code <NUM> in the memory <NUM>, described below as a computer readable medium, to perform some or all of the operations and methods for one or more of the embodiments disclosed herein for a server <NUM>.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein.

Claim 1:
A second electronic device (<NUM>) for measuring a position of a first electronic device (<NUM>) relative to the second electronic device (<NUM>), the second electronic device (<NUM>) comprising:
a display device (<NUM>) which allows a user to see real-world objects through the display device (<NUM>);
a user-facing camera (<NUM>);
a processor (<NUM>); and
a memory (<NUM>) storing program code that is executed by the processor (<NUM>) to perform operations comprising:
capturing (<NUM>) a digital picture of a reflection from a user's eye using the user-facing camera (<NUM>), wherein the reflection from the user's eye includes a first component that is a reflection of a virtual computer-generated object displayed on the display device (<NUM>) and a second component that is a reflection of light from a real-world object;
processing (<NUM>) the digital picture to extract a measure of misalignment between the virtual computer-generated object and the real-world object; and
responsive to the measure of misalignment not satisfying a defined alignment rule, initiating (<NUM>) operations to generate an updated transformation matrix.